Memory consumption

All your suggestions, requests and ideas for future development
Post Reply
devster
Posts: 388
Joined: 06 Jun 2017, 22:56

Memory consumption

Post by devster »

I'm seeing some issues with memory management on a Docker container.

Code: Select all

devster   4763 19.4  1.3 12037212 920136 ?     Sl   04:52   1:06  | java -Dapplication.deployment=deb 
--module-path /usr/share/filebot/mod --add-modules ALL-MODULE-PATH -Dnet.filebot.AcoustID.fpcalc=/usr/share/filebot/lib/fpcalc -Dunixfs=false -DuseExtendedFileAttributes=true -DuseCreationDate=false -Djava.net.useSystemProxies=true -Djna.nosys=true -Djna.nounpack=true --illegal-access=permit --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.function=ALL-UNNAMED --add-opens=java.base/java.util.regex=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.nio.file=ALL-UNNAMED --add-opens=java.base/java.nio.file.attribute=ALL-UNNAMED --add-opens=java.base/java.nio.channels=ALL-UNNAMED --add-opens=java.base/java.nio.charset=ALL-UNNAMED --add-opens=java.base/java.time=ALL-UNNAMED --add-opens=java.base/java.time.chrono=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.logging/java.util.logging=ALL-UNNAMED --add-opens=java.desktop/java.awt=ALL-UNNAMED --add-opens=java.desktop/sun.awt=ALL-UNNAMED --add-opens=java.desktop/javax.swing.text.html=ALL-UNNAMED --add-opens=java.prefs/java.util.prefs=ALL-UNNAMED 
-Djna.boot.library.path=/usr/share/filebot/lib
-Djna.library.path=/usr/share/filebot/lib
-Djava.library.path=/usr/share/filebot/lib
-Dapplication.dir=/config/.filebot
-Dapplication.cache=/config/.filebot/cache
-Djava.io.tmpdir=/config/.filebot/tmp
-Dfile.encoding=UTF-8
-Dsun.jnu.encoding=UTF-8
-Dnet.filebot.UserFiles.fileChooser=JavaFX
-DuseGVFS=true
-Dnet.filebot.gio.GVFS=/gvfs
-Xmx4g
-Dapplication.deployment=docker
-Dapplication.cache=/filebot/cache
-Dapplication.dir=/filebot
-Djava.io.tmpdir=/tmp/filebot
-Djava.library.path=/usr/lib/x86_64-linux-gnu
-Djna.library.path=/usr/lib/x86_64-linux-gnu
-Dnet.filebot.archive.extractor=ShellExecutables
-Dnet.filebot.AcoustID.fpcalc=/usr/bin/fpcalc
-Dnet.filebot.license=/filebot/license.txt
-Dnet.filebot.util.prefs.file=/filebot/prefs.properties
-Duser.home=/filebot
-jar /usr/share/filebot/jar/filebot.jar
-script fn:amc
--action keeplink
--output /cephfs/Media
--conflict skip
--filter "!readLines(/scripts/excludes.txt).contains(n)"
-non-strict
--log-file amc.log
--def excludeList=.excludes
--def ut_dir=/path/to/movies/movie.1080p.bluray.dts.x264-group.mkv ut_kind=multi ut_title=movie.1080p.bluray.dts.x264-group.mkv ut_label=movies
--def @/scripts/notify.txt
--def [email protected]/scripts/movieFormat.groovy
--def [email protected]/scripts/seriesFormat.groovy
--def [email protected]/scripts/animeFormat.groovy
I copied the output of ps that includes the entire command which got launched by my post-processing script. It shows usage of about 12 Gb of ram, despite having set -Xmx4g.

FileBot is copying the file from a mount referring to a local SSD to a ceph filesystem locally mounted.
I'm almost certain this isn't a specific FileBot issue but rather an underlying Java issue encountered in this edge case: container mount to container mount moves and possibly with the specificities of the ceph mount (although that's a kernel driver and shouldn't have issues) but I wouldn't know where to start debugging.
I only work in black and sometimes very, very dark grey. (Batman)
User avatar
rednoah
The Source
Posts: 18834
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Memory consumption

Post by rednoah »

Sorry, I'm completely lost here as well.

The only thing that I can think of, is to use VisualVM, or any other java debugging tool, to see how much memory filebot / java thinks it's using:
https://visualvm.github.io/

I'd assume when it's hogging so much memory, something has gone wrong, and it's stuck in some infinite loop allocating memory, until it's eventually crashing. A java heapdump might give some clues as to what it's doing before it crashes.
:idea: Please read the FAQ and How to Request Help.
devster
Posts: 388
Joined: 06 Jun 2017, 22:56

Re: Memory consumption

Post by devster »

It's actually not crashing, not even when it reached 30+ Gb. It just keeps doing what it's doing.
I only work in black and sometimes very, very dark grey. (Batman)
Post Reply