Limiting memory of filebot (java)?

[ittayd]

So it seems the java process for filebot uses 3.3G of my measly 1G qnap. How can I configure it to limit the memory, and what is the recommended size?

[rednoah]

What stats are you looking at? What does filebot -script fn:sysinfo say?

The memory limit is always below physically available RAM, and it can't be higher. You can configure -Xmx though. -Xmx256m or higher should be enough.

[rednoah]

You're probably using FileBot Node, which means FileBot Node will configure -Xmx to 80% of physical memory:

Code: Select all

CPU/MEM: 4 Core / 796 MB Max Memory / 21 MB Used Memory

The memory limit is 796 MB.

[ittayd]

Well, it is killing my qnap... Xmx is not really limiting the total memory of the jvm, just the memory it uses for its own heap.

Code: Select all

[~] # java -XX:+PrintFlagsFinal -Xmx796m -version | grep -Ei "maxheapsize|maxram"
   size_t MaxHeapSize                              = 834666496                                 {product} {command line}
 uint64_t MaxRAM                                   = 137438953472                           {pd product} {default}
    uintx MaxRAMFraction                           = 4                                         {product} {default}
   double MaxRAMPercentage                         = 25.000000                                 {product} {default}
   size_t SoftMaxHeapSize                          = 834666496                              {manageable} {ergonomic}

So it'll happily use 137GB of memory....

Code: Select all

[~] # java -XX:+PrintFlagsFinal -XX:MaxRAM=796m -version | grep -Ei "maxheapsize|maxram"
   size_t MaxHeapSize                              = 209715200                                 {product} {ergonomic}
 uint64_t MaxRAM                                   = 834666496                              {pd product} {command line}
    uintx MaxRAMFraction                           = 4                                         {product} {default}
   double MaxRAMPercentage                         = 25.000000                                 {product} {default}
   size_t SoftMaxHeapSize                          = 209715200                              {manageable} {ergonomic}

That is more like it...

And I think best is:

Code: Select all

[~] # java -XX:+PrintFlagsFinal -XX:MaxRAM=1g -XX:MaxRAMFraction=2 -version | grep -Ei "maxheapsize|maxram"
OpenJDK 64-Bit Server VM warning: Option MaxRAMFraction was deprecated in version 10.0 and will likely be removed in a future release.
   size_t MaxHeapSize                              = 536870912                                 {product} {ergonomic}
 uint64_t MaxRAM                                   = 1073741824                             {pd product} {command line}
    uintx MaxRAMFraction                           = 2                                         {product} {command line}
   double MaxRAMPercentage                         = 50.000000                                 {product} {default}
   size_t SoftMaxHeapSize                          = 536870912                              {manageable} {ergonomic}

which will use 50% of the available 1GB of memory

How can I control these options, other than editing the filebot script?

[ittayd]

So I tried these options and it doesn't really help. Does filebot read files into memory? That may explain how it gets to 3.5GB of memory.

[ittayd]

Seems like using XX:MaxMetaspaceSize helped the situation. Can it be that filebot includes many unrequired classes that get parsed? Or maybe static content in some classes that makes them bloat?

[rednoah]

Are you doing anything out of the ordinary the somehow nobody else is doing? Seems to work fine for mostly everyone else, so perhaps finding out what's different in your case will give as a hint as to what to do about it.

At least for filebot -script fn:sysinfo it's only using 21 MB Used Memory so I guess we could do a bit of systematic testing from there, and see if there's any particular aspect in your particular use case that breaks things.


EDIT:

So what exactly are you doing? Just a amc script call via FileBot Node? On a large set of files? What's the command-line call? What does the console log say? How far does it get before the system stalls?

[ittayd]

22 files . It goes past writing the inputs, warning the index is out of date, then stalls for a long time with high cpu and the memory jumps when it starts moving files.

I can't see how java anywhere users 21MB. Just the JVM is more. I'm using Java 14 from java-install

[ittayd]

Can something similar to this be the culprit?: https://hibernate.atlassian.net/browse/ ... Order=desc

"The method reads the bytes from the InputStream to a byte array,
however, the GZIPInputStream is never closed explicitly. From my understanding, this causes the leak, because the GZIP decompression is done via JNI. Thus, memory outside the JVM is allocated and never released."

[ittayd]

I've set every aggressive flag I could find. The process starts at 20% memory and grows to 35%. At least now it doesn't cause my entire qnap to stall, but takes forever to run (even while running interpreted, there are python applications that do similar stuff)

Code: Select all

JAVA_OPTS="-XX:+TieredCompilation -XX:TieredStopAtLevel=1 -Xint -XX:+UseSerialGC -XX:CompressedClassSpaceSize=75m -XX:MaxMetaspaceSize=75m -Xshare:off  -XX:NativeMemoryTracking=summary -XX:MaxRAM=1g -XX:MaxRAMPercentage=70"

[rednoah]

21 MB refers to the memory that FileBot is using within the JVM. The JVM itself does its own thing, but we have little to no control over, so not something you'd normally worry about. Manually tuning JVM memory and GC certainly shouldn't be necessary for even the most hardcore filebot stress tests.


The key hint we've got, isn't that something is weird, it's that something is weird exclusively on you specific device, setup, use case, set of files, etc. Because it works for me and seemingly works for everyone else. The next step would be to systematically narrow down what makes things work / not work for you.

The first step is finding a simplified example use case on your machine that works, and then gradually add complexity step by step to find out when it stops working.



Test Case #1

Code: Select all

filebot -script fn:sysinfo
filebot -clear-cache
touch Alias.1x01.avi
time filebot -rename Alias.1x01.avi -non-strict --action TEST --log INFO

Result Set #1 from my QNAP TS-228A (aarch64 with 1 GB RAM)

Code: Select all

...
CPU/MEM: 4 Core / 518 MB Max Memory / 25 MB Used Memory
OS: Linux (aarch64)
...
[TEST] from [Alias.1x01.avi] to [Alias - 1x01 - Truth Be Told.avi]

real	0m46.755s
user	1m20.938s
sys	0m0.781s

Result Set #2 from my Synology DS213J (armv7l with 512 MB RAM)

Code: Select all

...
CPU/MEM: 1 Core / 127 MB Max Memory / 24 MB Used Memory
OS: Linux (arm)
...
[TEST] from [Alias.1x01.avi] to [Alias - 1x01 - Truth Be Told.avi]

real	3m57.927s
user	3m50.890s
sys	0m1.290s

What do you get if you run the same test?