java.lang.OutOfMemoryError: Java heap space

Support for Synology NAS, QNAP NAS and other Embedded Linux systems
Post Reply
otr
Posts: 7
Joined: 06 Aug 2018, 20:37

java.lang.OutOfMemoryError: Java heap space

Post by otr » 06 Aug 2018, 21:13

Hi,

I am running into issues with Java heap space. This is very possibly due to me being new at everything Linux/Synology.
My setup is a "JDtoFilebot" script that I call from the command line (for now, will figure out the JDEventScripter later).

This setup does work, generally speaking. However, I am getting Java heap space errors in two scenarios:
1) When I run this on a lot of files (8 upwards) -> Workaround: Process less files at once (but is that best practice?).
2) When I run this on one specific file. -> No workaround.

I am posting my console output below. Also, I would like to understand: Is it better to run filebot many times (one file each time), or should filebot be able to handle 100+ files at each go and serialize them internally?

My script

Code: Select all

LOG="/volume1/Downloads/amc.log"
EXC="/volume1/Downloads/amc.txt"
SERIEN="/volume1/Media/{Plex}"
FILME="/volume1/Media/{Plex}"
/var/packages/filebot/target/filebot.sh -script fn:amc --lang en --log-file $LOG --action move "$1" --def "seriesFormat=$SERIEN" "movieFormat=$FILME" "animeFormat=$ANIME" "unsortedFormat=$UNSORTIERT" -non-strict --def unsorted=y "artwork=y" --def clean=y
/var/packages/filebot/target/filebot.sh -script fn:cleaner "$1" --def root=y
My call:

Code: Select all

sh jdtofilebot.sh /volume1/Downloads/B/
My call, trying to increase Java memory - however I have no idea whether that can even work this way.

Code: Select all

sh jdtofilebot.sh /volume1/Downloads/B/ -xmx300M
Sysenv: Pastebin -script fn:sysenv

Console Output:

Code: Select all

sh jdtofilebot.sh /volume1/Downloads/B/
Locking /volume1/Downloads/amc.log
Run script [fn:amc] at [Mon Aug 06 23:06:44 CEST 2018]
Parameter: seriesFormat = /volume1/Media/{Plex}
Parameter: movieFormat = /volume1/Media/{Plex}
Parameter: animeFormat =
Parameter: unsortedFormat =
Parameter: unsorted = y
Parameter: artwork = y
Parameter: clean = y
Argument[0]: /volume1/Downloads/B
Input: /volume1/Downloads/B/DDLValley.me_The.15.17.to.Paris.2018.BDRip.x264-GECKOS.mkv
Aug 06, 2018 11:06:50 PM net.sf.ehcache.store.disk.DiskStorageFactory <init>
WARNING: The index for data file /var/packages/filebot/target/data/.../cache/0/data_0.data is out of date, probably due to an unclean shutdown. Deleting index file /var/packages/filebot/target/data/.../cache/0/data_0.index
Java heap space
java.lang.OutOfMemoryError: Java heap space
        at org.tukaani.xz.ArrayCache.getByteArray(Unknown Source)
        at org.tukaani.xz.lz.LZDecoder.<init>(Unknown Source)
        at org.tukaani.xz.LZMA2InputStream.<init>(Unknown Source)
        at org.tukaani.xz.LZMA2Decoder.getInputStream(Unknown Source)
        at org.tukaani.xz.BlockInputStream.<init>(Unknown Source)
  at org.tukaani.xz.SingleXZInputStream.read(Unknown Source)
        at org.tukaani.xz.XZInputStream.read(Unknown Source)
        at net.filebot.util.ByteBufferOutputStream.transferFrom(ByteBufferOutputStream.java:102)
        at net.filebot.util.ByteBufferOutputStream.transferFully(ByteBufferOutputStream.java:112)
        at net.filebot.util.ByteBufferOutputStream.transferFully(ByteBufferOutputStream.java:106)
        at net.filebot.CachedResource.lambda$getBytes$3(CachedResource.java:154)
        at net.filebot.CachedResource$$Lambda$28/963269035.transform(Unknown Source)
        at net.filebot.CachedResource.lambda$get$1(CachedResource.java:99)
        at net.filebot.CachedResource$$Lambda$31/517210187.apply(Unknown Source)
        at net.filebot.Cache.computeIf(Cache.java:90)
        at net.filebot.CachedResource.get(CachedResource.java:82)
        at net.filebot.media.ReleaseInfo.lambda$resource$19(ReleaseInfo.java:490)
        at net.filebot.media.ReleaseInfo$$Lambda$38/914507705.get(Unknown Source)
        at net.filebot.TransformedResource.get(Resource.java:58)
        at net.filebot.MemoizedResource.get(Resource.java:36)
        at net.filebot.media.ReleaseInfo.getSeriesMappings(ReleaseInfo.java:368)
        at net.filebot.media.MediaDetection.matchSeriesMappings(MediaDetection.java:467)
        at net.filebot.media.MediaDetection.detectSeriesNames(MediaDetection.java:353)
        at net.filebot.media.MediaDetection.detectSeriesNames(MediaDetection.java:323)
        at net.filebot.cli.ScriptShellBaseClass.detectSeriesName(ScriptShellBaseClass.java:261)
        at net.filebot.cli.ScriptShellBaseClass.detectSeriesName(ScriptShellBaseClass.java:249)

Failure (°_°)
Done ヾ(@⌒ー⌒@)ノ

Thanks!

User avatar
rednoah
The Source
Posts: 16092
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Java Heap Space - certain files only

Post by rednoah » 07 Aug 2018, 06:40

1.
Please run filebot -script fn:sysinfo and post the output. That'll tell us how much memory the java process can work with on your device.


2.
The -Xmx limit should be set to 70% of physical memory by default on Synology NAS devices:

Code: Select all

export JAVA_OPTS=`free | awk -vm=1024 -vp=0.7 '/Mem:/ {printf "-Xmx%dm", ($2*p)/m; exit}'` # set -Xmx to 0.7 of physical memory
:idea: Please read the FAQ and How to Request Help.

otr
Posts: 7
Joined: 06 Aug 2018, 20:37

Re: java.lang.OutOfMemoryError: Java heap space

Post by otr » 07 Aug 2018, 06:53

Ah, I had somehow confused sysenv and sysinfo. Here you go:

Code: Select all

filebot -script fn:sysinfo
FileBot 4.8.2 (r5768)
JNA Native: 5.2.2
MediaInfo: net.filebot.mediainfo.MediaInfoException: Unable to load aarch64 (64-bit) native library libmediainfo.so: Unable to load library 'zen': Native library (linux-aarch64/libzen.so) not found in resource path ([file:/volume1/@appstore/filebot/jar/filebot.jar])
Apache Commons VFS: [zip, rar]
Chromaprint: java.io.IOException: Cannot run program "fpcalc": error=2, No such file or directory
ffprobe: java.io.IOException: Cannot run program "/volume1/@appstore/MediaServer/bin/ffprobe": error=2, No such file or directory
Extended Attributes: OK
Unicode Filesystem: OK
Script Bundle: 2018-07-15 (r530)
Groovy: 2.5.0
JRE: Java(TM) SE Runtime Environment 1.8.0_181
JVM: 64-bit Java HotSpot(TM) 64-Bit Server VM
CPU/MEM: 4 Core / 156 MB Max Memory / 14 MB Used Memory
OS: Linux (aarch64)
HW: Linux Syn... 4.4.59+ #23739 SMP Tue Jul 3 19:47:55 CST 2018 aarch64 GNU/Linux synology_rtd1296_ds218play
DATA: /volume1/@appstore/filebot/data/...
Package: SPK
License: FileBot License ... (Valid-Until: 2019-07-30)
Done ヾ(@⌒ー⌒@)ノ

User avatar
rednoah
The Source
Posts: 16092
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: java.lang.OutOfMemoryError: Java heap space

Post by rednoah » 07 Aug 2018, 06:57

DiskStation DS218play should have 1 GB of memory:
https://www.synology.com/en-global/prod ... play#specs

What do these commands say on your device:

Code: Select all

free

Code: Select all

free | awk -vm=1024 -vp=0.7 '/Mem:/ {printf "-Xmx%dm", ($2*p)/m; exit}'
:idea: Please read the FAQ and How to Request Help.

otr
Posts: 7
Joined: 06 Aug 2018, 20:37

Re: java.lang.OutOfMemoryError: Java heap space

Post by otr » 07 Aug 2018, 07:03

Code: Select all

 free
              total        used        free      shared  buff/cache   available
Mem:         658464      227588      212136        1784      218740      204184
Swap:       2097084      217116     1879968
and

Code: Select all

free | awk -vm=1024 -vp=0.7 '/Mem:/ {printf "-Xmx%dm", ($2*p)/m; exit}'
-Xmx450m

User avatar
rednoah
The Source
Posts: 16092
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: java.lang.OutOfMemoryError: Java heap space

Post by rednoah » 07 Aug 2018, 07:06

Looks like the 70% memory limit is set by FileBot Node by default, but if you're using filebot directly then the choice is left to you.


Here's how you can correctly set the memory limit on your device:

Code: Select all

export JAVA_OPTS="-Xmx512m"
filebot -script fn:sysinfo

:idea: How come your device only has 650 MB of memory, and not 1 GB of memory as per hardware specification? Strange indeed! Still plenty though to run FileBot on heavy workloads.
:idea: Please read the FAQ and How to Request Help.

otr
Posts: 7
Joined: 06 Aug 2018, 20:37

Re: java.lang.OutOfMemoryError: Java heap space

Post by otr » 07 Aug 2018, 07:15

Thanks, I will inquire about the memory in the Synology forum /with their support.

About Filebot / Filebot Node: My goal is to use Filebot Node (via JD's EventScripter). Will the snippet you gave above

Code: Select all

export JAVA_OPTS="-Xmx512m"
work for Filebot Node as well? If not, how do I do that?

I can also report that the "tricky" file from before has been processed now, with the higher memory setting.

User avatar
rednoah
The Source
Posts: 16092
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: java.lang.OutOfMemoryError: Java heap space

Post by rednoah » 07 Aug 2018, 07:20

1.
FileBot Node will set JAVA_OPTS to 450 MB automatically before calling filebot, so you won't need to worry about anything.


2.
FileBot Node and JD aren't really things you can hook up to each other, as either of them can be used to make filebot calls directly. So you'll be using either FileBot Node / Synology Scheduler or JD (but not both) to automate filebot calls.
:idea: Please read the FAQ and How to Request Help.

otr
Posts: 7
Joined: 06 Aug 2018, 20:37

Re: java.lang.OutOfMemoryError: Java heap space

Post by otr » 07 Aug 2018, 07:38

Ahh, just got it - my statement above was wrong.
Filebot node is the GUI, and I will not use that. My goal is to use filebot via the EventScripter. Now, for that one, do I need to set the memory manually frequently (e.g. after every reboot)? Or will that setting stay?

By the way, the low memory seems to be due to the system reserving memory for the hardware transcoding functionality:

Transcoding enabled:

Code: Select all

free
              total        used        free      shared  buff/cache   available
Mem:         658464      186500      196340        1432      275624      245488
Swap:       2097084      262664     1834420
Transcoding disabled:

Code: Select all

free
              total        used        free      shared  buff/cache   available
Mem:         883744      322308      127716       25932      433720      305300
Swap:       2097084           0     2097084

User avatar
rednoah
The Source
Posts: 16092
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: java.lang.OutOfMemoryError: Java heap space

Post by rednoah » 07 Aug 2018, 08:08

1.
If you're using the JD setup instructions here:
viewtopic.php?f=4&t=3478

Then you'll need to set your memory limit in jdownloader-postprocess.sh before the filebot command:

Code: Select all

#!/bin/sh -xu
export JAVA_OPTS="-Xmx512m"
...

2.
Modifying /etc/profile and thus setting JAVA_OPTS system-wide is an option too, but DSM will probably override your changes with each update, so it's best to just set JAVA_OPTS before each filebot call.
:idea: Please read the FAQ and How to Request Help.

Post Reply