Filebot 4.7.9 on Synology ds213j DSM 6 Out of memory using amc

[qvazzler]

Hi, thanks for your great support so far!

Started noticing that my filebot script is no longer running as it should when I scan many shows at once.

I tried changing from copy action to move, and copying the exact same memory allocation profile for java as filebot's bash executable. Have also installed java via your unofficial java installer.

Any ideas? I have 512mb ram. I have raised these issues before and the things I did back then helped, but here I am again!

[rednoah]

What command are you running?

What's the log say when it crashes with out-of-memory?

What does sysinfo say?

[qvazzler]

Command:

Code: Select all

filebot -script fn:amc
 --output /share2/
 --log ALL
 --def subtitles=en
 --def seriesFormat=TV Shows/{n}/{episode.special ? 'Special' : 'Season '+s.pad(2)}/{n} - {episode.special ? 'S00E'+special.pad(2) : s00e00} - {t.replaceAll(/[\`´ââÊ»]/, /'/).replaceAll(/[!?.]+$/).replacePart(', Part $1')}{' - '+group}{'.'+lan--def movieFormat=Movies/{n} ({y})/{n} ({y}){' CD'+pi}{'.'+lang}{' - '+group}
 --def animeFormat=Anime/{primaryTitle}/{primaryTitle} - {sxe} - {t.replaceAll(/[!?.]+$/).replaceAll(/[\`´ââÊ»]/, /'/).replacePart(', Part $1')}{' - '+group}
 --def musicFormat=Music/{n}/{album+'/'}{pi.pad(2)+'. '}{artist} - {t}
 --def skipExtract=n
 --def unsorted=y
 --conflict override -non-strict
 --def skipExtract=y
 --def music=y
 --def artwork=y
 --def ut_kind=multi
 --def clean=y
 --def deleteAfterExtract=n
 --def pushbullet=[token]
 --def kodi=myhost.com,myhost2.com
 --def excludeList=/volume1/Scripts/excludelists/FileBot_ExcludeList__2017-October.txt
 --action move
 --def ut_dir=/share/Any 

Command initial output

Code: Select all

Python: 2017-10-23 16:14:02,740 torrent_finish CRITICAL error code: 1 Run script [fn:amc] at [Mon Oct 23 15:51:07 CEST 2017]
Parameter: subtitles = en
Parameter: seriesFormat = TV Shows/{n}/{episode.special ? 'Special' : 'Season '+s.pad(2)}/{n} - {episode.special ? 'S00E'+special.pad(2) : s00e00} - {t.replaceAll(/[\`´ââÊ»]/, /'/).replaceAll(/[!?.]+$/).replacePart(', Part $1')}{' - '+gParameter: movieFormat = Movies/{n} ({y})/{n} ({y}){' CD'+pi}{'.'+lang}{' - '+group}
Parameter: animeFormat = Anime/{primaryTitle}/{primaryTitle} - {sxe} - {t.replaceAll(/[!?.]+$/).replaceAll(/[\`´ââÊ»]/, /'/).replacePart(', Part $1')}{' - '+group}
Parameter: musicFormat = Music/{n}/{album+'/'}{pi.pad(2)+'. '}{artist} - {t}
Parameter: skipExtract = y
Parameter: unsorted = y
Parameter: music = y
Parameter: artwork = y
Parameter: ut_kind = multi
Parameter: clean = y
Parameter: deleteAfterExtract = n
Parameter: pushbullet = *****
Parameter: kodi = *****
Parameter: excludeList = /volume1/Scripts/excludelists/FileBot_ExcludeList__2017-October.txt
Parameter: ut_dir = /volumeUSB1/usbshare/Any
Use excludes: /volume1/Scripts/excludelists/FileBot_ExcludeList__2017-October.txt

The error:

Code: Select all

Java heap space
java.lang.OutOfMemoryError: Java heap space
        at Script1.run(Script1.groovy:318)
        at net.filebot.cli.ScriptShell.evaluate(ScriptShell.java:64)
        at net.filebot.cli.ScriptShell.runScript(ScriptShell.java:74)
        at net.filebot.cli.ArgumentProcessor.runScript(ArgumentProcessor.java:119)

Failure (°_°)

sysinfo

Code: Select all

qvazzler@x:/volume1/Scripts$ filebot -script fn:sysinfo
FileBot 4.7.9 (r4984)
JNA Native: 5.1.0
MediaInfo: 0.7.93
Apache Commons VFS: [zip, rar]
Chromaprint: 1.3.2
Extended Attributes: OK
Unicode Filesystem: OK
Script Bundle: 2017-05-15 (r500)
Groovy: 2.4.10
JRE: Java(TM) SE Runtime Environment 1.8.0_131
JVM: 32-bit Java HotSpot(TM) Client VM
CPU/MEM: 1 Core / 121 MB Max Memory / 9 MB Used Memory
OS: Linux (arm)
Package: SPK
uname: Linux BrickStation 3.2.40 #15101 Thu Apr 20 03:56:30 CST 2017 armv7l GNU/Linux synology_armada370_213j
Done ã¾(ï¼ âã¼âï¼ )ã

[qvazzler]

Ok, I thought FILEBOT_OPTS was already in effect but jokes on me, I'm not using the right version for that.

Gonna try now with

Code: Select all

JAVA_OPTS=`free | awk -vm=1024 -vp=0.7 '/Mem:/ {printf "-Xmx%dm", ($2*p)/m; exit}'`

[qvazzler]

Ok, still having the same issue with memory exceptions. Hm. Different source though.

Code: Select all

Java heap space
java.lang.OutOfMemoryError: Java heap space
        at Script1.run(Script1.groovy:318)
        at net.filebot.cli.ScriptShell.evaluate(ScriptShell.java:64)
        at net.filebot.cli.ScriptShell.runScript(ScriptShell.java:74)
        at net.filebot.cli.ArgumentProcessor.runScript(ArgumentProcessor.java:119)

Failure (°_°)

[meowmoo]

I added

Code: Select all

export JAVA_OPTS="-Xmx256m"

to my .sh file and it works fine, i got the same issue on dsm6 with a ds413j

[qvazzler]

I do have a lot of files for Filebot to process though.. Sometimes my scripts break and there's a ton of backlog that I need to run through.

I'm experimenting a bit now with a simplified command, but we'll see how that goes...

PS. There is a lot of "Ignore hidden" and "Ignore extra" messages in my log. Could that be the source of a memory leak? Feels pretty far-fetched but thought I'd mention it.

[qvazzler]

Ok, progress is better so far.. It's been going through half of the files.

I see memory usage creeping up slowly, but so far the maximum is 276 mb. It seems to climb a few mb every 2-5 minutes or so.

[rednoah]

What's the output on DSM 6 if you run this command?

Code: Select all

free | awk -vm=1024 -vp=0.7 '/Mem:/ {printf "-Xmx%dm", ($2*p)/m; exit}'

[qvazzler]

-Xmx348m

I am trying that now actually, unsure if it somehow didn't get picked up before. It seems to work better now, and the memory actually got garbage collected. Still not done yet, we'll see what happens.

[rednoah]

You can use sysinfo calls to check if max. memory is getting picked up or not. 300+ MB should be more than enough to process very large amounts of files.

[qvazzler]

After a long time (and 2 retries), filebot managed to work through 350+ entries.

Thanks for the help!