many hanging tasks and instances concerning filebot

[DarthWeber]

Is there any reason why the scripts cummulates many many instances of ffilebot ? They are filling my process list and i need to kill them manually or reboot my system. After that filebot is running well and even doen't hang.

I am calling filebot from jdownloader btw.

[rednoah]

What does the log say? Where does filebot hang according to the log?


What filebot -script fn:sysinfo say? Are you running out of memory?


How long does each filebot call take? How many filebot instances per minute are you calling?


If you call many filebot instances at the same time, then they will run in sequence. Each new filebot instance will wait for the previous filebot instance to complete. If each filebot call takes a very long time, perhaps due to network issues, perhaps due to being low on memory, perhaps due to some othe reason, then new filebot calls will queue up.


If you have very-low end hardware (i.e. not enough RAM) that is suddenly faced with a very heavy workload (i.e. hundreds of filebot calls at the same time) then that might bring the system to it's knees as it starts swapping application memory from RAM to disk. In this case, you could use flock to enforce one-filebot-call-at-a-time on your end to make things more memory-efficient.

[DarthWeber]

What does the log say? Where does filebot hang according to the log?

It seems not to be touched as the last entry is

Code: Select all

"Done ヾ(＠⌒ー⌒＠)ノ"

from 9th of july. But there are several tasks hanging again:

Code: Select all

32317 ?        Sl     0:34 java -Dapplication.deployment=deb --module-path /usr/share/openjfx/lib --add-modules ALL-MODULE-PATH -Djna.boot.library.name=jnidispatch.system -Dnet.filebot.archive.extractor=ShellExecutables --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.function=ALL-UNNAMED --add-opens=java.base/java.util.regex=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.nio.file=ALL-UNNAMED --add-opens=java.base/java.nio.file.attribute=ALL-UNNAMED --add-opens=java.base/java.nio.channels=ALL-UNNAMED --add-opens=java.base/java.nio.charset=ALL-UNNAMED --add-opens=java.base/java.time=ALL-UNNAMED --add-opens=java.base/java.time.chrono=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.text=ALL-UNNAMED --add-opens=java.base/sun.nio.fs=ALL-UNNAMED --add-opens=java.logging/java.util.logging=ALL-UNNAMED --add-opens=java.desktop/java.awt=ALL-UNNAMED --add-opens=java.desktop/sun.awt=ALL-UNNAMED --add-opens=java.desktop/sun.swing=ALL-UNNAMED --add-opens=java.desktop/javax.swing.text.html=ALL-UNNAMED --add-opens=java.prefs/java.util.prefs=ALL-UNNAMED -XX:+DisableAttachMechanism -Djna.boot.library.path=/usr/lib/x86_64-linux-gnu/jni -Djna.library.path=/usr/lib/x86_64-linux-gnu/jni -Djava.library.path=/usr/lib/x86_64-linux-gnu/jni -Dapplication.dir=/root/.filebot -Dapplication.cache=/root/.filebot/cache -Djava.io.tmpdir=/root/.filebot/tmp -Dfile.encoding=UTF-8 -Dsun.jnu.encoding=UTF-8 -Dawt.useSystemAAFontSettings=on -Dprism.order=sw -Dnet.filebot.theme=Darcula -DuseGVFS=true -Dnet.filebot.gio.GVFS=/gvfs -jar /usr/share/filebot/jar/filebot.jar -script fn:amc --lang de --output /volume1/media --log-file /volume1/media/amc.log --action move /backup/jdownloads --def seriesFormat=Serien/{n}/{s}/{n}.{s00e00} - {t} ({y}) {vf} movieFormat=Video/{n} ({y}) {source}({vf}@{vc}) [{rating}]/{n} ({y}) {source}({vf}@{vc}) [{rating}] animeFormat= unsortedFormat=Unsortiert/{file.structurePathTail} -non-strict --def unsorted=y artwork=y --def clean=y --def exec=chmod -R 777 "{output}/" ; chown -R avahi:avahi "{output}/"

What filebot -script fn:sysinfo say? Are you running out of memory?

Code: Select all

FileBot 4.9.6 (r9125)
JNA Native: 5.2.0
MediaInfo: 17.12
Tools: fpcalc/1.4.3 7z/16.02 unrar/5.50 mkvpropedit/19.0.0
Extended Attributes: OK
Unicode Filesystem: OK
Script Bundle: 2022-06-28 (r831)
Groovy: 3.0.9
JRE: OpenJDK Runtime Environment 11.0.15
JVM: OpenJDK 64-Bit Server VM
CPU/MEM: 4 Core / 2.0 GB Max Memory / 35 MB Used Memory
OS: Linux (amd64)
HW: Linux r2d2 4.15.0-176-generic #185-Ubuntu SMP Tue Mar 29 17:40:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
CPU/MEM: Intel(R) Celeron(R) J4105 CPU @ 1.50GHz [MemTotal: 7 GB | MemFree: 135 MB | MemAvailable: 3.8 GB | SwapCached: 11 MB | SwapTotal: 4.3 GB]
STORAGE: ext4 [/] @ 123 GB | ext4 [/volume1] @ 1.7 TB | fuse [/mnt/webdav/GymWild] @ 6 TB
USER: root
DATA: /root/.filebot
Package: DEB
Activate License [PX21881167] on [Tue Jul 12 06:05:28 CEST 2022]
License: FileBot License PX21881167 (Valid-Until: 2071-01-04)
Done ヾ(＠⌒ー⌒＠)ノ

How long does each filebot call take? How many filebot instances per minute are you calling?

I am calling from jdownload

If you call many filebot instances at the same time, then they will run in sequence. Each new filebot instance will wait for the previous filebot instance to complete. If each filebot call takes a very long time, perhaps due to network issues, perhaps due to being low on memory, perhaps due to some othe reason, then new filebot calls will queue up.


If you have very-low end hardware (i.e. not enough RAM) that is suddenly faced with a very heavy workload (i.e. hundreds of filebot calls at the same time) then that might bring the system to it's knees as it starts swapping application memory from RAM to disk. In this case, you could use flock to enforce one-filebot-call-at-a-time on your end to make things more memory-efficient.

If i kill all filebot processes and java processes like above and start filebot manually: it takes long time (some couple of minutes), but it works. So i think it must have to do something with my call from jdownloader. I am calling it with the event handler from jdownloader like:

Code: Select all

[{"eventTrigger":"ON_ARCHIVE_EXTRACTED", "eventTriggerSettings":{}, "id":1576604595593, "name":"FileBot", "script":"var amcFile = \"/opt/filebot/filebot.sh\";var path = archive.getFolder(); callAsync(function() {}, amcFile, path);", "enabled":true}]

which may cause to many lineups. Is there any way to call it from jdownloader ?

[rednoah]

The log would indicate that no filebot process is currently locking the log file and doing things and writing things to the log. That would be most strange. The file lock on the log file. Note that if any of these filebot processes where to be locking the log file, then you wouldn't be able to run the same filebot command with the same log file either.


Memory looks fine. Plenty of free RAM available. You'll want to check if there's any available RAM at the point in time where you have lots of filebot processes and the system starts stalling though. 4 GB is plenty, but if you accidentally call filebot dozens of times per second then you will run into bottlenecks in no time.


Note that FileBot shouldn't take multiple minutes. Which log line is followed by multiple minutes of nothing? That might be a hint.


What does your /opt/filebot/filebot.sh custom shell script do? You'll want to name your custom script /path/to/jdownloader-postprocess.sh so that you don't confuse your own filebot.sh entry point script with the filebot launcher script.


Note that your JD event script is also very different from the ArchiveExtracted.js sample code. Your code is likely calling your filebot.sh entry point script much much much more often (dozens of times per completed download, in rapid succession, repeatedly with the same input arguments) than you think. You'll want to test your assumptions there.


The ps output above also seems to indicate that you're running everything as root which is generally a bad idea. If you accidentally DoS attack your own system by spawning lots of processes, then you definitely don't wanna do so as root, because root processes probably get priority over everything else.


You'll want to update your entry point script and add additional logging, e.g. print time, arguments, process id, etc and generally redirect all console output to a file (different file for each call) so you can get more information for what's going on with each command, and track each time your script is called by JD.


You can do kill -3 <pid> to make a java process dump all stack traces to console output. That will tell you exactly what that process is currently doing and might give us a clue as to where it's stuck.

[DarthWeber]

Could you please give me a hint how to install the ArchiveExtracted.js ? Is there eny doc about ist ?

[rednoah]

You already set up an event script, no? How did you set up your current event script?

Code: Select all

[{"eventTrigger":"ON_ARCHIVE_EXTRACTED", "eventTriggerSettings":{}, "id":1576604595593, "name":"FileBot", "script":"var amcFile = \"/opt/filebot/filebot.sh\";var path = archive.getFolder(); callAsync(function() {}, amcFile, path);", "enabled":true}]

This is the event script you have currently configured:

Code: Select all

var amcFile = "/opt/filebot/filebot.sh";var path = archive.getFolder(); callAsync(function() {}, amcFile, path);

Maybe via JD Desktop application? Maybe via JD WebUI? Maybe by writing the JSON configuration file directly? Since you did it before, I'm assuming that you can do it again. Please read [JDownloader] Setup for Windows, Linux and Mac OS X for what we have on the topic. If you need click-by-click instructions on how to configure JD event scripts though, you may need to ask the JD community for guidance.

You may edit the configuration files directly if you are running JDownloader on a headless server:
org.jdownloader.extensions.eventscripter.EventScripterExtension.scripts.json


Known Issues:

• The PackageFinished script is called before extraction is complete. The ArchiveExtracted script is called for each individual archive. In order to avoid FileBot processing partially extracted files it is recommended that extraction is disabled in JDownloader and enabled in FileBot via the --def skipExtract=n option.