Filebot-node docker stuck for hours

[FabulousTiger]

I seem to be having a problem with filebot-node starting out fine but after a a while it seemingly stops and sometimes it gets killed and doesn't start again. I thought it might have been that I ran the crontab schedule too often with every two hours. So I increased it to 4 hours, but then I thought of if there is a way to run it only if it's not running. I found "run-one" which I hope did something but it is still killing itself presumably because it hangs for too long. I really enjoyed the interface so it would be a shame if I would have too try a different approach. I liked that I had quick (if not annoying to have too scroll down to see end) access to the log at the same place as I can tweak and monitor the progress. So any tips would be much appreciated.

[rednoah]

Note that FileBot Node merely runs filebot commands, and that Schedule merely creates a shell script that you can run which calls filebot with predefined arguments. The components are completely independent. FileBot Node just uses the "filebot" command, which could be your shell script of your own as far as FileBot Node is aware. The filebot command just runs and is not at all aware of FileBot Node. Since FileBot Node and filebot are completely separate and independent components, you'll want to starting with figuring out what exactly is stuck and where exactly it is stuck. FileBot Node itself most likely is not stuck, assuming that FileBot Node is still serving the logs every second, which just so happen to not change for some reason.

You'll want to check the ps -A process table to see what's going on inside the container. The filebot commands generated by FileBot Node call filebot with a specific --log-file path, and each new filebot instance will require an exclusive read/write lock to that log file. So if you run filebot many times, only one filebot instance will actually be running and processing files at any given time, and so if you're doing something unusual (calling filebot many times in rapid succession; filebot taking an unexpectedly long time to process files; etc) you could theoretically end up with a large queue of filebot instances waiting to run.

How long does each filebot call take? (typically; a few seconds; perhaps minutes at most; so not usually an issue if filebot is called once per day)

[FabulousTiger]

I saw in the console that the process was killed because of lack of memory so I have now 4x it from 512mb to 2048mb might increase it if it requires it but I want to stay within reason. I also have moved the the original files to a new completed folder when I am satisfied they have been hardlinked too remove a lot of skips as there are a few extras(as extras doesn't get added to the exclude list) that I skip as well as "Exclude superfluous link" which I think is smart I don't really know what it really does. But now I have paused the container to figure out if its easier to have it on my truenas scale file server as it would talk better maybe with the files and have network delay and finish quicker maybe. As well as trying to perfect my format to have less mass redo's to get it consistent. What is expected ram usage for filebot amc script with say 100 files, 1000 files and 10000 files? Of course not linear but a ballpark guess, of what I can expect.

[rednoah]

I saw in the console that the process was killed because of lack of memory

What did the console say? Did FileBot crash? Did the OS kill the FileBot process?

What was FileBot doing when it was killed? What did the console output before the process ended?

512 is plenty. 10000 isn't an unreasonable large number of files, at least for a local file system. Presumably, they're not all in the same folder, so you could easily process files one folder at a time, if "all at once" isn't feasible for one reason or another.

Note that you not post command & console output so I don't actually have any clues as to what your use case might be, so my ability to give good advice is somewhat limited.

[FabulousTiger]

I didn't take note of exactly what but it was on the lines of "java process nr x killed because of memory" something around that so yeah the OS killed the process. (i should have grabbed the console output for later context yeah)

It happened multiple times as it was ruining the cronjob every 4 hours to search the media folder for newly added files older than 3 hours. By lookin at the log i count 37 killed counts and it varies what happened right before but a few examples

Auto-detect movie from context X
Group: {Series=X}
Fetching episode data for [X]
Failed to process X because X is an exact copy and already exists (which is annoying that it is not in the exclusion, but that might be from the first few tests that did not have exclusions)

I am moving it to the same vm as the files are housed on so that its "local" and not over SMB don't know if it will help. And I have disabled the "skip superfluous links" incase it would help if a file is linked. I have also noticed that "ingore extra" takes a long time multiple minutes per skip.

Console Output: Select all

Run script [fn:amc] at [Fri Jul 14 10:00:06 GMT 2023]

[PSA] Important Discussion of Proposed Changes:
https://www.filebot.net/forums/viewtopic.php?t=13406

Parameter: excludeLink = y
Parameter: artwork = y
Parameter: skipExtract = y
Parameter: minFileAge = 0.125
Parameter: storeReport = /mnt/hydra-media/Appdata/filebot/reports/
Parameter: seriesFormat = {vf =~ /2160p/ ? '4k ' : ''}Tv Shows/{ ~plex.id % { " {edition-$edition}" } }{director ? ' (' + director + ')' : ''} {allOf{hdr}{vf}{vcf}{ac}{channels}.join(' ')}
Parameter: movieFormat = /mnt/hydra-media/{fn.match(/Open.Matte/) ?  '/Open matte/ ' : "" }/{vf =~ /2160p/ ? '4k ' : ''}Movies/{ ~plex.id % { " {edition-$edition}" } % {fn.match(/Open.Matte/) ? ' (edition-' + {fn.match(/Open.Matte/)} + ')' : '' } }{director ? ' (' + director + ')' : ''} {allOf{hdr}{vf}{vcf}{ac}{channels}.join(' ')}
Parameter: movieDB = TheMovieDB
Parameter: seriesDB = TheMovieDB::TV
Parameter: excludeList = .excludes
Argument[0]: /mnt/hydra-media/Downloads/Media
Use excludes: /mnt/hydra-media/.excludes (1650)

Was the config from the log.

[rednoah]

What's the filebot command?

Shell: Select all

filebot -script fn:amc ...

The number of files is fairly irrelevant if you're filtering out the old ones before processing anyway.

You can manually generate an exclude list (i.e. all files you have in your input folder right now; assumed to be processed already) with the find command, it's just a list of file paths after all:

Shell: Select all

find /path/to/input > excludes.txt

--def excludeLink=y means that you want to ignore files which have more than 1 hard link, i.e. only process files that have not yet been hardlinked. If you're using a remote file system then the link count may or may not be reported correctly.

If you have extra video files, trailers, samples, etc, then FileBot may use libmediainfo or ffprobe to check the video duration, i.e. partially read the file which can be slow on a remote file system. Please read Optimizations for Remote File Systems for details.




EDIT:

I didn't take note of exactly what but it was on the lines of "java process nr x killed because of memory" something around that so yeah the OS killed the process.

Notably, java has its own memory limits that are set via the -Xmx option. See filebot -script fn:sysinfo output for details. If you have memory limits for the container, then you want to ensure that the java process running within that container is configured to work within those limits.

[FabulousTiger]

Shell: Select all

/opt/filebot-node/task 21

Shell: Select all

filebot -script 'fn:amc' /mnt/hydra-media/Downloads/Media/ --output /mnt/hydra-media --action duplicate -non-strict --order Airdate --conflict auto --lang en --def 'excludeLink=y' 'artwork=y' 'skipExtract=y' 'minFileAge=0.125' 'storeReport=/mnt/hydra-media/Appdata/filebot/reports/' 'seriesFormat={vf =~ /2160p/ ? '\''4k '\'' : '\'\''}Tv Shows/{ ~plex.id % { " {edition-$edition}" } }{director ? '\'' ('\'' + director + '\'')'\'' : '\'\''} {allOf{hdr}{vf}{vcf}{ac}{channels}.join('\'' '\'')}' 'movieFormat=/mnt/hydra-media/{fn.match(/Open.Matte/) ?  '\''/Open matte/ '\'' : "" }/{vf =~ /2160p/ ? '\''4k '\'' : '\'\''}Movies/{ ~plex.id % { " {edition-$edition}" } % {fn.match(/Open.Matte/) ? '\'' (edition-'\'' + {fn.match(/Open.Matte/)} + '\'')'\'' : '\'\'' } }{director ? '\'' ('\'' + director + '\'')'\'' : '\'\''} {allOf{hdr}{vf}{vcf}{ac}{channels}.join('\'' '\'')}' 'movieDB=TheMovieDB' 'seriesDB=TheMovieDB::TV' 'excludeList=.excludes' --apply metadata refresh --log all --log-file '/data/filebot/node/filebot.log'

That is what I used, I should have figured you would want that line instead. But I have now moved it to the truenas VM, and it seems to perform better. That could also be because I changed some proxmox settings to improve nested virtualization. As you can see, I had an exclusion file, but it does not exclude extras in the folders, so it took a long time to go through them every time as I skip the extras. If it starts to slow down again I guess I have to use the -no-probe option. Would that still skip extras and or have false positives?

[rednoah]

I guess I have to use the -no-probe option. Would that still skip extras and or have false positives?

It'll work. Notably, it'll incorrectly ignore files like "Trailer Park Boys 1x01.mkv" as an extra:
viewtopic.php?t=12986

That said, mediainfo is 2-3 orders of magnitude faster on local file systems, maybe ~50ms per file, so FileBot can easily check the video length for a few hundred files in seconds.

And if you somehow do have a large number of files you want to preemptively exclude via the exclude list text file, then you can just add those paths once in a while:

Shell: Select all

filebot -find /input --filter "f.video && minutes < 15" >> excludes.txt

[FabulousTiger]

I will try that exclude less than 15min video and maybe implement it into a schedule once a month or week. Is there a test I can do to see if it understands that it is using a local mnt? Like if I can run a test to see how long time it would take to run mediainfo on the files. It should be local now but as it's all virtual I idk might get confused.

[rednoah]

Use -mediainfo --format to test custom bindings on all your files:

Shell: Select all

filebot -mediainfo -r /path/to/files --format "{vc} | {ac} | {channels} | {resolution} | {duration} | {f}"

Add time and do a test run:

Shell: Select all

time filebot -mediainfo -r /path/to/files --format "{vc} | {ac} | {channels} | {resolution} | {duration} | {f}"

filebot itself has some overhead of course, but that'll become fairly negligible if your sample file set a good number of files.

It should be local now but as it's all virtual I idk might get confused.

If the file system is local, then IO operations will be near instant. I reckon that docker mounts add little to no overhead.

You might be confused by mv doing a slow copy+delete instead of an instant atomic-move if you're moving files from one volume mount to another, even if both volume mounts happen to be on the same file system from the host point of view. But that's got nothing to do with disk latency / throughput.