FileBot Running Very Slow on Large Libraries—Anyone Else?

[gordonpiper]

I’ve noticed FileBot gets really slow when I try to process large media libraries (thousands of files). It sometimes freezes or takes a very long time to complete tasks.
Is this a common issue?
Does splitting tasks into smaller batches help, or is there a better solution?

[rednoah]

What does System Information say?

Are you using the GUI or the CLI?

What specifically is slow? Loading files? Matching files? Renaming files? Does the UI freeze or become sluggish when scrolling through the list of loaded / matched files?

How many files? 10.000 files? 100.000 files? Which database are you using? How may TV Shows / Episodes (1 API request per series) or Movies (1 API request per movie) roughly? You may get rate-limited by the API if you hammer, i.e. smooth UI / no UI freezes but a progress wheel that continues to spin while requests are happing (and possibly failing with a wait-for-retry timeout) in the background.

Are the files on a fast local drive or a slow network drive?

Are you hitting the memory limit?

Does the log say anything?






EDIT:

I did a quick local test with 1000 movies. 0 byte files for testing. All in the same folder. All using Name (Year) naming. Matching against TheMovieDB took less than a minute. I noticed no issues when scrolling through the list after matching. Memory Usage was at 200 MB / 400 MB at the end. Matching the same files again (no API requests thanks to caching) took around 2 seconds. Running the same test with 10.000 movies I noticed no UI sluggishness. Matching proportionally took a few minutes. Memory Usage was 400 MB / 800 MB.

That can serve as a baseline for future testing, if you get different results, when can then look into what's different and narrow down on the culprit(s) from there.

[gordonpiper]

I'm using the GUI.
What's slow: The slowdown mainly happens during matching and renaming. The UI can become sluggish, especially when scrolling through long lists after files have been loaded.
Library size: Typically between 8,000–12,000 files at once. These are a mix of movies and TV shows. Since TV shows require per-series API calls, I suspect that's contributing to the slowdown.
Memory: I haven’t seen out-of-memory errors, but usage spikes significantly during processing.

[rednoah]

Can you copy & paste the System Information output? This will tell us the memory limit on your particular system, and lots of additional context that may yield additional clues:




What does Memory Usage say when it's being sluggish? This will tell us if you're hitting the memory limit:




Since it's a lot of files, can we assume that files are located on a slow remote network drive?


Since you process movies and tv shows, can we assume that you load in all the files and then click on Smart Mode: Automatic to match all of them?

Since TV shows require per-series API calls, I suspect that's contributing to the slowdown.

Since many files can be processed using the episode list information from a single API call, the API is actually less of a bottleneck when processing TV shows with many episodes. If the UI is sluggish just because many files have been loaded in, while matching is already complete and no longer hogging CPU resources, then that would suggest a memory problem or a programming problem. Since I cannot seem to reproduce the issue with 10k files loaded and matched here on my end, memory is the likely culprit at this point.

Can you confirm that the UI is / or is not sluggish when scrolling after just loading in all your files? Or does the UI work smoothly at first but then become sluggish after matching?

[rednoah]

Can you copy & paste the System Information output? This will tell us the memory limit on your particular system, and lots of additional context that may yield additional clues:

I'd really need the System Information output at the very least to continue this topic. I'd be very interested in figuring out whats going on here, but since I cannot reproduce the issue on my end, that will need a bit of back-and-forth to figure out whats going on on your end.