Removing unnecessary content in exclude list

[qvazzler]

Hello! This is my first post, please be gentle.

I have arrived at a problem where my exclude list for my fn:amc script is becoming crazy large.

I'm talking at least thousands of lines...!

Since the exclude list is very simple and doesn't contain any dates when they were added, the only optimal solution I can think of is to do a complete purge of the list every week or month. If I don't do any cleaning, the consequences will be massive (and unnecessary) slowdowns when running Filebot as often as I do.

Please let me know what I can do. I really don't want to become banned from whatever place I'll get banned from, but I also want a self-maintaining system.

Thanks in advance!

PS. I thought about purging 10-50 lines every time I run filebot, but this solution hardly feels optimal.

[rednoah]

thousands of lines is nothing. Are you sure that the exclude causes a noticeable delay in pre- and post-processing phases of the amc script? If yes, send me your exclude list (e.g. via dropbox link).

The exclude list is not necessary if you can ensure that no file is passed into the amc script more than once, including any and all corner cases and error scenarios.

EDIT: I can unexpectedly performance issues when adding 100k+ excludes.

[qvazzler]

thousands of lines is nothing. Are you sure that the exclude causes a noticeable delay in pre- and post-processing phases of the amc script? If yes, send me your exclude list (e.g. via dropbox link).

The exclude list is not necessary if you can ensure that no file is passed into the amc script more than once, including any and all corner cases and error scenarios.

EDIT: I can unexpectedly performance issues when adding 100k+ excludes.

Hi rednoah, thanks for the quick response!

I've checked again and it is 35158 lines. The reason why it is so big is because a ton of split .rar files are added to it as well.

Actually I think I am getting performance issues in many places since I don't have any torrent- or log-rotation put in place from my overhead script, but since Filebot does not print date and time in the log outputs it is hard for me to pinpoint where I get the actual slowdowns. I am working on reducing large amounts of data in all kinds of places currently, as I want a maintenance free solution. (I'm using FlexGet, FileBot, Transmission and some personal python scripts.. The whole shabang)

Do you still want the dropbox dl?

I am not 100% sure if it will check anything twice.. I mean, maybe I delete a show I no longer like, but then suddenly my gf decides that she wants to see the show and downloads it again.. The exclude-list will become problematic if that were the case, although I do have a seperate re-scan list to somewhat resolve this.

[qvazzler]

Ok, I will rephrase my request a bit.

Since the split .rar files are what's causing the concern with a crazy amount of excluded files, would it be possible for Filebot to simply skip searching them?

Then I would not need to delete anything from the exclude list.

Mind you, I'm thinking 5 years ahead here, where my exclude list will become so large it'd be crazy to leave it like that.

[rednoah]

1.
Use --script dev:amc and see if it makes a difference. On may 2012 MBP it takes 2s to load 1.000.000 lines and that takes about 200 MB memory.

2.
The exclude lists will grow indefinitely, but with modern hardware you should be good for at least 10 million entries. That being said, the exclude list mechanism is just there for the noobs to not break everything for everyone.

3.
Do a proper rtorrent setup. No need for the exclude list if your torrent client passes only newly downloaded folders/files to filebot.

@see http://www.filebot.net/forums/viewtopic ... =215#p5316

[qvazzler]

1.
Use --script dev:amc and see if it makes a difference. On may 2012 MBP it takes 2s to load 1.000.000 lines and that takes about 200 MB memory.

2.
The exclude lists will grow indefinitely, but with modern hardware you should be good for at least 10 million entries. That being said, the exclude list mechanism is just there for the noobs to not break everything for everyone.

3.
Do a proper rtorrent setup. No need for the exclude list if your torrent client passes only newly downloaded folders/files to filebot.

@see http://www.filebot.net/forums/viewtopic ... =215#p5316

Thank you for the info about the entries, very appreciated.

I have a very elaborate script running when a torrent is done. Filebot is only executed on new downloads. The way you're putting it, you mean I could possibly skip the exclude list if my filebot execution is optimal?

[rednoah]

Yes, in an optimal setup the torrent client would call filebot on exactly once and only the files that have been completed.

If you happen to download the same files twice half a year apart then it naturally doesn't matter that it's processed "twice". Just don't process everything over and over every day. That's what the exclude lists prevents regardless setup issues.

[qvazzler]

Yes, in an optimal setup the torrent client would call filebot on exactly once and only the files that have been completed.

If you happen to download the same files twice half a year apart then it naturally doesn't matter that it's processed "twice". Just don't process everything over and over every day. That's what the exclude lists prevents regardless setup issues.

May I ask, out of sheer curiosity, what is it that causes strain on your services for using Filebot on the same file multiple times?

The only performance drain I see is that your script is downloaded remotely, but surely the TV/Movie db's belong to other groups/companies?

After what you're telling me I feel somewhat better, although I do have a script that currently runs every midnight to double-check if any files have not been processed for some reason. Maybe I could still use this feature but extend it to a bi-weekly basis?

[rednoah]

1)
FileBot only uses regularly updated & heavily cached static resources. Nothing short of a DDOS attack will strain that.

2)
External services like TheTVDB / TheMovieDB / AcoustID / OpenSubtitles / etc are a lot more complicated and do need significant resources to keep running. Stupid noobs calling filebot on their 10.000 movies collection every day for potentially years on end doesn't help.

FileBot does A LOT of caching, and any data is cached for at least 2 days. But requesting (most likely identical, but possibly updated) over and over indefinitely because of some badly conceived automated setup (i.e. with deluge, transmission, etc) must not be allowed and certainly not the default.

[qvazzler]

I understand your frustration.

I will do what I can to avoid any kind of hammering.