HELP: Filebot 4.9.4 AniDB Caching different from 4.9.3 for Groovy Script!

Any questions? Need some help?
Post Reply
rv23
Posts: 64
Joined: 19 Jul 2020, 15:05

HELP: Filebot 4.9.4 AniDB Caching different from 4.9.3 for Groovy Script!

Post by rv23 »

I switched my extensive Anime renaming script from 4.9.3 r8311 to 4.9.4 r8736 and the # of times I am banned by AniDB has significantly increased.
With 4.9.4 r8736 I got banned 19 times out of 76 runs over 3 months.
with 4.9.3 r8311 I got banned 0 times out of 146 runs over 6 months.

As there are lots of variables involved, I decided to start with asking for assistance with how can I update my script to account for the backend changes either with how Filebot is handling the AniDB caching, or changes to the methods I am using (some of which are not documented at https://www.filebot.net/docs/api/index.html).

The methods I'm using that I *think* might deal with AniDB lookups are;

Code: Select all

WebServices.AniDB.search
WebServices.AniDb.getSeriesInfo
WebServices.AniDb.getEpisodeList
WebServices.AnimeList.model.anime
and of course rename

I don't think I have any configuration or are sending any command that might be changing/clearing cache.

I have recently switched back to 4.9.3 r8311, but would appreciate any assistance in using 4.9.4 r8736 with the same level of robustness (for my script) that I was seeing with r8311.

Thanks.
rv23
Posts: 64
Joined: 19 Jul 2020, 15:05

Re: HELP: Filebot 4.9.4 AniDB Caching different from 4.9.3 for Groovy Script!

Post by rv23 »

oh, I didn't include how I "know" if I've been banned by AniDB. I am using this try/catch block..

Code: Select all

	try {
        rfsTemp = rename(*:wrapperArgs)
        if (rfsTemp) {
          rfs += rfsTemp
        }
      } catch (Exception IllegalStateException) {
        println 'AniDB BanHammer Detected. Please stop hitting AniDB for at least 24 hours'
        aniDBBanHammer = true as Boolean
        rfsIncomplete = false as Boolean
        rfs = []
      }
User avatar
rednoah
The Source
Posts: 22898
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: HELP: Filebot 4.9.4 AniDB Caching different from 4.9.3 for Groovy Script!

Post by rednoah »

No idea. Pretty sure the haven't been any relevant direct or indirect changes between those revisions in particular. You can enable debug logging and track requests to confirm that FileBot is or is not caching API requests correctly.


:idea: The underlying code has not be changed:
https://pastebin.com/A2PNcHCb


:idea: Note that killing the filebot process may result in the cache not shutting down cleaning, and thus initializing from scratch the next time filebot is called.


:idea: The filebot process does count requests internally and apply rate limiting preemptively. However, there is no persistence, and so it only works if everything is done in a single filebot process. This filebot process also won't be able track other machines or processes accessing AniDB services from the same public IP.


:!: catch (Exception e) doesn't tell you anything. You'll want to add at least println e so you can actually see the exception you have caught, which may or may not be related to AniDB rate limiting. Your code just so happens to print "AniDB BanHammer Detected" for any and all errors, which is going to be very misleading for all the possible errors except the one. ;)


EDIT:

Did your catch block every work in the first place? rename() will catch all exceptions internally, print an error message, and then return null. So your catch block will never actually catch anything as far as I can tell.
:idea: Please read the FAQ and How to Request Help.
rv23
Posts: 64
Joined: 19 Jul 2020, 15:05

Re: HELP: Filebot 4.9.4 AniDB Caching different from 4.9.3 for Groovy Script!

Post by rv23 »

re: Catch block:
It was catching *something*, or I expect I wouldn't have seen the messages in my script output for the AniDB Ban.

Re: Caching
hmm... Your right that I should have been saving the actual exception messages as well, so I will add some additional output to my script and see if I can at least have the actual error/message that my script is actually hitting and see if that's an AniDB ban or not.

I'll post an update in a week or two (since the issue doesn't happen every time it's run) with what I have been able to find.

Thanks.
rv23
Posts: 64
Joined: 19 Jul 2020, 15:05

Re: HELP: Filebot 4.9.4 AniDB Caching different from 4.9.3 for Groovy Script!

Post by rv23 »

You are correct, the error I am getting is not related to AniDB, but a "filebot" error :)

Code: Select all

Map [Focus T25 - Special 1 - Stretch] to [...] failed: net.filebot.format.BindingException: Binding "e": undefined
renameWrapper() - Caught error:[net.filebot.cli.CmdlineException: Syntax Error: net.filebot.format.BindingException: Binding "e": undefined]
I *am* using a mapper of

Code: Select all

[order.absolute.episode.derive(e)]
DB is TheTVDB
Order is Airedate
No filter
Query is 394354
Strict is true

Is the error I got something expected given the options I am using, or should it not be throwing an error?
User avatar
rednoah
The Source
Posts: 22898
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: HELP: Filebot 4.9.4 AniDB Caching different from 4.9.3 for Groovy Script!

Post by rednoah »

How would you like to deal with special episodes?


Note that special episodes don't have an episode number "e" and, though possible, you probably don't want your mapper to turn them into regular episodes. You'll probably want to not map specials, or map them to null.


EDIT:

e.g. exclude episode object if custom mapping is invalid:

Code: Select all

any{ order.absolute.episode.derive(e) }{ null }
e.g. use original episode object if custom mapping is invalid:

Code: Select all

any{ order.absolute.episode.derive(e) }{ episode }
:idea: Please read the FAQ and How to Request Help.
rv23
Posts: 64
Joined: 19 Jul 2020, 15:05

Re: HELP: Filebot 4.9.4 AniDB Caching different from 4.9.3 for Groovy Script!

Post by rv23 »

Good question ..

I totally forgot you can use the any {} with the mapper option, I'll have to update that specific case to something that in theory should at least not cause filebot to throw out an error.
I had expected that if the mapper couldn't be used for an episode (or special in this case), it would just do nothing or skip that object vs throw an error.

Thanks for the good code suggestions (as usual).
User avatar
rednoah
The Source
Posts: 22898
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: HELP: Filebot 4.9.4 AniDB Caching different from 4.9.3 for Groovy Script!

Post by rednoah »

rv23 wrote: 02 Dec 2021, 21:15 I had expected that if the mapper couldn't be used for an episode (or special in this case), it would just do nothing or skip that object vs throw an error.
The error case is subject to change, and my change from version to version depending on the feedback and kind of issues users run into in production. So it's best to have your format be explicit about how you want to handle mapping errors.


:idea: The current code (FileBot r8952) will catch and log mapper errors. So it won't crash the process any longer, but it also means that it can act like an accidental --filter and lead to seemingly obvious mismatches if the correct episode happens to not be considered at all due to having been filtered out due to a mapper error.
:idea: Please read the FAQ and How to Request Help.
Post Reply