[AniDB] Client-side Limits and Server-side Bans

[rednoah]

Anti Leech Protection
The API should not be used to "download" AniDB. If such attempts are detected you will get banned."

This is a reasonable limit, but effectively indistinguishable from someone with a large A-Z library processing all their files. As for a specific HTTP request limit, I did bite the sour apple and got myself banned, and I got ~250 requests via the HTTP API before the ban hammer struck. Stopped. Tried again two days later and everything worked again. So the ban does time out if you stop hitting AniDB for a while.


FileBot r7750 implements various new behaviors and now things work as follows:
* No more than one active request at any point in time
* No more than 5 requests within a 1 minute time frame (i.e. the first 5 requests are very fast, one after another, but the 6th request will have to wait for ~60 seconds, heavily slowing down processing)
* No more than 200 request per 24 hour time frame (i.e. the first 200 requests will be relatively fast at 5 requests per 60 seconds, but if we hit the limit, then FileBot will deadlock and suspend processing for up to 24 hours)
* All information is cached for 6 days (i.e. FileBot will not request the same information twice, unless the information we have is more than 6 days old already)
* If our 200 request per 24 hour limit is 50% exhausted, then FileBot will prefer previously cached information regardless of age to conserve requests if possible (i.e. you may end up with episode data that is 2-3 months old that was cached in a previous run long ago)
* A single banned response will switch the AniDB client into offline mode for the remaining process life-time and all subsequent requests that cannot be served by the cache will fail immediately

The limits above can only be enforced within a single running filebot process. Running multiple filebot processes parallel or in sequence will effectively disable all these limits.

FileBot r7750 implements fine-grained logging and additional error messages:

Code: Select all

AniDB daily limit: FloodLimit(200)

Code: Select all

AniDB minutely limit: FloodLimit(5)

Code: Select all

AniDB cache expiration temporarily disabled due to heavy usage: 26: FloodLimit(50)

Code: Select all

AniDB HTTP API daily limit has been exceeded. Processing will continue in 24 hours...

Code: Select all

AniDB has banned your IP due to excessive usage. Please try again in 24 hours or later.

Code: Select all

AniDB has already banned your IP. Please stop hitting AniDB for at least 24 hours.

Please read AniDB error: banned for a complete discussion of various issues and workarounds.

[rv23]

Awesome, Thanks for both adding the additional error message logging and detailing it specifically here (I've bookmarked this info now)

Do I need to set a specific log level or will they be logged by the default log level?

[rednoah]

If the log message is relevant to the user, then the default log level will be enough. The log messages that accompany every single request will be logged only if debug logging is enabled though.