[AniDB] Client-side Limits and Server-side Bans
Posted: 22 Aug 2020, 02:54
rednoah wrote: ↑23 Jul 2020, 13:12This is a reasonable limit, but effectively indistinguishable from someone with a large A-Z library processing all their files. As for a specific HTTP request limit, I did bite the sour apple and got myself banned, and I got ~250 requests via the HTTP API before the ban hammer struck. Stopped. Tried again two days later and everything worked again. So the ban does time out if you stop hitting AniDB for a while.
FileBot r7750 implements various new behaviors and now things work as follows:
* No more than one active request at any point in time
* No more than 5 requests within a 1 minute time frame (i.e. the first 5 requests are very fast, one after another, but the 6th request will have to wait for ~60 seconds, heavily slowing down processing)
* No more than 200 request per 24 hour time frame (i.e. the first 200 requests will be relatively fast at 5 requests per 60 seconds, but if we hit the limit, then FileBot will deadlock and suspend processing for up to 24 hours)
* All information is cached for 6 days (i.e. FileBot will not request the same information twice, unless the information we have is more than 6 days old already)
* If our 200 request per 24 hour limit is 50% exhausted, then FileBot will prefer previously cached information regardless of age to conserve requests if possible (i.e. you may end up with episode data that is 2-3 months old that was cached in a previous run long ago)
* A single banned response will switch the AniDB client into offline mode for the remaining process life-time and all subsequent requests that cannot be served by the cache will fail immediately
The limits above can only be enforced within a single running filebot process. Running multiple filebot processes parallel or in sequence will effectively disable all these limits.
FileBot r7750 implements fine-grained logging and additional error messages:
Code: Select all
AniDB daily limit: FloodLimit(200)Code: Select all
AniDB minutely limit: FloodLimit(5)Code: Select all
AniDB cache expiration temporarily disabled due to heavy usage: 26: FloodLimit(50)Code: Select all
AniDB HTTP API daily limit has been exceeded. Processing will continue in 24 hours...Code: Select all
AniDB has banned your IP due to excessive usage. Please try again in 24 hours or later.Code: Select all
AniDB has already banned your IP. Please stop hitting AniDB for at least 24 hours.
Please read AniDB error: banned for a complete discussion of various issues and workarounds.