WARNING: AniDB error: banned

[Fribb]

SysInfo:

FileBot 4.9.1 (r7372)
JNA Native: 6.1.0
MediaInfo: 19.09
7-Zip-JBinding: 9.20
Chromaprint: 1.4.3
Extended Attributes: OK
Unicode Filesystem: OK
Script Bundle: 2020-06-18 (r655)
Groovy: 3.0.3
JRE: OpenJDK Runtime Environment 14
JVM: 64-bit OpenJDK 64-Bit Server VM
CPU/MEM: 12 Core / 4.3 GB Max Memory / 68 MB Used Memory
OS: Windows 10 (amd64)
STORAGE: NTFS [(C:)] @ 171 GB | NTFS [Games] @ 320 GB | NTFS [Data] @ 995 GB
DATA: C:\Users\Fribb\AppData\Roaming\FileBot
Package: MSI
License: FileBot License PX11041992 (Valid-Until: 2070-01-03)

Saturday I started working on my AniDB renaming again when I got the AniDB error: banned Warning message.

While looking for a solution I noticed this forum post which told me that apparently my IP got banned because of too many requests. I also have seen that there is some code that prevents too many requests in a short amount of time, IIRC it was 2 requests in 5 seconds?! But apparently this either does not work correctly or there is some issue that lets you do more requests than you should be able to.

[rednoah]

AFAIK, AniDB has a daily limit of ~100 ~200 requests (i.e. episode listings for ~100 ~200 shows) per IP per day, so if you run FileBot against a large library, you will hit the limit. The "2 requests in 5 seconds" flood limit will space requests out a little bit, just to be nice to the server, but you'll still hit the daily limit eventually.


Depending on your files, TheTVDB with Absolute Episode Order might be more suitable for the task at hand.


EDIT:

Newer revisions use a flood limit of "1 requests in 5 seconds" so you'll hit the daily limit in ~10 ~20 minutes if you run FileBot against a large library.

[Fribb]

Is that somewhere listed in Filebot itself? Because the daily cap is something new to me (and I have renamed a lot more than 200 the weeks before)

Maybe a Daily counter would be a good way to make this more visible

For the TheTVDB option, that is not really an option for me. Even AniDB is just a workaround since most of my Media file actually should be in a MyAnimeList format but since there is no renamer for this (except the mess of a CLI program I wrote) filebot with AniDB is a better solution.

[rednoah]

Is that somewhere listed in Filebot itself? Because the daily cap is something new to me (and I have renamed a lot more than 200 the weeks before)

We don't know what the cap is. The ~100 ~200 numbers is just a random guess. We only know that it's relatively low. If you process 800 episodes of One Piece, then you will never run into the issue because it's just 1 episode list request. If you process 500 different Anime shows, with only one episode each, then you will almost certainly hit the limit. Also, the limit may change over time. Also, the limit may change every time you hit the limit. Also, FileBot does cache requests, so if you process the same Anime shows over and over every day, then FileBot will not actually request anything most of the time.

Maybe a Daily counter would be a good way to make this more visible

At best, the FileBot process can track it's own requests, and that's where "1 requests in 5 seconds" comes from, but has no way of tracking request count to AniDB from your public IP. There could be other filebot instances on other machines on your local network. There could be other tools talking to AniDB and using up requests.


If you must use AniDB, then I'd recommend processing less than 100 Anime shows per day. AFAIK, the limit is reset every day, so you can play it and see where the limit is.

[Fribb]

If you must use AniDB, then I'd recommend processing less than 100 Anime shows per day. AFAIK, the limit is reset every day, so you can play it and see where the limit is.

Alright, but for that, I need to get unbanned first or get a new IP which, so far, hasn't worked. Thanks.

[rednoah]

Alright, but for that, I need to get unbanned first or get a new IP which, so far, hasn't worked. Thanks.

Well, you can always use your phone as hotspot and viola.

But remember that the "daily limit" again is pure conjecture. If you repeatedly run into the limit, you might get banned for longer periods of time. We don't know. It's conceivable though.

[anidbban]

Same issue. I've never encountered this issue using anidb in Plex, nor in Jellyfin. But as soon as I pay for filebot and start to use it, within an hour I'm banned from anidb and have lost functionality not only in filebot.

If I google "AniDB filebot banned" I immediately see a half dozen posts exactly like this, and in each one I see someone from filebot act dismissive about the possibility that it's your software doing something wrong.

You sold a me tool that immediately bricks itself, and in the process degrades functionality in unrelated applications, and it looks like this problem has been reported over and over and you've done nothing to address it, pushed blame on others. It's really shameful, and I'll discourage people from paying for this software at every opportunity going forward.

I guess the workaround is maybe to only ever use filebot on VPN, and be ready to cycle to a new server periodically, but man, what a disappointment after reading all the raving reviews for filebot.

[rednoah]

Since different applications don't share AniDB data, the more applications you use, the more HTTP requests are sent to AniDB services, and thus the faster you will hit the limit. A typical FileBot usage scenario will also look different from a typical Media Center usage scenario. In that FileBot is usually used to process a large number of Anime in a short time, while typical Media Center usage is more stretched out over time, adding a few new Anime per day, thus not running in the limits.


If you suspect an issue with FileBot, then you can enable full debug logging to trace each HTTP request and confirm the issue for yourself:
viewtopic.php?t=3913


I'm looking forward to your analysis of FileBot logs and suggestions for improvement. FileBot is as good as I can make it. Perhaps you can find a flaw that I cannot.


EDIT:

Out of curiosity, do you mind sharing your usage scenario? GUI? CLI? Are you keeping logs? Are you processing hundreds of Anime at once, or just a few new episodes per day? etc

[anidbban]

I mean, I'm able to spin up a new instance of a media center like plex, give it my entire library to scan and identify at once, walk away while it works, and never once had to worry that I'm going to get banned.

To address the point about multiple applications, I've now had to set it up so that plex, filebot, and anything else that talks to anidb all are on separate VPN connections with different IP addresses. Having done this, I started up filebot, started to rename more of my library, and within an hour got banned again.

My goal was to move away from plex to Jellyfin, which, being a fork of emby, is much more finicky about file locations and file names. So I need to rename my entire library essentially. You advertise and sell a tool for doing this. Can it really be your position that it's not a valid use case for filebot to try to batch rename large numbers of files?

[rednoah]

It’s certainly a valid use case, but AniDB unilaterally makes this use case impossible, if you want to process more than ~250 anime per day.


If you have a small library, it’ll work just fine. If you have a large library, then you may need to consider alternative databases, or alternatively spread out processing over time. FileBot does not force you to process all your files in one go.


The HAMA plugin for Plex (NOTE: Plex doesn’t officially support AniDB for some reason) is running in the background, so it can take its time, and just go to sleep for a day or two whenever it hits the limit.



Now consider a user-interactive GUI tool like FileBot:

(A) Should we error out immediately once our IP has been banned? That’s the current behavior.

(B) Should we just go to sleep for 24 hours after each batch of 200 requests to make sure we never hit the limit? This could work on the CLI (assuming that there aren't any other tools using up requests too) but it’s somewhat unacceptable for a GUI tool. Either way, a typical end user will likely kill FileBot after a few minutes of FileBot not doing anything, instead of waiting for 24 hours for FileBot to resume its work.

(C) Remove AniDB support altogether as not to inadvertently misleading a small minority of users with marketing that "only" applies to the vast majority of users that use TheTVDB, or TheMovieDB, and even AniDB users that just so happen to never hit the limit? The silent majority of happy users might not take kindly to that.

It is my position that these are the options that we are given, and that we have to choose one of them. I'd like to stress that all of these options are bad options, and that none of these options will allow you to process a large library with 250+ anime against AniDB in one go within a day.



I encourage a technical discussion with well-reasoned arguments. We're on the same side. We all want things to just work, but wanting things does not change reality, and neither does complaining. If there is a technical solution, then I’m happy to hear it and implement it.

[rv23]

AniDB has if I recall handles their UDP API request differently then their HTTP API requests. This UDP API wiki page https://wiki.anidb.net/UDP_API_Definition has some information on expected behaviors on clients, due to rate limiting/banning.
Excerpt
"Connection Problems
There are many ways for a client to end up banned or the API might currently be handling too many concurrent connections. If a client does not get any reply to an AUTH command it should start to increase the retry delay exponentially with every failed login attempt. (i.e. try again after 30 seconds if the first login attempt failed, if the second fails too retry after 2 minutes, then 5 minutes, then 10 minutes, then 30 minutes, ... until you reach a retry delay of ~2-4h.)"
and Rate Limiting (which is probably the most influencial on banning)
"Flood Protection
To prevent high server load the UDP API server enforces a strict flood protection policy.

Short Term:
A Client MUST NOT send more than 0.5 packets per second (that's one packet every two seconds, not two packets a second!)
The server will start to enforce the limit after the first 5 packets have been received.
Long Term:
A Client MUST NOT send more than one packet every four seconds over an extended amount of time.
An extended amount of time is not defined. Use common sense.
Once a client exceeds a rate limit all further UDP packets from that client will be dropped without feedback until the client's packet rate is back down to acceptable levels.

Note Dropped packets are still taken into account for the packet rate. Meaning if you continuously send packets your client will be banned forever.
Generally clients should be written in a way to minimize server and network load. You should always keep that in mind.

Abusive clients may be banned completely.

Note If you suddenly stop getting replies from the AniDB API or you normally don't have a notable packetloss and from one point on you suddenly note a high packetloss percentage you have most likely entered a critical flood ratio.
If the AniDB API isn't answering at all it might either be down or you have been banned, this is normally caused by violating the API specs (i.e. too many connections using different ports from one IP, too many auth failures, ...) (usually lasts 30 minutes). If you're experiencing packet loss you have probably reached the rate limit and the API starts to randomly drop incoming packets from your IP. (can be solved by enforcing a delay between multiple commands when you're sending a batch)

Anti Leech Protection
The API should not be used to "download" AniDB. If such attempts are detected you will get banned"

The HTTP API page here: https://wiki.anidb.net/HTTP_API_Definition is even less detailed on what could get you banned (tho it probably is similiar to their UDP rate limiting)

Excerpt:
"Flooding and Caching
All users of this API should employ heavy local caching. Requesting the same dataset multiple times on a single day can get you banned. The same goes for request flooding. You should not request more than one page every two seconds.

Anti Leech Protection
The API should not be used to "download" AniDB. If such attempts are detected you will get banned."

The Official AniAdd client (java based) uses their UDP API, and in it's current iteration rarely get's someone banned, but on large datasets it can take quite some time. Of course it's also only good for renaming files that are actually *IN* AniDB.

unfortunately the SourceCode for AniAdd is not Open Source, but there is an older version of it (perhaps when it was OSS) that you can find on GitHub.

The list of Software/Clients for AniDB is here: https://anidb.net/software

The API policy (and likely the API itself) is very 1990's/2000's, which fits into when AniDB was first open to the public (2002). Rest type API's where not that common I believe in the early 2000's, so it's not that unexpected that they don't have one. They also don't rely on ad revenue etc, which probably limits their options as well. They currently get their "funding" from people on Patreon (https://www.patreon.com/AniDB).

None of these things make it less annoying to deal with

[rv23]

Older (not current) source of AniAdd variant: https://github.com/Tyderion/AniDb-AniAdd-CLI

[rednoah]

Anti Leech Protection
The API should not be used to "download" AniDB. If such attempts are detected you will get banned."

This is a reasonable limit, but effectively indistinguishable from someone with a large A-Z library processing all their files. As for a specific HTTP request limit, I did bite the sour apple and got myself banned, and I got ~250 requests via the HTTP API before the ban hammer struck. Stopped. Tried again two days later and everything worked again. So the ban does time out if you stop hitting AniDB for a while.


FileBot r7750 implements various new behaviors and now things work as follows:
* No more than one active request at any point in time
* No more than 5 requests within a 1 minute time frame (i.e. the first 5 requests are very fast, one after another, but the 6th request will have to wait for ~60 seconds, heavily slowing down processing)
* No more than 200 request per 24 hour time frame (i.e. the first 200 requests will be relatively fast at 5 requests per 60 seconds, but if we hit the limit, then FileBot will deadlock and suspend processing for up to 24 hours)
* All information is cached for 6 days (i.e. FileBot will not request the same information twice, unless the information we have is more than 6 days old already)
* If our 200 request per 24 hour limit is 50% exhausted, then FileBot will prefer previously cached information regardless of age to conserve requests if possible (i.e. you may end up with episode data that is 2-3 months old that was cached in a previous run long ago)
* A single banned response will switch the AniDB client into offline mode for the remaining process life-time and all subsequent requests that cannot be served by the cache will fail immediately

The limits above can only be enforced within a single running filebot process. Running multiple filebot processes parallel or in sequence will effectively disable all these limits.