WARNING: AniDB error: banned

All your suggestions, requests and ideas for future development
Post Reply
Fribb
Posts: 3
Joined: 23 Jun 2020, 16:17

WARNING: AniDB error: banned

Post by Fribb » 23 Jun 2020, 16:24

SysInfo:
FileBot 4.9.1 (r7372)
JNA Native: 6.1.0
MediaInfo: 19.09
7-Zip-JBinding: 9.20
Chromaprint: 1.4.3
Extended Attributes: OK
Unicode Filesystem: OK
Script Bundle: 2020-06-18 (r655)
Groovy: 3.0.3
JRE: OpenJDK Runtime Environment 14
JVM: 64-bit OpenJDK 64-Bit Server VM
CPU/MEM: 12 Core / 4.3 GB Max Memory / 68 MB Used Memory
OS: Windows 10 (amd64)
STORAGE: NTFS [(C:)] @ 171 GB | NTFS [Games] @ 320 GB | NTFS [Data] @ 995 GB
DATA: C:\Users\Fribb\AppData\Roaming\FileBot
Package: MSI
License: FileBot License PX11041992 (Valid-Until: 2070-01-03)
Saturday I started working on my AniDB renaming again when I got the AniDB error: banned Warning message.

While looking for a solution I noticed this forum post which told me that apparently my IP got banned because of too many requests. I also have seen that there is some code that prevents too many requests in a short amount of time, IIRC it was 2 requests in 5 seconds?! But apparently this either does not work correctly or there is some issue that lets you do more requests than you should be able to.

User avatar
rednoah
The Source
Posts: 18537
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: WARNING: AniDB error: banned

Post by rednoah » 23 Jun 2020, 16:48

AFAIK, AniDB has a daily limit of ~100 ~200 requests (i.e. episode listings for ~100 ~200 shows) per IP per day, so if you run FileBot against a large library, you will hit the limit. The "2 requests in 5 seconds" flood limit will space requests out a little bit, just to be nice to the server, but you'll still hit the daily limit eventually.


:arrow: Depending on your files, TheTVDB with Absolute Episode Order might be more suitable for the task at hand.


EDIT:

Newer revisions use a flood limit of "1 requests in 5 seconds" so you'll hit the daily limit in ~10 ~20 minutes if you run FileBot against a large library.
:idea: Please read the FAQ and How to Request Help.

Fribb
Posts: 3
Joined: 23 Jun 2020, 16:17

Re: WARNING: AniDB error: banned

Post by Fribb » 23 Jun 2020, 16:57

Is that somewhere listed in Filebot itself? Because the daily cap is something new to me (and I have renamed a lot more than 200 the weeks before)

Maybe a Daily counter would be a good way to make this more visible

For the TheTVDB option, that is not really an option for me. Even AniDB is just a workaround since most of my Media file actually should be in a MyAnimeList format but since there is no renamer for this (except the mess of a CLI program I wrote) filebot with AniDB is a better solution.

User avatar
rednoah
The Source
Posts: 18537
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: WARNING: AniDB error: banned

Post by rednoah » 23 Jun 2020, 17:20

Fribb wrote:
23 Jun 2020, 16:57
Is that somewhere listed in Filebot itself? Because the daily cap is something new to me (and I have renamed a lot more than 200 the weeks before)
We don't know what the cap is. The ~100 ~200 numbers is just a random guess. We only know that it's relatively low. If you process 800 episodes of One Piece, then you will never run into the issue because it's just 1 episode list request. If you process 500 different Anime shows, with only one episode each, then you will almost certainly hit the limit. Also, the limit may change over time. Also, the limit may change every time you hit the limit. Also, FileBot does cache requests, so if you process the same Anime shows over and over every day, then FileBot will not actually request anything most of the time.

Fribb wrote:
23 Jun 2020, 16:57
Maybe a Daily counter would be a good way to make this more visible
At best, the FileBot process can track it's own requests, and that's where "1 requests in 5 seconds" comes from, but has no way of tracking request count to AniDB from your public IP. There could be other filebot instances on other machines on your local network. There could be other tools talking to AniDB and using up requests.


:arrow: If you must use AniDB, then I'd recommend processing less than 100 Anime shows per day. AFAIK, the limit is reset every day, so you can play it and see where the limit is.
:idea: Please read the FAQ and How to Request Help.

Fribb
Posts: 3
Joined: 23 Jun 2020, 16:17

Re: WARNING: AniDB error: banned

Post by Fribb » 23 Jun 2020, 17:49

If you must use AniDB, then I'd recommend processing less than 100 Anime shows per day. AFAIK, the limit is reset every day, so you can play it and see where the limit is.
Alright, but for that, I need to get unbanned first or get a new IP which, so far, hasn't worked. Thanks.

User avatar
rednoah
The Source
Posts: 18537
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: WARNING: AniDB error: banned

Post by rednoah » 24 Jun 2020, 03:53

Fribb wrote:
23 Jun 2020, 17:49
Alright, but for that, I need to get unbanned first or get a new IP which, so far, hasn't worked. Thanks.
Well, you can always use your phone as hotspot and viola.

But remember that the "daily limit" again is pure conjecture. If you repeatedly run into the limit, you might get banned for longer periods of time. We don't know. It's conceivable though.
:idea: Please read the FAQ and How to Request Help.

anidbban
Posts: 2
Joined: Yesterday, 08:05

Re: WARNING: AniDB error: banned

Post by anidbban » Yesterday, 08:13

Same issue. I've never encountered this issue using anidb in Plex, nor in Jellyfin. But as soon as I pay for filebot and start to use it, within an hour I'm banned from anidb and have lost functionality not only in filebot.

If I google "AniDB filebot banned" I immediately see a half dozen posts exactly like this, and in each one I see someone from filebot act dismissive about the possibility that it's your software doing something wrong.

You sold a me tool that immediately bricks itself, and in the process degrades functionality in unrelated applications, and it looks like this problem has been reported over and over and you've done nothing to address it, pushed blame on others. It's really shameful, and I'll discourage people from paying for this software at every opportunity going forward.

I guess the workaround is maybe to only ever use filebot on VPN, and be ready to cycle to a new server periodically, but man, what a disappointment after reading all the raving reviews for filebot.

User avatar
rednoah
The Source
Posts: 18537
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: WARNING: AniDB error: banned

Post by rednoah » Yesterday, 09:46

Since different applications don't share AniDB data, the more applications you use, the more HTTP requests are sent to AniDB services, and thus the faster you will hit the limit. A typical FileBot usage scenario will also look different from a typical Media Center usage scenario. In that FileBot is usually used to process a large number of Anime in a short time, while typical Media Center usage is more stretched out over time, adding a few new Anime per day, thus not running in the limits.


:idea: If you suspect an issue with FileBot, then you can enable full debug logging to trace each HTTP request and confirm the issue for yourself:
viewtopic.php?t=3913


I'm looking forward to your analysis of FileBot logs and suggestions for improvement. FileBot is as good as I can make it. Perhaps you can find a flaw that I cannot.


EDIT:

:?: Out of curiosity, do you mind sharing your usage scenario? GUI? CLI? Are you keeping logs? Are you processing hundreds of Anime at once, or just a few new episodes per day? etc
:idea: Please read the FAQ and How to Request Help.

anidbban
Posts: 2
Joined: Yesterday, 08:05

Re: WARNING: AniDB error: banned

Post by anidbban » Yesterday, 17:08

I mean, I'm able to spin up a new instance of a media center like plex, give it my entire library to scan and identify at once, walk away while it works, and never once had to worry that I'm going to get banned.

To address the point about multiple applications, I've now had to set it up so that plex, filebot, and anything else that talks to anidb all are on separate VPN connections with different IP addresses. Having done this, I started up filebot, started to rename more of my library, and within an hour got banned again.

My goal was to move away from plex to Jellyfin, which, being a fork of emby, is much more finicky about file locations and file names. So I need to rename my entire library essentially. You advertise and sell a tool for doing this. Can it really be your position that it's not a valid use case for filebot to try to batch rename large numbers of files?

User avatar
rednoah
The Source
Posts: 18537
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: WARNING: AniDB error: banned

Post by rednoah » Today, 05:54

:!: It’s certainly a valid use case, but AniDB unilaterally makes this use case impossible, if you want to process more than ~250 anime per day.


:idea: If you have a small library, it’ll work just fine. If you have a large library, then you may need to consider alternative databases, or alternatively spread out processing over time. FileBot does not force you to process all your files in one go.


:idea: The HAMA plugin for Plex (NOTE: Plex doesn’t officially support AniDB for some reason) is running in the background, so it can take its time, and just go to sleep for a day or two whenever it hits the limit.



Now consider a user-interactive GUI tool like FileBot:

:arrow: (A) Should we error out immediately once our IP has been banned? That’s the current behavior.

:arrow: (B) Should we just go to sleep for 24 hours after each batch of 200 requests to make sure we never hit the limit? This could work on the CLI (assuming that there aren't any other tools using up requests too) but it’s somewhat unacceptable for a GUI tool. Either way, a typical end user will likely kill FileBot after a few minutes of FileBot not doing anything, instead of waiting for 24 hours for FileBot to resume its work.

:arrow: (C) Remove AniDB support altogether as not to inadvertently misleading a small minority of users with marketing that "only" applies to the vast majority of users that use TheTVDB, or TheMovieDB, and even AniDB users that just so happen to never hit the limit? The silent majority of happy users might not take kindly to that.

:idea: It is my position that these are the options that we are given, and that we have to choose one of them. I'd like to stress that all of these options are bad options, and that none of these options will allow you to process a large library with 250+ anime against AniDB in one go within a day.



:arrow: I encourage a technical discussion with well-reasoned arguments. We're on the same side. We all want things to just work, but wanting things does not change reality, and neither does complaining. If there is a technical solution, then I’m happy to hear it and implement it.
:idea: Please read the FAQ and How to Request Help.

Post Reply