[Feature] anidb UDP file hashing+lookup?

All your suggestions, requests and ideas for future development
Post Reply
andy22
Posts: 10
Joined: 17 Aug 2020, 16:59

[Feature] anidb UDP file hashing+lookup?

Post by andy22 »

It seems FB is not actually utilizing the AniDB UDP api to match files, yet it already has a lot of the skeleton to-do the work. It seems it has/had a ED2k hasher and already know the AniDB endpoint as DB + Mapper. So having the ability to directly hash a file and use the UDP API, would result in perfect matches and also remove the need for third party scripts/tools.

Would really help if FB could directly do the job, so we can remove the extra tools/steps from our import pipeline.
User avatar
rednoah
The Source
Posts: 22991
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: [Feature] anidb UDP file hashing+lookup?

Post by rednoah »

AniDB UDP API requires a user login per user and is thus fundamentally ill-suited for how FileBot works. The additional I/O would make things infinitely slower (especially when processing files on SMB shares as many do) for every single file, but won't have a noticeable effect on matching accuracy for the vast majority of users since things already work well.

It certainly makes for a nice-to-have 3rd party tool (or FileBot Groovy script if preferred) though. Just build a simple tool that talks to AniDB UDP to do the lookup for each file overnight and write xattr metadata. Then run through Local Xattr Mode the next day in a second step.
:idea: Please read the FAQ and How to Request Help.
andy22
Posts: 10
Joined: 17 Aug 2020, 16:59

Re: [Feature] anidb UDP file hashing+lookup?

Post by andy22 »

rednoah wrote: 18 Aug 2020, 10:50 AniDB UDP API requires a user login per user and is thus fundamentally ill-suited for how FileBot works.
mhh ok i understand, but does OMDB not also have a scrape limit and require additional keys?
The additional I/O would make things infinitely slower (especially when processing files on SMB shares as many do) for every single file
Not sure what you mean, but the hashing via the modern/updated tools/script is as fast as the filesystem. So via ssd/hdd/smb normal episodes take like 1-2 seconds and movies 3-6s.
The required timeouts on AniDB via http/udp are usually enough to hash the next file, so you are more likely to actually wait for the enforced timeouts to avoid banns, than waiting for the hashing.

I think a groovy script could do the job and would need a fast ED2K (preferably multicore) hasher and be able to parse the AniDB UDP file api results.
User avatar
rednoah
The Source
Posts: 22991
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: [Feature] anidb UDP file hashing+lookup?

Post by rednoah »

The other databases do use API keys, but that's just to identify the client application, so that's built-in.

The AniDB UDP API requires a user to create an account on AniDB and then enter username/password into FileBot before anything can work. This would make FileBot not work out of the box.

A large number of users process files on their NAS from their Desktop and expect this to not involve reading all bytes over the network. Though it might have been ok if we had done so all along, starting to do so now in a future update would be a bit jarring. IMHO the cons outweigh the pros.
:idea: Please read the FAQ and How to Request Help.
andy22
Posts: 10
Joined: 17 Aug 2020, 16:59

Re: [Feature] anidb UDP file hashing+lookup?

Post by andy22 »

rednoah wrote: 18 Aug 2020, 11:28 IMHO the cons outweigh the pros.
ok i understand, maybe keep the idea in mind.

I can get something to work, with a prepass external script that prepares the files for the main FB pass.
Post Reply