How to get series synonym from AniDB in filebot when using groovy Script.

[rv23]

The net.filebot.web.SeriesInfo object returned from AniDB.getSeriesInfo("Sword Art Online: Alicization - War of Underworld (2020)", locale) doesn't look to include any synonyms, only official titles ..

Code: Select all

[runtime:null, startDate:2020-07-12, genres:[], certification:null, rating:4.69, id:15146, name:Sword Art Online: Alicization - War of Underworld (2020), network:null, ratingCount:null, type:Anime, class:class net.filebot.web.SeriesInfo, spokenLanguages:[], order:Absolute, status:null, language:en, aliasNames:[Sword Art Online: Alicization - War of Underworld, ???????????? ???????? War of Underworld (2020), ???????????? ???????? War of Underworld, ??????????? ???????? War of Underworld], database:AniDB]

Is this the correct way to get the Synonym Information?
Does filebot not support retrieving it from AniDB?

In theory the http api does support returning that information:
https://wiki.anidb.net/HTTP_API_Definition

[rednoah]

What specific synonym are you looking for? Presumably, we're using only "good" alias names such as official titles to accidental mismatches caused by synonym names (which tend to be invisible to end users, making accidental behaviour seem particularly random and stupid).

[rv23]

I'm experimenting with using jaroWinklerDistance (https://en.wikipedia.org/wiki/Jaro%E2%8 ... r_distance) from Apache Commons (https://commons.apache.org/proper/commo ... mmary.html) to determine the "highest" matching entries in TheTVDB/AniDB.
First I query for the Series (TheTVDB.search/AniDB.Search), then the Aliases for each Series returned (TheTVDB.getSeriesInfo/AniDB.getSeriesInfo) and then perform a jaroWinklerDistance on those (vs the Series name from the file) and see if I can use that as a "hint" on the best way to resolve the file with higher accuracy then a basic AMC strict/non-strict (which for some current shows is very much a hit or miss because of some silly filenames people release).

If I can figure out how to download/parse the XML AniDB Title database, I'll add that into the mix for AniDB (which should greatly increase the 'accuracy' of the AniDB search as I can search for the highest match from all Anime Titles/Synonyms), that would also help with looking up info when switching between AniDB/TheTVDB as a "if the first one doesn't succeed, try the other one" process.

I am doing this because I do NOT want to manually babysit the Anime sorting at this part of the process (This is the first sorting of the raw inbound files), but the base accuracy of AMC is not as high as I want when in non-strict (which is unfortunately required for ALOT of anime, I DO run a strict pass FIRST before resorting to Non-Strict). So far of the current season, a number of the release groups for Re Zero, Latest Sword Art Online and surprisingly Black Clover anime have filenames that just confuse filebot (with some good reason for some of them).

So right now, there are a number of release groups that look to be purposely trying to make it hard to accurately get the series right (much less the episode), with names like the following (these are from the last few day's, and strict amc wasn't working)

None of these match using strict for me (recently), and this is not an exhaustive list of files that strict has issues with, and non-strict doesn't always work right .. and this is when using AMC, so when AMC doesn't group files together "accurately", it can also make for some wildly off matches using non-strict (reason #2 I don't like to use non-strict with amc).

Code: Select all

(CBB) Re-Zero kara Hajimeru Isekai Seikatsu - 35 (1080p)(HEVC)(10bit-AAC).mkv
[FFA] Re_Zero kara Hajimeru Isekai Seikatsu 2nd Season - 10 [1080p][HEVC][AAC].mkv - This does actually work correctly if using non-strict (but I don't like using non-strict unless all other options are exhausted)
[EMBER] Re Zero kara Hajimeru Isekai Seikatsu S02 - 10.mkv
[shadow.jp.net] Re Zero kara Hajimeru Isekai Seikatsu - 35 [720p] [MULTi-SUB].mkv - This does actually work correctly if using non-strict (but I don't like using non-strict unless all other options are exhausted)
[Edge] ReZero kara Hajimeru Isekai Seikatsu (Season 2) - 35- I Know Hell [1080p].mkv
[HorribleSubs] Re Zero kara Hajimeru Isekai Seikatsu - 35 [720p].mkv
[shadow.jp.net] Re Zero kara Hajimeru Isekai Seikatsu - 35 [720p] [MULTi-SUB].mkv
[AkihitoSubs] Sword Art Online Alicization - War of Underworld - S02E09 (21).mkv
[BakedFish] Sword Art Online_ Alicization - War of Underworld 2nd Season - 09 [720p][AAC].mp4
[HR] Black Clover 142 - S03E40.mkv
[Edge] Black Clover (TV) - 142 [1080p][Multiple Subtitle][10Bit][x265].mkv
[mal lu zen] Black Clover - 142 [720p].mkv

I've given up on a generic programmatic method of dealing with Sword Art Online Alicization War of the Underworld, and just hard code any "season 2" stuff to Sword Art Online War of the Underworld (2020) on AniDB (non-strict) and at least I get the right season, same deal with Black Clover as there is only the SINGLE season on TheTVDB, the episodes rarely match but at least it's the right series (but since the episodes rarely match, the time of year aka 'anime season' usually doesn't match, but it's the best that I can do with that filename). I'm at a bit of a loss on why some of the simple black cover filenames don't match the episodes frequently either..

I don't rename the files to the episode(s) at this point, nor add xattr because even a 20-30% mismatch is still thousands of files for me in this stage. I collect ALOT of fansub Anime (I also purchase the titles I like too), so accuracy is very important for me (I'm an avid AniDB user), and I also have alot of stuff that isn't in TheTVDB as well.

I have been adding Synonyms like Sword Art Online_ Alicization - War of Underworld 2nd Season and Re_Zero kara Hajimeru Isekai Seikatsu 2nd Season to AniDB so I can improve the accuracy of my filebot script as well as help increase the ability of other's to match against some of the silly names release groups are using.

If it's possible to get the AniDB synonyms info from filebot that would be my preferred method, even if it's a flag to pass to getSeriesInfo or even a dedicated method is fine. It *seems* like filebot will match against the synonym when in non-strict, but it could also be just getting "lucky"

Does this help explain what I'm looking for and why I'm asking for a method to retrieve the synonym info using filebot (in a groovy script)?

[rednoah]

1.

If it's possible to get the AniDB synonyms info from filebot that would be my preferred method, even if it's a flag to pass to getSeriesInfo or even a dedicated method is fine. It *seems* like filebot will match against the synonym when in non-strict, but it could also be just getting "lucky"

I reviewed some of the code, and at first glance I don't see any code that explicitly excludes synonyms. But I also can't explain why they're not there. Presumably, there's caching every step of the way, and so information newly added on the website might need a few weeks to flow onto your local machine.




2.
At a glance, would it be safe to say that you're only ever processing newly released episodes within days of the release date? Because that would vastly narrow down the scope of the problem, since we could use --filter to focus on episodes released during a very specific window of time:

Code: Select all

--filter "age < 3" --def "ut_label=Anime"

Advanced Fine-Tuning
You can (and should) force Movie/Series/Anime mode by setting --def ut_label accordingly, and you can further define your own --filter rules to fine-tune matching to your exact needs, and avoid mismatches. For example, if one show is confused with another due to bad naming or missing data you can simply exclude the bad one, or if you only process recently aired episodes you could exclude any episode of any show that has been aired for more than a few days.

e.g. if you only process recently aired episodes you can virtually guarantee 100% auto-detection accuracy by adding simple filter rules such as these:

Code: Select all

--filter "age < 5" --def "ut_label=TV"

[rv23]

Well, I think I have a better idea of why the Re: Zero stuff frequently doesn't get matched (against TheTVDB), and why using -q "Re:Zero kara Hajimeru Isekai Seikatsu" just doesn't work ..
It seems that Re:Zero kara Hajimeru Isekai Seikatsu (which is what SeriesInfo returns, and has the highest similiarity/match against most of the file names) .. is in some odd state in TheTVDB, as it shows up in a search as an object, but you can't actually browse the show ..

https://www.thetvdb.com/search?query=Re ... 20Seikatsu

and gives a 404 when you try to click on it ..

It *IS* also an Alias for Re:ZERO -Starting Life in Another World-, which is the "correct" show to use with -q/query .. What a PITA.

hmm.. will querying the series for episodes on TheTVDB to weed out stuff like "Re:Zero kara Hajimeru Isekai Seikatsu" which is returned by TheTVDB.Search, but is in fact not a valid show?

[rv23]

Re: AniDB Synonyms

I ran filebot -clear-cache and I still do see the AniDB Synonyms for Sword Art Online: Alicization - War of Underworld (2020)

Code: Select all

Our AniDB Aliases: [Sword Art Online: Alicization - War of Underworld, ???????????? ???????? War of Underworld (2020), ???????????? ???????? War of Underworld, ??????????? ???????? War of Underworld] for Sword Art Online: Alicization - War of Underworld (2020)

an a "properties" from the object returned from AniDB.getSeriesInfo

Code: Select all

properties for Sword Art Online: Alicization - War of Underworld (2020): [runtime:null, startDate:2020-07-12, genres:[], certification:null, rating:4.61, id:15146, name:Sword Art Online: Alicization - War of Underworld (2020), network:null, ratingCount:null, type:Anime, class:class net.filebot.web.SeriesInfo, spokenLanguages:[], order:Absolute, status:null, language:en, aliasNames:[Sword Art Online: Alicization - War of Underworld, ???????????? ???????? War of Underworld (2020), ???????????? ???????? War of Underworld, ??????????? ???????? War of Underworld], database:AniDB]

looks to pretty much be just the Official/Main titles ..

[rednoah]

2.
At a glance, would it be safe to say that you're only ever processing newly released episodes within days of the release date? Because that would vastly narrow down the scope of the problem, since we could use --filter to focus on episodes released during a very specific window of time:

Code: Select all

--filter "age < 3" --def "ut_label=Anime"

Advanced Fine-Tuning
You can (and should) force Movie/Series/Anime mode by setting --def ut_label accordingly, and you can further define your own --filter rules to fine-tune matching to your exact needs, and avoid mismatches. For example, if one show is confused with another due to bad naming or missing data you can simply exclude the bad one, or if you only process recently aired episodes you could exclude any episode of any show that has been aired for more than a few days.

e.g. if you only process recently aired episodes you can virtually guarantee 100% auto-detection accuracy by adding simple filter rules such as these:

Code: Select all

--filter "age < 5" --def "ut_label=TV"

Does this not bypass all the problems listed above simply and elegantly?

[rv23]

Re: Filter by Age

My apologies, I used examples of currently airing shows primarily because the example info was readily available for the post, and as currently airing shows I'm interesting in increasing the accuracy of their matching first. However without any real in-depth analysis, from the files collected over the last 2 months, currently airing shows accounts for 5.3k of the almost 29.5k files (however the actual number of 2020 shows is likely a bit higher due to mismatches not identifying it as a 2020 show).

A quick AMC run adding age < 5 and -non-strict to the files that amc strict failed on last night does show some promise, but I'm still a bit leery of non-strict due to false matches (even if not that many)

Code: Select all

[TEST] from [Z:\FB-3\anime\[Erai-raws] Appare-Ranman! - 10 [720p].mkv] to [Z:\1-InitialSort\2020\summer\Appare-Ranman! [anidb-15145]\[Erai-raws] Appare-Ranman! - 10 [720p].mkv]
[TEST] from [Z:\FB-3\anime\[Erai-raws] Boruto - Naruto Next Generations - 162 [720p][Multiple Subtitle].mkv] to [Z:\1-InitialSort\2020\summer\Boruto Naruto Next Generations [anidb-12661]\[Erai-raws] Boruto - Naruto Next Generations - 162 [720p][Multiple Subtitle].mkv]
[TEST] from [Z:\FB-3\anime\[Erai-raws] Deca-Dence - 08 [720p].mkv] to [Z:\1-InitialSort\2020\summer\Deca-Dence [anidb-14994]\[Erai-raws] Deca-Dence - 08 [720p].mkv]
[TEST] from [Z:\FB-3\anime\[Erai-raws] Dokyuu Hentai HxEros - 08 [720p].mkv] to [Z:\1-InitialSort\2020\summer\Super HxEros [anidb-15241]\[Erai-raws] Dokyuu Hentai HxEros - 08 [720p].mkv]
[TEST] from [Z:\FB-3\anime\[Erai-raws] Fruits Basket 2nd Season - 21 [720p].mkv] to [Z:\1-InitialSort\2020\summer\Fruits Basket Season 2 [anidb-14988]\[Erai-raws] Fruits Basket 2nd Season - 21 [720p].mkv]
[TEST] from [Z:\FB-3\anime\[FFA] Fugou Keiji_ Balance_Unlimited - 09 [1080p][HEVC][AAC].mkv] to [Z:\1-InitialSort\2020\summer\The Millionaire Detective - Balance Unlimited [anidb-15364]\[FFA] Fugou Keiji_ Balance_Unlimited - 09 [1080p][HEVC][AAC].mkv]
[TEST] from [Z:\FB-3\anime\[Erai-raws] Houkago Teibou Nisshi - 08 [720p].mkv] to [Z:\1-InitialSort\2020\summer\Diary of Our Days at the Breakwater [anidb-14827]\[Erai-raws] Houkago Teibou Nisshi - 08 [720p].mkv]
[TEST] from [Z:\FB-3\anime\[Erai-raws] Lapis Re-LiGHTs - 10 [720p].mkv] to [Z:\1-InitialSort\2020\summer\Lapis ReLights [anidb-13973]\[Erai-raws] Lapis Re-LiGHTs - 10 [720p].mkv]
[TEST] from [Z:\FB-3\anime\[Erai-raws] Muhyo to Rouji no Mahouritsu Soudan Jimusho 2nd Season - 09 [720p].mkv] to [Z:\1-InitialSort\2020\summer\Muhyo & Roji`s Bureau of Supernatural Investigation (2020) [anidb-14953]\[Erai-raws] Muhyo to Rouji no Mahouritsu Soudan Jimusho 2nd Season - 09 [720p].mkv]
[TEST] from [Z:\FB-3\anime\[Erai-raws] Ninja Collection - 07 [720p][Multiple Subtitle].mkv] to [Z:\1-InitialSort\2020\summer\Ninja Collection [anidb-15603]\[Erai-raws] Ninja Collection - 07 [720p][Multiple Subtitle].mkv]
[TEST] from [Z:\FB-3\anime\[Erai-raws] Uzaki-chan wa Asobitai! - 08 [720p].mkv] to [Z:\1-InitialSort\2020\summer\Uzaki-chan Wants to Hang Out! [anidb-15382]\[Erai-raws] Uzaki-chan wa Asobitai! - 08 [720p].mkv]
[TEST] from [Z:\FB-3\anime\[leorio1009] One Piece Special Edition - 101-150 [WEB 720p] SUBTITLES\[leorio1009] One Piece SE - 141 [720p].ass] to [Z:\1-InitialSort\2020\summer\One Piece [anidb-69]\[leorio1009] One Piece SE - 141 [720p].ass]
[TEST] from [Z:\FB-3\anime\Pokemon The Movie 22 - Mewtwo Strikes Back Evolution [BDRip.DE.720p.Russian.v2020.09.by.LightEssence].mkv] to [Z:\1-InitialSort\2020\summer\Pokemon Journeys The Series [anidb-15067]\Pokemon The Movie 22 - Mewtwo Strikes Back Evolution [BDRip.DE.720p.Russian.v2020.09.by.LightEssence].mkv]

Re: Filter by Series Name
The Filter topic link: viewtopic.php?t=2127 has this example:

Code: Select all

--filter "n in lines('/path/to/anime-names.txt')"

I was going to do some tests on this with TheTVDB/AniDB, but I just got API banned from AniDB so no testing using AniDB

Does the title need to match 100% the title(s) returned by TheTVDB or AniDB? (This does look to be the case) If yes, does it need to match the English Title for AniDB, and the official title for TheTVDB?.

This seems like it might work a bit better at increasing the "current" show matching when using strict, and avoid most of the odd matching from non-strict when run against a wide gambit of shows with varying "ages". I just need to write something to get that list of current AniDB/TheTVDB titles

Still doesn't help with increasing the accuracy of matching "older" titles, but it should make a good first pass for currently airing shows, and then a 2nd pass for more "in-depth" matching for everything else.

[rednoah]

1.
I see. I guess in your case Filter by Age makes only sense as a first pass. Process newly aired shows with high reliability, and fail naturally if no potential episode match passes the filter, to be dealt with in subsequent passes.

A mismatch here would require a similarly named series airing at the same time within the same N days window. Possible for sure, but probably less likely than accidental String similarity / dissimilarity that we have to content with otherwise.

We would expect FileBot to work better on more established shows, and work less well on newly added shows that nobody has heard of before. A first pass with Filter by Age might balance out this particular weakness.


2.
n in --filter is whatever n is in your --format, so I guess that would be the English series name. You can of course use id or something else there as well, whatever works best for you.

[rv23]

Re: Filter - n/primaryTitle matching

For the filter of x in lines('file'), do the line endings matter, or are there special (non-ascii for example, * or @ 'special' characters) that need to be stripped out for the match to work? would you expect a file with a few hundred lines to present problems with the filter?

Would a reason filebot doesn't show any "options passed" because the filter didn't match or because there were no files that matched, so it didn't need to apply the filter?

Output from a Strict run of AMC

Code: Select all

Rename episodes using [AniDB] with [Absolute Order]
Auto-detected query: [Maou, maou gakuin no futekigousha]
Fetching episode data for [Kyou kara Maou!]
Fetching episode data for [Maou Dante]
Fetching episode data for [Maou-sama, Retry!]
Fetching episode data for [Maou-jou de Oyasumi]
Fetching episode data for [Maou Gakuin no Futekigousha: Shijou Saikyou no Maou no Shiso, Tensei Shite Shison-tachi no Gakkou e Kayou]
Apply filter [primaryTitle in lines('C:\\Users\\xxxx\\Desktop\\scripts\\2020-SUMMER-Anime-Titles.txt') || n in lines('C:\\Users\\xxxx\\Desktop\\scripts\\2020-SUMMER-Anime-Titles.txt')] on [118] options
[0 / 118] options passed
CmdlineException: Failed to identify or process any files

-
has me wondering if I still do not quite understand how the filter & episode matching work together. However in this case, the file (2020-SUMMER-Anime-Titles.txt) contains 339 lines, and there is an exact match (capitalization and everything) for "Maou Gakuin no Futekigousha: Shijou Saikyou no Maou no Shiso, Tensei Shite Shison-tachi no Gakkou e Kayou".
My Filter is this: "primaryTitle in lines('C:\\Users\\vitki\\Desktop\\scripts\\2020-SUMMER-Anime-Titles.txt') || n in lines('C:\\Users\\vitki\\Desktop\\scripts\\2020-SUMMER-Anime-Titles.txt')"

So it is that the Filter didn't match n or primaryTitle to a line in the file, so it didn't have any episode data in which to try to perform an episode match? if one of these n/primaryTitle is empty would it cause the entire filter to fail?

[rednoah]

lines() is a helper function that will give you a List<String> of lines in the text file. The rest is vanilla Groovy code. The code effectively boils down to doing this, no magic there:

Code: Select all

'A' in ['A', 'B']

This means that your filter excluded every single option, out of the 118 episodes that are taken into consideration:

Code: Select all

[0 / 118] options passed

You're effectively doing --filter false so to say.


One might debug like so:

Code: Select all

--filter "
println n
def lines = lines('C:\\Users\\xxxx\\Desktop\\scripts\\2020-SUMMER-Anime-Titles.txt')
println lines
def accept = n in lines
println accept
return accept
"

The best place to debug expressions is the Format Editor GUI though. Paste in your --filter expression, and see if it says true or false.

[rv23]

ah so in general groovy stuff works in the filter .. that is useful..

I kinda suspected it was one of the bindings was silently throwing an error and killing the expression ..

Code: Select all

Apply filter [println db.AniDB.n ? println db.AniDB.primaryTitle ? println db.TheTVDB.n ? def lines = lines('C:\\Users\\vitki\\Desktop\\Scripts\\current-Anime-Titles.txt') ? def accept = (db.AniDB.n in lines || db.AniDB.primaryTitle in lines || db.TheTVDB.n in lines || n in lines) && age < 14 ? println accept ? return accept] on [12] options

so I should see 3 titles, then a true/false ..

Code: Select all

The Misfit of Demon King Academy: History`s Strongest Demon King Reincarnates and Goes to School with His Descendants
Maou Gakuin no Futekigousha: Shijou Saikyou no Maou no Shiso, Tensei Shite Shison-tachi no Gakkou e Kayou
The Misfit of Demon King Academy
true

except not so much when one of the name bindings doesn't work ..

Code: Select all

Apply filter [println db.AniDB.n ? println db.AniDB.primaryTitle ? println db.TheTVDB.n ? def lines = lines('C:\\Users\\vitki\\Desktop\\Scripts\\current-Anime-Titles.txt') ? def accept = (db.AniDB.n in lines || db.AniDB.primaryTitle in lines || db.TheTVDB.n in lines || n in lines) && age < 14 ? println accept ? return accept] on [12] options
Peter Grill and the Philosopher`s Time
Peter Grill to Kenja no Jikan
Peter Grill and the Philosopher`s Time

can I use the any {} construct here?

Code: Select all

def accept = any {{ db.AniDB.n in lines } { db.AniDB.primaryTitle in lines } { db.TheTVDB.n in lines } { n in lines }}

Threw an error ..

Code: Select all

def accept = any {  (db.AniDB.n in lines)  (db.AniDB.primaryTitle in lines) (db.TheTVDB.n in lines) ( n in lines) }

just doesn't work (accept never prints a true or false)

Code: Select all

def accept = { any { db.AniDB.n in lines } { db.AniDB.primaryTitle in lines } { db.TheTVDB.n in lines } { n in lines } }

works as well as the first attempt not using the any closure ..

so I simplified it to *just* the primaryTitle ..
with mixed results, in this very very small test the short name (without any punctuation) matched as expected, while the long one (with punctuation) did not. Both are in the file ..

Code: Select all

Apply filter [println primaryTitle ? def lines = lines('C:\\Users\\vitki\\Desktop\\Scripts\\current-Anime-Titles.txt') ? def accept = primaryTitle in lines ? println accept ? return accept ? ] on [12] options
Peter Grill to Kenja no Jikan
true
Maou Gakuin no Futekigousha: Shijou Saikyou no Maou no Shiso, Tensei Shite Shison-tachi no Gakkou e Kayou
false

So, I started to play around with the string(s) to see if there is something about it that is confusing the groovy x in list code .. like removing the : and , (no luck) .. and then I started to remove entries in the file to see if that helps .. didn't seem to help..

The only thing that worked is to cut and paste the value from the filebot console output into the file .. ah hell.. Case Sensitivity .. The S in Shite it capitalized in the primaryTitle value, while in the file it is lowercase..

okay, well I'll switch all the entries in the file to lowercase AND try switching primaryTitle to lowercase and see how that affects things (stupid case sensitivity).

[rv23]

okay, that was the primary issue with values not matching when they "should", as the windows utilities I was using to search for are case insensitive they worked just fine with the mixed case, switching all the entries to lowercase and doing a lowercase search solves that problem.

At this point I'm good on my questions regarding the --filter usage trying to match shows ..

Thank you for your help.

[rednoah]

I usually don't recommend using {id} for --filter expressions, numeric ids aren't very human-friendly. But I guess String values aren't that human-friendly either in this case... id in ... might be easier if you have an easy way to get a list of numeric ids of Anime you're processing.