[DEPRECATED] support for XEM?

All your suggestions, requests and ideas for future development
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: support for XEM?

Post by rednoah »

ZyFinity wrote: 06 Nov 2022, 23:00 Is AnimeLists actually better and if so in what way?
ZyFinity wrote: 07 Nov 2022, 03:06 I just ran some tests, and it seems animelist mapped incorrectly here with that example folder structure above, what would the reasoning for that be?
The console output you posted shows that AnimeLists correctly maps Attack on Titan The Final Season (2022) - 01 to 4x17:

Code: Select all

Map [Attack on Titan The Final Season (2022) - 01 - Judgment] to [Attack on Titan - 4x17]

EDIT: I get confused. I don't watch that series so I don't actually know what the correct mapping should be for that one. I guess the mapping is right, but matching doesn't work out for other reasons. I'll check again some other day when I actually have time to research that series. The series / season names seem to be especially confusing.


The XEM mapping is definitely wrong, though FileBot does seem to arrive at the correct match in your specific use case, that's mostly by happy accident:

Code: Select all

Map [Attack on Titan The Final Season (2022) - 01 - Judgment] to [Attack on Titan - 1x01]
:!: TheXEM can't work reliably for similarly-named shows because the API doesn't provide AniDB ID / TheTVDB ID mappings. The API might have improved over time since we started this thread here in 2018 but I haven't looked into TheXEM since then.


EDIT: Map [A] to [M] indicates that you might be using --mapper the wrong way around. The [A] is what you want to see in the destination file path. The [M] is what you have in your source file names and should match that, e.g. Map [Attack on Titan The Final Season (2022) - 01 - Judgment] to [Attack on Titan - 4x17] means that you want to match files currently named "4x17" and then rename them to "01".



ZyFinity wrote: 06 Nov 2022, 23:00 I wanted to also ask why filebot only connects to a centralised "filebot hosted" animelists db that gets pulled monthly rather than just allowing us the user to download the database at our wanted set interval locally or even whenever there's a commit to the github repo or something. As I would be wanting to potentially have this setup with airing shows and be automated, however I guess this is where a fallback to the xem could come in handy.
* GitHub is not a CDN
* FileBot uses a modified XML file which adds in additional information, strips unnecessary entries, adds support for XZ compression and If-Modified-Since requests, etc
* FileBot does check for updates (of our modified XML file) bi-weekly-ish nowadays (I believe it's only updated every 1-2 weeks on the server-side though; but this can be adjusted if there is a actual use case that requires regular frequent updates; but the mappings are maintained by humans so entries are added slowly, and almost never change once added)



ZyFinity wrote: 06 Nov 2022, 23:00 One thing i noticed however is that file bot pulls a mapping for the show from anidb -> tvdb correctly, however then does a "reversemap" for whatever reason?
and then ends up getting a wrong match depending on the file layout.
The --mapper option allows you to match an Episode object as if it was another Episode object, but rename with the original Episode object:

Code: Select all

Start with A
Map A to M (for matching)
Reverse Map M to A (for renaming)
:idea: Please read the FAQ and How to Request Help.
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: support for XEM?

Post by rednoah »

I've re-created your file structure and ran a few tests:

Code: Select all

Anime/Attack on Titan The Final Season (2022) [anidb-16177]/Attack on Titan The Final Season - 1 - Judgement [1080p] [Subsplease].mp4
Anime/Attack on Titan The Final Season (2022) [anidb-16177]/Attack on Titan The Final Season - 2 - Sneak Attack [1080p] [Subsplease].mp4
Anime/Attack on Titan The Final Season (2022) [anidb-16177]/Attack on Titan The Final Season - 12 - The Dawn of Humanity [1080p] [Subsplease].mp4

In this case, you'll want to use --db AniDB and match files with AniDB information (i.e. no mapper) but then translate to TheTVDB episode information at format time by using the {db} binding (uses AnimeLists internally) in your --format expression. The --q option is very useful in this case because it'll allows us to force lookup-by-ID and skip search-by-name entirely:

Code: Select all

$ filebot -rename -r Anime --q "{ f.path.match(/anidb-([0-9]+)/) }" --db AniDB --output "TV Shows" --format "{ db.TheTVDB.plex.id }" -non-strict --action TEST --log INFO
[TEST] from [Anime/Attack on Titan The Final Season (2022) [anidb-16177]/Attack on Titan The Final Season - 1 - Judgement [1080p] [Subsplease].mp4] to [TV Shows/TV Shows/Attack on Titan {tvdb-267440}/Season 04/Attack on Titan - S04E17 - Judgment.mp4]
[TEST] from [Anime/Attack on Titan The Final Season (2022) [anidb-16177]/Attack on Titan The Final Season - 12 - The Dawn of Humanity [1080p] [Subsplease].mp4] to [TV Shows/TV Shows/Attack on Titan {tvdb-267440}/Season 04/Attack on Titan - S04E28 - The Dawn of Humanity.mp4]
[TEST] from [Anime/Attack on Titan The Final Season (2022) [anidb-16177]/Attack on Titan The Final Season - 2 - Sneak Attack [1080p] [Subsplease].mp4] to [TV Shows/TV Shows/Attack on Titan {tvdb-267440}/Season 04/Attack on Titan - S04E18 - Sneak Attack.mp4]



In this case, --mapper does not work well because [anidb-16177] isn't used during episode matching, and the ID is meaningless to TheTVDB for series lookup purposes, and the file name doesn't match AniDB primary naming either:

Code: Select all

$ filebot -rename -r Anime --db TheTVDB --mapper AnimeList.AniDB --output "TV Shows" --format "{ plex.id }" -non-strict --action TEST --log INFO
[TEST] from [Anime/Attack on Titan The Final Season (2022) [anidb-16177]/Attack on Titan The Final Season - 12 - The Dawn of Humanity [1080p] [Subsplease].mp4] to [TV Shows/TV Shows/Attack on Titan {tvdb-267440}/Season 01/Attack on Titan - S01E12 - Wound - The Struggle for Trost.mp4]
e.g. In this case, the --mapper instructs FileBot to match "Shingeki no Kyojin: The Final Season (2022) - 12" but that's completely different from the file name "Attack on Titan The Final Season - 12 - The Dawn of Humanity" and so FileBot cannot arrive at the correct match based on series name or episode title, and so it'll just end up at some 12th Episode of some series vaguely named Attack on Titan:

Code: Select all

Map [Attack on Titan - 4x28 - The Dawn of Humanity] to [Shingeki no Kyojin: The Final Season (2022) - 12]
:idea: Please read the FAQ and How to Request Help.
ZyFinity
Posts: 10
Joined: 28 Jan 2022, 10:25

Re: [DEPRECATED] support for XEM?

Post by ZyFinity »

Thanks for that, I actually wrote up a whole reply to this however I accidentally closed the tab.

Anyways to keep it short,
What I don't understand is why in:
In this case, --mapper does not work well because [anidb-16177] isn't used during episode matching, and the ID is meaningless to TheTVDB for series lookup purposes, and the file name doesn't match AniDB primary naming either:
Works perfectly fine (I tested your config myself), while my config using AniDB as the DB, AnimeList.TheTVDB as the mapper and {plex} as the format expression led to the incorrect results as seen in my previous post?
They seem to be doing or atleast be tasked to do the same thing, right? or am I missing something?
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: [DEPRECATED] support for XEM?

Post by rednoah »

--mapper and --format are programmable elements so they're not tasked with anything specific, but can be used for anything. --mapper comes in before matching and only applies to matching. --format is just some arbitrary display value generated by your code for a given match object.

:idea: This may be important if you need xattr metadata to match the file names. Because if you do --db AniDB --format {db.TheTVDB.plex} then FileBot is working with AniDB episode information and thus write that xattr metadata accordingly, which then won't match the file names, because FileBot is completely unaware of what your format is doing.




:?: Do you get different results from me? What is your command? What is your console output?

Code: Select all

$ filebot -rename -r Anime --db TheTVDB --mapper AnimeList.AniDB --output "TV Shows" --format "{ plex.id }" -non-strict --action TEST --log INFO
[TEST] from [Anime/Attack on Titan The Final Season (2022) [anidb-16177]/Attack on Titan The Final Season - 12 - The Dawn of Humanity [1080p] [Subsplease].mp4] to [TV Shows/TV Shows/Attack on Titan {tvdb-267440}/Season 01/Attack on Titan - S01E12 - Wound - The Struggle for Trost.mp4]
:idea: Attack on Titan The Final Season - 12 - The Dawn of Humanity => Attack on Titan - S01E12 - Wound - The Struggle for Trost is wrong. --db TheTVDB --mapper AnimeList.AniDB does mean "I want TheTVDB episode information; but match files as if I was working with AniDB episode information" which does make sense but unfortunately doesn't work in this specific use case, though its known to work in most use cases.



:idea: --db AniDB --mapper AnimeList.TheTVDB means that you want AniDB naming but have TheTVDB SxE numbered files:

Code: Select all

$ filebot -rename *.mp4 --db AniDB --q 16177 --mapper AnimeList.TheTVDB --format "{ plex.id }" -non-strict --action TEST --log INFO
[TEST] from [Attack on Titan - 4x17.mp4] to [Anime/Shingeki no Kyojin - The Final Season (2022) {anidb-16177}/Shingeki no Kyojin - The Final Season (2022) - 01 - Judgment.mp4]
:!: I had to add --q 16177 because AniDB has too many "Attack on Titan" series entries, and a search for "Attack on Titan" unfortunately does not include Shingeki no Kyojin - The Final Season (2022) in the Top 5 results.

:idea: TheTVDB => AniDB translation is not a common use case, and is not your use case as far as I can tell. You have AniDB naming and want TheTVDB naming, so you mean to convert from AniDB to TheTVDB.
:idea: Please read the FAQ and How to Request Help.
ZyFinity
Posts: 10
Joined: 28 Jan 2022, 10:25

Re: [DEPRECATED] support for XEM?

Post by ZyFinity »

So I get these results from the following:

Code: Select all

filebot -rename -r Anime --db AniDB--mapper AnimeList.TheTVDB --output "TV Shows" --format "{plex}" -non-strict --action TEST --log INFO
[TEST] from [F:\ANIME\testRename\Attack on Titan The Final Season (2022) [anidb-16177]\Attack on Titan The Final Season - 1 - Judgment [1080p] [Subsplease].mkv] to [F:\ANIME\testRename\Attack on Titan The Final Season (2022) [anidb-16177]\Attack on Titan The Final Season (2022) - S04E22 - Thaw.mkv]
[TEST] from [F:\ANIME\testRename\Attack on Titan The Final Season (2022) [anidb-16177]\Attack on Titan The Final Season - 12 - The Dawn of Humanity [1080p] [Subsplease].mkv] to [F:\ANIME\testRename\Attack on Titan The Final Season (2022) [anidb-16177]\Attack on Titan The Final Season (2022) - S04E28 - The Dawn of Humanity.mkv]
[TEST] from [F:\ANIME\testRename\Attack on Titan The Final Season (2022) [anidb-16177]\Attack on Titan The Final Season - 2 - Sneak Attack [1080p] [Subsplease].mkv] to [F:\ANIME\testRename\Attack on Titan The Final Season (2022) [anidb-16177]\Attack on Titan The Final Season (2022) - S04E18 - Sneak Attack.mkv]

While doing what you said does work:

Code: Select all

filebot -rename -r Anime --q "{ f.path.match(/anidb-([0-9]+)/) }" --db AniDB --output "TV Shows" --format "{ db.TheTVDB.plex.id }" -non-strict --action TEST --log INFO
[TEST] from [Anime/Attack on Titan The Final Season (2022) [anidb-16177]/Attack on Titan The Final Season - 1 - Judgement [1080p] [Subsplease].mp4] to [TV Shows/TV Shows/Attack on Titan {tvdb-267440}/Season 04/Attack on Titan - S04E17 - Judgment.mp4]
[TEST] from [Anime/Attack on Titan The Final Season (2022) [anidb-16177]/Attack on Titan The Final Season - 12 - The Dawn of Humanity [1080p] [Subsplease].mp4] to [TV Shows/TV Shows/Attack on Titan {tvdb-267440}/Season 04/Attack on Titan - S04E28 - The Dawn of Humanity.mp4]
[TEST] from [Anime/Attack on Titan The Final Season (2022) [anidb-16177]/Attack on Titan The Final Season - 2 - Sneak Attack [1080p] [Subsplease].mp4] to [TV Shows/TV Shows/Attack on Titan {tvdb-267440}/Season 04/Attack on Titan - S04E18 - Sneak Attack.mp4]
Do you get different results from me? What is your command? What is your console output?
Nope, I get the same results (shown above) and it works correctly.


:?: My question is, If I have a large library I want to convert from AniDB Absolute -> TVDB Season naming with every single anime having the anidb ID in the folder that I can extract with a groovy script. Is the method you posted of just having "{db.TheTVDB.plex.id}" in the format expression, the best way to go about it and the most reliable way?
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: [DEPRECATED] support for XEM?

Post by rednoah »

Yes, in your case using AniDB as primary database will likely work best, because that's what matches your files best.


Using --format to simply output TheTVDB file paths based on AniDB episode of information has the advantage that if the format doesn't print anything due to the {db} binding not working for the episode at hand, you will get nice errors instead of missing file path parts (i.e. the result of a format partially working and partially not working).


If you need TheTVDB episode information in xattr metadata you will have to reprocess with --db TheTVDB again just to fix the xattr metadata attached to each file.


Please let us know how it goes. We can only guess what the best solution might be, but can't know for sure until we run things for real. ;)
:idea: Please read the FAQ and How to Request Help.
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: [DEPRECATED] support for XEM?

Post by rednoah »

UPDATE: XEM

TheTVDB ➔ AniDB mapping not really useful for typical use cases due to API limitations.

e.g.

Code: Select all

TheTVDB::267440 | Attack on Titan | Season 4 | Episode 28
cannot be mapped to

Code: Select all

AniDB::16177 | Anime: Shingeki no Kyojin: The Final Season (2022) | Episode 12
In particular, the AniDB ID nor season-specific series name AniDB::16177 | Anime: Shingeki no Kyojin: The Final Season (2022) cannot be retrieved via the API. We can map to 6x12 but Season 6 in the context of AniDB means nothing, and there's no API to retrieve the corresponding AniDB series name and ID, and Episode 12 isn't useful because every AniDB series/season as a 12th episode, and so the AniDB series/season name is required to identify files.

The necessary information is available on the website, at least the season-specific AniDB series ID:
* https://thexem.info/xem/show/4469

But not via the machine-readable API:
* https://thexem.info/map/all?id=267440&origin=tvdb


e.g.

Code: Select all

$ filebot -list --q 267440 --db TheTVDB --format "{xem.AniDB} <= {episode}"
null - 01 <= Attack on Titan - 1x01 - To You, in 2000 Years: The Fall of Shiganshina (1)
...
Shingeki no Kyojin S2 - 01 <= Attack on Titan - 2x01 - Beast Titan
...
Shingeki no Kyojin S3 - 01 <= Attack on Titan - 3x01 - Smoke Signal
...
Shingeki no Kyojin S3 - 01 <= Attack on Titan - 3x13 - The Town Where Everything Began
...
Shingeki no Kyojin S4 - 01 <= Attack on Titan - 4x01 - The Other Side of the Sea
...
Shingeki no Kyojin S4 - 01 <= Attack on Titan - 4x17 - Judgment
...





EDIT:

FileBot r9354 updates {xem} and removes AniDB season/series name guesswork to make {xem} work or not work in a more predictable manner. TheTVDB ➔ AniDB mapping now passes back API records unmodified. AniDB ➔ TheTVDB mapping will now throw an error instead of maybe working, maybe not working, maybe working incorrectly.

Code: Select all

$ filebot -list --q 267440 --db TheTVDB --format "{xem.AniDB} <= {episode}"
Attack on Titan - 1x01 <= Attack on Titan - 1x01 - To You, in 2000 Years: The Fall of Shiganshina (1)
...
Attack on Titan - 2x01 <= Attack on Titan - 2x01 - Beast Titan
...
Attack on Titan - 3x01 <= Attack on Titan - 3x01 - Smoke Signal
...
Attack on Titan - 5x01 <= Attack on Titan - 4x01 - The Other Side of the Sea
...
Attack on Titan - 6x01 <= Attack on Titan - 4x17 - Judgment
...
:!: The Attack on Titan - 6x01 mapping would generally be useless for matching AniDB named files, because AniDB named files don't have season numbers, and instead identify "seasons" via a unique series name per "season".
:idea: Please read the FAQ and How to Request Help.
Locked