[DEPRECATED] support for XEM?

All your suggestions, requests and ideas for future development
devster
Posts: 417
Joined: 06 Jun 2017, 22:56

Re: support for XEM?

Post by devster »

It doesn’t, but it was my understanding that new versions made the match implicitly, via AnimeList.
See viewtopic.php?f=6&t=6037&start=50#p44526
I only work in black and sometimes very, very dark grey. (Batman)
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: support for XEM?

Post by rednoah »

{sc} and {sy} are based on {episodelist} so it doesn't work if --db AniDB is being used.

:idea: Implicit conversion is only implemented for {s} and {sxe} and {s00e00} at this point.

:arrow: If {sc} and {sy} is desired, then I recommend using --db TheTVDB, optionally with --mapper AnimeList.AniDB to help with matching AniDB-named files.


EDIT:

Fixed {sy} to work for AniDB data. No mapper involved. Just the years from the first to the last episode of the given AniDB series.
:idea: Please read the FAQ and How to Request Help.
devster
Posts: 417
Joined: 06 Jun 2017, 22:56

Re: support for XEM?

Post by devster »

rednoah wrote: 06 Jul 2019, 02:37 {sc} and {sy} are based on {episodelist} so it doesn't work if --db AniDB is being used.
For the sake of argument, why is {episodelist} undefined for AniDB?
rednoah wrote: 06 Jul 2019, 02:37 :idea: Implicit conversion is only implemented for {s} and {sxe} and {s00e00} at this point.
Use case would be the following:

Code: Select all

	{
		Integer count = any{ sc }{ 0 }
		if (episode.special) {
			"S$special"
		} else if (count > 1) {
			s00e00
		} else {
			"EP" + absolute
		}
	}
As this is is not foreseen, how would I use the amc.groovy script to have a single season folder for all shows in the same bucket (One-Punch Man and One-Punch Man S2), I mean, is there a way to add the mapper option to the script? Is this doable with ut_label=anime?
I only work in black and sometimes very, very dark grey. (Batman)
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: support for XEM?

Post by rednoah »

1.
Sounds like you don't want AniDB series organization. I recommend using --db TheTVDB so all seasons belong to the same series {n}.


2.
If you are set on using --db AniDB for reasons I can't imagine, then you can access the TheTVDB series name like so, which would be the same for both One Punch man series seasons:

Code: Select all

{AnimeList.TheTVDB.seriesName}

3.
The amc script will inherit the --mapper option, and that's a Groovy expressions, so you can check the label, or other things, and map episodes accordingly:

Code: Select all

--mapper "label =~ /anime/ ? [episode, AnimeList.TheTVDB] : [episode, XEM.AniDB]"
:idea: Consider multiple mappings, derived from different sources, based on the label parameter. Not sure if this actually makes sense though. :lol:


4.
{sc} is always undefined for AniDB, since all season values will be undefined. {sc} is going to be 1 or more for TheTVDB data since there's generally at least one season for any give series.

:idea: Are you using {sc} to check if the database is AniDB or TheTVDB? Because that's effectively what you're doing, and using {info.database} would be more readable. ;)
:idea: Please read the FAQ and How to Request Help.
devster
Posts: 417
Joined: 06 Jun 2017, 22:56

Re: support for XEM?

Post by devster »

rednoah wrote: 06 Jul 2019, 14:27 :idea: Are you using {sc} to check if the database is AniDB or TheTVDB? Because that's effectively what you're doing, and using {info.database} would be more readable. ;)
Not really, I only wanted to have absolute episode numbering for single-season animes, season-based for multi-season animes, even with different names. TheTVDB is a simpler solution, with the mapper as suggested on point 3, but I looked here since the category from the torrent client is Anime and amc.groovy defaults to AniDB for those.
I only work in black and sometimes very, very dark grey. (Batman)
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: support for XEM?

Post by rednoah »

Now that we have the --mapper feature, it's probably feasible to always force TV Mode / TheTVDB while using --mapper to make it work for AniDB-numbered files.
:idea: Please read the FAQ and How to Request Help.
devster
Posts: 417
Joined: 06 Jun 2017, 22:56

Re: support for XEM?

Post by devster »

I apologize for asking, but as source is no longer available, is it possible to have a recap of the available mappers and what they match to?
I only work in black and sometimes very, very dark grey. (Batman)
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: support for XEM?

Post by rednoah »

The --mapper option accepts a Groovy expression, that can yield Episode or Episode[] objects. (i.e. map on episode to one or more other episodes)


e.g. use and rename with TheTVDB episode data, but use AnimeList.AniDB episode mappings (if available) for matching:

Code: Select all

filebot -rename . --db TheTVDB --mapper AnimeList.AniDB -non-strict

--mapper expressions you will most likely find useful:

Code: Select all

AnimeList.AniDB

Code: Select all

AnimeList.TheTVDB

Code: Select all

XEM.TheTVDB

Code: Select all

XEM.AniDB
e.g. map to itself (i.e. identity mapper, do nothing)

Code: Select all

episode
e.g. re-number episode to Season 1 Episode (e+1) and match accordingly:

Code: Select all

episode.derive(1, e + 1)
e.g. use different mapper for different TV Shows:

Code: Select all

n =~ /Monogatari/ ? AnimeList.AniDB : episode
e.g. use multiple mappers at the same time, and let the matcher try to figure out the rest given the available options (NOTE: more options ➔ more room for error)

Code: Select all

[episode, AnimeList.AniDB, XEM.AniDB]
e.g. do your own thing entirely:

Code: Select all

[seriesName: /My Series/, season: 1, episode: 1, title: /Hello World/]

:idea: You can use the Format Editor to test and prototype --mapper expressions.


:idea: Time will tell which approach is best. The --mapper option allows you to do anything, but just like --filter, how to best use it probably depends on the exact use case, and maybe over time we can come up with a "standard" --mapper that works well enough for most use cases.
:idea: Please read the FAQ and How to Request Help.
devster
Posts: 417
Joined: 06 Jun 2017, 22:56

Re: support for XEM?

Post by devster »

Thanks, this may be worth a pinned post.
I only work in black and sometimes very, very dark grey. (Batman)
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: support for XEM?

Post by rednoah »

I guess I could paste this in into a --mapper man page, and then update as things change.
:idea: Please read the FAQ and How to Request Help.
crawfs
Posts: 24
Joined: 01 Feb 2019, 09:26

Re: support for XEM?

Post by crawfs »

Hey, do you have a solution for the fact that files named with absolute ordering which are from anime that have the show split into 3 separate seasons doesn't work with the mapper? Unfortunately it's a bit of a deal breaker for using this feature as it will likely break a lot of stuff for me.

To clarify this is the issue I raised in this comment: viewtopic.php?p=44573#p44573
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: support for XEM?

Post by rednoah »

The use cases discussed in June should work nowadays.

What have you tried so far? What doesn't work as expected?

I'll need specific examples, and logs, please so I can help you out with concrete solutions. ;)

:idea: I've been testing with the Monogatari series where 7 AniDB entities correlate to 7 seasons of a single TheTVDB entity. So it should generally work.
:idea: Please read the FAQ and How to Request Help.
crawfs
Posts: 24
Joined: 01 Feb 2019, 09:26

Re: support for XEM?

Post by crawfs »

Alright, I'll take another look and post based on how it resolves, I haven't tested it since back then.
Corey3443
Posts: 23
Joined: 03 Sep 2014, 02:55

Re: support for XEM?

Post by Corey3443 »

I just saw this thread and was wondering if the mapper function allows you to use —dB tvdb but keep the original Romanized Japanese title for anime? I was using a csv generated from XML to map titles but this looks like it could be cleaner.

An example would be great if this is possible.
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: support for XEM?

Post by rednoah »

Yes. Give me an example file path and example result, and I'll give you an example command to make that happen. ;)
:idea: Please read the FAQ and How to Request Help.
Corey3443
Posts: 23
Joined: 03 Sep 2014, 02:55

Re: support for XEM?

Post by Corey3443 »

"Boku no Hero Academia" is probably a good example. TVDB uses My Hero Academia

So I would be expecting it stored like \path\to\Boku no Hero Academia\Season 4\Boku.no.Hero.Academia.S04E01.A064.720p.GROUP.mkv

My current name string. I would, of course, add the path later, but currently, a bash script has been handling moving files as part of my download-task; but if i can get filebot naming the way I like it, ill change it so filebot does the move.

Code: Select all

{n.space('.').replace(/]/,'.')+'.'}{s00e00}{episode.special ? '.'+t.space('_') : ''}{'.'+vf}{any{'.'+source}{'.'+fn.match(/BD|Blu-ray/)}}{bitdepth == 10 ? '.10bit' : null}{any{'.'+group}{'.NOGRP'}}
Since sleeping on the idea, it got me thinking that will work for 90% of cases, but for shows like "Nanatsu no Taizai" that have different titles for each season, would it still be possible to resolve the seasons to just "Nanatsu no Taizai" rather than like "Nanatsu no Taizai - Kamigami no Gekirin" which is season 3?

ie: \path\to\Nanatsu no Taizai\Season 3\Nanatsu.no.Taizai.S03E01.720p.GROUP.mkv
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: support for XEM?

Post by rednoah »

e.g.

Code: Select all

{db.AniDB.primaryTitle}

filebot -list is a good tool for testing on a complete set of episodes:

Code: Select all

filebot -list --q "Boku no Hero Academia" --format "{db.AniDB.primaryTitle}/Season {s}/{db.AniDB.primaryTitle} - {s00e00} - {t}" --filter "e == 1" --log INFO
Boku no Hero Academia/Season 1/Boku no Hero Academia - S01E01 - Izuku Midoriya: Origin
Boku no Hero Academia (2017)/Season 2/Boku no Hero Academia (2017) - S02E01 - That's the Idea, Ochaco
Boku no Hero Academia (2018)/Season 3/Boku no Hero Academia (2018) - S03E01 - Game Start
Boku no Hero Academia (2019)/Season 4/Boku no Hero Academia (2019) - S04E01 - The Scoop on U.A. Class 1-A
:idea: The dynamic database binding {db} like so to get the AniDB romanji title: {db.AniDB.primaryTitle}

:idea: Note that --mapper isn't necessary for this use case if files are already named using TheTVDB SxE numbering, though {db} does of course utilize AniDB / TheTVDB mapping data internally.

:idea: Looks like "Boku no Hero Academia" is already a good example for multi-season Anime, since each season is a different entry on AniDB it it seems to be working correctly here.
:idea: Please read the FAQ and How to Request Help.
Corey3443
Posts: 23
Joined: 03 Sep 2014, 02:55

Re: support for XEM?

Post by Corey3443 »

So how about files with the original sourced names? I should have probably included them above instead of just "what I wanted". The above as mentioned already had SxE data which was sourced separately, so realistically not a fair comparison.

Here are a few examples (hopefully these will be varied enough to cover all bases):

Code: Select all

[Group] Boku no Hero Academia - 64 [720p].mkv (tvdb s4)
[Group] Nanatsu no Taizai - Kamigami no Gekirin - 01 [720p].mkv (tvdb s3)
[Group] Sword Art Online - Alicization - War of Underworld - 01 [720p].mkv (tvdb s4)
My main goal is to keep romanji titles while having access to sxe and absolute numbers for my naming scheme. I would also like to keep the main series romanji title for the series folder.

ie: "\path\to\sword art online\" no matter the season
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: support for XEM?

Post by rednoah »

In this case, the --mapper option will make sure you get the correct match according to AniDB naming and numbering, even though we're using TheTVDB (SxE Airdate Order) as database for naming and formatting:

Code: Select all

filebot -rename "Boku no Hero Academia - 64.mkv" --db TheTVDB --mapper AnimeList.AniDB -non-strict --action TEST --log INFO
[TEST] from [Boku no Hero Academia - 64.mkv] to [My Hero Academia - 4x01 - The Scoop on U.A. Class 1-A.mkv]

Now we just use the format expression discussed previously, and we get the desired destination path:

Code: Select all

--format "{db.AniDB.primaryTitle}/Season {s}/{db.AniDB.primaryTitle} - {s00e00} - {t}"

Code: Select all

[TEST] from [Boku no Hero Academia - 64.mkv] to [Boku no Hero Academia (2019)/Season 4/Boku no Hero Academia (2019) - S04E01 - The Scoop on U.A. Class 1-A.mkv]


EDIT:

You'll probably want to use replaceTrailingBrackets() on the AniDB/Romanji-based series name, at least on the folder level, to combine different AniDB entries (with distinct series names) into the same folder (if possible).

Code: Select all

db.AniDB.primaryTitle.replaceTrailingBrackets()

Code: Select all

filebot -rename "Boku no Hero Academia - 64.mkv" --db TheTVDB --mapper AnimeList.AniDB -non-strict --action TEST --log INFO --format "{db.AniDB.primaryTitle.replaceTrailingBrackets()}/Season {s}/{db.AniDB.primaryTitle} - {s00e00} - {t}"
:idea: Please read the FAQ and How to Request Help.
Corey3443
Posts: 23
Joined: 03 Sep 2014, 02:55

Re: support for XEM?

Post by Corey3443 »

I'll have a play and see if I can come up with any oddities. thanks rednoah =)

edit: doesn't look like it will do what I want for anything that has an alternative name for the season. ie: sword art online s3 title var is "Sword Art Online Alicization" instead of just "Sword Art Online".

It's fine, I'll just continue using what I had in place; I just thought I might be able to simplify the process.
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: support for XEM?

Post by rednoah »

:o I found a new use case for --mapper today:

Code: Select all

--mapper "episode.derive(0, 0)"
8-) Set SxE to 0x0 to let Match by Title take over:

Code: Select all

$ filebot -rename *.mkv --db TheTVDB --mapper "episode.derive(0, 0)" -non-strict --action TEST --log INFO
[TEST] from [South Park S01E01 Tegridy Farms.mkv] to [South Park - 22x04 - Tegridy Farms.mkv]
:arrow: viewtopic.php?f=4&t=11221
:idea: Please read the FAQ and How to Request Help.
ZyFinity
Posts: 10
Joined: 28 Jan 2022, 10:25

Re: support for XEM?

Post by ZyFinity »

@rednoah

I believe the XEM mapping is not working at the moment as XEM changed their domain from "thexem.de" to "thexem.info"
Where the mapping requests are getting "301 Moved Permanently" errors in the latest filebot version
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: support for XEM?

Post by rednoah »

FileBot r9352 changes the API endpoint to https://thexem.info/ so it should work again.


:?: While you're here. Are you actually using the {XEM} binding for some use case successfully? I've not found it useful for anything, since AnimeLists already takes care of AniDB/TheTVDB mappings and better at that.
:idea: Please read the FAQ and How to Request Help.
ZyFinity
Posts: 10
Joined: 28 Jan 2022, 10:25

Re: support for XEM?

Post by ZyFinity »

Is AnimeLists actually better and if so in what way?
I wanted to also ask why filebot only connects to a centralised "filebot hosted" animelists db that gets pulled monthly rather than just allowing us the user to download the database at our wanted set interval locally or even whenever there's a commit to the github repo or something. As I would be wanting to potentially have this setup with airing shows and be automated, however I guess this is where a fallback to the xem could come in handy.

Secondly, the reason I want this, is that I have a very large library sorted perfectly following AniDB with absolute numbering e.g. foldername: (animename [anidb-id])
What I've done is create a simple groovy script that extracts the anidb id from the folder name so that every anime should be perfectly mapped to the TVDB format.

One thing i noticed however is that file bot pulls a mapping for the show from anidb -> tvdb correctly, however then does a "reversemap" for whatever reason?
and then ends up getting a wrong match depending on the file layout.

I tested this by having the following dummy folder setup:

Code: Select all

/Attack on Titan The Final Season (2022) [anidb-16177]/
                                                                                   /Attack on Titan The Final Season - 1 - Judgement [1080p] [Subsplease].mkv
                                                                                   /Attack on Titan The Final Season - 2 - Sneak Attack [1080p] [Subsplease].mkv
                                                                                   /Attack on Titan The Final Season - 12 - The Dawn of Humanity [1080p] [Subsplease].mkv
And it matched I believe only episode 2 correctly and mismatched the others to other episodes within the same season. This test purposely included missing episodes for testing reasons.

When I changed the test so instead of episode 12 it was episode 3, it seemed to map as expected. or is it simply that the original naming of the episodes are sub optimal?
ZyFinity
Posts: 10
Joined: 28 Jan 2022, 10:25

Re: support for XEM?

Post by ZyFinity »

I just ran some tests, and it seems animelist mapped incorrectly here with that example folder structure above, what would the reasoning for that be?


Using THE XEM

Code: Select all

filebot.exe -script "C:/Users/Documents/AnidbToTvdb/anidbToTvdb.groovy" -non-strict --action TEST
Rename episodes using [AniDB] with [Absolute]
Lookup via [16177]
Fetching episode data for [16177]
└─ 12 episodes
Apply mapper [XEM.TheTVDB] on [12] episodes
Map [Attack on Titan The Final Season (2022) - 01 - Judgment] to [Attack on Titan - 1x01]
Map [Attack on Titan The Final Season (2022) - 02 - Sneak Attack] to [Attack on Titan - 1x02]
Map [Attack on Titan The Final Season (2022) - 03 - Two Brothers] to [Attack on Titan - 1x03]
Map [Attack on Titan The Final Season (2022) - 04 - Memories of the Future] to [Attack on Titan - 1x04]
Map [Attack on Titan The Final Season (2022) - 05 - From You, 2,000 Years Ago] to [Attack on Titan - 1x05]
Map [Attack on Titan The Final Season (2022) - 06 - Thaw] to [Attack on Titan - 1x06]
Map [Attack on Titan The Final Season (2022) - 07 - Sunset] to [Attack on Titan - 1x07]
Map [Attack on Titan The Final Season (2022) - 08 - Pride] to [Attack on Titan - 1x08]
Map [Attack on Titan The Final Season (2022) - 09 - Night of the End] to [Attack on Titan - 1x09]
Map [Attack on Titan The Final Season (2022) - 10 - Traitor] to [Attack on Titan - 1x10]
Map [Attack on Titan The Final Season (2022) - 11 - Retrospective] to [Attack on Titan - 1x11]
Map [Attack on Titan The Final Season (2022) - 12 - The Dawn of Humanity] to [Attack on Titan - 1x12]
Reverse Map [Attack on Titan - 1x01] to [Attack on Titan The Final Season (2022) - 01 - Judgment]
Reverse Map [Attack on Titan - 1x12] to [Attack on Titan The Final Season (2022) - 12 - The Dawn of Humanity]
Reverse Map [Attack on Titan - 1x02] to [Attack on Titan The Final Season (2022) - 02 - Sneak Attack]
[TEST] from [F:\ANIME\testRename\Attack on Titan The Final Season (2022) [anidb-16177]\Attack on Titan The Final Season - 1 - Judgment [1080p] [Subsplease].mkv] to [F:\ANIME\testRename\Attack on Titan The Final Season (2022) [anidb-16177]\Attack on Titan The Final Season (2022) - S04E17 - Judgment.mkv]
[TEST] from [F:\ANIME\testRename\Attack on Titan The Final Season (2022) [anidb-16177]\Attack on Titan The Final Season - 12 - The Dawn of Humanity [1080p] [Subsplease].mkv] to [F:\ANIME\testRename\Attack on Titan The Final Season (2022) [anidb-16177]\Attack on Titan The Final Season (2022) - S04E28 - The Dawn of Humanity.mkv]
[TEST] from [F:\ANIME\testRename\Attack on Titan The Final Season (2022) [anidb-16177]\Attack on Titan The Final Season - 2 - Sneak Attack [1080p] [Subsplease].mkv] to [F:\ANIME\testRename\Attack on Titan The Final Season (2022) [anidb-16177]\Attack on Titan The Final Season (2022) - S04E18 - Sneak Attack.mkv]
Processed 3 files
Done ?(?????)?

Using AnimeList

Code: Select all

filebot.exe -script "C:/Users/Documents/AnidbToTvdb/anidbToTvdb.groovy" -non-strict --action TEST
Rename episodes using [AniDB] with [Absolute]
Lookup via [16177]
Fetching episode data for [16177]
└─ 12 episodes
Apply mapper [AnimeList.TheTVDB] on [12] episodes
Map [Attack on Titan The Final Season (2022) - 01 - Judgment] to [Attack on Titan - 4x17]
Map [Attack on Titan The Final Season (2022) - 02 - Sneak Attack] to [Attack on Titan - 4x18]
Map [Attack on Titan The Final Season (2022) - 03 - Two Brothers] to [Attack on Titan - 4x19]
Map [Attack on Titan The Final Season (2022) - 04 - Memories of the Future] to [Attack on Titan - 4x20]
Map [Attack on Titan The Final Season (2022) - 05 - From You, 2,000 Years Ago] to [Attack on Titan - 4x21]
Map [Attack on Titan The Final Season (2022) - 06 - Thaw] to [Attack on Titan - 4x22]
Map [Attack on Titan The Final Season (2022) - 07 - Sunset] to [Attack on Titan - 4x23]
Map [Attack on Titan The Final Season (2022) - 08 - Pride] to [Attack on Titan - 4x24]
Map [Attack on Titan The Final Season (2022) - 09 - Night of the End] to [Attack on Titan - 4x25]
Map [Attack on Titan The Final Season (2022) - 10 - Traitor] to [Attack on Titan - 4x26]
Map [Attack on Titan The Final Season (2022) - 11 - Retrospective] to [Attack on Titan - 4x27]
Map [Attack on Titan The Final Season (2022) - 12 - The Dawn of Humanity] to [Attack on Titan - 4x28]
Reverse Map [Attack on Titan - 4x22] to [Attack on Titan The Final Season (2022) - 06 - Thaw]
Reverse Map [Attack on Titan - 4x28] to [Attack on Titan The Final Season (2022) - 12 - The Dawn of Humanity]
Reverse Map [Attack on Titan - 4x18] to [Attack on Titan The Final Season (2022) - 02 - Sneak Attack]
[TEST] from [F:\ANIME\testRename\Attack on Titan The Final Season (2022) [anidb-16177]\Attack on Titan The Final Season - 1 - Judgment [1080p] [Subsplease].mkv] to [F:\ANIME\testRename\Attack on Titan The Final Season (2022) [anidb-16177]\Attack on Titan The Final Season (2022) - S04E22 - Thaw.mkv]
[TEST] from [F:\ANIME\testRename\Attack on Titan The Final Season (2022) [anidb-16177]\Attack on Titan The Final Season - 12 - The Dawn of Humanity [1080p] [Subsplease].mkv] to [F:\ANIME\testRename\Attack on Titan The Final Season (2022) [anidb-16177]\Attack on Titan The Final Season (2022) - S04E28 - The Dawn of Humanity.mkv]
[TEST] from [F:\ANIME\testRename\Attack on Titan The Final Season (2022) [anidb-16177]\Attack on Titan The Final Season - 2 - Sneak Attack [1080p] [Subsplease].mkv] to [F:\ANIME\testRename\Attack on Titan The Final Season (2022) [anidb-16177]\Attack on Titan The Final Season (2022) - S04E18 - Sneak Attack.mkv]
Processed 3 files
Done ?(?????)?
Locked