support for XEM?

All your suggestions, requests and ideas for future development
howdymilsap
Posts: 34
Joined: 14 Jul 2018, 17:22

support for XEM?

Post by howdymilsap » 16 Jul 2018, 20:45

Hi,

I am wondering if support for XEM has been considered (to augment tvdb for example). I have run across several shows that have discrepancies between naming/numbering from the scene and what thetvdb lists. These are always fixable in a manual way, but having it automated would be even better.

many thanks.


-hm

User avatar
rednoah
The Source
Posts: 15956
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: support for XEM?

Post by rednoah » 16 Jul 2018, 20:52

1.
What is XEM?


2.
Are you using the amc script? The Advanced File-Tuning section will explain a good solution for making things pretty much error-proof for newly aired episodes.
:idea: Please read the FAQ and How to Request Help.

howdymilsap
Posts: 34
Joined: 14 Jul 2018, 17:22

Re: support for XEM?

Post by howdymilsap » 16 Jul 2018, 21:15

XEM is the cross entity map for tv shows.

http://thexem.de/

Basically, discrepancies in tvdb, anidb and the scene are managed here. The episode mismatches for shows like Mythbusters and SNL have already been re-mapped in XEM. Season 43 of SNL is a great example of the issue, where the scene episode numbering is off by 2 compared to tvdb. This is because of how tvdb handled 2 specials that aired this season.

Fixing this is not that difficult, but i have to manually edit the files and change the filename. if we could use XEM, this wouldn't be necessary as someone has already "fixed" this mismatch in the XEM data. Just for clarity, the mismatch is in the scene numbering and what tvdb thinks that episode is. For example, SNL S43E23 is Tina Fey/Nicki Minaj per the scene numbering. SNL S43E23 does not exist in tvdb, S43E21 is the Tina Fey/Nicki Minaj episode in their database.

Lastly, I am using your AMC script (it is fantastic) and am using the advanced file tuning suggestions. This doesn't remedy this particular situation however.


-hm

User avatar
rednoah
The Source
Posts: 15956
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: support for XEM?

Post by rednoah » 16 Jul 2018, 21:30

A few examples of original file names that get matched incorrectly by the amc script would be helpful.
:idea: Please read the FAQ and How to Request Help.

howdymilsap
Posts: 34
Joined: 14 Jul 2018, 17:22

Re: support for XEM?

Post by howdymilsap » 16 Jul 2018, 23:32

filebot output is not all that useful (in this example) as the file is seemingly properly identified as Saturday Night live S43E21 and filebot using tvdb applies S43E21 metadata.

However, the scene numbering for S43E21 is really the Donald Glover/Childish Gambino episode. the TVDB has this as the Tina Fey/Nicki Minage episode instead.

Here is theTVDB page for this episode https://www.thetvdb.com/series/saturday ... es/6681723
Here is the official NBC page indicating S43E21 is the Donald Glover episode https://www.nbc.com/saturday-night-live ... er/3712034
Here is theXEM page showing how it fixes this issue http://thexem.de/xem/show/4294

Finally, here is the output from when I run filebot:

Code: Select all

Run script [fn:amc] at [Mon Jul 16 23:43:36 CEST 2018]

Parameter: ut_dir = /home/hd1/torrents/data/Saturday.Night.Live.S43E21.1080p.HEVC.x265-MeGusta/Saturday.Night.Live.S43E21.1080p.HEVC.x265-MeGusta.mkv/.

Parameter: subtitles = en

Parameter: artwork = false

Parameter: extractFolder = /home/hd1/torrents/extracted

Parameter: skipExtract = n

Parameter: ut_label = TV

Parameter: music = y

Parameter: clean = y

Input: /home/hd1/torrents/data/Saturday.Night.Live.S43E21.1080p.HEVC.x265-MeGusta/Saturday.Night.Live.S43E21.1080p.HEVC.x265-MeGusta.mkv

Group: [tvs:saturday night live] => [Saturday.Night.Live.S43E21.1080p.HEVC.x265-MeGusta.mkv]

Get [English] subtitles for 1 files

Looking up subtitles by hash via OpenSubtitles

No matching subtitles found: /home/hd1/torrents/data/Saturday.Night.Live.S43E21.1080p.HEVC.x265-MeGusta/Saturday.Night.Live.S43E21.1080p.HEVC.x265-MeGusta.mkv

Rename episodes using [TheTVDB]

Auto-detected query: [Saturday Night Live]

Fetching episode data for [Saturday Night Live]

Fetching episode data for [SNL Poland]

Fetching episode data for [Saturday Night Live (JP)]

Fetching episode data for [Saturday Night Live Korea]

Fetching episode data for [Weekend Update]

[OVERRIDE] Delete [/home/hd1/media/TV Shows/Saturday Night Live/Season 43/Saturday Night Live - S43E21 - Tina Fey Nicki Minaj.mkv]

[COPY] From [/home/hd1/torrents/data/Saturday.Night.Live.S43E21.1080p.HEVC.x265-MeGusta/Saturday.Night.Live.S43E21.1080p.HEVC.x265-MeGusta.mkv] to [/home/hd1/media/TV Shows/Saturday Night Live/Season 43/Saturday Night Live - S43E21 - Tina Fey Nicki Minaj.mkv]

Processed 1 files
and here is a screenshot of this episode playing in Plex showing Donald Glover. https://imgur.com/a/eii2huU

this is a screengrab from inside Plex, showing this episode identified as Tina fey https://imgur.com/a/NpdUXns

User avatar
rednoah
The Source
Posts: 15956
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: support for XEM?

Post by rednoah » 17 Jul 2018, 05:57

Thanks for the logs. I always need the original file paths so I can add them to my test dataset.

Support for XEM won't make it into the upcoming release, but maybe the next.
:idea: Please read the FAQ and How to Request Help.

bradleytf
Posts: 10
Joined: 11 Feb 2013, 17:48

Re: support for XEM?

Post by bradleytf » 02 Jan 2019, 07:02

+1 for interest in this.
Any update on if we'll see it in a future release?

User avatar
rednoah
The Source
Posts: 15956
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: support for XEM?

Post by rednoah » 02 Jan 2019, 09:45

It's on my TODO list, but I'm not quite sure on how to implement this well in a consistent way that fits into how things currently work. It won't be in 4.8.5 but possibly in the next release, especially if if there's lots of demand for this particular feature.
:idea: Please read the FAQ and How to Request Help.

bradleytf
Posts: 10
Joined: 11 Feb 2013, 17:48

Re: support for XEM?

Post by bradleytf » 03 Feb 2019, 17:00

Great thanks!
I'd note that I don't think you'll have a lot of vocal requests, just because I think most people have no idea it exists.
But it would certainly help a lot of people with their problems with having to manually correct mismatches.
Demands there, people just aren't aware that there's a centralized solution to it.

User avatar
rednoah
The Source
Posts: 15956
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: support for XEM?

Post by rednoah » 03 Feb 2019, 17:31

I usually ask people to include file paths when reporting mismatches, so that should give some indication what kind of files are troublesome and if XEM could be a solution for those.
:idea: Please read the FAQ and How to Request Help.

devster
Posts: 275
Joined: 06 Jun 2017, 22:56

Re: support for XEM?

Post by devster » 22 Mar 2019, 15:21

+1 for this feature
American Dad is an example:

Code: Select all

$ filebot -rename --action test American\ Dad\ -\ 108\ Bullocks\ to\ Stan.mkv --format '{plex}' --q 'American Dad' --filter 's == 1' --db TheTVDB
Rename episodes using [TheTVDB]
Fetching episode data for [American Dad!]
Apply filter [s == 1] on [263] items
Include [American Dad! - 1x01 - Pilot]
Include [American Dad! - 1x02 - Threat Levels]
Include [American Dad! - 1x03 - Stan Knows Best]
Include [American Dad! - 1x04 - Francine's Flashback]
Include [American Dad! - 1x05 - Roger Codger]
Include [American Dad! - 1x06 - Homeland Insecurity]
Include [American Dad! - 1x07 - Deacon Stan, Jesus Man]
Failed to match files to episode data
Failure (°_°)
Scene releases include all episodes up to 23 in the first season, TVDB splits first and second season in 7 episodes and the rest. XEN contains the mapping from scene to TVDB http://thexem.de/xem/show/4606
I only work in black and sometimes very, very dark grey. (Batman)

crawfs
Posts: 20
Joined: 01 Feb 2019, 09:26

Re: support for XEM?

Post by crawfs » 30 Apr 2019, 08:57

Adding my +1 to this, I found out only today that sonarr uses thexem to map anime from anidb to their TVDB counterpart which I think would be a wonderful feature for filebot.

User avatar
rednoah
The Source
Posts: 15956
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: support for XEM?

Post by rednoah » 30 Apr 2019, 09:55

crawfs wrote:
30 Apr 2019, 08:57
Adding my +1 to this, I found out only today that sonarr uses thexem to map anime from anidb to their TVDB counterpart which I think would be a wonderful feature for filebot.
Please post the original file name, and the desired file name, so I can add that to my test data.
:idea: Please read the FAQ and How to Request Help.

crawfs
Posts: 20
Joined: 01 Feb 2019, 09:26

Re: support for XEM?

Post by crawfs » 30 Apr 2019, 11:38

So I think a good example of what I'm talking about is a series like Monogatari:

https://www.thetvdb.com/series/monogatari/seasons/all

Where season 1 is known as Bakemonogatari, season 2 is Nisemonogatari etc, each season that is listed in TVDB is actually a completely seperate series so if you want the series named according TVDB metadata AMC just doens't really know what to do with a file called Bakemonogatari 01.mkv or Nisemonogatari 01.mkv (Monogatari S01E01 and S02E01 respectively).

Cross referencing anime on AniDB with thexem entries should help to solve that problem to rename the file to match TVDB metadata.

Have I effectively answered your request? When I made that comment I didn't have anything in mind in particular just the fact that aniDB and tvdb entries vary wildly and anime releases are more commonly named in such a way that makes matching to anidb more consistent than matching to TVDB.

User avatar
rednoah
The Source
Posts: 15956
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: support for XEM?

Post by rednoah » 30 Apr 2019, 12:59

Alright, we wanna rename this:

Code: Select all

Nisemonogatari 01.mkv
to this:

Code: Select all

Monogatari S02E01.mkv
By first matching to AniDB as per usual, and then using XEM to find the corresponding TheTVDB entry.
:idea: Please read the FAQ and How to Request Help.

crawfs
Posts: 20
Joined: 01 Feb 2019, 09:26

Re: support for XEM?

Post by crawfs » 30 Apr 2019, 22:50

Yeah pretty much

User avatar
rednoah
The Source
Posts: 15956
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: support for XEM?

Post by rednoah » 01 May 2019, 04:40

I'd like to also appropriate this thread for possible solution.

* How would XEM support look like on the CLI?
* How would it look like in the GUI?
:idea: Please read the FAQ and How to Request Help.

devster
Posts: 275
Joined: 06 Jun 2017, 22:56

Re: support for XEM?

Post by devster » 01 May 2019, 11:21

At first glance XEM doesn't seem to qualify as standalone DB as it links between them mainly:
  1. provide mapping between scene releases and databases
  2. provide source of truth between database for same show with different names according to season
  3. provide mapping between AniDB order and TheTVDB order
  4. provide name mapping between different shows
XEN allows specifying both a source and a destination, so the double colon notation could be used like this
AniDB::XEN::TheTVDB.
This could potentially solve point 2, 3 but likely not 1 and 4.
For number 4 the above notation would be used like AniDB::XEN to specify that the matching process should query XEN for all available names before renaming, however the response doesn't seem to return a preferred name or be filterable by show (it returns all names in the XEN database).
Still no ideas about point 1 which is the fundamental one (mapping a single file to a specific episode or multiple episodes).
I only work in black and sometimes very, very dark grey. (Batman)

kim
Power User
Posts: 738
Joined: 15 May 2014, 16:17

Re: support for XEM?

Post by kim » 01 May 2019, 18:56

American Dad!
http://thexem.de/xem/show/4606

Update? =
2012-03-26 20:47:14 UTC / 2019-02-18 03:47:19 UTC
if so and not updated the day a new episode comes out it's USELESS

But I made this so maybe you can use it:

Code: Select all

{
def currentSeason = 16
def currentEpisode = 4
def tvdbSeason, tvdbEpisode
def id = 73141
def show = 'American Dad!'

def slurper = new groovy.json.JsonSlurper()
def url = "http://thexem.de/map/all?id=$id&origin=tvdb"
def data = any{slurper.parseText( new URL( url ).text ).data
	.find{ (it.scene.season == currentSeason && it.scene.episode == currentEpisode) || (it.tvdb.season == currentSeason && it.tvdb.episode == currentEpisode)}
	.collectEntries{println it; [ (it.key) :  [ 'season' : it.getValue().season, 'episode' : it.getValue().episode ] ] }}{}

any{"$show - (TVDb - Season ${data.tvdb.season.pad(2)} Episode ${data.tvdb.episode.pad(2)}) vs (Scene - Season ${data.scene.season.pad(2)} Episode ${data.scene.episode.pad(2)})"}{'Wrong info'}
}
Output:
American Dad! - (TVDb - Season 16 Episode 04) vs (Scene - Season 15 Episode 04)
FYI: when scaped from tvdb just replace the first part to:

Code: Select all

def currentSeason = s
def currentEpisode = e
def tvdbSeason, tvdbEpisode
def id = id
def show = n

kim
Power User
Posts: 738
Joined: 15 May 2014, 16:17

Re: support for XEM?

Post by kim » 01 May 2019, 19:50

Anime
you can do a lookup for "alias" e.g. Nisemonogatari
http://thexem.de/map/allNames?origin=anidb

and then
http://thexem.de/map/all?id=6327&origin=anidb


Saturday Night Live
http://thexem.de/xem/show/4294
it only has season 43 ? USELESS

User avatar
rednoah
The Source
Posts: 15956
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: support for XEM?

Post by rednoah » 02 May 2019, 03:11

1.
My idea is a new xem.[AniDB, TheTVDB, Scene, ...] binding, so that we can map the output via xem for naming, but use the original value for matching.


2.
Additionally, a new --xem option that'll let you re-map each original to a new episode, or multi-episode, or nothing at all, before matching, via Groovy code and existing bindings of course:

e.g. map to itself, and effectively do nothing

Code: Select all

--xem episode
e.g. allow arbitrary Groovy code to come up with new Episode objects:

Code: Select all

--xem "new Episode(/Name/, 1, 1, /Title/)"

Code: Select all

--xem "episode.derive(1, 1)"
e.g. make the standard use case easy:

Code: Select all

--xem "xem.scene"

3.
A use case for matching with scene, but renaming with original TheTVDB data, might then look like this:

Code: Select all

--xem "xem.scene" --format "{xem.thetvdb.plex}"
Or just this, depending on whatever we decide to pass the xem'ed episodes on to the formatter, or really just use them for matching, but reverse-map them to the original Episode object before formatting:

Code: Select all

--xem "xem.scene" --format "{plex}"
:idea: Please read the FAQ and How to Request Help.

crawfs
Posts: 20
Joined: 01 Feb 2019, 09:26

Re: support for XEM?

Post by crawfs » 04 May 2019, 01:47

Is your posted example solution relying on thexem for matching in that case? That is to say if in your example the show you're matching to doesn't have an entry in thexem will it fail? Or will it fallback to TVDB/aniDB?

Because thexem only does mappings for shows where actually required so it won't be a complete catalogue like TVDB for example.

Just making sure you are aware of that aspect.

User avatar
rednoah
The Source
Posts: 15956
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: support for XEM?

Post by rednoah » 04 May 2019, 03:47

I assume the implementation of {xem} would just return the original episode object if it can't be mapped to a new one.
:idea: Please read the FAQ and How to Request Help.

crawfs
Posts: 20
Joined: 01 Feb 2019, 09:26

Re: support for XEM?

Post by crawfs » 04 May 2019, 07:01

Yeah that sounds pretty much ideal.

User avatar
rednoah
The Source
Posts: 15956
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: support for XEM?

Post by rednoah » 06 May 2019, 12:40

How would I match Owarimonogatari (2017) - 01 to Monogatari - S05E01 using the available API methods?

I only get back numbers. But I also need to somehow match AniDB ID to TheTVDB ID / SEASON.

e.g.
http://thexem.de/map/all?id=13033&origin=anidb
http://thexem.de/map/single?id=13033&or ... absolute=1

How do I know which "Episode 1" belongs to series AniDB::13033 since I'm not sure how to make sense of the numbers.
:idea: Please read the FAQ and How to Request Help.

Post Reply