support for XEM?

All your suggestions, requests and ideas for future development
devster
Posts: 332
Joined: 06 Jun 2017, 22:56

Re: support for XEM?

Post by devster » 06 May 2019, 14:58

Also the results seem wrong, ID 13033 absolute 01 should return tvdb season 5 episode 1 and 2, absolute 62 and 63
I only work in black and sometimes very, very dark grey. (Batman)

User avatar
rednoah
The Source
Posts: 16419
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: support for XEM?

Post by rednoah » 06 May 2019, 15:20

Yep, that's what I'm thinking. Just double checking with you guys.
:idea: Please read the FAQ and How to Request Help.

crawfs
Posts: 22
Joined: 01 Feb 2019, 09:26

Re: support for XEM?

Post by crawfs » 09 May 2019, 12:37

Unfortunately I don't know the answer to that, not much of a programmer myself...

It might be worth taking a look at sonarr by eithertaking a look at their source code or hang around their discord and figure out how sonarr handles it as that project is how I first learned that this is possible.

devster
Posts: 332
Joined: 06 Jun 2017, 22:56

Re: support for XEM?

Post by devster » 26 May 2019, 10:58

Found a possible solution:
http://thexem.de/map/allNames?origin=tv ... ultNames=1
the above link should return the entire list of shows but in a specific format.
The ID is still the basic one, corresponding to the default name (so for Monogatari/AniDB is 6327) however associated to that name there is an array of string and objects which map the default name (the string) and all alternative name to a season number (they correspond to the query strings above, default name can be skipped).
EDIT: I forgot to mention that -1 exists as a season and represents an alias of the default name.

Code: Select all

  "6327": [
    "Monogatari",
    {
      "Bakemonogatari": 1
    },
    {
      "Nisemonogatari": 2
    },
    {
      "Monogatari Series Second Season": 3
    },
    {
      "Owarimonogatari": 4
    },
    {
      "Owarimonogatari S2": 5
    },
    {
      "Owarimonogatari Second Season": 5
    }
  ]
So, in theory, Owarimonogatari S2 can be mapped to S5.
Requesting http://thexem.de/map/all?id=13033&origi ... ation=tvdb and filtering for season 5 should then yield the correct corresponding episodes (or multiple episodes separated by _2, _3 and so on).
By the way, thexem.de already caches responses so it may be useful to reduce the default cache time for these requests.
Final useful method is http://thexem.de/map/havemap?origin=tvdb which should return every ID which has a mapping associated.

A possible flow could be matching against AniDB as usual, with the resulting name, not ID, search for correspondence in TheXEM, replace series name with "default" found and add season info, alternatively use absolute numbering from theTVDB, there is also a "master" category but I haven't seen it represented.
I only work in black and sometimes very, very dark grey. (Batman)

User avatar
rednoah
The Source
Posts: 16419
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: support for XEM?

Post by rednoah » 27 May 2019, 09:07

FileBot r6419 now supports the --mapper <expression> option.

:!: Note that nothing specifically related to TheXEM is implemented at this point. (i.e. fairly useless unless you're fairly comfortable with writing Groovy code)

e.g. identity mapper:

Code: Select all

--mapper "episode"
e.g. constant mapper:

Code: Select all

--mapper "new net.filebot.web.Episode(/Series Name/, 1, 1, /Episode Title/)"
e.g. arbitrary custom mapper that you have written yourself:

Code: Select all

--mapper /path/to/mapper.groovy
:idea: Please read the FAQ and How to Request Help.

devster
Posts: 332
Joined: 06 Jun 2017, 22:56

Re: support for XEM?

Post by devster » 27 May 2019, 09:20

Where can I find more info on how it's supposed to work?
For example, does mapper accept a class? Does it need to implement any specific methods?
What would be the flow for a custom mapper?
Please forgive the barrage, I believe it could be helpful for users.
I only work in black and sometimes very, very dark grey. (Batman)

User avatar
rednoah
The Source
Posts: 16419
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: support for XEM?

Post by rednoah » 27 May 2019, 09:42

--mapper works exactly the same as --filter, except where --filter is expected to return type Boolean objects, --filter is expected to return type Episode objects.

* All the usual bindings, such as episode, n, s, e, t and friends work exactly the same way as in format / filter / exec expressions.

* The mapper is applied to each Episode object (unless it has already been excluded by the filter) then all the new Episode objects generated by your mapper are passed on to matching as usual.

* Just run with the useless examples from my last post, and then the console log will show you what's happening.



EDIT:

Here's another cute example:

Code: Select all

filebot -list --q "Firefly" --mapper "[seriesName: ny, season: s + 1, episode: e, title: /Episode / + absolute]"

Code: Select all

Firefly (2002) - 2x01 - Episode 2
Firefly (2002) - 2x02 - Episode 3
Firefly (2002) - 2x03 - Episode 6
Firefly (2002) - 2x04 - Episode 7
Firefly (2002) - 2x05 - Episode 8
...
* Add year to series name
* Add +1 to season number
* Replace episode title with absolute number
:idea: Please read the FAQ and How to Request Help.

devster
Posts: 332
Joined: 06 Jun 2017, 22:56

Re: support for XEM?

Post by devster » 27 May 2019, 11:20

Perfect, thanks, will play with it and see what happens.
However, trying to think about XEM, I'm not finding fields in the Episode class that refer to databases though, so, for example, if I changed season/episode but also mapped the season to another database entirely (AniDB > TheTVDB) the ID wouldn't match anymore (13033 becomes 102261 which is missing in AniDB).
Unless I misunderstood the "new Episode objects generated by your mapper are passed on to matching as usual". Is matching against the DB done before or after?
I only work in black and sometimes very, very dark grey. (Batman)

User avatar
rednoah
The Source
Posts: 16419
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: support for XEM?

Post by rednoah » 27 May 2019, 12:01

1. Retrieve episode objects from database
2. (Optional) Map episode objects to other episode objects
3. Match episode objects with file objects
4. Format matches


The episode objects you get as input are typically more complex than just [seriesName, season, episode, title] and so it might make sense to just copy over into the new Episode object all the fields you don't want to modify:

Code: Select all

Episode​(String seriesName, Integer season, Integer episode, String title, Integer absolute, Integer special, SimpleDate airdate, Integer id, SeriesInfo seriesInfo)
:arrow: https://www.filebot.net/docs/api/net/fi ... isode.html


:!: {d} requires episode airdate, {order} requires series id, etc, etc, etc. Depending on what information you keep, remove or change in the Episode object effects behavior later on.
:idea: Please read the FAQ and How to Request Help.

devster
Posts: 332
Joined: 06 Jun 2017, 22:56

Re: support for XEM?

Post by devster » 27 May 2019, 16:14

Got it now, I hope, thanks again for the explanation.
I only work in black and sometimes very, very dark grey. (Batman)

devster
Posts: 332
Joined: 06 Jun 2017, 22:56

Re: support for XEM?

Post by devster » 27 May 2019, 22:41

It's extremely rough and untested but it seems to work (barely):

Code: Select all

// test with filebot -list --q "Monogatari" --db AniDB --mapper xem.groovy
import groovy.json.JsonSlurper

String origin = anime ? "anidb" : "tvdb"
Integer seas
if (anime) {
    seas = 1
} else {
    seas = s
}
def spec = call{special}
def ep = call{e}
def baseURL = new URL("http://thexem.de")
def reqHeaders = [:]
def params = [
                "origin": origin
            ]
def query = params.collect { k, v -> "$k=$v" }.join('&')
def getResponse = new URL(baseURL, "/map/havemap?$query").get(reqHeaders)
String stringID = id.toString()
Map    json     = new JsonSlurper().parseText(getResponse.text)
def    item     = json.data.any{ it == stringID }
if (item) {
    def paramsName = [
                "origin": origin,
                "id": id,
                "defaultNames": 1,
            ]
    def  queryName = paramsName.collect { k, v -> "$k=$v" }.join('&')
    def getResName = new URL(baseURL, "/map/names?$queryName").get(reqHeaders)
    Map   jsonName = new JsonSlurper().parseText(getResName.text)
    def    mat     = jsonName.data.collect{
        if (it.value instanceof Map) {
            def name = it.value.entrySet().value
            [(it.key): name.flatten()]
        } else if (it.value instanceof String) {
            [(it.key): it.value]
        }
    }

    String  newN   = mat.findAll{ it.all }?.all.first()
    Integer foundS = mat.findAll{
		it.entrySet().value.any{ v -> v =~ /(?i)$n/ }
    }.first().entrySet().key.first().toInteger()
    Integer newS   = (foundS < 0) ? seas : foundS

    def paramsMap = [
                "origin": origin,
                "id": id,
                "season": newS,
                "episode": ep,
            ]
    def queryMap = paramsMap.collect { k, v -> "$k=$v" }.join('&')
    def getResponseMap = new URL(baseURL, "/map/single?$queryMap").get(reqHeaders)
    Map jsonMap = new JsonSlurper().parseText(getResponseMap.text)
    // assuming tvdb destination, could be included in the query
    def result  = jsonMap.data.entrySet().findAll{ it.key.matches(/tvdb.*/) }
    if (result.size() < 2) {
        return new net.filebot.web.Episode(newN, newS, result.first().value.episode, t, result.first().value.absolute, spec, d, id, series)
    } else {
        def multi = []
        for ( i in 0..result.size()-1 ) {
            multi << new net.filebot.web.Episode(newN, newS, result[i].value.episode, t, result[i].value.absolute, spec, d, id, series)
        }
        return new net.filebot.web.MultiEpisode(*multi)
    }
}
// hopefully return the episode untouched if not matched
return new net.filebot.web.Episode(n, seas, ep, t, absolute, spec, d, id, series)
Hopefully all multi-episodes are just multi-part because I couldn't find a way to merge titles.
I'm no groovy dev so forgive my horrible code.
I only work in black and sometimes very, very dark grey. (Batman)

User avatar
rednoah
The Source
Posts: 16419
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: support for XEM?

Post by rednoah » 28 May 2019, 04:57

Wow! Impressive! I'd have added the --mapper option much sooner (that part was easy) if I had known you'd do all the heavy lifting afterwards! :D

I'll play with it as well next week, and see what I can add into the core to make this a built-in feature in upcoming releases. :mrgreen:



EDIT:

And so the student has become the master! I had no idea you can use the spread operator to spread an array into a method as arguments! :shock: :o

Code: Select all

def x = [1, 2, 3]
def f = { a, b, c -> a * b * c }
f(*x)

EDIT 2:

I'd add a bit of caching as well, like so:

Code: Select all

def cache = Cache.getCache('xem', CacheType.Daily)

def url = 'https://www.filebot.net/update.xml'
def content = cache.text(url, String.&toURL).get()

println content
:idea: Please read the FAQ and How to Request Help.

devster
Posts: 332
Joined: 06 Jun 2017, 22:56

Re: support for XEM?

Post by devster » 28 May 2019, 07:17

Will try, thanks.
I just noticed my script seems to halt after the first result in the -list (e.g. for Firefly it prints only the first item).
Not sure what's happening.
I only work in black and sometimes very, very dark grey. (Batman)

User avatar
rednoah
The Source
Posts: 16419
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: support for XEM?

Post by rednoah » 28 May 2019, 07:24

Can you paste your command so I can try and see what's going on?
:idea: Please read the FAQ and How to Request Help.

devster
Posts: 332
Joined: 06 Jun 2017, 22:56

Re: support for XEM?

Post by devster » 28 May 2019, 08:00

Code: Select all

# this returns only the first 2 episodes despite XEM having 3 mappings, but it's an issue with AniDB only reporting the first 2 as regular episodes and the third as special.
# they also join titles on their own, so that shouldn't be a byproduct of the MultiEpisode object
filebot -list --q "Owarimonogatari Second Season" --db AniDB --mapper xem.groovy
# this should return episodes unchanged
filebot -list --q "Firefly" --db TheTVDB --mapper xem.groovy
I only work in black and sometimes very, very dark grey. (Batman)

User avatar
rednoah
The Source
Posts: 16419
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: support for XEM?

Post by rednoah » 28 May 2019, 08:20

I see, since the mapper will automatically get rid of duplicates for you, and since you're using the series id as episode id, you end up with only one episode object per series id:

Code: Select all

filebot -list --q "Owarimonogatari Second Season" --db AniDB --format "{id} | {episode.id}"
:arrow: Try using episode.id instead of id when creating episode objects.


e.g. tail tail xem.groovy

Code: Select all

    if (result.size() < 2) {
        return new net.filebot.web.Episode(newN, newS, result.first().value.episode, t, result.first().value.absolute, spec, d, episode.id, series)
    } else {
        def multi = []
        for ( i in 0..result.size()-1 ) {
            multi << new net.filebot.web.Episode(newN, newS, result[i].value.episode, t, result[i].value.absolute, spec, d, episode.id, series)
        }
        return new net.filebot.web.MultiEpisode(*multi)
    }
}
// hopefully return the episode untouched if not matched
return episode
:idea: Please read the FAQ and How to Request Help.

devster
Posts: 332
Joined: 06 Jun 2017, 22:56

Re: support for XEM?

Post by devster » 28 May 2019, 18:13

Code: Select all

// test with filebot -list --q "Monogatari" --db AniDB --mapper xem.groovy
import groovy.json.JsonSlurper
import net.filebot.Cache
import net.filebot.CacheType

Closure<Object> request = { Map headers = [:], String base = "http://thexem.de", String path, Map params ->
    Cache  cache    = net.filebot.Cache.getCache('xem', CacheType.Daily)
    URL    baseURL  = new URL(base)
    String query    = params.collect { k, v -> "$k=$v" }.join('&')
    Object response = new URL(baseURL, "$path?$query").get(headers)
    response
    // TODO: daily caching
    // def content = cache.text(url, String.&toURL).get()
}

String origin = anime ? "anidb" : "tvdb"
Integer seas = anime ? 1 : episode?.season

Object  hasMap  = request("/map/havemap", ["origin": origin])
Map     jHasMap = new JsonSlurper().parseText(hasMap.text)
Boolean item    = jHasMap.data.any{ it == id.toString() }
if (item) {
    Object names = request("/map/names", [
        "origin": origin,
        "id": id,
        "defaultNames": 1,
    ])
    Map       jName   = new JsonSlurper().parseText(names.text)
    ArrayList reflect = jName.data.collect{
        if (it.value instanceof Map) {
            def name = it.value.entrySet().value
            [(it.key): name.flatten()]
        } else if (it.value instanceof String) {
            [(it.key): it.value]
        }
    }

    // String  newN   = reflect.findAll{ it instanceof String }.first()
    String  newN   = reflect.findAll{ it.all }?.all.first()
    Integer foundS = reflect.findAll{
		it.entrySet().value.any{ v -> v =~ /(?i)$episode.seriesName/ }
    }.first().entrySet().key.first().toInteger()
    // Integer foundS = item.findAll{ it instanceof Map }*.find{ k, v -> k.match(/$n/) }.find{ it != null }?.value
    Integer newS   = (foundS < 0) ? seas : foundS

    Map old = [
        ep: episode.episode ? episode.episode : episode.special,
        se: episode.special ? 0 : newS,
    ]
    // assuming TVDB destination, could be included in the query
    Object mapping = request("/map/single", [
        "origin": origin,
        "id": id,
        "season": old.se,
        "episode": old.ep,
    ])
    Map jMapping = new JsonSlurper().parseText(mapping.text)
    // also assuming TVDB destination
    if (jMapping.data.isEmpty()) {
        return episode
    }
    def result  = jMapping.data.entrySet().findAll{ it.key.matches(/tvdb.*/) }
    if (result.size() < 2) {
        return new net.filebot.web.Episode(newN, newS, result.first().value.episode, episode?.title, result.first().value.absolute, episode?.special, episode?.airdate, episode.id, series)
    } else {
        def multi = []
        for ( i in 0..result.size()-1 ) {
            // hopefully all multi-episodes are just multi-part because I couldn't find a way to merge titles
            multi << new net.filebot.web.Episode(newN, newS, result[i].value.episode, episode?.title, result[i].value.absolute, episode?.special, episode?.airdate, episode.id, series)
        }
        return new net.filebot.web.MultiEpisode(*multi)
    }
}
// hopefully return the episode untouched if not matched
return episode
update, should work a bit better.
I only work in black and sometimes very, very dark grey. (Batman)

User avatar
rednoah
The Source
Posts: 16419
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: support for XEM?

Post by rednoah » 01 Jun 2019, 10:46

I've added the {xem} binding so you can now do this:

Code: Select all

XEM.TheTVDB

Code: Select all

filebot -list --q "Monogatari" --db AniDB --mapper XEM.TheTVDB
:idea: It should work exactly the same as the Groovy version written by devster so his Groovy implementation is probably more useful for debugging.

:?: How well does it work different series and corner cases? No idea. Fairly untested.
:idea: Please read the FAQ and How to Request Help.

kim
Power User
Posts: 797
Joined: 15 May 2014, 16:17

Re: support for XEM?

Post by kim » 02 Jun 2019, 18:17

How do I get this working ?
rednoah wrote:
28 May 2019, 04:57
I'd add a bit of caching as well, like so:

Code: Select all

def cache = Cache.getCache('xem', CacheType.Daily)

def url = 'https://www.filebot.net/update.xml'
def content = cache.text(url, String.&toURL).get()

println content
Error:
Expression yields empty value: No signature of method: java.lang.String.toURL() is applicable for argument types: (String) values: [https://www.filebot.net/update.xml]
Possible solutions: toURL(), toURL(), toURI(), toURI(), toSet(), toFile(java.lang.String)

User avatar
rednoah
The Source
Posts: 16419
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: support for XEM?

Post by rednoah » 03 Jun 2019, 03:24

How are you using this code?

I think I only tested this in the Groovy Pad script. Does it not work out if used inside a format?


EDIT:

This should work better:

Code: Select all

def cache = Cache.getCache('xem', CacheType.Daily)
def url = 'https://www.filebot.net/update.xml'
def content = cache.text(url, { new URL(it) }).get()
println content
:idea: Please read the FAQ and How to Request Help.

crawfs
Posts: 22
Joined: 01 Feb 2019, 09:26

Re: support for XEM?

Post by crawfs » 04 Jun 2019, 06:14

will --mapper work for AMC scripts? Or will it throw an error if it doesn't find a matching xem entry for an item it's trying to match?

User avatar
rednoah
The Source
Posts: 16419
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: support for XEM?

Post by rednoah » 04 Jun 2019, 09:39

Untested. Should work though.

What happens if there is no match is entirely up to your --mapper code.

The xem binding will either translate to xem mappings, or keep the episode object as is.
:idea: Please read the FAQ and How to Request Help.

User avatar
rednoah
The Source
Posts: 16419
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: support for XEM?

Post by rednoah » 07 Jun 2019, 21:56

I'm also looking into support for Anime Lists for mapping information:
https://github.com/ScudLee/anime-lists

e.g.

Code: Select all

filebot -list --q 14444 --db AniDB --mapper "AnimeLists.TheTVDB"
Apply mapper [AnimeLists.TheTVDB] on [10] items
Map [Attack on Titan Season 3 (2019) - 01 - The Town Where Everything Began] to [Attack on Titan Season 3 (2019) - 3x13 - The Town Where Everything Began]
Map [Attack on Titan Season 3 (2019) - 02 - Thunder Spears] to [Attack on Titan Season 3 (2019) - 3x14 - Thunder Spears]
Map [Attack on Titan Season 3 (2019) - 03 - Descent] to [Attack on Titan Season 3 (2019) - 3x15 - Descent]
Map [Attack on Titan Season 3 (2019) - 04 - Perfect Game] to [Attack on Titan Season 3 (2019) - 3x16 - Perfect Game]
Map [Attack on Titan Season 3 (2019) - 05 - Hero] to [Attack on Titan Season 3 (2019) - 3x17 - Hero]
Map [Attack on Titan Season 3 (2019) - 06 - Midnight Sun] to [Attack on Titan Season 3 (2019) - 3x18 - Midnight Sun]
Map [Attack on Titan Season 3 (2019) - 07 - The Basement] to [Attack on Titan Season 3 (2019) - 3x19 - The Basement]
Map [Attack on Titan Season 3 (2019) - 08 - Episode 8] to [Attack on Titan Season 3 (2019) - 3x20 - Episode 8]
Map [Attack on Titan Season 3 (2019) - 09 - Episode 9] to [Attack on Titan Season 3 (2019) - 3x21 - Episode 9]
Map [Attack on Titan Season 3 (2019) - 10 - Episode 10] to [Attack on Titan Season 3 (2019) - 3x22 - Episode 10]
Attack on Titan Season 3 (2019) - 3x13 - The Town Where Everything Began
Attack on Titan Season 3 (2019) - 3x14 - Thunder Spears
Attack on Titan Season 3 (2019) - 3x15 - Descent
Attack on Titan Season 3 (2019) - 3x16 - Perfect Game
Attack on Titan Season 3 (2019) - 3x17 - Hero
Attack on Titan Season 3 (2019) - 3x18 - Midnight Sun
Attack on Titan Season 3 (2019) - 3x19 - The Basement
Attack on Titan Season 3 (2019) - 3x20 - Episode 8
Attack on Titan Season 3 (2019) - 3x21 - Episode 9
Attack on Titan Season 3 (2019) - 3x22 - Episode 10
:idea: Please read the FAQ and How to Request Help.

kim
Power User
Posts: 797
Joined: 15 May 2014, 16:17

Re: support for XEM?

Post by kim » 09 Jun 2019, 19:27

rednoah wrote:
03 Jun 2019, 03:24
This should work better:

Code: Select all

def cache = Cache.getCache('xem', CacheType.Daily)
def url = 'https://www.filebot.net/update.xml'
def content = cache.text(url, { new URL(it) }).get()
println content
thx

how do I read/load an already written file into filebot ?
and also the, when will it expire and the other properties ?

User avatar
rednoah
The Source
Posts: 16419
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: support for XEM?

Post by rednoah » 09 Jun 2019, 19:33

kim wrote:
09 Jun 2019, 19:27
how do I read/load an already written file into filebot ?
The code above will either do a network request to get the data, or read the data from local disk cache. It's all handled implicitly if you use it like in the example above. Your code doesn't know or care if the data came from cache or via web request.

kim wrote:
09 Jun 2019, 19:27
and also the, when will it expire and the other properties ?
If you use the code above, then it'll get cached for about a day. If you tell me what expiration times or properties you're looking for, then I can give you examples.
:idea: Please read the FAQ and How to Request Help.

Post Reply