[DEPRECATED] support for XEM?

[crawfs]

Unfortunately I don't know the answer to that, not much of a programmer myself...

It might be worth taking a look at sonarr by eithertaking a look at their source code or hang around their discord and figure out how sonarr handles it as that project is how I first learned that this is possible.

[devster]

Found a possible solution:
http://thexem.de/map/allNames?origin=tv ... ultNames=1
the above link should return the entire list of shows but in a specific format.
The ID is still the basic one, corresponding to the default name (so for Monogatari/AniDB is 6327) however associated to that name there is an array of string and objects which map the default name (the string) and all alternative name to a season number (they correspond to the query strings above, default name can be skipped).
EDIT: I forgot to mention that -1 exists as a season and represents an alias of the default name.

Code: Select all

  "6327": [
    "Monogatari",
    {
      "Bakemonogatari": 1
    },
    {
      "Nisemonogatari": 2
    },
    {
      "Monogatari Series Second Season": 3
    },
    {
      "Owarimonogatari": 4
    },
    {
      "Owarimonogatari S2": 5
    },
    {
      "Owarimonogatari Second Season": 5
    }
  ]

So, in theory, Owarimonogatari S2 can be mapped to S5.
Requesting http://thexem.de/map/all?id=13033&origi ... ation=tvdb and filtering for season 5 should then yield the correct corresponding episodes (or multiple episodes separated by _2, _3 and so on).
By the way, thexem.de already caches responses so it may be useful to reduce the default cache time for these requests.
Final useful method is http://thexem.de/map/havemap?origin=tvdb which should return every ID which has a mapping associated.

A possible flow could be matching against AniDB as usual, with the resulting name, not ID, search for correspondence in TheXEM, replace series name with "default" found and add season info, alternatively use absolute numbering from theTVDB, there is also a "master" category but I haven't seen it represented.

[rednoah]

FileBot r6419 now supports the --mapper <expression> option.

Note that nothing specifically related to TheXEM is implemented at this point. (i.e. fairly useless unless you're fairly comfortable with writing Groovy code)

e.g. identity mapper:

Code: Select all

--mapper "episode"

e.g. constant mapper:

Code: Select all

--mapper "new net.filebot.web.Episode(/Series Name/, 1, 1, /Episode Title/)"

e.g. arbitrary custom mapper that you have written yourself:

Code: Select all

--mapper /path/to/mapper.groovy

[devster]

Where can I find more info on how it's supposed to work?
For example, does mapper accept a class? Does it need to implement any specific methods?
What would be the flow for a custom mapper?
Please forgive the barrage, I believe it could be helpful for users.

[rednoah]

--mapper works exactly the same as --filter, except where --filter is expected to return type Boolean objects, --filter is expected to return type Episode objects.

* All the usual bindings, such as episode, n, s, e, t and friends work exactly the same way as in format / filter / exec expressions.

* The mapper is applied to each Episode object (unless it has already been excluded by the filter) then all the new Episode objects generated by your mapper are passed on to matching as usual.

* Just run with the useless examples from my last post, and then the console log will show you what's happening.



EDIT:

Here's another cute example:

Code: Select all

filebot -list --q "Firefly" --mapper "[seriesName: ny, season: s + 1, episode: e, title: /Episode / + absolute]"

Code: Select all

Firefly (2002) - 2x01 - Episode 2
Firefly (2002) - 2x02 - Episode 3
Firefly (2002) - 2x03 - Episode 6
Firefly (2002) - 2x04 - Episode 7
Firefly (2002) - 2x05 - Episode 8
...

* Add year to series name
* Add +1 to season number
* Replace episode title with absolute number

[devster]

Perfect, thanks, will play with it and see what happens.
However, trying to think about XEM, I'm not finding fields in the Episode class that refer to databases though, so, for example, if I changed season/episode but also mapped the season to another database entirely (AniDB > TheTVDB) the ID wouldn't match anymore (13033 becomes 102261 which is missing in AniDB).
Unless I misunderstood the "new Episode objects generated by your mapper are passed on to matching as usual". Is matching against the DB done before or after?

[rednoah]

1. Retrieve episode objects from database
2. (Optional) Map episode objects to other episode objects
3. Match episode objects with file objects
4. Format matches


The episode objects you get as input are typically more complex than just [seriesName, season, episode, title] and so it might make sense to just copy over into the new Episode object all the fields you don't want to modify:

Code: Select all

Episode​(String seriesName, Integer season, Integer episode, String title, Integer absolute, Integer special, SimpleDate airdate, Integer id, SeriesInfo seriesInfo)

https://www.filebot.net/docs/api/net/fi ... isode.html


{d} requires episode airdate, {order} requires series id, etc, etc, etc. Depending on what information you keep, remove or change in the Episode object effects behavior later on.

[devster]

Got it now, I hope, thanks again for the explanation.

[devster]

It's extremely rough and untested but it seems to work (barely):

Code: Select all

// test with filebot -list --q "Monogatari" --db AniDB --mapper xem.groovy
import groovy.json.JsonSlurper

String origin = anime ? "anidb" : "tvdb"
Integer seas
if (anime) {
    seas = 1
} else {
    seas = s
}
def spec = call{special}
def ep = call{e}
def baseURL = new URL("http://thexem.de")
def reqHeaders = [:]
def params = [
                "origin": origin
            ]
def query = params.collect { k, v -> "$k=$v" }.join('&')
def getResponse = new URL(baseURL, "/map/havemap?$query").get(reqHeaders)
String stringID = id.toString()
Map    json     = new JsonSlurper().parseText(getResponse.text)
def    item     = json.data.any{ it == stringID }
if (item) {
    def paramsName = [
                "origin": origin,
                "id": id,
                "defaultNames": 1,
            ]
    def  queryName = paramsName.collect { k, v -> "$k=$v" }.join('&')
    def getResName = new URL(baseURL, "/map/names?$queryName").get(reqHeaders)
    Map   jsonName = new JsonSlurper().parseText(getResName.text)
    def    mat     = jsonName.data.collect{
        if (it.value instanceof Map) {
            def name = it.value.entrySet().value
            [(it.key): name.flatten()]
        } else if (it.value instanceof String) {
            [(it.key): it.value]
        }
    }

    String  newN   = mat.findAll{ it.all }?.all.first()
    Integer foundS = mat.findAll{
		it.entrySet().value.any{ v -> v =~ /(?i)$n/ }
    }.first().entrySet().key.first().toInteger()
    Integer newS   = (foundS < 0) ? seas : foundS

    def paramsMap = [
                "origin": origin,
                "id": id,
                "season": newS,
                "episode": ep,
            ]
    def queryMap = paramsMap.collect { k, v -> "$k=$v" }.join('&')
    def getResponseMap = new URL(baseURL, "/map/single?$queryMap").get(reqHeaders)
    Map jsonMap = new JsonSlurper().parseText(getResponseMap.text)
    // assuming tvdb destination, could be included in the query
    def result  = jsonMap.data.entrySet().findAll{ it.key.matches(/tvdb.*/) }
    if (result.size() < 2) {
        return new net.filebot.web.Episode(newN, newS, result.first().value.episode, t, result.first().value.absolute, spec, d, id, series)
    } else {
        def multi = []
        for ( i in 0..result.size()-1 ) {
            multi << new net.filebot.web.Episode(newN, newS, result[i].value.episode, t, result[i].value.absolute, spec, d, id, series)
        }
        return new net.filebot.web.MultiEpisode(*multi)
    }
}
// hopefully return the episode untouched if not matched
return new net.filebot.web.Episode(n, seas, ep, t, absolute, spec, d, id, series)

Hopefully all multi-episodes are just multi-part because I couldn't find a way to merge titles.
I'm no groovy dev so forgive my horrible code.

[rednoah]

Wow! Impressive! I'd have added the --mapper option much sooner (that part was easy) if I had known you'd do all the heavy lifting afterwards!

I'll play with it as well next week, and see what I can add into the core to make this a built-in feature in upcoming releases.



EDIT:

And so the student has become the master! I had no idea you can use the spread operator to spread an array into a method as arguments!

Code: Select all

def x = [1, 2, 3]
def f = { a, b, c -> a * b * c }
f(*x)

EDIT 2:

I'd add a bit of caching as well, like so:

Code: Select all

def cache = Cache.getCache('xem', CacheType.Daily)

def url = 'https://www.filebot.net/update.xml'
def content = cache.text(url, String.&toURL).get()

println content

[devster]

Will try, thanks.
I just noticed my script seems to halt after the first result in the -list (e.g. for Firefly it prints only the first item).
Not sure what's happening.

[rednoah]

Can you paste your command so I can try and see what's going on?

[devster]

Code: Select all

# this returns only the first 2 episodes despite XEM having 3 mappings, but it's an issue with AniDB only reporting the first 2 as regular episodes and the third as special.
# they also join titles on their own, so that shouldn't be a byproduct of the MultiEpisode object
filebot -list --q "Owarimonogatari Second Season" --db AniDB --mapper xem.groovy
# this should return episodes unchanged
filebot -list --q "Firefly" --db TheTVDB --mapper xem.groovy

[rednoah]

I see, since the mapper will automatically get rid of duplicates for you, and since you're using the series id as episode id, you end up with only one episode object per series id:

Code: Select all

filebot -list --q "Owarimonogatari Second Season" --db AniDB --format "{id} | {episode.id}"

Try using episode.id instead of id when creating episode objects.


e.g. tail tail xem.groovy

Code: Select all

    if (result.size() < 2) {
        return new net.filebot.web.Episode(newN, newS, result.first().value.episode, t, result.first().value.absolute, spec, d, episode.id, series)
    } else {
        def multi = []
        for ( i in 0..result.size()-1 ) {
            multi << new net.filebot.web.Episode(newN, newS, result[i].value.episode, t, result[i].value.absolute, spec, d, episode.id, series)
        }
        return new net.filebot.web.MultiEpisode(*multi)
    }
}
// hopefully return the episode untouched if not matched
return episode

[devster]

Code: Select all

// test with filebot -list --q "Monogatari" --db AniDB --mapper xem.groovy
import groovy.json.JsonSlurper
import net.filebot.Cache
import net.filebot.CacheType

Closure<Object> request = { Map headers = [:], String base = "http://thexem.de", String path, Map params ->
    Cache  cache    = net.filebot.Cache.getCache('xem', CacheType.Daily)
    URL    baseURL  = new URL(base)
    String query    = params.collect { k, v -> "$k=$v" }.join('&')
    Object response = new URL(baseURL, "$path?$query").get(headers)
    response
    // TODO: daily caching
    // def content = cache.text(url, String.&toURL).get()
}

String origin = anime ? "anidb" : "tvdb"
Integer seas = anime ? 1 : episode?.season

Object  hasMap  = request("/map/havemap", ["origin": origin])
Map     jHasMap = new JsonSlurper().parseText(hasMap.text)
Boolean item    = jHasMap.data.any{ it == id.toString() }
if (item) {
    Object names = request("/map/names", [
        "origin": origin,
        "id": id,
        "defaultNames": 1,
    ])
    Map       jName   = new JsonSlurper().parseText(names.text)
    ArrayList reflect = jName.data.collect{
        if (it.value instanceof Map) {
            def name = it.value.entrySet().value
            [(it.key): name.flatten()]
        } else if (it.value instanceof String) {
            [(it.key): it.value]
        }
    }

    // String  newN   = reflect.findAll{ it instanceof String }.first()
    String  newN   = reflect.findAll{ it.all }?.all.first()
    Integer foundS = reflect.findAll{
		it.entrySet().value.any{ v -> v =~ /(?i)$episode.seriesName/ }
    }.first().entrySet().key.first().toInteger()
    // Integer foundS = item.findAll{ it instanceof Map }*.find{ k, v -> k.match(/$n/) }.find{ it != null }?.value
    Integer newS   = (foundS < 0) ? seas : foundS

    Map old = [
        ep: episode.episode ? episode.episode : episode.special,
        se: episode.special ? 0 : newS,
    ]
    // assuming TVDB destination, could be included in the query
    Object mapping = request("/map/single", [
        "origin": origin,
        "id": id,
        "season": old.se,
        "episode": old.ep,
    ])
    Map jMapping = new JsonSlurper().parseText(mapping.text)
    // also assuming TVDB destination
    if (jMapping.data.isEmpty()) {
        return episode
    }
    def result  = jMapping.data.entrySet().findAll{ it.key.matches(/tvdb.*/) }
    if (result.size() < 2) {
        return new net.filebot.web.Episode(newN, newS, result.first().value.episode, episode?.title, result.first().value.absolute, episode?.special, episode?.airdate, episode.id, series)
    } else {
        def multi = []
        for ( i in 0..result.size()-1 ) {
            // hopefully all multi-episodes are just multi-part because I couldn't find a way to merge titles
            multi << new net.filebot.web.Episode(newN, newS, result[i].value.episode, episode?.title, result[i].value.absolute, episode?.special, episode?.airdate, episode.id, series)
        }
        return new net.filebot.web.MultiEpisode(*multi)
    }
}
// hopefully return the episode untouched if not matched
return episode

update, should work a bit better.

[rednoah]

I've added the {xem} binding so you can now do this:

Code: Select all

XEM.TheTVDB

Code: Select all

filebot -list --q "Monogatari" --db AniDB --mapper XEM.TheTVDB

It should work exactly the same as the Groovy version written by devster so his Groovy implementation is probably more useful for debugging.

How well does it work different series and corner cases? No idea. Fairly untested.

[kim]

How do I get this working ?

I'd add a bit of caching as well, like so:

Code: Select all

def cache = Cache.getCache('xem', CacheType.Daily)

def url = 'https://www.filebot.net/update.xml'
def content = cache.text(url, String.&toURL).get()

println content

Error:

Expression yields empty value: No signature of method: java.lang.String.toURL() is applicable for argument types: (String) values: [https://www.filebot.net/update.xml]
Possible solutions: toURL(), toURL(), toURI(), toURI(), toSet(), toFile(java.lang.String)

[rednoah]

How are you using this code?

I think I only tested this in the Groovy Pad script. Does it not work out if used inside a format?


EDIT:

This should work better:

Code: Select all

def cache = Cache.getCache('xem', CacheType.Daily)
def url = 'https://www.filebot.net/update.xml'
def content = cache.text(url, { new URL(it) }).get()
println content

[crawfs]

will --mapper work for AMC scripts? Or will it throw an error if it doesn't find a matching xem entry for an item it's trying to match?

[rednoah]

Untested. Should work though.

What happens if there is no match is entirely up to your --mapper code.

The xem binding will either translate to xem mappings, or keep the episode object as is.

[rednoah]

I'm also looking into support for Anime Lists for mapping information:
https://github.com/ScudLee/anime-lists

e.g.

Code: Select all

filebot -list --q 14444 --db AniDB --mapper "AnimeLists.TheTVDB"
Apply mapper [AnimeLists.TheTVDB] on [10] items
Map [Attack on Titan Season 3 (2019) - 01 - The Town Where Everything Began] to [Attack on Titan Season 3 (2019) - 3x13 - The Town Where Everything Began]
Map [Attack on Titan Season 3 (2019) - 02 - Thunder Spears] to [Attack on Titan Season 3 (2019) - 3x14 - Thunder Spears]
Map [Attack on Titan Season 3 (2019) - 03 - Descent] to [Attack on Titan Season 3 (2019) - 3x15 - Descent]
Map [Attack on Titan Season 3 (2019) - 04 - Perfect Game] to [Attack on Titan Season 3 (2019) - 3x16 - Perfect Game]
Map [Attack on Titan Season 3 (2019) - 05 - Hero] to [Attack on Titan Season 3 (2019) - 3x17 - Hero]
Map [Attack on Titan Season 3 (2019) - 06 - Midnight Sun] to [Attack on Titan Season 3 (2019) - 3x18 - Midnight Sun]
Map [Attack on Titan Season 3 (2019) - 07 - The Basement] to [Attack on Titan Season 3 (2019) - 3x19 - The Basement]
Map [Attack on Titan Season 3 (2019) - 08 - Episode 8] to [Attack on Titan Season 3 (2019) - 3x20 - Episode 8]
Map [Attack on Titan Season 3 (2019) - 09 - Episode 9] to [Attack on Titan Season 3 (2019) - 3x21 - Episode 9]
Map [Attack on Titan Season 3 (2019) - 10 - Episode 10] to [Attack on Titan Season 3 (2019) - 3x22 - Episode 10]
Attack on Titan Season 3 (2019) - 3x13 - The Town Where Everything Began
Attack on Titan Season 3 (2019) - 3x14 - Thunder Spears
Attack on Titan Season 3 (2019) - 3x15 - Descent
Attack on Titan Season 3 (2019) - 3x16 - Perfect Game
Attack on Titan Season 3 (2019) - 3x17 - Hero
Attack on Titan Season 3 (2019) - 3x18 - Midnight Sun
Attack on Titan Season 3 (2019) - 3x19 - The Basement
Attack on Titan Season 3 (2019) - 3x20 - Episode 8
Attack on Titan Season 3 (2019) - 3x21 - Episode 9
Attack on Titan Season 3 (2019) - 3x22 - Episode 10

[kim]

This should work better:

Code: Select all

def cache = Cache.getCache('xem', CacheType.Daily)
def url = 'https://www.filebot.net/update.xml'
def content = cache.text(url, { new URL(it) }).get()
println content

thx

how do I read/load an already written file into filebot ?
and also the, when will it expire and the other properties ?

[rednoah]

how do I read/load an already written file into filebot ?

The code above will either do a network request to get the data, or read the data from local disk cache. It's all handled implicitly if you use it like in the example above. Your code doesn't know or care if the data came from cache or via web request.

and also the, when will it expire and the other properties ?

If you use the code above, then it'll get cached for about a day. If you tell me what expiration times or properties you're looking for, then I can give you examples.

[rednoah]

FileBot r6486 now supports both AnimeLists and XEM to map between AniDB and TheTVDB.


AnimeLists mappings are now used implicitly used when using season bindings like {s} {sxe} and {s00e00} on Anime-type Episode objects.

[kim]

but you can't use this code to read e.g. data_0.*

I'm writing a IMDb scraper, here is data from cache file:
index:

Code: Select all

"¬í t Top250Moviessr 7net.sf.ehcache.store.disk.DiskStorageFactory$DiskMarkerþ-U\†å J expiryJ hitCountJ positionI sizeL keyt Ljava/lang/Object;xp  kw æÐ                 ¤¼q ~  "

data:

Code: Select all

¬í sr net.sf.ehcache.Element>é_”  	Z cacheDefaultLifespanJ hitCountJ idJ lastUpdateTimeI 
timeToIdleI 
timeToLiveJ versionL keyt Ljava/lang/Object;L valueq 

so how do i read the expiry, hitCount...