Anime names with mixed DBs

All about user-defined episode / movie format expressions
Post Reply
devster
Posts: 335
Joined: 06 Jun 2017, 22:56

Anime names with mixed DBs

Post by devster » 12 Aug 2019, 00:10

Hello,
I'm trying to refine my naming scheme for Anime.
The issue is that AniDB is the only database that has properly romanized original titles, whereas TheTVDB only has the original Japanese ones.
However I would also like to use TheTVDB for organization, as the most common scraper is TheTVDB (both for Plex and Kodi).
Final result I would like to have is something like:

Code: Select all

Anime/Mob Psycho 100 (2016)/Season 2 [Mob Psycho 100 II] (2019)/[HorribleSubs] Mob Psycho 100 - EP13 - Biribiri, Dareka ga Mite Iru [Ripped Apart, Someone Is Watching] [720p x264 - 2.0 AAC LC Jpn][0AE0673C].mkv
Anime/Neon Genesis Evangelion (1995)/Season 1 [Shinseiki Evangelion] (1995)/[NTb] Shinseiki Evangelion - EP01 - Shito, Shuurai [Angel Attack!] [720p x264 - 2.0 FLAC Jpn, 5.1 AC-3 Jpn, 5.1 AC-3 Eng - BluRay][D11B2120].mkv
The first name of the folder is the english one, it has in brackets the romanized one in case it's different; this would contain all names for various seasons (this works as Kodi Regex matches from the beginning but doesn't check for an end.
Episode would be absolute numbering from TheTVDB (Mob Psycho 100 II would start from 13).

I'm currently using --db AniDB, --mapper "label =~ /anime/ ? [episode, XEM.TheTVDB] : [episode, XEM.AniDB]", and the following animeFormat (comments where it's broken):

Code: Select all

{
  def norm = { it.replaceAll(/[`´‘’ʻ""“”]/, "'")
                 .replaceAll(/[|]/, " - ")
                 .replaceAll(/[?]/, "\uFE56")
                 .replaceAll(/[*\p{Zs}]+/, " ")
                 .replaceAll(/\b[IiVvXx]+\b/, { it.upper() })
                 .replaceAll(/\b[0-9](?i:th|nd|rd)\b/, { it.lower() }) }

  def isEng = any{ audio.language.first() ==~ /en/ }{ true }
  def isJpn = any{ languages.first().ISO2 ==~ /ja/ || audio.language.first() ==~ /ja/ }{ false }

  String theTVDBseriesName = XEM.TheTVDB.seriesName 
  String mainTitle = any{ theTVDBseriesName }{ norm(n).colon(" - ").replaceTrailingBrackets() }
  String primTitle = norm(primaryTitle).colon(" - ")

allOf
  { "Anime" }
  {
    allOf
      { mainTitle }
      { "($y)" } // can't use it as it refers to AniDB year e.g. 2019 for Mob Psycho 100 S2
    .join(" ")
  }
  {
    if (episode.special) {
      "Specials"
    } else {
    	 allOf
    	   { "Season" }
    	   { s }
        { if (mainTitle != primTitle) "[" + norm(primaryTitle).colon(" - ") + "]" }
        { "($sy)" } // missing as AniDB doesn't have it, but would match $y
      .join(" ")
    }
  }
  { allOf
  	{ allOf
      { def grp = net.filebot.media.MediaDetection.releaseInfo.getReleaseGroup(fn.replaceAll(/\[.*\]$/, ""))
        (grp) ? "[$grp]" : "[$group]" }
      { mainTitle }
      .join(" ") }
    { episode.special ? "S$special" : "EP" + XEM.TheTVDB.absolute.pad(2) }
    { allOf
      { def trLang = any{ if (isJpn) "x-jat" }{ if (isEng) "eng" }{ audio.language.first() }{"eng"}
        switch (trLang) {
          case { it == "x-jat" }:
          allOf
            { norm(localize."$trLang".t).colon(", ").slash("\u2571") }
            { "[" + norm(t).colon(", ").slash("\u2571") + "]" }
            .join(" ")
          break
        case { it == "eng" }:
          norm(t).colon(", ").slash("\u2571")
          break
        default:
          norm(localize."$trLang".t).colon(", ").slash("\u2571")
        }
      }
      // simplified rest of the naming (not relevant)
      { allOf
        {" ["}
        { allOf{vf}{vc}.join(" ") }  
        {"]"}
        .join("") }
      .join(" ") }
    .join(" - ") }
  .join("/") }
Any ideas on how to solve the years?
I only work in black and sometimes very, very dark grey. (Batman)

User avatar
rednoah
The Source
Posts: 16942
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Anime names with mixed DBs

Post by rednoah » 12 Aug 2019, 09:26

:idea: If you target TheTVDB for Plex, then I highly recommend using --db TheTVDB, so that xattr metadata is TheTVDB metadata, so that you can use the FileBot Xattr Metadata Scanners or Plex, so that Plex can interpret your files correctly even if you're not using the {plex} naming standard.


If you have trouble with matching the correct TheTVDB episodes, then --mapper AnimeList.AniDB might help:

Code: Select all

--mapper AnimeList.AniDB

Since we'll be formatting TheTVDB episode data now, but want the AniDB series name, we'll have to again use the {AnimeList} binding like so:

Code: Select all

{AnimeList.AniDB.seriesName}

EDIT:

:!: I'm not sure if there's currently an easy way to get the AniDB primary title (x-jat) when formatting TheTVDB episode data, assuming AnimeList or XEM doesn't already list mappings with that name already. I'll look into it.
:idea: Please read the FAQ and How to Request Help.

devster
Posts: 335
Joined: 06 Jun 2017, 22:56

Re: Anime names with mixed DBs

Post by devster » 12 Aug 2019, 11:17

I'm currently using the default Plex TheTVDB scraper, it's working flawlessly for AniDB-tagged stuff when the name matches TheTVDB name; indeed one of the reasons for mixing DB info is because I don't want to install additional scrapers.

In addition to seriesName (which is currently accessible), I also would use localizable AniDB episode names (which I currently don't know how to access).
The previous example also uses 2 things which are currently broken or missing: the years of the series (broken as AniDB uses the specific season) and the season year (not available for AniDB); both seem currently unavailable through XEM.TheTVDB.
One solution could be to have the mapper return the full record, maybe with a separate query to TheTVDB, with bindings accessible with the same names as the original ones.
e.g. using AniDB and --mapper XEM.TheTVDB for the anime Mob Psycho 100 S2.

Code: Select all

n = 'Mob Psycho 100 II'
XEM.TheTVDB.n = 'Mob Psycho 100'
sy = null
XEM.TheTVDB.sy = [2016,2019]
y = 2019
XEM.TheTVDB.y = 2016
absolute = 1
XEM.TheTVDB.absolute = 13
AnimeList.TheTVDB.absolute = null // can't use this for this purpose
Depending on how localize works it may also be possible to do the reverse, match with TheTVDB and use XEM.AniDB to localize the titles.
I only work in black and sometimes very, very dark grey. (Batman)

User avatar
rednoah
The Source
Posts: 16942
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Anime names with mixed DBs

Post by rednoah » 12 Aug 2019, 15:20

I see. Unfortunately, {XEM} and {AnimeList} can't be optimized for both mapper (return Episode object) and format (return associative script object) usage. The needs of the mapper use case take precedence for these two bindings.



EDIT:


r6579 introduces the {db} binding to dynamically cross-references between databases.

e.g.

Code: Select all

{db.AniDB.primaryTitle}

Code: Select all

{db.AniDB.localize.Japanese.t}
:idea: The current implementation uses AnimeLists data implicitly and only works for AniDB / TheTVDB episode objects.
:idea: Please read the FAQ and How to Request Help.

devster
Posts: 335
Joined: 06 Jun 2017, 22:56

Re: Anime names with mixed DBs

Post by devster » 12 Aug 2019, 19:30

That's actually exactly what I was looking for, the name of the binding doesn't really matter.
Updated naming scheme:

Code: Select all

{
  def norm = { it.replaceAll(/[`´‘’ʻ""“”]/, "'")
                 .replaceAll(/[|]/, " - ")
                 .replaceAll(/[?]/, "\uFE56")
                 .replaceAll(/[*\p{Zs}]+/, " ")
                 .replaceAll(/\b[IiVvXx]+\b/, { it.upper() })
                 .replaceAll(/\b[0-9](?i:th|nd|rd)\b/, { it.lower() }) }

  def isEng = any{ audio.language.first() ==~ /en/ }{ true }
  def isJpn = any{ languages.first().ISO2 ==~ /ja/ || audio.language.first() ==~ /ja/ }{ false }

  String mainTitle = any{ db.TheTVDB.n }{ norm(n).colon(" - ").replaceTrailingBrackets() }
  String primTitle = norm(primaryTitle).colon(" - ")

allOf
  { "Anime" }
  {
    allOf
      { mainTitle }
      { "(" + db.TheTVDB.y + ")" }
    .join(" ")
  }
  {
    if (episode.special) {
      "Specials"
    } else {
    	 allOf
    	   { "Season" }
    	   { s }
        { if (mainTitle != primTitle) "[" + norm(primaryTitle).colon(" - ") + "]" }
        { "(" + db.TheTVDB.sy.bounds().join("-") + ")" }
      .join(" ")
    }
  }
  { allOf
  	{ allOf
      { def grp = net.filebot.media.MediaDetection.releaseInfo.getReleaseGroup(fn.replaceAll(/\[.*\]$/, ""))
        (grp) ? "[$grp]" : "[$group]" }
      { mainTitle }
      .join(" ") }
    { episode.special ? "S$special" : "EP" + db.TheTVDB.absolute.pad(2) }
    { allOf
      { def trLang = any{ if (isJpn) "x-jat" }{ if (isEng) "eng" }{ audio.language.first() }{"eng"}
        switch (trLang) {
          case { it == "x-jat" }:
          allOf
            { norm(localize."$trLang".t).colon(", ").slash("\u2571") }
            { "[" + norm(t).colon(", ").slash("\u2571") + "]" }
            .join(" ")
          break
        case { it == "eng" }:
          norm(t).colon(", ").slash("\u2571")
          break
        default:
          norm(localize."$trLang".t).colon(", ").slash("\u2571")
        }
      }
      // simplified rest of the naming (not relevant)
      { allOf
        {" ["}
        { allOf{vf}{vc}.join(" ") }  
        {"]"}
        .join("") }
      .join(" ") }
    .join(" - ") }
  .join("/") }
Still uses AniDB as db and doesn't need to use a specific mapper; now it could easily be rewritten for TheTVDB.
Output:

Code: Select all

Anime/Mob Psycho 100 (2016)/Season 2 [Mob Psycho 100 II] (2019)/[HorribleSubs] Mob Psycho 100 - EP13 - Biribiri, Dareka ga Mite Iru [Ripped Apart, Someone Is Watching] [1080p x264]
Is there a plan for 4.8.6 stable?
I only work in black and sometimes very, very dark grey. (Batman)

User avatar
rednoah
The Source
Posts: 16942
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Anime names with mixed DBs

Post by rednoah » 13 Aug 2019, 04:58

devster wrote:
12 Aug 2019, 19:30
Is there a plan for 4.8.6 stable?
Yes, last week. :lol: But it gets pushed back 1-2 weeks every time there's a potentially breaking change or a useful feature gets added to let things settle down.

The latest builds are pretty stable these days, but I wouldn't quite want everyone to upgrade just yet:
https://get.filebot.net/filebot/BETA/
:idea: Please read the FAQ and How to Request Help.

devster
Posts: 335
Joined: 06 Jun 2017, 22:56

Re: Anime names with mixed DBs

Post by devster » 13 Aug 2019, 10:56

Between reading the reply, going back home and trying it out my current version is 3 commits ahead of r6579, It's moving a bit too fast for me to upgrade without concerns.
Also, as it's quite difficult to use the CLI within a container from another container, so currently it's bundled in the same image as my torrent client, which makes rebuilding everything a bit too long to stay on-par with the BETA.
I only work in black and sometimes very, very dark grey. (Batman)

User avatar
rednoah
The Source
Posts: 16942
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Anime names with mixed DBs

Post by rednoah » 13 Aug 2019, 20:52

No worries, the final release should be finalized and ready by the end of August.
:idea: Please read the FAQ and How to Request Help.

devster
Posts: 335
Joined: 06 Jun 2017, 22:56

Re: Anime names with mixed DBs

Post by devster » 18 Aug 2019, 17:18

Found a possible issue with {db} binding when trying to rename Cannon Buster, a recently released anime from Netflix.
I get:

Code: Select all

{
  allOf
    { mainTitle }
    { db.TheTVDB.y.toString().surround() }
  .join()
} // returns "Cannon Busters" without the year

try {
	db.TheTVDB
} catch (Exception e) {
	assert e.getMessage() == 'Binding "TheTVDB": Mapping not found'
}
However Cannon Busters has a strict match on TheTVDB (https://www.thetvdb.com/series/cannon-busters).
Is this expected?
I only work in black and sometimes very, very dark grey. (Batman)

User avatar
rednoah
The Source
Posts: 16942
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Anime names with mixed DBs

Post by rednoah » 18 Aug 2019, 18:16

Yep, looks like nobody has entered mapping data for this one yet:
:arrow: https://github.com/ScudLee/anime-lists
:arrow: https://raw.githubusercontent.com/ScudL ... e-list.xml

If it's newer shows, or just not very popular shows, then there's a good chance that mapping won't work correctly, since the mapping data is curated manually.

:idea: {db} and {AnimeList} use AniDB ID ➔ TheTVDB ID known mappings. FileBot does not perform any search queries to guess corresponding entries.
:idea: Please read the FAQ and How to Request Help.

devster
Posts: 335
Joined: 06 Jun 2017, 22:56

Re: Anime names with mixed DBs

Post by devster » 18 Aug 2019, 18:35

Mh, I was hoping db would send a request for the TVDB in case no mapping was found, but I realize it the possibility for mistakes and the added complexity.
I'm trying to work around it using:

Code: Select all

def alt = net.filebot.web.TheTVDBClient.fetchSearchResult​(n, Locale.ENGLISH)
but I get:

Code: Select all

Expression yields empty value: No signature of method: static net.filebot.web.TheTVDBClient.fetchSearchResult​() is applicable for argument types: (String, Locale) values: [Cannon Busters, en]
Possible solutions: fetchSearchResult(java.lang.String, java.util.Locale)
Not sure why.
I only work in black and sometimes very, very dark grey. (Batman)

User avatar
rednoah
The Source
Posts: 16942
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Anime names with mixed DBs

Post by rednoah » 18 Aug 2019, 18:50

This will help:

Code: Select all

{
	def query = "Cannon Busters"
	def options = net.filebot.WebServices.TheTVDB.search(query, Locale.ENGLISH)
	def selection = options.find{ it.name == query }
	def seriesInfo = net.filebot.WebServices.TheTVDB.getSeriesInfo(selection, Locale.ENGLISH)
}
WARNING: INTERNAL API ➔ UNSUPPORTED
:idea: Please read the FAQ and How to Request Help.

devster
Posts: 335
Joined: 06 Jun 2017, 22:56

Re: Anime names with mixed DBs

Post by devster » 18 Aug 2019, 19:27

Not too much, I'm unable to find those methods in the docs and can't get season or episode-level info.
This fallback was intended only to match identical name and identical absolute ep number.

This is the page I used as reference: https://www.filebot.net/docs/api/net/fi ... il.Locale) I thought these were intended to be supported.
I'll hope they update the mappings soon.
I only work in black and sometimes very, very dark grey. (Batman)

User avatar
rednoah
The Source
Posts: 16942
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Anime names with mixed DBs

Post by rednoah » 18 Aug 2019, 20:04

@kim wanted javadocs for the client classes, so I've just added them. The EpisodeListProvider class hierarchy is not designed with end users in mind though, especially not specific implementations for the various databases, but the alternative to giving you an internal solution is no solution, so here we are.


Here's the common interface for TV database clients that lists the method I'm using and probably the ones you're missing:
https://www.filebot.net/docs/api/net/fi ... vider.html


EDIT:

Added a lookup function:

Code: Select all

{
	def tvdb = getService('TheTVDB')
	def query = 'Cannon Busters'

	def options = tvdb.search(query, Locale.ENGLISH)
	def selection = options.find{ it.name == query }
	def seriesInfo = tvdb.getSeriesInfo(selection, Locale.ENGLISH)
}
:idea: Please read the FAQ and How to Request Help.

Post Reply