Season number wrong when renaming by xattr metadata

[nls]

I'm generating metadata info into xattr attributes in my files. Specials are handled with a custom name template, setting season to 0. When I place variables conatining season identifiers, I get 1 instead of 0 for those template variables. There're a few extra values in the json, they're silently ignored by filebot, no problem there.

Sample formatted xattr:

Code: Select all

{
   "absolute" : 1,
   "season" : 0,
   "seriesInfo" : {
      "genres" : [],
      "name" : "Amagami SS",
      "startDate" : {
         "day" : 2,
         "year" : 2010,
         "month" : 7
      },
      "order" : "Airdate",
      "id" : 7506,
      "database" : "AniDB",
      "rating" : 6.4,
      "language" : "en",
      "aliasNames" : [
         "アマガミSS",
         "Amagami SS"
      ]
   },
   "title" : "Miya Tachibana: Little Sister",
   "specType" : "Special",
   "airdate" : {
      "year" : 2011,
      "day" : 29,
      "month" : 4
   },
   "seriesName" : "Amagami SS",
   "id:" : 126361,
   "@type" : "Episode",
   "origEpno" : "S1",
   "episode" : 1
}

Test run (added some bits to see the bug):

Code: Select all

$ filebot --format '{plex[1]}/{s00e00}/{sxe}/{s}/{plex[2]}' -non-strict -rename --action test --db xattr --output . .
Rename files using [Extended Attributes]
...
[TEST] from [/.../xxx.mkv] to [/.../Amagami SS/S01E01/1x01/1/Amagami SS - 0x01 - Miya Tachibana - Little Sister.mkv]
...
Processed 42 files

The {plex} bits work correctly but the {s}, etc. aren't. Not providing any template works too.

[rednoah]

{s} for example behaves differently for TheTVDB and AniDB data. Since your metadata specifies AniDB, FileBot knows that this episode object can't have a season, and then goes ahead to cross-reference your episode data with TheTVDB data online to guess the correct TheTVDB season number you might probably want when using {s} for AniDB episode objects.

If you change "database" : "AniDB" to "TheTVDB" then it'll probably just use the season field as specified by your xattr instead of doing magic.


EDIT:

You can do {episode.season} to access the season field directly, instead of using {s} which might do more complex things implicitly.

[nls]

I don't want seasons with anime, their season numbering and relations are all over the place. 0 is for handling specials with Plex exlicitly. I'll try episode.season as you suggested, thanks. Though I must note that filebot ought to obey the user and use the database source the user specifies (xattr in this case) and not go online to do its matching magic and use something else that seems to fit... Or at least it could be documented to avoid making bug reports about intended(?) behavior.

[rednoah]

1.
All bindings are evaluated dynamically, and most of them involve more than a simple field access. This case is tricky, because {s} generally works as expected for the vast majority of users (since the alternative to doing magic and working most of the time is not doing anything and failing every time).


2.
The main reason special processing doesn't work out of the box in this case is xattr not being what FileBot expects for specials. If it's a special episode, you'd get a "special" episode number field in the xattr json, and not necessarily season: 0 (as specials are often part of a specific season), and that's how bindings like {plex} yield Season 00 folders (regardless of season / episode numbers).


3.
While it might not make sense to you in this particular use case, the format logic should always work the same way for the same object regardless how this object came to be, or whether --db TheTVDB or --db xattr was originally specified on the command-line.


4.
Bindings in particular are fairly well-documented in code and should be as readable as any verbose handwritten document:
https://github.com/filebot/filebot/blob ... .java#L151

[nls]

OK, thanks for the explanation. I can use any fields and numbering that seems working, so what would you suggest for handling special episodes in the json? Should I just add a "special" field with the number of the special episode and omit the episode, absolute and season fields?

[rednoah]

I've never really had anyone write xattr manually, so the best course of action is to just have a look at what json FileBot generates for a bunch of episodes and specials, and then follow the same patterns. If you write xattr that FileBot itself wouldn't normally write, then behavior is effectively untested and undefined, and I myself wouldn't know what happens without running tests and checking the underlying code.

e.g.

Code: Select all

filebot -list --q "Amagami SS" --format "{plex.name} => {json}"

e.g. Output for {plex.name} for the last special episode:

Code: Select all

Amagami SS - S00E15 - Valentine's Day Short Voice Drama

e.g. Output for {json} for the last special episode:

Code: Select all

{
   "@type":"Episode",
   "seriesName":"Amagami SS",
   "title":"Valentine's Day Short Voice Drama",
   "special":15,
   "airdate":{
      "year":2012,
      "month":2,
      "day":14
   },
   "id":4774029,
   "seriesInfo":{
      "database":"TheTVDB",
      "order":"Airdate",
      "language":"en",
      "id":172921,
      "name":"Amagami SS",
      "aliasNames":[
         "Amagami SS Plus",
         "Amagami SS+",
         "Amagami SS (2010)"
      ],
      "certification":"TV-PG",
      "startDate":{
         "year":2010,
         "month":7,
         "day":2
      },
      "genres":[
         "Animation"
      ],
      "network":"Tokyo Broadcasting System",
      "rating":7.2,
      "ratingCount":24,
      "runtime":25,
      "status":"Ended"
   }
}

You can try the same with --db AniDB and see how the object changes for episodes / specials compared to TheTVDB.

[nls]

Yeah, thanks. It didn't occur to me that specials have a different field for the ep. number, but apparently this is the case. It's fine this way too, I'll use that for generating json.

I'm generating my own json data source mainly for several reasons: 1 - I want to keep the original directory structure for various reasons and point symlinks to the files, whatever/wherever they are; 2 - in that my only (but very effective) aid is AniDB's hash+filesize based query method in their UDP API; 3 - I want to effectively support the Plex scanners to avoid any manual work if possible; 4 - it migt be useful sometimes to have good metadata already stored in the files, but I guess that's why filebot adds xattrs by default.

No single tool has proper support for these things, AniDB's own tools are somewhat outdated and don't directly support Plex. I can ID my files with the ADB API, make the proper json structure in xattr and then use filebot to do the actual renaming (symlinking) because it's the best tool for the task. I need to do some coding (ADB framework is Python, rest is bash shell and filebot, maybe some Groovy coding later), but when I'm done with that I can properly catalog anime using a single shell call on the NAS (which I plan to integrate into Windows context menu later). Sadly, using only filebot is error-prone with the randomly named group release files, mixing up seasons, then Plex mistaking specials for episodes, etc. My current tests using the outlined technique show almost no mistakes.

[rednoah]

Using dedicated AniDB tools to identify files is definitely a good approach for a first pass. But I'd use that as a pre-filebot step, and the run FileBot (possibly with special --filter settings, possibly skipping auto-detection all together, possibly just renaming files in a way that makes things easier for FileBot in the second pass).


e.g. you could process each file individually, forcing both series ID and episode ID, making mismatches impossible:

Code: Select all

filebot -rename --db AniDB --q "172921" --filter "episode.id == 4774029" --format "{s} - {absolute.pad(5)} - {t}" ...

Alternatively, the airdate (e.g. 2018-09-30) and episode title are usually a very good way to identify episodes across different databases, so if you pre-rename files with a format akin to "{s} {[airdate]} {t}" then that might be a pretty good shortcut (however, I can think of rare corner cases like multiple episodes with the same title aired on the same date that could potentially throw a spanner in the works there).


This thread might give you some good ideas on how to translate AniDB data to TheTVDB data in a third step:
viewtopic.php?f=3&t=2769

[nls]

Thanks for the suggestions. What I do is essentially already using AniDB for a first pass and then filebot for a second pass (third is in Plex for additional stuff like BG images, posters, other metadata). Passing exact episode IDs to filebot could be a good idea. I still want to maintain the json in xattrs, this has multiple benefits. Though I don't want any mapping/xrefs between AniDB and other databases, I'm settling exclusively with AniDB as it makes IDing easier for me. I'm using the Absolute Series Scanner and HAMA in Plex that makes good use of the files named like I described. So I'm trying to standardize around AniDB and tools that support it in some way. I don't care about imagery and other series info in Plex though, they can come from any usable source for all I care.

On another note, would it be possible to support using arbitrary db info for a DB or for at least a well-defined structure source such as xattr? It would make life easier sometimes, as I can pull out pre-generated info from the json I prepared instead of long Groovy name templates full of ifs and regexes (those are for specials).

[rednoah]

Yep, if you use HAMA then you can skip the AniDB -> TheTVDB conversion step.

You can access any xattr directly like this:

Code: Select all

f.xattr['MyKey']

[nls]

Thanks, very useful!

[nls]

The xattr array access doesn't seem to work, or I'm using the wrong syntax. In my sample in the first post, there's an attribute named "origEpno", which I try to access using f.xattr["origEpno"] but it gets substituted with an empty string, as if it wasn't recognized. Do you have any suggestions? Considering it works, how would I access deeper structures like eg. seriesInfo.rating?

[nls]

OK, I seem to get what you meant. I can do {f.xattr["net.filebot.metadata"]} and get the whole xattr content of that xattr attribute. I can also do {new groovy.json.JsonSlurper().parseText(f.xattr["net.filebot.metadata"]).origEpno} which is... meh.

What I meant was "seeing into" the provided json structure, and do something like {xattr.origEpno}, {xattr.seriesInfo.aliasNames[0]} etc.

Would something like that be possible? If not, can I parse the attribute easier and shorter?

[rednoah]

It's probably easier to use your own xattr fields in the first place instead of merging it into the FileBot metadata.

e.g.

Code: Select all

f.xattr.origEpno

[nls]

Yeah, I was thinking of that, though since there's that nice json serialized object you provide provisioning for, I wanted to put my stuff there. I don't want a dozen extra xattr fields, either. But if there's no way nor you plan on implementing something like accessing those fields directly, there's no other choice. The Groovy filename expression ability is rather flexible, so I might stick with that in that case, despite the complicated expressions required.

[nls]

BTW.

e.g. you could process each file individually, forcing both series ID and episode ID, making mismatches impossible:

Code: Select all

filebot -rename --db AniDB --q "172921" --filter "episode.id == 4774029" --format "{s} - {absolute.pad(5)} - {t}" ...

I tried this with a special:

Code: Select all

$ filebot -rename --db AniDB --action test --q "09134" --filter "episode.id == 150671" --format "{n} - {absolute.pad(2)} - {t}" Specials/\[MewSubs\]\ Ore\ no\ Imouto\ ga\ Konna\ ni\ Kawaii\ Wake\ ga\ Nai\ 2\ -\ Special\ 01v0\ \[21FD67D3\].mkv
Rename episodes using [AniDB]
Apply filter [episode.id == 150671] on [33] items
Include [Oreimo 2 - Special 4 - Love Doll]
Failed to match files to episode data
Failure (°_°)

Even though the hit matches the data provided. Any hints?

[rednoah]

I guess -non-strict is required too to force FileBot to accept the match.

[nls]

I got a bit closer by adding -non-strict:

Code: Select all

$ filebot -non-strict -rename --db AniDB --action test --q "09134" --filter "episode.id == 150671" --format "{n} - {absolute.pad(2)} - {t}" Specials/\[MewSubs\]\ Ore\ no\ Imouto\ ga\ Konna\ ni\ Kawaii\ Wake\ ga\ Nai\ 2\ -\ Special\ 01v0\ \[21FD67D3\].mkv
Rename episodes using [AniDB]
Apply filter [episode.id == 150671] on [33] items
Include [Oreimo 2 - Special 4 - Love Doll]
[TEST] from [/pool1/shared/media/anime/series/Ore no Imouto S2/Specials/[MewSubs] Ore no Imouto ga Konna ni Kawaii Wake ga Nai 2 - Special 01v0 [21FD67D3].mkv] to [/pool1/shared/media/anime/series/Ore no Imouto S2/Specials/Oreimo 2 - - Love Doll.mkv]
Processed 1 files

But the episode number is unspecified. {e} isn't working, either.

[rednoah]

{special} will give you the special number. {absolute} may or may not work depending on the situation. Should work as expected for TheTVDB, but not sure if AniDB specials can have absolute numbers at all actually. I think Specials are just numbered S1, S2, and so on and don't have a unified absolute number system that covers both normal episodes / special episodes / openings / endings / etc.


I recommend using {airdate} if {absolute} doesn't work. That should give you the temporal order you probably want.


You can use this command to check what kind of data we're working with:

Code: Select all

filebot -list --db AniDB --q "09134" --filter "episode.id == 150671" --format "{json}"

[nls]

Ah, probably my bad, it needs {special} then, thanks (yes, the example gives an episode number using {special}). But... shouldn't it be uniform across episode categories? I mean, it could be easier on the users using custom expressions not to expect them to write conditional expressions based on episode type. I don't mind it too much, just an idea.

[rednoah]

Yes, and all the higher level bindings such as {sxe} {s00e00} {plex} etc handle these things correctly implicitly (specials, multi-episode files, etc).

Having {e} default to the special number is a bad idea, as you will unexpectedly end up with the same episode number for Episode 1 and Special 1 and possibly even override your Episode 1 file with your Special 1 file if you use a format such as {n} - E{e}. At the very least it would mess with future processing if specials aren't marked properly.

[nls]

I'm trying various methods for consistent file metadata queries, and again tried one of your suggestions on a credits special:

Code: Select all

$ filebot -non-strict -list --db AniDB --q "09134" --filter "episode.id == 149621" --format "{json}"
Apply filter [episode.id == 149621] on [33] items
<empty result set>

It skips the specified episode id. In fact it skips everything in a -list without filters that isn't an episode or a special, leaving out AniDB's other types of specials like credits (usually creditless OP/ED), trailers, etc. The episode above has the C17 episode/special number. Specifying a "standard" special like 151205 prints output like usual.

[rednoah]

What exactly do you expect to happen?

There is no episode with ID = 149621 in the first place as far as I can tell:

Code: Select all

filebot -list --db AniDB --q "09134" --format "{json}" | grep 149621

EDIT:

FileBot does completely ignore non-episode / extra / opening / ending / etc entries from the very beginning, as those entries tend to interfere with matching. You won't be able to process these kinds of files with FileBot.

In AniDB terms, FileBot implicitly filters outs anything that is not of Type 1 or Type 2 right away.

[nls]

This is the one in the http API xml:

Code: Select all

<episode id="149621" update="2015-12-07"><epno type="3">C17</epno><length>2</length><airdate>2013-04-14</airdate><title xml:lang="en">Ending 2</title></episode>

There are 80+ various episodes per AniDB for this anime and filebot only recognizes 33. So I'll go on with my own tool for IDing the files and the proceed with the renaming using filebot if possible. Well, unless you plan on adding recognition of all files in the http API.

[rednoah]

I guess Type C / 3 is Openings / Endings and I don't plan on creating Episode objects for these entries. That would likely introduce possible mismatches downstream as a normal episode might just end up getting matched to some Opening / Ending, and that's almost never actually what we want.