Plex Episode Preset removing (1) / (2) from Episode Title

[dokuro]

This is my first post so be nice

I've been using { ~plex.unix * {" ($y) {tmdb-$id}"} } as my TV episode naming for a while. With version FileBot 5.0.3 for multi part episodes the (1) was added to the title. So, for example Avatar: The Last Airbender - S01E07 - The Spirit World: Winter Solstice became Avatar: The Last Airbender - S01E07 - The Spirit World: Winter Solstice (1). Notice the (1) at the end. I was happy with this so went over my entire library and renamed everything to use this multi part naming.

However, with FileBot 5.1.1 this has now been removed going back to how things where pre FileBot 5.0.3. IE, renaming the episode to Avatar: The Last Airbender - S01E07 - The Spirit World: Winter Solstice. Is this intentional or a bug which will be fixed again in the future.

I'm using FileBot on ubuntu 22.04 in case that's important. Here is the actual episode on TMDB -> https://www.themoviedb.org/tv/246-avata ... /episode/7.

[rednoah]

Yes, the {plex} binding will implicitly remove trailing (1) (2) (1/2) patterns. You can however use {t} for less sanitized values or {episode.title} for unsanitized values.


If you are keen on "(1)" or ", Part 1" then you can add the part in your custom format:

Format: Select all

{ plex % { ", Part " + t.match(/[(]([0-9]+)[)]/) } }

Format: Select all

{ plex % { " (" + t.match(/[(]([0-9]+)[)]/) + ")" } }

EDIT: Add (1) (2) part numbers to the episode title



Older versions removed any trailing (...) pattern, not just (1) (2) patterns, which was a problem. Removing trailing (1/2) was intended but removing trailing (XYZ) was not intended. FileBot 5.0.3 had removed the logic entirely, thus fixed the unintended behaviour but also broke the intended behaviour.


I have a personal preference for removing (1) (2) but if there's a good reason to keep it, or if a large number of users prefer to keep it, then I'm open to changes. That said, you can also use your own custom format that does things the way you want.

[dokuro]

Okay, so basically 5.0.3 had changes which removed the intended behaviour to trim the trailing (1) (2) (1/2) patterns and this is now which is now fixed in 5.1.1. That's fine, I'm more checking what the intended / final behaviour is so I can plan for that in my naming and not have flux when new version of the FileBot come out. So, I'll go with the addition of % { " (" + t.match(/[(]([0-9]+)[)]/) + ")" } in my naming scheme.

For your reference, I personally rather have the full unsanitized name as I have a secondary use for FileBot over and above just bulk renaming of media files. I use it to detect episode name changes over at TMDB. So, I fire up FileBot, load media I've already renamed and look for changes. If changes are detected then I can go and check if they are intended changes at TMDB or some rouge user badly changing things. The reason I do this is 1) to ensure TMDB renaming accurate and 2) plex does not like changes in episode ordering of shows, it can mess up watched status information within their application.

Anyhow, thanks for your response, very helpful.

[rednoah]

For your reference, I personally rather have the full unsanitized name as I have a secondary use for FileBot over and above just bulk renaming of media files. I use it to detect episode name changes over at TMDB. So, I fire up FileBot, load media I've already renamed and look for changes. If changes are detected then I can go and check if they are intended changes at TMDB or some rouge user badly changing things.

This is an interesting use case. I'd probably write a custom script to specifically check for metadata changes, instead of loading everything into FileBot, rematching everything, then manually checking through the list, might work, but seems a bit of a clunky hands-on approach. You could start with the miss script but instead of checking for missing episodes, you could check for changes in episode numbers / episode titles.

[dokuro]

Thanks, I'll take a look at that miss script and see if its a better approach than what I'm doing today.

[rednoah]

I've added "episode information has changed" logging to the miss script:
https://github.com/filebot/scripts/comm ... 8704cbf950

[dokuro]

Thanks. Only had a few minutes to test and while it did find missing episodes the script did not find when episode information had changed. Syntax I used was filebot -script miss.groovy <path to media>.

I tried to simulate a change by changing the actual filename, net.filebot.filename and net.filebot.metadata so it was different to what was on TMDB. I could be doing something stupid so will come back to it later. Thanks again.

[rednoah]

Detection is triggered based on Episode object String representation not being the same when comparing episode list objects and previously stored metadata objects:
https://github.com/filebot/scripts/blob ... groovy#L53


e.g. online episode list episode information matches xattr metadata episode information:

Console Output: Select all

$ filebot -list --q "Alias" --db TheMovieDB::TV
Alias - 1x01 - Truth Be Told
$ filebot -mediainfo *.mkv --format "{episode}"
Alias - 1x01 - Truth Be Told

e.g. console output after manually editing xattr metadata:

Console Output: Select all

$ filebot -mediainfo *.mkv --format "{episode}"
Alias - 1x01 - XYZ
$ filebot -script dev:miss *.mkv
...
TheMovieDB::TV::2046 | Alias - 1x01 - XYZ | /path/to/Alias - S01E01 - Truth Be Told.mkv
* Episode #141453 has changed: Alias - 1x01 - XYZ -> Alias - 1x01 - Truth Be Told
...

-script dev:miss instructs filebot to fetch the latest script revision from GitHub.

Notably, the file name does not matter. We only check if online episode information matches xattr metadata episode information. Please read Metadata and Extended Attributes for details.

[dokuro]

Thanks again. How are you manually editing xattr metadata ? I just edited the file (net.filebot.metadata) and change the title within the file. Is that correct. Reason I ask is that when I run filebot -mediainfo *.mp4 --format "{episode}" after editing the data its not picking up the edit.

[rednoah]

Did you configure FileBot to use .xattr folders instead of native file system xattr?


FileBot uses native file system xattr unless configured otherwise. Please read Metadata and Extended Attributes for details.


You can read / write xattr like so:

View xattr for files:

Shell: Select all

filebot -script fn:xattr /path/to/files

Set xattr on files:

Shell: Select all

filebot -script fn:xattr /path/to/files --def name="value"

[dokuro]

Brilliant thank you, that was it. I had incorrectly assumed that xattr was supported by my file system. Once enabled and I wrote the xattr attributes then the miss script picked up the stuff that changed. This will save me a lot of manual error prone work in the future. Couple of followup questions if you don't mind

• I've never exported .xattr folders. Is the best way for me to now get this to rename again with the post process, export .xattr folders enabled ? Asking in case there is a better way for me to do it.

• I assume the answer to this is no but wanted to ask anyhow. Is there anyway with FileBot to detect episode filename on disk and compare that to the online source name ?

• Could this updated script be adopted to also detect movie title changes ? While name changes are much rarer in movies over episodes they do happen from time to time.

Again, cannot thank you enough for your help and support.

[rednoah]

FileBot writes xattr by default, which may or may not work depending on the underlying file system. You will get warnings in the log if it doesn't work.

The Export .xattr folders post-processing feature will explicitly export metadata to .xattr folders in addition to the default behaviour. By default, FileBot does not read / write .xattr dolders. If you have configured FileBot to use .xattr folders internally instead of native file system xattr, then FileBot will read / write to those files by default, and Export .xattr folders has no effect, as you just end up writing the same data to the same file paths twice.

If you have native file system xattr then you can use Export .xattr folders to export native file system xattr to .xattr folders. If you don't have native file system xattr then you will have to move / rename files with FileBot as usual.




FileBot cannot compare file names to online database entries without xattr metadata. Processing files without metadata based on the file name alone is effectively the same as processing files from scratch, detect series name, lookup series online, fetch episode list, match files to episodes, etc, just without the file renaming at the very end.




Checking for modified movie metadata would be fairly easy, but is out-of-scope for the miss.groovy script, because you can't really have "missing movies" in the way that a series can have missing episodes.

You can however do something like this:

Console Output: Select all

$ filebot -mediainfo . --filter movie --format "{n} | {y} | { info.name } | { info.released.year } | {f}"
Avatar | 2009 | Avatar | 2009 | /path/to/Avatar (2009).mp4

{n} movie name and {y} movie year are based local xattr metadata. {info} extended information is based on online information. So you can compare the two to check if something has changed. This check could be done as --filter expression to have filebot print only file paths where something has changed.

[dokuro]

Super. Again, thanks for all your help.

[rednoah]

This check could be done as --filter expression to have filebot print only file paths where something has changed.

e.g. find files where movie name or movie year metadata has changed:

Shell: Select all

filebot -mediainfo -r /path/to/movies --filter "n != info.name || y != info.released.year" --format "{n} | {y} | { info.name } | { info.released.year } | {f}"

[dokuro]

Thanks again for all your help with this thread.

As I'd never configured .xattr folders previously it took me a little time to add them for my 35,000+ episodes and 3,900+ movies. Having safely added them, both the miss script the --filter expression above are working perfectly. These take just over 5 minutes to run and will save me a massive amount time to detect these type of changes and that's not to mention its no longer error prone.

Much appreciated and thank you.

[rednoah]

Since we're now completely off topic, and as there's a new use case that may very well be useful to others, I would appreciate it if you could make a new tutorial topic for the benefit of all (and possibly your future self) so that others can learn from your experience and find your write-up and commands via a Google search.

[dokuro]

Just so I'm clear. Your asking me to create a how to topic under viewforum.php?f=3 outling the change use case and the steps I use in FileBot to identify them ? If so, sure, I can do that. However, it might be better to augment any topic which already exists for the miss use case.

[rednoah]

Just so I'm clear. Your asking me to create a how to topic under viewforum.php?f=3 outling the change use case and the steps I use in FileBot to identify them ? If so, sure, I can do that. However, it might be better to augment any topic which already exists for the miss use case.

Yep, think of helping out your past self, and thus future others. The forum can be here or another place on the internet. The topic is up to you. The content is up to you. The idea is to come up with some documentation / tutorial / explanation that would have made things easy and clear if past self had stumbled up them. Perhaps you even had some search term in mind at the time but didn't find anything. We can certainly add to the miss script but I think your past self would not have search along those lines at first.




EDIT:

I've created a new topic for my own future reference as well: Find files where online metadata has changed

[dokuro]

I had a look at the update to the miss script topic where you added This script additionally checks for changes in metadata where previously stored xattr metadata does not exactly match current online episode metadata. (e.g. find files where the episode title was changed in the online database) and to be honest this seems sufficient to me. It's clear, concise and to the point and while it I might not have found it right away when search I suspect I would have got there in the end or would have been pointed to the topic/reference.

I also took a look at the new topic -> Find files where online metadata has changed. I noticed you change the syntax for the movies from

filebot -mediainfo -r /path/to/movies --filter "n != info.name || y != info.released.year" --format "{n} | {y} | { info.name } | { info.released.year } | {f}"
to
filebot -mediainfo -r "/path/to/Movies" --filter "none{ n == info.name && y == info.released.year }" --format "{n} | {y} | { info.name } | { info.released.year } | {f}"

Just some feedback. I didn't find this change really that useful. I have poster.jpg and fanart.jpg within each movie folder and this new syntax outputs all of them which is not very useful for me. Also, the one you added find episode files, I could not get working at all. It just printed every episode and I've not had a chance to debug what wrong with it, the miss script is much better.

Also, I've noticed I need to clear the cache to get accurate results. I am trying to detect changes over at TMDB so cached data is not great for this use case. I've read the doc's viewtopic.php?t=1996 & viewtopic.php?t=9594 regarding cache clear and wanted to check if weekly manual clearing is okay ?

[rednoah]

Good point. I've updated Find files where online metadata has changed to pass over files without appropriate xattr metadata.

You do not need to clear the cache, unless you have made changes to the database yourself very recently, and want to run tests with the updated information immediately. By default (e.g. if you were to check for changed information every day, which I would strongly discourage as being extremely wasteful and will likely get you banned) then FileBot will typically fetch the latest episode information ~2 times a week, varying by chosen database and series at hand, possibly daily for ongoing daily shows where things are likely to change multiple times a week, possibly monthly for shows that have long since ended and are unlikely to ever change.

If you check for changed episode / movie information monthly then clearing the cache beforehand does nothing; except prevent FileBot from sending efficient If-Modified-Since / If-None-Match requests that allow the database to send back nothing if nothing which is very efficient when things mostly stay the same.

If you check for changed episode / movie information daily or weekly then I have to ask why? (NOTE: behaviour that can be interpreted as mirroring the database will likely get you preemptively banned from our end; so our end doesn't get banned by the database)

[dokuro]

Not sure what to say regarding the clear cache but my experience in the past week is that when TMDB changed the titles for these two episodes https://www.themoviedb.org/tv/114461-ah ... /episode/4 and https://www.themoviedb.org/tv/693-despe ... episode/13 that the miss script did NOT detect the changes until I cleared the cache. Maybe I just did not wait long enough for the cache to organically get refreshed itself.

I'd like to run this script weekly to pickup changes, its certainly not daily. I'll also add to your point possibly monthly for shows that have long since ended and are unlikely to ever change is exactly the change I'm trying to detect and the ones I'm worried about. As an example (only one example), TMDB changed the season 1 episode order of Star Trek Enterprise https://www.themoviedb.org/tv/314-star- ... e/season/1 which then messed up the metadata on my plex server as the names etc... all changed. This is an old show and changes like this I'd like to scan for weekly which I had been doing manually for a while now but am now using the miss script.

[rednoah]

You'll want to run your own tests to confirm. You can schedule a command like this to run every day and then check the changes over time at the end of the week:

Shell: Select all

filebot -list --q 114461 --db TheMovieDB::TV --format "{today} | {episode}" | tee -a 114461.txt

Ahsoka is a new ongoing series, so FileBot will always get the latest episode list if you process a new episode of Ahsoka once per week. If FileBot were to cache episode listings for too long, then you would see dozens of bug reports here in the forums, since FileBot can't process files that are not in the episode list yet.

Desperate Housewives is a ~20 year old series that ended ~10 years ago. Since this is a series that has long since ended and is thus not expected to be updated frequently, FileBot will check for updates monthly. Do you expect weekly changes in episode numbers and episode titles in Sep 2023? And if there are, do you mind if you only detect those changes in Oct 2023? Does Plex really refresh metadata periodically by default for all files in the library for all Plex users? (IDK if Plex does that; maybe; but seems like excessive load on TheMovieDB and friends to just to make Plex work less well)




EDIT:

Note that in the context of current versions of FileBot, "daily" means every 16 hours, "weekly" means every 4 days, "monthly" means every 24 days, to stay well below the targeted refresh period:

Groovy: Select all

ONE_DAY = Duration.ofHours(16)
ONE_WEEK = Duration.ofDays(4)
ONE_MONTH = Duration.ofDays(24)

[rednoah]

It just printed every episode and I've not had a chance to debug what wrong with it, the miss script is much better.

Here's how you can run your own tests and see where things go differently for you:

1. Use .xattr folders so we can edit xattr metadata easily and fake changes:

Console Output: Select all

$ filebot -script fn:properties --def net.filebot.xattr.store=.xattr
...

2. Rename sample file and generate xattr metadata:

Console Output: Select all

$ filebot -rename Alias.1x01.mp4
...
[MOVE] from [Alias.1x01.mp4] to [Alias - 1x01 - Truth Be Told.mp4]
...

3. Modify xattr metadata to create an artificial difference:

Console Output: Select all

$ vi ".xattr/Alias - 1x01 - Truth Be Told.mp4/net.filebot.metadata"
...
$ cat ".xattr/Alias - 1x01 - Truth Be Told.mp4/net.filebot.metadata"
{"@type":"Episode","seriesName":"Alias","season":1,"episode":1,"title":"XYZ",...

4. Find changes:

Console Output: Select all

$ filebot -mediainfo -r . --filter "episode && none{ episode as String == episodelist.find{ it.episode == episode }.episode as String }" --format "{episode} | {episodelist.find{ it.episode == episode }.episode} | {f}"
Alias - 1x01 - XYZ | Alias - 1x01 - Truth Be Told | /path/to/Alias - 1x01 - Truth Be Told.mp4

[dokuro]

Thank you again. The updated movie syntax to include -file-filter "f.video" --filter "movie && none has returned the expected results. Much appreciated. However, I'm not having much luck testing the episodes one.

When I run (which is directed at the show level directory)

Shell: Select all

filebot -mediainfo -r "/data/MediaServer/8b-TV/Sci-Fi & Fantasy/Ahsoka (2023) {tmdb-114461}" --filter "episode && none{ episode as String == episodelist.find{ it.episode == episode }.episode as String }" --format "{episode} | {episodelist.find{ it.episode == episode }.episode} | {f}"

I get this output ...

Console Output: Select all

Ahsoka - 1x04 - Part Four | Ahsoka - 1x04 - Part Four: Fallen Jedi | /data/MediaServer/8b-TV/Sci-Fi & Fantasy/Ahsoka (2023) {tmdb-114461}/Season 01/Ahsoka - S01E04 - Part Four.mp4

All good and what we want to accomplish but when I then run (which is directed at the root directory)

Format: Select all

filebot -mediainfo -r "/data/MediaServer/8b-TV" --filter "episode && none{ episode as String == episodelist.find{ it.episode == episode }.episode as String }" --format "{episode} | {episodelist.find{ it.episode == episode }.episode} | {f}"

I get this output ...

Console Output: Select all

...
Ahsoka - 1x01 - Part One: Master and Apprentice |  | /data/MediaServer/8b-TV/Sci-Fi & Fantasy/Ahsoka (2023) {tmdb-114461}/Season 01/Ahsoka - S01E01 - Part One: Master and Apprentice.mp4
Ahsoka - 1x02 - Part Two: Toil and Trouble |  | /data/MediaServer/8b-TV/Sci-Fi & Fantasy/Ahsoka (2023) {tmdb-114461}/Season 01/Ahsoka - S01E02 - Part Two: Toil and Trouble.mp4
Ahsoka - 1x03 - Part Three: Time to Fly |  | /data/MediaServer/8b-TV/Sci-Fi & Fantasy/Ahsoka (2023) {tmdb-114461}/Season 01/Ahsoka - S01E03 - Part Three: Time to Fly.mp4
Ahsoka - 1x04 - Part Four |  | /data/MediaServer/8b-TV/Sci-Fi & Fantasy/Ahsoka (2023) {tmdb-114461}/Season 01/Ahsoka - S01E04 - Part Four.mp4
...

The ... in this output represents every file found for every show under this directory. I get the same results when using "/data/MediaServer/8b-TV/Sci-Fi & Fantasy" as the path of TV shows.

BTW... on the plex and your comment Does Plex really refresh metadata periodically by default for all files in the library for all Plex users?. Plex does Refresh library metadata periodically as part of their maintenance tasks nightly. Just in case you did not know, individual plex servers no longer directly request metadata from TMDB (or any other online metadata source). Plex implemented their own back-end cloud based metadata solution. This cloud based plex metadata updates from the online sources such as TMDB, TVDB, IMDB etc... but when we as plex server owners issue a refresh of the metadata its done off the plex cloud based solution. They implemented this as to not impact performance at those sites.

[rednoah]

Different results for the same files? There's a mystery. Something is different. I'd start by comparing the episode data we have with the episode data we're getting back from the database. If you're really dealing with the same files, then my first guess would be differently configured filebot commands working (maybe one with xattr folders and another one with native xattr) with different xattr metadata for each file.

I'd start by checking if we're working with the same episode metadata:

Shell: Select all

filebot -mediainfo -r /input --filter 'episode' --format '{json}'

Shell: Select all

filebot -list --db TheMovieDB::TV --q 114461 --format '{json}'

** you'll get a lot of output; just paste the first line so we can compare the metadata you have with the metadata you get from the episode list for one particular episode of your choice where things don't work

The second column being empty in your output would suggest that the episode you have is not in the episode list, either because the series ID has changed or because the episode ID has changed. You might find new information if you look at the actual values.

Refresh metadata periodically
Over the course of the month, the server will refresh the metadata for musical artists and TV shows in your library. This will help ensure that your artists have good tour date information (if you’ve enabled that feature) and that Plex Pass users of music libraries will automatically get new lyrics from LyricFind (if that’s enabled), for instance.

Note: This metadata refreshing currently only occurs for music and TV libraries (assuming the library is on the latest metadata agent).

https://support.plex.tv/articles/201553 ... led-tasks/

Looks like you'll want to disable Refresh metadata periodically if you don't want any surprises, and instead check for database changes first, and update your file names as necessary, before giving Plex the refresh command.