[amc] Fixing bad matches & improving auto-detection

[rednoah]

Try the --filter option. If the US and DE version pass first round of series-detection, then you can eliminate the misleading US matches via your --filter expression.

But since the DE version has 0 ratings, it might be excluded from the start as "too unpopular for anyone to care". So just give it a 10 rating if you want FileBot to give it more consideration.

@see viewtopic.php?f=3&t=2127

[hellmaker]

Mhhh it becomes Unsorted...........
I´m doing anything wrong with the filter?

It seems that this filter is applied AFTER it is identify the episode........

Call:

Code: Select all

filebot -script 'fn:amc' /volume1/TV-Aufnahmen/sort --output /volume1/TV-Aufnahmen/Fertig --filter 'n != /Undercover Boss (US)/' --action test -non-strict --conflict override --lang de --def 'ut_label=TV' 'unsorted=y' 'artwork=y' 'clean=y' 'deleteAfterExtract=y' 'minLengthMS=600000' --log all --log-file '/volume1/@appstore/filebot-node/filebot.log' >> '/volume1/TV-Aufnahmen/filebot.log' 2>&1

Code: Select all

Locking /volume1/@appstore/filebot-node/filebot.log
Run script [fn:amc] at [Wed Jan 20 11:59:13 CET 2016]
Parameter: ut_label = TV
Parameter: unsorted = y
Parameter: artwork = y
Parameter: clean = y
Parameter: deleteAfterExtract = y
Parameter: minLengthMS = 600000
Argument: /volume1/TV-Aufnahmen/sort
Input: /volume1/TV-Aufnahmen/sort/Stargate-Universe.3.Angedockt.mkv
Input: /volume1/TV-Aufnahmen/sort/Undercover Boss.MeinFernbus FlixBus.mkv
Group: [tvs:stargate universe] => [Stargate-Universe.3.Angedockt.mkv]
Group: [tvs:undercover boss us] => [Undercover Boss.MeinFernbus FlixBus.mkv]
Rename episodes using [TheTVDB]
Auto-detected query: [Stargate Universe]
Fetching episode data for [Stargate Universe]
Apply Filter: {n != /Undercover Boss (US)/}
Skipped [/volume1/TV-Aufnahmen/sort/Stargate-Universe.3.Angedockt.mkv] because [/volume1/TV-Aufnahmen/Fertig/TV Shows/Stargate Universe/Season 02/Stargate Universe - S02E03 - Angedockt.mkv] already exists
Processed 1 files
Rename episodes using [TheTVDB]
Auto-detected query: [Undercover Boss (US)]
Fetching episode data for [Undercover Boss (US)]
Apply Filter: {n != /Undercover Boss (US)/}
No matching episode: Undercover Boss.MeinFernbus FlixBus.mkv
CmdlineException: Unable to match files to episode data
Processing 1 unsorted files
[TEST] Rename [/volume1/TV-Aufnahmen/sort/Undercover Boss.MeinFernbus FlixBus.mkv] to [/volume1/TV-Aufnahmen/Fertig/Unsorted/Undercover Boss.MeinFernbus FlixBus.mkv]
Processed 1 files
Done ヾ(＠⌒ー⌒＠)ノ

[rednoah]

--filter is a FileBot feature that works on episode object data, not series object data. Series/Movie differentiation as feature specific to the amc script.

The --filter rules is good. It eliminates the bad matches. Too bad the good matches aren't even there to be considered in the first place.

After a few weeks it might magically start working (or start giving you different mismatches from the Canadian/Australian/Dutch version of that show).

[rednoah]

So I ran this test with clean data myself:

Code: Select all

Group: [tvs:undercover boss] => [Undercover Boss.MeinFernbus FlixBus.mkv]
Rename episodes using [TheTVDB]
Auto-detected query: [Undercover Boss]
Fetching episode data for [Undercover Boss]
Fetching episode data for [Undercover Boss (US)]
Fetching episode data for [Undercover Boss (AU)]
Fetching episode data for [Undercover Boss (NO)]
Fetching episode data for [Undercover Boss (CA)]

This is how it should look like. All these shows match "Undercover Boss". However, FileBot will only consider the Top 5 results and episode data thereof. Hence, you cannot make Undercover Boss (DE) work. There's just too many shows with the same alias in this very specific case!

In this case, you'll have to replace occurrences of "Undercover.Boss" with "Undercover.Boss.DE" in all filenames before you call the amc script.

EDIT:

At this point, only has Undercover Boss (US) has any ratings, but since we added ratings to Undercover Boss (DE) it should be listed as part of the Top 5 results (pushing out out AU/NO/CA) sooner or later.

The reason you and I get different results is probably caused by xattr. My test files are pristine, while your files are already xattr tagged (due to previous rename operations). Remember to test with --action test otherwise your test results will be tainted.

[hellmaker]

Ok lets wait for the next episode and see whats happens.

Lets see how the magic is working

xattr? i dont know? isnt it a mac os x thing?

Thanks for your help so far. I will be back ;P

[rednoah]

It's called xattr on Unix-based systems, and Extended Attributes on Windows. Same thing though, allows you to store arbitrary data as file attributes.

@see viewtopic.php?t=5#p5394

[hellmaker]

ok i get that.

I have another question:
Using "--conflict auto" says it only overwrite, when the new file is better.
But what mean better? Is it filesize bigger?
Or what is the indicator for the file to be better?

I´m looking for a "--conflict index" function which keeps both files, but the new one becomes an index like #2 and so.
So i have both files listed and can terminate so wrong one by my own.

[rednoah]

"better" is vague on purpose. Currently video resolution and filesize is used, but behaviour may change if someone comes up with better logic.

What you want is not impossible and can be done via the format, although it's very tricky and won't be pretty.

[hellmaker]

Ok after testing it for some days with real recordings, i want to share my minds:

Filebot is awesome and i realy love it. Its working fine and saves a lot of work and time for me.

The key for have it working perfect is the name of the recording file.
And here starts my problem.
Sometime i´m not able to create good filenames so filebot is not able to do a good job.

I started a feature request for my tvheadend server which will make me able to mix individual epg data into the filename.
Hopefully it will be aproved.

An additionally option could be to run a script before filebot which will grab the epg data and rename the file how it should be.
Are there some scipts out there which do that job already?

[rednoah]

The last use case here seems to be somewhat similar to what you're asking for:
viewtopic.php?f=3&t=2072

Also, most files with bad names can still be detected. If you find issues with some of them, just keep posting the paths in the forums here. I might be able to fix things for some of them.

[hellmaker]

ok, here i´m back........

After running filebot for some days i still have problems which i can not solve by myself.

The last case was a wrong assignment like:

Code: Select all

Group: [tvs:csi crime scene investigation] => [CSI---Den-Tätern-auf-der-Spur.15.Der-Verlust-von-Leben-und-Moral-1.ts, CSI---Den-Tätern-auf-der-Spur.16.Das-Cabinet-des-Dr.-Aden.ts, Stargate.3.Das-Opfer.ts, Stargate.4.Der-fünfte-Mann.ts, Stargate.6.Das-Übergangsritual.ts]

So i want to fix this by adding a hardcoded txt list.
I can image to tell filebot when a filename starts with "Stargate.*" it should always match this the according TV series Stargat SG-1. This match should be fixed for ALL files with begining "Stargate".
Is there any way to create such a hardcoded list for fixed matching? How can i do this?
Having such a list would fix nearly up to 90% off my problems.

so i could do a list like:
CSI* -> tvs:csi crime scene investigation
Stargate* -> tvs:stargate universe
Ich bin ein Star* -> tvs:ich bin ein star holt mich hier raus

Another problem is:

Code: Select all

Apply Filter: {n != /The Hobbit (1977)/}
Include [The Hobbit (1977)]
Include [Die (2010)]
[MOVE] Rename [/volume1/TV-Aufnahmen/sortmovie/Der-Hobbit_-Die-Schlacht-der-fünf-Heere.Fantasyfilm.ts] to [/volume1/TV-Aufnahmen/Fertig/Movies/The Hobbit (1977)/The Hobbit (1977).ts]
Processed 1 files

I exclude the movie and its still matching it. Dunno why?

[rednoah]

1.
What's the full log that shows the mismatch? The "Group" step is just rough preprocessing by the amc script. It does not necessary affect the final match I the rename step. Just means that the group will be processed as one batch.

2.
The movie name {n} is the name, just the name, as opposed to {ny} which is name/year.


EDIT:


Keep in mind that --filter rules are about your behaviour. It's not about fixing bad matches (although it can be used that way) but about preventing bad matches in the first place.

e.g.
{ny != /The Hobbit (1977)/} means "I will not process The Hobbit from 1977". That's oddly specific.

e.g.
{y >= 2000} means "I will not process old movies, from before 2000". That's much more generic, and it'll kill off all the potentially bad matches you haven't encountered yet.


Other approaches might be {info.votes >= 50} roughly translates to "I only process well-known/popular movies" but it's of course somewhat unpredictable to "popular" being defined by TheMovieDB vote count.


EDIT 2:

TheMovieDB didn't contain "Der Hobbit: Die Schlacht der Fünf Heere", instead it only contained "Der Hobbit - Die Schlacht der fuenf Heere", so fuenf != fünf, no substring match.

I've updated the TheMovieDB info:
https://www.themoviedb.org/movie/122917 ... lease-info

[hellmaker]

Find the full log here:
http://pastebin.com/Cd6Z4b1L

I can not explain myself why Stargate is matched to CSI.

And yes adding the filter rule with {ny != /The Hobbit (1977)/} means it is only excluding one movie but it is still using this movie even when i exclude it with that filter rule.
Maybe its a good decision to add {y >= 2000} but for that movie it should work, shouldnt it?

Find the full log of the movie sorting here: http://pastebin.com/9gU3zn7r

So implementing such a hardcoded match file is not possible? Maybe you can identify the problem with that stargate thing. That´s my main problem.

[rednoah]

1.
Yes, both year-filter or exact-name/year-filter both will work in excluding that specific movie. Also, if the data in TheMovieDB was complete and correct, this would not have been an issue in the first place. So you can always help out by adding/correcting the German alternative titles there.

2.
The Stargate thing is probably more complicated, but boils down to Stargate SG-1 != Stargate while at the same time "Den" being identified as possible query (because query-blacklist isn't that optimized for German yet).

[hellmaker]

1.
The movie Der-Hobbit_-Die-Schlacht-der-fünf-Heere.Fantasyfilm.ts is listed with perfect matching subtitle on themoviedb.org.
https://www.themoviedb.org/movie/122917 ... ive-armies

Also adding filter rule "age" does not help to recognize it.
See log file -> http://pastebin.com/XCyDBu39

2.
Yes excatly that is the problem. How can i make Stargate SG-1 == Stargate
I want to override the normal process with a command where Stargate SG-1 == Stargate

[rednoah]

1.
FileBot has the English title, as well as all alternative titles, in its local index. However, it seems that TheMovieDB doesn't automatically list the primary titles of other languages in the alternative titles:
http://pastebin.com/7VkLgFgG

So FileBot doesn't know that Der Hobbit - Die Schlacht der fünf Heere is a movie, and it doesn't know that "Fantasyfilm" is bad, so it'll search online for Der Hobbit - Die Schlacht der fünf Heere Fantasyfilm. Adding "Fantasyfilm" to the query-excludelist will fix the latter issue. The first one can only be fixed by TheMovieDB.

2.
I've added Stargate as an alias for Stargate SG-1. Might start working at some point.

However, if you had 5 or more episodes of Stargate then it would work, or if the folder only contained Stargate episodes, or if the folder contains only "badly named" episodes of "CSI" and "Stargate".

Stargate (no match, bad numbers) VS CSI DenTätern auf der Spur (perfect match, double-digit numbers) => "Stargate" might as well be "XXYYZZ" for all FileBot knows.

Adding Stargate as known alias will fix that.

[hellmaker]

1. Sure it does. Look in line 160 of your pastebin. There you see the german subtitle listed correct as "alternative_titles".

I will make a post in the query-excludelist for eleminate such bad words like Fantasyfilm.....


2. Thanks. Hopefully it will fix my Stargate problems.

[rednoah]

1.
Der Hobbit - Die Schlacht der fuenf Heere (known to filebot) is not the same as Die Schlacht der fünf Heere (not known to filebot). FileBot can infer u|ue from ü but it can't/won't do it the other way around.

In a second step, it could make a match based on text similarity, but because it already finds a perfect match for "Der Hobbit" in the first step, there is no (more much more time-consuming / generally less reliable) second step.

[hellmaker]

Ok i think i can use an option in tvheadend which will convert unsave characters. It should convert ü to ue, hopefully...........

EDIT: Going crazy:

Change Ü to UE does not help

http://pastebin.com/5TKS6Csy

[rednoah]

Sorry, can't help with that one. Works fine here. Make sure that, there's no xattr in the file metadata, or nfo files, etc

Code: Select all

Rename movies using [TheMovieDB]
Auto-detect movie from context: [Der-Hobbit_-Die-Schlacht-der-fuenf-Heere.ts]
Stripping invalid characters from new path: Movies/The Hobbit: The Battle of the Five Armies (2014)/The Hobbit: The Battle of the Five Armies (2014)
[TEST] Rename [Der-Hobbit_-Die-Schlacht-der-fuenf-Heere.ts] to [Movies/The Hobbit The Battle of the Five Armies (2014)/The Hobbit The Battle of the Five Armies (2014).ts]

[hellmaker]

again a new problem i dont understand.

I call filebot with that line:

Code: Select all

filebot -script 'fn:amc' /volume1/TV-Aufnahmen/sortmovie --output /volume1/TV-Aufnahmen/Fertig --action move --filter 'y >= 2000' -non-strict --conflict override --lang de --def 'ut_label=MOVIE' 'unsorted=y' 'artwork=y' 'clean=y' 'deleteAfterExtract=y' 'minLengthMS=600000' --log all --log-file '/volume1/@appstore/filebot-node/filebot.log' >> '/volume1/TV-Aufnahmen/filebotmovie.log' 2>&1

But it give me an exception no file??? I dont understand.
http://pastebin.com/UxzyN3XQ

[rednoah]

There was some odd internal behaviour when --filter is set when unsorted logic is executed.

Fixed with r3485.

[rednoah]

--conflict index has been added with r3496.

[hellmaker]

Thanks.

Going to test it.

[rednoah]

Q: How do I fix misidentified files?