--filter and fine-tuning

[rednoah]

Description:

The --filter option is probably the the most powerful, yet least used option the CLI has to offer. As long as everything "just works" you will never need --filter but if you get mismatches then --filter will allow you to pass in arbitrary rules to help FileBot make the correct match. Plus, once you're familiar with format expressions you'll immediately get filter expressions.

The --filter expression allows you to control which episode or movie object is considered for matching. So if one show is confused with another due to bad naming or missing data you can simply exclude the bad one.



Examples:

If you only process recently aired episodes you could exclude any episode of any show that has been aired more than a few days ago, immediately eliminating any and all potential mismatches with similarly named TV Shows aired at different times.

Code: Select all

--filter "age < 7"

A more advanced version of the above might only include episode data of recently aired episodes if there are any recently aired episodes, but otherwise default to accepting any episode data.

Code: Select all

--filter 'age < 7 || !model.any{ it.age < 7 }'

If misleading data from a specific series is making trouble you can simply exclude it.

Code: Select all

--filter "n != /Doctor Who/"

You can manage your list of explicitly excluded (or included) shows with an external text file.

Code: Select all

--filter '!readLines('/path/to/excludes.txt').contains(n)'

Since filter expressions allow arbitrary code, you can come up with more smart or more specialized logic yourself. Feel free to share.

[Laynezilla]

Is there a complete list of filters anywhere?

[rednoah]

The number of possible filters is infinite. The available bindings (n, y, s, etc) are the same as in format expressions.

What's the problem? What are you trying to accomplish?

[sblack55]

You can manage your list of explicitly excluded (or included) shows with an external text file.

Code: Select all

--filter '!readLines('/path/to/excludes.txt').contains(n)'

Having some trouble with this syntax, probably due my failure to grasp the vagaries of script/cmdline quoting. Here are the key lines from my bash script:

Code: Select all

PVR_LOC="/volume1/PVR"
...
sFilter='!readLines('${PVR_LOC}/ExcludeShows.txt').contains(n)'
…
filebot "${sSourceFile}" ${FB_OPTS} -script fn:amc --output "${DEST}" ${sMapper} --filter "${sFilter}" --def seriesFormat="${FB_FMT_SERIES}" --def movieFormat="${FB_FMT_MOVIE}" --def "ut_label=${UTLABEL}">/tmp/fblog.txt 2>&1

And here is my log:

Code: Select all

Dec 08 15:03:38: filebot "/volume1/PVR/record/The Carbonaro Effect/The Carbonaro Effect.S01E12.An Elevated Ferret Alert.ts" -non-strict --action hardlink --conflict auto -script fn:amc --output "/volume1/PVR/General" --mapper episode.derive(0,0) --filter "!readLines(/volume1/PVR/ExcludeShows.txt).contains(n)" --def seriesFormat="TV Shows/{ny}/Season {s}/{n}.{s00e00}.{t}" --def movieFormat="" --def "ut_label=TV"
Dec 08 15:03:50: Run script [fn:amc] at [Sun Dec 08 15:03:42 EST 2019]
Dec 08 15:03:50: Parameter: seriesFormat = TV Shows/{ny}/Season {s}/{n}.{s00e00}.{t}
Dec 08 15:03:50: Parameter: movieFormat =
Dec 08 15:03:50: Parameter: ut_label = TV
Dec 08 15:03:50: Argument[0]: /volume1/PVR/record/The Carbonaro Effect/The Carbonaro Effect.S01E12.An Elevated Ferret Alert.ts
Dec 08 15:03:50: Input: /volume1/PVR/record/The Carbonaro Effect/The Carbonaro Effect.S01E12.An Elevated Ferret Alert.ts
Dec 08 15:03:50: Group: {Series=true} => [The Carbonaro Effect.S01E12.An Elevated Ferret Alert.ts]
Dec 08 15:03:50: ScriptException: org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
Dec 08 15:03:50: Script2.groovy: 1: unexpected token: PVR @ line 1, column 21.
Dec 08 15:03:50: !readLines(/volume1/PVR/ExcludeShows.txt).contains(n)
Dec 08 15:03:50: ^
Dec 08 15:03:50: 
Dec 08 15:03:50: 1 error
Dec 08 15:03:50: 
Dec 08 15:03:50: Finished without processing any files
Dec 08 15:03:50: Abort (×_×)

SysInfo:

Code: Select all

FileBot 4.8.6 (r6868)
JNA Native: 6.1.0
MediaInfo: 19.04
7-Zip-JBinding: 9.20
Chromaprint: 1.4.3
Extended Attributes: OK
Unicode Filesystem: OK
Script Bundle: 2019-12-07 (r610)
Groovy: 2.5.8
JRE: OpenJDK Runtime Environment 13.0.1
JVM: 64-bit OpenJDK 64-Bit Server VM
CPU/MEM: 4 Core / 522 MB Max Memory / 26 MB Used Memory
OS: Linux (amd64)
HW: Linux NetStore 3.10.105 #24922 SMP Wed Jul 3 16:37:24 CST 2019 x86_64 GNU/Linux synology_avoton_1815+
STORAGE: ext4 [/] @ 1.2 GB | btrfs [/volume1] @ 7 TB
DATA: /volume1/@appstore/filebot/data/admin
Package: SPK
License: FileBot License P9013368 (Valid-Until: 2020-08-23)
Done ヾ(＠⌒ー⌒＠)ノ

[rednoah]

YES:

Code: Select all

!readLines('/path/to/excludes.txt').contains(n)

This is the value you want to pass in.

NO:

Code: Select all

!readLines(/volume1/PVR/ExcludeShows.txt).contains(n)

This is the value you are actually passing in, which is syntactically incorrect Groovy code, hence syntax errors.


Hence:

Code: Select all

--filter '!readLines("/path/to/excludes.txt").contains(n)'

Please read Cmdline and Argument Passing for details.

[sblack55]

Hence:

Code: Select all

--filter '!readLines("/path/to/excludes.txt").contains(n)'

Got it working, thanks!

(Single-quotes, double-quotes, substitutions, no substitutions... sheesh, I'm dizzy!)

[bilaalpeters]

That worked, thanks so much!

[JD2005]

What's the correct syntax inside of the excludes.txt file? I'm finding contradicting information, some saying you just list the episodes line by line, others saying you need to include the 'n !=' ahead of it.

[rednoah]

The code is literally reading the file, so this piece of code implicitly specifies the format, which is values line by line:

Code: Select all

!readLines("/path/to/excludes.txt").contains(n)

e.g. /path/to/excludes.txt

Code: Select all

Alias
Firefly

This would be functionally equivalent just specifying the list of values in code instead of an external text file:

Code: Select all

![/Alias/, /Firefly/].contains(n)

--filter allows you to specify arbitrary code. Maybe your code reads a file and checks if n is in there. Maybe your code checks if n is a specific value or not. Maybe your code does something I haven't thought of yet. It's your code. Anything is possible. Hence you will find different solutions made by different people hand-crafted for different use cases.


You may find "conflicting" information because some people use the @file syntax for reading command-line arguments from external text files. That's completely different though. But if all args (including the filter option) a read from a text file anyway, then some users might prefer to specify this data inline rather than in another external text file.

[JD2005]

Thank you, I wasn't sure as Filebot comes back with the words 'Include' next to each line, as well it doesn't seem to be excluding what I'm specifying as it's being labeled incorrectly anyway. I'm using the syntax laid out in this thread so just wanted to make sure I was applying this correctly before trying to figure out where else the issue is.

In case you could give me any pointers, the show 'Watchmen' is being converted to 'Vekterne' no matter what I do... All other shows seem to be working fine except this one. I realize Vekterne is Norwegian for Watchmen, but still not sure why it's choosing this language for just this show. The show and input filename going into it is english. I am also using '--lang en' in my script, so again not sure what's happening. If you have any advice I certainly appreciate it.

Code: Select all

filebot -script fn:amc --output "/home/user/Media" --action move --conflict skip -non-strict "/home/user/Stage" --log-file amc.log --def unsorted=y music=y --lang en --filter "!readLines('/home/user/uScripts/excludes.txt').contains(n)"
"

[rednoah]

You can use filebot -list for testing:

Code: Select all

filebot -list --db TheTVDB --q Watchmen -non-strict

e.g.

Code: Select all

$ filebot -list --db TheTVDB --q Watchmen -non-strict --filter "!(n ==~ /WatchMojo|Pitchmen| Vekterne|Watchmen: Motion Comic/)"
Apply filter [!(n ==~ /WatchMojo|Pitchmen| Vekterne|Watchmen: Motion Comic/)] on [249] items
Include [Watchmen - 1x01 - It's Summer and We're Running Out of Ice]
Include [Watchmen - 1x02 - Martial Feats of Comanche Horsemanship]
Include [Watchmen - 1x03 - She Was Killed by Space Junk]
Include [Watchmen - 1x04 - If You Don't Like My Story, Write Your Own]
Include [Watchmen - 1x05 - Little Fear of Lightning]
Include [Watchmen - 1x06 - This Extraordinary Being]
Include [Watchmen - 1x07 - An Almost Religious Awe]
Include [Watchmen - 1x08 - A God Walks into Abar.]
Include [Watchmen - 1x09 - See How They Fly]
Watchmen - 1x01 - It's Summer and We're Running Out of Ice
Watchmen - 1x02 - Martial Feats of Comanche Horsemanship
Watchmen - 1x03 - She Was Killed by Space Junk
Watchmen - 1x04 - If You Don't Like My Story, Write Your Own
Watchmen - 1x05 - Little Fear of Lightning
Watchmen - 1x06 - This Extraordinary Being
Watchmen - 1x07 - An Almost Religious Awe
Watchmen - 1x08 - A God Walks into Abar.
Watchmen - 1x09 - See How They Fly

Turns out " Vekterne" has a leading space in the database, which is probably an errors, so as far as the exclude list is concerned, I'd add both values:

Code: Select all

Vekterne
 Vekterne

readLines() may or may not trim() lines implicitly. It don't know. It might. If it does, then this won't work either.


Newer revisions of FileBot implicitly trim() all values we get back from TheTVDB, as to avoid invisible leading/trailing spaces messing with us: viewtopic.php?t=1609

[JD2005]

Well in the meantime I was researching and found that my Ubuntu server's locale was blank, so I set it to en_US.utf-8. At the same time I added the ' Vekterne' to the exceptions list and reran the script and it worked! So not sure which one solved it but either way I really appreciate your assistance!

[kim]

someone renamed show in English to Vekterne