FileBot

Posted: **07 Oct 2021, 17:15**

When using CLI in a bash script, I was excluding this show using this filter command successfully:

--filter "n != /Magnum, P.I/"

I've decided to start using the command for excludes.txt for another show. This filter command is now failing:

Code: Select all

--filter '!(n in lines("/home/moi/excludes.txt"))'

excludes.txt contains:

Code: Select all

Magnum, P.I
CSI Crime Scene Investigation

File permissions are correct, but neither show is filtered out. What am I doing wrong?

Thanks

Alfke

Posted: **07 Oct 2021, 18:30**

maybe the missing dot at end ?

Magnum, P.I.

https://thetvdb.com/series/magnum-pi

and the colon ?

CSI: Crime Scene Investigation

https://thetvdb.com/series/csi-crime-sc ... estigation

Posted: **07 Oct 2021, 18:36**

ffs...I totally missed that.

Everything is working, thank you.

Alfke

Posted: **08 Oct 2021, 03:31**

You could use plex[1] (i.e. {plex} path series folder name) instead of n if that makes things easier. I'm not entirely sure if it does make things easier though... Easier if you copy & paste from already incorrectly organized files / folders into your exclude list. Worse if you copy & paste from TheTVDB website into your exclude list.

Using the numeric series id is also an option. That will bypass spelling mistakes entirely but might make it harder for a human to maintain the exclude list.

Posted: **08 Oct 2021, 16:37**

rednoah wrote: ↑08 Oct 2021, 03:31 Using the numeric series id is also an option. That will bypass spelling mistakes entirely but might make it harder for a human to maintain the exclude list.

This sounds interesting...so my list would look like this instead: ?

74380
72546

Posted: **08 Oct 2021, 17:14**

Presumably. I'd start by doing test runs in the Format Editor GUI. It'll say true or false.

Posted: **08 Oct 2021, 17:19**

I'll give it a shot. Thanks a bunch!

Posted: **09 Oct 2021, 02:06**

e.g. like so

Code: Select all

--filter "!readLines('%FilterPath%').join().findAll(/\d+[^(19|20)\d{2}]/)*.toInteger().contains(id)"

tvdb_excludes.txt

Code: Select all

 id	Name of TV Show / Movie
-------------------------------
74380	Magnum, P.I. (1980)
72546	CSI: Crime Scene Investigation (2000)

we only use the id other info just for easy read

Posted: **09 Oct 2021, 04:30**

If you're using a lookup table like the one suggest above, then you can use the csv() function to read it in as a Map object:

Code: Select all

none{ id.toString() in csv('/path/to/excludes.tsv') }

id in csv(...) won't work because id is an Integer value, but the Map keys are String values.

Posted: **10 Oct 2021, 20:21**

rednoah wrote: ↑09 Oct 2021, 04:30 If you're using a lookup table like the one suggest above, then you can use the csv() function to read it in as a Map object:
Code: Select all
none{ id.toString() in csv('/path/to/excludes.tsv') }
id in csv(...) won't work because id is an Integer value, but the Map keys are String values.

I couldn't get this to work (syntax errors) but Kim's solution with the table worked.

Thanks All!

Alfke

Posted: **11 Oct 2021, 18:29**

I've been watching this thread because of two shows being misnamed consistently. I made an "excludes.txt" file, and put in the tvdb ID of the incorrect shows, but it does not seem to work for me. I've used the two --filter pieces of code the original poster had tried, but neither work. I have TV and movies being moved to the same directory, otherwise I'd use a different --filter command.

My script being called:

Code: Select all

 filebot -script fn:amc /mnt/media/Downloads  --output /mnt/media --action move --conflict auto  --def clean=y -non-strict --def deleteAfterExtract=y  --log-file /mnt/media/Logs/amc.log --filter '!readLines('/mnt/media/scripts/excludes.txt').contains(n)'

Using that, I get an error:

Code: Select all

 args[18] = !readLines(/mnt/media/scripts/excludes.txt).contains(n)

Syntax Error: org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
Script3.groovy: 1: unexpected token: media @ line 1, column 17.
   !readLines(/mnt/media/scripts/excludes.txt).contains(n)
                   ^

1 error

I'm not sure where the error is, the absolute path is correct. The excludes.txt is just tvdb ID's, one line for each:

Code: Select all

If I use this:

Code: Select all

 --filter '!(n in lines("/mnt/media/scripts/excludes.txt"))'

...I get a lot of noise about includes, but in the end it doesn't use the tvdb ID's to stop the misnaming from happening. (I couldn't locate the exact shows in the log, this is a sample of the 'includes')

Code: Select all

Fetching episode data for [La Brea]
Apply filter [!(n in lines("/mnt/media/scripts/excludes.txt"))] on [10] options
Include [La Brea - 1x01 - Pilot]
Include [La Brea - 1x02 - Day Two]
Include [La Brea - 1x03 - The Hunt]
Include [La Brea - 1x04 - The New Arrival]
Include [La Brea - 1x05 - The Way Home]
Include [La Brea - 1x06 - The Fort]
Include [La Brea - 1x07 - The Storm]
Include [La Brea - 1x08 - Origins]
Include [La Brea - 1x09 - Fathers and Sons]
Include [La Brea - 1x10 - Topanga]
[10] options remaining

I don't know enough about CSV's to incorporate the other answers that seemed to help the OP. Any advice/slaps upside the head for obvious overlooks on my part are welcome.

Posted: **12 Oct 2021, 02:04**

1. Create a file that looks like this:

Code: Select all

74380	Magnum, P.I. (1980)
72546	CSI: Crime Scene Investigation (2000)

Note that we use the TAB character to separate ID ➔ NAME columns.

2.
Add this filter option to your commad:

Code: Select all

--filter "none{ id.toString() in csv('/path/to/excludes.tsv') }"

Replace /path/to/excludes.tsv with the actual path. The '...' are required.

You have removed the '...' (perhaps accidentally due to quoting the argument incorrectly) and so the format code is syntactically incorrect:

snowybunting wrote: ↑11 Oct 2021, 18:29
Code: Select all
args[18] = !readLines(/mnt/media/scripts/excludes.txt).contains(n)

You are checking if the series name {n} (as opposed to series id {id}) is in your exclude list, which is never true, and so your format code is logically incorrect:

snowybunting wrote: ↑11 Oct 2021, 18:29
Code: Select all
246641
...
Code: Select all
n in lines("/mnt/media/scripts/excludes.txt")

Posted: **12 Oct 2021, 13:11**

rednoah wrote: ↑12 Oct 2021, 02:04 1. Create a file that looks like this:
Code: Select all
74380	Magnum, P.I. (1980)
72546	CSI: Crime Scene Investigation (2000)
Note that we use the TAB character to separate ID ➔ NAME columns.

2.
Add this filter option to your commad:
Code: Select all
--filter "none{ id.toString() in csv('/path/to/excludes.tsv') }"
Replace /path/to/excludes.tsv with the actual path. The '...' are required.

OK, that makes sense. I honestly looked for a long time on explanations for lines after "--filter" and what they mean and do, but could not find anything besides old forum posts.

You have removed the '...' (perhaps accidentally due to quoting the argument incorrectly) and so the format code is syntactically incorrect:
snowybunting wrote: ↑11 Oct 2021, 18:29
Code: Select all
args[18] = !readLines(/mnt/media/scripts/excludes.txt).contains(n)

I just grabbed the error message from the log file, the '...' were truncated in that.

You are checking if the series name {n} (as opposed to series id {id}) is in your exclude list, which is never true, and so your format code is logically incorrect:
snowybunting wrote: ↑11 Oct 2021, 18:29
Code: Select all
246641
...
Code: Select all
n in lines("/mnt/media/scripts/excludes.txt")

Again, perfect sense, but at the time I was thinking somehow your code magically turned the tvdb ID into a show name for the --filter to do its work. Lots of wishful thinking/hoping on my part.

Thanks for the response, really appreciate the help.

Posted: **12 Oct 2021, 13:53**

--format expressions and --filter expressions are the same thing, so all documentation related to naming applies equally to filtering, and vice versa.

1. use --format for testing --filter expressions, so we can see what your filter expression would include / exclude:

Code: Select all

filebot -list --q CSI -non-strict  --format "{id in [1431]} | {id} | {n} | {s00e00}" | grep S01E01
true | 1431 | CSI: Crime Scene Investigation | S01E01
false | 1620 | CSI: Miami | S01E01
false | 2458 | CSI: NY | S01E01
false | 122194 | CSI: Vegas | S01E01
false | 61811 | CSI: Cyber | S01E01

2. use --filter only once you've thoroughly prototyped your filter expression:

Code: Select all

filebot -list --q CSI -non-strict --filter "id in [1431]" --format "{id} | {n} | {s00e00}" | grep S01E01
1431 | CSI: Crime Scene Investigation | S01E01

You'll want to use the Format Editor GUI for prototyping filter expressions, but here I'm using filebot -list because it's more copy & paste friendly.

Posted: **12 Oct 2021, 19:23**

Maybe someone can use this (filter by id / name / year):

sample file:

Code: Select all

82618	Watchmen: Motion Comic	(2009)
90001	Pitchmen	(2009)
297138	WatchMojoq	(2016)

Code: Select all

{
	def list = lines('PathTo/tvdb_excludes.txt')*.tokenize('\t()').collect{ [ 'id': it[0]?.toInteger(), 'name': it[1], 'year': it[2]?.toInteger() ] }
	!list.id.contains(id)
}

using older version of filebot replace "lines" with "readLines"

same on one line and using "readLines":

Code: Select all

{ def list = readLines('PathTo/tvdb_excludes.txt')*.tokenize('\t()').collect{ [ 'id': it[0]?.toInteger(), 'name': it[1], 'year': it[2]?.toInteger() ] }; !list.id.contains(id) }

same but shorter:

Code: Select all

{ !readLines('PathTo/tvdb_excludes.txt')*.tokenize('\t()').collect{ [ 'id': it[0]?.toInteger(), 'name': it[1], 'year': it[2]?.toInteger() ] }.id.contains(id) }

you can also use it to search

Code: Select all

list.findAll{it.name =~ /(?i)watch/}.name

output sample:

Code: Select all

[Watchmen: Motion Comic, WatchMojoq]

btw:

Code: Select all

\t

= use 'tab' in file between id / name / year

Posted: **12 Oct 2021, 21:54**

rednoah wrote: ↑12 Oct 2021, 13:53 --format expressions and --filter expressions are the same thing, so all documentation related to naming applies equally to filtering, and vice versa.

1. use --format for testing --filter expressions, so we can see what your filter expression would include / exclude:
Code: Select all
filebot -list --q CSI -non-strict  --format "{id in [1431]} | {id} | {n} | {s00e00}" | grep S01E01
true | 1431 | CSI: Crime Scene Investigation | S01E01
false | 1620 | CSI: Miami | S01E01
false | 2458 | CSI: NY | S01E01
false | 122194 | CSI: Vegas | S01E01
false | 61811 | CSI: Cyber | S01E01
2. use --filter only once you've thoroughly prototyped your filter expression:
Code: Select all
filebot -list --q CSI -non-strict --filter "id in [1431]" --format "{id} | {n} | {s00e00}" | grep S01E01
1431 | CSI: Crime Scene Investigation | S01E01
You'll want to use the Format Editor GUI for prototyping filter expressions, but here I'm using filebot -list because it's more copy & paste friendly.

This is good stuff, very helpful. I'd not seen the --format page before, obviously that would have helped me. I've followed your tips, everything seems to be working as expected, just curious if having an "excludes" file for filebot to work from lets it work more efficiently, at least according to how I'm using it? Does using the --filter before testing has succeeded cause filebot to not work as I'm expecting, is that why you were fairly emphatic about that?

I'm using filebot on an Ubuntu box with no gui, so the 'filebot -list' worked just fine. I guess I could install it on my windows machine to test filter expression more easily?

Posted: **13 Oct 2021, 03:30**

The id in [1431] bit is just one example. It's completely up to you and your code. Include / Exclude by id / name / airdate / country / genre / etc. Load data from local files, the internet, etc. Any code that some how comes up with true | false will do.

e.g. include ID

Code: Select all

id in [1431]

e.g. exclude ID

Code: Select all

none{ id in [1431] }

e.g. include ID, where list of allowed IDs is loaded from an external text file:

Code: Select all

id in lines('/path/to/ids')

e.g. exclude ID, where list of allowed IDs is loaded from an external text file:

Code: Select all

none{ id in lines('/path/to/ids') }

e.g. exclude classic TV Shows:

Code: Select all

y < 1990

That's just a few examples that are known to work well for some users. But you don't have to do it that way. You can do absolutely anything and the possibilities are infinite.

You can use filebot-xpra docker container to run a Linux Desktop environment with the FileBot Desktop application on any device.

Posted: **08 Jul 2022, 14:20**

What would be the construction for a --filter that had both includes and excludes, by referencing two different text files?

Also, it looks like the command you give above gets the ID from TheMovieDB by default
So how do we specify on the command line which database to get the ID from in order to test our include/exclude file?
The script I use has specified seriesDB=TheTVDB

After some random digging found this "--db TheTVDB" for the command line ( Surprised that when running "filebot" the default database was "TheMovieDB" )

This is a working solution for me with a text file that includes the following line:

Code: Select all

382389	Star Trek: Strange New Worlds

Code: Select all

filebot -list --q "Star Trek: Strange New Worlds" --db TheTVDB -non-strict --filter "id.toString() in csv('/Users/john/.filebot/tvdb_includes.txt') " --format "{id} | {n} | {s00e00}"
[2022-07-08 10:03:52.923] Apply filter [id.toString() in csv('/Users/john/.filebot/tvdb_includes.txt')] on [61] options
[2022-07-08 10:03:53.079] Include [Star Trek: Strange New Worlds - 1x01 - Strange New Worlds]
[2022-07-08 10:03:53.082] Include [Star Trek: Strange New Worlds - 1x02 - Children of the Comet]
[2022-07-08 10:03:53.083] Include [Star Trek: Strange New Worlds - 1x03 - Ghosts of Illyria]
[2022-07-08 10:03:53.084] Include [Star Trek: Strange New Worlds - 1x04 - Memento Mori]
[2022-07-08 10:03:53.086] Include [Star Trek: Strange New Worlds - 1x05 - Spock Amok]
[2022-07-08 10:03:53.087] Include [Star Trek: Strange New Worlds - 1x06 - Lift Us Where Suffering Cannot Reach]
[2022-07-08 10:03:53.088] Include [Star Trek: Strange New Worlds - 1x07 - The Serene Squall]
[2022-07-08 10:03:53.089] Include [Star Trek: Strange New Worlds - 1x08 - The Elysian Kingdom]
[2022-07-08 10:03:53.090] Include [Star Trek: Strange New Worlds - 1x09 - All Those Who Wander]
[2022-07-08 10:03:53.091] Include [Star Trek: Strange New Worlds - 1x10 - A Quality of Mercy]
[2022-07-08 10:03:53.092] Include [Star Trek: Strange New Worlds - Special 2 - Inside The Series]
[2022-07-08 10:03:53.115] [11] options remaining
382389 | Star Trek: Strange New Worlds | S01E01
382389 | Star Trek: Strange New Worlds | S01E02
382389 | Star Trek: Strange New Worlds | S01E03
382389 | Star Trek: Strange New Worlds | S01E04
382389 | Star Trek: Strange New Worlds | S01E05
382389 | Star Trek: Strange New Worlds | S01E06
382389 | Star Trek: Strange New Worlds | S01E07
382389 | Star Trek: Strange New Worlds | S01E08
382389 | Star Trek: Strange New Worlds | S01E09
382389 | Star Trek: Strange New Worlds | S01E10
382389 | Star Trek: Strange New Worlds | S00E02

Posted: **08 Jul 2022, 21:49**

1.
--filter allows for arbitrary code that evaluates to true or false. You can use && and or || or in your code:
https://docs.groovy-lang.org/latest/htm ... _operators

e.g. id in a.csv or id in b.csv

Code: Select all

id in csv('a.csv') || id in csv(/'b.csv'/)

2.
You can use {id} in your format during testing so that you can see the value {id} has for any given use case. {id} is the ID for the database that is being used. If you are using TheTVDB, then it'll be the TheTVDB ID.

e.g. for debugging purposes, we can just print binding values so that we know what we are working with, and then do nothing by having the expression always return true:

Code: Select all

--filter "println id; println series; return true"

3.
TheMovieDB is a website that supports both Movie and Series information and FileBot uses it by default unless otherwise specified. TheTVDB is a website that also supports both Movie and Series information, though FileBot only supports the latter. Older versions of FileBot used TheTVDB by default. The default databases are configurable with newer versions.

Posted: **12 Jul 2022, 19:16**

I added a list of tv shows (that seem to consistently be incorrectly matched) in my includes file to set FB straight, unfortunately if I download something else, a movie for instance, that's NOT in the includes list, it is not acted on. I really don't want to have to add a movie to the includes list before I download it every time.

What is the proper way to deal with this?

In FB jargon, "include" really means "only match items on this list, forget about everything else" - not "prefer these items among other items".

Confusing for a user since including something means we are adding it to a group of existing things not isolating it from existing things.

This leaves me with the option of finding every possible mismatch and entering those in an exclude list? I think we've been over this before... maybe I am just senile

Posted: **12 Jul 2022, 19:25**

e.g. there are many ways, but I'd do it like this:

Code: Select all

any{ movie }{ n in ['Firefly'] }

The movie binding will throw an error if the object at hand is not a movie, and so all Movie type objects evaluate to true and thus all Movie type objects are included. The remainder of the code is the series episode include / exclude logic.

Posted: **12 Jul 2022, 19:32**

rednoah wrote: ↑12 Jul 2022, 19:25 e.g. there are many ways, but I'd do it like this:
Code: Select all
any{ movie }{ n in ['Firefly'] }
The movie binding will throw an error if the object at hand is not a movie, and so all Movie type objects evaluate to true and thus all Movie type objects are included. The remainder of the code is the series episode include / exclude logic.

Thank you.
That still leaves the issue of tv shows not on the include list being ignored...
I am using an amc script and running a shell script after torrent download.

Code: Select all

{
  /usr/local/bin/filebot -script fn:amc --output "/Volumes/PlexMedia/PlexServer_1" \
--action copy \
-non-strict \
--conflict index \
--log-file amc.log \
--def minFileSize=10000000 \
--def minLengthMS=300000 \
--def "ut_kind"="multi" \
--def "ut_title"="${1}" \
--def "ut_label"="${2}" \
--def "ut_dir"="${3}" \
--def "ut_rpath"="${4}" \
--def "ut_spath"="${5}" \
--def "ut_files"="${6}" \
--def "ut_bytes"="${7}" \
--def "ut_track"="${8}" \
--def "ut_info"="${9}" \
--def "ut_tags"="${10}" \
--def excludeList="/Users/john/.filebot/amc_excludes.txt" \
--def unsorted=y \
--def music=y \
--def skipExtract=y \
--def movieDB=TheMovieDB seriesDB=TheTVDB animeDB=AniDB musicDB=ID3 \
--def seriesFormat="/Volumes/PlexMedia/PlexServer_1/{plex.derive{' {thetvdb-'}{id}{'}'}{' - ['+allOf{tags}{vf}{vs}{crc32}.join(' ')}{']'}}{if (dc > 1) '.'+di}" \
--def movieFormat="/Volumes/PlexMedia/PlexServer_1/{plex.derive{' {imdb-'}{imdbid}{'}'}{' ['+allOf{tags}{audio.language}{if ('Documentary' in genres)'[doc]'}{info:video[0].displayAspectRatioString.colon('"∶"').replace('?', '')}{ws}{vf}{vs}{vc}{runtime}{crc32}.join(' ')}{']'}}{if (dc > 1) '.'+di}" \
"ut_title"="${1}" "ut_label"="${2}" "ut_dir"="${3}" \
--filter "true && id.toString() in csv('/Users/john/.filebot/tvdb_includes.txt') " \
-exec /Users/john/.filebot/scripts/fb_remove_non-eng_subs.sh {folder}
} >> /Users/john/.filebot/scripts/qBT_e-program_out.txt 2>&1

Posted: **12 Jul 2022, 20:17**

Yep, if you're using --filter then conceptually it's always going to a whitelist approach (i.e. you know what you want) or a blacklist approach (i.e. you know what you don't want) but it doesn't have to be based on the series name.

I'd typically recommend some generic rule such as "include only episodes aired within the last 7 days". If your use case is "do everything" that you're gonna have a hard time, because filebot would already do that if it could do that, but if you can somehow simplify the problem and limit the use case, then there might be simple and effective solutions.

You may want to write your own top-level script that then performs different actions for different files, i.e. run tailored commands for some files, but more generic commands for other files. Please read How do I process specific files with specific command-line options? for details.

Please please paste the log whenever you encounter a mismatch.

FileBot

Excluding multiple shows with excludes.txt

Excluding multiple shows with excludes.txt

Re: Excluding multiple shows with excludes.txt

Re: Excluding multiple shows with excludes.txt

Re: Excluding multiple shows with excludes.txt

Re: Excluding multiple shows with excludes.txt

Re: Excluding multiple shows with excludes.txt

Re: Excluding multiple shows with excludes.txt

Re: Excluding multiple shows with excludes.txt

Re: Excluding multiple shows with excludes.txt

Re: Excluding multiple shows with excludes.txt

Re: Excluding multiple shows with excludes.txt

Re: Excluding multiple shows with excludes.txt

Re: Excluding multiple shows with excludes.txt

Re: Excluding multiple shows with excludes.txt

Re: Excluding multiple shows with excludes.txt

Re: Excluding multiple shows with excludes.txt

Re: Excluding multiple shows with excludes.txt

Re: Excluding multiple shows with excludes.txt

Re: Excluding multiple shows with excludes.txt

Re: Excluding multiple shows with excludes.txt

Re: Excluding multiple shows with excludes.txt

Re: Excluding multiple shows with excludes.txt

Re: Excluding multiple shows with excludes.txt