[Bug] [Linux CLI] Binding genres sometimes being undefined and sometimes being defined?!?

[dgnrt3]

Hi all,

have some quite weird behavior here and I'm not able to reproduce the error I'm seeing in the logs:

Console Output: Select all

[2024-03-12 20:35:29.596] Rename episodes using [TheMovieDB] with [Airdate Order]
[2024-03-12 20:35:32.469] Lookup via [] [234201]
[2024-03-12 20:35:32.470] Fetching episode data for [234201]
[2024-03-12 20:35:33.187] * Sexuell verfügbar [234201] | 5 episodes | 1x01-05
[2024-03-12 20:35:33.454] Lookup via [] [234201]
[2024-03-12 20:35:33.454] Fetching episode data for [234201]
[2024-03-12 20:35:33.455] * Sexuell verfügbar [234201] | 5 episodes | 1x01-05
[2024-03-12 20:35:33.457] Lookup via [] [234201]
[2024-03-12 20:35:33.458] * Sexuell verfügbar [234201] | 5 episodes | 1x01-05
[2024-03-12 20:35:33.461] Lookup via [] [234201]
[2024-03-12 20:35:33.461] Fetching episode data for [234201]
[2024-03-12 20:35:33.461] * Sexuell verfügbar [234201] | 5 episodes | 1x01-05
[2024-03-12 20:35:33.464] Lookup via [] [234201]
[2024-03-12 20:35:33.464] Fetching episode data for [234201]
[2024-03-12 20:35:33.464] * Sexuell verfügbar [234201] | 5 episodes | 1x01-05
[2024-03-12 20:35:35.287] Binding "genres": undefined
net.filebot.format.SuppressedThrowables: Expression yields empty value: Binding "genres": undefined
at net.filebot.format.ExpressionFormat.format(Unknown Source)
at net.filebot.format.ExpressionFormat.format(Unknown Source)
at net.filebot.cli.CmdlineOperations.formatMatch(Unknown Source)
at net.filebot.cli.CmdlineOperations.getRenameMap(Unknown Source)
at net.filebot.cli.CmdlineOperations.renameSeries(Unknown Source)
at net.filebot.cli.CmdlineOperations.rename(Unknown Source)
at net.filebot.cli.ArgumentProcessor.runCommand(Unknown Source)
at net.filebot.cli.ArgumentProcessor.run(Unknown Source)
at net.filebot.Main.main(Unknown Source)
Caused by: net.filebot.format.BindingException: Binding "genres": undefined
at net.filebot.format.MediaBindingBean.undefined(Unknown Source)
at net.filebot.format.ExpressionBindings$MethodBinding.invoke(Unknown Source)
at net.filebot.format.ExpressionBindings.get(Unknown Source)
at net.filebot.format.ExpressionFileFormat$1.get(Unknown Source)
at Script1.run(Script1.groovy:2)
at Script.run(Script:3)
at net.filebot.format.ExpressionEngine.evaluateScriptlet(Unknown Source)
... 9 more

[2024-03-12 20:35:35.289] Error (o_O)
[2024-03-12 20:37:13.008] Rename episodes using [TheTVDB] with [Airdate Order]
[2024-03-12 20:37:15.775] Lookup via [] [447198]
[2024-03-12 20:37:15.777] Fetching episode data for [447198]
[2024-03-12 20:37:17.364] * Sexuell verfügbar [447198] | 5 episodes | 1x01-05
[2024-03-12 20:37:17.647] Lookup via [] [447198]
[2024-03-12 20:37:17.647] Fetching episode data for [447198]
[2024-03-12 20:37:17.648] * Sexuell verfügbar [447198] | 5 episodes | 1x01-05
[2024-03-12 20:37:17.651] Lookup via [] [447198]
[2024-03-12 20:37:17.651] Fetching episode data for [447198]
[2024-03-12 20:37:17.651] * Sexuell verfügbar [447198] | 5 episodes | 1x01-05
[2024-03-12 20:37:17.654] Lookup via [] [447198]
[2024-03-12 20:37:17.654] Fetching episode data for [447198]
[2024-03-12 20:37:17.655] * Sexuell verfügbar [447198] | 5 episodes | 1x01-05
[2024-03-12 20:37:17.657] Lookup via [] [447198]
[2024-03-12 20:37:17.657] Fetching episode data for [447198]
[2024-03-12 20:37:17.658] * Sexuell verfügbar [447198] | 5 episodes | 1x01-05
[2024-03-12 20:37:21.105] [MOVE] from [/glftpd/site/incoming/at_sl/serie/Sexuell.verfuegbar.S01E01.Miki.und.Ben.GERMAN.720p.WEB.x264-TMSF/tmsf-sexuellverfuegbar-s01e01-720p.mkv] to [/glftpd/site/sorted/1080p/serie/allgemein/s/sexuell.verfügbar.2024/s01/sexuell.verfügbar.2024.s01e01.720p.x264.aac.2ch.de.mkv]
[2024-03-12 20:37:21.130] [MOVE] from [/glftpd/site/incoming/at_sl/serie/Sexuell.verfuegbar.S01E02.Du.musst.kaempfen.GERMAN.720p.WEB.x264-TMSF/tmsf-sexuellverfuegbar-s01e02-720p.mkv] to [/glftpd/site/sorted/1080p/serie/allgemein/s/sexuell.verfügbar.2024/s01/sexuell.verfügbar.2024.s01e02.720p.x264.aac.2ch.de.mkv]
[2024-03-12 20:37:21.130] [MOVE] from [/glftpd/site/incoming/at_sl/serie/Sexuell.verfuegbar.S01E03.Lady.Sheriffs.GERMAN.720p.WEB.x264-TMSF/tmsf-sexuellverfuegbar-s01e03-720p.mkv] to [/glftpd/site/sorted/1080p/serie/allgemein/s/sexuell.verfügbar.2024/s01/sexuell.verfügbar.2024.s01e03.720p.x264.aac.2ch.de.mkv]
[2024-03-12 20:37:21.130] [MOVE] from [/glftpd/site/incoming/at_sl/serie/Sexuell.verfuegbar.S01E04.Porno.Girls.GERMAN.720p.WEB.x264-TMSF/tmsf-sexuellverfuegbar-s01e04-720p.mkv] to [/glftpd/site/sorted/1080p/serie/allgemein/s/sexuell.verfügbar.2024/s01/sexuell.verfügbar.2024.s01e04.720p.x264.aac.2ch.de.mkv]
[2024-03-12 20:37:21.131] [MOVE] from [/glftpd/site/incoming/at_sl/serie/Sexuell.verfuegbar.S01E05.Liebe.und.Zeit.GERMAN.720p.WEB.x264-TMSF/tmsf-sexuellverfuegbar-s01e05-720p.mkv] to [/glftpd/site/sorted/1080p/serie/allgemein/s/sexuell.verfügbar.2024/s01/sexuell.verfügbar.2024.s01e05.720p.x264.aac.2ch.de.mkv]
[2024-03-12 20:37:21.143] Processed 5 files

Format looks like this:

Format: Select all

((none{genre} == true ? 'kein_genre' : (genre.replace(csv("/scripts/fb/media.list/media.genre.list.csv"))))+"/"+
((n.match(/.*-*|\d*/) ? (n.substring(0,1)) : (localize.deu.n.clean().substring(0,1))).toLowerCase())+"/"+
((n.match(/.*-*|\d*/) ? (localize.deu.n.lower().space('.')) : (localize.deu.n.clean())).lower().space('.'))+"."+y+"/"+
(episode.special ? 'specials' : 's'+s.pad(2))+"/"+
allOf
{(n.match(/.*-*|\d*/) ? (localize.deu.n.lower().space('.')) : (localize.deu.n.clean())).lower().space('.')}{y}{s00e00}
{vf.match(/480[pP]|576[pP]|720[pP]|1080[pP]|2160[pP]/)}{bitdepth == 10 ? '10bit' : null}{vc}{ac}{af}{any{audioLanguages[0].ISO2}{'und'}}
.join('.').space('.').lower().removeAll(','))

as you can see there already is a none{genre} condition to catch all cases in which the DBs dont have a genre for given content but as you can also see in the logs sometimes it doesnt work and only 2mins later the exact same process works.

Questions:
As my none{genre} condition doesn't seem to cover the case that this binding is 'undefined' which seems to be something different than being 'empty' (?)...in which other way am I suppose to catch those cases? I tried with any {genre} but in the case it's indeed empty this should also fail which is the reason i chose the none route.
How can one and the same filebot operation with the same variables (like files, used format, used arguments, same DB etc) fail at a given moment and minutes later it works flawlessly?
This isnt a once in a lifetime occurance/statistical anomaly. Got several cases in my logfiles if some one is interested

Any input is appreciated. Cheers

[rednoah]

Can you provide a reproducible command? (i.e. complete command and console output)


Can you provide context as to how your automated setup works? Container? How does automation work? How is FileBot called? What happens if multiple FileBot instances are called at the same time? Is it possible that there are multiple FileBot instances called with different arguments / different application data that thus behave differently?




Here's what I get:

Console Output: Select all

$ filebot -list --q 447198 --db TheTVDB --lang German --format "{genre}/{episode}"
Mini-Series/Sexuell verfügbar - 1x01 - Miki und Ben
...

Here's how it works if something does not work:

Console Output: Select all

$ filebot -list --q 447198 --db TheTVDB --lang German --format "1 { any{ xyz }{ 'No XYZ' } }/2 { none{ xyz } }/3 { none { '' } }/4 { none{ null } }/{ episode }"
1 No XYZ/2 true/3 true/4 true/Sexuell verfügbar - 1x01 - Miki und Ben
...

[dgnrt3]

Sure thing. Here's my setup:

- Linux Root Server (Debian 6.1.76-1 (2024-02-01) x86_64 GNU/Linux 6.1.0-18-amd64 #1)
- direct filebot installation via APT (installed via DEB package)
- got a self made .bash file which is sourced by .bashrc
- as far as I'm aware it shouldn't happen that multiple filebot instances are called at the same time (and haven't tested it for that reason tbh)
- i checked my logs and got myself the files on which it was happening. Tried it via CLI, via the same loops that where used in the first place, via GUI and wasnt able to reproduce it once let alone consistently.
- contents of .bash file:

Shell: Select all

#defining needed paths
flmpath=("/media/incoming/at_sl/film" "/media/incoming/p2p/film")
serpath=("/media/incoming/at_sl/serie" "/media/incoming/p2p/serie")

#sourcing needed scripts
for file in /scripts/*.sh; do
source "$file"
done

# media management aliases

alias fbflm='for i in "${flmpath[@]}"; do uacln "$i"; done; for i in "${flmpath[@]}"; do filebot "$i" \@/scripts/fb/args/inc.film.tmdb.args.txt; done '
alias fbser='for i in "${serpath[@]}"; do uacln "$i"; done; for i in "${serpath[@]}"; do filebot "$i" \@/scripts/fb/args/inc.serie.tvdb.args.txt ; filebot "$i" \@/scripts/fb/args/inc.serie.tmdbtv.args.txt; done '

Note: the command uacln is an external .sh script sourced from /scripts and does nothing more then running unrarall over the arrays.

Content of .args.txt

Groovy: Select all

cat fb/args/inc.serie.tmdbtv.args.txt
-r
-rename
--file-order
[vc,height]
--conflict
INDEX
-non-strict
--db
TheMovieDB::TV
--lang
German
--output
/glftpd/site/sorted/
--format
/scripts/fb/formats/lang_check.groovy
--log-file
/var/log/fb/inc.serie.log

Groovy: Select all

cat fb/args/inc.film.tmdb.args.txt
-r
-rename
--file-order
[vc,height]
--conflict
INDEX
-non-strict
--db
TheMovieDB
--lang
German
--output
/glftpd/site/sorted/
--format
/scripts/fb/formats/lang_check.groovy
--log-file
/var/log/fb/inc.film.log

Format: Select all

cat fb/formats/lang_check.groovy
{evaluate(any{localize.deu.n} ?
new File("/scripts/fb/formats/de.format.groovy").text :
new File("/scripts/fb/formats/en.format.groovy").text
)

Fornat: https://pastebin.com/4TkhA5Rn

[rednoah]

I see you are using --db TheMovieDB::TV in your argument array, but the console output in the OP suggests that you're switching between --db TheMovieDB::TV and --db TheTVDB, or doing both in this case, one works, the other one does not work:

Code: Select all

[2024-03-12 20:35:29.596] Rename episodes using [TheMovieDB] with [Airdate Order]
[2024-03-12 20:35:32.469] Lookup via [] [234201]

Code: Select all

[2024-03-12 20:37:13.008] Rename episodes using [TheTVDB] with [Airdate Order]
[2024-03-12 20:37:15.775] Lookup via [] [447198]

{genre} is indeed undefined for TheMovieDB::234201 so we can run tests with that:

Console Output: Select all

$ filebot -list --q 234201 --db TheMovieDB::TV --lang German --format "{ any{ genre }{ 'No Genre' } }/{ episode }"
No Genre/Sexuell verfügbar - 1x01 - Miki und Ben
...

You are loading format code at runtime, depending on the series at hand. This can be confusing. If en.format.groovy and de.format.groovy are mostly copy & paste (i.e. mostly identical with minor edits) then you can probably express what you want within a single Groovy script instead and thus simplify things. The code you posted itself seems to be the same 10 lines copy & pasted over and over with minor edits. You will have a very *very* difficult editing and maintaining code like that over time. I don't know what your code does, so I can't rewrite it for you, but it's certainly possible to rewrite good clean code without copy & paste instead. It's a time & effort investment that you will want to make rather sooner than later.



In any case, the script you posted will immediately fail at line 2 if {genres} is undefined according to the The Unwind-on-Undefined Behaviour:

Groovy: Select all

genres.contains('Reality')

Console Output: Select all

$ filebot -list --q 234201 --db TheMovieDB::TV --lang German --format "{ genres.contains('Reality') }"
Expression yields empty value: Binding "genres": undefined
...

I'm not sure why you would purge_with_fire any series from before 2010 in the first place... Maybe you meant && and instead of || or?

Groovy: Select all

( genres.contains('Reality') || genres.contains('Soap') || y < 2010 ) ? 'purge_with_fire/' +

You probably meant to write something like this:

Groovy: Select all

any{ genres =~ /Reality|Soap/ && y < 2010 ? 'Old Soap' : 'New Soap or Non-Soap' }{ 'No Genre or No Year' }

Console Output: Select all

$ filebot -list --q 234201 --db TheMovieDB::TV --lang German --format "{ any{ genres =~ /Reality|Soap/ && y < 2010 ? 'Old Soap' : 'New Soap or Non-Soap' }{ 'No Genre or No Year' } }"
No Genre or No Year
...

[dgnrt3]

You are loading format code at runtime, depending on the series at hand. This can be confusing. If en.format.groovy and de.format.groovy are mostly copy & paste (i.e. mostly identical with minor edits) then you can probably express what you want within a single Groovy script instead and thus simplify things. The code you posted itself seems to be the same 10 lines copy & pasted over and over with minor edits. You will have a very *very* difficult editing and maintaining code like that over time. I don't know what your code does, so I can't rewrite it for you, but it's certainly possible to rewrite good clean code without copy & paste instead. It's a time & effort investment that you will want to make rather sooner than later.

There are several reasons for the code to be like that. We have quite big media server (roughly 200 TB) and there are plex libraries like: Anime-Series, Documentation Series, Documentation Movies, 4K Movies etc. So there's a need to categorize the content based on several factors:
- Height (4K or not?)
- Series or Movie?
- Content for Children or not?
- Anime or not?

I agree that there is much similarity for the individual blocks....only thing important is the logic for categories and corresponding output path
The only reason to make it simpler I see at this moment is to split up the code blocks into several files and call them as needed as the only real diffences are the appendage to the output path (from args file) and series have S00E00 in them while movies obviously dont.

I'm not sure why you would purge_with_fire any series from before 2010 in the first place... Maybe you meant && and instead of || or?

nope, that's totally intended (ofc with a small joke towards my colleagues which have some tools that automatically fetch re-releases of huge reality/soap tv series and old tv - stuff which just has been re-released by some dude) but as i dont wanna delete them all together as there may have been someone who specifically requested something this is the solution for this.

Got the existing code from: viewtopic.php?p=60568#p60568
But i see the potential point of failure you pointed out and really appreciate your new code for it (will just replace && with ||)

I see you are using --db TheMovieDB::TV in your argument array, but the console output in the OP suggests that you're switching between --db TheMovieDB::TV and --db TheTVDB, or doing both in this case, one works, the other one does not work:

this is correct. First loop trough the array is with TMDB::TV and the second one is supposed to catch the content which isnt listed at TMDB::TV and try it on TVDB.

[rednoah]

The only reason to make it simpler I see at this moment is to split up the code blocks into several files and call them as needed

Take the bit at the end of your code for instance. You only switch between 2160p and 1080p, and yet you duplicate everything:

Format: Select all

{
( hd == /UHD/ ?
( '2160p/film/' +
((none{genre} == true ? 'kein_genre' : (genre.replace(csv("/scripts/fb/media.list/media.genre.list.csv"))))+"/"+
((n.match(/.*-*|\d*/) ? (n.substring(0,1)) : (localize.deu.n.clean().substring(0,1))).toLowerCase())+"/"+
((n.match(/.*-*|\d*/) ? (localize.deu.n.lower().space('.')) : (localize.deu.n.clean())).lower().space('.'))+"."+y+"/"+
allOf
{(n.match(/.*-*|\d*/) ? (localize.deu.n.lower().space('.')) : (localize.deu.n.clean()))}{y}
{vf.match(/480[pP]|576[pP]|720[pP]|1080[pP]|2160[pP]/)}{bitdepth == 10 ? '10bit' : null}{vc}{ac}{af}{any{audioLanguages[0].ISO2}{'und'}}
.join('.').space('.').lower().removeAll(','))
)
:
( '1080p/film/' +
((none{genre} == true ? 'kein_genre' : (genre.replace(csv("/scripts/fb/media.list/media.genre.list.csv"))))+"/"+
((n.match(/.*-*|\d*/) ? (n.substring(0,1)) : (localize.deu.n.clean().substring(0,1))).toLowerCase())+"/"+
((n.match(/.*-*|\d*/) ? (localize.deu.n.lower().space('.')) : (localize.deu.n.clean())).lower().space('.'))+"."+y+"/"+
allOf
{(n.match(/.*-*|\d*/) ? (localize.deu.n.lower().space('.')) : (localize.deu.n.clean()))}{y}
{vf.match(/480[pP]|576[pP]|720[pP]|1080[pP]|2160[pP]/)}{bitdepth == 10 ? '10bit' : null}{vc}{ac}{af}{any{audioLanguages[0].ISO2}{'und'}}
.join('.').space('.').lower().removeAll(','))
))
}

You could write the exact same logic without copy & paste:

Format: Select all

{
	(hd == /UHD/ ? '2160p/film/' : '1080p/film/')+
	((none{genre} == true ? 'kein_genre' : (genre.replace(csv("/scripts/fb/media.list/media.genre.list.csv"))))+"/"+
	((n.match(/.*-*|\d*/) ? (n.substring(0,1)) : (localize.deu.n.clean().substring(0,1))).toLowerCase())+"/"+
	((n.match(/.*-*|\d*/) ? (localize.deu.n.lower().space('.')) : (localize.deu.n.clean())).lower().space('.'))+"."+y+"/"+
	allOf
	{(n.match(/.*-*|\d*/) ? (localize.deu.n.lower().space('.')) : (localize.deu.n.clean()))}{y}
	{vf.match(/480[pP]|576[pP]|720[pP]|1080[pP]|2160[pP]/)}{bitdepth == 10 ? '10bit' : null}{vc}{ac}{af}{any{audioLanguages[0].ISO2}{'und'}}
	.join('.').space('.').lower().removeAll(','))
}

Now, the code is still pretty ugly and unreadable, but it's half the code, and does exactly the same. A lot better with minimal effort.



You'll want to refactor your code by understanding which bits & pieces are distinct and then write your code so that you can easily enable / disable / switch each bit & piece individually depending on the file / movie / episode at hand.

[dgnrt3]

ooooohhhhh i see...you mean something like this?

Now the only part being a necessary copy-pasta is at the the start cuz for our later workflow it's easier to have our usual folder structure duplicated under purged_with_fire/

Format: Select all

{
	(	
		( any{ genres =~ /Reality|Soap/ } || y < 2010 ) ? 'purged_with_fire/' + 
		(
			( hd == /UHD/ ? '2160p/' : '1080p/' ) + 
			( type == /Episode/ ? 'serie/' : 'film/') + 
			( anime == true ? 'anime/' : 
								( genre.replace (csv("/scripts/fb/media.list/media.genre.list.csv") ) )
						
			 )
	   
		)  
	:
		(
			( hd == /UHD/ ? '2160p/' : '1080p/' ) + 
			( type == /Episode/ ? 'serie/' : 'film/') + 
			( anime == true ? 'anime/' :
						( none {genre} == true ? 'kein_genre/' : 
							( genre.replace ( csv("/scripts/fb/media.list/media.genre.list.csv") ) )
						+"/")
			)
	    
		)
	) +"/" 
}

{
	(
			(
				( n.match (/.*-*|\d*/) ? ( n.substring(0,1) ) :
									( localize.deu.n.clean().substring(0,1) ) 
					).toLowerCase()
			)
			+"/"+
			(
				( n.match (/.*-*|\d*/) ? ( localize.deu.n.lower().space('.') ) : 
									( localize.deu.n.clean() )
					).lower().space('.')
			)	+"."+y+"/"+
			( type == /Episode/ ? ( episode.special ? 'specials' : 
						's'+s.pad(2)) : '' 
			)+"/"+
		allOf
			{
				( n.match(/.*-*|\d*/) ? ( localize.deu.n.lower().space('.') ) : 
									( localize.deu.n.clean() ) 
				).lower().space('.')
			}
			{y}
			{s00e00}
			{vf.match(/480[pP]|576[pP]|720[pP]|1080[pP]|2160[pP]/)}
			{bitdepth == 10 ? '10bit' 
							: null}
			{vc}{ac}{af}
			{any
				{audioLanguages[0].ISO2}
				{'und'}
			}
		.join('.').space('.').lower().removeAll(',')
	)
}

and yeah...agree that this is way nicer and easier to read. Thanks so much for pushing me into this direction!
Got some more best practices/something along those lines?

[rednoah]

Well, whenever you write the same code twice, then maybe there's a better way to doing things. If you write all your code as a single multi-line {code} block then all the general programming best-practices apply. Your code does not use variables. Might be useful.


e.g. use variables for each of the reoccuring components, and then put things together at the end:

Format: Select all

{
	def germanName = localize.deu.n
	def myName = germanName.clean().lower().space('.')
	def firstLetter = myName[0]

	def prefix = y < 2010 || any{ genres =~ /Reality|Soap/ } ? 'purged_with_fire' : ''
	def myGenre = anime ? 'anime' : any{ genre.replace(csv("/scripts/fb/media.list/media.genre.list.csv")) }{ 'No Genre' }
	
	return prefix + '/' + myGenre + '/' + firstLetter + '/' + myName
}

[dgnrt3]

Mhm i see. Figured out how to do it without repeating anything!

Groovy: Select all

{
	(	
		 y < 2010 || ( any{ genres =~ /Reality|Soap/ } ) ? 'purged_with_fire/'
		 : (
			( hd == /UHD/ ? '2160p/' : '1080p/' ) + 
			( type == /Episode/ ? 'serie/' : 'film/') + 
			( none {genre} == true ? 'kein_genre/' :
			( anime == true ? 'anime/' :
							( genre.replace ( csv("/scripts/fb/media.list/media.genre.list.csv") ) )
						+"/")
			)
	   
		)  
	
	) +"/" +
		(any{localize.deu.n} ? ( n.clean().lower().space('.').substring(0,1) + "/" + n.clean().lower().space('.') )  :
							 ( localize.deu.n.clean().lower().space('.').substring(0,1) + "/" + localize.deu.n.clean().lower().space('.') )
							) + "."+ y + "/" +
				( type == /Episode/ ? ( episode.special ? 'specials' : 
					's'+s.pad(2)) : '' 
				)  +"/" +
					allOf							
	    					{ any {localize.deu.n} ? ( n.clean().lower().space('.') ) :
	    										 ( localize.deu.n.clean().lower().space('.') )
	    					}
	    					{y}
						{ type == /Episode/ ? s00e00 : null }
						{vf.match(/480[pP]|576[pP]|720[pP]|1080[pP]|2160[pP]/)}
						{bitdepth == 10 ? '10bit' : null}
						{vc}{ac}{af}
						{any
							{audioLanguages[0].ISO2}
							{'und'}
						}
					.join('.').space('.').lower().removeAll(',')
}

Nice! Variables work quite similar to bash which makes that a hell lot easier. Thanks so much for ur input and have a nice weekend

[rednoah]

A good days work. Looks a lot more readable now!