Skipping/moving files with damaged/defective media stream + Timestamps in logfile when not using a script

[dgnrt3]

Hi all, hi rednoah,

I'm currently in the process of reorganising a vast media collection and I'm letting filebot go over our series collection which is somewhat around 25TB (~25k files) and some of the files seem to have a damaged media stream which naturally causes filebot to throw an exception as binding "media" yields an empty value which in turn causes the running filebot command to fail.

As I let filebot work over the whole series folder at once one can imagine that this is a quite time-consuming task and it would be awesome if filebot would skip the damaged files and just log the damaged file (maybe even move the damaged files into a separate "damaged" folder or something along those lines) but keep on going.

Additionally I'd really love timestamps in the logfile as it would make looking through them quite a bit easier for me.
I tested if

Shell: Select all

log=info

would maybe print timestamps and upon searching in the forums I saw that there's

Groovy: Select all

--def net.filebot.logging.time=true

to print timestamps in the logfile but I'm not working with a filebot script and only use text files to pass along command arguments and format options but couldn't find a solution to pass

Groovy: Select all

net.filebot.logging.time=true

to filebot.

Please point me into the correct direction once again!

Thanks a lot and cheers

[rednoah]

Can you provide a sample command with console output? Please read How to Request Help.



Note that it is normal for bindings to fail and yield nothing, so what you're asking for is already default behavior.

e.g. format yields a value even though some bindings fail:

Console Output: Select all

$ touch test.mkv
$ filebot -mediainfo *.mkv --format "{f.name} | {vc} | {ac}"
test.mkv |  |

e.g. format yielding no value for one file or another is not a problem either, the process will continue:

Console Output: Select all

$ touch a.mkv b.mkv c.mkv
$ filebot -mediainfo *.mkv --format "{vc}"
Expression yields empty value: Binding "media": Failed to read media info: Invalid media file: a.mkv
Expression yields empty value: Binding "media": Failed to read media info: Invalid media file: b.mkv
Expression yields empty value: Binding "media": Failed to read media info: Invalid media file: c.mkv

You can of course find files based on the existence or non-existence of a given binding via the filebot -find command:

e.g. find video files where {vc} is undefined:

Shell: Select all

filebot -find . --filter "f.video && none{ vc }"

e.g. find video files where {vc} is undefined and then move them somewhere:

Shell: Select all

filebot -find . --filter "f.video && none{ vc }" -exec mv -v {f} "/path/to/Trash"

You can permanently enable timestamps via the configuration file. You can write the configuration file yourself, or have the properties script do it for you:

Print timestamp for each log record: net.filebot.logging.time = true | false

Shell: Select all

filebot -script fn:properties --def net.filebot.logging.time=true

e.g.

Console Output: Select all

$ filebot -script fn:properties --def net.filebot.logging.time=true
Update user-defined System Properties
* Set net.filebot.logging.time = true
Store user-defined System Properties
* Write ~/.filebot/system.properties
#FileBot System Properties
#Sat Jul 29 12:30:00 UTC 2023
net.filebot.logging.time=true

[dgnrt3]

Mhm haven't seen the behavior you described yet. Mhm...haven't seen the behavior you described in the time I've been using filebot now but can't recall an option where I could've changed this setting.
Sorry for my very low quality post! Here you go:

Command:

Shell: Select all

filebot /mnt/drive3/1080p/serie/allgemein/b/ @/scripts/fb/args/watch.reorg.drv3.serie.tmdbtv.args.txt

Content args file:

Groovy: Select all

-r
-rename
-non-strict
--conflict
/scripts/fb/conflicts/res-x265-x264.groovy
--db
TheMovieDB::TV
--lang
German
--output
/
--format
/scripts/fb/formats/watch.reorg.serie.drv3.format.groovy
--log-file
/var/log/fb/reorg.drv3.serie.log
-no-xattr

Logfile:

Console Output: Select all

Invalid media file: /media/1080p/serie/allgemein/b/bosch.2015/s03/bosch.2015.s03e01.mkv
net.filebot.format.SuppressedThrowables: Expression yields empty value: Binding "media": Failed to read media info: Invalid media file: /media/1080p/serie/allgemein/b/bosch.2015/s03/bosch.2015.s03e01.mkv
at net.filebot.format.ExpressionFormat.format(Unknown Source)
at net.filebot.format.ExpressionFormat.format(Unknown Source)
at net.filebot.cli.CmdlineOperations.formatMatch(Unknown Source)
at net.filebot.cli.CmdlineOperations.getRenameMap(Unknown Source)
at net.filebot.cli.CmdlineOperations.renameSeries(Unknown Source)
at net.filebot.cli.CmdlineOperations.rename(Unknown Source)
at net.filebot.cli.ArgumentProcessor.runCommand(Unknown Source)
at net.filebot.cli.ArgumentProcessor.run(Unknown Source)
at net.filebot.Main.main(Unknown Source)
Caused by: net.filebot.format.BindingException: Binding "media": Failed to read media info: Invalid media file: /media/1080p/serie/allgemein/b/bosch.2015/s03/bosch.2015.s03e01.mkv
at net.filebot.format.MediaBindingBean.getMediaInfo(Unknown Source)
at net.filebot.format.MediaBindingBean.lambda$getVideoResolution$11(Unknown Source)
at net.filebot.format.MediaBindingBean.getVideoResolution(Unknown Source)
at net.filebot.format.MediaBindingBean.getVideoDefinitionCategory(Unknown Source)
at net.filebot.format.ExpressionBindings$MethodBinding.call(Unknown Source)
at net.filebot.format.ExpressionBindings.get(Unknown Source)
at net.filebot.format.ExpressionFileFormat$1.get(Unknown Source)
at __script_b4c78d10ca1eff8f39c52359691d7758_11b9.run(__script_b4c78d10ca1eff8f39c52359691d7758_11b9:6)
at net.filebot.format.ExpressionEngine.evaluateScriptlet(Unknown Source)
... 9 more
Caused by: net.filebot.mediainfo.MediaInfoException: Invalid media file: /media/1080p/serie/allgemein/b/bosch.2015/s03/bosch.2015.s03e01.mkv
at net.filebot.format.MediaBindingBean.lambda$new$42(Unknown Source)
at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2406)
at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2404)
at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2387)
at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:108)
at com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:62)
at net.filebot.MemoryCache.get(Unknown Source)
at com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:62)
at net.filebot.MemoryCache.get(Unknown Source)
at net.filebot.format.MediaBindingBean.lambda$new$43(Unknown Source)
at net.filebot.Resource$Memoized.get(Unknown Source)
... 18 more
Caused by: java.lang.IllegalArgumentException: Invalid media file: /media/1080p/serie/allgemein/b/bosch.2015/s03/bosch.2015.s03e01.mkv
at net.filebot.mediainfo.MediaInfo.open(Unknown Source)
... 27 more

Error (o_O)

Here is the format file (most probably unnecessary complicated/complex in some places and as the name suggests this is a format to reorg and has been stitched together from pre-existing formats ...has been grown historically as one would say lol ).

Groovy: Select all

{
(none{genres} || genres.contains('Reality') || genres.contains('Soap') || y < 2010 ) ? 
'mnt/drive3/purge_with_fire/' + 
( hd == /UHD/ ? 
( '2160p/' + ( type == /Episode/ ? 
		(
			'serie/'+
			(none{genre} == true ? 'kein_genre' : (genre.replace(csv("/scripts/fb/media.list/media.genre.list.csv"))))+"/"+
			((n.match(/.*-*|\d*/) ? (n.substring(0,1)) : (localize.deu.n.substring(0,1))).toLowerCase())+"/"+
			((n.match(/.*-*|\d*/) ? (localize.deu.n.lower().space('.')) : (localize.deu.n.clean())).lower().space('.'))+"."+y+"/"+
			(episode.special ? 'specials' : 's'+s.pad(2))+"/"+
			allOf
			{(n.match(/.*-*|\d*/) ? (localize.deu.n.lower().space('.')) : (localize.deu.n.clean())).lower().space('.')}{y}{s00e00}.join('.').space('.').lower().removeAll(',')
		)
	: 
		(
			'film/'+
			(none{genre} == true ? 'kein_genre' : (genre.replace(csv("/scripts/fb/media.list/media.genre.list.csv"))))+"/"+
			((n.match(/.*-*|\d*/) ? (n.substring(0,1)) : (localize.deu.n.substring(0,1))).toLowerCase())+"/"+
			((n.match(/.*-*|\d*/) ? (localize.deu.n.lower().space('.')) : (localize.deu.n.clean())).lower().space('.'))+"."+y+"/"+
			allOf
			{(n.match(/.*-*|\d*/) ? (localize.deu.n.lower().space('.')) : (localize.deu.n.clean())).lower().space('.')}{y}.join('.').space('.').lower().removeAll(',')
		)
	)
)
: 
( '1080p/' + ( type == /Episode/ ? 
		(
			'serie/'+
			(none{genre} == true ? 'kein_genre' : (genre.replace(csv("/scripts/fb/media.list/media.genre.list.csv"))))+"/"+
			((n.match(/.*-*|\d*/) ? (n.substring(0,1)) : (localize.deu.n.substring(0,1))).toLowerCase())+"/"+
			((n.match(/.*-*|\d*/) ? (localize.deu.n.lower().space('.')) : (localize.deu.n.clean())).lower().space('.'))+"."+y+"/"+
			(episode.special ? 'specials' : 's'+s.pad(2))+"/"+
			allOf
			{(n.match(/.*-*|\d*/) ? (localize.deu.n.lower().space('.')) : (localize.deu.n.clean())).lower().space('.')}{y}{s00e00}.join('.').space('.').lower().removeAll(',')
		)
: 
		(
			'film/'+
			(none{genre} == true ? 'kein_genre' : (genre.replace(csv("/scripts/fb/media.list/media.genre.list.csv"))))+"/"+
			((n.match(/.*-*|\d*/) ? (n.substring(0,1)) : (localize.deu.n.substring(0,1))).toLowerCase())+"/"+
			((n.match(/.*-*|\d*/) ? (localize.deu.n.lower().space('.')) : (localize.deu.n.clean())).lower().space('.'))+"."+y+"/"+
			allOf
			{(n.match(/.*-*|\d*/) ? (localize.deu.n.lower().space('.')) : (localize.deu.n.clean())).lower().space('.')}{y}.join('.').space('.').lower().removeAll(',')
		)
	)
)
)
: 
(
(
( type == /Episode/ ) 
? 
'mnt/drive3/' + 
( hd == /UHD/
? 
( '2160p/serie/' + ( anime == true ? 
	('anime/'+
		((n.match(/.*-*|\d*/) ? (n.substring(0,1)) : (localize.deu.n.clean().substring(0,1))).toLowerCase())+"/"+
		((n.match(/.*-*|\d*/) ? (localize.deu.n.lower().space('.')) : (localize.deu.n.clean())).lower().space('.'))+"."+y+"/"+
		(episode.special ? 'specials' : 's'+s.pad(2))+"/"+
		allOf
		{(n.match(/.*-*|\d*/) ? (localize.deu.n.lower().space('.')) : (localize.deu.n.clean())).lower().space('.')}{y}{s00e00}.join('.').space('.').lower().removeAll(',')
	)
: 
	(
		(none{genre} == true ? 'kein_genre' : (genre.replace(csv("/scripts/fb/media.list/media.genre.list.csv"))))+"/"+
		((n.match(/.*-*|\d*/) ? (n.substring(0,1)) : (localize.deu.n.clean().substring(0,1))).toLowerCase())+"/"+
		((n.match(/.*-*|\d*/) ? (localize.deu.n.lower().space('.')) : (localize.deu.n.clean())).lower().space('.'))+"."+y+"/"+
		(episode.special ? 'specials' : 's'+s.pad(2))+"/"+
		allOf
		{(n.match(/.*-*|\d*/) ? (localize.deu.n.lower().space('.')) : (localize.deu.n.clean())).lower().space('.')}{y}{s00e00}.join('.').space('.').lower().removeAll(',')
	)
)
)
: 
( '1080p/serie/' + ( anime == true ? 
	('anime/'+
		((n.match(/.*-*|\d*/) ? (n.substring(0,1)) : (localize.deu.n.clean().substring(0,1))).toLowerCase())+"/"+
		((n.match(/.*-*|\d*/) ? (localize.deu.n.lower().space('.')) : (localize.deu.n.clean())).lower().space('.'))+"."+y+"/"+
		(episode.special ? 'specials' : 's'+s.pad(2))+"/"+
		allOf
		{(n.match(/.*-*|\d*/) ? (localize.deu.n.lower().space('.')) : (localize.deu.n.clean())).lower().space('.')}{y}{s00e00}.join('.').space('.').lower().removeAll(',')
	)
: 
	(
		(none{genre} == true ? 'kein_genre' : (genre.replace(csv("/scripts/fb/media.list/media.genre.list.csv"))))+"/"+
		((n.match(/.*-*|\d*/) ? (n.substring(0,1)) : (localize.deu.n.clean().substring(0,1))).toLowerCase())+"/"+
		((n.match(/.*-*|\d*/) ? (localize.deu.n.lower().space('.')) : (localize.deu.n.clean())).lower().space('.'))+"."+y+"/"+
		(episode.special ? 'specials' : 's'+s.pad(2))+"/"+
		allOf
		{(n.match(/.*-*|\d*/) ? (localize.deu.n.lower().space('.')) : (localize.deu.n.clean())).lower().space('.')}{y}{s00e00}.join('.').space('.').lower().removeAll(',')
	)
)))
: 
(( 'mnt/drive3/' + 
( hd == /UHD/
? 
'fb_match_falsch/' +
( '2160p/film/' + ( anime == true ? 
	(
		'anime/'+
		((n.match(/.*-*|\d*/) ? (n.substring(0,1)) : (localize.deu.n.clean().substring(0,1))).toLowerCase())+"/"+
		((n.match(/.*-*|\d*/) ? (localize.deu.n.lower().space('.')) : (localize.deu.n.clean())).lower().space('.'))+"."+y+"/"+
		allOf
		{(n.match(/.*-*|\d*/) ? (localize.deu.n.lower().space('.')) : (localize.deu.n.clean())).lower().space('.')}{y}.join('.').space('.').lower().removeAll(',')
	)
: 
	(
		(none{genre} == true ? 'kein_genre' : (genre.replace(csv("/scripts/fb/media.list/media.genre.list.csv"))))+"/"+
		((n.match(/.*-*|\d*/) ? (n.substring(0,1)) : (localize.deu.n.clean().substring(0,1))).toLowerCase())+"/"+
		((n.match(/.*-*|\d*/) ? (localize.deu.n.lower().space('.')) : (localize.deu.n.clean())).lower().space('.'))+"."+y+"/"+
		allOf
		{(n.match(/.*-*|\d*/) ? (localize.deu.n.lower().space('.')) : (localize.deu.n.clean())).lower().space('.')}{y}.join('.').space('.').lower().removeAll(',')
	)
))
: 
( '1080p/film/' + ( anime == true ? 
	(
		'anime/'+
		((n.match(/.*-*|\d*/) ? (n.substring(0,1)) : (localize.deu.n.clean().substring(0,1))).toLowerCase())+"/"+
		((n.match(/.*-*|\d*/) ? (localize.deu.n.lower().space('.')) : (localize.deu.n.clean())).lower().space('.'))+"."+y+"/"+
		allOf
		{(n.match(/.*-*|\d*/) ? (localize.deu.n.lower().space('.')) : (localize.deu.n.clean())).lower().space('.')}{y}.join('.').space('.').lower().removeAll(',')
	)
: 
	(
		(none{genre} == true ? 'kein_genre' : (genre.replace(csv("/scripts/fb/media.list/media.genre.list.csv"))))+"/"+
		((n.match(/.*-*|\d*/) ? (n.substring(0,1)) : (localize.deu.n.clean().substring(0,1))).toLowerCase())+"/"+
		((n.match(/.*-*|\d*/) ? (localize.deu.n.lower().space('.')) : (localize.deu.n.clean())).lower().space('.'))+"."+y+"/"+
		allOf
		{(n.match(/.*-*|\d*/) ? (localize.deu.n.lower().space('.')) : (localize.deu.n.clean())).lower().space('.')}{y}.join('.').space('.').lower().removeAll(',')
	)
)))))))
}

[rednoah]

If your format yields nothing, then filebot cannot rename the file to nothing, and fails. You can make filebot fail or not fail by making your format yield nothing or something.


e.g. format that yields nothing if {vc} is undefined:

Console Output: Select all

$ filebot -rename *.mkv --db file --format "{vc}"
Rename files using [Plain File]
Invalid media file: test.mkv
net.filebot.format.SuppressedThrowables: Expression yields empty value: Binding "media": Failed to read media info: Invalid media file: ~/test.mkv

e.g. format that always yields something, because we add something that cannot fail:

Console Output: Select all

$ filebot -rename *.mkv --db file --format "{fn} {vc}"
Rename files using [Plain File]
[MOVE] Skipped [test.mkv] because [test.mkv] already exists

e.g. format that always yields something, because we catch errors and provide a default value instead:

Console Output: Select all

$ filebot -rename *.mkv --db file --format "{ any{vc}{/NO CODEC/} }"
Rename files using [Plain File]
[MOVE] from [test.mkv] to [NO CODEC.mkv]

[dgnrt3]

Well..now you're explaining it, it makes a hell lot of sense....
Me be like right now:


Thanks so much for explaining and supporting again!

...time to update my formats!

[rednoah]

EDIT: Adding the solution at the end.


You can of course find / include / exclude files based on the existence or non-existence of a given binding via the filebot -find command or the --file-filter option:


e.g. find video files where {vc} is undefined:

Shell: Select all

filebot -find . --filter "f.video && none{ vc }"

e.g. find video files where {vc} is undefined and then move them somewhere:

Shell: Select all

filebot -find . --filter "f.video && none{ vc }" -exec mv -v {f} "/path/to/Trash"

e.g. exclude files from processing if a given binding is undefined:

Shell: Select all

filebot -rename /input --file-filter "vc"

Console Output: Select all

$ filebot -rename *.mkv --file-filter "vc"
test.mkv: Binding "media": Failed to read media info: Invalid media file: test.mkv
No input files: [test.mkv]

NOTE: --file-filter works on input arguments before processing, so filebot will appear to do nothing for some time if you are passing in a large number of files that need to be checked before processing can start.