FileBot

Posted: **29 Jun 2023, 04:55**

Hi all,

we have a vast media collection in which there are often times multiple resolutions and codecs available and i want filebot to only keep the one with the highest resolution and when there are several files with the same resolution it should favor HEVC files.

viewtopic.php?t=12891 according to this i came up with this:

Groovy: Select all

{ from, to ->
	def score = 
	{ f -> ['2160p':30, '1080p':20, '720p':10][f.mediaCharacteristics.videoFormat] ?: 0 }
	{ f -> ['HEVC':20, 'AVC':10][f.mediaCharacteristics.videoCodec] ?: 0 }
	score(from) > score(to) ? to : null
}

I'm used to test formats with the filebot GUI but afaik there's no easily accessible possibility to test this kind of code and I'm not sure how to link the two scores together so filebot considers them in a correct way.

Any help is highly appreciated, thanks in advance

Posted: **29 Jun 2023, 06:05**

e.g.

Groovy: Select all

{ from, to ->
	def score = { f ->
		def x = ['2160p': 30, '1080p': 20, '720p': 10][getMediaInfo(f, '{vf}')] ?: 0
		def y = ['HEVC': 2, 'AVC': 1][getMediaInfo(f, '{vcf}')] ?: 0
		return x + y
	}
	score(from) > score(to) ? to : null
}

Use Groovy Pad to prototype custom Groovy code:

Posted: **29 Jun 2023, 09:10**

Hey rednoah,

you literal god amongst filebot users. Thanks a lot for your fast reply and the input.

Cheers

Posted: **29 Jun 2023, 22:14**

FileBot r9844 now includes "HEVC over AVC" heuristics in its "better video quality" logic.

Posted: **30 Jun 2023, 05:38**

ouh very sweet...so what does the logic now all consider with which weight? I'm fine with a short link to the code in the source code

Posted: **30 Jun 2023, 06:22**

The source code will update after the next release:
https://www.filebot.net/docs/api/src-ht ... ality.html

Here's a snapshot for FileBot r9844 in the meantime:

Posted: **30 Jun 2023, 07:37**

looks like a rock solid solution! Thanks for keeping filebot alive and always improving it

Posted: **01 Jul 2023, 04:46**

One thing I noticed with the solution I'm currently using which you might want to consider for your improved quality logic.

Apart from that: why does filebot behave the way shown below?

Working env:
- Movie collection gets searched with find with a regex test to only match folders which have '.2010' and higher at the end of their name

- These folder are given to fb via -exec and are further processed by an external args file which includes -r to consider all files in the given folder

- The idea was that filebot checks all files, assigns scores on the above mentioned parameters and then only transfers the file with the highest score

Right now I'm seeing several (can't say if it's the same with every folder) cases where the following happens:

Console Output: Select all

Auto-detect movie from context [/mnt/drive1/1080p/film/allgemein/d/die.ausgrabung.2021/die.ausgrabung.2021.1080p.x265.ac3.6ch.de.mkv]                                   
[MOVE] from [/mnt/drive1/1080p/film/allgemein/d/die.ausgrabung.2021/die.ausgrabung.2021.1080p.x264.eac3.6ch.de.mkv] to [/media/1080p/film/allgemein/d/die.ausgrabung.2021/die.ausgrabung.2021.mkv]                            
[CLOSURE] Delete [/media/1080p/film/allgemein/d/die.ausgrabung.2021/die.ausgrabung.2021.mkv]
[MOVE] from [/mnt/drive1/1080p/film/allgemein/d/die.ausgrabung.2021/die.ausgrabung.2021.1080p.x265.ac3.6ch.de.mkv] to [/media/1080p/film/allgemein/d/die.ausgrabung.2021/die.ausgrabung.2021.mkv]
Processed 2 files

Cheers

Posted: **01 Jul 2023, 09:27**

I've fixed the log above. You'll want to read logs in plain/text. If you're using some kind of web-based management software, then you'll want to make sure that it doesn't remove newlines in the console output text area, as that will make things hard to read.

Here's what happens: You have A and B, and both will be moved to X, first A is moved successfully without conflict, then B is to be moved to X as well, and now we have a conflict. Your custom conflict code decides that B is better, so the existing X (i.e. original A) is deleted to resolve the conflict, and then B is moved to X without conflict.

Console Output: Select all

[MOVE] from [A] to [X]                            
[CLOSURE] Delete [X]
[MOVE] from [B] to [X]

Because, according to the hints in the file name:

Properties: Select all

# ['HEVC': 2, 'AVC': 1]
A ➔ x264.eac3 ➔ AVC ➔ 1
B ➔ x265.ac3 ➔ HEVC ➔ 2

** you maybe confused because one file has better video / worse audio, while the other has worse video / better audio, but video is the deciding factor, because our code does not take audio into account

Please add lots of debug logging to your custom conflict code, so that we can see what it is doing, and what numbers it is working with for each file at hand:

rednoah wrote: ↑29 Jun 2023, 06:05 Use Groovy Pad to prototype custom Groovy code:

dgnrt3 wrote: ↑01 Jul 2023, 04:46 - The idea was that filebot checks all files, assigns scores on the above mentioned parameters and then only transfers the file with the highest score

Interesting use case. I don't recall this use case being mentioned before. Unfortunately, the --conflict option cannot be used to preemptively resolve predicted conflicts, since conflicts are typically with previously processed files. If the target file system is remote, then I would recommend processing files within the same file system where move operations are instant, and then rsync the final file structure to the remote file system at the end.

Posted: **02 Jul 2023, 10:22**

I've fixed the log above. You'll want to read logs in plain/text. If you're using some kind of web-based management software, then you'll want to make sure that it doesn't remove newlines in the console output text area, as that will make things hard to read.

I'm sorry, was laying on the couch and just checked the logs via termux on my phone

Usually I'm only working with linux shell commands on my desktop which was turned off at that moment

Because, according to the hints in the file name:

Oh ok, I was under the impression filebot would use mediainfo to get the videoFormat from the actual files as the code uses

Code: Select all

[getMediaInfo(f, '{vf}')]

and not match the filename against a regex.
It's interesting to see that the code doesn't check for conflicts between A and B and writes A to X (A having a lower score, first, then estimates the score of B and overwrites X) would have thought only B with the highest scrore getting written in the first place.

Interesting use case. I don't recall this use case being mentioned before. Unfortunately, the --conflict option cannot be used to preemptively resolve predicted conflicts, since conflicts are typically with previously processed files. If the target file system is remote, then I would recommend processing files within the same file system where move operations are instant, and then rsync the final file structure to the remote file system at the end.

That's the reason I created this post in the first place as I couldn't find it anywhere else and wouldn't want to bother you or anyone else with something that has been documented before somehwere.
You are correct on the remote filesystem part (we are moving away from it to a local FS). I thought of your solution with moving/renaming on the remote FS before but the real pain in the ass is the fact that this remote file system also has several API constraints set on it so any call to it would count against those limits and makes any I/O on it painfully slow and I have to process somewhat around 400TB of files so I chose the route I'm heading now.

If you're interested in exploring this use case further I'm happy to provide you with any debugging data you want me to provide.

Posted: **02 Jul 2023, 10:53**

1.
{vf} is indeed based on the file content on not the file name. Since you didn't post the MediaInfo table for each file, I had to guess the value of {vf} based on the file path that you did share with us, just to illustrate what is probably happening with some example numbers.

2.
The current conflict action API is unfortunately ill-suited to compare arbitrary files. A conflict script assumes that the second parameter is the target file, and may choose to rename the current X to X.backup to resolve the conflict, before confirming X as target file path, or choose to confirm a different target file path, so things may go very badly if the "to" parameter is suddenly unexpectedly one of the input files in some use cases (i.e. process both versions at once) but not others (i.e. process one file at a time).

I see no issue with hardlinking files into a temporary staging location (instant, no extra disk space required) to resolve conflicts locally, and then moving files ot the the remote location, ideally with rsync.

EDIT:

If you could find and sort files ahead of time, i.e. custom program that collects files, and then groups / sorts them according to your custom quality metric, and then passes them on to the filebot call then - assuming that FileBot processes files in order of input arguments - do that with --conflict skip since the first occurance in the process is going to be the better quality one, and the second lower quality one will run into a conflict and get ignored.

We can mostly reuse the code above for that and then | xargs the lines into a new filebot call:

Groovy: Select all

#!/usr/bin/env filebot -script
args.files.findAll{ it.video }.toSorted{ from, to ->
	def score = { f ->
		def x = ['2160p': 30, '1080p': 20, '720p': 10][getMediaInfo(f, '{vf}')] ?: 0
		def y = ['HEVC': 2, 'AVC': 1][getMediaInfo(f, '{vcf}')] ?: 0
		return x + y
	}
	return score(to) - score(from)
}.each{ println it }

Console Output: Select all

$ ls
[email protected]
[email protected]
MPEG-4 [email protected]
$ find-and-sort.groovy . --log OFF | xargs -I {} filebot -rename --db file --format "{today}" -- {}
Rename files using [Plain File]
[MOVE] from [[email protected]] to [2023-07-02.mkv]
Processed 1 file
Rename files using [Plain File]
[SKIP] Skipped [[email protected]] because [2023-07-02.mkv] already exists
Processed 0 files
Rename files using [Plain File]
[SKIP] Skipped [MPEG-4 [email protected]] because [2023-07-02.mkv] already exists
Processed 0 files

tl;dr No custom conflict action necessary. The default --conflict skip option works just fine. You just need to pass in the input files in your preferred order.

EDIT 2:

xargs is finicky AF. \n isn't the default separator. macOS xargs doesn't have the -d option. -I enables \n as separator but implicitly disables -n resulting in one filebot command per one file. I'm always surprised how ill-suited xargs can be for the most most typical xargs use cases one can think of, just run a command for every line, how hard can it be...

Maybe better to avoid xargs, and instead write arguments in order to a text file, and then have filebot read arguments from that text file in order:

Console Output: Select all

$ find-and-sort.groovy . --log OFF > input.txt
$ cat input.txt
[email protected]
[email protected]
MPEG-4 [email protected]
$ filebot -rename --db file --format "{today}" -- @input.txt
Rename files using [Plain File]
[MOVE] from [[email protected]] to [2023-07-02.mkv]
[SKIP] Skipped [[email protected]] because [2023-07-02.mkv] already exists
[SKIP] Skipped [MPEG-4 [email protected]] because [2023-07-02.mkv] already exists
Processed 1 file

EDIT 3:

FileBot r9859 adds the --file-order option to re-order the input argument array before processing:
viewtopic.php?t=13796

Posted: **05 Jul 2023, 20:02**

rednoah wrote: ↑02 Jul 2023, 10:53 1.
{vf} is indeed based on the file content on not the file name. Since you didn't post the MediaInfo table for each file, I had to guess the value of {vf} based on the file path that you did share with us, just to illustrate what is probably happening with some example numbers.

Can't provide them now anymore as files got deleted. Will post in the future. Sry, my bad.

2.
The current conflict action API is unfortunately ill-suited to compare arbitrary files. A conflict script assumes that the second parameter is the target file, and may choose to rename the current X to X.backup to resolve the conflict, before confirming X as target file path, or choose to confirm a different target file path, so things may go very badly if the "to" parameter is suddenly unexpectedly one of the input files in some use cases (i.e. process both versions at once) but not others (i.e. process one file at a time).

I see no issue with hardlinking files into a temporary staging location (instant, no extra disk space required) to resolve conflicts locally, and then moving files ot the the remote location, ideally with rsync.

EDIT:

If you could find and sort files ahead of time, i.e. custom program that collects files, and then groups / sorts them according to your custom quality metric, and then passes them on to the filebot call then - assuming that FileBot processes files in order of input arguments - do that with --conflict skip since the first occurance in the process is going to be the better quality one, and the second lower quality one will run into a conflict and get ignored.

Thats quite difficult and time consuming for my use case/environment as there are the beforementioned API limits in place.

EDIT 3:

FileBot r9859 adds the --file-order option to re-order the input argument array before processing:
viewtopic.php?t=13796

that's awesome stuff, mate! Is there ANY way you'd accept a donation or some other way of appreciation to thank you? 'Cause that was way above some "normal" support you've shown here (and on MANY, MANY other occasions on this forum! <3 <3

Posted: **06 Jul 2023, 08:38**

dgnrt3 wrote: ↑05 Jul 2023, 20:02 Can't provide them now anymore as files got deleted. Will post in the future. Sry, my bad.

No worries. I could reasonably guess from the filename in this case, so there was no need need for an additional question / answer round trip.

dgnrt3 wrote: ↑05 Jul 2023, 20:02 that's awesome stuff, mate! Is there ANY way you'd accept a donation or some other way of appreciation to thank you? 'Cause that was way above some "normal" support you've shown here (and on MANY, MANY other occasions on this forum! <3 <3

--file-order much like --file-filter was already on the radar, just looking for an actual use case, so cheers for finding a good one. But you can always purchase an extra license and then gift it to a friend.

Posted: **06 Jul 2023, 17:51**

FileBot

Need little help with conflict groovy script

Need little help with conflict groovy script

Re: Need little help with conflict groovy script

Re: Need little help with conflict groovy script

Re: Need little help with conflict groovy script

Re: Need little help with conflict groovy script

Re: Need little help with conflict groovy script

Re: Need little help with conflict groovy script

Re: Need little help with conflict groovy script

Re: Need little help with conflict groovy script

Re: Need little help with conflict groovy script

Re: Need little help with conflict groovy script

Re: Need little help with conflict groovy script

Re: Need little help with conflict groovy script

Re: Need little help with conflict groovy script