AV1 codec support for duplicates script

[WhyDoINeedAUsername]

Hi,
Firstly thank you for creating Filebot - I've been using it for years to remove duplicates etc (via https://github.com/filebot/scripts/blob ... tes.groovy).

However I'm now converting my entire library to AV1 to free up a lot of disk space and improve streaming, but Filebot doesn't seem to know that AV1 is better than x264, HEVC, etc and is effectively discarding the better files.

When running duplicate detection this is what is happening:
[*] SomeVideo
Unknown Video Codec: AV1
[+] 1. M:\SomeVideo - [2160p x264 8bit AC3 2.0 21 Mbps].mkv
[-] 2. M:\SomeVideo - [2160p AV1 10bit AC3 2.0 4.6 Mbps].mkv

For reference this was the same video transcoded from x264 (15.3 GB) to AV1 (3.3 GB) with a VMAF score of 95% (https://github.com/Netflix/vmaf) - visually the files basically look identical, which is amazing.

I assume Filebot will do the same with all AV1 files because it hasn't been coded to explicitly know about AV1 yet.?
So I'm assuming it'll discard all AV1 files, which means I won't be able to use it anymore

I hope this is an easy fix because I'd love to continue to use Filebot.

I'd also love to know what the bitrate cutoffs are between codecs - i.e. at what % bitrate is x265 seen as better than x264 etc.

[rednoah]

So AV1 @ 4.6 Mbps should be considered "better" than x264 @ 21 Mbps then? I guess we could do that, but a fair number of users might reasonably say that the 20 GB files is better than the 5 GB file, no? On a related note, how does AV1 compare to HEVC? Better? Same? Worse?


Can you paste the MediaInfo table for the AV1 file?


Unknown Video Codec: AV1 means "neither better nor worse" so other things like bitrate and file size will probably be playing a deciding role. The implementation details are fairly simple. There are no bitrate cutoffs are between codecs.


The duplicates script does actually allow to pass in your own code to decide the order of things. Here's a simple example that compares by file size where larger is better:

Shell: Select all

--order '{ a, b -> b.length() <=> a.length() }'

** you will find comparing non-numeric values a bit tricky though; that'll require a lot more code;

[WhyDoINeedAUsername]

Thanks for the quick response.

I think the problem is around line 100.
It looks like there is a missing switch statement resulting in (Unknown Video Codec: AV1):
// print missing codec name
110 throw new IllegalArgumentException("Unknown Video Codec: " + vc);

[rednoah]

Yes. A fix would be easy. But AV1 hasn't come up as an issue before, so I would need a bit of convincing to unilaterally make a change based on a single persons request. If FileBot were to unexpectedly delete a 20 GB HEVC file in favour of a 5 GB AV1 file, that could be a serious problem for some users.


Can you paste the MediaInfo table for the AV1 file?


A custom --order function for "AV1 always wins" might make more sense for your specific use case, since that is what you're trying to express it seems. That'd be 1 line of easy code, and you wouldn't have to wait for FileBot updates.

[WhyDoINeedAUsername]

Yeah I fully agree with you, I wouldn't want my much better quality stuff getting deleted.

I'm still very new to AV1, but super impressed so far!

I'd need to do a lot more testing, so currently I don't know what the ideal cutoff point would be yet, but at the same bitrates I think AV1 will definitely beat everything else (except x266 perhaps).

Each file compresses differently, with some having amazing differences (like the 15GB -> 3GB example, for a B&W video),
but in general it seems to me that there's about a 30-50% improvement over HEVC, and even more over x264.

But at a minimum AV1 > HEVC > x264 at the same bitrate anyway.

[rednoah]

Can you paste the MediaInfo table for the AV1 file please?

[WhyDoINeedAUsername]

I'm not sure if I've done it correctly - I used the command line to create it

I've done it for two videos (2nd was converted from about 15GB to 7.4GB)
Sorry I've already deleted the originals, but can wait for another video too.

[WhyDoINeedAUsername]

MediaInfos:
https://pastebin.com/0b5wcSa0
https://pastebin.com/NH9re3gH


Here's some more data
---
Source [7.8GB HEVC] -> [2.4GB AV1 @ 95% of source quality]
https://pastebin.com/nRfYRryU -> https://pastebin.com/m0V2nZsu

----
Source [7.95 AVC/x264] -> [3GB AV1 @ 95% of source quality]
https://pastebin.com/BVqC41uF -> https://pastebin.com/LKVgbGwy

I've also found that some files don't get smaller when being re-encoded, especially videos with a lot of film grain.
I'm also converting from previously compressed files, so I'm not sure what the numbers would look like when encoding from a completely uncompressed source to each of the different codecs - that's an experiment for someone else perhaps. These results are good enough for me because I can't see a visual difference between them.

[rednoah]

Thanks for the MediaInfo table. This should work:

Groovy: Select all

{
	def order = ['AV1': 100, 'HEVC': 10, 'AVC': 1].withDefault{ 0 }
	order[b] <=> order[a]
}

You can add that to your command like so:

Shell: Select all

--order "{
	def order = ['AV1': 100, 'HEVC': 10, 'AVC': 1].withDefault{ 0 }
	order[b] <=> order[a]
}"

[WhyDoINeedAUsername]

Thanks

[WhyDoINeedAUsername]

Hi Noah,
I just came across this related error during duplicate detection:
Unknown Video Codec: xvid

Though not super important anymore, it can probably be fixed at the same time as the AV1 issue.

[rednoah]

I just came across this related error during duplicate detection:
Unknown Video Codec: xvid

Yep, we'll want to add that. Can you paste the MediaInfo table for the xvid file?

[WhyDoINeedAUsername]

Sorry the files have been deleted already.
I just noticed the message afterwards.

[rednoah]

No worries. Maybe next time.