External .xattr and CRC32

[cheaters]

Hi Red
I have my .xattr data set to be saved externally as sidecar files.
I am also using crc32 in my filename format.
My format also copies files from Seed Drive to Plex Drive when renaming... (A to B Drives)

When renaming using a preset in the GUI or a shell script (and copying) the files the CRC32 is calculated for the filename and two .xattr sidecar files are saved in the downloaded torrent's directory on Seed Drive (A):
complete/The.Expanse.S04.1080p.BluRay/.xattr/The.Expanse.S04E01.1080p.BluRay.mp4/CRC32
complete/The.Expanse.S04.1080p.BluRay/.xattr/The.Expanse.S04E01.1080p.BluRay.mp4/CRC32.mtime

When filebot is finished renaming and copying the movie and subtitle files to (B) Drive, those two files are not copied or moved over. They remain in the original torrent's directory.

What are my options? Calcuating CRC32 is time consuming, I would rather not have to do it twice for the same files on every copy. And there is the housekeeping issue of maintaining the original torrent file directory as downloaded without the added .xattr directories. Can you add a setting or option to copy them over with the files?

[rednoah]

Do you need CRC32 metadata for any particular reason if the file name itself already contains the same information? If you must have .xattr/<name>/CRC32 files then you could probably auto-generate them with a custom script fairly instantly.


If you use {crc32} then FileBot will automatically store computed values to xattr metadata. That said, if the CRC32 is already part of the file name, then FileBot will just take that and not compute it. So having CRC32 xattr metadata as well adds nothing for the purposes of FileBot at the very least.



EDIT:

What is your use case? GUI or CLI? I would consider adding a new --apply crc32 post-processing feature to either compute or copy CRC32 checksum from the source file path and / or create / update *.sfv files in the target folder.

[cheaters]

Do you need CRC32 metadata for any particular reason if the file name itself already contains the same information? If you must have .xattr/<name>/CRC32 files then you could probably auto-generate them with a custom script fairly instantly.

At some point in the past it was suggested to make sure there were no collisions with filenames. I use the CRC32 in the filename in combination with --conflict index in my script. Files always have an original name and always get moved over.

If you use {crc32} then FileBot will automatically store computed values to xattr metadata. That said, if the CRC32 is already part of the file name, then FileBot will just take that and not compute it. So having CRC32 xattr metadata as well adds nothing for the purposes of FileBot at the very least.

How should the filename be formatted for FileBot to recognize the CRC32? As it stand now, since I use Plex, I have it in square braces at the end of my format - just the number.

EDIT:

What is your use case? GUI or CLI? I would consider adding a new --apply crc32 post-processing feature to either compute or copy CRC32 checksum from the source file path and / or create / update *.sfv files in the target folder.

I use a script that runs upon download. If that fails to recognize a file properly, usually due to badly named files (wrong year) - or series issues (production vs airdate for season/episode), then I move to the GUI but in both cases I use the same format.

If you could add that feature it would be brilliant.

Another issue with keeping sidecar .xattr files is that when files are named incorrectly and then renamed their old sidecar data is not removed from the .xattr folder. This can be a manual housekeeping problem and I have seen conflicts with renaming files when there are too many iterations in that .xattr folder.

[rednoah]

At some point in the past it was suggested to make sure there were no collisions with filenames. I use the CRC32 in the filename in combination with --conflict index in my script. Files always have an original name and always get moved over.

So you do not actually need CRC32 xattr metadata? You merely need CRC32 in the file name to generate unique file paths.

--conflict index has no effect in this case, because adding {crc32} to the target file path guarantees a unique file path per unique file, so you won't ever have a file path conflict. --conflict index is useful resolving file path conflicts by adding .1 .2 .3 etc as needed to find a unique file path for each unique file.

How should the filename be formatted for FileBot to recognize the CRC32? As it stand now, since I use Plex, I have it in square braces at the end of my format - just the number.

[12345678] at the end of the file name will work best.

e.g.

Code: Select all

$ filebot -mediainfo *.mkv --format "{fn} | {bytes} | {crc32}"
Test File [12345678] | 0 bytes | 12345678

** a 0 byte file always has CRC32 checksum 00000000 so this example shows that FileBot is simply copying the CRC32 from the file name

I use a script that runs upon download. If that fails to recognize a file properly, usually due to badly named files (wrong year) - or series issues (production vs airdate for season/episode), then I move to the GUI but in both cases I use the same format.

So you do not actually need CRC32 xattr metadata on the target file path?

I do not see a use case where .xattr/<name>/CRC32 has an effect on reprocessing:
(1) if you reprocess the incorrectly named target path, then CRC32 is already in the file name
(2) if you manually rename the incorrectly named target path before reprocessing then .xattr/<name>/CRC32 has no effect because the file name no longer matches, and you can just keep the CRC32 in the file name
(3) if you delete the incorrectly named target path, and then reprocess the original file path then .xattr/<name>/CRC32 already exists

Another issue with keeping sidecar .xattr files is that when files are named incorrectly and then renamed their old sidecar data is not removed from the .xattr folder. This can be a manual housekeeping problem and I have seen conflicts with renaming files when there are too many iterations in that .xattr folder.

The xattr script can be used to prune orphaned .xattr folders:
viewtopic.php?p=5394#p5394

That said, having a few extraneous small files and folders lying around has no effect. If you didn't check the .xattr folders you wouldn't know or worry about them. But if you are OCD about these things, then xattr script can help you find and delete them, but it's not strictly necessary.

[cheaters]

I feel ambushed. Let's try thinking this way.... what "Use Case" would you have considered worthy of the effort required to add this feature?

I have read on these forums that we can and perhaps should use crc32 to check for corrupt files routinely.

I will continue to use crc32 in my filenames because it's an excellent way of differentiating between similar movie files and associating multiple (more than one per movie) .srt files with their matching movie files.

[rednoah]

Well, if I add a feature that doesn't solve the actually problem that you are trying to solve... It's not about "worthy". It's simply about "Does it solve the problem?" or not...

The post above is not an "ambush" but a detailed explanation (which takes a significant amount of time to write; you are welcome) as to why I think that your feature request does not actually solve the problem that you are trying to solve, to the best of my understanding of the problem at hand, based on the information that you have provided.

You may think that adding CRC32 xattr metadata files will help you, but I have reason to believe that this is not so. Primarily, because you already have CRC32 checksums in the file path.



I could image a use case like "I want to be able to verify CRC32 checksums in the future, but I do not want to embed CRC32 checksums in the file name because I just don't like that." In this case I would propose a feature that auto-generates and auto-updates standard *.sfv files. You'll want standard file formats (and not FileBot-specific .xattr folders) for long-term archival purposes. If you're not a programmer yourself, then you'll be relying on standard GUI tools (FileBot or otherwise) to verify those CRC32 checksums.



Adding the CRC32 at the end of the filename is already a good choice, IMHO the best solution for the problem at hand. It's simple and it works. It's been the standard in the Anime community for decades. FileBot has 1st class support for checking embedded CRC32 checksums.

[rednoah]

Read external CSV / TSV / JSON / XML files