Metadata and Extended Attributes

[rednoah]

FileBot (r1255) uses xattr (i.e. Extended Attributes / NTFS Alternative Streams) to store full metadata in filesystem attributes. These xattr are not to be confused with traditional tags that are embedded in the file content, but additional data streams stored in the file system for each file that are not part of the file itself. As such, xattr requires a filesystem that supports xattr, and xattr may be silently lost if files are moved to a filesystem that doesn't support xattr.


Details:

• [xattr:net.filebot.metadata] Episode or Movie Object (as json)
• [xattr:net.filebot.filename] Original filename
• [basic:creationTime] Episode airdate or Movie release year (disabled by default for compatibility reasons)

Usage:
FileBot and other programs have the original filename and all episode / movie metadata available regardless how the file is named. Sure nice to have full metadata instantly / offline available should you need it.


Disable this feature via Java System Properties:

Code: Select all

-DuseExtendedFileAttributes=false

(requires FileBot 4.8.5) Configure FileBot to store xattr metadata in plain text files instead of filesystem xattr streams:

Code: Select all

-Dnet.filebot.xattr.store=.xattr

View or Clear Extended Attributes
with the filebot command-line tools and xattr script:

Code: Select all

filebot -script fn:xattr /path/to/files

[rednoah]

1. Rename files with FileBot:

Code: Select all

$ filebot -rename *.mp4 --db TheMovieDB -non-strict
Rename movies using [TheMovieDB]
Auto-detect movie from context: [avatar.mp4]
[MOVE] From [avatar.mp4] to [Avatar (2009).mp4]
Processed 1 files

2. Confirm that FileBot metadata has been stored as xattr in the filesystem:

Code: Select all

$ filebot -script fn:xattr *.mp4
/path/to/Avatar (2009).mp4
	net.filebot.filename: avatar.mp4
	net.filebot.metadata: {"@type":"Movie","year":2009,"imdbId":499549,"tmdbId":19995,"language":"en","id":19995,"name":"Avatar","aliasNames":[]}

3. Use xattr metadata in filter and format expressions:

Code: Select all

$ filebot -mediainfo -r . --filter "y >= 2000" --format "{n} by {director}"
Avatar by James Cameron

4. Process xattr tagged files based on existing xattr metadata:

Code: Select all

filebot -rename -r . --db xattr --action copy --output /Volumes/USB --filter "episode && age < 30" --format "{n} - {sxe} - {t}"

[oneguynick]

Code: Select all

filebot -rename *.mp4 --db TVDB -non-strict
Illegal Argument: java.nio.file.InvalidPathException: Illegal char <*> at index 0: *.mp4 (*.mp4)
Failed to read xattr: InvalidPathException: Illegal char <*> at index 0: *.mp4
Failed to read xattr: InvalidPathException: Illegal char <*> at index 0: *.mp4
Failed to read xattr: InvalidPathException: Illegal char <*> at index 0: *.mp4
Failed to read xattr: InvalidPathException: Illegal char <*> at index 0: *.mp4

^C

Example filename: Penoza - S01E02 - Unpleasant Surprises HDTV-720p.mkv

[rednoah]

*.mp4 refers to all mp4 files. If you're testing with mkv files then *.mkv will select those.

@see https://mywiki.wooledge.org/glob

[oneguynick]

My mistake. I forgot they were *.mkv, good catch. With that said I am now testing the same working commands against my samba share with the same issue. Copy locally just isn't an option. Is there a better way to do this? NFS or SFTP or something?

[oneguynick]

Alright so I took the network share out of the picture for troubleshooting and tried docker:

Code: Select all

docker run -it -v /tv:/volume1 -v /opt/docker/filebot:/data rednoah/filebot -script fn:xattr -r *
docker run -it -v /tv:/volume1 -v /opt/docker/filebot:/data rednoah/filebot -script fn:duplicates -r --action test *

Done ヾ(＠⌒ー⌒＠)ノ in both cases. I specifically copied a folder of shows to ensure there were exact duplicates.

I had considered that maybe it was the ZFS pool. I verified the pool is enabled for xattr:

Code: Select all

NAME                                                    PROPERTY  VALUE  SOURCE
share                                                   xattr     on     default

[rednoah]

NFS might work. SFTP probably doesn't preserve attributes, but RSYNC might (with --xattrs option).

If you're using docker with some host folder mounted into the docker, then there's a good chance xattr won't work. I remember filesystem events not working either. Not sure if that's something that can be enabled or if it's just not supported.

If you care about xattr, then the best solution is usually to process files on the NAS or server where reading/writing xattr usually works out of the box.

FileBot generally doesn't require xattr, it just stores some xattr info if possible, but will work just fine without. It's just that the duplicates script requires them to work, so it'll instantly know which file is which movie/episode without having to bother any web services.