Metadata and Extended Attributes

How-to guides, frequently asked questions, not-so-obvious features, etc
Post Reply
User avatar
rednoah
The Source
Posts: 16990
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Metadata and Extended Attributes

Post by rednoah » 24 Oct 2012, 11:16

FileBot uses xattr (i.e. Extended Attributes / NTFS Alternative Streams) to store full metadata in filesystem attributes. These xattr are not to be confused with traditional tags that are embedded in the file content, but additional data streams stored in the file system for each file that are not part of the file itself. As such, xattr requires a filesystem that supports xattr, and xattr may be silently lost if files are moved to a filesystem that doesn't support xattr.


Details:
  • [xattr:net.filebot.metadata] Episode or Movie Object (as json)
  • [xattr:net.filebot.filename] Original filename
  • [basic:creationTime] Episode airdate or Movie release year (disabled by default for compatibility reasons)

Usage:
FileBot and other programs have the original filename and all episode / movie metadata available regardless how the file is named. Sure nice to have full metadata instantly / offline available should you need it.


Disable this feature via the -no-xattr command-line flag:

Code: Select all

-no-xattr
or via system properties:

Code: Select all

filebot -script fn:properties --def useExtendedFileAttributes=false

(requires FileBot 4.8.5) Configure FileBot to store xattr metadata in plain text files instead of filesystem xattr streams:

Code: Select all

filebot -script fn:properties --def net.filebot.xattr.store=.xattr
:idea: If you require cross-platform compatibility, then using .xattr folders and files is highly recommended. The FileBot Xattr Metadata Scanners & Plug-ins for Plex
support both native filesystem xattr and .xattr folders by default.


View or Clear Extended Attributes
with the filebot command-line tools and xattr script:

Code: Select all

filebot -script fn:xattr /path/to/files
:idea: Please read the FAQ and How to Request Help.

User avatar
rednoah
The Source
Posts: 16990
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Usage Examples

Post by rednoah » 05 Apr 2017, 09:27

1. Rename files with FileBot:

Code: Select all

$ filebot -rename *.mp4 --db TheMovieDB -non-strict
Rename movies using [TheMovieDB]
Auto-detect movie from context: [avatar.mp4]
[MOVE] From [avatar.mp4] to [Avatar (2009).mp4]
Processed 1 files
2. Confirm that FileBot metadata has been stored as xattr in the filesystem:

Code: Select all

$ filebot -script fn:xattr *.mp4
/path/to/Avatar (2009).mp4
	net.filebot.filename: avatar.mp4
	net.filebot.metadata: {"@type":"Movie","year":2009,"imdbId":499549,"tmdbId":19995,"language":"en","id":19995,"name":"Avatar","aliasNames":[]}
3. Use xattr metadata in filter and format expressions:

Code: Select all

$ filebot -mediainfo -r . --filter "y >= 2000" --format "{n} by {director}"
Avatar by James Cameron
4. Process xattr tagged files based on existing xattr metadata:

Code: Select all

filebot -rename -r . --db xattr --action copy --output /Volumes/USB --filter "episode && age < 30" --format "{n} - {sxe} - {t}"
:idea: Please read the FAQ and How to Request Help.

oneguynick
Posts: 7
Joined: 06 Jan 2018, 13:45

Re: Metadata and Extended Attributes

Post by oneguynick » 07 Jan 2018, 12:58

Code: Select all

filebot -rename *.mp4 --db TVDB -non-strict
Illegal Argument: java.nio.file.InvalidPathException: Illegal char <*> at index 0: *.mp4 (*.mp4)
Failed to read xattr: InvalidPathException: Illegal char <*> at index 0: *.mp4
Failed to read xattr: InvalidPathException: Illegal char <*> at index 0: *.mp4
Failed to read xattr: InvalidPathException: Illegal char <*> at index 0: *.mp4
Failed to read xattr: InvalidPathException: Illegal char <*> at index 0: *.mp4
^C

Example filename: Penoza - S01E02 - Unpleasant Surprises HDTV-720p.mkv

User avatar
rednoah
The Source
Posts: 16990
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Metadata and Extended Attributes

Post by rednoah » 07 Jan 2018, 13:08

*.mp4 refers to all mp4 files. If you're testing with mkv files then *.mkv will select those.

@see https://mywiki.wooledge.org/glob
:idea: Please read the FAQ and How to Request Help.

oneguynick
Posts: 7
Joined: 06 Jan 2018, 13:45

Re: Metadata and Extended Attributes

Post by oneguynick » 08 Jan 2018, 10:36

My mistake. I forgot they were *.mkv, good catch. With that said I am now testing the same working commands against my samba share with the same issue. Copy locally just isn't an option. Is there a better way to do this? NFS or SFTP or something?

oneguynick
Posts: 7
Joined: 06 Jan 2018, 13:45

Re: Metadata and Extended Attributes

Post by oneguynick » 08 Jan 2018, 10:58

Alright so I took the network share out of the picture for troubleshooting and tried docker:

Code: Select all

docker run -it -v /tv:/volume1 -v /opt/docker/filebot:/data rednoah/filebot -script fn:xattr -r *
docker run -it -v /tv:/volume1 -v /opt/docker/filebot:/data rednoah/filebot -script fn:duplicates -r --action test *
Done ヾ(@⌒ー⌒@)ノ in both cases. I specifically copied a folder of shows to ensure there were exact duplicates.

I had considered that maybe it was the ZFS pool. I verified the pool is enabled for xattr:

Code: Select all

NAME                                                    PROPERTY  VALUE  SOURCE
share                                                   xattr     on     default

User avatar
rednoah
The Source
Posts: 16990
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Metadata and Extended Attributes

Post by rednoah » 08 Jan 2018, 17:03

NFS might work. SFTP probably doesn't preserve attributes, but RSYNC might (with --xattrs option).

If you're using docker with some host folder mounted into the docker, then there's a good chance xattr won't work. I remember filesystem events not working either. Not sure if that's something that can be enabled or if it's just not supported.

:idea: If you care about xattr, then the best solution is usually to process files on the NAS or server where reading/writing xattr usually works out of the box.

:arrow: FileBot generally doesn't require xattr, it just stores some xattr info if possible, but will work just fine without. It's just that the duplicates script requires them to work, so it'll instantly know which file is which movie/episode without having to bother any web services.
:idea: Please read the FAQ and How to Request Help.

b2un0
Posts: 10
Joined: 09 Jun 2014, 10:41

Re: Metadata and Extended Attributes

Post by b2un0 » 06 Jul 2019, 06:38

Hi,

is there any option to add filebot xattr to already sorted files?

eg, i have multiple series where on some seasons the xattr from filebot are missing, so ne the " filebot missing script" does not work correct and think the season is missing.

User avatar
rednoah
The Source
Posts: 16990
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Metadata and Extended Attributes

Post by rednoah » 06 Jul 2019, 06:46

You'll have to process files through FileBot again to ensure that xattr are set this time.

You can find files without xattr metadata like so:

Code: Select all

filebot -mediainfo . --filter '!f.metadata' --format {f}
:idea: Please read the FAQ and How to Request Help.

b2un0
Posts: 10
Joined: 09 Jun 2014, 10:41

Re: Metadata and Extended Attributes

Post by b2un0 » 06 Jul 2019, 06:49

thanks for your fast reply!

thats the first thing i did, but filebot says "Skipped, already exists". :(

User avatar
rednoah
The Source
Posts: 16990
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Metadata and Extended Attributes

Post by rednoah » 06 Jul 2019, 06:55

1.
You can move all untagged files elsewhere first:

Code: Select all

filebot -mediainfo . --filter '!f.metadata' -exec mv -v {f} /path/to/untagged/files/{f.name}

2.
However, the general smart approach is to never touch files in your Media folder, and only ever move files there via FileBot. That way you can ensure that everything is always in perfect order.

i.e. just process all your files into a new folder, and then remove the old one if you're happy with the new one

Code: Select all

--format /path/to/new/folder/{plex}
:idea: Please read the FAQ and How to Request Help.

Post Reply