Problem loading files with tildes (e.g. Señor) on NFS mount (without NFC option)

[8i6juyhrggvf]

I have made sure this is isolated to my Mac only, on my Linux install all the files load as expected. But today I went to rename a couple files that had tilde's in them and noticed filebot did not load them on my Mac. I have tried purging my filebot install and setting it back up on my Mac, including deleting the dot directory in my users home dir. I just updated to 13.4.1 and using version 5.4.3 of filebot. Not sure how long this has been going on as I don't get many files with them in it. I also checked the logs and they have output nothing in the timeframes I have tried opening those files. Am I missing something? Thank you for any help with this.

[edit]
Forgot to mention that if I manually edit the file of just the tilde character it then loads as expected.

[rednoah]

GUI or CLI?


If you're using the filebot command with --log-file and don't see any output, then there's a strong possibility that filebot is not called in the first place. Do you perhaps have bash script that automatically calls filebot that might sometimes error out?


I see no fundamental issues with the ~ character on macOS:

Console Output: Select all

$ filebot -rename *.mp4 --log INFO
[MOVE] from [Alias.1x01.~TILDE.mp4] to [Alias - 1x01 - Truth Be Told.mp4]

Please post screenshots or files paths as text so that we can see what you are seeing and perhaps allow us to reproduce the issue ourselves.


Please read How to Request Help for details:

• Include Screenshots (absolutely required for new topics)
• Include file paths as text when reporting mismatches (press F7 to copy debug information)

[8i6juyhrggvf]

Apologies forgot to mention it's with the GUI I was trying, then opening the logfile in the dot filebot dir of my user. Here is me doing it with the cli

Code: Select all

filebot -rename Señor.mkv --log INFO
Initialize new disk cache: /Users/USERNAME/.filebot/cache/1
Failed to read xattr: Señor.mkv [Last-Modified: Thu Jun 22 22:20:52 PDT 2023]: java.nio.file.NoSuchFileException: Señor.mkv
Failed to process group: {Series=30 Rock} [Señor.mkv]
Failed to identify or process any files

If I change the file to Senor.mkv it sees the file but fails to identify it because I didn't leave enough info in the filename I am assuming I really dont use the cli tool even though I should. In the GUI I can get it to load the file and properly find the right season and episode once the tilde is gone.

[edit]
And so you know the file is really there

Code: Select all

-rwxrwxrwx  1 1024  _lpoperator  542794462 Jun 22 22:20 Señor.mkv

[rednoah]

I see. We're talking about combining diacritics and not the ~ character. Diacritics are tricky because you can have different byte sequences that are rendered the same, e.g. ñ vs n+◌̃, i.e. file names that look the same but are not the same. Notably, copy & paste operations, posting on the internet, etc may normalize n+◌̃ to ñ hiding the underlying issue when you try to investigate.


How did you get those files? rsync from a Linux server?


What composition do your files use?

Console Output: Select all

$ filebot -mediainfo *.mkv --format "{ fn } | { fn.transliterate(/any-name/) }"
Señor | \N{LATIN CAPITAL LETTER S}\N{LATIN SMALL LETTER E}\N{LATIN SMALL LETTER N WITH TILDE}\N{LATIN SMALL LETTER O}\N{LATIN SMALL LETTER R}

What if you type ñ yourself? Does ls -l Señor.mkv work? Because ñ at least works for me:

Console Output: Select all

$ filebot -rename Señor.mkv -non-strict --log INFO
[MOVE] from [Señor.mkv] to [Señor (2019).mkv]

[8i6juyhrggvf]

Thanks for the help as always, learned something new about the proper term.

I get the file onto a Linux system through transmission. From there a folder is nfs mounted onto the MacOS host. I open the GUI click the .mkv or whatever extension the file is and load in. I know at one time this was working just fine on the Mac host as I had to write some format syntax to replace those. I thought it was my format syntax possibly causing this and switched to just {plex}, have been using only {plex} since discovering in hopes that kept it simpler on debugging.

All below ran on my MacOS host

Code: Select all

filebot -mediainfo Señor.mkv --format "{ fn } | { fn.transliterate(/any-name/) }"
Failed to read xattr: /Volumes/Media/Señor.mkv [Last-Modified: Thu Jun 22 22:20:52 PDT 2023]: java.nio.file.NoSuchFileException: /Volumes/Media/Señor.mkv
Señor | \N{LATIN CAPITAL LETTER S}\N{LATIN SMALL LETTER E}\N{LATIN SMALL LETTER N WITH TILDE}\N{LATIN SMALL LETTER O}\N{LATIN SMALL LETTER R}

Code: Select all

$ ls -lah Señor.mkv
-rwxrwxrwx  1 1024  _lpoperator   518M Jun 22 22:20 Señor.mkv

$ mv Señor.mkv Senor.mkv

$ ls -lah Senor.mkv
-rwxrwxrwx  1 1024  _lpoperator   518M Jun 22 22:20 Senor.mkv

$ filebot -mediainfo Senor.mkv --format "{ fn } | { fn.transliterate(/any-name/) }"
Senor | \N{LATIN CAPITAL LETTER S}\N{LATIN SMALL LETTER E}\N{LATIN SMALL LETTER N}\N{LATIN SMALL LETTER O}\N{LATIN SMALL LETTER R}

$ mv Senor.mkv Señor.mkv

$ filebot -mediainfo Señor.mkv --format "{ fn } | { fn.transliterate(/any-name/) }"
Failed to read xattr: /Volumes/Media/Señor.mkv [Last-Modified: Thu Jun 22 22:20:52 PDT 2023]: java.nio.file.NoSuchFileException: /Volumes/Media/Señor.mkv
Señor | \N{LATIN CAPITAL LETTER S}\N{LATIN SMALL LETTER E}\N{LATIN SMALL LETTER N WITH TILDE}\N{LATIN SMALL LETTER O}\N{LATIN SMALL LETTER R}

Code: Select all

filebot -rename Señor.mkv -non-strict --log INFO
Failed to read xattr: /Volumes/Media/Señor.mkv [Last-Modified: Thu Jun 22 22:20:52 PDT 2023]: java.nio.file.NoSuchFileException: /Volumes/Media/Señor.mkv
Activate License [********] on [Fri Jun 23 06:50:07 PDT 2023]
[MOVE] from [/Volumes/Media/Señor.mkv] to [/Volumes/Media/30 Rock - 3x07 - Señor Macho Solo.mkv]
[MOVE] from [/Volumes/Media/Señor.mkv] to [/Volumes/Media/30 Rock - 3x07 - Señor Macho Solo.mkv] failed due to I/O error [No Such File: /Volumes/Media/Señor.mkv]

$ ls -lah Señor.mkv
-rwxrwxrwx  1 1024  _lpoperator   518M Jun 22 22:20 Señor.mkv

$ mv Señor.mkv Senor.mkv

$ filebot -rename Senor.mkv -non-strict --log INFO
[MOVE] from [/Volumes/Media/Senor.mkv] to [/Volumes/Media/30 Rock - 3x07 - Señor Macho Solo.mkv]

If I forgot anything or can provide more info just let me know.

[rednoah]

I get the file onto a Linux system through transmission. From there a folder is nfs mounted onto the MacOS host.

What NFS mount option are you using? If you mount a Linux file system on macOS, then you may need to enable certain compatibility options:
https://superuser.com/a/757570/349721


Since you seem to be able to address the file with Terminal, getting the character codes from console output might shed some more light on the issue:

Shell: Select all

ls | xxd

** run in an empty folder that only contains Señor.mkv to keep the hex dump readable



EDIT:

Further investigation reveals that NFC / NFD doesn't matter on APFS because the underlying file system will accept either unicode sequence to address the file. The shell will always print NFD unicode and FileBot will always print NFC unicode, since arguments are passed in as NFC unicode, and both will work. The issue is probably specific to the NFS file system, which, depending on your options, treats NFC / NFD unicode sequences as "different file name". I did not run any tests on an NFS remote file system though.

[8i6juyhrggvf]

No other files have the access issues only ones with similar diacritics just FYI.

I have not changed mount options in probably 2ish years on this host and it did work before as I have other files with diacritics that were loaded and renamed just not sure of when it broke. I know that doesn't mean much, but figured might be useful

Mount options

Code: Select all

(nfs, nodev, nosuid)

I will explore some different mount options to see if I can get any different results. But wouldn't that mean that everything on the MacOS system shouldn't be able to interact with that file? VLC can load and play it, I can move it, I can ls it, the only thing on the MacOS host that seems to not play nice is Filebot. Happy to test the file use with these mount options in any method that is more useful to you if VLC loading and playing it just fine isn't a good choice.

Code: Select all

$ ls | xxd
00000000: 5365 6ecc 836f 722e 6d6b 760a            Sen..or.mkv.
$ mv Señor.mkv Senor.mkv
$ ls
Senor.mkv
$ ls | xxd
00000000: 5365 6e6f 722e 6d6b 760a                 Senor.mkv.
$ mv Senor.mkv Señor.mkv
$ ls | xxd
00000000: 5365 6ecc 836f 722e 6d6b 760a            Sen..or.mkv.

[rednoah]

No other files have the access issues only ones with similar diacritics just FYI.

You will only ever run into this issue with combining diacritics. Characters like "a" and "好" only have one unicode representation and so you will never run into this issue. ñ has multiple unicode representations - "\u00F1" and "\u006E\u0303" - which is as different as "a" and "xy" for a Linux file system, and so you can have multiple file paths that look the same, unicode-equivalent even, but are actually canonically different.

I have not changed mount options in probably 2ish years on this host and it did work before as I have other files with diacritics that were loaded and renamed just not sure of when it broke. I know that doesn't mean much, but figured might be useful

My educated guess would be that ñ "\u006E\u0303" (NFD; normal form decomposed) works and ñ "\u00F1" (NFC; normal form composed) does not work, if the software at hand happens to use / expect NFD unicode forms as opposed to NFC unicode forms, or just retains the "bag of bytes" path whatever it may be. FileBot happens to normalize any input to NFC, somehow somewhere, which is unlucky if the input files use NFD, on a file system that doesn't normalize unicode forms. If the files at the source were to use NFC then maybe it would just work.

e.g. create 2 files with the "same" name on Linux:

Console Output: Select all

$ touch ñ.mkv ñ.mkv
$ ls | cat
ñ.mkv
ñ.mkv
$ ls | xxd -g1
00000000: 6e cc 83 2e 6d 6b 76 0a c3 b1 2e 6d 6b 76 0a     n...mkv....mkv.

** [NFD] 6e cc 83 = n + ◌̃ | [NFC] c3 b1 = ñ
** If you run the same command on macOS / APFS file system only one ñ.mkv file will be created, because macOS APFS normalizes unicode forms by default, while Linux does not. This is the crux of the problem.




I'd start by creating test files for both unicode forms on the Linux machine, and then try to process them with FileBot from the macOS host, and see if one of them works.

[8i6juyhrggvf]

You were totally right about the differences and that I just never caught it before or a recent change in MacOS updates made me require mount changes. Not Filebot at all but my nfs mount itself.

Using the following works now again as I was expecting in my /etc/nfs.conf on the MacOS client. The `nfc` being the key options for the mount

Code: Select all

nfs.client.mount.options = tcp,vers=4,nfc

[rednoah]

You were totally right about the differences and that I just never caught it before or a recent change in MacOS updates made me require mount changes.

No worries. NFC / NFD is tricky and just not something you think about because unicode forms are not relevant for most characters / most languages, and most file systems normalize things implicitly by default anyway. It's a bit of a rare corner case that would never cross your mind until you actually have the problem.


If you ever rsync between macOS and Linux then you may also need to use the --iconv option to normalize unicode forms for the target file system:

Shell: Select all

rsync --iconv=utf-8-mac,utf-8

[8i6juyhrggvf]

Thanks for the rsync tip! Really appreciate all your time and efforts in Filebot and support.

I think what threw me the most at first was that ls, mv, etc was all "OK" appearing. Another good lesson to have learned for strange edge cases.