Problem loading files with tildes (e.g. Señor) on NFS mount (without NFC option)
-
- Posts: 14
- Joined: 19 Mar 2023, 02:21
Problem loading files with tildes (e.g. Señor) on NFS mount (without NFC option)
I have made sure this is isolated to my Mac only, on my Linux install all the files load as expected. But today I went to rename a couple files that had tilde's in them and noticed filebot did not load them on my Mac. I have tried purging my filebot install and setting it back up on my Mac, including deleting the dot directory in my users home dir. I just updated to 13.4.1 and using version 5.4.3 of filebot. Not sure how long this has been going on as I don't get many files with them in it. I also checked the logs and they have output nothing in the timeframes I have tried opening those files. Am I missing something? Thank you for any help with this.
[edit]
Forgot to mention that if I manually edit the file of just the tilde character it then loads as expected.
[edit]
Forgot to mention that if I manually edit the file of just the tilde character it then loads as expected.
Re: Problem loading files with tildes
![Question :?:](./images/smilies/icon_question.gif)
![Idea :idea:](./images/smilies/icon_idea.gif)
![Arrow :arrow:](./images/smilies/icon_arrow.gif)
Console Output: Select all
$ filebot -rename *.mp4 --log INFO
[MOVE] from [Alias.1x01.~TILDE.mp4] to [Alias - 1x01 - Truth Be Told.mp4]
![Arrow :arrow:](./images/smilies/icon_arrow.gif)
![Idea :idea:](./images/smilies/icon_idea.gif)
rednoah wrote: ↑31 Jul 2014, 16:40
- Include Screenshots (absolutely required for new topics)
- Include file paths as text when reporting mismatches (press F7 to copy debug information)
-
- Posts: 14
- Joined: 19 Mar 2023, 02:21
Re: Problem loading files with tildes
Apologies forgot to mention it's with the GUI I was trying, then opening the logfile in the dot filebot dir of my user. Here is me doing it with the cli
If I change the file to Senor.mkv it sees the file but fails to identify it because I didn't leave enough info in the filename I am assuming I really dont use the cli tool even though I should. In the GUI I can get it to load the file and properly find the right season and episode once the tilde is gone.
[edit]
And so you know the file is really there
Code: Select all
filebot -rename Señor.mkv --log INFO
Initialize new disk cache: /Users/USERNAME/.filebot/cache/1
Failed to read xattr: Señor.mkv [Last-Modified: Thu Jun 22 22:20:52 PDT 2023]: java.nio.file.NoSuchFileException: Señor.mkv
Failed to process group: {Series=30 Rock} [Señor.mkv]
Failed to identify or process any files
[edit]
And so you know the file is really there
Code: Select all
-rwxrwxrwx 1 1024 _lpoperator 542794462 Jun 22 22:20 Señor.mkv
Re: Problem loading files with tildes
I see. We're talking about combining diacritics and not the ~ character. Diacritics are tricky because you can have different byte sequences that are rendered the same, e.g. ñ vs n+◌̃, i.e. file names that look the same but are not the same. Notably, copy & paste operations, posting on the internet, etc may normalize n+◌̃ to ñ hiding the underlying issue when you try to investigate.
How did you get those files? rsync from a Linux server?
What composition do your files use?
What if you type ñ yourself? Does ls -l Señor.mkv work? Because ñ at least works for me:
![Question :?:](./images/smilies/icon_question.gif)
![Question :?:](./images/smilies/icon_question.gif)
Console Output: Select all
$ filebot -mediainfo *.mkv --format "{ fn } | { fn.transliterate(/any-name/) }"
Señor | \N{LATIN CAPITAL LETTER S}\N{LATIN SMALL LETTER E}\N{LATIN SMALL LETTER N WITH TILDE}\N{LATIN SMALL LETTER O}\N{LATIN SMALL LETTER R}
![Question :?:](./images/smilies/icon_question.gif)
Console Output: Select all
$ filebot -rename Señor.mkv -non-strict --log INFO
[MOVE] from [Señor.mkv] to [Señor (2019).mkv]
-
- Posts: 14
- Joined: 19 Mar 2023, 02:21
Re: Problem loading files with tildes (e.g. Señor)
Thanks for the help as always, learned something new about the proper term.
I get the file onto a Linux system through transmission. From there a folder is nfs mounted onto the MacOS host. I open the GUI click the .mkv or whatever extension the file is and load in. I know at one time this was working just fine on the Mac host as I had to write some format syntax to replace those. I thought it was my format syntax possibly causing this and switched to just {plex}, have been using only {plex} since discovering in hopes that kept it simpler on debugging.
All below ran on my MacOS host
If I forgot anything or can provide more info just let me know.
I get the file onto a Linux system through transmission. From there a folder is nfs mounted onto the MacOS host. I open the GUI click the .mkv or whatever extension the file is and load in. I know at one time this was working just fine on the Mac host as I had to write some format syntax to replace those. I thought it was my format syntax possibly causing this and switched to just {plex}, have been using only {plex} since discovering in hopes that kept it simpler on debugging.
All below ran on my MacOS host
Code: Select all
filebot -mediainfo Señor.mkv --format "{ fn } | { fn.transliterate(/any-name/) }"
Failed to read xattr: /Volumes/Media/Señor.mkv [Last-Modified: Thu Jun 22 22:20:52 PDT 2023]: java.nio.file.NoSuchFileException: /Volumes/Media/Señor.mkv
Señor | \N{LATIN CAPITAL LETTER S}\N{LATIN SMALL LETTER E}\N{LATIN SMALL LETTER N WITH TILDE}\N{LATIN SMALL LETTER O}\N{LATIN SMALL LETTER R}
Code: Select all
$ ls -lah Señor.mkv
-rwxrwxrwx 1 1024 _lpoperator 518M Jun 22 22:20 Señor.mkv
$ mv Señor.mkv Senor.mkv
$ ls -lah Senor.mkv
-rwxrwxrwx 1 1024 _lpoperator 518M Jun 22 22:20 Senor.mkv
$ filebot -mediainfo Senor.mkv --format "{ fn } | { fn.transliterate(/any-name/) }"
Senor | \N{LATIN CAPITAL LETTER S}\N{LATIN SMALL LETTER E}\N{LATIN SMALL LETTER N}\N{LATIN SMALL LETTER O}\N{LATIN SMALL LETTER R}
$ mv Senor.mkv Señor.mkv
$ filebot -mediainfo Señor.mkv --format "{ fn } | { fn.transliterate(/any-name/) }"
Failed to read xattr: /Volumes/Media/Señor.mkv [Last-Modified: Thu Jun 22 22:20:52 PDT 2023]: java.nio.file.NoSuchFileException: /Volumes/Media/Señor.mkv
Señor | \N{LATIN CAPITAL LETTER S}\N{LATIN SMALL LETTER E}\N{LATIN SMALL LETTER N WITH TILDE}\N{LATIN SMALL LETTER O}\N{LATIN SMALL LETTER R}
Code: Select all
filebot -rename Señor.mkv -non-strict --log INFO
Failed to read xattr: /Volumes/Media/Señor.mkv [Last-Modified: Thu Jun 22 22:20:52 PDT 2023]: java.nio.file.NoSuchFileException: /Volumes/Media/Señor.mkv
Activate License [********] on [Fri Jun 23 06:50:07 PDT 2023]
[MOVE] from [/Volumes/Media/Señor.mkv] to [/Volumes/Media/30 Rock - 3x07 - Señor Macho Solo.mkv]
[MOVE] from [/Volumes/Media/Señor.mkv] to [/Volumes/Media/30 Rock - 3x07 - Señor Macho Solo.mkv] failed due to I/O error [No Such File: /Volumes/Media/Señor.mkv]
$ ls -lah Señor.mkv
-rwxrwxrwx 1 1024 _lpoperator 518M Jun 22 22:20 Señor.mkv
$ mv Señor.mkv Senor.mkv
$ filebot -rename Senor.mkv -non-strict --log INFO
[MOVE] from [/Volumes/Media/Senor.mkv] to [/Volumes/Media/30 Rock - 3x07 - Señor Macho Solo.mkv]
Re: Problem loading files with tildes (e.g. Señor)
8i6juyhrggvf wrote: ↑23 Jun 2023, 14:00 I get the file onto a Linux system through transmission. From there a folder is nfs mounted onto the MacOS host.
![Question :?:](./images/smilies/icon_question.gif)
https://superuser.com/a/757570/349721
![Question :?:](./images/smilies/icon_question.gif)
Shell: Select all
ls | xxd
EDIT:
Further investigation reveals that NFC / NFD doesn't matter on APFS because the underlying file system will accept either unicode sequence to address the file. The shell will always print NFD unicode and FileBot will always print NFC unicode, since arguments are passed in as NFC unicode, and both will work. The issue is probably specific to the NFS file system, which, depending on your options, treats NFC / NFD unicode sequences as "different file name". I did not run any tests on an NFS remote file system though.
-
- Posts: 14
- Joined: 19 Mar 2023, 02:21
Re: Problem loading files with tildes (e.g. Señor)
No other files have the access issues only ones with similar diacritics just FYI.
I have not changed mount options in probably 2ish years on this host and it did work before as I have other files with diacritics that were loaded and renamed just not sure of when it broke. I know that doesn't mean much, but figured might be useful
Mount options
I will explore some different mount options to see if I can get any different results. But wouldn't that mean that everything on the MacOS system shouldn't be able to interact with that file? VLC can load and play it, I can move it, I can ls it, the only thing on the MacOS host that seems to not play nice is Filebot. Happy to test the file use with these mount options in any method that is more useful to you if VLC loading and playing it just fine isn't a good choice.
I have not changed mount options in probably 2ish years on this host and it did work before as I have other files with diacritics that were loaded and renamed just not sure of when it broke. I know that doesn't mean much, but figured might be useful
Mount options
Code: Select all
(nfs, nodev, nosuid)
Code: Select all
$ ls | xxd
00000000: 5365 6ecc 836f 722e 6d6b 760a Sen..or.mkv.
$ mv Señor.mkv Senor.mkv
$ ls
Senor.mkv
$ ls | xxd
00000000: 5365 6e6f 722e 6d6b 760a Senor.mkv.
$ mv Senor.mkv Señor.mkv
$ ls | xxd
00000000: 5365 6ecc 836f 722e 6d6b 760a Sen..or.mkv.
Re: Problem loading files with tildes (e.g. Señor)
You will only ever run into this issue with combining diacritics. Characters like "a" and "好" only have one unicode representation and so you will never run into this issue. ñ has multiple unicode representations - "\u00F1" and "\u006E\u0303" - which is as different as "a" and "xy" for a Linux file system, and so you can have multiple file paths that look the same, unicode-equivalent even, but are actually canonically different.8i6juyhrggvf wrote: ↑23 Jun 2023, 16:09 No other files have the access issues only ones with similar diacritics just FYI.
My educated guess would be that ñ "\u006E\u0303" (NFD; normal form decomposed) works and ñ "\u00F1" (NFC; normal form composed) does not work, if the software at hand happens to use / expect NFD unicode forms as opposed to NFC unicode forms, or just retains the "bag of bytes" path whatever it may be. FileBot happens to normalize any input to NFC, somehow somewhere, which is unlucky if the input files use NFD, on a file system that doesn't normalize unicode forms. If the files at the source were to use NFC then maybe it would just work.8i6juyhrggvf wrote: ↑23 Jun 2023, 16:09 I have not changed mount options in probably 2ish years on this host and it did work before as I have other files with diacritics that were loaded and renamed just not sure of when it broke. I know that doesn't mean much, but figured might be useful
e.g. create 2 files with the "same" name on Linux:
Console Output: Select all
$ touch ñ.mkv ñ.mkv
$ ls | cat
ñ.mkv
ñ.mkv
$ ls | xxd -g1
00000000: 6e cc 83 2e 6d 6b 76 0a c3 b1 2e 6d 6b 76 0a n...mkv....mkv.
** If you run the same command on macOS / APFS file system only one ñ.mkv file will be created, because macOS APFS normalizes unicode forms by default, while Linux does not. This is the crux of the problem.
![Arrow :arrow:](./images/smilies/icon_arrow.gif)
-
- Posts: 14
- Joined: 19 Mar 2023, 02:21
Re: Problem loading files with tildes (e.g. Señor) on NFS mount (without NFC option)
You were totally right about the differences and that I just never caught it before or a recent change in MacOS updates made me require mount changes. Not Filebot at all but my nfs mount itself.
Using the following works now again as I was expecting in my /etc/nfs.conf on the MacOS client. The `nfc` being the key options for the mount
Using the following works now again as I was expecting in my /etc/nfs.conf on the MacOS client. The `nfc` being the key options for the mount
Code: Select all
nfs.client.mount.options = tcp,vers=4,nfc
Re: Problem loading files with tildes (e.g. Señor) on NFS mount (without NFC option)
No worries. NFC / NFD is tricky and just not something you think about because unicode forms are not relevant for most characters / most languages, and most file systems normalize things implicitly by default anyway. It's a bit of a rare corner case that would never cross your mind until you actually have the problem.8i6juyhrggvf wrote: ↑24 Jun 2023, 03:36 You were totally right about the differences and that I just never caught it before or a recent change in MacOS updates made me require mount changes.
![Idea :idea:](./images/smilies/icon_idea.gif)
Shell: Select all
rsync --iconv=utf-8-mac,utf-8
-
- Posts: 14
- Joined: 19 Mar 2023, 02:21
Re: Problem loading files with tildes (e.g. Señor) on NFS mount (without NFC option)
Thanks for the rsync tip! Really appreciate all your time and efforts in Filebot and support.
I think what threw me the most at first was that ls, mv, etc was all "OK" appearing. Another good lesson to have learned for strange edge cases.
I think what threw me the most at first was that ls, mv, etc was all "OK" appearing. Another good lesson to have learned for strange edge cases.