Match issue due to special characters when file name contains < and >

Any questions? Need some help?
Post Reply
starsys
Posts: 11
Joined: 13 May 2019, 12:46

Match issue due to special characters when file name contains < and >

Post by starsys »

After several trials, I finally understand why filebot do not recognize my file correctly.
Episode detection error comes from a special character : "<" or ">". When source file contains such character, filebot do not consider the filename at all but a strange random one like "GR2RV~.MKV" (and leads of course to matching issue)

As soon as I remove these characters from original filename, everything runs fine.

I'm using {plex} format.
Is there any solution for ignoring these special characters ?
Is it possible to ignore complete set of characters like " --> Torrent9.uno <--" (ignore pattern in original filename)
And thus help accurate episode detection ?

I'm french. Should I expect any issues with "éèàç..." characters ?

Thanks and congratulation for this great program (I bought the lifetime version ;-)
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Match issue due to special characters

Post by rednoah »

< and > are not allowed on Windows, so you can't have those file paths in the first place, unless you're using a Linux-based NAS (which isn't mentioned in your OP) then it's allowed there of course, but if you then access those files from Windows via SMB shares, then the SMB share will "normalize" the names so Windows can handle them.

:idea: There's nothing FileBot can do on your Windows client, because even Windows itself probably doesn't know the real file path on the NAS.

:idea: You can run FileBot on the NAS itself (i.e. the CLI, not the GUI) and then it'll have direct access to the file system and see the real file paths, and not the fake compatibility names as seen through the SMB share.

:idea: You can also use Desktop Linux with NFS shares, keeping the pipeline all Linux, so you won't run into Windows limitations.

:idea: éèàç are allowed in Windows file paths so those should be fine.
:idea: Please read the FAQ and How to Request Help.
starsys
Posts: 11
Joined: 13 May 2019, 12:46

Re: Match issue due to special characters when file name contains < and >

Post by starsys »

Hello.
Thanks for your feedback.
I'm using filebot running on a synology DS918+.
Input files are in a remote folder (pointing my local internet provider box, where torrents are downloaded. I assume it's also a linux based system) mounted via CIFS in the NAS.

I didn't find if I can mount my box via NFS on my synology.
If you've any idea, thanks in advance.
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Match issue due to special characters when file name contains < and >

Post by rednoah »

AFAIK, if you're on Windows, then SMB is your only option.

Depending on your use case, using FileBot Node running filebot locally on your NAS might work better:
viewtopic.php?f=13&t=1802
:idea: Please read the FAQ and How to Request Help.
starsys
Posts: 11
Joined: 13 May 2019, 12:46

Re: Match issue due to special characters when file name contains < and >

Post by starsys »

I'm not using windows.
I'm using my internet provider Box : "Freebox" for downloading : it's a Linux based system. (https://fr.wikipedia.org/wiki/Freebox#C ... s_internes)
I'm using Filebot node runnning on my Synology DS918+
The download drive of my Freebox is mounted in my NAS via CIFS (I can only link my Freebox to my NAS via CIFS or FTP, I cannot connect it via NFS)

I've seen that maybe editing file smb.conf would help (avoiding mangle and changing charset). Should I try ?
https://serverfault.com/questions/12461 ... -filenames

Thanks.
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Match issue due to special characters when file name contains < and >

Post by rednoah »

I see. Well, CIFS is a Windows thing so it probably defaults to Windows compatibility options.

There could be options to make it work as expected. Worth a try. Please report back on your findings.
:idea: Please read the FAQ and How to Request Help.
starsys
Posts: 11
Joined: 13 May 2019, 12:46

Re: Match issue due to special characters when file name contains < and >

Post by starsys »

I tried to edit smb.conf with different charset : it doesn't work.
I tried to change an advanced SMB setting in my NAS (It really seemed to match my problem) :
"Enable VFS module to convert Mac special characters: Select to convert SMB-reserved characters (e.g. < > " * ?) in filenames to Basic Latin during file creation. Such files will be recognizable via AFP" : it doesn't work

So my original input file looks like something like "my-file-name--> Torrent9.uno <--.mkv" on my freebox and is transformed to "GRVGK2~W.MKV" on my NAS...
I'm going crazy ! Why I cannot just remove / transform / ignore this fuc*ing characters !

I'll try to contact synology and understand why the special characters options doesn't work
starsys
Posts: 11
Joined: 13 May 2019, 12:46

Re: Match issue due to special characters when file name contains < and >

Post by starsys »

By the way : I just updated my NAS with the last firmware version : filebot node seems not to work anymore (Synology error window)
starsys
Posts: 11
Joined: 13 May 2019, 12:46

Re: Match issue due to special characters when file name contains < and >

Post by starsys »

I had to repair the package. Now I can launch the filebot node but it get stucked in this state :
Locking /volume1/@appstore/filebot-node/data/filebot.log

and nothing happens
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Match issue due to special characters when file name contains < and >

Post by rednoah »

That's fine. You might be running multiple instances of FileBot at the same time, and so it'll wait for previously invoked instances to finish first.
:idea: Please read the FAQ and How to Request Help.
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Match issue due to special characters when file name contains < and >

Post by rednoah »

My best guess is that the file names get mangled on the remote server, and so there's not much any device on your end can do about it:
https://www.oreilly.com/openbook/samba/ ... 05_04.html

You'll want to talk to however manages your Freebox thing rather than your Synology NAS.
:idea: Please read the FAQ and How to Request Help.
starsys
Posts: 11
Joined: 13 May 2019, 12:46

Re: Match issue due to special characters when file name contains < and >

Post by starsys »

Hello. Thanks for the info, really interesting.
Unfortunatly, NFS is not enabled in my Freebox : so I cannot use this protocol, and I've no access to its SMB.conf file.
Otherwise I would have tried this option :

Code: Select all

[data]
	mangled names = no
I think this setting has to be applied from the server side (My Freebox) not from client (My NAS) ?
starsys
Posts: 11
Joined: 13 May 2019, 12:46

Re: Match issue due to special characters when file name contains < and >

Post by starsys »

I still have problem with filebot node since the last synology update.
I stuck for hours (no update):

Code: Select all

filebot -script 'fn:amc' '/volume1/freebox/disque/Téléchargements/' --output /volume1/freebox/disque --action TEST -non-strict --order Airdate --conflict auto --lang fr --def 'music=y' 'unsorted=y' 'clean=y' 'skipExtract=y' 'seriesFormat=series/{plex.tail}' 'movieFormat=films/{plex.name}' 'unsortedFormat=films/{plex.name}' 'excludeList=.excludes' --log all --log-file '/volume1/@appstore/filebot-node/data/filebot.log'

------------------------------------------

Locking /volume1/@appstore/filebot-node/data/filebot.log
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.codehaus.groovy.vmplugin.v7.Java7$1 (file:/volume1/@appstore/filebot/jar/groovy.jar) to constructor java.lang.invoke.MethodHandles$Lookup(java.lang.Class,int)
WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.vmplugin.v7.Java7$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Run script [fn:amc] at [Wed May 15 09:47:09 CEST 2019]
Parameter: music = y
Parameter: unsorted = y
Parameter: clean = y
Parameter: skipExtract = y
Parameter: seriesFormat = series/{plex.tail}
Parameter: movieFormat = films/{plex.name}
Parameter: unsortedFormat = films/{plex.name}
Parameter: excludeList = .excludes
Argument[0]: /volume1/freebox/disque/Téléchargements
Use excludes: /volume1/freebox/disque/.excludes (5)
Nothing happens for more than 30 minutes........................

I also noticed that system detect a "OS: Linux (amd64)" whereas my NAS is using a Intel Celeron

Code: Select all

filebot -script 'fn:sysinfo'

------------------------------------------

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.codehaus.groovy.vmplugin.v7.Java7$1 (file:/volume1/@appstore/filebot/jar/groovy.jar) to constructor java.lang.invoke.MethodHandles$Lookup(java.lang.Class,int)
WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.vmplugin.v7.Java7$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
FileBot 4.8.5 (r6224)
JNA Native: 5.2.2
MediaInfo: 17.12
7-Zip-JBinding: 9.20
Chromaprint: java.io.IOException: Cannot run program "fpcalc": error=2, No such file or directory
Extended Attributes: OK
Unicode Filesystem: OK
Script Bundle: 2019-05-05 (r564)
Groovy: 2.5.6
JRE: OpenJDK Runtime Environment 12.0.1
JVM: 64-bit OpenJDK 64-Bit Server VM
CPU/MEM: 4 Core / 2 GB Max Memory / 37 MB Used Memory
OS: Linux (amd64)
HW: Linux DS918 4.4.59+ #24922 SMP PREEMPT Fri May 10 02:59:42 CST 2019 x86_64 GNU/Linux synology_apollolake_918+
DATA: /volume1/@appstore/filebot/data/admin
Package: SPK
License: FileBot License PX7914368 (Valid-Until: 2069-05-13)
Done ヾ(@⌒ー⌒@)ノ
------------------------------------------

[Process completed]
Is it normal ?
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Match issue due to special characters when file name contains < and >

Post by rednoah »

1.
starsys wrote: 15 May 2019, 08:41 Nothing happens for more than 30 minutes........................

Is it normal ?
Not at all. No idea why it might get stuck at this point though, as not much is happening. I reckon that some file system operation (which is usually fairly instant) is somehow taking a very long time.

You could try running things via SSH yourself, to see if you can reproduce the issue. You can also use strace to track file system calls, which might help you figure out where exactly it's stuck:
https://linux.die.net/man/1/strace

You can also enable more logging in FileBot:
viewtopic.php?f=3&t=3913

:idea: But in this case my educated guess is definitely on blocking IO blocking things, so I gather it's a network filesystem issue rather than a filebot issue, so extra filebot logging might not help much. Worth a shot thing. See if gives us more fine-grained information as to what it's doing and where exactly it's stuck.


2.
starsys wrote: 15 May 2019, 07:44 I think this setting has to be applied from the server side (My Freebox) not from client (My NAS) ?
Yes, your local NAS probably only ever sees the mangled names, and has no way of knowing the unmangled names.
:idea: Please read the FAQ and How to Request Help.
starsys
Posts: 11
Joined: 13 May 2019, 12:46

Re: Match issue due to special characters when file name contains < and >

Post by starsys »

Can the CPU wrong detection (AMD instead of Intel) be an issue ?
I deleted my complete input folder and created it again (it seems it was full of corrupted hardlinks).
Now the log go a bit further but I still have an issue :

Code: Select all

Rename movies using [TheMovieDB]
Auto-detect movie from context: [/volume1/freebox/disque/Téléchargements/[ Torrent9.NZ ] La.Derniere.Folie.de.Claire.Darling.2018.FRENCH.1080p.WEB.H264-EXTREME.mkv]
CmdlineException: Failed to identify or process any files
Processing 1 unsorted files
[TEST] from [/volume1/freebox/disque/Téléchargements/[ Torrent9.NZ ] La.Derniere.Folie.de.Claire.Darling.2018.FRENCH.1080p.WEB.H264-EXTREME.mkv] to [/volume2/temp/Unsorted/freebox/disque/Téléchargements/[ Torrent9.NZ ] La.Derniere.Folie.de.Claire.Darling.2018.FRENCH.1080p.WEB.H264-EXTREME.mkv]
It seems no information is found for this movie whereas the windows version of filebot perfectly match and rename "Claire darling (2019)".
What is the search difference between CLI and windows version of Filebot ? Thanks.
User avatar
rednoah
The Source
Posts: 22923
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Match issue due to special characters when file name contains < and >

Post by rednoah »

1.
starsys wrote: 15 May 2019, 17:53 Can the CPU wrong detection (AMD instead of Intel) be an issue ?
The CPU architecture is correctly displayed as amd64 which is a CPU instruction set originally invented by AMD and now adopted by all modern Intel and AMD CPUs:
https://en.wikipedia.org/wiki/AMD64


2.
starsys wrote: 15 May 2019, 17:53 I deleted my complete input folder and created it again (it seems it was full of corrupted hardlinks).
You can have corrupted data. You can't have corrupted or broken hardlinks. The concept you think of as "file" is actually a hardlink, so files and hardlinks are quite literally exactly the same thing. ;)

:idea: You can have broken symlinks though.


3.
IDK. Works for me:

Code: Select all

[TEST] from [volume1/freebox/disque/Téléchargements/[ Torrent9.NZ ] La.Derniere.Folie.de.Claire.Darling.2018.FRENCH.1080p.WEB.H264-EXTREME.mp4] to [Movies/Claire Darling (2019)/Claire Darling (2019).mp4]
:idea: Make sure you're using the latest version. Other than that, only trial and error can help you figure out what's different.

:idea: Adding additional logging might help you figure things out: viewtopic.php?f=3&t=3913



:arrow: Please create new threads for new issues unrelated to the OP: viewtopic.php?f=3&t=1868
:idea: Please read the FAQ and How to Request Help.
starsys
Posts: 11
Joined: 13 May 2019, 12:46

Re: Match issue due to special characters when file name contains < and >

Post by starsys »

I changed nothing and today it works !
Really strange...
Post Reply