InvalidPathException when dealing with files that have Chinese character in name

[virtualidacc]

Hi. I realised every single time if there is any Chinese character in the filename, filebot throws an InvalidPathException because of illegal character <?> (shown as ? because it cannot recognise it).

Code: Select all

Run script [fn:amc] at [Mon Jul 30 18:36:21 BST 2018]
Parameter: excludeList = amc.excludes
Parameter: unsorted = y
Parameter: music = y
Parameter: artwork = y
Parameter: movieFormat = Z:/*****/*****/{plex.name}
Parameter: seriesFormat = Z:/*****/*****/{plex.tail}
Parameter: animeFormat = Z:/*****/*****/{plex.tail}
Parameter: plex = *****
Parameter: minFileSize = 1
Parameter: minLengthMS = 1
Parameter: ut_label = Anime
Argument[0]: File does not exist: C:\Users\*****\*****\*****\*****\????????~.jpg
Use excludes: Z:\*****\*****\amc.excludes (5)
Illegal char <?> at index 82: C:\*****\*****\*****\*****\*****\????????~.jpg
java.nio.file.InvalidPathException: Illegal char <?> at index 82: C:\*****\*****\*****\*****\*****\????????~.jpg
        at net.filebot.util.FileSet.getPath(FileSet.java:90)
        at net.filebot.util.FileSet.contains(FileSet.java:82)
        at java_util_Set$contains$0.call(Unknown Source)
        at Script1.acceptFile(Script1.groovy:238)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at Script1$_run_closure47.doCall(Script1.groovy:301)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at Script1.run(Script1.groovy:301)
        at net.filebot.cli.ScriptShell.evaluate(ScriptShell.java:64)
        at net.filebot.cli.ScriptShell.runScript(ScriptShell.java:74)
        at net.filebot.cli.ArgumentProcessor.runScript(ArgumentProcessor.java:148)
        at net.filebot.cli.ArgumentProcessor.run(ArgumentProcessor.java:36)
        at net.filebot.Main.main(Main.java:126)

Failure (°_°)

Sysinfo:

Code: Select all

FileBot 4.8.2 (r5673)
JNA Native: 5.2.0
MediaInfo: 18.05
7-Zip-JBinding: 9.20
Chromaprint: 1.4.2
Extended Attributes: OK
Unicode Filesystem: OK
Script Bundle: 2018-07-15 (r530)
Groovy: 2.5.0
JRE: Java(TM) SE Runtime Environment 10.0.1
JVM: 64-bit Java HotSpot(TM) 64-Bit Server VM
CPU/MEM: 1 Core / 989 MB Max Memory / 19 MB Used Memory
OS: Windows 10 (amd64)
Package: MSI
License: FileBot License *****(Valid-Until: 2019-**-**)
Done ?(?????)?

I am very sure unicode is enabled. I tried to run with chcp 65001 by executing a batch file like following. But it did not work.

Code: Select all

chcp 65001
filebot %*

[rednoah]

1.
Not sure. It should work. Works for my test case anyway:

Code: Select all

PS C:\Test> filebot -rename 阿凡达2009.mp4 --db TheMovieDB
Rename movies using [TheMovieDB]
Auto-detect movie from context: [C:\Test\???2009.mp4]
[MOVE] from [C:\Test\???2009.mp4] to [C:\Test\Avatar (2009).mp4]
Processed 1 files

I'm using PowerShell. Can you try the same? Wouldn't be surprised if it's a CMD issue, because CMD has serious issues.


2.
Presumably, since you pass the argument directly, whatever gets passed in is already mangled, so the issue is probably upstream, and lies with the tool that is calling filebot:

Code: Select all

Argument[0]: File does not exist: C:\Users\*****\*****\*****\*****\????????~.jpg

If you're using --log-file then you can check the log file with your favorite UTF-8 / Unicode-enabled editor (e.g Sublime) and see what file path is passed in. PS and CMD don't display unicode character correctly, so you can't trust what you see in the PS / CMD window.

[virtualidacc]

1.
Not sure. It should work. Works for my test case anyway:

Code: Select all

PS C:\Test> filebot -rename 阿凡达2009.mp4 --db TheMovieDB
Rename movies using [TheMovieDB]
Auto-detect movie from context: [C:\Test\???2009.mp4]
[MOVE] from [C:\Test\???2009.mp4] to [C:\Test\Avatar (2009).mp4]
Processed 1 files

I'm using PowerShell. Can you try the same? Wouldn't be surprised if it's a CMD issue, because CMD has serious issues.


2.
Presumably, since you pass the argument directly, whatever gets passed in is already mangled, so the issue is probably upstream, and lies with the tool that is calling filebot:

Code: Select all

Argument[0]: File does not exist: C:\Users\*****\*****\*****\*****\????????~.jpg

If you're using --log-file then you can check the log file with your favorite UTF-8 / Unicode-enabled editor (e.g Sublime) and see what file path is passed in. PS and CMD don't display unicode character correctly, so you can't trust what you see in the PS / CMD window.

Thanks for the prompt reply! Are you using the same version (as I am using the latest beta version)? The script is called from a batch file and qbittorrent. The results are the same. Tried powershell and it is still the same. The character is not recognised properly and is just a question mark in log. The system display language is en-gb. However, if I use echo command to print any Chinese character and set the correct font, the system can display the character correctly. Therefore, I believe it is the JRE or the software that do not read in unicode.

[rednoah]

1.
Yes, I'm using the latest version.

Please post your raw --log-file on pastebin:
https://pastebin.com/

Make sure to select the log file. Please do not use copy & paste.

You can confirm if it's a Java / FileBot problem by running the test case I have posted in my last post yourself. It shows that Java / FileBot works with unicode arguments, at least in the context / environment I've tested.


2.
It clearly works on my machine. Have you tried the same? Does it not work on your machine?

Code: Select all

PS C:\Test> filebot -rename 阿凡达2009.mp4 --db TheMovieDB
Rename movies using [TheMovieDB]
Auto-detect movie from context: [C:\Test\???2009.mp4]
[MOVE] from [C:\Test\???2009.mp4] to [C:\Test\Avatar (2009).mp4]
Processed 1 files

Windows CMD is stupid and will show ? instead of the actual unicode character that is printed by FileBot. Checking the --log-file is the only way to confirm with what unicode strings FileBot is working internally.

[virtualidacc]

1.
Yes, I'm using the latest version.

Please post your raw --log-file on pastebin:
https://pastebin.com/

Make sure to select the log file. Please do not use copy & paste.

You can confirm if it's a Java / FileBot problem by running the test case I have posted in my last post yourself. It shows that Java / FileBot works with unicode arguments, at least in the context / environment I've tested.


2.
It clearly works on my machine. Have you tried the same? Does it not work on your machine?

Code: Select all

PS C:\Test> filebot -rename 阿凡达2009.mp4 --db TheMovieDB
Rename movies using [TheMovieDB]
Auto-detect movie from context: [C:\Test\???2009.mp4]
[MOVE] from [C:\Test\???2009.mp4] to [C:\Test\Avatar (2009).mp4]
Processed 1 files

Windows CMD is stupid and will show ? instead of the actual unicode character that is printed by FileBot. Checking the --log-file is the only way to confirm with what unicode strings FileBot is working internally.

Hi. Sorry for not replying for so long!

I do not think it is possible to upload files to pastebin.com. There is simply no such option or it is not obvious to guests. Still, I can try to paste it here. With any viewer, it is not possible to recognise the character.

I used vscode and pasted this to pastebin.com using chrome https://pastebin.com/3MNFhvCW

Is it because I am using windows 10 in English (I installed language package though)?

Code: Select all

Illegal Argument: java.nio.file.InvalidPathException: Illegal char <?> at index 10: E:\[YMDR][?????][Goblin Slayer][2018][04][720p][AVC][JAP][BIG5][MP4-AAC][??]\[YMDR][?????][Goblin Slayer][2018][04][720p][AVC][JAP][BIG5][MP4-AAC][??].mp4 (E:\[YMDR][?????][Goblin Slayer][2018][04][720p][AVC][JAP][BIG5][MP4-AAC][??]\[YMDR][?????][Goblin Slayer][2018][04][720p][AVC][JAP][BIG5][MP4-AAC][??].mp4)
Run script [fn:amc] at [Sun Oct 28 01:02:24 BST 2018]
Parameter: excludeList = amc.excludes
Parameter: unsorted = y
Parameter: music = y
Parameter: artwork = y
Parameter: movieFormat = Z:/My Drive/Movies/{plex.name}
Parameter: seriesFormat = Z:/My Drive/Series/{plex.tail}
Parameter: animeFormat = Z:/My Drive/Series/{plex.tail}
Parameter: minFileSize = 1
Parameter: minLengthMS = 1
Parameter: ut_label = Anime
Argument[0]: File does not exist: E:\[YMDR][?????][Goblin Slayer][2018][04][720p][AVC][JAP][BIG5][MP4-AAC][??]\[YMDR][?????][Goblin Slayer][2018][04][720p][AVC][JAP][BIG5][MP4-AAC][??].mp4
Use excludes: Z:\My Drive\dl\amc.excludes (120)
Illegal char <?> at index 10: E:\[YMDR][?????][Goblin Slayer][2018][04][720p][AVC][JAP][BIG5][MP4-AAC][??]\[YMDR][?????][Goblin Slayer][2018][04][720p][AVC][JAP][BIG5][MP4-AAC][??].mp4
java.nio.file.InvalidPathException: Illegal char <?> at index 10: E:\[YMDR][?????][Goblin Slayer][2018][04][720p][AVC][JAP][BIG5][MP4-AAC][??]\[YMDR][?????][Goblin Slayer][2018][04][720p][AVC][JAP][BIG5][MP4-AAC][??].mp4
	at net.filebot.util.FileSet.getPath(FileSet.java:90)
	at net.filebot.util.FileSet.contains(FileSet.java:82)
	at java_util_Set$contains$0.call(Unknown Source)
	at Script1.acceptFile(Script1.groovy:239)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at Script1$_run_closure47.doCall(Script1.groovy:302)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at Script1.run(Script1.groovy:302)
	at net.filebot.cli.ScriptShell.evaluate(ScriptShell.java:64)
	at net.filebot.cli.ScriptShell.runScript(ScriptShell.java:74)
	at net.filebot.cli.ArgumentProcessor.runScript(ArgumentProcessor.java:154)
	at net.filebot.cli.ArgumentProcessor.run(ArgumentProcessor.java:36)
	at net.filebot.Main.main(Main.java:131)

Failure (°_°)

[virtualidacc]

1.
Yes, I'm using the latest version.

Please post your raw --log-file on pastebin:
https://pastebin.com/

Make sure to select the log file. Please do not use copy & paste.

You can confirm if it's a Java / FileBot problem by running the test case I have posted in my last post yourself. It shows that Java / FileBot works with unicode arguments, at least in the context / environment I've tested.


2.
It clearly works on my machine. Have you tried the same? Does it not work on your machine?

Code: Select all

PS C:\Test> filebot -rename 阿凡达2009.mp4 --db TheMovieDB
Rename movies using [TheMovieDB]
Auto-detect movie from context: [C:\Test\???2009.mp4]
[MOVE] from [C:\Test\???2009.mp4] to [C:\Test\Avatar (2009).mp4]
Processed 1 files

Windows CMD is stupid and will show ? instead of the actual unicode character that is printed by FileBot. Checking the --log-file is the only way to confirm with what unicode strings FileBot is working internally.

I tried again and found it is a problem with qbittorrent. qbittorent does not pass arguments in unicode.

[rednoah]

Not quite sure what to do about that. Best to report this issue to the qBT guys and see what they say.