SFV Verification do not recognize special characters as á,é,í,ó,ú

[lemon389]

Scenario:
I create a *.svf file for some videos, then, I drag and drop this *.svf file into the SFV in the filebot, and, all the videos are checked based on their CRC.

Problem:
When there are files with 'special' characters, the *.svf file recognize them and store the name correctly, but, the SFV tool inside the filebot do not recognize those charcaters and change them with random chars, therefore, the tool do not compare this files against the *.svf tool as they 'do not exist'.

*.svf file example:
Pokémon/S01/Pokémon_S01E01_The Road Most Traveled!.mkv 2F2AC255

SVF tool in Filebot example:
Column Name: Pokémon/S01/Pokémon_S01E01_The Road Most Traveled!.mkv
Column *.svf: 2F2AC255
Column Folder: "File not found"


Question:
What can I do to avoid this situation with file names with characters as á,é,ó or others?

Thanks!

Format: Select all

FileBot 5.1.2 (r10125)
JNA Native: 6.1.6
MediaInfo: 23.10
7-Zip-JBinding: 16.02
Tools: fpcalc/1.5.0 mkvpropedit/80.0
Extended Attributes: OK
Unicode Filesystem: OK
Script Bundle: 2023-11-06 (r952)
Groovy: 4.0.15
JRE: OpenJDK Runtime Environment 17.0.8
JVM: OpenJDK 64-Bit Server VM
CPU/MEM: 12 Core / 8 GB Max Memory / 95 MB Used Memory
OS: Windows 11 (amd64)
STORAGE: NTFS [(C:)] @ 100 GB 
DATA: C:\Users\user\AppData\Roaming\FileBot
Package: MSI
License: FileBot License PX51386286 (Valid-Until: 2073-07-11)

[rednoah]

Can you provide a sample file? (please provide text as as binary file; copy & paste of text won't be useful for debugging text encoding issues; i.e. share via gdrive, dropbox, etc)


What tool / text encoding is used to generate the *.sfv files at hand?


FileBot generally expecteds UTF-8 encoded text files. Other UTF text encodings are supported if the file at hand uses the appropriate BOM marker. FileBot may also try to guess the text encoding which may not always be possible depending on the text encoding at hand. é vs é would suggest an issue with ISO-8859-1 being decoded as UTF-8 or vice versa.




EDIT:

Here is what I get with a sample *.sfv file generated by FileBot:

Code: Select all

; Generated by FileBot 5.1.2 on 2023-11-21 at 11:55:19
; charset=UTF-8
;
Pokémon_S01E01_The Road Most Traveled!.mkv 40E5A043

EDIT 2:

I've tried the sample *.sfv file above with common Windows text encodings for various locales and was not able to find one where encoding auto-detection does not work.


Please share the *.sfv file that you have via Google Drive / DropBox / etc. If you know, please also tell us which program was used to generate the *.sfv file you have and which text encoding was used.

[lemon389]

Hello,

Thanks for the quick response.

The *.sfv file are generated with Filebot 5.1.2, and, this situation happens very frequently.

Note:
I trimmed the *.sfv file for only one video, and, the é was recognized... but, when using the complete *.sfv export, the problem appeared again.
This version attached has the problem with "é", but, if you delete one more line, now the "é" is recognized correctly...

https://drive.google.com/file/d/1J__SrB ... sp=sharing

[rednoah]

You'll want to create one *.sfv file per series. Then it'll likely always work. Alternatively, you can add any UTF-8 character into the first 8000 bytes (e.g. Pokémon lines first; or add a character like ✔ to the comments on the first line) to nudge the charset detector towards the correct charset.



The issue is that charset detection only takes the first 8000 bytes into account, and those happen to be the same for UTF-8 and ISO-8859-1 and unfortunately the latter charset is then chosen to decode the entire UTF-8 encoded test file.

[lemon389]

That's unfortunate, but then I will split the ones with special chars, thanks for the support!

[rednoah]

Alternatively, you can add any UTF-8 character into the first 8000 bytes to nudge the charset detector towards the correct charset:

Code: Select all

; ✔
; Generated by FileBot 5.1.2 on 2023-11-20 at 13:04:50
; charset=UTF-8
;

[lemon389]

Ohhh got it, now is crystal clear, thanks a lot for your support!

[rednoah]

FileBot r10152 fixes charset auto-detection for UTF-8 encoded files generated by FileBot itself.