Bad system encoding: ANSI_X3.4-1968

Support for Ubuntu and other Desktop Linux distributions
Post Reply
User avatar
rednoah
The Source
Posts: 22899
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Bad system encoding: ANSI_X3.4-1968

Post by rednoah »

If your environment is not correctly configured to use UTF-8 for encoding and decoding file paths, then filebot may fail to process non-ASCII file paths:

Code: Select all

Bad system encoding: ANSI_X3.4-1968

Code: Select all

* Please configure your locale with LANG=C.UTF-8 or LANG=en_US.UTF-8

Please configure your environment correctly before calling filebot:

Code: Select all

# force JVM language and encoding settings
export LANG="en_US.UTF-8"
export LC_ALL="en_US.UTF-8"

:idea: You may add these lines to your .profile to configure your login shell on login.
:idea: Please read the FAQ and How to Request Help.
cm2003
Posts: 19
Joined: 30 Oct 2018, 14:57

Re: Bad system encoding: ANSI_X3.4-1968

Post by cm2003 »

Hi.
I'm having this exact issue and I just can't fix this.
Everything seems to be fine, but filebot keeps giving me this error under debian 12. This is my locale output:

Code: Select all

root@vpn-filebot-nzbget:/mnt/media/Series# locale
LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=en_US.UTF-8
I'm running this under proxmox. Triggering filebot from nzbget using a script.
I've rebooted, reinstalled, echo commands, etc., but it just won't work :(
Result is that files with special characters are sorted into "Others" folder and won't be renamed - which is annoying :(

Any help would be very much appreciated!

Thanks

-edit-
Here also my sysinfo:

Code: Select all

root@vpn-filebot-nzbget:~# filebot -script fn:sysinfo
Low Disk Space (5.0 GB) on / (/dev/mapper/pve-vm--101--disk--0)
FileBot 5.1.2 (r10125)
JNA Native: 6.1.6
MediaInfo: 23.04
Tools: fpcalc/1.5.1 7z/16.02 unrar/0.1.3
Extended Attributes: OK
Unicode Filesystem: OK
Script Bundle: 2023-11-06 (r952)
Groovy: 4.0.15
JRE: OpenJDK Runtime Environment 17.0.10
JVM: OpenJDK 64-Bit Server VM
CPU/MEM: 4 Core / 8 GB Max Memory / 30 MB Used Memory
OS: Linux (amd64)
HW: Linux vpn-filebot-nzbget 6.5.11-8-pve #1 SMP PREEMPT_DYNAMIC PMX 6.5.11-8 (2024-01-30T12:27Z) x86_64 GNU/Linux
CPU/MEM: : Intel(R) N100 / MemTotal: 4.3 GB / MemFree: 3.8 GB / MemAvailable: 4.1 GB / SwapTotal: 536 MB / SwapFree: 536 MB
STORAGE: ext4 [/] @ 5.0 GB | zfs [/mnt/download] @ 3.9 TB | zfs [/mnt/media] @ 3.9 TB
UID/GID: uid=0(root) gid=0(root) groups=0(root)
DATA: /root/.filebot
Package: DEB
License: FileBot License PX6070926 (Valid-Until: 2068-11-01)
Done ヾ(@⌒ー⌒@)ノ
-edit2-
Just spun up Debian 11 (again using proxmox) as unpriviliged container - there it seems to be working after entering both echo commands. (actually have to enter them before using filebot for the first time)
User avatar
rednoah
The Source
Posts: 22899
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Bad system encoding: ANSI_X3.4-1968

Post by rednoah »

cm2003 wrote: 17 Feb 2024, 00:07

Code: Select all

Unicode Filesystem: OK
:idea: sysinfo output suggests no locale issues. (NOTE: assuming that printenv is the same for all filebot calls)

cm2003 wrote: 17 Feb 2024, 00:07 Result is that files with special characters are sorted into "Others" folder and won't be renamed - which is annoying :(
:arrow: Please post the complete console output of the filebot call that does not work, so that we can see warning messages and file paths at hand. (NOTE: Please read How to Request Help for details)
:idea: Please read the FAQ and How to Request Help.
cm2003
Posts: 19
Joined: 30 Oct 2018, 14:57

Re: Bad system encoding: ANSI_X3.4-1968

Post by cm2003 »

Good morning.

Thank you for your fast support.
I really don't know why... but I've turned off the entire proxmox system over night, turned it back on, was just about to provide you the output you were asking for, and it's working from the console window of proxmox...
However, if I run it from the nzbget script, the error comes up again :(

Here the two results.
Run from console:

Code: Select all

Low Disk Space (5.0 GB) on / (/dev/mapper/pve-vm--101--disk--0)
Run script [fn:amc] at [Sat Feb 17 08:21:40 UTC 2024]

[PSA] Important Discussion of Changes effective as of 28 Apr 2023:
https://www.filebot.net/forums/viewtopic.php?t=13406

Parameter: unsorted = y
Parameter: clean = y
Parameter: skipExtract = y
Parameter: seriesFormat = Series/{n}/{'Season '+s}/{n} - {s00e00} - {t}
Parameter: movieFormat = Movies/{ny}/{ny}{' CD'+pi}{subt}
Parameter: exec = chmod -R 777 /mnt/media && setfacl -m g:users:rwx {file}
Argument[0]: /mnt/download/completed
Ignore extra: /mnt/download/completed/TV-Series/SERIESNAME/Sample
Input: /mnt/download/completed/TV-Series/SERIESNAME/FILENAME.mkv
Group files by movie or series
Group: {Series=SERIESNAME (2007)} => [FILENAME.mkv]
Rename episodes using [TheMovieDB] with [Airdate Order]
Lookup via [SERIESNAME] []
Fetching episode data for [SERIESNAME]
* SERIESNAME [1418] | 279 episodes | 1x01-17 .. 2x01-23 .. 3x01-23 .. 4x01-24 .. 5x01-24 .. 6x01-24 .. 7x01-24 .. 8x01-24 .. 9x01-24 .. 10x01-24 .. 11x01-24 .. 12x01-24
[MOVE] from [/mnt/download/completed/TV-Series/SERIESNAME/FILENAME.mkv] to [/mnt/media/Series/SERIESNAME/SEASON/SERIESNAME - SEASONEPISODE - TITLE.mkv]
Processed 1 file
Execute: chmod -R 777 /mnt/media && setfacl -m g:users:rwx /mnt/media/Series/SERIESNAME/SEASON/SERIESNAME - SEASONEPISODE - TITLE.mkv
Clean clutter files and empty folders
Delete /mnt/download/completed/TV-Series/SERIESNAME/Sample/FILENAME-sample.mkv
Delete /mnt/download/completed/TV-Series/SERIESNAME/FILENAME.nfo
Delete /mnt/download/completed/TV-Series/SERIESNAME/Sample
Delete /mnt/download/completed/TV-Series/SERIESNAME
Delete /mnt/download/completed/TV-Series
Keep /mnt/download/completed (root folder)
Done ヾ(@⌒ー⌒@)ノ
and here when run from the nzbget script:

Code: Select all

Low Disk Space (5.0 GB) on / (/dev/mapper/pve-vm--101--disk--0)
Bad system encoding: ANSI_X3.4-1968
* Please configure your locale with LANG=C.UTF-8 or LANG=en_US.UTF-8
Run script [fn:amc] at [Sat Feb 17 08:23:24 UTC 2024]
[PSA] Important Discussion of Changes effective as of 28 Apr 2023:
https://www.filebot.net/forums/viewtopic.php?t=13406

Parameter: unsorted = y
Parameter: clean = y
Parameter: skipExtract = y
Parameter: seriesFormat = Series/{n}/{'Season '+s}/{n} - {s00e00} - {t}
Parameter: movieFormat = Movies/{ny}/{ny}{' CD'+pi}{subt}
Parameter: exec = chmod -R 777 /mnt/media && setfacl -m g:users:rwx {file}
Argument[0]: /mnt/download/completed
Ignore extra: /mnt/download/completed/TV-Series/SERIESNAME/Sample
Input: /mnt/download/completed/TV-Series/SERIESNAME/FILENAME.mkv
Group files by movie or series
Group: {Series=SERIESNAME (2007)} => [FILENAME.mkv]
Rename episodes using [TheMovieDB] with [Airdate Order]
Lookup via [SERIESNAME] []
Fetching episode data for [SERIESNAME]
* SERIESNAME [1418] | 279 episodes | 1x01-17 .. 2x01-23 .. 3x01-23 .. 4x01-24 .. 5x01-24 .. 6x01-24 .. 7x01-24 .. 8x01-24 .. 9x01-24 .. 10x01-24 .. 11x01-24 .. 12x01-24
Processed 0 files
No Such File: Malformed input or input contains unmappable characters: /mnt/media/Series/SERIESNAME/SEASON/SERIESNAME - SEASONEPISODE - TITLE.mkv
Processing 1 unsorted files
[MOVE] from [/mnt/download/completed/TV-Series/SERIESNAME/FILENAME.mkv] to [/mnt/media/Unsorted/SERIESNAME/FILENAME.mkv]
Processed 1 file
Execute: chmod -R 777 /mnt/media && setfacl -m g:users:rwx /mnt/media/Unsorted/SERIESNAME/FILENAME.mkv
Clean clutter files and empty folders
Delete /mnt/download/completed/TV-Series/SERIESNAME/Sample/FILENAME-sample.mkv
Delete /mnt/download/completed/TV-Series/SERIESNAME/FILENAME.nfo
Delete /mnt/download/completed/TV-Series/SERIESNAME/Sample
Delete /mnt/download/completed/TV-Series/SERIESNAME
Delete /mnt/download/completed/TV-Series
Keep /mnt/download/completed (root folder)
Done ヾ(@⌒ー⌒@)ノ
The nzbget script looks as follows:

Code: Select all

#!/bin/sh

#######################################
### NZBGET POST-PROCESSING SCRIPT   ###

# Trigger Filebot
#
# This is a script to trigger Filebot. It starts sorting,
# renaming and cleaning up the downloaded file.

### NZBGET POST-PROCESSING SCRIPT   ###
#######################################

filebot -script fn:amc /mnt/download/completed --output /mnt/media --action move -non-strict --order Airdate --conflict auto --lang de --def unsorted=y clean=y skipExtract=y "seriesFormat=Series/{n}/{'Season '+s}/{n} - {s00e00} - {t}" "movieFormat=Movies/{ny}/{ny}{' CD'+pi}{subt}" "exec=chmod -R 777 /mnt/media && setfacl -m g:users:rwx {file}" --log all --log-file /mnt/download/filebot.log

# NZBGet Exit Codes
if [ $? = 0 ]; then
	exit 93 # SUCCESS
	else
	exit 94 # FAILURE 
fi
PS: the filebot command is obviously exactly the same. In fact I copy paste from the script into the console.

-edit-
I think I've found a workaround.
I've just added
export LANG="en_US.UTF-8"
export LC_ALL="en_US.UTF-8"
to my nzbget script before the filebot trigger. This seems to do the trick :)
Also double checked that the script is UTF-8 encoded, but this didn't change anything (until I've added your two commands to the script itself).
User avatar
rednoah
The Source
Posts: 22899
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Bad system encoding: ANSI_X3.4-1968

Post by rednoah »

:idea: Yes, calling commands yourself and some service calling a command in some other context is not exactly the same. The user context and environment variables may be different. See Shell Script Debugging for Beginners for details.

:arrow: Fixing the environment variables in your script code before the filebot call is indeed the solution.
:idea: Please read the FAQ and How to Request Help.
Post Reply