Parameters and options for localized data

Running FileBot from the console, Groovy / FileBot scripting, shell scripts, etc
Post Reply
devster
Posts: 332
Joined: 06 Jun 2017, 22:56

Parameters and options for localized data

Post by devster » 24 Mar 2019, 09:08

I'm having some issues trying to integrate FileBot in a docker container.
Based on the rednoah/filebot image, linuxserver/transmission and others I stitched something together; I tried using the official image but it couldn't run as non-root user inside the container - almost required for file permissions.

I'm having some issues with the filebot.sh scripts though.
The way I set up the user running the torrent client, and therefore FileBot, is as a system user, without shell or a home directory, UIDs and GIDs get reassigned based on env variables (using s6-overlay).
Transmission config is in /transmission and FileBot data is supposed to be in /filebot, this is useful to have persistent volumes per-application.
The deb package, however, seems to require HOME to be set; it seems an easy fix, but assigning it would also move transmission config files which use XDG_CONFIG_HOME by default.
In short, which options are required and overridable to have everything FileBot needs to persist in a single directory (e.g. /filebot, not hidden), regardless of the user?
Below my attempt (unsuccessful as still tries to use HOME of the running user for some reason (4.8.5 from deb repository https://get.filebot.net/deb/), what did I miss?

Code: Select all

FILEBOT_OPTS=-Dapplication.deployment=docker \
	-Dapplication.dir=/filebot \
	-Dapplication.cache=/filebot/cache \ # persistent
	-Djava.io.tmpdir="/tmp/filebot \ # volatile
	-Dnet.filebot.util.prefs.file="/filebot/prefs.properties" \
	-Duser.home="/filebot"
	-Dnet.filebot.license="/filebot/license.txt" # is this a file or a directory?
and for reference the 2 files I'm using as basis:

Code: Select all

### DEB filebot.sh ###
FILEBOT_HOME="/usr/share/filebot"
JAVA_HOME="$FILEBOT_HOME/jre"

if [ -z "$HOME" ]; then
	echo '$HOME must be set' # WHY?
	exit 1
fi

# select application data folder
APP_DATA="$HOME/.filebot"
LIBRARY_PATH="$FILEBOT_HOME/lib"

"$JAVA_HOME/bin/java" \
    -Dapplication.deployment=deb \ # overridden
    -Dnet.filebot.AcoustID.fpcalc="$LIBRARY_PATH/fpcalc" \
    -Dunixfs=false \
    -DuseExtendedFileAttributes=true \
    -DuseCreationDate=false \
    -Djava.net.useSystemProxies=true \
    -Djna.nosys=true \
    -Djna.nounpack=true \
    [illegal-access and add-opens] \
    -Djna.boot.library.path="$LIBRARY_PATH" \
    -Djna.library.path="$LIBRARY_PATH" \
    -Djava.library.path="$LIBRARY_PATH" \
    -Dapplication.dir="$APP_DATA" \ # overridden (?)
    -Dapplication.cache="$APP_DATA/cache" \
    -Djava.io.tmpdir="$APP_DATA/tmp" \
    -Dfile.encoding="UTF-8" \
    -Dsun.jnu.encoding="UTF-8" \
    -Djdk.gtk.version=2 \
    -Dsun.java2d.xrender=true \
    -Dawt.useSystemAAFontSettings=on \
    -Dswing.aatext=true \
    -Dapplication.help=show \
    -Dnet.filebot.UserFiles.fileChooser=JavaFX \
    -DuseGVFS=true \
    -Dnet.filebot.gio.GVFS="$XDG_RUNTIME_DIR/gvfs" \
    $JAVA_OPTS \
    $FILEBOT_OPTS \
    -jar "$FILEBOT_HOME/jar/filebot.jar" "[email protected]"


### TAR filebot.sh ###
#!/bin/sh
PRG="$0"

# resolve relative symlinks
while [ -h "$PRG" ] ; do
	ls=`ls -ld "$PRG"`
	link=`expr "$ls" : '.*-> \(.*\)$'`
	if expr "$link" : '/.*' > /dev/null; then
		PRG="$link"
	else
		PRG="`dirname "$PRG"`/$link"
	fi
done

# get canonical path
PRG_DIR=`dirname "$PRG"`
FILEBOT_HOME=`cd "$PRG_DIR" && pwd` # should default to directory where filebot.sh is

# make sure required environment variables are set
if [ -z "$USER" ]; then
	export USER=`whoami`
fi

# add package lib folder to library path
PACKAGE_LIBRARY_ARCH="$(uname -s)-$(uname -m)"
PACKAGE_LIBRARY_PATH="$FILEBOT_HOME/lib/$PACKAGE_LIBRARY_ARCH"

# add fpcalc to the $PATH by default
export PATH="$PATH:$PACKAGE_LIBRARY_PATH"

# force JVM language and encoding settings
export LANG="en_US.UTF-8"
export LC_ALL="en_US.UTF-8"

# choose archive extractor / media characteristics parser
case $PACKAGE_LIBRARY_ARCH in
	Linux-x86_64|Linux-i686)
		# i686 or x86_64
		ARCHIVE_EXTRACTOR="SevenZipNativeBindings"
		MEDIA_PARSER="libmediainfo"
	;;
	*)
		# armv7l or aarch64
		ARCHIVE_EXTRACTOR="ShellExecutables"
		MEDIA_PARSER="ffprobe"
	;;
esac

# select application data folder
APP_DATA="$FILEBOT_HOME/data/$USER"
LIBRARY_PATH="$PACKAGE_LIBRARY_PATH:$LD_LIBRARY_PATH"

# start filebot
java \
    -Dapplication.deployment=tar  
    -Dnet.filebot.license="$FILEBOT_HOME/data/.license" \ # does this mean it's shared among users (?)
    -Dnet.filebot.media.parser="$MEDIA_PARSER" \
    -Dnet.filebot.archive.extractor="$ARCHIVE_EXTRACTOR" \
    -Dunixfs=false \
    -DuseExtendedFileAttributes=true \
    -DuseCreationDate=false \
    -Djava.net.useSystemProxies=true \
    -Djna.nosys=true \
    -Djna.nounpack=true \
    -Djna.boot.library.path="$LIBRARY_PATH" \
    -Djna.library.path="$LIBRARY_PATH" \
    -Djava.library.path="$LIBRARY_PATH" \
    -Dapplication.dir="$APP_DATA" \
    -Dapplication.cache="$APP_DATA/cache" \
    -Djava.io.tmpdir="$APP_DATA/tmp" \
    -Dfile.encoding="UTF-8" \
    -Dsun.jnu.encoding="UTF-8" \
    -Duser.home="$APP_DATA" \
    -Djava.util.prefs.PreferencesFactory=net.filebot.util.prefs.FilePreferencesFactory \
    -Dnet.filebot.util.prefs.file="$APP_DATA/prefs.properties" \
    $JAVA_OPTS \
    $FILEBOT_OPTS \
    -jar "$FILEBOT_HOME/jar/filebot.jar" "[email protected]"
I only work in black and sometimes very, very dark grey. (Batman)

User avatar
rednoah
The Source
Posts: 16546
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Parameters and options for localized data

Post by rednoah » 24 Mar 2019, 09:59

My docker file uses FILEBOT_OPTS to override the user.home Java System Property so that all state is stored to /data:
https://github.com/filebot/plugins/blob ... erfile#L24

Due to the sanity check in filebot.sh, and many many other reasons, $HOME should still be set to something reasonable, but as far as FileBot or Java code is concerned, user.home will take precedence and all application data will be stored there.
:idea: Please read the FAQ and How to Request Help.

devster
Posts: 332
Joined: 06 Jun 2017, 22:56

Re: Parameters and options for localized data

Post by devster » 24 Mar 2019, 10:18

Thank you, will fix it as suggested.
I only work in black and sometimes very, very dark grey. (Batman)

devster
Posts: 332
Joined: 06 Jun 2017, 22:56

Re: Parameters and options for localized data

Post by devster » 24 Mar 2019, 22:43

FileBot starts now and correctly runs, it also stores everything in /filebot, it seems some error is still getting thrown though.

Code: Select all

+ TR_APP_VERSION=2.94
+ TR_TIME_LOCALTIME='Sun Mar 24 22:29:13 2019'
+ TR_TORRENT_HASH=38a29d4f2f9bf149cc9d683c91ce29c84d3d105b
+ TR_TORRENT_ID=7
+ ARG_PATH='/downloads/tv_shows/<show>'
+ ARG_NAME='<show>'
+ ARG_LABEL=N/A
+ CONFIG_OUTPUT=/cephfs/Media
+ FILEBOT=/usr/bin/filebot
+ LINKS=($(find "$ARG_PATH" -type l))
++ find '/downloads/tv_shows/<show>' -type l
+ [[ 0 -gt 0 ]]
+ transmission-remote --auth <user>:<pass> -t 7 -S
localhost:9091/transmission/rpc/ responded: "success"
+ export JAVA_OPTS=-Xmx256M
+ JAVA_OPTS=-Xmx256M
+ [[ /downloads/tv_shows =~ ^/downloads/completed/(tv_shows|anime).* ]]
+ /usr/bin/filebot -script fn:amc --action keeplink --output /cephfs/Media --conflict skip --filter '!readLines("/filebot/scripts/movie_excludes.txt").contains(n)' --log-file amc.log --def subtitles=en artwork=y excludeList=.excludes 'ut_dir=/downloads/tv_shows/<show>' ut_kind=multi 'ut_title=<show>' ut_label=N/A 'exec=chmod 664 {quote file}' --def @/scripts/notify.txt --def [email protected]/scripts/movieFormat.groovy --def [email protected]/scripts/seriesFormat.groovy --def [email protected]/scripts/animeFormat.groovy
java.nio.file.AccessDeniedException: /"
java.nio.file.AccessDeniedException: /"
	at net.filebot.ApplicationFolder.<init>(ApplicationFolder.java:34)
	at net.filebot.ApplicationFolder.<clinit>(ApplicationFolder.java:21)
	at net.filebot.Main.initializeSystemProperties(Main.java:388)
	at net.filebot.Main.main(Main.java:106)

Locking /filebot/logs/amc.log
PWD seems the obvious culprit, this the output of fn:sysenv and fn:sysinfo:

Code: Select all

java.nio.file.AccessDeniedException: /"
java.nio.file.AccessDeniedException: /"
	at net.filebot.ApplicationFolder.<init>(ApplicationFolder.java:34)
	at net.filebot.ApplicationFolder.<clinit>(ApplicationFolder.java:21)
	at net.filebot.Main.initializeSystemProperties(Main.java:388)
	at net.filebot.Main.main(Main.java:106)

# Environment Variables #
PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
S6_SERVICES_GRACETIME: 0
PUID: 1000
TZ: Europe/London
TERM: xterm
LANG: C.UTF-8
FILEBOT_OPTS: -Dapplication.deployment=docker -Dapplication.dir=/filebot -Dapplication.cache=/filebot/cache -Djava.io.tmpdir="/tmp/filebot -Dnet.filebot.util.prefs.file=/filebot/prefs.properties -Duser.home=/filebot -Dnet.filebot.license=/filebot/license.txt
HOSTNAME: 401f91df22f0
PGID: 1000
<censored TR_AUTH var>
PWD: /
HOME: /var/empty


# Java System Properties #
awt.toolkit: sun.awt.X11.XToolkit
java.specification.version: 10
file.encoding.pkg: sun.io
sun.cpu.isalist:
sun.jnu.encoding: UTF-8
java.class.path: /usr/share/filebot/jar/filebot.jar
net.filebot.AcoustID.fpcalc: /usr/share/filebot/lib/fpcalc
java.vm.vendor: "Oracle Corporation"
grape.root: /filebot/grape
useExtendedFileAttributes: true
sun.arch.data.model: 64
jna.library.path: /usr/share/filebot/lib
swing.aatext: true
java.vendor.url: http://java.oracle.com/
user.timezone:
jna.nounpack: true
os.name: Linux
java.vm.specification.version: 10
sun.net.client.defaultReadTimeout: 60000
sun.java2d.xrender: true
jna.boot.library.path: /usr/share/filebot/lib
sun.java.launcher: SUN_STANDARD
sun.boot.library.path: /usr/share/filebot/jre/lib
sun.java.command: /usr/share/filebot/jar/filebot.jar -script fn:sysenv
jdk.debug: release
useGVFS: true
sun.cpu.endian: little
user.home: /filebot
user.language: en
java.specification.vendor: Oracle Corporation
java.version.date: 2018-07-17
java.home: /usr/share/filebot/jre
file.separator: /
java.vm.compressedOopsMode: Zero based
line.separator:

unixfs: false
useCreationDate: false
java.specification.name: Java Platform API Specification
java.vm.specification.vendor: Oracle Corporation
java.awt.graphicsenv: sun.awt.X11GraphicsEnvironment
application.cache: /filebot/cache
net.filebot.UserFiles.fileChooser: JavaFX
http.agent: FileBot/4.8.2
awt.useSystemAAFontSettings: on
sun.management.compiler: HotSpot 64-Bit Tiered Compilers
java.runtime.version: 10.0.2+13
user.name: abc
sun.net.client.defaultConnectTimeout: 10000
path.separator: :
java.net.useSystemProxies: true
application.help: show
net.filebot.license: /filebot/license.txt
os.version: 4.15.0-46-generic
jna.nosys: true
java.runtime.name: Java(TM) SE Runtime Environment
file.encoding: UTF-8
java.vm.name: Java HotSpot(TM) 64-Bit Server VM
java.vendor.version: 18.3
java.vendor.url.bug: http://bugreport.java.com/bugreport/
java.io.tmpdir: "/tmp/filebot
swing.crossplatformlaf: javax.swing.plaf.nimbus.NimbusLookAndFeel
java.version: 10.0.2
user.dir: /
os.arch: amd64
net.filebot.util.prefs.file: /filebot/prefs.properties
java.vm.specification.name: Java Virtual Machine Specification
net.filebot.gio.GVFS: /gvfs
java.awt.printerjob: sun.print.PSPrinterJob
sun.os.patch.level: unknown
application.deployment: docker
java.library.path: /usr/share/filebot/lib
java.vendor: Oracle Corporation
java.vm.info: mixed mode
java.vm.version: 10.0.2+13
jdk.gtk.version: 2
application.dir: /filebot
sun.io.unicode.encoding: UnicodeLittle
org.apache.commons.logging.Log: org.apache.commons.logging.impl.NoOpLog
java.class.version: 54.0


# Arguments #
args[0] = -script
args[1] = fn:sysenv


Done ヾ(@⌒ー⌒@)ノ

java.nio.file.AccessDeniedException: /"
java.nio.file.AccessDeniedException: /"
	at net.filebot.ApplicationFolder.<init>(ApplicationFolder.java:34)
	at net.filebot.ApplicationFolder.<clinit>(ApplicationFolder.java:21)
	at net.filebot.Main.initializeSystemProperties(Main.java:388)
	at net.filebot.Main.main(Main.java:106)

FileBot 4.8.2 (r5789)
JNA Native: 5.2.2
MediaInfo: 18.05
7-Zip-JBinding: 9.20
Chromaprint: 1.4.2
Extended Attributes: OK
Unicode Filesystem: OK
Script Bundle: 2019-02-26 (r552)
Groovy: 2.5.1
JRE: Java(TM) SE Runtime Environment 10.0.2
JVM: 64-bit Java HotSpot(TM) 64-Bit Server VM
CPU/MEM: 32 Core / 15 GB Max Memory / 39 MB Used Memory
OS: Linux (amd64)
HW: Linux 401f91df22f0 4.15.0-46-generic #49-Ubuntu SMP Wed Feb 6 09:33:07 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
DATA: /filebot
Package: DOCKER
License: FileBot License xxx (Valid-Until: xxx)
Done ヾ(@⌒ー⌒@)ノ
I only work in black and sometimes very, very dark grey. (Batman)

User avatar
rednoah
The Source
Posts: 16546
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Parameters and options for localized data

Post by rednoah » 25 Mar 2019, 05:03

1.
Your opening "... are never closed, so everything after that goes to shit after that:

Code: Select all

-Djava.io.tmpdir="/tmp/filebot
Which somehow results in you instructing FileBot to use a folder called " in your filesystem root / which FileBot fails to create of course:

Code: Select all

java.nio.file.AccessDeniedException: /"

2.
The easiest solution is often the best. Why not just take things out of the box, and then just add RUN ln -s $HOME/.filebot /filebot at the end of your docker file? This simple approach will work for all tools and you don't have to worry about FileBot implementation details.


3.
Out of curiosity, why are you so keen on not having a normal $HOME folder? No developer has ever tested their tools with /var/empty as $HOME older, so you're inviting strange unforeseeable issues. It doesn't have to be persistent, but it should at least be persistent for the current docker session.
:idea: Please read the FAQ and How to Request Help.

devster
Posts: 332
Joined: 06 Jun 2017, 22:56

Re: Parameters and options for localized data

Post by devster » 25 Mar 2019, 09:28

1. wow, apologies for the silly mistake

2. I'll see if I can do that, I was worried that during container creation HOME is /root and it'd link /root/.filebot to /filebot making it unwritable by a regular user.

3. fair enough, I was a bit biased by the fact that it's common to have a directory /var/empty to be used by certain daemons (e.g. sshd) but it's complicating stuff, I'll remove and follow convention, maybe even scrap everything and just add an exec=chown/chmod to the script.

Thanks again for the help.
I only work in black and sometimes very, very dark grey. (Batman)

User avatar
rednoah
The Source
Posts: 16546
Joined: 16 Nov 2011, 08:59
Location: Taipei
Contact:

Re: Parameters and options for localized data

Post by rednoah » 26 Mar 2019, 04:08

2.
That's a good point. I haven't tried it myself, so there might be road blocks if you take this path as well. Keep us updated!

3.
I read up on /var/empty and it makes sense as a place where you chroot / jail a untrusted peer, so they can't create write a super large file that fills up the drive and thus crashes the entire server. :lol:
:idea: Please read the FAQ and How to Request Help.

Post Reply