Single executable

[devster]

I'm opening this feature request for an easier way to run FileBot.
The new versions of Java seem to include a utility (jlink) to create a single executable file with everything included (app code and dependencies, an embedded and optimized JRE, and a launcher).
This would greatly simplify packaging and possibly also the current command to launch FileBot.

[rednoah]

jlink will not give you a single executable. It'll give you an application folder, with executables, jre files, application files, etc. Just like what we already have right now.


The benefit of jlink is that it can strip down the JRE and only include the required modules, reducing the size of the application folder.


jlink requires modularization, which means I'd have to drop support for Java 8, which means dropping support for 32-bit Windows and a couple of Linux devices.


jlink does not support cross-platform packaging, which means you need to run a build for each native platform on a native host for that platform, i.e. can't build Linux (aarch64) package on Linux (x64) machine, can't build Mac package on Windows machine, etc.


Check out GraalVM. That's the future. It already works well for compiling simple applications into a single executable. FileBot is far too complex, but my experiments are not not promising.

[devster]

My misunderstanding, I tried very rapidly and I saw the option to include launcher scripts.
It would still be really good (and the end goal IMHO) to have a single file, one of the main reasons I really like Go.

[rednoah]

My understanding is that we won't be able to get that far with FileBot. Since Groovy is dynamic code, it can't be pre-compiled, so GraalVM will always have to work in mixed mode, with some pre-compiled native code in the executable, but with additional bundled JRE to run Java byte code dynamically as necessary, e.g. any code that you pass in via formats, filters, mappers, etc.

[devster]

On a conceptual level, how is --format code different from say a templating engine like https://golang.org/pkg/text/template/ ?
I'm thinking about Hugo as an example.

[rednoah]

On a conceptual level it's the same. But rewriting Groovy to work within GraalVM and support dynamic code execution without a fallback JRE, that would be a task of monumental proportions.

[devster]

Your docker image seems to produce a single binary as output, and it seems to work.
--static and --no-fallback don't seem to work.

Code: Select all

bash-4.2# ./filebot -version
FileBot 4.8.6 (r6605) / OpenJDK Runtime Environment 1.8.0_222 / Linux 4.4.0-154-generic (amd64)
bash-4.2# ./filebot -list --q 'cannon busters' --db TheTVDB
Initialize new disk cache: /opt/filebot/data/root/cache/0
Aug 20, 2019 8:29:36 AM java.util.prefs.FileSystemPreferences$1 run
INFO: Created user preferences directory.
Cannon Busters - 1x01 - High Risk, Low Reward!
Cannon Busters - 1x02 - Grifters Gonna Grift
Cannon Busters - 1x03 - Watch Out For The Wet Spot
Cannon Busters - 1x04 - 9INE
Cannon Busters - 1x05 - 21 the Hard Way
Cannon Busters - 1x06 - Unfettered
Cannon Busters - 1x07 - Lady & The Kid
Cannon Busters - 1x08 - Turnbuckle Ex-Machina
Cannon Busters - 1x09 - Lullaby of The Stars
Cannon Busters - 1x10 - Squeaking Springs Afternoon
Cannon Busters - 1x11 - Innocence Lost Pt. 1
Cannon Busters - 1x12 - Innocence Lost Pt. 2
bash-4.2# file filebot
filebot: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=e95e7472844c9a0571afacefc0f0034f8424b488, not stripped

But something is wrong:

Code: Select all

bash-4.2# ./filebot -script fn:sysinfo
FileBot 4.8.6 (r6605)
JNA Native: 6.1.0
MediaInfo: net.filebot.mediainfo.MediaInfoException: Unable to load amd64 (64-bit) native library libmediainfo.so: Unable to load library 'zen':
/lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /opt/filebot/lib/Linux-x86_64/libzen.so)
/lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /opt/filebot/lib/Linux-x86_64/libzen.so)
Native library (linux-x86-64/libzen.so) not found in resource path ([file:/opt/filebot/jar/activation.jar, file:/opt/filebot/jar/ant-commons-net.jar, file:/opt/filebot/jar/ant-javamail.jar, file:/opt/filebot/jar/ant-jsch.jar, file:/opt/filebot/jar/ant-launcher.jar, file:/opt/filebot/jar/ant.jar, file:/opt/filebot/jar/args4j.jar, file:/opt/filebot/jar/autocomplete.jar, file:/opt/filebot/jar/bcpg.jar, file:/opt/filebot/jar/bcprov.jar, file:/opt/filebot/jar/bencode.jar, file:/opt/filebot/jar/caffeine.jar, file:/opt/filebot/jar/commons-io.jar, file:/opt/filebot/jar/commons-logging.jar, file:/opt/filebot/jar/commons-net.jar, file:/opt/filebot/jar/commons-vfs2.jar, file:/opt/filebot/jar/controlsfx.jar, file:/opt/filebot/jar/darcula.jar, file:/opt/filebot/jar/ehcache.jar, file:/opt/filebot/jar/failureaccess.jar, file:/opt/filebot/jar/glazedlists.jar, file:/opt/filebot/jar/groovy-ant.jar, file:/opt/filebot/jar/groovy-datetime.jar, file:/opt/filebot/jar/groovy-dateutil.jar, file:/opt/filebot/jar/groovy-json.jar, file:/opt/filebot/jar/groovy-jsr223.jar, file:/opt/filebot/jar/groovy-nio.jar, file:/opt/filebot/jar/groovy-swing.jar, file:/opt/filebot/jar/groovy-templates.jar, file:/opt/filebot/jar/groovy-xml.jar, file:/opt/filebot/jar/groovy.jar, file:/opt/filebot/jar/guava.jar, file:/opt/filebot/jar/icu4j.jar, file:/opt/filebot/jar/imgscalr.jar, file:/opt/filebot/jar/ivy.jar, file:/opt/filebot/jar/javax.mail.jar, file:/opt/filebot/jar/jaxb-api.jar, file:/opt/filebot/jar/jaxb-impl.jar, file:/opt/filebot/jar/jna-platform.jar, file:/opt/filebot/jar/jna.jar, file:/opt/filebot/jar/jsch.jar, file:/opt/filebot/jar/json-io.jar, file:/opt/filebot/jar/jsoup.jar, file:/opt/filebot/jar/junrar.jar, file:/opt/filebot/jar/language-detector.jar, file:/opt/filebot/jar/lanterna.jar, file:/opt/filebot/jar/metadata-extractor.jar, file:/opt/filebot/jar/miglayout-core.jar, file:/opt/filebot/jar/miglayout-swing.jar, file:/opt/filebot/jar/rsyntaxtextarea.jar, file:/opt/filebot/jar/sevenzipjbinding.jar, file:/opt/filebot/jar/simmetrics-core.jar, file:/opt/filebot/jar/slf4j-api.jar, file:/opt/filebot/jar/slf4j-jdk14.jar, file:/opt/filebot/jar/streamex.jar, file:/opt/filebot/jar/xmlrpc.jar, file:/opt/filebot/jar/xmpcore.jar, file:/opt/filebot/jar/xz.jar, file:/opt/filebot/jar/filebot.jar])
7-Zip-JBinding: net.sf.sevenzipjbinding.SevenZipNativeInitializationException: Failed to load 7z-JBinding: no 7-Zip-JBinding in java.library.path
Chromaprint: java.io.IOException: Cannot run program "fpcalc": error=2, No such file or directory
Extended Attributes: OK
Unicode Filesystem: OK
Script Bundle: 2019-08-18 (r584)
Groovy: 2.5.8
JRE: OpenJDK Runtime Environment 1.8.0_222
JVM: 64-bit OpenJDK 64-Bit GraalVM CE 19.1.1
CPU/MEM: 8 Core / 6 GB Max Memory / 66 MB Used Memory
OS: Linux (amd64)
HW: Linux 5b712ad5f1e7 4.4.0-154-generic #181-Ubuntu SMP Tue Jun 25 05:29:03 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
DATA: /opt/filebot/data/root
Package: TAR
License: UNREGISTERED
Done ヾ(＠⌒ー⌒＠)ノ

It seems related to libs missing and JNI (https://github.com/oracle/graal/blob/ma ... rface--jni) and (https://github.com/oracle/graal/blob/ma ... evm/JNI.md)

[rednoah]

It’s just a prototype. Feel free to play with it and see if you can make things work.

The GraalVM docker images aren’t based on Debian, so the native libraries don’t work, or maybe JNA just doesn’t work in GraalVM, most probably both.

This is a AOT executable does speed up startup, but still requires a full JRE to run.

[devster]

On a conceptual level it's the same. But rewriting Groovy to work within GraalVM and support dynamic code execution without a fallback JRE, that would be a task of monumental proportions.

Is this the same as the documented limit of GraalVM called dynamic class loading?

It’s just a prototype. The GraalVM docker images aren’t based on Debian, so the native libraries don’t work. Or maybe JNA just doesn’t work in GraalVM. Probably both.

This is a AOT executable does speed up startup, but still requires a full JRE to run.

I moved it to a new container without JRE before running, is the JRE embedded?

[rednoah]

Not sure. FileBot code should be alright, but all bets are off for any of its dependencies. Most notably, the Groovy script engine.

I’d be very surprised if it could run without a JRE. If you just do filebot -version then that specific code path might work without though.

[devster]

Running your container with: --no-fallback needs the following:

Code: Select all

java -agentlib:native-image-agent=config-output-dir=conf/ -cp "$FILEBOT_HOME/jar/filebot.jar" net.filebot.Main

to generate all config files required for the build, although it seems some classes such as Script1 are others including the $ sign, are identified as a standalone class, and need to be removed manually and most notably JNI seems empty.

• -H:ConfigurationFileDirectories=conf/ is associated to the above, and despite this, the following error still appears:

  Code: Select all

  WARNING: Could not register reflection metadata for net.sf.ehcache.transaction.manager.DefaultTransactionManagerLookup. Reason: java.lang.NoClassDefFoundError: javax/transaction/TransactionManager.
  

• --initialize-at-run-time=org.codehaus.groovy,net.sf.ehcache.store,sun.awt.X11,sun.awt.dnd and --initialize-at-build-time=sun.awt.X11.XToolkit
• graphics will also throw a long:

  Code: Select all

  WARNING: Exception during peer initialization
  java.awt.AWTError: Local GraphicsEnvironment must not be null
  ...
  

• --report-unsupported-elements-at-runtime and --allow-incomplete-classpath are needed to at least allow the compilation to finish, they don't seem enough for the build to succeed...
• Final error log is:

  Code: Select all

  bash-4.2# $FILEBOT_HOME/filebot.sh --verbose --no-server --no-fallback -H:ConfigurationFileDirectories=conf/ --initialize-at-run-time=org.codehaus.groovy --report-unsupported-elements-at-runtime --allow-incomplete-classpath --initialize-at-run-time=sun.awt.X11,sun.awt.dnd and --initialize-at-build-time=sun.awt.X11.XToolkit
  Executing [
  /opt/graalvm-ce-19.1.1/jre/bin/java \
  -XX:+UnlockExperimentalVMOptions \
  -XX:+EnableJVMCI \
  -Dtruffle.TrustAllTruffleRuntimeProviders=true \
  -Dtruffle.TruffleRuntime=com.oracle.truffle.api.impl.DefaultTruffleRuntime \
  -Dgraalvm.ForcePolyglotInvalid=true \
  -Dgraalvm.locatorDisabled=true \
  -d64 \
  -XX:-UseJVMCIClassLoader \
  -XX:+UseJVMCINativeLibrary \
  -Xss10m \
  -Xms1g \
  -Xmx13454016512 \
  -Duser.country=US \
  -Duser.language=en \
  -Dorg.graalvm.version=19.1.1 \
  -Dorg.graalvm.config=CE \
  -Dcom.oracle.graalvm.isaot=true \
  -Djvmci.class.path.append=/opt/graalvm-ce-19.1.1/jre/lib/jvmci/graal.jar \
  -Dapplication.deployment=tar \
  -Dnet.filebot.license=/opt/filebot/data/.license \
  -Dnet.filebot.media.parser=libmediainfo \
  -Dnet.filebot.archive.extractor=SevenZipNativeBindings \
  -Dunixfs=false \
  -DuseExtendedFileAttributes=true \
  -DuseCreationDate=false \
  -Dnative-image.net.useSystemProxies=true \
  -Djna.nosys=true \
  -Djna.nounpack=true \
  -Djna.boot.library.path=/opt/filebot/lib/Linux-x86_64:/opt/rh/llvm-toolset-7/root/usr/lib64 \
  -Djna.library.path=/opt/filebot/lib/Linux-x86_64:/opt/rh/llvm-toolset-7/root/usr/lib64 \
  -Dnative-image.library.path=/opt/filebot/lib/Linux-x86_64:/opt/rh/llvm-toolset-7/root/usr/lib64 \
  -Dapplication.dir=/opt/filebot/data/root \
  -Dapplication.cache=/opt/filebot/data/root/cache \
  -Dnative-image.io.tmpdir=/opt/filebot/data/root/tmp \
  -Dfile.encoding=UTF-8 \
  -Dsun.jnu.encoding=UTF-8 \
  -Duser.home=/opt/filebot/data/root \
  -Dnative-image.util.prefs.PreferencesFactory=net.filebot.util.prefs.FilePreferencesFactory \
  -Dnet.filebot.util.prefs.file=/opt/filebot/data/root/prefs.properties \
  -Xbootclasspath/a:/opt/graalvm-ce-19.1.1/jre/lib/boot/graal-sdk.jar:/opt/graalvm-ce-19.1.1/jre/lib/boot/graaljs-scriptengine.jar \
  -cp \
  /opt/graalvm-ce-19.1.1/jre/lib/svm/builder/llvm-platform-specific.jar:/opt/graalvm-ce-19.1.1/jre/lib/svm/builder/svm-llvm.jar:/opt/graalvm-ce-19.1.1/jre/lib/svm/builder/objectfile.jar:/opt/graalvm-ce-19.1.1/jre/lib/svm/builder/graal-llvm.jar:/opt/graalvm-ce-19.1.1/jre/lib/svm/builder/llvm-wrapper.jar:/opt/graalvm-ce-19.1.1/jre/lib/svm/builder/pointsto.jar:/opt/graalvm-ce-19.1.1/jre/lib/svm/builder/svm.jar:/opt/graalvm-ce-19.1.1/jre/lib/svm/builder/javacpp.jar:/opt/graalvm-ce-19.1.1/jre/lib/jvmci/jvmci-api.jar:/opt/graalvm-ce-19.1.1/jre/lib/jvmci/jvmci-hotspot.jar:/opt/graalvm-ce-19.1.1/jre/lib/jvmci/graal.jar:/opt/graalvm-ce-19.1.1/jre/lib/jvmci/graal-management.jar \
  com.oracle.svm.hosted.NativeImageGeneratorRunner \
  -watchpid \
  1652 \
  -imagecp \
  /opt/graalvm-ce-19.1.1/jre/lib/boot/graal-sdk.jar:/opt/graalvm-ce-19.1.1/jre/lib/boot/graaljs-scriptengine.jar:/opt/graalvm-ce-19.1.1/jre/lib/svm/builder/llvm-platform-specific.jar:/opt/graalvm-ce-19.1.1/jre/lib/svm/builder/svm-llvm.jar:/opt/graalvm-ce-19.1.1/jre/lib/svm/builder/objectfile.jar:/opt/graalvm-ce-19.1.1/jre/lib/svm/builder/graal-llvm.jar:/opt/graalvm-ce-19.1.1/jre/lib/svm/builder/llvm-wrapper.jar:/opt/graalvm-ce-19.1.1/jre/lib/svm/builder/pointsto.jar:/opt/graalvm-ce-19.1.1/jre/lib/svm/builder/svm.jar:/opt/graalvm-ce-19.1.1/jre/lib/svm/builder/javacpp.jar:/opt/graalvm-ce-19.1.1/jre/lib/jvmci/jvmci-api.jar:/opt/graalvm-ce-19.1.1/jre/lib/jvmci/jvmci-hotspot.jar:/opt/graalvm-ce-19.1.1/jre/lib/jvmci/graal.jar:/opt/graalvm-ce-19.1.1/jre/lib/jvmci/graal-management.jar:/opt/graalvm-ce-19.1.1/jre/lib/svm/library-support.jar:/opt/filebot/jar/activation.jar:/opt/filebot/jar/ant-commons-net.jar:/opt/filebot/jar/ant-javamail.jar:/opt/filebot/jar/ant-jsch.jar:/opt/filebot/jar/ant-launcher.jar:/opt/filebot/jar/ant.jar:/opt/filebot/jar/args4j.jar:/opt/filebot/jar/autocomplete.jar:/opt/filebot/jar/bcpg.jar:/opt/filebot/jar/bcprov.jar:/opt/filebot/jar/bencode.jar:/opt/filebot/jar/caffeine.jar:/opt/filebot/jar/commons-io.jar:/opt/filebot/jar/commons-logging.jar:/opt/filebot/jar/commons-net.jar:/opt/filebot/jar/commons-vfs2.jar:/opt/filebot/jar/controlsfx.jar:/opt/filebot/jar/darcula.jar:/opt/filebot/jar/ehcache.jar:/opt/filebot/jar/failureaccess.jar:/opt/filebot/jar/glazedlists.jar:/opt/filebot/jar/groovy-ant.jar:/opt/filebot/jar/groovy-datetime.jar:/opt/filebot/jar/groovy-dateutil.jar:/opt/filebot/jar/groovy-json.jar:/opt/filebot/jar/groovy-jsr223.jar:/opt/filebot/jar/groovy-nio.jar:/opt/filebot/jar/groovy-swing.jar:/opt/filebot/jar/groovy-templates.jar:/opt/filebot/jar/groovy-xml.jar:/opt/filebot/jar/groovy.jar:/opt/filebot/jar/guava.jar:/opt/filebot/jar/icu4j.jar:/opt/filebot/jar/imgscalr.jar:/opt/filebot/jar/ivy.jar:/opt/filebot/jar/javax.mail.jar:/opt/filebot/jar/jaxb-api.jar:/opt/filebot/jar/jaxb-impl.jar:/opt/filebot/jar/jna-platform.jar:/opt/filebot/jar/jna.jar:/opt/filebot/jar/jsch.jar:/opt/filebot/jar/json-io.jar:/opt/filebot/jar/jsoup.jar:/opt/filebot/jar/junrar.jar:/opt/filebot/jar/language-detector.jar:/opt/filebot/jar/lanterna.jar:/opt/filebot/jar/metadata-extractor.jar:/opt/filebot/jar/miglayout-core.jar:/opt/filebot/jar/miglayout-swing.jar:/opt/filebot/jar/rsyntaxtextarea.jar:/opt/filebot/jar/sevenzipjbinding.jar:/opt/filebot/jar/simmetrics-core.jar:/opt/filebot/jar/slf4j-api.jar:/opt/filebot/jar/slf4j-jdk14.jar:/opt/filebot/jar/streamex.jar:/opt/filebot/jar/xmlrpc.jar:/opt/filebot/jar/xmpcore.jar:/opt/filebot/jar/xz.jar:/opt/filebot/jar/filebot.jar \
  -H:Path=/opt/filebot \
  -H:Class=net.filebot.Main \
  -H:FallbackThreshold=0 \
  -H:ConfigurationFileDirectories=conf/ \
  -H:ClassInitialization=org.codehaus.groovy:run_time \
  -H:+ReportUnsupportedElementsAtRuntime \
  -H:+AllowIncompleteClasspath \
  -H:ClassInitialization=sun.awt.X11:run_time,sun.awt.dnd:run_time \
  -H:ClassInitialization=sun.awt.X11.XToolkit:build_time \
  -H:CLibraryPath=/opt/graalvm-ce-19.1.1/jre/lib/svm/clibraries/linux-amd64 \
  -H:Name=and
  ]
  [and:1669]    classlist:  15,414.59 ms
  [and:1669]        (cap):   2,143.53 ms
  [and:1669]        setup:   7,636.72 ms
  [and:1669]     analysis: 144,478.26 ms
  Fatal error: java.lang.NullPointerException
  	at sun.reflect.annotation.TypeAnnotationParser.mapTypeAnnotations(TypeAnnotationParser.java:356)
  	at sun.reflect.annotation.AnnotatedTypeFactory$AnnotatedTypeBaseImpl.<init>(AnnotatedTypeFactory.java:139)
  	at sun.reflect.annotation.AnnotatedTypeFactory$AnnotatedTypeVariableImpl.<init>(AnnotatedTypeFactory.java:222)
  	at sun.reflect.annotation.AnnotatedTypeFactory.buildAnnotatedType(AnnotatedTypeFactory.java:70)
  	at sun.reflect.annotation.TypeAnnotationParser.buildAnnotatedTypes(TypeAnnotationParser.java:137)
  	at java.lang.reflect.Executable.getAnnotatedParameterTypes(Executable.java:688)
  	at com.oracle.svm.reflect.target.Target_java_lang_reflect_Executable$AnnotatedParameterTypesComputer.compute(Target_java_lang_reflect_Executable.java:154)
  	at com.oracle.svm.hosted.substitute.ComputedValueField.readValue(ComputedValueField.java:264)
  	at com.oracle.svm.core.meta.ReadableJavaField.readFieldValue(ReadableJavaField.java:35)
  	at com.oracle.svm.hosted.ameta.AnalysisConstantReflectionProvider.readValue(AnalysisConstantReflectionProvider.java:100)
  	at com.oracle.svm.hosted.ameta.AnalysisConstantReflectionProvider.readFieldValue(AnalysisConstantReflectionProvider.java:75)
  	at com.oracle.graal.pointsto.ObjectScanner.scanField(ObjectScanner.java:175)
  	at com.oracle.graal.pointsto.ObjectScanner.doScan(ObjectScanner.java:344)
  	at com.oracle.graal.pointsto.ObjectScanner.access$400(ObjectScanner.java:62)
  	at com.oracle.graal.pointsto.ObjectScanner$3$1.run(ObjectScanner.java:414)
  	at com.oracle.graal.pointsto.util.CompletionExecutor.lambda$execute$0(CompletionExecutor.java:171)
  	at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402)
  	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
  	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
  	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
  	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
  Error: Image build request failed with exit status 1
  com.oracle.svm.driver.NativeImage$NativeImageError: Image build request failed with exit status 1
  	at com.oracle.svm.driver.NativeImage.showError(NativeImage.java:1405)
  	at com.oracle.svm.driver.NativeImage.build(NativeImage.java:1183)
  	at com.oracle.svm.driver.NativeImage.performBuild(NativeImage.java:1145)
  	at com.oracle.svm.driver.NativeImage.main(NativeImage.java:1104)
  

• anything that uses java.lang.ClassLoader will likely never be supported ("Dynamic class loading is not supported, and cannot be supported by our execution model" from https://github.com/oracle/graal/blob/ma ... TATIONS.md) from what I understand

On the bright side allowing a fallback returns a single executable, that kinda counts.