[AMC] Very Slow execution

Greg · Post by **Greg** » 03 Jul 2015, 20:21

Hello,

I use Filebot 4.6 on a Netgear ReadyNas 100. A shell script is called by qBittorrent after a download ended and is actually very very slow (about 20 minutes) while processing FileBot AMC script.

Please find bellow logs :

Code: Select all

2015-07-03 21:45:38 INFO ### Automated Media Integration v.0.3.0 ###
2015-07-03 21:45:38 INFO filePath = /data/Media/__A_RANGER/download_end/[ www.CpasBien.pw ] Under.the.Dome.S03E03.FASTSUB.VOSTFR.HDTV.XviD-ADDiCTiON.avi

2015-07-03 21:45:38 DEBUG filebot -version
FileBot 4.6 (r3052) / OpenJDK Runtime Environment 1.8.0_45-internal (headless)

2015-07-03 21:45:42 DEBUG filebot -clear-cache
Clear cache and temporary files

2015-07-03 21:45:51 DEBUG filebot -clear-pref
Reset preferences

2015-07-03 21:45:53 DEBUG filebot -script fn:sysinfo
FileBot 4.6 (r3052)
JNA Native: /usr/share/filebot/bin/temp/jna-92668751/jna5701918287452194698.tmp: /usr/share/filebot/bin/temp/jna-92668751/jna5701918287452194698.tmp: cannot open shared object file: No such file or directory
MediaInfo: java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
p7zip: Version 9.20 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,1 CPU)
chromaprint-tools: fpcalc version 1.1.0 (fpcalc)
Extended Attributes: OK
Groovy Engine: 2.4.3
JRE: OpenJDK Runtime Environment 1.8.0_45-internal (headless)
JVM: 32-bit OpenJDK Zero VM
CPU/MEM: 1 Core / 119 MB Max Memory / 13 MB Used Memory
OS: Linux (arm)
uname: Linux MyNAS 3.0.101.RN_ARM.4 #1 Wed Apr 1 16:06:27 PDT 2015 armv7l GNU/Linux
Done ヾ(＠⌒ー⌒＠)ノ

2015-07-03 21:47:05 INFO Search & rename /data/Media/__A_RANGER/download_end/[ www.CpasBien.pw ] Under.the.Dome.S03E03.FASTSUB.VOSTFR.HDTV.XviD-ADDiCTiON.avi [type=F] with Filebot
2015-07-03 21:47:05 DEBUG filebot -script fn:amc --action move --conflict override -non-strict --def "seriesFormat=/data/Media/Video/Serie/{n}/Saison {s}/{n} - {s}x{e.pad(2)} - {t}" --def "movieFormat=/data/Media/Video/Movie/{n} [{y}]/{n} [{y}] ({director} - {genres})" --def "clean=y" --def "excludeList=/data/Private/06-Divers/dev/shell/amc.txt" --def "xbmc=myraspberry" --def "minFileSize=50000000" "ut_kind=multi" "ut_dir=/data/Media/__A_RANGER/download_end/[ www.CpasBien.pw ] Under.the.Dome.S03E03.FASTSUB.VOSTFR.HDTV.XviD-ADDiCTiON.avi"  
Run script [fn:amc] at [Fri Jul 03 21:48:04 CEST 2015]
Parameter: seriesFormat = /data/Media/Video/Serie/{n}/Saison {s}/{n} - {s}x{e.pad(2)} - {t}
Parameter: movieFormat = /data/Media/Video/Movie/{n} [{y}]/{n} [{y}] ({director} - {genres})
Parameter: clean = y
Parameter: excludeList = /data/Private/06-Divers/dev/shell/amc.txt
Parameter: xbmc = myraspberry
Parameter: minFileSize = 50000000
Parameter: ut_kind = multi
Parameter: ut_dir = /data/Media/__A_RANGER/download_end/[ www.CpasBien.pw ] Under.the.Dome.S03E03.FASTSUB.VOSTFR.HDTV.XviD-ADDiCTiON.avi
Using excludes: /data/Private/06-Divers/dev/shell/amc.txt (1)
Input: /data/Media/__A_RANGER/download_end/[ www.CpasBien.pw ] Under.the.Dome.S03E03.FASTSUB.VOSTFR.HDTV.XviD-ADDiCTiON.avi
Group: [tvs:under the dome] => [[ www.CpasBien.pw ] Under.the.Dome.S03E03.FASTSUB.VOSTFR.HDTV.XviD-ADDiCTiON.avi]
Rename episodes using [TheTVDB]
Auto-detected query: [Under the Dome]
Fetching episode data for [Under the Dome]
[MOVE] Rename [/data/Media/__A_RANGER/download_end/[ www.CpasBien.pw ] Under.the.Dome.S03E03.FASTSUB.VOSTFR.HDTV.XviD-ADDiCTiON.avi] to [/data/Media/Video/Serie/Under the Dome/Saison 3/Under the Dome - 3x03 - Redux.avi]
Processed 1 files
Notify XBMC: myraspberry
Done ヾ(＠⌒ー⌒＠)ノ
2015-07-03 22:10:15 INFO END OK

The only exception I notice is in sysconfig (JNA Native: /usr/share/filebot/bin/temp/jna-92668751/jna5701918287452194698.tmp: /usr/share/filebot/bin/temp/jna-92668751/jna5701918287452194698.tmp: cannot open shared object file: No such file or directory
MediaInfo: java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native).

Thanks in advance for your help.

Regards

Greg

Post by **rednoah** » 04 Jul 2015, 07:06

1.
sysinfo?

@see viewtopic.php?f=4&t=1868

2.
On under powered ARM CPUs it's indeed VERY slow. Depends on the device, an how much memory you have. If you have less than 256-512 RAM it'll probably start swapping and that will make things slower by multiple magnitudes. But you can try Oracle Java 8 which should be a bit faster than the OpenJDK runtime.

Greg · Post by **Greg** » 04 Jul 2015, 10:20

Hello,

Thanks for your answer. Sorry if I'd not been clear. I decompose my logs :

Version of Filebot and Java :

Code: Select all

2015-07-03 21:45:38 DEBUG filebot -version
FileBot 4.6 (r3052) / OpenJDK Runtime Environment 1.8.0_45-internal (headless)

System Info

Code: Select all

2015-07-03 21:45:53 DEBUG filebot -script fn:sysinfo
FileBot 4.6 (r3052)
JNA Native: /usr/share/filebot/bin/temp/jna-92668751/jna5701918287452194698.tmp: /usr/share/filebot/bin/temp/jna-92668751/jna5701918287452194698.tmp: cannot open shared object file: No such file or directory
MediaInfo: java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
p7zip: Version 9.20 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,1 CPU)
chromaprint-tools: fpcalc version 1.1.0 (fpcalc)
Extended Attributes: OK
Groovy Engine: 2.4.3
JRE: OpenJDK Runtime Environment 1.8.0_45-internal (headless)
JVM: 32-bit OpenJDK Zero VM
CPU/MEM: 1 Core / 119 MB Max Memory / 13 MB Used Memory
OS: Linux (arm)
uname: Linux MyNAS 3.0.101.RN_ARM.4 #1 Wed Apr 1 16:06:27 PDT 2015 armv7l GNU/Linux
Done ヾ(＠⌒ー⌒＠)ノ

I notice I have a exception with JNA.
For information, I follow this tutorial [GUIDE] Filebot + MediaInfo + Raspberry Pi to install Filebot on my ReadyNas (ARM CPU)

Filebot Processing :

Code: Select all

2015-07-03 21:47:05 INFO Search & rename /data/Media/__A_RANGER/download_end/[ www.CpasBien.pw ] Under.the.Dome.S03E03.FASTSUB.VOSTFR.HDTV.XviD-ADDiCTiON.avi [type=F] with Filebot
2015-07-03 21:47:05 DEBUG filebot -script fn:amc --action move --conflict override -non-strict --def "seriesFormat=/data/Media/Video/Serie/{n}/Saison {s}/{n} - {s}x{e.pad(2)} - {t}" --def "movieFormat=/data/Media/Video/Movie/{n} [{y}]/{n} [{y}] ({director} - {genres})" --def "clean=y" --def "excludeList=/data/Private/06-Divers/dev/shell/amc.txt" --def "xbmc=myraspberry" --def "minFileSize=50000000" "ut_kind=multi" "ut_dir=/data/Media/__A_RANGER/download_end/[ www.CpasBien.pw ] Under.the.Dome.S03E03.FASTSUB.VOSTFR.HDTV.XviD-ADDiCTiON.avi"  
Run script [fn:amc] at [Fri Jul 03 21:48:04 CEST 2015]
Parameter: seriesFormat = /data/Media/Video/Serie/{n}/Saison {s}/{n} - {s}x{e.pad(2)} - {t}
Parameter: movieFormat = /data/Media/Video/Movie/{n} [{y}]/{n} [{y}] ({director} - {genres})
Parameter: clean = y
Parameter: excludeList = /data/Private/06-Divers/dev/shell/amc.txt
Parameter: xbmc = myraspberry
Parameter: minFileSize = 50000000
Parameter: ut_kind = multi
Parameter: ut_dir = /data/Media/__A_RANGER/download_end/[ www.CpasBien.pw ] Under.the.Dome.S03E03.FASTSUB.VOSTFR.HDTV.XviD-ADDiCTiON.avi
Using excludes: /data/Private/06-Divers/dev/shell/amc.txt (1)
Input: /data/Media/__A_RANGER/download_end/[ www.CpasBien.pw ] Under.the.Dome.S03E03.FASTSUB.VOSTFR.HDTV.XviD-ADDiCTiON.avi
Group: [tvs:under the dome] => [[ www.CpasBien.pw ] Under.the.Dome.S03E03.FASTSUB.VOSTFR.HDTV.XviD-ADDiCTiON.avi]
Rename episodes using [TheTVDB]
Auto-detected query: [Under the Dome]
Fetching episode data for [Under the Dome]
[MOVE] Rename [/data/Media/__A_RANGER/download_end/[ www.CpasBien.pw ] Under.the.Dome.S03E03.FASTSUB.VOSTFR.HDTV.XviD-ADDiCTiON.avi] to [/data/Media/Video/Serie/Under the Dome/Saison 3/Under the Dome - 3x03 - Redux.avi]
Processed 1 files
Notify XBMC: myraspberry
Done ヾ(＠⌒ー⌒＠)ノ
2015-07-03 22:10:15 INFO END OK

Please have a look on the dates (started at 21:47:05, ended at 22:10:15) so more than 23 minutes to find and rename the file.

Maybe you right, my machine should not have enought ressources :

Resource consumption while running FileBot :

I would like to know of it exists a WebService version of Filebot. I could passe as input the path of my file and filebot configuration (pattern, script, ...) and the WS returns the ordered filepath. So small machines (raspberrypi, nas, ...) could use Filebot in remote way (no resource consumption)

For example :
input :

Code: Select all

{
	script : "fn:amc",
	action : "move",
	conflict : "override",
	def : {
		seriesFormat: "/data/Media/Video/Serie/{n}/Saison {s}/{n} - {s}x{e.pad(2)} - {t}"
		movieFormat : "/data/Media/Video/Movie/{n} [{y}]/{n} [{y}] ({director} - {genres})"
		excludeList : "/data/Private/06-Divers/dev/shell/amc.txt"
	},
	filepath : "/data/Media/__A_RANGER/download_end/[ www.CpasBien.pw ] Under.the.Dome.S03E03.FASTSUB.VOSTFR.HDTV.XviD-ADDiCTiON.avi"
}

Output :

Code: Select all

{
	result : {
		source : "/data/Media/__A_RANGER/download_end/[ www.CpasBien.pw ] Under.the.Dome.S03E03.FASTSUB.VOSTFR.HDTV.XviD-ADDiCTiON.avi",
		target : "/data/Media/Video/Serie/Under the Dome/Saison 3/Under the Dome - 3x03 - Redux.avi",
		metadata : {
			database : "TheTVDB",
			tvShow : "Under the Dome"
		}
	}
}

This WebService could be deploy on a AWS machines.

Anyway, just an idea. If you're interesting, I could think about it.

Thanks again for you're answer.

Regards.

Grég

Post by **rednoah** » 04 Jul 2015, 11:22

1.
I'd start by using the Oracle Java 8 runtime:
http://www.oracle.com/technetwork/java/ ... 87472.html

2.
Since there's not much CPU usage it's either a Memory / Swap IO issue, or a network IO issue. I'd check network usage statistics to check when filebot is fetching data from from online services.

According to top you have 512 MB RAM? But the graph only shows 140 MB? My Synology DS213J has 512 MB and it works alright. 140 MB is definitely not enough though.

3.
There's no plan to run filebot as a service. But you can run everything on a remote server and then only rsync the organised file structure to your underpowered NAS. That's how people here seem to do it.

Here's a related thread. Though no solution:
viewtopic.php?f=8&t=2388

Greg · Post by **Greg** » 04 Jul 2015, 12:03

Hi,

1. Ok i'll try Oracle JDK 8 instead of open-jdk-8

2. The network seems OK :

You should right. It must be a memory/swap issue. It only has 23 MB free memory ...

3. You will laugh, but I only have unpowered machines at home (2 raspberryPi - 256 MB RAM and 1 NAS 512 MB RAM).
I try to make a remote solution using AWS machines.

Thanks for your help.

Regards

Grég

[AMC] Very Slow execution

[AMC] Very Slow execution

Re: [AMC] Very Slow execution

Re: [AMC] Very Slow execution

Re: [AMC] Very Slow execution

Re: [AMC] Very Slow execution