[Synology DS213j][FileBot 4.6.1] FileBot extraction speed

[qvazzler]

Hi RedNoah.

After your 4.6.1 update, I think a lot of Synology users were very happy to be able to rely on FileBot for rar extraction.
I'll be the first to thank you for this.. My auto-extract script was downloaded bare bones from a site, heavily modified by me, and still did not completely satisfy me due to lack of integration with FileBot.

A question remains though, based on what you stated below:

https://sourceforge.net/projects/filebo ... ebot/HEAD/

RAR is now supported (to a degree) by a pure-Java implementation. No support for password protected RARs though.

Assuming I can rely on this pure-Java implementation, I'd like to know if there is room for code efficiency, and if this is something you're looking to work on in the future.
The reason why I'm asking is because a 4gb rar archive seems to take about 30-40 minutes to extract on my Synology machine.

This will help me weigh down on whether I should start relying on your implementation (which I'd really much rather do) instead of my home-cooked auto-extracting scripts that uses unrar.

Thanks in advance!

[qvazzler]

As a brief update to this, it seems extraction speed has varied a bit for me.

Now I've extracted a 1.5gb file in less than 2 minutes..

Not sure if the issue is on my end..

[rednoah]

Most rar files we're talking about here, especially the multi-volume ones, aren't actually compressed. They just use rar as a container format, but use "Store" compressions, which means no compression at all.

Considering junrar has been abandoned many years ago, I'm surprised it works at all, especially for rar archives that actually use rar compression.

It'd be interesting to know why one archives take forever and another one works just fine though. Is it slow most of the time? Or only in rare cases? Because of compression level? Or maybe it just doesn't deal with large archives well, and what happens if you give the JVM more memory? These things are pretty much untested.

@see https://github.com/edmund-wagner/junrar

[qvazzler]

Hmm ok. Interesting indeed!

Any chance of being able enable switching to proprietary unrar?

[rednoah]

FileBot also supports extraction via the p7zip command-line tool, which can support rar archives. But the Synology p7zip build doesn't support the proprietary rar extensions. But if you could get a p7zip build that does support rar, then you could make it work with that.

[qvazzler]

Gotcha! I will stick with your implementation I think.

Are you still looking for Synology testers?

[rednoah]

Hardware-wise, not particularly, since I have a DS213J myself. But if you're a dev or generally good with testing I don't mind people mapping out various issues. For example, with the junrar thing I'd be quite interested in figuring out what works and what doesn't.

But you can always sign up here:
viewtopic.php?f=13&t=1165

[qvazzler]

You have a DS213j as well? Then maybe you just ran into the same issue as me.

I ran AMC on some directories with my "full-scan" method, but it runs out of memory.
I never had this issue when I had my python auto-extract script.. Maybe I'll have to start using it again..

I'm testing my way through what you said here.

You could try setting -Xmx100m (or lower or higher depending on what works) in the java call in filebot.sh to increase the default memory limit.

I'm an amateur python scripter so it was kinda sad to see this error not until after 3 nights of dabbling..

Edit: It seems like Java is only allowed around 100mb inherently on my DS213j. I assigned minimum and initial heap size to 256m, so far so good. The DSM Web GUI reports about 80% memory usage, from 50% before FileBot was started, so it looks like it's hanging in there. The script is processing a ton of backlog media right now, so in 6-12 hours I'll know for sure and update here.

Edit: Unfortunately, I'm still getting "Out of Memory"... Looks like I'll be reverting back to my old script...

[rednoah]

So you're getting Out of Memory problems immediately because you extract one big archive? If it can't even extract a single archive then it's a problem.

Or do you get Out of Memory, after some amount of time, because you extract & process & rename hundreds of archives and media files in one go?

FileBot (and especially amc) is not designed to run on everything at once, but rather in small bits, whenever a download finishes.

Have a shell script call filebot amc on one media folder at a time and see what happens.

[qvazzler]

It can manage for a while, but eventually it stops. Maybe after 3-6 things.. I didn't look closely enough.

Filebot used to run well before on massive folders like this, back when it did not extract anything, just moved and renamed media.

I'd like to avoid a shell script.

Maybe I can modify your groovy script to run my unrar script?

[rednoah]

1.
Yep. The junrar library is probably not well tested, and honestly it's surprising it works as well as it works. As long as it can deal with 1-2 archives per run I'll do for most people.

2.
Modifying the groovy script to call your own unrar tool is pretty easy. Though instead of modifying the script I'd recommend calling your own extract-all-archives-that-haven't-been-extracted-before script before the amc script.

A simple unrar call before the filebot/amc call might do the trick:
http://www.beginninglinux.com/home/data ... es-at-once

[qvazzler]

Yep, this is pretty much what my old script did. I'll have to work some more to reincorporate it.

[rednoah]

I ran a few tests with multi-gigabyte rar archive on my DS213J the other day:

7z-JBindings and junrar both took about 4 minutes, and unrar took 2 minutes. Not that much of a difference.

Can't reproduce the memory issues. Extracting a huge rar doesn't seem to cause it.

[qvazzler]

I ran a few tests with multi-gigabyte rar archive on my DS213J the other day:

7z-JBindings and junrar both took about 4 minutes, and unrar took 2 minutes. Not that much of a difference.

Can't reproduce the memory issues. Extracting a huge rar doesn't seem to cause it.

Very odd.. For me it refused. Then again my java might be slightly outdated?

I've gotten it to work using my old script now. Actually it was better to use a separate script because filebot doesn't know if the files belong to the torrent or not, so I devised a cleaning script that connects to transmission and removes any file (like extracted ones) that are not in the torrent.