Exponential delays when subtitle-searching multiple folders

[Ithiel]

Hi Red,

I'm trying out the subtitle downloader via the GUI and have noticed a strange issue...

If I select a single season folder of a show (i.e. \\mynas\media\TV Shows\Castle (2009 TV-PG)\Season 06), it loads the list of episodes, find/matches subtitles, and we're ready to download in a few seconds.

However, if I browse to \\mynas\media\TV Shows\Castle (2009 TV-PG)\ and select multiple folders (i.e. season 04 and season 05), it will take several minutes to find/match subtitles and be ready to download (an exponential increase).

Generally, I would expect that if 10 files in 1 folder takes 10 seconds, 20 files across 2 folders would take 20 seconds... but it taking 5+ minutes seems like it's getting stuck somewhere.

The more folders I add, the more exponential the increase - so when I browsed to \\mynas\media\TV Shows\ and selected a dozen or so different shows an hour ago, it's still loading :-/

Any ideas?


For reference, I'm running r1959, and file/folder selections were made via the file browser window as Win8.1 is still stopping the drag and drop from working in Filebot, lol

Edit: the multi show scan got to about 90min, and I gave up and cancelled it. :-/

[rednoah]

Mmhhh... that may potentially be the case since i started using my matching logic for file/subtitle matches as well. I guess I should do some per-folder clustering before throwing everything into the Matcher.

Try r1961.

[Ithiel]

It still seems to be taking its time (started a scan of 2x 1 season shows a few min ago, and it hasn't loaded yet), but this could be in part to having hit the anti-leech limit, and OpenSubtitles being all snotty at me for the next few hours, lol.

[rednoah]

Send me a list of file paths and I'll try myself.

[Ithiel]

lol, actually - looks like we're all good today, and that it was just OpenSubtitles being snotty at me for all my testing yesterday >.<

Edit: Just quickly though, I noticed that you also added an option to match subs with the file name (dropping the .eng) - thanks for that!

Is there a way to use that option from the cli/context menu for both individual files and folders?

I've currently got this for individual files:

Code: Select all

@="cmd /c filebot -get-subtitles \"%1\" -non-strict --lang en --log-file context.log"

And this for folders:

Code: Select all

@="cmd /c filebot -script fn:suball \"%1\" -non-strict --lang en --log-file context.log"

[rednoah]

Nope, for that would completely break the -get-missing-subtitles feature.

But for scripting it's not really important, if you can call one script to grab subs, you might as well call another to remove the language tags, if you really need to.

[Ithiel]

righto... I think I got it:

command (file)

Code: Select all

cmd /c "C:\Program Files\FileBot\cmdlets\get-subtitles-single-rename.cmd"

get-subtitles-single-rename.cmd

Code: Select all

filebot -get-subtitles "%1" -non-strict --lang en --log-file context.log
filebot -script fn:replace --filter "[.](srt|sub|ass)$" --def "e=^(.+)([.]\w+)([.]\w+)$" "r=$1$3" "%1"

command (folder)

Code: Select all

cmd /c "C:\Program Files\FileBot\cmdlets\get-subtitles-folder-rename.cmd"

get-subtitles-folder-rename.cmd

Code: Select all

filebot -script fn:suball "%1" -non-strict --lang en --log-file context.log
filebot -script fn:replace --filter "[.](srt|sub|ass)$" --def "e=^(.+)([.]\w+)([.]\w+)$" "r=$1$3" "%1"

edit: argh.. this method doesn't accept unc paths. I'll need to come at this another way, lol

[Ithiel]

hm.. I can get this to work without issue (download subtitle for single file):

Code: Select all

cmd /c filebot -get-subtitles "%1" -non-strict --lang en --log-file context.log

and I can get this to work just as well on the sub file (renaming sub to remove .eng):

Code: Select all

cmd /c filebot -script fn:replace --filter "[.](srt|sub|ass)$" --def "e=^(.+)([.]\w+)([.]\w+)$" "r=$1$3" "%1"

and if I am not using UNC paths, I can call a batch file that does both (per previous post)..

However, all my attempts at doing both within the single command have thus far failed:

Code: Select all

cmd /c filebot -get-subtitles "%1" -non-strict --lang en --log-file context.log && filebot -script fn:replace --filter "[.](srt|sub|ass)$" --def "e=^(.+)([.]\w+)([.]\w+)$" "r=$1$3" "%1"

(download works, but rename doesn't)

and I even tried this:

Code: Select all

cmd /c filebot -get-subtitles "%1" -non-strict --lang en --log-file context.log && "C:\Program Files\FileBot\cmdlets\get-subtitles-single.bat"

but alas, it doesn't seem to want to pass %1 after it was already used in the first command.

you sure there isn't a way to add a "-no-lang-tag" option (or whatever you want to call it) to the CLI without the world imploding? :-/

[rednoah]

It's probably easier if you make a single .bat file that calls make both of these filebot calls. This should be as easy as just calling two commands but cmd is always troublesome.

[Ithiel]

Hi Red,

I tried that (here: http://www.filebot.net/forums/viewtopic ... 7380#p7376), and it works, but only for local files - it doesn't work for UNC paths.

If I use cmd in the context menu command, UNC paths works, but if I use the context menu to call a batch file, UNC paths breaks everything.

The only way to get UNC to work is to have both the subtitle download and the rename within the content menu cmd command :-/

I've been trying to get the best of both worlds (UNC working through batch file), but it doesn't seem to want to play ball.. lol

[Ithiel]

While the above is still driving me crazy, I've flicked it back to just the sub download via the context menu cmd so I could run some tests..

When using the GUI to scan \\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\, it sits there for an eternity (20min and counting) and still nothing, however when I use my context menu option to download them via the CLI, it zips through the whole thing lightning fast.

File list is below:

Code: Select all

\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 04
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Special
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 02
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 01
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 03
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 04\Battlestar Galactica (2003) S04E01 He That Believeth In Me [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 04\Battlestar Galactica (2003) S04E05 Road Less Traveled, The [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 04\Battlestar Galactica (2003) S04E18 Islanded In A Stream Of Stars [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 04\Battlestar Galactica (2003) S04E06 Faith [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 04\Battlestar Galactica (2003) S04E08 Sine Qua Non [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 04\Battlestar Galactica (2003) S04E12 A Disquiet Follows My Soul [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 04\Battlestar Galactica (2003) S04E03 Ties That Bind, The [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 04\Battlestar Galactica (2003) S04E20 Daybreak (2) [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 04\Battlestar Galactica (2003) S04E14 Blood On The Scales [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 04\Battlestar Galactica (2003) S04E15 No Exit [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 04\Battlestar Galactica (2003) S04E02 Six Of One [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 04\Battlestar Galactica (2003) S04E19 Daybreak (1) [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 04\Battlestar Galactica (2003) S04E13 Oath, The [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 04\Battlestar Galactica (2003) S04E16 Deadlock [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 04\Battlestar Galactica (2003) S04E07 Guess What's Coming To Dinner [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 04\Battlestar Galactica (2003) S04E04 Escape Velocity [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 04\Battlestar Galactica (2003) S04E11 Sometimes A Great Notion [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 04\Battlestar Galactica (2003) S04E09 Hub, The [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 04\Battlestar Galactica (2003) S04E17 Someone To Watch Over Me [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Special\Battlestar Galactica (2003) S00E05 Razor (2) [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Special\Battlestar Galactica S00E01 Battlestar Galactica - The Miniseries [240p On2 MP3 2ch].nfo
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Special\Battlestar Galactica (2003) S00E22 Plan, The [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Special\Battlestar Galactica (2003) S00E04 Razor (1) [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Special\Battlestar Galactica (2003) S00E06 Revealed [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Special\Battlestar Galactica (2003) S00E06 Razor (3) [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Special\Battlestar Galactica S00E01 Battlestar Galactica - The Miniseries [240p On2 MP3 2ch].url
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Special\Battlestar Galactica (2003) S00E07 Phenomenon, The [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Special\Battlestar Galactica S00E01 Battlestar Galactica - The Miniseries [240p On2 MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 02\Battlestar Galactica (2003) S02E20 Lay Down Your Burdens (2) [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 02\Battlestar Galactica (2003) S02E09 Flight Of The Phoenix [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 02\Battlestar Galactica (2003) S02E17 Captain's Hand, The [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 02\Battlestar Galactica (2003) S02E16 Sacrifice [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 02\Battlestar Galactica (2003) S02E08 Final Cut [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 02\Battlestar Galactica (2003) S02E13 Epiphanies [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 02\Battlestar Galactica (2003) S02E02 Valley Of Darkness [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 02\Battlestar Galactica (2003) S02E06 Home (1) [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 02\Battlestar Galactica (2003) S02E12 Resurrection Ship (2) [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 02\Battlestar Galactica (2003) S02E04 Resistance [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 02\Battlestar Galactica (2003) S02E18 Downloaded [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 02\Battlestar Galactica (2003) S02E11 Resurrection Ship (1) [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 02\Battlestar Galactica (2003) S02E07 Home (2) [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 02\Battlestar Galactica (2003) S02E19 Lay Down Your Burdens (1) [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 02\Battlestar Galactica (2003) S02E14 Black Market [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 02\Battlestar Galactica (2003) S02E05 Farm, The [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 02\Battlestar Galactica (2003) S02E03 Fragged [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 02\Battlestar Galactica (2003) S02E15 Scar [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 02\Battlestar Galactica (2003) S02E10 Pegasus [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 02\Battlestar Galactica (2003) S02E01 Scattered [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 01\Battlestar Galactica (2003) S01E08 Flesh And Bone [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 01\Battlestar Galactica (2003) S01E02 Water [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 01\Battlestar Galactica (2003) S01E09 Tigh Me Up, Tigh Me Down [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 01\Battlestar Galactica (2003) S01E07 Six Degrees Of Separation [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 01\Battlestar Galactica (2003) S01E01 33 [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 01\Battlestar Galactica (2003) S01E04 Act Of Contrition [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 01\Battlestar Galactica (2003) S01E06 Litmus [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 01\Battlestar Galactica (2003) S01E05 You Can't Go Home Again [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 01\Battlestar Galactica (2003) S01E03 Bastille Day [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 01\Battlestar Galactica (2003) S01E13 Kobol's Last Gleaming (2) [480p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 01\Battlestar Galactica (2003) S01E10 Hand Of God, The [480p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 01\Battlestar Galactica (2003) S01E12 Kobol's Last Gleaming (1) [480p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 01\Battlestar Galactica (2003) S01E11 Colonial Day [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 03\Battlestar Galactica (2003) S03E13 Taking A Break From All Your Worries [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 03\Battlestar Galactica (2003) S03E12 Rapture (2) [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 03\Battlestar Galactica (2003) S03E09 Unfinished Business [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 03\Battlestar Galactica (2003) S03E14 Woman King, The [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 03\Battlestar Galactica (2003) S03E08 Hero [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 03\Battlestar Galactica (2003) S03E19 Crossroads (1) [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 03\Battlestar Galactica (2003) S03E04 Exodus (2) [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 03\Battlestar Galactica (2003) S03E10 Passage, The [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 03\Battlestar Galactica (2003) S03E15 A Day In The Life [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 03\Battlestar Galactica (2003) S03E02 Precipice [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 03\Battlestar Galactica (2003) S03E11 Eye Of Jupiter (1), The [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 03\Battlestar Galactica (2003) S03E16 Dirty Hands [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 03\Battlestar Galactica (2003) S03E18 Son Also Rises, The [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 03\Battlestar Galactica (2003) S03E03 Exodus (1) [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 03\Battlestar Galactica (2003) S03E07 A Measure Of Salvation (2) [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 03\Battlestar Galactica (2003) S03E05 Collaborators [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 03\Battlestar Galactica (2003) S03E20 Crossroads (2) [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 03\Battlestar Galactica (2003) S03E01 Occupation [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 03\Battlestar Galactica (2003) S03E17 Maelstrom [360p XviD MP3 2ch].avi
\\mynas\media\TV Shows\Battlestar Galactica (2003 TV-14)\Season 03\Battlestar Galactica (2003) S03E06 Torn (1) [360p XviD MP3 2ch].avi

Summary:
GUI allows subtitle downloading of files via UNC Path, supports removal of .eng tag, but has issues with multiple folders
Context menu calling Batch file allows subtitle downloading of multiple folders, supports removal of .eng tag via code referenced earlier, but has issues with UNC paths
Context menu calling cmd directly allows for subtitle downloading of multiple folders, allows for unc paths, but doesn't currently support removal of .eng tag (or at least I haven't figured out how to make it work yet, lol)

It's the age old story of "pick 2 out of 3", haha.

[rednoah]

Try r1966+

[Ithiel]

Righto - will test tmro.

Is this fixing the GUI multi folder scan issue or adding a -something option to the CLI to remove the language tag?

(I hit the leech limit, upgraded to VIP... and then hit the leech limit again, lol)

[rednoah]

I'm asking about that OpenSubtitles VIP thing right now:
http://forum.opensubtitles.org/viewtopi ... 11&t=14569

[Ithiel]

haha, it's all good.

Let me assure you, once I upgraded to VIP, I downloaded SIGNIFICANTLY more than 200 subtitles *cough* more like 20 times 200 *cough*

I think that out of everyone in the history of forever that hit the anti-leech limit, I probably deserved it the most, haha..

Of course, this does mean that I now need to run over 4000 files through the little .eng removal tool (in retrospect, I probably should have waited until after you "--format MATCH_VIDEO" before going at the subtitles download like a crazy person, but such is life... now I can test how many files remove-language-tag.cmd can handle, LOL!)

All the same, I am in the time-out bin until I get un-banhammered >.<

[rednoah]

4000 files, regex replace, how long can it take? 1 second? I can assure you that 99.99% of the time it runs will be spent waiting for the the remote filesystem to respond.

I think the main reason things were significantly slower on your side is because you're working with remote files. So even simple things like getting the file size will take time while it's almost instant for local files.

You can do the replace script with --action test first.

btw the osdb scripts are now:

Code: Select all

filebot -script dev:osdb.stats

Code: Select all

filebot -script dev:osdb.login

configure:login will be removed after the next update

[Ithiel]

configure:login will be removed after the next update

Oh, what's it being replaced with? (or how should I update things to log into OpenSubtitles.org?)

[rednoah]

It's the same as filebot -script fn:osdb.login now. With the newer jars that use GitHub.

[Ithiel]

ah, righto - thanks. I've updated my central installer accordingly.

Anyone running the live installer will automatically receive the updated setup file