hearing impaired subtitle files (.HI) problems

beamer145 · Post by **beamer145** » 27 Jan 2013, 02:01

Filebot renaming (V3.3) seems to not like subtitle files which have a .HI tag (hearing impaired) in the name : both the group and the lang bindings give often wrong results.

I can understand it gets confused wrt to the lang binding. Though it should be possible to deal with a lot of cases by checking if another language tag is present right in front of the .hi , and if so then that is likely the real language (still gives problems for "Johnny English" etc, but maybe this is solvable as well, because it is known that that English is part of the title)

I don't get why the group gets mixed up though.

Eg with the format string {n}.{s00e00}.{t}.{vc}.{ac}{".$group"}{".$lang"} I get:
xxxxxxx.- Pilot.xvid.asap.English.HI => xxxxx.XviD.MP3.ASAP.hin (lang not ok)
xxxxxxx.FQM.English.HI.C.orig.Addic7ed.com => xxxxx.XviD.MP3.ASAP.eng (lang ok but group not ok)
xxxxxxx.HDTV.XviD-FQM.HI => xxxxx.XviD.MP3.ASAP.hin (lang not ok but he cannot know, group nok)
(the resulting wrong group always seems to be ASAP but that can be a coincidence)

Note that i started from a really messy folder, where (originally manually downloaded) subtitles and video files did not match.
After the rename step, sometimes they still did not match for the .HI files with problems (eg a different group for the subtitle file and for the video file)

beamer145 · Post by **beamer145** » 27 Jan 2013, 02:15

Mmm sometimes the vc tag of the subtitle file also differs from the video file. How is this tag determined for the subtitle file (I assumed it just took the corresponding tags from the video file, but apparently that is not (always) the case.....) ?

Post by **rednoah** » 27 Jan 2013, 02:45

I suppose .hi gets interpreted as Hindi. I can add a special case for that.

With the other things I need all your filename to test. Except for {lang} all the other bindings will be based on the corresponding video file. That assumes that video and subtitle file are at least named well enough to make that connection.

I guess the naming engine can't currently guess things like Alias.101.english.hi.srt matching Alias.S01E01.Title.avi

EDIT: Fixed with r1424. Hindi is now exempt from 2-letter language codes. The second one was extremely tricky but it should work as you're expecting now.

beamer145 · Post by **beamer145** » 27 Jan 2013, 15:03

I just tried r1424 and it seems to solve both problems, so I assume there is no longer a need for a list of example filenames which show the problem.
Thanks !!

beamer145 · Post by **beamer145** » 27 Jan 2013, 15:42

I just ran into a related minor issue with the r1424
When I have 2 versions of the subtitles file, eg a normal and a .HI version, they will both be transformed to the same filename.

So I get a conflict message when I select rename.

I can fix this by deleting one of the subtitle files before running filebot, but I was hoping for a different solution.
Probably I can do this automatically by checking for ".HI" myself with the groovy syntax (still need to look into that) and preserve the .HI indication if it is present.
But anyway, for now I tried to manually remove one version from the filelist from within filebot, and I wanted to do that by selecting the name in the output column with a mouseclick and then pressing delete on the keyboard.
Bad idea

:
- For some reason this resulted in the entry below the selected one to be removed (in this case the movie name)
- The group name of the file that should have been removed becomes wrong during the reparsing of all the files after/on the deletion point (BAJSKORV io LOL). This is btw the same incorrect group I got when using the original v3.3, so it is probably related somehow.
- The filenames behind the removed one the movie files are marked in red which probably indicates some sort of trouble.
- There is something strange going on with the matching of input to output elements, eg no empty slot in the output column for the removed element, but there is one for the last element. The output column gets shifted somehow.

Maybe it is also interesting to add tooltips to the buttons below the input and output lists, because when looking for a way to solve the conflict it is not really obvious what these are for.
Eg if I press the up button below the output list, he seems to reparse the filename, but all the movie names get marked in red (as if the reparsing fails the second time ??)

Edit: forgot to add the screenshots I made illustrating the delete problem (before and after pressing delete)
http://imgur.com/BmBUWCS
http://imgur.com/5q8Pc1q

Post by **rednoah** » 27 Jan 2013, 16:13

1. Yep, if you have hi and non-hi you'll have to take care of that in the naming scheme via {fn.match(/[.]hi$/)}

2. I don't understand what you mean. I tried a few things and all works. You can check via {media.filename} in the format what filebot things is the corresponding video file.

3. The Up-Down buttons? They just move entries up and down, and since drag-n-reorder is much more efficient for doing that kinda just the for decoration.

4. File<->Movie matches get highlighted in red? Are you sure it's not File<->Episode matches where SxE doesn't match?

5. Did you read the docs on {group}? It doesn't magically know all groups. It only knows the ones I added. Match preference goes like this: original filename > current filename > current foldername.
http://filebot.sourceforge.net/forums/v ... hp?f=5&t=4

Post by **rednoah** » 27 Jan 2013, 16:16

Yes, it's two stacks. If you delete something on one side then the other side is one short doesn't align anymore => One click on "Match" will align everything again.

beamer145 · Post by **beamer145** » 27 Jan 2013, 17:05

>Yes, it's two stacks.
Ah crap, with that in mind everything I am seeing makes perfect sense.

I thought pressing delete was suppose to affect both lists, as the fact that both lists have matching solid blue selections kind of insinuates that ( if only one is affected, it would be more intuitive if there was a visible difference in the way entries are selected. Eg the 'active' list could have a blue selection background, the inactive one a gray background for selected entries, like eg 2xExplorer ).

Anyway, sorry to have bothered you.
And thanks for the app !

Post by **rednoah** » 27 Jan 2013, 17:10

Fixed the other issue as well. Because of all that "C.hi.orig" it couldn't find the corresponding video file and then just defaulted to the first one.

beamer145 · Post by **beamer145** » 27 Jan 2013, 18:38

Ah, that explains the strange occurrence of always the same group for some files.

BTW maybe interesting for other ppl ending up in this thread: i added an extra match for the xxx.hi.xxx case (yours was only for xxx.hi)
{fn.match(/[.]HI(?=\.)/)}{fn.match(/[.]HI$/)}

( or my full string {n}.{s00e00}.{t}.{vc}.{ac}{".$group"}{".$lang"}{fn.match(/[.]HI(?=\.)/)}{fn.match(/[.]HI$/)} )

Note: maybe there are more optimal ways eg to merge them into one match operation, or to avoid the lookahead needed to not have a double '.' at the end, but I don't really know what is possible in the match operator's pattern argument, eg I am already surprised/happy he does not get confused about the double use of the round brackets '(' and ')' (inside the regexp and around the argument)

Post by **rednoah** » 28 Jan 2013, 00:43

Maybe like this:

Code: Select all

{fn.match(/\.HI\b/)}

beamer145 · Post by **beamer145** » 28 Jan 2013, 23:27

.HI\b will get confused for http://www.imdb.com/title/tt0132213/?ref_=fn_al_tt_5

(I think, now I am not sure if a - counts

)
And there is no way anyone can deal with http://www.imdb.com/title/tt1588694/?ref_=fn_al_tt_1

I just ran into a new unrelated problem
Eg Terra.Nova.S01E01.720p.HDTV.x264 is matched on the show NOVA instead of terra nova.
This happens both with tvrage as with thetvdb.
When searching for subtitles they were retrieved correctly though.
If I manually search for Terra Nova then I get the correct show, so apparently he is looking for Nova, or is giving that search result a higher probabilty then the match for Terra Nova.
I am still on r1424.

Post by **rednoah** » 29 Jan 2013, 04:43

1. Match for ".HI."

Code: Select all

{file.name.match(/[.]HI[.]/}

Actually I'd be more fun to auto-detect that from the subtitle content:

Code: Select all

{file.length() < 10e6 && {file.text.match(/(?m)^\(.+\)$/)} ? 'HI' : ''}

e.g. for files less than 10 MB, match subtitle text for (something) lines. I'm assuming that HI subtitles contain lines like (whistling) (singing a song) etc

2. Please do some more testing and tell me what might be throwing things off, cause it works for me:

Code: Select all

Parameter: ut_kind = single
Parameter: ut_dir = E:\testdata\utorrent-postprocess-test
Parameter: ut_file = Terra.Nova.S01E01.720p.HDTV.x264.avi
Parameter: ut_title = Deluge Test Folder
Input: E:\testdata\utorrent-postprocess-test\Terra.Nova.S01E01.720p.HDTV.x264.avi
Group: [tvs:Terra Nova] => [Terra.Nova.S01E01.720p.HDTV.x264.avi]
Rename episodes using [TheTVDB]
Auto-detected query: [Terra Nova, Nova]
Fetching episode data for [NOVA]
Fetching episode data for [Terra Nova]
Fetching episode data for [NOVA scienceNOW]
[TEST] Rename [E:\testdata\utorrent-postprocess-test\Terra.Nova.S01E01.720p.HDTV.x264.avi] to [E:\testdata\Deluge\output\TV Shows\Terra Nova\Season 1\Terra Nova - S01E01 - Genesis.avi]
Processed 1 files
Done ヾ(＠⌒ー⌒＠)ノ

beamer145 · Post by **beamer145** » 29 Jan 2013, 23:36

[1] {file.name.match(/[.]HI[.]/} : that fails for subtitles ending on .HI? That is why I had to split it up for the 2 cases. Also it returns .HI. giving you an extra . at the end (i think) ?
The second expression is a cool idea

If I open a few .HI subtitles is seems [] are the delimiters....but I guess there is not really a standard.
So if we go into full paranoia mode we get something like this ?

:
{file.length() < 10e6 && {file.text.match(/(?m)^[$\[].+[$\]]$/)} && ( {file.name.match(/[.]HI[.]/} || {file.name.match(/[.]HI$/} )? '.HI' : ''}

[2] How do I enable the debug output you added ?
I tried a few things from the CLI page but that did not seem to work....
filebot-3.3-r1424-revref.jar --log all => just starts the gui with nothing on the command line stdout (i immediately get a prompt again), and I cannot immediately find a log file
filebot-3.3-r1424-revref.jar --log all -rename D:\Series\TerraNova\test\ => no reaction as far as I can tell , I just go to the prompt again, file has not been renamed
filebot-3.3-r1424-revref.jar -help => also no reaction, just the prompt again

I also tried a few filename variations in the gui, both 'Terra.Nova' and 'Terra Nova' lead to the incorrect result, but 'TerraNova' returns the correct series.

Gui screenshot of the Terra.Nova result to prove that i am not entirely dreaming : http://imgur.com/UUBDRBU

Post by **rednoah** » 30 Jan 2013, 01:09

1. {fn} is name without extension, {file.name} however is with extension so there'll always be a .srt or something. The dot we need.

2. For cmdline I recommend replacing the jar in your install folder and then using my filebot startup script. If you installed the 'filebot' command will do that.

3. I'll have a look at terra nova. Please send me the full path of the file. And other files in the same folder.

EDIT:
E:\testdata\p3252\Terra.Nova.S01E01.720p.HDTV.x264.avi => WORKS

Can't reproduce the issue. What's the full path?

beamer145 · Post by **beamer145** » 30 Jan 2013, 11:16

[2] Mmmm, I did not use the installer, I just downloaded the jar file
filebot-3.3-r1424-revref.jar (at the moment, originally I was using the one linked from the home page)
and the libs
http://sourceforge.net/p/filebot/code/H ... win32-x64/
and put them together in a folder.

I don't have access to my home pc at the moment, but will give it a try later today to see if installing makes any difference, and check if I can get logging enabled.
Other possibility : is there maybe a different behavior in the libs between x64 and x86 ?

[3] For testing, I made a folder with just the single file
D:\Series\TerraNova\test\Terra.Nova.S01E01.720p.HDTV.x264.mkv

After starting the jar file, I think i drag/dropped either the file in the gui or else the 'test' folder, I am not sure anymore.

Post by **rednoah** » 30 Jan 2013, 11:54

The folder "TerraNova" parent folder messed things up due to missing space. Doesn't match "Terra Nova" while "NOVA" is a substring.

Fixed with r1435.

PS: In anycase I recommend using the platform package (e.g. the .msi) cause that includes optimized settings.

beamer145 · Post by **beamer145** » 30 Jan 2013, 19:32

>The folder "TerraNova" parent folder messed things up due to missing space. Doesn't match "Terra Nova" while "NOVA" is a substring.
Ah i was not aware the folder name was taken into account when doing the match. I am not a big fan of spaces in my names, especially not folder names so I camelcase everything.
That gave me loads of fun modifying the XMBC scrapers as well (especially since they do a ToLower() on all the input file/folder names before you arrive to the part where you can modify something in the add-on system.
Is there anything I can modify for filebot in the way he generates search querys from file/folder names, or is that in code (which is still modifyable of course). From the looks of the r1435 changelist, it seems to be in code.

Anyway, all files are good now except for the double episode:
Terra.Nova.S01E12E13.HDTV.XviD-LOL.avi

That is for some reason still mapped to a nova episode :
NOVA.S01E12-E13.Fusion: The Energy Of Promise & Te mystery Of The Anasazi

Full path:
D:\Series\TerraNova\Season01\Terra.Nova.S01E12E13.HDTV.XviD-LOL.avi

Post by **rednoah** » 31 Jan 2013, 01:13

Code: Select all

C:\>filebot -list --q "terra nova" --db thetvdb
Terra Nova - 1x01 - Genesis
Terra Nova - 1x02 - Instinct
Terra Nova - 1x03 - What Remains
Terra Nova - 1x04 - The Runaway
Terra Nova - 1x05 - Bylaw
Terra Nova - 1x06 - Nightfall
Terra Nova - 1x07 - Proof
Terra Nova - 1x08 - Vs.
Terra Nova - 1x09 - Now You See Me
Terra Nova - 1x10 - Within
Terra Nova - 1x11 - Occupation / Resistance

No 1x12 & 1x13. Can't work if your episode is invalid.

PS: Found some other Multi-Episode related issues that are fixed now.

beamer145 · Post by **beamer145** » 31 Jan 2013, 09:39

IMDB does list 13 episodes (http://www.imdb.com/title/tt1641349/episodes).

Ah, the reason is mentioned on the wikipedia page (http://en.wikipedia.org/wiki/Terra_Nova_%28TV_series%29) :
"Note: As first and last episodes are two hours, some senders cut them up to air as 13 episodes."

Now I am wondering what he will have done for the subtitles

I guess this one will require manual intervention.

Thanks for the support !

Post by **rednoah** » 31 Jan 2013, 10:44

TVRage lists it as 13 episodes. So if you match with TVRage data it should work.

beamer145 · Post by **beamer145** » 01 Feb 2013, 16:57

Sorry to bother you again, but I am bumping into something new.

I have files names:
Haven.S01E01.DVDRip.XviD-SAiNTS.avi
Haven.S01E01.DVDRip.XviD-SAiNTS.idx
Haven.S01E01.DVDRip.XviD-SAiNTS.sub
...

I don't know if .idx/.sub is supported by Filebot, since he tries to pick a language for the idx files I am assuming it is. If not, ignore the rest.

For some reason, he is really slow at processing the sub files when he does the final output name generation (but ok, that is not yet really an issue) + he thinks the .idx files are slovenian (or whatever the the .slo language code stands for) + the sub files are not tagged with the language (maybe because he does not associates them with the .idx and therefore does not know the language connecton ?) .
The idx files contain:
....
# Language index in use
langidx: 0

# English
id: en, index: 0
# Decomment next line to activate alternative name in DirectVobSub / Windows Media Player 6.x
# alt: English
....

So i am not really sure why he picks .slo (for regular srt files he always picked .eng when the language is not part of the filename)

Folder:
D:\Series\Haven\S01\*

Screenshot :
http://imgur.com/l46qyNK
(also shows the slow .sub renaming, it takes 10-20 seconds for each .sub file so plenty of time to take a screenshot)

Post by **rednoah** » 01 Feb 2013, 17:48

{lang} is based on the filename if possible but fall back to statistical language detection based on the text. I guess .idx mostly contains frame numbers, while .sub is binary.

Please send me a idx/sub pair and I'll have a look. I'll have to exclude VobSub subtitles from any normal subtitle processing.

EDIT: VobSub idx/sub pairs will be excluded from text-based processing with r1439

hearing impaired subtitle files (.HI) problems

hearing impaired subtitle files (.HI) problems

Re: hearing impaired subtitle files (.HI) problems

Re: hearing impaired subtitle files (.HI) problems

Re: hearing impaired subtitle files (.HI) problems

Re: hearing impaired subtitle files (.HI) problems

Re: hearing impaired subtitle files (.HI) problems

Re: hearing impaired subtitle files (.HI) problems

Re: hearing impaired subtitle files (.HI) problems

Re: hearing impaired subtitle files (.HI) problems

Re: hearing impaired subtitle files (.HI) problems

Re: hearing impaired subtitle files (.HI) problems

Re: hearing impaired subtitle files (.HI) problems

Re: hearing impaired subtitle files (.HI) problems

Re: hearing impaired subtitle files (.HI) problems

Re: hearing impaired subtitle files (.HI) problems

Re: hearing impaired subtitle files (.HI) problems

Re: hearing impaired subtitle files (.HI) problems

Re: hearing impaired subtitle files (.HI) problems

Re: hearing impaired subtitle files (.HI) problems

Re: hearing impaired subtitle files (.HI) problems

Re: hearing impaired subtitle files (.HI) problems

Re: hearing impaired subtitle files (.HI) problems

Re: hearing impaired subtitle files (.HI) problems