Filebot CLI (Linux) truncating subtitle downloads

[tlarcombe]

This is a problem that I have been working on troubleshooting for a few months - but finally decided to ask for help.

Firstly, I use Manjaro Linux - rolling release, all up to date.

Here are the filebot install details. It is licenced, but I have obscured the licence key

filebot -script fn:sysinfo
FileBot 4.9.1 (r7372) JDK8
JNA Native: 6.1.0
MediaInfo: 19.09
7-Zip-JBinding: 9.20
Chromaprint: 1.4.3
Extended Attributes: OK
Unicode Filesystem: OK
Script Bundle: 2020-04-20 (r634)
Groovy: 3.0.3
JRE: OpenJDK Runtime Environment 1.8.0_242
JVM: 64-bit OpenJDK 64-Bit Server VM
CPU/MEM: 4 Core / 1.8 GB Max Memory / 58 MB Used Memory
OS: Linux (amd64)
HW: Linux Hathi 5.6.7-1-MANJARO #1 SMP PREEMPT Thu Apr 23 10:50:31 UTC 2020 x86_64 GNU/Linux
STORAGE: ext4 [/] @ 92 GB
DATA: /home/tlarcombe/.config/filebot
Package: AUR
License: FileBot License PXXXXXXXX (Valid-Until: 2021-03-05)

Normally I don't use the GUI - the reason is, that because of something (which might be my desktop theme), when I click the match button to choose TheMovieDB, TheTVDB etc - the whole menu is blank. It doesn't worry me too much because I prefer to work with the CLI anyway.

Onto the problem:

When I download subtitles, the resulting .srt file is not complete. Example:

filebot -get-subtitles "The.Birds.1963.HDTV.x264.YIFY.mp4" --lang en --encoding utf8
Get [English] subtitles for 1 files
Looking up subtitles by hash via OpenSubtitles
Fetching [English] subtitles [The.Birds.1963.HDTV.x264.YIFY.en.srt] from [OpenSubtitles]
Export [The.Birds.1963.HDTV.x264.YIFY.en.srt] as [null / UTF-8]
Writing [The.Birds.1963.HDTV.x264.YIFY.en.srt] to [The.Birds.1963.HDTV.x264.YIFY.eng.srt]

It all seems to work ok, but the last few lines of the .srt file are:

1055
01:35:58,951 --> 01:36:00,409
- Where will we go?
- I don't know.

1056
01:36:00,577 --> 01:36:02,534
I think we're safe here
for the time being.

The file is 4877 lines long and finishes about 20mins short of the movie length.

As a comparison, I used the Filebot GUI to get the same subtitles. The end of this new file is:

1117
01:57:51,025 --> 01:57:55,354
- They haven't harmed anyone.
- Alright. Bring them.

1118
01:57:56,305 --> 01:58:02,832
Support us and become VIP member
to remove all ads from www.OpenSubtitles.org

Now 5158 lines.

This problem doesn't just affect movie subtitles, or English subtitles. It's not 'big' files - because I notice the same behaviour downloading subtitles for series episodes in English and Greek.

Often the file are truncated in the middle of subtitle line.

There doesn't seem to be any specific cut off point, except it loses about the last 15% of a file - so in a 1hr episode it will download maybe 50(ish) minutes. In a movie it will go way past the 50m mark - maybe losing the last 10-20 mins of the movie.

I can't figure out why GUI works, but CLI fails like this.

Thank you in advance for any guidance.

[rednoah]

1.
You can configure a different Look-and-Feel if the System default doesn't work well for you:
viewtopic.php?t=9827


2.
Do you have an OpenSubtitles link for that specific subtitle? Can you share the raw subtitle file via Google Drive?

What happens if you don't transcode the file?

Code: Select all

filebot -get-subtitles "The.Birds.1963.HDTV.x264.YIFY.mp4" --lang en

[tlarcombe]

Nice call. Well done. If I don't transcode the file to UTF8, the whole file downloads correctly.

This works for both English and Greek.

However, without transcoding to UTF8 the Greek subtitles are unreadable because the characters are incorrect.

Thank you for your help so far.

[tlarcombe]

The Java Swing look and feel is pretty cool - I learn something new every day.

It didn't solve the GUI issue though - with any of the four themes applied, clicking the 'match' button still drops down the menu (and the menu is working because as I trace the mouse down the menu the cursor changes) but it is just an empty box. I can screenshot it if you wish to see it.

Not only are the menu items there, but invisible, they are clickable and do the right functions. It just takes me time to remember how far down to click

[rednoah]

1.
I think there might a problem with the code path that transcodes raw text. Try using --output srt --encoding utf8 to rewrite the subtitle file from scratch:

Code: Select all

 --output srt --encoding utf8

2.
Screenshots are always appreciated. If it's a general problem, you might get lucky on Google, since this issue would likely affect all Java Swing applications, not just FileBot. My first try would be disabling any kind of graphics acceleration that might be enabled by default.

You could use Presets to work around those action popups altogether.

[tlarcombe]

Adding --output srt --encoding utf8 worked perfectly in both English and Greek.

Rednoah, you are truly a god amongst men! Thank you for all your help, your quick and detailed responses, and all round nice-guyishness.

T.

[rednoah]

* Note that I did find the underlying issue. There was an implicit buffer in the transcoder, which meant that my buffer sometimes didn't end up with the complete output since the transcode still had some of the data in it's own buffer and didn't flush it into mine.