Using Fiddler to Troubleshoot Subtitle Download

[viking]

I am breaking out the partial discussion of using Fiddler to troubleshoot poor subtitle downloads from this topic: http://www.filebot.net/forums/viewtopic ... 446#p16446.

I am trying to determine how/why Filebot / OpenSubtitles sometimes picks a subtitle with Sync issues. Here is an example of downloading subtitles for:
Cosmos.A.Space.Time.Odyssey.S01E02.720p.HDTV.TommyJay.mkv

Using the GUI, I see 3 subtitles:
However, the first two have Sync issues while the third (DIMENSION) is fine.

Using the Command mode, the first (poor one; SWOLLED) is chosen:

Code: Select all

C:\MyProgs\Filebot>filebot -script fn:suball "P:\Ultimate_Server\Television\Cosmos- A Spacetime Odyssey\Season 1\test/" -non-strict --log all --log-file D:\testlog21.txt
Java HotSpot(TM) Client VM warning: TieredCompilation is disabled in this release.
Locking D:\test\log21.txt
Fetch subtitles for [P:\Ultimate_Server\Television\Cosmos- A Spacetime Odyssey\Season 1\test]
Get [English] subtitles for 1 files
Looking up subtitles by hash via OpenSubtitles
Matched [Cosmos.A.Space.Time.Odyssey.S01E02.720p.HDTV.TommyJay.mkv] to [Cosmos.A.Space.Time.Odyssey.S01E02.720p.HDTV.x264-SWOLLED-eng] via hash
Fetching [Cosmos.A.Space.Time.Odyssey.S01E02.720p.HDTV.x264-SWOLLED-eng.srt]
Writing [Cosmos.A.Space.Time.Odyssey.S01E02.720p.HDTV.x264-SWOLLED-eng.srt] to [Cosmos.A.Space.Time.Odyssey.S01E02.720p.HDTV.TommyJay.eng.srt]
Done pâ+(n+áGîÆpâ+GîÆn+á)pâÄ

I wanted to find out why the first one, in particular was chosen, so I used Fiddler. I saw the following communications:

Session #6 shows the request from Filebot to OpenSubtitles:
Filebot sends 1. Desired Language; 2. Hash (1961faa834f41e4c) and 3. Media Size. I am not sure how the media title is submitted? Maybe in the string "kmfrfldqisleb764ct6muakmd4"?? There is also another unknown parameter "limit" with value i4_2000. What is that?

The #6 response from OpenSubtitles was (the xml image would be too big so I am pasting the text instead):

Code: Select all

<?xml version="1.0" encoding="utf-8"?>
<methodResponse>
<params>
 <param>
  <value>
   <struct>
    <member>
     <name>status</name>
     <value>
      <string>200 OK</string>
     </value>
    </member>
    <member>
     <name>data</name>
     <value>
      <array>
       <data>
        <value>
         <struct>
          <member>
           <name>MatchedBy</name>
           <value>
            <string>moviehash</string>
           </value>
          </member>
          <member>
           <name>IDSubMovieFile</name>
           <value>
            <string>9308640</string>
           </value>
          </member>
          <member>
           <name>MovieHash</name>
           <value>
            <string>1961faa834f41e4c</string>
           </value>
          </member>
          <member>
           <name>MovieByteSize</name>
           <value>
            <string>1722075354</string>
           </value>
          </member>
          <member>
           <name>MovieTimeMS</name>
           <value>
            <string>0</string>
           </value>
          </member>
          <member>
           <name>IDSubtitleFile</name>
           <value>
            <string>1954173645</string>
           </value>
          </member>
          <member>
           <name>SubFileName</name>
           <value>
            <string>cosmos.a.space.time.odyssey.102.hdtv-lol.eng.srt</string>
           </value>
          </member>
          <member>
           <name>SubActualCD</name>
           <value>
            <string>1</string>
           </value>
          </member>
          <member>
           <name>SubSize</name>
           <value>
            <string>56701</string>
           </value>
          </member>
          <member>
           <name>SubHash</name>
           <value>
            <string>c4995d7aac5db80b9aca93e0be6b1056</string>
           </value>
          </member>
          <member>
           <name>SubLastTS</name>
           <value>
            <string>00:42:15</string>
           </value>
          </member>
          <member>
           <name>IDSubtitle</name>
           <value>
            <string>5587891</string>
           </value>
          </member>
          <member>
           <name>UserID</name>
           <value>
            <string>790019</string>
           </value>
          </member>
          <member>
           <name>SubLanguageID</name>
           <value>
            <string>eng</string>
           </value>
          </member>
          <member>
           <name>SubFormat</name>
           <value>
            <string>srt</string>
           </value>
          </member>
          <member>
           <name>SubSumCD</name>
           <value>
            <string>1</string>
           </value>
          </member>
          <member>
           <name>SubAuthorComment</name>
           <value>
            <string/>
           </value>
          </member>
          <member>
           <name>SubAddDate</name>
           <value>
            <string>2014-03-17 10:51:30</string>
           </value>
          </member>
          <member>
           <name>SubBad</name>
           <value>
            <string>0</string>
           </value>
          </member>
          <member>
           <name>SubRating</name>
           <value>
            <string>0.0</string>
           </value>
          </member>
          <member>
           <name>SubDownloadsCnt</name>
           <value>
            <string>31407</string>
           </value>
          </member>
          <member>
           <name>MovieReleaseName</name>
           <value>
            <string> cosmos.a.space.time.odyssey.102.hdtv-lol</string>
           </value>
          </member>
          <member>
           <name>MovieFPS</name>
           <value>
            <string>0.000</string>
           </value>
          </member>
          <member>
           <name>IDMovie</name>
           <value>
            <string>173630</string>
           </value>
          </member>
          <member>
           <name>IDMovieImdb</name>
           <value>
            <string>2987754</string>
           </value>
          </member>
          <member>
           <name>MovieName</name>
           <value>
            <string>"Cosmos: A Spacetime Odyssey" Some of the Things That Molecules Do</string>
           </value>
          </member>
          <member>
           <name>MovieNameEng</name>
           <value>
            <string/>
           </value>
          </member>
          <member>
           <name>MovieYear</name>
           <value>
            <string>2014</string>
           </value>
          </member>
          <member>
           <name>MovieImdbRating</name>
           <value>
            <string>9.2</string>
           </value>
          </member>
          <member>
           <name>SubFeatured</name>
           <value>
            <string>0</string>
           </value>
          </member>
          <member>
           <name>UserNickName</name>
           <value>
            <string>hippik</string>
           </value>
          </member>
          <member>
           <name>ISO639</name>
           <value>
            <string>en</string>
           </value>
          </member>
          <member>
           <name>LanguageName</name>
           <value>
            <string>English</string>
           </value>
          </member>
          <member>
           <name>SubComments</name>
           <value>
            <string>0</string>
           </value>
          </member>
          <member>
           <name>SubHearingImpaired</name>
           <value>
            <string>0</string>
           </value>
          </member>
          <member>
           <name>UserRank</name>
           <value>
            <string>gold member</string>
           </value>
          </member>
          <member>
           <name>SeriesSeason</name>
           <value>
            <string>1</string>
           </value>
          </member>
          <member>
           <name>SeriesEpisode</name>
           <value>
            <string>2</string>
           </value>
          </member>
          <member>
           <name>MovieKind</name>
           <value>
            <string>episode</string>
           </value>
          </member>
          <member>
           <name>SubHD</name>
           <value>
            <string>1</string>
           </value>
          </member>
          <member>
           <name>SeriesIMDBParent</name>
           <value>
            <string>2395695</string>
           </value>
          </member>
          <member>
           <name>SubEncoding</name>
           <value>
            <string>ASCII</string>
           </value>
          </member>
          <member>
           <name>SubDownloadLink</name>
           <value>
            <string>http://dl.opensubtitles.org/en/download/file/src-api/vrf-19b60c56/sid-kmfrfldqisleb764ct6muakmd4/1954173645.gz</string>
           </value>
          </member>
          <member>
           <name>ZipDownloadLink</name>
           <value>
            <string>http://dl.opensubtitles.org/en/download/sub/src-api/vrf-f5790bc4/sid-kmfrfldqisleb764ct6muakmd4/5587891</string>
           </value>
          </member>
          <member>
           <name>SubtitlesLink</name>
           <value>
            <string>http://www.opensubtitles.org/en/subtitles/5587891/sid-kmfrfldqisleb764ct6muakmd4/cosmos-a-spacetime-odyssey-some-of-the-things-that-molecules-do-en</string>
           </value>
          </member>
         </struct>
        </value>
        <value>
         <struct>
          <member>
           <name>MatchedBy</name>
           <value>
            <string>moviehash</string>
           </value>
          </member>
          <member>
           <name>IDSubMovieFile</name>
           <value>
            <string>9314984</string>
           </value>
          </member>
          <member>
           <name>MovieHash</name>
           <value>
            <string>1961faa834f41e4c</string>
           </value>
          </member>
          <member>
           <name>MovieByteSize</name>
           <value>
            <string>1722075354</string>
           </value>
          </member>
          <member>
           <name>MovieTimeMS</name>
           <value>
            <string>0</string>
           </value>
          </member>
          <member>
           <name>IDSubtitleFile</name>
           <value>
            <string>1954174855</string>
           </value>
          </member>
          <member>
           <name>SubFileName</name>
           <value>
            <string>Cosmos.A.Space.Time.Odyssey.S01E02.720p.HDTV.x264-SWOLLED-eng.srt</string>
           </value>
          </member>
          <member>
           <name>SubActualCD</name>
           <value>
            <string>1</string>
           </value>
          </member>
          <member>
           <name>SubSize</name>
           <value>
            <string>57494</string>
           </value>
          </member>
          <member>
           <name>SubHash</name>
           <value>
            <string>1f57567a0ea7445a22d8a1ff9650a4b7</string>
           </value>
          </member>
          <member>
           <name>SubLastTS</name>
           <value>
            <string>00:42:15</string>
           </value>
          </member>
          <member>
           <name>IDSubtitle</name>
           <value>
            <string>5589103</string>
           </value>
          </member>
          <member>
           <name>UserID</name>
           <value>
            <string>0</string>
           </value>
          </member>
          <member>
           <name>SubLanguageID</name>
           <value>
            <string>eng</string>
           </value>
          </member>
          <member>
           <name>SubFormat</name>
           <value>
            <string>srt</string>
           </value>
          </member>
          <member>
           <name>SubSumCD</name>
           <value>
            <string>1</string>
           </value>
          </member>
          <member>
           <name>SubAuthorComment</name>
           <value>
            <string/>
           </value>
          </member>
          <member>
           <name>SubAddDate</name>
           <value>
            <string>2014-03-18 05:13:28</string>
           </value>
          </member>
          <member>
           <name>SubBad</name>
           <value>
            <string>0</string>
           </value>
          </member>
          <member>
           <name>SubRating</name>
           <value>
            <string>0.0</string>
           </value>
          </member>
          <member>
           <name>SubDownloadsCnt</name>
           <value>
            <string>6599</string>
           </value>
          </member>
          <member>
           <name>MovieReleaseName</name>
           <value>
            <string> Cosmos.A.Space.Time.Odyssey.S01E02.720p.HDTV.x264-SWOLLED</string>
           </value>
          </member>
          <member>
           <name>MovieFPS</name>
           <value>
            <string>0.000</string>
           </value>
          </member>
          <member>
           <name>IDMovie</name>
           <value>
            <string>173630</string>
           </value>
          </member>
          <member>
           <name>IDMovieImdb</name>
           <value>
            <string>2987754</string>
           </value>
          </member>
          <member>
           <name>MovieName</name>
           <value>
            <string>"Cosmos: A Spacetime Odyssey" Some of the Things That Molecules Do</string>
           </value>
          </member>
          <member>
           <name>MovieNameEng</name>
           <value>
            <string/>
           </value>
          </member>
          <member>
           <name>MovieYear</name>
           <value>
            <string>2014</string>
           </value>
          </member>
          <member>
           <name>MovieImdbRating</name>
           <value>
            <string>9.2</string>
           </value>
          </member>
          <member>
           <name>SubFeatured</name>
           <value>
            <string>0</string>
           </value>
          </member>
          <member>
           <name>UserNickName</name>
           <value>
            <string/>
           </value>
          </member>
          <member>
           <name>ISO639</name>
           <value>
            <string>en</string>
           </value>
          </member>
          <member>
           <name>LanguageName</name>
           <value>
            <string>English</string>
           </value>
          </member>
          <member>
           <name>SubComments</name>
           <value>
            <string>0</string>
           </value>
          </member>
          <member>
           <name>SubHearingImpaired</name>
           <value>
            <string>0</string>
           </value>
          </member>
          <member>
           <name>UserRank</name>
           <value>
            <string/>
           </value>
          </member>
          <member>
           <name>SeriesSeason</name>
           <value>
            <string>1</string>
           </value>
          </member>
          <member>
           <name>SeriesEpisode</name>
           <value>
            <string>2</string>
           </value>
          </member>
          <member>
           <name>MovieKind</name>
           <value>
            <string>episode</string>
           </value>
          </member>
          <member>
           <name>SubHD</name>
           <value>
            <string>1</string>
           </value>
          </member>
          <member>
           <name>SeriesIMDBParent</name>
           <value>
            <string>2395695</string>
           </value>
          </member>
          <member>
           <name>SubEncoding</name>
           <value>
            <string>UTF-8</string>
           </value>
          </member>
          <member>
           <name>SubDownloadLink</name>
           <value>
            <string>http://dl.opensubtitles.org/en/download/file/src-api/vrf-19c20c5a/sid-kmfrfldqisleb764ct6muakmd4/1954174855.gz</string>
           </value>
          </member>
          <member>
           <name>ZipDownloadLink</name>
           <value>
            <string>http://dl.opensubtitles.org/en/download/sub/src-api/vrf-f55c0bb8/sid-kmfrfldqisleb764ct6muakmd4/5589103</string>
           </value>
          </member>
          <member>
           <name>SubtitlesLink</name>
           <value>
            <string>http://www.opensubtitles.org/en/subtitles/5589103/sid-kmfrfldqisleb764ct6muakmd4/cosmos-a-spacetime-odyssey-some-of-the-things-that-molecules-do-en</string>
           </value>
          </member>
         </struct>
        </value>
        <value>
         <struct>
          <member>
           <name>MatchedBy</name>
           <value>
            <string>moviehash</string>
           </value>
          </member>
          <member>
           <name>IDSubMovieFile</name>
           <value>
            <string>9318907</string>
           </value>
          </member>
          <member>
           <name>MovieHash</name>
           <value>
            <string>1961faa834f41e4c</string>
           </value>
          </member>
          <member>
           <name>MovieByteSize</name>
           <value>
            <string>1722075354</string>
           </value>
          </member>
          <member>
           <name>MovieTimeMS</name>
           <value>
            <string>0</string>
           </value>
          </member>
          <member>
           <name>IDSubtitleFile</name>
           <value>
            <string>1954174330</string>
           </value>
          </member>
          <member>
           <name>SubFileName</name>
           <value>
            <string>Cosmos.A.Space.Time.Odyssey.S01E02.PROPER.720p.HDTV.X264-DIMENSION-eng.srt</string>
           </value>
          </member>
          <member>
           <name>SubActualCD</name>
           <value>
            <string>1</string>
           </value>
          </member>
          <member>
           <name>SubSize</name>
           <value>
            <string>53522</string>
           </value>
          </member>
          <member>
           <name>SubHash</name>
           <value>
            <string>940c8e1cf55a6726ffe8ed296b387fd3</string>
           </value>
          </member>
          <member>
           <name>SubLastTS</name>
           <value>
            <string>00:42:15</string>
           </value>
          </member>
          <member>
           <name>IDSubtitle</name>
           <value>
            <string>5588572</string>
           </value>
          </member>
          <member>
           <name>UserID</name>
           <value>
            <string>0</string>
           </value>
          </member>
          <member>
           <name>SubLanguageID</name>
           <value>
            <string>eng</string>
           </value>
          </member>
          <member>
           <name>SubFormat</name>
           <value>
            <string>srt</string>
           </value>
          </member>
          <member>
           <name>SubSumCD</name>
           <value>
            <string>1</string>
           </value>
          </member>
          <member>
           <name>SubAuthorComment</name>
           <value>
            <string/>
           </value>
          </member>
          <member>
           <name>SubAddDate</name>
           <value>
            <string>2014-03-17 21:23:27</string>
           </value>
          </member>
          <member>
           <name>SubBad</name>
           <value>
            <string>0</string>
           </value>
          </member>
          <member>
           <name>SubRating</name>
           <value>
            <string>0.0</string>
           </value>
          </member>
          <member>
           <name>SubDownloadsCnt</name>
           <value>
            <string>11173</string>
           </value>
          </member>
          <member>
           <name>MovieReleaseName</name>
           <value>
            <string>Cosmos.A.Space.Time.Odyssey.S01E02.PROPER.720p.HDTV.X264-DIMENSION</string>
           </value>
          </member>
          <member>
           <name>MovieFPS</name>
           <value>
            <string>0.000</string>
           </value>
          </member>
          <member>
           <name>IDMovie</name>
           <value>
            <string>173630</string>
           </value>
          </member>
          <member>
           <name>IDMovieImdb</name>
           <value>
            <string>2987754</string>
           </value>
          </member>
          <member>
           <name>MovieName</name>
           <value>
            <string>"Cosmos: A Spacetime Odyssey" Some of the Things That Molecules Do</string>
           </value>
          </member>
          <member>
           <name>MovieNameEng</name>
           <value>
            <string/>
           </value>
          </member>
          <member>
           <name>MovieYear</name>
           <value>
            <string>2014</string>
           </value>
          </member>
          <member>
           <name>MovieImdbRating</name>
           <value>
            <string>9.2</string>
           </value>
          </member>
          <member>
           <name>SubFeatured</name>
           <value>
            <string>0</string>
           </value>
          </member>
          <member>
           <name>UserNickName</name>
           <value>
            <string/>
           </value>
          </member>
          <member>
           <name>ISO639</name>
           <value>
            <string>en</string>
           </value>
          </member>
          <member>
           <name>LanguageName</name>
           <value>
            <string>English</string>
           </value>
          </member>
          <member>
           <name>SubComments</name>
           <value>
            <string>0</string>
           </value>
          </member>
          <member>
           <name>SubHearingImpaired</name>
           <value>
            <string>0</string>
           </value>
          </member>
          <member>
           <name>UserRank</name>
           <value>
            <string/>
           </value>
          </member>
          <member>
           <name>SeriesSeason</name>
           <value>
            <string>1</string>
           </value>
          </member>
          <member>
           <name>SeriesEpisode</name>
           <value>
            <string>2</string>
           </value>
          </member>
          <member>
           <name>MovieKind</name>
           <value>
            <string>episode</string>
           </value>
          </member>
          <member>
           <name>SubHD</name>
           <value>
            <string>1</string>
           </value>
          </member>
          <member>
           <name>SeriesIMDBParent</name>
           <value>
            <string>2395695</string>
           </value>
          </member>
          <member>
           <name>SubEncoding</name>
           <value>
            <string>UTF-8</string>
           </value>
          </member>
          <member>
           <name>SubDownloadLink</name>
           <value>
            <string>http://dl.opensubtitles.org/en/download/file/src-api/vrf-19aa0c4e/sid-kmfrfldqisleb764ct6muakmd4/1954174330.gz</string>
           </value>
          </member>
          <member>
           <name>ZipDownloadLink</name>
           <value>
            <string>http://dl.opensubtitles.org/en/download/sub/src-api/vrf-f5710bc1/sid-kmfrfldqisleb764ct6muakmd4/5588572</string>
           </value>
          </member>
          <member>
           <name>SubtitlesLink</name>
           <value>
            <string>http://www.opensubtitles.org/en/subtitles/5588572/sid-kmfrfldqisleb764ct6muakmd4/cosmos-a-spacetime-odyssey-some-of-the-things-that-molecules-do-en</string>
           </value>
          </member>
         </struct>
        </value>
       </data>
      </array>
     </value>
    </member>
    <member>
     <name>seconds</name>
     <value>
      <double>0.011</double>
     </value>
    </member>
   </struct>
  </value>
 </param>
</params>
</methodResponse>

As can be seen (based on the hash) OpenSubtitles sends back the 3 subtitles shown in the GUI but in a different order (SWOLLED is 2nd instead of first):

Code: Select all

cosmos.a.space.time.odyssey.102.hdtv-lol.eng.srt
Cosmos.A.Space.Time.Odyssey.S01E02.720p.HDTV.x264-SWOLLED-eng.srt
Cosmos.A.Space.Time.Odyssey.S01E02.PROPER.720p.HDTV.X264-DIMENSION-eng.srt

The download path for the SWOLLED subtitle is

Code: Select all

http://dl.opensubtitles.org/en/download/file/src-api/vrf-19c20c5a/sid-kmfrfldqisleb764ct6muakmd4/1954174855.gz

Filebot next send out the download request for SWOLLED in session #10, and the subtitles for SWOLLED are downloaded.
Why does Filebot ask for this 2nd found (poorly Synced) subtitle rather than the 1st or the 3rd one?

[rednoah]

First of all, thanks for spending time on trying to debugging these things.

1.
If the the GUI shows a match with white background (not orange or red) then it means it's 100% accurate match, i.e. a hash match. That means somebody uploaded these subtitles for exactly that file, hence sync issues shouldn't be an issue, unless there's idiots out there that upload subtitles before checking if there in sync at all. Evidently there's some idiots out there...

2.
hash-lookup means that it will search for subtitles for exactly that file using a hash computed based on the file data, not by name. So if you have FILEHASH and FILESIZE you can lookup subtitles for that particular file.

If hash-lookup yields 0 results, then FileBot will fall back to search by movie/series name like a human would.

3.
"kmfrfldqisleb764ct6muakmd4" is your session ID, i.e. your API access token you get after calling LogIn. When you close FileBot it will call LogOut invalidating the access token, so posting this online shouldn't be an issue.

4.
i4 means 4-byte Integer and 2000 is the maximum number of results. Not sure if that even works since OpenSubtitles may limit results to 500 on the server-side anyway.

[rednoah]

Because of people uploading bad subtitles (or the rare cases of hash collisions) FileBot will try to pick the best match out of all the subtitle options it gets back from a hash-lookup (even though hash-lookup should only yield perfect matches in the first place).

Currently it will select the best match mostly based on the similarity between filename and subtitle name. SubBad/SubRating/SubDownloadsCnt don't seem to help much in finding the right one. Just using the first result doesn't help neither.

This is where things get tricky. Looking at the data the OpenSubtitles API sends back. How would you select the best subtitle? Is there anything that indicates that SWOLLED might be the best choice out of the 3 options?

* SubLastTS is new. Excluding subtitles that exceed the video length might help in some cases.
* Using SubAddDate to pick the latest one works in this case. May or may not work well in other cases.

Thoughts? Ideas?

[rednoah]

Here's a link for the bad subtitle:
http://www.opensubtitles.org/en/subtitl ... ules-do-en

* might be a good idea to penalize or completely ignore Anonymous subtitle uploads (extensive testing required) or at least prefer certain UserRanks

[viking]

Actually, SubAddDate was not a good parameter in this case as that would have have picked the poor SWOLLED. The good one, DIMENSION, was published 2nd.
UserRank is also not relevant as the only uploader with a rank (gold), for hdtv-lol, also had bad Sync.

Code: Select all

2014-03-17 10:51:30: cosmos.a.space.time.odyssey.102.hdtv-lol.eng.srt (bad Sync); UserRank = gold member
2014-03-18 05:13:28: Cosmos.A.Space.Time.Odyssey.S01E02.720p.HDTV.x264-SWOLLED-eng.srt (bad Sync); UserRank = NA
2014-03-17 21:23:27:  Cosmos.A.Space.Time.Odyssey.S01E02.PROPER.720p.HDTV.X264-DIMENSION-eng.srt (OK Sync); UserRank = NA

BTW, from your responses, it was not clear to me why Filebot picked SWOLLED out of the three possible hash matches?

[viking]

I just got some very strange results;
The file I used above was "Cosmos.A.Space.Time.Odyssey.S01E02.720p.HDTV.TommyJay.mkv".

I decided now to also download the file with the file name "Cosmos.A.Space.Time.Odyssey.S01E02.PROPER.720p.HDTV.X264-DIMENSION.mkv" (which had a corresponding good subtitle).
Using the GUI, Filebot finds the same subtitles as for the original file (as expected; albeit in a different order):

Code: Select all

Cosmos.A.Space.Time.Odyssey.S01E02.PROPER.720p.HDTV.X264-DIMENSION-eng.srt
cosmos.a.space.time.odyssey.102.hdtv-lol.eng.srt
Cosmos.A.Space.Time.Odyssey.S01E02.720p.HDTV.x264-SWOLLED-eng.srt

In this case, and using the command prompt, Filebot downloads the previously good subtitle (Cosmos.A.Space.Time.Odyssey.S01E02.PROPER.720p.HDTV.X264-DIMENSION-eng.srt).
Why does Filebot pick the DIMENSION rather than SWOLLED in this case? Maybe because the names match?

Here comes the strange parts:
Strange #1. The two media files have different hash and file sizes!

Code: Select all

Cosmos.A.Space.Time.Odyssey.S01E02.720p.HDTV.TommyJay.mkv: 1722075354 bytes
Cosmos.A.Space.Time.Odyssey.S01E02.PROPER.720p.HDTV.X264-DIMENSION.mkv: 1346233417 bytes 

How can two different files find the same hash matched subtitles!?

Strange #2. When watching Cosmos.A.Space.Time.Odyssey.S01E02.PROPER.720p.HDTV.X264-DIMENSION.mkv, any of the 3 downloaded subtitles are OK without any Sync issues!
Why does one media file only work with one of the subtitles, while the other works with all three?


p.s. When calculating the hash, what type of checksum is being used, e.g. MD5, SHA-1 etc...?

[rednoah]

How does FileBot pick the "best" subtitle option?

Selecting the best option is based on:
* Matching episode numbers
* Matching original filename (via xattr metadata)
* NameSubstringSequenceExists
* Similarity of number patterns
* Matching CD[0-9] patterns
* Matching video properties (video duration, frame rate)
* Similarity of names

All results (hash-lookup or name-search) are sorted using these metrics, so the "best" result should end up being the first option.

In this case, most of these metrics yield nothing, so it's effectively asking "Which subtitle filename is most similar to this video filename?" which doesn't really help since it's impossible to tell good/bad ones apart based on the filename alone in this case.

If the filename is "Cosmos.A.Space.Time.Odyssey.S01E02.PROPER.720p.HDTV.X264-DIMENSION" then "Cosmos.A.Space.Time.Odyssey.S01E02.PROPER.720p.HDTV.X264-DIMENSION-eng" is clearly the most similar option (plus there's a exact substring match).



Strange #1
The same subtitles can match (and be in sync with) many video files (different encodings, different formats, etc) as long as the timing is the same.

And there is plenty of room for user errors:
e.g. Person A uploads subtitles S1 for movie M1. Person B downloads subtitles S1 for movie M2 because it's the only option. Person B uploads subtitles S1 for M2 without making sure the subtitles are in sync. Now S1 is hash-matched to both M1 and M2.


Strange #2
The subtitle options you get from a hash-lookup SHOULD ALL BE PERFECT so any option should be a good option.

The only reason FileBot tries to pick the "best" option instead of just picking the first one is because stupid uses keep hash-matching subtitles with the wrong movie files and FileBot tries to work around that.


OpenSubtitles Hash => http://trac.opensubtitles.org/projects/ ... ourceCodes

[viking]

As always, thanks for your quick response.
I understand now that the same subtitle can match different movies (with different hashes).

However, I am still confused why the media file "Cosmos.A.Space.Time.Odyssey.S01E02.PROPER.720p.HDTV.X264-DIMENSION.mkv" works fine with all 3 subtitles, while the media file "Cosmos.A.Space.Time.Odyssey.S01E02.720p.HDTV.TommyJay.mkv" only works with one subtitle (Cosmos.A.Space.Time.Odyssey.S01E02.PROPER.720p.HDTV.X264-DIMENSION-eng.srt)? Any ideas?

[rednoah]

Two different video files yield two different sets of subtitle hash-matches.

It doesn't matter that both files happen to be the same episode. You might as well rename the files Avatar.mkv and Thor.mkv and you should still get the same Cosmos Episode 2 results (though FileBot might sort them differently).

* A typical hash-lookup response should be like for DIMENSION one => one or more options, all of which are perfectly synced

* The issue with the TommyJay release is that the hash-lookup response contains badly synced options which shouldn't exist (unless stupid users or extremely rare hash/size collision)

[viking]

Sorry, I still don't understand;
If "Cosmos.A.Space.Time.Odyssey.S01E02.PROPER.720p.HDTV.X264-DIMENSION.mkv" works with all 3 subtitles then shouldn't the subtitles all be in sync? In other words. the dialogues would all occur at the same times in all 3 subtitles?

If that is the case, then shouldn't all 3 also work with "Cosmos.A.Space.Time.Odyssey.S01E02.720p.HDTV.TommyJay.mkv" (as long as one of them works)?

What am I missing?

EDIT: I checked all 3 subtitles using (an old version of) KMPlayer Subtitle Explorer; I sxlected the dialogue at Time=30.01 sec and the same dialogue occurred at that time in all 3 subtitles.
(There is also a column labeled "Sync". I am not sure what that is for, but it also shows the same time-stamp: 1801368 ms, which is 30.01 sec)

[rednoah]

One subtitle might work. Another might not. Depends on wether or not somebody uploaded bad subtitles for that video file.


If you're using hash lookup then all the subtitles should have the same timing.

You understand how it works in theory.

QUESTION:
What happens if User Jimmy uploads subtitle Avatar.srt for video file Terminator.avi and User John then comes along and searches for subtitles for the exact same video file Terminator.avi?

[rednoah]

You have two completely different video files: M1 and M2

You have 3 completely different subtitles files for each video files: S11 S12 S13 S21 S22 S23

All the subtitles (e.g. S11) are provided by users, and all the associations (e.g. S11=>M1) are provided by users, so if users mess these things up, then you may get out-of-sync subtitles, or subtitles for the wrong movie/episode altogether. Either because the association (e.g. S11=>M1) is wrong, or because the the subtitle content (e.g. S11) is simply wrong.

[rednoah]

What's the output of:

Code: Select all

filebot -script fn:osdb.explain /path/to/cosmos/episodes

[viking]

I understand how an uploaded subtitle can be incorrect for a particular media file. What I am trying to understand is what the difference is between different subtitles; if they have the same timings shouldn't they all work the same? In other words, they should all be in Sync (or not in Sync) for a particular media file?

Here is the osdb.explain results for "Cosmos.A.Space.Time.Odyssey.S01E02.720p.HDTV.TommyJay.mkv"

Code: Select all

P:\Ultimate_Server\!TEMP_HOLD\Test\Cosmos.A.Space.Time.Odyssey.S01E02.720p.HDTV.TommyJay.mkv

	cosmos.a.space.time.odyssey.102.hdtv-lol.eng [English]
		IDSubtitle = 5587891
		IDSubtitleFile = 1954173645
		IDSubMovieFile = 9308640
		IDMovie = 173630
		IDMovieImdb = 2987754
		SubFileName = cosmos.a.space.time.odyssey.102.hdtv-lol.eng.srt
		SubFormat = srt
		SubHash = c4995d7aac5db80b9aca93e0be6b1056
		SubSize = 56701
		MovieHash = 1961faa834f41e4c
		MovieByteSize = 1722075354
		MovieName = "Cosmos: A Spacetime Odyssey" Some of the Things That Molecules Do
		MovieNameEng = 
		MovieYear = 2014
		MovieReleaseName = cosmos.a.space.time.odyssey.102.hdtv-lol
		MovieTimeMS = 0
		MovieFPS = 0.000
		MovieImdbRating = 9.2
		MovieKind = episode
		SeriesSeason = 1
		SeriesEpisode = 2
		SeriesIMDBParent = 2395695
		SubLanguageID = eng
		ISO639 = en
		LanguageName = English
		UserID = 790019
		UserRank = gold member
		UserNickName = hippik
		SubAddDate = 2014-03-17 10:51:30
		SubAuthorComment = 
		SubFeatured = 0
		SubComments = 0
		SubDownloadsCnt = 31506
		SubHearingImpaired = 0
		SubRating = 0.0
		SubHD = 1
		SubBad = 0
		SubActualCD = 1
		SubSumCD = 1
		MatchedBy = moviehash
		QueryNumber = null
		SubtitlesLink = http://www.opensubtitles.org/en/subtitles/5587891/sid-2krkvlm0vhkovnj0f5hkuj0vn2/cosmos-a-spacetime-odyssey-some-of-the-things-that-molecules-do-en
		SubDownloadLink = http://dl.opensubtitles.org/en/download/file/src-api/vrf-19b60c56/sid-2krkvlm0vhkovnj0f5hkuj0vn2/1954173645.gz
		ZipDownloadLink = http://dl.opensubtitles.org/en/download/sub/src-api/vrf-f5790bc4/sid-2krkvlm0vhkovnj0f5hkuj0vn2/5587891

	Cosmos.A.Space.Time.Odyssey.S01E02.720p.HDTV.x264-SWOLLED-eng [English]
		IDSubtitle = 5589103
		IDSubtitleFile = 1954174855
		IDSubMovieFile = 9314984
		IDMovie = 173630
		IDMovieImdb = 2987754
		SubFileName = Cosmos.A.Space.Time.Odyssey.S01E02.720p.HDTV.x264-SWOLLED-eng.srt
		SubFormat = srt
		SubHash = 1f57567a0ea7445a22d8a1ff9650a4b7
		SubSize = 57494
		MovieHash = 1961faa834f41e4c
		MovieByteSize = 1722075354
		MovieName = "Cosmos: A Spacetime Odyssey" Some of the Things That Molecules Do
		MovieNameEng = 
		MovieYear = 2014
		MovieReleaseName = Cosmos.A.Space.Time.Odyssey.S01E02.720p.HDTV.x264-SWOLLED
		MovieTimeMS = 0
		MovieFPS = 0.000
		MovieImdbRating = 9.2
		MovieKind = episode
		SeriesSeason = 1
		SeriesEpisode = 2
		SeriesIMDBParent = 2395695
		SubLanguageID = eng
		ISO639 = en
		LanguageName = English
		UserID = 0
		UserRank = 
		UserNickName = 
		SubAddDate = 2014-03-18 05:13:28
		SubAuthorComment = 
		SubFeatured = 0
		SubComments = 0
		SubDownloadsCnt = 6633
		SubHearingImpaired = 0
		SubRating = 0.0
		SubHD = 1
		SubBad = 0
		SubActualCD = 1
		SubSumCD = 1
		MatchedBy = moviehash
		QueryNumber = null
		SubtitlesLink = http://www.opensubtitles.org/en/subtitles/5589103/sid-2krkvlm0vhkovnj0f5hkuj0vn2/cosmos-a-spacetime-odyssey-some-of-the-things-that-molecules-do-en
		SubDownloadLink = http://dl.opensubtitles.org/en/download/file/src-api/vrf-19c20c5a/sid-2krkvlm0vhkovnj0f5hkuj0vn2/1954174855.gz
		ZipDownloadLink = http://dl.opensubtitles.org/en/download/sub/src-api/vrf-f55c0bb8/sid-2krkvlm0vhkovnj0f5hkuj0vn2/5589103

	Cosmos.A.Space.Time.Odyssey.S01E02.PROPER.720p.HDTV.X264-DIMENSION-eng [English]
		IDSubtitle = 5588572
		IDSubtitleFile = 1954174330
		IDSubMovieFile = 9318907
		IDMovie = 173630
		IDMovieImdb = 2987754
		SubFileName = Cosmos.A.Space.Time.Odyssey.S01E02.PROPER.720p.HDTV.X264-DIMENSION-eng.srt
		SubFormat = srt
		SubHash = 940c8e1cf55a6726ffe8ed296b387fd3
		SubSize = 53522
		MovieHash = 1961faa834f41e4c
		MovieByteSize = 1722075354
		MovieName = "Cosmos: A Spacetime Odyssey" Some of the Things That Molecules Do
		MovieNameEng = 
		MovieYear = 2014
		MovieReleaseName = Cosmos.A.Space.Time.Odyssey.S01E02.PROPER.720p.HDTV.X264-DIMENSION
		MovieTimeMS = 0
		MovieFPS = 0.000
		MovieImdbRating = 9.2
		MovieKind = episode
		SeriesSeason = 1
		SeriesEpisode = 2
		SeriesIMDBParent = 2395695
		SubLanguageID = eng
		ISO639 = en
		LanguageName = English
		UserID = 0
		UserRank = 
		UserNickName = 
		SubAddDate = 2014-03-17 21:23:27
		SubAuthorComment = 
		SubFeatured = 0
		SubComments = 0
		SubDownloadsCnt = 11190
		SubHearingImpaired = 0
		SubRating = 0.0
		SubHD = 1
		SubBad = 0
		SubActualCD = 1
		SubSumCD = 1
		MatchedBy = moviehash
		QueryNumber = null
		SubtitlesLink = http://www.opensubtitles.org/en/subtitles/5588572/sid-2krkvlm0vhkovnj0f5hkuj0vn2/cosmos-a-spacetime-odyssey-some-of-the-things-that-molecules-do-en
		SubDownloadLink = http://dl.opensubtitles.org/en/download/file/src-api/vrf-19aa0c4e/sid-2krkvlm0vhkovnj0f5hkuj0vn2/1954174330.gz
		ZipDownloadLink = http://dl.opensubtitles.org/en/download/sub/src-api/vrf-f5710bc1/sid-2krkvlm0vhkovnj0f5hkuj0vn2/5588572
Done ヾ(＠⌒ー⌒＠)ノ

And for "Cosmos.A.Space.Time.Odyssey.S01E02.PROPER.720p.HDTV.X264-DIMENSION.mkv"

Code: Select all

P:\Ultimate_Server\!TEMP_HOLD\test2\Cosmos.A.Space.Time.Odyssey.S01E02.PROPER.720p.HDTV.X264-DIMENSION.mkv

	cosmos.a.space.time.odyssey.102.hdtv-lol.eng [English]
		IDSubtitle = 5587891
		IDSubtitleFile = 1954173645
		IDSubMovieFile = 9308609
		IDMovie = 173630
		IDMovieImdb = 2987754
		SubFileName = cosmos.a.space.time.odyssey.102.hdtv-lol.eng.srt
		SubFormat = srt
		SubHash = c4995d7aac5db80b9aca93e0be6b1056
		SubSize = 56701
		MovieHash = 53c9e1b40875d8ff
		MovieByteSize = 1346233417
		MovieName = "Cosmos: A Spacetime Odyssey" Some of the Things That Molecules Do
		MovieNameEng = 
		MovieYear = 2014
		MovieReleaseName = cosmos.a.space.time.odyssey.102.hdtv-lol
		MovieTimeMS = 0
		MovieFPS = 0.000
		MovieImdbRating = 9.2
		MovieKind = episode
		SeriesSeason = 1
		SeriesEpisode = 2
		SeriesIMDBParent = 2395695
		SubLanguageID = eng
		ISO639 = en
		LanguageName = English
		UserID = 790019
		UserRank = gold member
		UserNickName = hippik
		SubAddDate = 2014-03-17 10:51:30
		SubAuthorComment = 
		SubFeatured = 0
		SubComments = 0
		SubDownloadsCnt = 31463
		SubHearingImpaired = 0
		SubRating = 0.0
		SubHD = 1
		SubBad = 0
		SubActualCD = 1
		SubSumCD = 1
		MatchedBy = moviehash
		QueryNumber = null
		SubtitlesLink = http://www.opensubtitles.org/en/subtitles/5587891/sid-1uchmr7mp29phtg5e10785sdi4/cosmos-a-spacetime-odyssey-some-of-the-things-that-molecules-do-en
		SubDownloadLink = http://dl.opensubtitles.org/en/download/file/src-api/vrf-19b60c56/sid-1uchmr7mp29phtg5e10785sdi4/1954173645.gz
		ZipDownloadLink = http://dl.opensubtitles.org/en/download/sub/src-api/vrf-f5790bc4/sid-1uchmr7mp29phtg5e10785sdi4/5587891

	Cosmos.A.Space.Time.Odyssey.S01E02.PROPER.720p.HDTV.X264-DIMENSION-eng [English]
		IDSubtitle = 5588572
		IDSubtitleFile = 1954174330
		IDSubMovieFile = 9311309
		IDMovie = 173630
		IDMovieImdb = 2987754
		SubFileName = Cosmos.A.Space.Time.Odyssey.S01E02.PROPER.720p.HDTV.X264-DIMENSION-eng.srt
		SubFormat = srt
		SubHash = 940c8e1cf55a6726ffe8ed296b387fd3
		SubSize = 53522
		MovieHash = 53c9e1b40875d8ff
		MovieByteSize = 1346233417
		MovieName = "Cosmos: A Spacetime Odyssey" Some of the Things That Molecules Do
		MovieNameEng = 
		MovieYear = 2014
		MovieReleaseName = Cosmos.A.Space.Time.Odyssey.S01E02.PROPER.720p.HDTV.X264-DIMENSION
		MovieTimeMS = 0
		MovieFPS = 0.000
		MovieImdbRating = 9.2
		MovieKind = episode
		SeriesSeason = 1
		SeriesEpisode = 2
		SeriesIMDBParent = 2395695
		SubLanguageID = eng
		ISO639 = en
		LanguageName = English
		UserID = 0
		UserRank = 
		UserNickName = 
		SubAddDate = 2014-03-17 21:23:27
		SubAuthorComment = 
		SubFeatured = 0
		SubComments = 0
		SubDownloadsCnt = 11182
		SubHearingImpaired = 0
		SubRating = 0.0
		SubHD = 1
		SubBad = 0
		SubActualCD = 1
		SubSumCD = 1
		MatchedBy = moviehash
		QueryNumber = null
		SubtitlesLink = http://www.opensubtitles.org/en/subtitles/5588572/sid-1uchmr7mp29phtg5e10785sdi4/cosmos-a-spacetime-odyssey-some-of-the-things-that-molecules-do-en
		SubDownloadLink = http://dl.opensubtitles.org/en/download/file/src-api/vrf-19aa0c4e/sid-1uchmr7mp29phtg5e10785sdi4/1954174330.gz
		ZipDownloadLink = http://dl.opensubtitles.org/en/download/sub/src-api/vrf-f5710bc1/sid-1uchmr7mp29phtg5e10785sdi4/5588572

	Cosmos.A.Space.Time.Odyssey.S01E02.720p.HDTV.x264-SWOLLED-eng [English]
		IDSubtitle = 5589103
		IDSubtitleFile = 1954174855
		IDSubMovieFile = 9315083
		IDMovie = 173630
		IDMovieImdb = 2987754
		SubFileName = Cosmos.A.Space.Time.Odyssey.S01E02.720p.HDTV.x264-SWOLLED-eng.srt
		SubFormat = srt
		SubHash = 1f57567a0ea7445a22d8a1ff9650a4b7
		SubSize = 57494
		MovieHash = 53c9e1b40875d8ff
		MovieByteSize = 1346233417
		MovieName = "Cosmos: A Spacetime Odyssey" Some of the Things That Molecules Do
		MovieNameEng = 
		MovieYear = 2014
		MovieReleaseName = Cosmos.A.Space.Time.Odyssey.S01E02.720p.HDTV.x264-SWOLLED
		MovieTimeMS = 0
		MovieFPS = 0.000
		MovieImdbRating = 9.2
		MovieKind = episode
		SeriesSeason = 1
		SeriesEpisode = 2
		SeriesIMDBParent = 2395695
		SubLanguageID = eng
		ISO639 = en
		LanguageName = English
		UserID = 0
		UserRank = 
		UserNickName = 
		SubAddDate = 2014-03-18 05:13:28
		SubAuthorComment = 
		SubFeatured = 0
		SubComments = 0
		SubDownloadsCnt = 6620
		SubHearingImpaired = 0
		SubRating = 0.0
		SubHD = 1
		SubBad = 0
		SubActualCD = 1
		SubSumCD = 1
		MatchedBy = moviehash
		QueryNumber = null
		SubtitlesLink = http://www.opensubtitles.org/en/subtitles/5589103/sid-1uchmr7mp29phtg5e10785sdi4/cosmos-a-spacetime-odyssey-some-of-the-things-that-molecules-do-en
		SubDownloadLink = http://dl.opensubtitles.org/en/download/file/src-api/vrf-19c20c5a/sid-1uchmr7mp29phtg5e10785sdi4/1954174855.gz
		ZipDownloadLink = http://dl.opensubtitles.org/en/download/sub/src-api/vrf-f55c0bb8/sid-1uchmr7mp29phtg5e10785sdi4/5589103
Done ヾ(＠⌒ー⌒＠)ノ

"

[rednoah]

Alright, so according to the logs do get the same 3 subtitles for both video files. If the subtitles all have exactly the same timing then they should of course work equally well.

If one subtitle works, and the other one doesn't, then something has too be different.

Are the subtitles downloaded by filebot for each case exactly the same?

If 2 subtitle files are byte-wise identical, and one works while the other doesn't, then there's something wrong with the way you're testing things.