submod retry; WIP

CHANGELOG.md

@@ -1,86 +1,3 @@
-2.6.5.3152
-
-subscene, addic7ed
-- either of those providers might impose a reCAPTCHA verification. In order to use those providers, please create an account at an AntiCaptcha service ([anti-captcha.com](http://getcaptchasolution.com/kkvviom7nh) or [deathbycaptcha.com](http://deathbycaptcha.com)), add funds, then supply your credentials/apikey in the configuration
-
-Changelog
-- core: fix core issue possibly impacting results on OpenSubtitles in certain conditions
-- core: fix default values of opensubtitles-skip-wrong-fps, use_https; fix #676
-- core: fix for determining whether to search under certain circumstances; fixes #666
-- core: #664 fix missing language processing of multiple videos refreshed at once
-- core: #661 fix match strictness when determining preexisting external subtitles
-- providers: titlovi: New implementation of Titlovi using API (thanks @viking1304)
-
-
-2.6.5.3124
-
-subscene, addic7ed and titlovi
-- either of those providers might impose a reCAPTCHA verification. In order to use those providers, please create an account at an AntiCaptcha service ([anti-captcha.com](http://getcaptchasolution.com/kkvviom7nh) or [deathbycaptcha.com](http://deathbycaptcha.com)), add funds, then supply your credentials/apikey in the configuration
-
-Changelog
-- core: http: fallback to default DNS when normal resolving fails; fixes #657
-- core: extract embedded/menu: fix detection of unknown streams; don't use unknown streams if a known language was previously found
-- core: language: use replacement map from bazarr
-- providers: titlovi: fix matching
-- providers: subscene: fix unknown language code error when "empty" result is returned
-- providers: subscene: add support for pt-BR (based on Diaoul/subliminal@b22cf08)
-- providers: subscene: explicitly set account filters for languages
-- providers: subscene: limit alternative searches to 3; set throttle to 8
-- providers: subscene: move login/cookies to initialization sequence
-- submod: generic: en: fix ";='s
-
-
-2.6.5.3109
-
-subscene, addic7ed and titlovi
-- either of those providers might impose a reCAPTCHA verification. In order to use those providers, please create an account at an AntiCaptcha service ([anti-captcha.com](http://getcaptchasolution.com/kkvviom7nh) or [deathbycaptcha.com](http://deathbycaptcha.com)), add funds, then supply your credentials/apikey in the configuration
-
-Changelog
-- providers: add Napisy24 (polish)
-- providers: subscene: reduce provider load by possibly half
-- providers: subscene: support logging in (username/password are now required)
-- providers: subscene: fallback to non year results if none found with year
-
-
-2.6.5.3099
-
-subscene, addic7ed and titlovi
-- either of those providers might impose a reCAPTCHA verification. In order to use those providers, please create an account at an AntiCaptcha service ([anti-captcha.com](http://getcaptchasolution.com/kkvviom7nh) or [deathbycaptcha.com](http://deathbycaptcha.com)), add funds, then supply your credentials/apikey in the configuration
-
-Changelog
-- core: allow system DNS again by putting "system" as the DNS
-- providers: subscene: fix again (subscene, contact us please, so we can end this)
-
-
-2.6.5.3092
-
-subscene, addic7ed and titlovi
-- either of those providers might impose a reCAPTCHA verification. In order to use those providers, please create an account at an AntiCaptcha service ([anti-captcha.com](http://getcaptchasolution.com/kkvviom7nh) or [deathbycaptcha.com](http://deathbycaptcha.com)), add funds, then supply your credentials/apikey in the configuration
-
-Changelog
-- providers: subscene: fix endpoint (hopefully for longer now)
-- providers: subscene: don't search for season packs (broken for now; relieves 50% of server load on provider)
-- providers: subscene: don't calculate video fn for now
-- providers: argenteam: backport fixes from bazarr
-- subtitle: try decoding with utf-16 by default as well (zho/farsi)
-- submod: HI: remove music tags by default
-- core: compat (bazarr): add env var SZ_KEEP_ENCODING to keep encoding of subtitles
-
-
-2.6.5.3074
-
-Changelog
-- core: cf: bypass cf 95% of the time without captchas
-- core: fix breaking line endings of certain languages (chinese, UTF-16); fixes #646
-- core: update pysubs2 to 0.2.3
-
-
-2.6.5.3062
-
-Changelog
-- core: cf: optimize
-- core: http: don't query DNS with IPs. thanks @fgump (fixes sonarr/radarr)
-
 
 2.6.5.3041
 

Contents/Code/__init__.py

@@ -119,23 +119,19 @@ def agent_extract_embedded(video_part_map):
             for plexapi_part in get_all_parts(plexapi_item):
                 item_count = item_count + 1
                 used_one_unknown_stream = False
-                used_one_known_stream = False
                 for requested_language in config.lang_list:
-                    skip_unknown = used_one_unknown_stream or used_one_known_stream
                     embedded_subs = stored_subs.get_by_provider(plexapi_part.id, requested_language, "embedded")
                     current = stored_subs.get_any(plexapi_part.id, requested_language) or \
                         requested_language in scanned_video.external_subtitle_languages
 
                     if not embedded_subs:
                         stream_data = get_embedded_subtitle_streams(plexapi_part, requested_language=requested_language,
-                                                                    skip_unknown=skip_unknown)
+                                                                    skip_unknown=used_one_unknown_stream)
 
-                        if stream_data and stream_data[0]["language"]:
+                        if stream_data:
                             stream = stream_data[0]["stream"]
                             if stream_data[0]["is_unknown"]:
                                 used_one_unknown_stream = True
-                            else:
-                                used_one_known_stream = True
 
                             to_extract.append(({scanned_video: part_info}, plexapi_part, str(stream.index),
                                                str(requested_language), not current))

Contents/Code/interface/item_details.py

@@ -12,10 +12,9 @@ from menu_helpers import debounce, SubFolderObjectContainer, default_thumb, add_
 from refresh_item import RefreshItem
 from subzero.constants import PREFIX
 from support.config import config, TEXT_SUBTITLE_EXTS
-from support.helpers import timestamp, df, get_language, display_language, get_language_from_stream
+from support.helpers import timestamp, df, get_language, display_language, get_language_from_stream, is_stream_forced
 from support.items import get_item_kind_from_rating_key, get_item, get_current_sub, get_item_title, save_stored_sub
-from support.plex_media import get_plex_metadata, get_part, get_embedded_subtitle_streams, is_stream_forced, \
-    update_stream_info
+from support.plex_media import get_plex_metadata, get_part, get_embedded_subtitle_streams
 from support.scanning import scan_videos
 from support.scheduler import scheduler
 from support.storage import get_subtitle_storage
@@ -119,8 +118,6 @@ def ItemDetailsMenu(rating_key, title=None, base_title=None, item_title=None, ra
             if not os.path.exists(part.file):
                 continue
 
-            update_stream_info(part)
-
             part_id = str(part.id)
             part_index += 1
 
@@ -673,28 +670,29 @@ def ListEmbeddedSubsForItemMenu(**kwargs):
             stream = stream_data["stream"]
             is_forced = stream_data["is_forced"]
 
-            oc.add(DirectoryObject(
-                key=Callback(TriggerExtractEmbeddedSubForItemMenu, randomize=timestamp(),
-                             stream_index=str(stream.index), language=language, with_mods=True, **kwargs),
-                title=_(u"Extract stream %(stream_index)s, %(language)s%(unknown_state)s%(forced_state)s"
-                        u"%(stream_title)s with default mods",
-                        stream_index=stream.index,
-                        language=display_language(language),
-                        unknown_state=_(" (unknown)") if is_unknown else "",
-                        forced_state=_(" (forced)") if is_forced else "",
-                        stream_title=" (\"%s\")" % stream.title if stream.title else ""),
-            ))
-            oc.add(DirectoryObject(
-                key=Callback(TriggerExtractEmbeddedSubForItemMenu, randomize=timestamp(),
-                             stream_index=str(stream.index), language=language, **kwargs),
-                title=_(u"Extract stream %(stream_index)s, %(language)s%(unknown_state)s%(forced_state)s"
-                        u"%(stream_title)s",
-                        stream_index=stream.index,
-                        language=display_language(language),
-                        unknown_state=_(" (unknown)") if is_unknown else "",
-                        forced_state=_(" (forced)") if is_forced else "",
-                        stream_title=" (\"%s\")" % stream.title if stream.title else ""),
-            ))
+            if language:
+                oc.add(DirectoryObject(
+                    key=Callback(TriggerExtractEmbeddedSubForItemMenu, randomize=timestamp(),
+                                 stream_index=str(stream.index), language=language, with_mods=True, **kwargs),
+                    title=_(u"Extract stream %(stream_index)s, %(language)s%(unknown_state)s%(forced_state)s"
+                            u"%(stream_title)s with default mods",
+                            stream_index=stream.index,
+                            language=display_language(language),
+                            unknown_state=_(" (unknown)") if is_unknown else "",
+                            forced_state=_(" (forced)") if is_forced else "",
+                            stream_title=" (\"%s\")" % stream.title if stream.title else ""),
+                ))
+                oc.add(DirectoryObject(
+                    key=Callback(TriggerExtractEmbeddedSubForItemMenu, randomize=timestamp(),
+                                 stream_index=str(stream.index), language=language, **kwargs),
+                    title=_(u"Extract stream %(stream_index)s, %(language)s%(unknown_state)s%(forced_state)s"
+                            u"%(stream_title)s",
+                            stream_index=stream.index,
+                            language=display_language(language),
+                            unknown_state=_(" (unknown)") if is_unknown else "",
+                            forced_state=_(" (forced)") if is_forced else "",
+                            stream_title=" (\"%s\")" % stream.title if stream.title else ""),
+                ))
     return oc
 
 

Contents/Code/interface/menu.py

@@ -368,7 +368,7 @@ def ValidatePrefs():
             "subtitle_destination_folder", "include", "include_exclude_paths", "include_exclude_sz_files",
             "new_style_cache", "dbm_supported", "lang_list", "providers", "normal_subs", "forced_only", "forced_also",
             "plex_transcoder", "refiner_settings", "unrar", "adv_cfg_path", "use_custom_dns",
-            "has_anticaptcha", "anticaptcha_cls", "mediainfo_bin"]:
+            "has_anticaptcha", "anticaptcha_cls"]:
 
         value = getattr(config, attr)
         if isinstance(value, dict):

Contents/Code/interface/menu_helpers.py

@@ -11,14 +11,14 @@ from subzero.lib.io import get_viable_encoding
 from subzero.language import Language
 from support.i18n import is_localized_string, _
 from support.items import get_kind, get_item_thumb, get_item, get_item_kind_from_item, refresh_item
-from support.helpers import get_video_display_title, pad_title, display_language, quote_args, \
+from support.helpers import get_video_display_title, pad_title, display_language, quote_args, is_stream_forced, \
     get_title_for_video_metadata, mswindows
 from support.history import get_history
 from support.ignore import get_decision_list
 from support.lib import get_intent
 from support.config import config
 from subzero.constants import ICON_SUB, ICON
-from support.plex_media import get_part, get_plex_metadata, is_stream_forced, update_stream_info
+from support.plex_media import get_part, get_plex_metadata
 from support.scheduler import scheduler
 from support.scanning import scan_videos
 from support.storage import save_subtitles
@@ -178,7 +178,6 @@ def extract_embedded_sub(**kwargs):
             metadata = get_plex_metadata(rating_key, part_id, item_type, plex_item=plex_item)
             scanned_videos = scan_videos([metadata], ignore_all=True, skip_hashing=True)
 
-        update_stream_info(part)
         for stream in part.streams:
             # subtitle stream
             if str(stream.index) == stream_index:

Contents/Code/support/config.py

@@ -22,7 +22,6 @@ from subzero.language import Language
 from subliminal.cli import MutexLock
 from subzero.lib.io import FileIO, get_viable_encoding
 from subzero.lib.dict import Dicked
-from subzero.lib.which import find_executable
 from subzero.util import get_root_path
 from subzero.constants import PLUGIN_NAME, PLUGIN_IDENTIFIER, MOVIE, SHOW, MEDIA_TYPE_TO_STRING
 from subzero.prefs import get_user_prefs, update_user_prefs
@@ -67,7 +66,6 @@ PROVIDER_THROTTLE_MAP = {
         DownloadLimitExceeded: (datetime.timedelta(hours=3), "3 hours"),
         ServiceUnavailable: (datetime.timedelta(minutes=20), "20 minutes"),
         APIThrottled: (datetime.timedelta(minutes=10), "10 minutes"),
-        AuthenticationError: (datetime.timedelta(hours=2), "2 hours"),
     },
     "opensubtitles": {
         TooManyRequests: (datetime.timedelta(hours=3), "3 hours"),
@@ -77,7 +75,6 @@ PROVIDER_THROTTLE_MAP = {
     "addic7ed": {
         DownloadLimitExceeded: (datetime.timedelta(hours=3), "3 hours"),
         TooManyRequests: (datetime.timedelta(minutes=5), "5 minutes"),
-        AuthenticationError: (datetime.timedelta(hours=24), "24 hours"),
     }
 }
 
@@ -156,7 +153,6 @@ class Config(object):
     anticaptcha_token = None
     anticaptcha_cls = None
     has_anticaptcha = False
-    mediainfo_bin = None
 
     store_recently_played_amount = 40
 
@@ -243,8 +239,6 @@ class Config(object):
         self.embedded_auto_extract = cast_bool(Prefs["subtitles.embedded.autoextract"])
         self.ietf_as_alpha3 = cast_bool(Prefs["subtitles.language.ietf_normalize"])
         self.use_custom_dns = self.parse_custom_dns()
-        if not self.advanced.dont_use_mediainfo_mp4:
-            self.mediainfo_bin = self.advanced.mediainfo_bin or find_executable("mediainfo")
         self.initialized = True
 
     def migrate_prefs(self):
@@ -767,7 +761,6 @@ class Config(object):
         return {'opensubtitles': cast_bool(Prefs['provider.opensubtitles.enabled']),
                      # 'thesubdb': Prefs['provider.thesubdb.enabled'],
                      'podnapisi': cast_bool(Prefs['provider.podnapisi.enabled']),
-                     'napisy24': cast_bool(Prefs['provider.napisy24.enabled']),
                      'titlovi': cast_bool(Prefs['provider.titlovi.enabled']),
                      'addic7ed': cast_bool(Prefs['provider.addic7ed.enabled']) and self.has_anticaptcha,
                      'tvsubtitles': cast_bool(Prefs['provider.tvsubtitles.enabled']),
@@ -808,7 +801,6 @@ class Config(object):
             providers["argenteam"] = False
             providers["assrt"] = False
             providers["subscene"] = False
-            providers["napisy24"] = False
             providers_forced_off = dict(providers)
 
         if not self.unrar and providers["legendastv"]:
@@ -849,8 +841,11 @@ class Config(object):
     providers = property(get_providers)
 
     def get_provider_settings(self):
-        os_use_https = self.advanced.providers.opensubtitles.get("use_https", True)
-        os_skip_wrong_fps = self.advanced.providers.opensubtitles.get("skip_wrong_fps", True)
+        os_use_https = self.advanced.providers.opensubtitles.use_https \
+            if self.advanced.providers.opensubtitles.use_https is not None else True
+
+        os_skip_wrong_fps = self.advanced.providers.opensubtitles.skip_wrong_fps \
+            if self.advanced.providers.opensubtitles.skip_wrong_fps is not None else True
 
         provider_settings = {'addic7ed': {'username': Prefs['provider.addic7ed.username'],
                                           'password': Prefs['provider.addic7ed.password'],
@@ -869,18 +864,8 @@ class Config(object):
                                  'only_foreign': self.forced_only,
                                  'also_foreign': self.forced_also,
                              },
-                             'titlovi': {
-                                 'username': Prefs['provider.titlovi.username'],
-                                 'password': Prefs['provider.titlovi.password'],
-                             },
-                             'napisy24': {
-                                 'username': Prefs['provider.napisy24.username'],
-                                 'password': Prefs['provider.napisy24.password'],
-                             },
                              'subscene': {
                                  'only_foreign': self.forced_only,
-                                 'username': Prefs['provider.subscene.username'],
-                                 'password': Prefs['provider.subscene.password'],
                              },
                              'legendastv': {'username': Prefs['provider.legendastv.username'],
                                             'password': Prefs['provider.legendastv.password'],
@@ -909,10 +894,10 @@ class Config(object):
         throttle_data = PROVIDER_THROTTLE_MAP.get(name, PROVIDER_THROTTLE_MAP["default"]).get(cls, None) or \
             PROVIDER_THROTTLE_MAP["default"].get(cls, None)
 
-        if throttle_data:
-            throttle_delta, throttle_description = throttle_data
-        else:
-            throttle_delta, throttle_description = datetime.timedelta(minutes=10), "10 minutes"
+        if not throttle_data:
+            return
+
+        throttle_delta, throttle_description = throttle_data
 
         if "provider_throttle" not in Dict:
             Dict["provider_throttle"] = {}
@@ -1099,12 +1084,11 @@ class Config(object):
     def parse_custom_dns(self):
         custom_dns = Prefs['use_custom_dns2'].strip()
         os.environ["dns_resolvers"] = ""
-
-        if custom_dns and custom_dns != "system":
+        if custom_dns:
             ips = filter(lambda x: x, [d.strip() for d in custom_dns.split(",")])
             if ips:
                 os.environ["dns_resolvers"] = json.dumps(ips)
-            return os.environ["dns_resolvers"]
+                return os.environ["dns_resolvers"]
 
     def init_subliminal_patches(self):
         # configure custom subtitle destination folders for scanning pre-existing subs

Contents/Code/support/download.py

@@ -33,14 +33,14 @@ def get_missing_languages(video, part):
     alpha3_map = {}
     if config.ietf_as_alpha3:
         for language in languages:
-            if language and language.country:
+            if language.country:
                 alpha3_map[language.alpha3] = language.country
                 language.country = None
 
     have_languages = video.subtitle_languages.copy()
     if config.ietf_as_alpha3:
         for language in have_languages:
-            if language and language.country:
+            if language.country:
                 alpha3_map[language.alpha3] = language.country
                 language.country = None
 
@@ -53,14 +53,14 @@ def get_missing_languages(video, part):
                 filter(lambda l: not l.forced, video.subtitle_languages)
             if langs:
                 Log.Debug("We have at least one subtitle for any configured language.")
-                return set()
+                return False
 
         elif "External subtitle" in config.any_language_is_enough:
-            langs = video.external_subtitle_languages if not not_in_forced else \
+            langs = video.subtitle_languages if not not_in_forced else \
                 filter(lambda l: not l.forced, video.external_subtitle_languages)
             if langs:
                 Log.Debug("We have at least one external subtitle for any configured language.")
-                return set()
+                return False
 
     # all languages are found if we either really have subs for all languages or we only want to have exactly one language
     # and we've only found one (the case for a selected language, Prefs['subtitles.only_one'] (one found sub matches any language))
@@ -70,7 +70,7 @@ def get_missing_languages(video, part):
             Log.Debug('Only one language was requested, and we\'ve got a subtitle for %s', video)
         else:
             Log.Debug('All languages %r exist for %s', languages, video)
-        return set()
+        return False
 
     # re-add country codes to the missing languages, in case we've removed them above
     if config.ietf_as_alpha3:
@@ -106,22 +106,21 @@ def language_hook(provider):
 def download_best_subtitles(video_part_map, min_score=0, throttle_time=None, providers=None):
     hearing_impaired = Prefs['subtitles.search.hearingImpaired']
     languages = set([Language.rebuild(l) for l in config.lang_list])
+    missing_languages = []
     if not languages:
         return
 
     use_videos = []
-    missing_languages = set()
     for video, part in video_part_map.iteritems():
         if not video.ignore_all:
-            p_missing_languages = get_missing_languages(video, part)
+            missing_languages = get_missing_languages(video, part)
         else:
-            p_missing_languages = languages
+            missing_languages = languages
 
-        if p_missing_languages:
-            Log.Info(u"%s has missing languages: %s", os.path.basename(video.name), p_missing_languages)
+        if missing_languages:
+            Log.Info(u"%s has missing languages: %s", os.path.basename(video.name), missing_languages)
             refine_video(video, refiner_settings=config.refiner_settings)
             use_videos.append(video)
-            missing_languages.update(p_missing_languages)
 
     # prepare blacklist
     blacklist = get_blacklist_from_part_map(video_part_map, languages)

Contents/Code/support/helpers.py

@@ -394,7 +394,7 @@ def get_language_from_stream(lang_code):
             return Language.fromietf(lang)
         elif lang:
             try:
-                return language_from_stream(lang_code)
+                return language_from_stream(lang)
             except LanguageError:
                 pass
 
@@ -437,10 +437,17 @@ def get_language(lang_short):
 
 
 def display_language(l):
-    if not l:
-        return "Unknown"
     return _(str(l.basename).lower()) + ((u" (%s)" % _("forced")) if l.forced else "")
 
 
+def is_stream_forced(stream):
+    stream_title = getattr(stream, "title", "") or ""
+    forced = getattr(stream, "forced", False)
+    if not forced and stream_title and "forced" in stream_title.strip().lower():
+        forced = True
+
+    return forced
+
+
 class PartUnknownException(Exception):
     pass

Contents/Code/support/localmedia.py

@@ -7,7 +7,6 @@ import helpers
 import subtitlehelpers
 
 from config import config as sz_config
-from subzero.language import ENDSWITH_LANGUAGECODE_RE
 
 
 SECONDARY_TAGS = ['forced', 'normal', 'default', 'embedded', 'embedded-forced', 'custom', 'hi', 'cc', 'sdh']
@@ -126,7 +125,7 @@ def find_subtitles(part, ignore_parts_cleanup=None):
                             root = split_tag[0]
 
                         # get associated media file name without language
-                        sub_fn = ENDSWITH_LANGUAGECODE_RE.sub("", root)
+                        sub_fn = subtitlehelpers.ENDSWITH_LANGUAGECODE_RE.sub("", root)
 
                         # subtitle basename and basename without possible language tag  not found in collected
                         # media files? kill.

Contents/Code/support/missing_subtitles.py

@@ -7,8 +7,7 @@ import os
 from babelfish import LanguageReverseError
 
 from support.config import config, TEXT_SUBTITLE_EXTS
-from support.helpers import get_plex_item_display_title, cast_bool, get_language_from_stream
-from support.plex_media import is_stream_forced, update_stream_info
+from support.helpers import get_plex_item_display_title, cast_bool, get_language_from_stream, is_stream_forced
 from support.items import get_item
 from support.lib import Plex
 from support.storage import get_subtitle_storage
@@ -36,7 +35,7 @@ def item_discover_missing_subs(rating_key, kind="show", added_at=None, section_t
     for media in item.media:
         existing_subs = {"internal": [], "external": [], "own_external": [], "count": 0}
         for part in media.parts:
-            update_stream_info(part)
+
             # did we already download an external subtitle before?
             if subtitle_target_dir and stored_subs:
                 for language in languages_set:

Contents/Code/support/plex_media.py

@@ -1,7 +1,6 @@
 # coding=utf-8
 
 import os
-import subprocess
 
 import helpers
 from items import get_item
@@ -27,9 +26,6 @@ tvdb_guid_identifier = "com.plexapp.agents.thetvdb://"
 
 
 def get_plexapi_stream_info(plex_item, part_id=None):
-    if not plex_item:
-        return
-
     d = {"stream": {}}
     data = d["stream"]
 
@@ -104,9 +100,6 @@ def media_to_videos(media, kind="series"):
                 plex_episode = get_item(ep.id)
                 stream_info = get_plexapi_stream_info(plex_episode)
 
-                if not stream_info:
-                    continue
-
                 for item in media.seasons[season].episodes[episode].items:
                     for part in item.parts:
                         videos.append(
@@ -128,24 +121,22 @@ def media_to_videos(media, kind="series"):
                         )
     else:
         stream_info = get_plexapi_stream_info(plex_item)
-
-        if stream_info:
-            imdb_id = None
-            if imdb_guid_identifier in media.guid:
-                imdb_id = media.guid[len(imdb_guid_identifier):].split("?")[0]
-            for item in media.items:
-                for part in item.parts:
-                    videos.append(
-                        get_metadata_dict(plex_item, part, dict(stream_info, **{"plex_part": part, "type": "movie",
-                                                                                 "title": media.title, "id": media.id,
-                                                                                 "super_thumb": plex_item.thumb,
-                                                                                 "series_id": None, "year": year,
-                                                                                 "season_id": None, "imdb_id": imdb_id,
-                                                                                 "original_title": original_title,
-                                                                                 "series_tvdb_id": None, "tvdb_id": None,
-                                                                                 "section": plex_item.section.title})
-                                          )
-                    )
+        imdb_id = None
+        if imdb_guid_identifier in media.guid:
+            imdb_id = media.guid[len(imdb_guid_identifier):].split("?")[0]
+        for item in media.items:
+            for part in item.parts:
+                videos.append(
+                    get_metadata_dict(plex_item, part, dict(stream_info, **{"plex_part": part, "type": "movie",
+                                                                             "title": media.title, "id": media.id,
+                                                                             "super_thumb": plex_item.thumb,
+                                                                             "series_id": None, "year": year,
+                                                                             "season_id": None, "imdb_id": imdb_id,
+                                                                             "original_title": original_title,
+                                                                             "series_tvdb_id": None, "tvdb_id": None,
+                                                                             "section": plex_item.section.title})
+                                      )
+                )
     return videos
 
 
@@ -183,89 +174,42 @@ def get_all_parts(plex_item):
     return parts
 
 
-def update_stream_info(part):
-    if config.mediainfo_bin and part.container == "mp4":
-        cmdline = '%s --Inform="Text;-%%ID%%_%%Title%%" %s' % (config.mediainfo_bin, helpers.quote(part.file))
-        result = subprocess.check_output(cmdline, stderr=subprocess.PIPE, shell=True)
-        if result:
-            try:
-                stream_titles = {}
-                for pair in result[1:].split("-"):
-                    sid, title = pair.split("_")
-                    stream_titles[int(sid.strip())] = title.strip()
-            except:
-                pass
-            else:
-                filled = []
-                for stream in part.streams:
-                    index = stream.index+1
-                    if index in stream_titles:
-                        stream.title = stream_titles[index]
-                        filled.append(index-1)
-                if filled:
-                    Log.Debug("Filled missing MP4 stream title info for streams: %s", filled)
-
-
-def is_stream_forced(stream):
-    stream_title = getattr(stream, "title", "") or ""
-    forced = getattr(stream, "forced", False)
-    if not forced and stream_title and "forced" in stream_title.strip().lower():
-        forced = True
-
-    return forced
-
-
 def get_embedded_subtitle_streams(part, requested_language=None, skip_duplicate_unknown=True, skip_unknown=False):
     streams = []
     streams_unknown = []
-    all_streams = []
     has_unknown = False
     found_requested_language = False
-    update_stream_info(part)
     for stream in part.streams:
         # subtitle stream
         if stream.stream_type == 3 and not stream.stream_key and stream.codec in TEXT_SUBTITLE_EXTS:
-            is_forced = is_stream_forced(stream)
+            is_forced = helpers.is_stream_forced(stream)
             language = helpers.get_language_from_stream(stream.language_code)
             if language:
                 language = Language.rebuild(language, forced=is_forced)
 
             is_unknown = False
             found_requested_language = requested_language and requested_language == language
-            stream_data = None
 
-            if not language:
+            if not language and config.treat_und_as_first:
                 # only consider first unknown subtitle stream
-                if config.treat_und_as_first:
-                    if has_unknown and skip_duplicate_unknown:
-                        Log.Debug("skipping duplicate unknown")
-                        continue
+                if has_unknown and skip_duplicate_unknown:
+                    continue
 
-                    language = Language.rebuild(list(config.lang_list)[0], forced=is_forced)
-                else:
-                    language = None
+                language = Language.rebuild(list(config.lang_list)[0], forced=is_forced)
                 is_unknown = True
                 has_unknown = True
-                stream_data = {"stream": stream, "is_unknown": is_unknown, "language": language,
-                               "is_forced": is_forced}
-                streams_unknown.append(stream_data)
+                streams_unknown.append({"stream": stream, "is_unknown": is_unknown, "language": language,
+                                        "is_forced": is_forced})
 
             if not requested_language or found_requested_language:
-                stream_data = {"stream": stream, "is_unknown": is_unknown, "language": language,
-                               "is_forced": is_forced}
-                streams.append(stream_data)
+                streams.append({"stream": stream, "is_unknown": is_unknown, "language": language,
+                                "is_forced": is_forced})
 
                 if found_requested_language:
                     break
 
-            if stream_data:
-                all_streams.append(stream_data)
-
-    if requested_language:
-        if streams_unknown and not found_requested_language and not skip_unknown:
-            streams = streams_unknown
-    else:
-        streams = all_streams
+    if streams_unknown and not found_requested_language and not skip_unknown:
+        streams = streams_unknown
 
     return streams
 
@@ -301,9 +245,6 @@ def get_plex_metadata(rating_key, part_id, item_type, plex_item=None):
 
     stream_info = get_plexapi_stream_info(plex_item, part_id)
 
-    if not stream_info:
-        return
-
     # get normalized metadata
     # fixme: duplicated logic of media_to_videos
     if item_type == "episode":

Contents/Code/support/scanning.py

@@ -4,7 +4,7 @@ import helpers
 from babelfish.exceptions import LanguageError
 
 from support.lib import Plex, get_intent
-from support.plex_media import get_stream_fps, is_stream_forced, update_stream_info
+from support.plex_media import get_stream_fps
 from support.storage import get_subtitle_storage
 from support.config import config, TEXT_SUBTITLE_EXTS
 from support.subtitlehelpers import get_subtitles_from_metadata
@@ -46,25 +46,23 @@ def prepare_video(pms_video_info, ignore_all=False, hints=None, rating_key=None,
     # fixme: skip the whole scanning process if known_embedded == wanted languages?
     audio_languages = []
     if plexpy_part:
-        update_stream_info(plexpy_part)
         for stream in plexpy_part.streams:
             if stream.stream_type == 2:
                 lang = None
                 try:
                     lang = language_from_stream(stream.language_code)
                 except LanguageError:
-                    Log.Info("Couldn't detect embedded audio stream language: %s", stream.language_code)
+                    Log.Debug("Couldn't detect embedded audio stream language: %s", stream.language_code)
 
                 # treat unknown language as lang1?
                 if not lang and config.treat_und_as_first:
                     lang = Language.rebuild(list(config.lang_list)[0])
-                    Log.Info("Assuming language %s for audio stream: %s", lang, getattr(stream, "index", None))
 
                 audio_languages.append(lang)
 
             # subtitle stream
             elif stream.stream_type == 3 and embedded_subtitles:
-                is_forced = is_stream_forced(stream)
+                is_forced = helpers.is_stream_forced(stream)
 
                 if ((config.forced_only or config.forced_also) and is_forced) or not is_forced:
                     # embedded subtitle
@@ -75,13 +73,11 @@ def prepare_video(pms_video_info, ignore_all=False, hints=None, rating_key=None,
                             try:
                                 lang = language_from_stream(stream.language_code)
                             except LanguageError:
-                                Log.Info("Couldn't detect embedded subtitle stream language: %s", stream.language_code)
+                                Log.Debug("Couldn't detect embedded subtitle stream language: %s", stream.language_code)
 
                             # treat unknown language as lang1?
                             if not lang and config.treat_und_as_first:
                                 lang = Language.rebuild(list(config.lang_list)[0])
-                                Log.Info("Assuming language %s for subtitle stream: %s", lang,
-                                         getattr(stream, "index", None))
 
                             if lang:
                                 if is_forced:
@@ -131,8 +127,7 @@ def prepare_video(pms_video_info, ignore_all=False, hints=None, rating_key=None,
             set_existing_languages(video, pms_video_info, external_subtitles=external_subtitles,
                                    embedded_subtitles=embedded_subtitles, known_embedded=known_embedded,
                                    stored_subs=stored_subs, languages=config.lang_list,
-                                   only_one=config.only_one, known_metadata_subs=known_metadata_subs,
-                                   match_strictness=config.ext_match_strictness)
+                                   only_one=config.only_one, known_metadata_subs=known_metadata_subs)
 
         # add video fps info
         video.fps = plex_part.fps

Contents/Code/support/subtitlehelpers.py

@@ -5,7 +5,6 @@ import helpers
 
 from config import config, SUBTITLE_EXTS, TEXT_SUBTITLE_EXTS
 from bs4 import UnicodeDammit
-from subzero.language import match_ietf_language
 
 
 class SubtitleHelper(object):
@@ -86,6 +85,19 @@ class VobSubSubtitleHelper(SubtitleHelper):
 #####################################################################################################################
 
 
+IETF_MATCH = ".+\.([^-.]+)(?:-[A-Za-z]+)?$"
+ENDSWITH_LANGUAGECODE_RE = re.compile("\.([^-.]{2,3})(?:-[A-Za-z]{2,})?$")
+
+
+def match_ietf_language(s):
+    language_match = re.match(".+\.([^\.]+)$" if not helpers.cast_bool(Prefs["subtitles.language.ietf_display"])
+                              else IETF_MATCH, s)
+    if language_match and len(language_match.groups()) == 1:
+        language = language_match.groups()[0]
+        return language
+    return s
+
+
 class DefaultSubtitleHelper(SubtitleHelper):
     @classmethod
     def is_helper_for(cls, filename):
@@ -121,7 +133,7 @@ class DefaultSubtitleHelper(SubtitleHelper):
         # Attempt to extract the language from the filename (e.g. Avatar (2009).eng)
         # IETF support thanks to
         # https://github.com/hpsbranco/LocalMedia.bundle/commit/4fad9aefedece78a1fa96401304351347f644369
-        lang_part = match_ietf_language(file, ietf=helpers.cast_bool(Prefs["subtitles.language.ietf_display"]))
+        lang_part = match_ietf_language(file)
         if lang_part != file:
             language = Locale.Language.Match(lang_part)
         elif config.only_one:

Contents/DefaultPrefs.json

@@ -288,7 +288,7 @@
   },
   {
     "id": "anticaptcha.service",
-    "label": "AntiCaptcha-Service (needs paid account; enables Addic7ed)",
+    "label": "AntiCaptcha-Service (needs paid account; enables Addic7ed, titlovi)",
     "type": "enum",
     "values": [
       "none",
@@ -335,26 +335,6 @@
     "type": "bool",
     "default": "true"
   },
-  {
-    "id": "provider.napisy24.enabled",
-    "label": "Provider: Enable Napisy24 (pl)",
-    "type": "bool",
-    "default": "false"
-  },
-  {
-    "id": "provider.napisy24.username",
-    "label": "Napisy24 Username",
-    "type": "text",
-    "default": ""
-  },
-  {
-    "id": "provider.napisy24.password",
-    "label": "Napisy24 Password",
-    "type": "text",
-    "option": "hidden",
-    "default": "",
-    "secure": "true"
-  },
   {
     "id": "provider.addic7ed.enabled",
     "label": "Provider: Enable Addic7ed (needs AntiCaptcha)",
@@ -409,24 +389,10 @@
   },
   {
     "id": "provider.titlovi.enabled",
-    "label": "Provider: Enable Titlovi.com (User and Password required)",
+    "label": "Provider: Enable Titlovi.com (might need AntiCaptcha)",
     "type": "bool",
     "default": "true"
   },
-  {
-    "id": "provider.titlovi.username",
-    "label": "Titlovi Username",
-    "type": "text",
-    "default": ""
-  },
-  {
-    "id": "provider.titlovi.password",
-    "label": "Titlovi Password",
-    "type": "text",
-    "option": "hidden",
-    "default": "",
-    "secure": "true"
-  },
   {
     "id": "provider.legendastv.enabled",
     "label": "Provider: Enable Legendas TV (mostly pt-BR; UNRAR NEEDED)",
@@ -465,20 +431,6 @@
     "type": "bool",
     "default": "true"
   },
-  {
-    "id": "provider.subscene.username",
-    "label": "SubScene Username",
-    "type": "text",
-    "default": ""
-  },
-  {
-    "id": "provider.subscene.password",
-    "label": "SubScene Password",
-    "type": "text",
-    "option": "hidden",
-    "default": "",
-    "secure": "true"
-  },
   {
     "id": "provider.supersubtitles.enabled",
     "label": "Provider: Enable feliratok.info (Hungarian)",
@@ -909,7 +861,7 @@
   },
   {
     "id": "use_custom_dns2",
-    "label": "Use custom DNS (IPs, comma-separated, set to 'system' for system DNS. Default: Google/CF)",
+    "label": "Use custom DNS (IPs, comma-separated, leave empty for system DNS. Default: Google/CF)",
     "type": "text",
     "default": "1.1.1.1, 8.8.8.8"
   },

Contents/Info.plist

@@ -13,7 +13,7 @@
         <key>CFBundleSignature</key>
         <string>????</string>
         <key>CFBundleVersion</key>
-        <string>2.6.5.3183</string>
+        <string>2.6.5.3074</string>
         <key>PlexFrameworkVersion</key>
         <string>2</string>
         <key>PlexPluginClass</key>
@@ -23,7 +23,7 @@
         <key>PlexPluginConsoleLogging</key>
         <string>0</string>
         <key>PlexPluginDevMode</key>
-        <string>0</string>
+        <string>1</string>
          <key>PlexPluginCodePolicy</key>
             <!-- this allows channels to access some python methods which are otherwise blocked, as well as import external code libraries, and interact with the PMS HTTP API -->
             <string>Elevated</string>
@@ -32,7 +32,7 @@
 
 &lt;h1&gt;Sub-Zero for Plex&lt;/h1&gt;&lt;i&gt;Subtitles done right&lt;/i&gt;
 
-Version 2.6.5.3183
+Version 2.6.5.3074 DEV
 
 Originally based on @bramwalet's awesome &lt;a href=&quot;https://github.com/bramwalet/Subliminal.bundle&quot;&gt;Subliminal.bundle&lt;/a&gt;
 

Contents/Libraries/Shared/subliminal_patch/converters/subscene.py

@@ -12,13 +12,6 @@ from_subscene = {
         'Malay': 'msa', 'Pashto': 'pus', 'Punjabi': 'pan', 'Swahili': 'swa'
 }
 
-from_subscene_with_country = {
-    'Brazillian Portuguese': ('por', 'BR')
-}
-
-to_subscene_with_country = {val: key for key, val in from_subscene_with_country.items()}
-
-
 to_subscene = {v: k for k, v in from_subscene.items()}
 
 exact_languages_alpha3 = [
@@ -41,12 +34,12 @@ language_ids = {
         'mkd': 48, 'mal': 64, 'mni': 65, 'mon': 72, 'pus': 67, 'pol': 31,
         'por': 32, 'pan': 66, 'rus': 34, 'srp': 35, 'sin': 58, 'slk': 36,
         'slv': 37, 'som': 70, 'tgl': 53, 'tam': 59, 'tel': 63, 'tha': 40,
-        'tur': 41, 'ukr': 56, 'urd': 42, 'yor': 71, 'pt-BR': 4
+        'tur': 41, 'ukr': 56, 'urd': 42, 'yor': 71
 }
 
 # TODO: specify codes for unspecified_languages
 unspecified_languages = [
-        'Big 5 code', 'Bulgarian/ English',
+        'Big 5 code', 'Brazillian Portuguese', 'Bulgarian/ English',
         'Chinese BG code', 'Dutch/ English', 'English/ German',
         'Hungarian/ English', 'Rohingya'
 ]
@@ -57,8 +50,6 @@ alpha3_of_code = {l.name: l.alpha3 for l in supported_languages}
 
 supported_languages.update({Language(l) for l in to_subscene})
 
-supported_languages.update({Language(lang, cr) for lang, cr in to_subscene_with_country})
-
 
 class SubsceneConverter(LanguageReverseConverter):
     codes = {l.name for l in supported_languages}
@@ -70,15 +61,9 @@ class SubsceneConverter(LanguageReverseConverter):
         if alpha3 in to_subscene:
             return to_subscene[alpha3]
 
-        if (alpha3, country) in to_subscene_with_country:
-            return to_subscene_with_country[(alpha3, country)]
-
         raise ConfigurationError('Unsupported language for subscene: %s, %s, %s' % (alpha3, country, script))
 
     def reverse(self, code):
-        if code in from_subscene_with_country:
-            return from_subscene_with_country[code]
-
         if code in from_subscene:
             return (from_subscene[code],)
 

Contents/Libraries/Shared/subliminal_patch/converters/titlovi.py

@@ -27,6 +27,16 @@ class TitloviConverter(LanguageReverseConverter):
                            }
         self.codes = set(self.from_titlovi.keys())
 
+        # temporary fix, should be removed as soon as API is used
+        self.lang_from_countrycode = {'ba': ('bos',),
+                                 'en': ('eng',),
+                                 'hr': ('hrv',),
+                                 'mk': ('mkd',),
+                                 'rs': ('srp',),
+                                 'rsc': ('srp', None, 'Cyrl'),
+                                 'si': ('slv',)
+                                 }
+
     def convert(self, alpha3, country=None, script=None):
         if (alpha3, country, script) in self.to_titlovi:
             return self.to_titlovi[(alpha3, country, script)]
@@ -39,5 +49,9 @@ class TitloviConverter(LanguageReverseConverter):
         if titlovi in self.from_titlovi:
             return self.from_titlovi[titlovi]
 
+        # temporary fix, should be removed as soon as API is used
+        if titlovi in self.lang_from_countrycode:
+            return self.lang_from_countrycode[titlovi]
+
         raise ConfigurationError('Unsupported language number for titlovi: %s' % titlovi)
 

Contents/Libraries/Shared/subliminal_patch/core.py

@@ -30,7 +30,7 @@ from subliminal.core import guessit, ProviderPool, io, is_windows_special_path,
     ThreadPoolExecutor, check_video
 from subliminal_patch.exceptions import TooManyRequests, APIThrottled
 
-from subzero.language import Language, ENDSWITH_LANGUAGECODE_RE
+from subzero.language import Language
 from scandir import scandir, scandir_generic as _scandir_generic
 
 logger = logging.getLogger(__name__)
@@ -62,7 +62,7 @@ class SZProviderPool(ProviderPool):
     def __init__(self, providers=None, provider_configs=None, blacklist=None, throttle_callback=None,
                  pre_download_hook=None, post_download_hook=None, language_hook=None):
         #: Name of providers to use
-        self.providers = providers
+        self.providers = providers or provider_registry.names()
 
         #: Provider configuration
         self.provider_configs = provider_configs or {}
@@ -186,9 +186,12 @@ class SZProviderPool(ProviderPool):
         except (requests.Timeout, socket.timeout):
             logger.error('Provider %r timed out', provider)
 
-        except Exception as e:
-            logger.exception('Unexpected error in provider %r: %s', provider, traceback.format_exc())
+        except (TooManyRequests, DownloadLimitExceeded, ServiceUnavailable, APIThrottled), e:
             self.throttle_callback(provider, e)
+            return
+
+        except:
+            logger.exception('Unexpected error in provider %r: %s', provider, traceback.format_exc())
 
     def list_subtitles(self, video, languages):
         """List subtitles.
@@ -280,10 +283,14 @@ class SZProviderPool(ProviderPool):
                 logger.debug("RAR Traceback: %s", traceback.format_exc())
                 return False
 
-            except Exception as e:
+            except (TooManyRequests, DownloadLimitExceeded, ServiceUnavailable, APIThrottled), e:
+                self.throttle_callback(subtitle.provider_name, e)
+                self.discarded_providers.add(subtitle.provider_name)
+                return False
+
+            except:
                 logger.exception('Unexpected error in provider %r, Traceback: %s', subtitle.provider_name,
                                  traceback.format_exc())
-                self.throttle_callback(subtitle.provider_name, e)
                 self.discarded_providers.add(subtitle.provider_name)
                 return False
 
@@ -302,8 +309,7 @@ class SZProviderPool(ProviderPool):
             logger.error('Invalid subtitle')
             return False
 
-        if not os.environ.get("SZ_KEEP_ENCODING", False):
-            subtitle.normalize()
+        subtitle.normalize()
 
         return True
 
@@ -466,7 +472,7 @@ if is_windows_special_path:
     SZAsyncProviderPool = SZProviderPool
 
 
-def scan_video(path, dont_use_actual_file=False, hints=None, providers=None, skip_hashing=False, hash_from=None):
+def scan_video(path, dont_use_actual_file=False, hints=None, providers=None, skip_hashing=False):
     """Scan a video from a `path`.
 
     patch:
@@ -531,34 +537,28 @@ def scan_video(path, dont_use_actual_file=False, hints=None, providers=None, ski
             video.alternative_titles.append(alt_guess["title"])
         logger.debug("Adding alternative title: %s", alt_guess["title"])
 
-    if dont_use_actual_file and not hash_from:
+    if dont_use_actual_file:
         return video
 
     # size and hashes
     if not skip_hashing:
-        hash_path = hash_from or path
-        video.size = os.path.getsize(hash_path)
+        video.size = os.path.getsize(path)
         if video.size > 10485760:
             logger.debug('Size is %d', video.size)
-            osub_hash = None
             if "opensubtitles" in providers:
-                video.hashes['opensubtitles'] = osub_hash = hash_opensubtitles(hash_path)
+                video.hashes['opensubtitles'] = hash_opensubtitles(path)
 
             if "shooter" in providers:
-                video.hashes['shooter'] = hash_shooter(hash_path)
+                video.hashes['shooter'] = hash_shooter(path)
 
             if "thesubdb" in providers:
-                video.hashes['thesubdb'] = hash_thesubdb(hash_path)
+                video.hashes['thesubdb'] = hash_thesubdb(path)
 
             if "napiprojekt" in providers:
                 try:
-                    video.hashes['napiprojekt'] = hash_napiprojekt(hash_path)
+                    video.hashes['napiprojekt'] = hash_napiprojekt(path)
                 except MemoryError:
-                    logger.warning(u"Couldn't compute napiprojekt hash for %s", hash_path)
-
-            if "napisy24" in providers:
-                # Napisy24 uses the same hash as opensubtitles
-                video.hashes['napisy24'] = osub_hash or hash_opensubtitles(hash_path)
+                    logger.warning(u"Couldn't compute napiprojekt hash for %s", path)
 
             logger.debug('Computed hashes %r', video.hashes)
         else:
@@ -567,16 +567,14 @@ def scan_video(path, dont_use_actual_file=False, hints=None, providers=None, ski
     return video
 
 
-def _search_external_subtitles(path, languages=None, only_one=False, scandir_generic=False, match_strictness="strict"):
+def _search_external_subtitles(path, languages=None, only_one=False, scandir_generic=False):
     dirpath, filename = os.path.split(path)
     dirpath = dirpath or '.'
-    fn_no_ext, fileext = os.path.splitext(filename)
-    fn_no_ext_lower = fn_no_ext.lower()
+    fileroot, fileext = os.path.splitext(filename)
     subtitles = {}
     _scandir = _scandir_generic if scandir_generic else scandir
-
     for entry in _scandir(dirpath):
-        if (not entry.name or entry.name in ('\x0c', '$', ',', '\x7f')) and not scandir_generic:
+        if not entry.name and not scandir_generic:
             logger.debug('Could not determine the name of the file, retrying with scandir_generic')
             return _search_external_subtitles(path, languages, only_one, True)
         if not entry.is_file(follow_symlinks=False):
@@ -585,11 +583,9 @@ def _search_external_subtitles(path, languages=None, only_one=False, scandir_gen
         p = entry.name
 
         # keep only valid subtitle filenames
-        if not p.lower().endswith(SUBTITLE_EXTENSIONS):
+        if not p.lower().startswith(fileroot.lower()) or not p.lower().endswith(SUBTITLE_EXTENSIONS):
             continue
 
-        # not p.lower().startswith(fileroot.lower()) or not
-
         p_root, p_ext = os.path.splitext(p)
         if not INCLUDE_EXOTIC_SUBS and p_ext not in (".srt", ".ass", ".ssa", ".vtt"):
             continue
@@ -607,33 +603,22 @@ def _search_external_subtitles(path, languages=None, only_one=False, scandir_gen
         if adv_tag:
             forced = "forced" in adv_tag
 
-        # remove possible language code for matching
-        p_root_bare = ENDSWITH_LANGUAGECODE_RE.sub("", p_root)
-
-        p_root_lower = p_root_bare.lower()
-
-        filename_matches = p_root_lower == fn_no_ext_lower
-        filename_contains = p_root_lower in fn_no_ext_lower
-
-        if not filename_matches:
-            if match_strictness == "strict" or (match_strictness == "loose" and not filename_contains):
-                continue
-
-        language = None
-
         # extract the potential language code
-        try:
-            language_code = p_root.rsplit(".", 1)[1].replace('_', '-')
+        language_code = p_root[len(fileroot):].replace('_', '-')[1:]
+
+        # default language is undefined
+        language = Language('und')
+
+        # attempt to parse
+        if language_code:
             try:
                 language = Language.fromietf(language_code)
                 language.forced = forced
-            except (ValueError, LanguageReverseError):
+            except ValueError:
                 logger.error('Cannot parse language code %r', language_code)
-                language_code = None
-        except IndexError:
-            language_code = None
+                language = None
 
-        if not language and not language_code and only_one:
+        elif not language_code and only_one:
             language = Language.rebuild(list(languages)[0], forced=forced)
 
         subtitles[p] = language
@@ -643,7 +628,7 @@ def _search_external_subtitles(path, languages=None, only_one=False, scandir_gen
     return subtitles
 
 
-def search_external_subtitles(path, languages=None, only_one=False, match_strictness="strict"):
+def search_external_subtitles(path, languages=None, only_one=False):
     """
     wrap original search_external_subtitles function to search multiple paths for one given video
     # todo: cleanup and merge with _search_external_subtitles
@@ -664,11 +649,10 @@ def search_external_subtitles(path, languages=None, only_one=False, match_strict
         if os.path.isdir(os.path.dirname(abspath)):
             try:
                 subtitles.update(_search_external_subtitles(abspath, languages=languages,
-                                                            only_one=only_one, match_strictness=match_strictness))
+                                                            only_one=only_one))
             except OSError:
                 subtitles.update(_search_external_subtitles(abspath, languages=languages,
-                                                            only_one=only_one, match_strictness=match_strictness,
-                                                            scandir_generic=True))
+                                                            only_one=only_one, scandir_generic=True))
     logger.debug("external subs: found %s", subtitles)
     return subtitles
 
@@ -861,9 +845,6 @@ def save_subtitles(file_path, subtitles, single=False, directory=None, chmod=Non
             logger.debug(u"Saving %r to %r", subtitle, subtitle_path)
             content = subtitle.get_modified_content(format=format, debug=debug_mods)
             if content:
-                if os.path.exists(subtitle_path):
-                    os.remove(subtitle_path)
-
                 with open(subtitle_path, 'w') as f:
                     f.write(content)
                 subtitle.storage_path = subtitle_path

Contents/Libraries/Shared/subliminal_patch/exceptions.py

@@ -9,8 +9,3 @@ class TooManyRequests(ProviderError):
 
 class APIThrottled(ProviderError):
     pass
-
-
-class ParseResponseError(ProviderError):
-    """Exception raised by providers when they are not able to parse the response."""
-    pass

Contents/Libraries/Shared/subliminal_patch/http.py

@@ -354,6 +354,7 @@ def patch_create_connection():
                         return _orig_create_connection((ip, port), *args, **kwargs)
                     except dns.exception.DNSException:
                         logger.warning("DNS: Couldn't resolve %s with DNS: %s", host, custom_resolver.nameservers)
+                        raise
 
         logger.debug("DNS: Falling back to default DNS or IP on %s", host)
         return _orig_create_connection((host, port), *args, **kwargs)

Contents/Libraries/Shared/subliminal_patch/providers/addic7ed.py

@@ -6,13 +6,11 @@ import subliminal
 import time
 
 from random import randint
-
-from dogpile.cache.api import NO_VALUE
 from requests import Session
 from subliminal.cache import region
-from subliminal.exceptions import DownloadLimitExceeded, AuthenticationError, ConfigurationError
+from subliminal.exceptions import DownloadLimitExceeded, AuthenticationError
 from subliminal.providers.addic7ed import Addic7edProvider as _Addic7edProvider, \
-    Addic7edSubtitle as _Addic7edSubtitle, ParserBeautifulSoup
+    Addic7edSubtitle as _Addic7edSubtitle, ParserBeautifulSoup, show_cells_re
 from subliminal.subtitle import fix_line_ending
 from subliminal_patch.utils import sanitize
 from subliminal_patch.exceptions import TooManyRequests
@@ -21,8 +19,6 @@ from subzero.language import Language
 
 logger = logging.getLogger(__name__)
 
-show_cells_re = re.compile(b'<td class="(?:version|vr)">.*?</td>', re.DOTALL)
-
 #: Series header parsing regex
 series_year_re = re.compile(r'^(?P<series>[ \w\'.:(),*&!?-]+?)(?: \((?P<year>\d{4})\))?$')
 
@@ -70,15 +66,11 @@ class Addic7edProvider(_Addic7edProvider):
     server_url = 'https://www.addic7ed.com/'
 
     sanitize_characters = {'-', ':', '(', ')', '.', '/'}
-    last_show_ids_fetch_key = "addic7ed_last_id_fetch"
 
     def __init__(self, username=None, password=None, use_random_agents=False):
         super(Addic7edProvider, self).__init__(username=username, password=password)
         self.USE_ADDICTED_RANDOM_AGENTS = use_random_agents
 
-        if not all((username, password)):
-            raise ConfigurationError('Username and password must be specified')
-
     def initialize(self):
         self.session = Session()
         self.session.headers['User-Agent'] = 'Subliminal/%s' % subliminal.__short_version__
@@ -109,18 +101,13 @@ class Addic7edProvider(_Addic7edProvider):
                     'remember': 'true'}
 
             tries = 0
-            while tries <= 3:
-                tries += 1
+            while tries < 3:
                 r = self.session.get(self.server_url + 'login.php', timeout=10, headers={"Referer": self.server_url})
-                if "g-recaptcha" in r.content or "grecaptcha" in r.content:
+                if "grecaptcha" in r.content:
                     logger.info('Addic7ed: Solving captcha. This might take a couple of minutes, but should only '
                                 'happen once every so often')
 
-                    for g, s in (("g-recaptcha-response", r'g-recaptcha.+?data-sitekey=\"(.+?)\"'),
-                                 ("recaptcha_response", r'grecaptcha.execute\(\'(.+?)\',')):
-                        site_key = re.search(s, r.content).group(1)
-                        if site_key:
-                            break
+                    site_key = re.search(r'grecaptcha.execute\(\'(.+?)\',', r.content).group(1)
                     if not site_key:
                         logger.error("Addic7ed: Captcha site-key not found!")
                         return
@@ -132,31 +119,23 @@ class Addic7edProvider(_Addic7edProvider):
 
                     result = pitcher.throw()
                     if not result:
-                        if tries >= 3:
-                            raise Exception("Addic7ed: Couldn't solve captcha!")
-                        logger.info("Addic7ed: Couldn't solve captcha! Retrying")
-                        time.sleep(4)
-                        continue
+                        raise Exception("Addic7ed: Couldn't solve captcha!")
 
-                    data[g] = result
+                    data["recaptcha_response"] = result
 
-                time.sleep(1)
                 r = self.session.post(self.server_url + 'dologin.php', data, allow_redirects=False, timeout=10,
                                       headers={"Referer": self.server_url + "login.php"})
 
                 if "relax, slow down" in r.content:
                     raise TooManyRequests(self.username)
 
-                if "Wrong password" in r.content or "doesn't exist" in r.content:
-                    raise AuthenticationError(self.username)
-
                 if r.status_code != 302:
-                    if tries >= 3:
-                        logger.error("Addic7ed: Something went wrong when logging in")
-                        raise AuthenticationError(self.username)
-                    logger.info("Addic7ed: Something went wrong when logging in; retrying")
-                    time.sleep(4)
-                    continue
+                    if "User <b></b> doesn't exist" in r.content and tries <= 2:
+                        logger.info("Addic7ed: Error, trying again. (%s/%s)", tries+1, 3)
+                        tries += 1
+                        continue
+
+                    raise AuthenticationError(self.username)
                 break
 
             store_verification("addic7ed", self.session)
@@ -164,12 +143,10 @@ class Addic7edProvider(_Addic7edProvider):
             logger.debug('Addic7ed: Logged in')
             self.logged_in = True
 
-            time.sleep(2)
-
     def terminate(self):
         self.session.close()
 
-    def get_show_id(self, series, year=None, country_code=None, ignore_cache=False):
+    def get_show_id(self, series, year=None, country_code=None):
         """Get the best matching show id for `series`, `year` and `country_code`.
 
         First search in the result of :meth:`_get_show_ids` and fallback on a search with :meth:`_search_show_id`.
@@ -181,45 +158,32 @@ class Addic7edProvider(_Addic7edProvider):
         :type country_code: str
         :return: the show id, if found.
         :rtype: int
+
         """
-        show_id = None
-        ids_to_look_for = {sanitize(series).lower(), sanitize(series.replace(".", "")).lower()}
+        series_sanitized = sanitize(series).lower()
         show_ids = self._get_show_ids()
-        if ignore_cache or not show_ids:
-            show_ids = self._get_show_ids.refresh(self)
+        show_id = None
 
-        logger.debug("Trying show ids: %s", ids_to_look_for)
-        for series_sanitized in ids_to_look_for:
-            # attempt with country
-            if not show_id and country_code:
-                logger.debug('Getting show id with country')
-                show_id = show_ids.get('%s %s' % (series_sanitized, country_code.lower()))
+        # attempt with country
+        if not show_id and country_code:
+            logger.debug('Getting show id with country')
+            show_id = show_ids.get('%s %s' % (series_sanitized, country_code.lower()))
 
-            # attempt with year
-            if not show_id and year:
-                logger.debug('Getting show id with year')
-                show_id = show_ids.get('%s %d' % (series_sanitized, year))
+        # attempt with year
+        if not show_id and year:
+            logger.debug('Getting show id with year')
+            show_id = show_ids.get('%s %d' % (series_sanitized, year))
 
-            # attempt clean
-            if not show_id:
-                logger.debug('Getting show id')
-                show_id = show_ids.get(series_sanitized)
+        # attempt clean
+        if not show_id:
+            logger.debug('Getting show id')
+            show_id = show_ids.get(series_sanitized)
 
-                if not show_id:
-                    now = datetime.datetime.now()
-                    last_fetch = region.get(self.last_show_ids_fetch_key)
-
-                    # re-fetch show ids once per day if any show ID not found
-                    if not ignore_cache and last_fetch != NO_VALUE and last_fetch + datetime.timedelta(days=1) < now:
-                        logger.info("Show id not found; re-fetching show ids")
-                        return self.get_show_id(series, year=year, country_code=country_code, ignore_cache=True)
-                    logger.debug("Not refreshing show ids, as the last fetch has been too recent")
-
-            # search as last resort
-            # broken right now
-            # if not show_id:
-            #     logger.warning('Series %s not found in show ids', series)
-            #     show_id = self._search_show_id(series)
+        # search as last resort
+        # broken right now
+        # if not show_id:
+        #     logger.warning('Series %s not found in show ids', series)
+        #     show_id = self._search_show_id(series)
 
         return show_id
 
@@ -233,8 +197,6 @@ class Addic7edProvider(_Addic7edProvider):
         """
         # get the show page
         logger.info('Getting show ids')
-        region.set(self.last_show_ids_fetch_key, datetime.datetime.now())
-
         r = self.session.get(self.server_url + 'shows.php', timeout=10)
         r.raise_for_status()
 
@@ -243,15 +205,14 @@ class Addic7edProvider(_Addic7edProvider):
         # Assuming the site's markup is bad, and stripping it down to only contain what's needed.
         show_cells = re.findall(show_cells_re, r.content)
         if show_cells:
-            soup = ParserBeautifulSoup(b''.join(show_cells).decode('utf-8', 'ignore'), ['lxml', 'html.parser'])
+            soup = ParserBeautifulSoup(b''.join(show_cells), ['lxml', 'html.parser'])
         else:
             # If RegEx fails, fall back to original r.content and use 'html.parser'
             soup = ParserBeautifulSoup(r.content, ['html.parser'])
 
         # populate the show ids
         show_ids = {}
-        shows = soup.select('td > h3 > a[href^="/show/"]')
-        for show in shows:
+        for show in soup.select('td > h3 > a[href^="/show/"]'):
             show_clean = sanitize(show.text, default_characters=self.sanitize_characters)
             try:
                 show_id = int(show['href'][6:])
@@ -269,9 +230,6 @@ class Addic7edProvider(_Addic7edProvider):
 
         logger.debug('Found %d show ids', len(show_ids))
 
-        if not show_ids:
-            raise Exception("Addic7ed: No show IDs found!")
-
         return show_ids
 
     @region.cache_on_arguments(expiration_time=SHOW_EXPIRATION_TIME)
@@ -371,7 +329,7 @@ class Addic7edProvider(_Addic7edProvider):
 
             # ignore incomplete subtitles
             status = cells[5].text
-            if "%" in status:
+            if status != 'Completed':
                 logger.debug('Ignoring subtitle with status %s', status)
                 continue
 

Contents/Libraries/Shared/subliminal_patch/providers/argenteam.py

@@ -23,10 +23,9 @@ class ArgenteamSubtitle(Subtitle):
     hearing_impaired_verifiable = False
     _release_info = None
 
-    def __init__(self, language, page_link, download_link, movie_kind, title, season, episode, year, release, version, source,
+    def __init__(self, language, download_link, movie_kind, title, season, episode, year, release, version, source,
                  video_codec, tvdb_id, imdb_id, asked_for_episode=None, asked_for_release_group=None, *args, **kwargs):
-        super(ArgenteamSubtitle, self).__init__(language, page_link=page_link, *args, **kwargs)
-        self.page_link = page_link
+        super(ArgenteamSubtitle, self).__init__(language, download_link, *args, **kwargs)
         self.download_link = download_link
         self.movie_kind = movie_kind
         self.title = title
@@ -136,8 +135,7 @@ class ArgenteamProvider(Provider, ProviderSubtitleArchiveMixin):
     provider_name = 'argenteam'
     languages = {Language.fromalpha2(l) for l in ['es']}
     video_types = (Episode, Movie)
-    BASE_URL = "https://argenteam.net/"
-    API_URL = BASE_URL + "api/v1/"
+    API_URL = "http://argenteam.net/api/v1/"
     subtitle_class = ArgenteamSubtitle
     hearing_impaired_verifiable = False
     language_list = list(languages)
@@ -242,15 +240,12 @@ class ArgenteamProvider(Provider, ProviderSubtitleArchiveMixin):
 
             for r in content['releases']:
                 for s in r['subtitles']:
-                    movie_kind = "episode" if is_episode else "movie"
-                    page_link = self.BASE_URL + movie_kind + "/" + str(aid)
-                    # use https and new domain
-                    download_link = s['uri'].replace('http://www.argenteam.net/', self.BASE_URL)
-                    sub = ArgenteamSubtitle(language, page_link, download_link, movie_kind, returned_title,
+                    sub = ArgenteamSubtitle(language, s['uri'], "episode" if is_episode else "movie", returned_title,
                                             season, episode, year, r.get('team'), r.get('tags'),
                                             r.get('source'), r.get('codec'), content.get("tvdb"), imdb_id,
                                             asked_for_release_group=video.release_group,
-                                            asked_for_episode=episode)
+                                            asked_for_episode=episode
+                                            )
                     subtitles.append(sub)
 
             if has_multiple_ids:

Contents/Libraries/Shared/subliminal_patch/providers/napisy24.py

@@ -1,124 +0,0 @@
-import logging
-import os
-from io import BytesIO
-from zipfile import ZipFile
-
-from requests import Session
-
-from subliminal_patch.subtitle import Subtitle
-from subliminal_patch.providers import Provider
-from subliminal import __short_version__
-from subliminal.exceptions import AuthenticationError, ConfigurationError
-from subliminal.subtitle import fix_line_ending
-from subzero.language import Language
-
-logger = logging.getLogger(__name__)
-
-
-class Napisy24Subtitle(Subtitle):
-    '''Napisy24 Subtitle.'''
-    provider_name = 'napisy24'
-
-    def __init__(self, language, hash, imdb_id, napis_id):
-        super(Napisy24Subtitle, self).__init__(language)
-        self.hash = hash
-        self.imdb_id = imdb_id
-        self.napis_id = napis_id
-
-    @property
-    def id(self):
-        return self.hash
-
-    def get_matches(self, video):
-        matches = set()
-
-        # hash
-        if 'napisy24' in video.hashes and video.hashes['napisy24'] == self.hash:
-            matches.add('hash')
-
-        # imdb_id
-        if video.imdb_id and self.imdb_id == video.imdb_id:
-            matches.add('imdb_id')
-
-        return matches
-
-
-class Napisy24Provider(Provider):
-    '''Napisy24 Provider.'''
-    languages = {Language(l) for l in ['pol']}
-    required_hash = 'napisy24'
-    api_url = 'http://napisy24.pl/run/CheckSubAgent.php'
-
-    def __init__(self, username=None, password=None):
-        if all((username, password)):
-            self.username = username
-            self.password = password
-        else:
-            self.username = 'subliminal'
-            self.password = 'lanimilbus'
-
-        self.session = None
-
-    def initialize(self):
-        self.session = Session()
-        self.session.headers['User-Agent'] = 'Subliminal/%s' % __short_version__
-        self.session.headers['Content-Type'] = 'application/x-www-form-urlencoded'
-
-    def terminate(self):
-        self.session.close()
-
-    def query(self, language, size, name, hash):
-        params = {
-            'postAction': 'CheckSub',
-            'ua': self.username,
-            'ap': self.password,
-            'fs': size,
-            'fh': hash,
-            'fn': os.path.basename(name),
-            'n24pref': 1
-        }
-
-        response = self.session.post(self.api_url, data=params, timeout=10)
-        response.raise_for_status()
-
-        response_content = response.content.split(b'||', 1)
-        n24_data = response_content[0].decode()
-
-        if n24_data[:2] != 'OK':
-            if n24_data[:11] == 'login error':
-                raise AuthenticationError('Login failed')
-            logger.error('Unknown response: %s', response.content)
-            return None
-
-        n24_status = n24_data[:4]
-        if n24_status == 'OK-0':
-            logger.info('No subtitles found')
-            return None
-
-        subtitle_info = dict(p.split(':', 1) for p in n24_data.split('|')[1:])
-        logger.debug('Subtitle info: %s', subtitle_info)
-
-        if n24_status == 'OK-1':
-            logger.info('No subtitles found but got video info')
-            return None
-        elif n24_status == 'OK-2':
-            logger.info('Found subtitles')
-        elif n24_status == 'OK-3':
-            logger.info('Found subtitles but not from Napisy24 database')
-            return None
-        
-        subtitle_content = response_content[1]
-
-        subtitle = Napisy24Subtitle(language, hash, 'tt%s' % subtitle_info['imdb'].zfill(7), subtitle_info['napisId'])
-        with ZipFile(BytesIO(subtitle_content)) as zf:
-            subtitle.content = fix_line_ending(zf.open(zf.namelist()[0]).read())
-
-        return subtitle
-
-    def list_subtitles(self, video, languages):
-        subtitles = [self.query(l, video.size, video.name, video.hashes['napisy24']) for l in languages]
-        return [s for s in subtitles if s is not None]
-
-    def download_subtitle(self, subtitle):
-        # there is no download step, content is already filled from listing subtitles
-        pass

Contents/Libraries/Shared/subliminal_patch/providers/subscene.py

@@ -4,34 +4,24 @@ import io
 import logging
 import os
 import time
-import traceback
-
-import requests
-
 import inflect
-import re
-import json
-import HTMLParser
-import urlparse
 
 from zipfile import ZipFile
 from babelfish import language_converters
 from guessit import guessit
-from dogpile.cache.api import NO_VALUE
 from subliminal import Episode, ProviderError
-from subliminal.exceptions import ConfigurationError, ServiceUnavailable
 from subliminal.utils import sanitize_release_group
-from subliminal.cache import region
 from subliminal_patch.http import RetryingCFSession
 from subliminal_patch.providers import Provider
 from subliminal_patch.providers.mixins import ProviderSubtitleArchiveMixin
 from subliminal_patch.subtitle import Subtitle, guess_matches
 from subliminal_patch.converters.subscene import language_ids, supported_languages
-from subscene_api.subscene import search, Subtitle as APISubtitle, SITE_DOMAIN
+from subscene_api.subscene import search, Subtitle as APISubtitle
 from subzero.language import Language
 
 p = inflect.engine()
 
+
 language_converters.register('subscene = subliminal_patch.converters.subscene:SubsceneConverter')
 logger = logging.getLogger(__name__)
 
@@ -122,106 +112,28 @@ class SubsceneProvider(Provider, ProviderSubtitleArchiveMixin):
     skip_wrong_fps = False
     hearing_impaired_verifiable = True
     only_foreign = False
-    username = None
-    password = None
 
-    search_throttle = 8  # seconds
-
-    def __init__(self, only_foreign=False, username=None, password=None):
-        if not all((username, password)):
-            raise ConfigurationError('Username and password must be specified')
+    search_throttle = 2  # seconds
 
+    def __init__(self, only_foreign=False):
         self.only_foreign = only_foreign
-        self.username = username
-        self.password = password
 
     def initialize(self):
         logger.info("Creating session")
         self.session = RetryingCFSession()
 
-        prev_cookies = region.get("subscene_cookies2")
-        if prev_cookies != NO_VALUE:
-            logger.debug("Re-using old subscene cookies: %r", prev_cookies)
-            self.session.cookies.update(prev_cookies)
-
-        else:
-            logger.debug("Logging in")
-            self.login()
-
-    def login(self):
-        r = self.session.get("https://subscene.com/account/login")
-        if "Server Error" in r.content:
-            logger.error("Login unavailable; Maintenance?")
-            raise ServiceUnavailable("Login unavailable; Maintenance?")
-
-        match = re.search(r"<script id='modelJson' type='application/json'>\s*(.+)\s*</script>", r.content)
-
-        if match:
-            h = HTMLParser.HTMLParser()
-            data = json.loads(h.unescape(match.group(1)))
-            login_url = urlparse.urljoin(data["siteUrl"], data["loginUrl"])
-            time.sleep(1.0)
-
-            r = self.session.post(login_url,
-                                  {
-                                      "username": self.username,
-                                      "password": self.password,
-                                      data["antiForgery"]["name"]: data["antiForgery"]["value"]
-                                  })
-            pep_content = re.search(r"<form method=\"post\" action=\"https://subscene\.com/\">"
-                                    r".+name=\"id_token\".+?value=\"(?P<id_token>.+?)\".*?"
-                                    r"access_token\".+?value=\"(?P<access_token>.+?)\".+?"
-                                    r"token_type.+?value=\"(?P<token_type>.+?)\".+?"
-                                    r"expires_in.+?value=\"(?P<expires_in>.+?)\".+?"
-                                    r"scope.+?value=\"(?P<scope>.+?)\".+?"
-                                    r"state.+?value=\"(?P<state>.+?)\".+?"
-                                    r"session_state.+?value=\"(?P<session_state>.+?)\"",
-                                    r.content, re.MULTILINE | re.DOTALL)
-
-            if pep_content:
-                r = self.session.post(SITE_DOMAIN, pep_content.groupdict())
-                try:
-                    r.raise_for_status()
-                except Exception:
-                    raise ProviderError("Something went wrong when trying to log in: %s", traceback.format_exc())
-                else:
-                    cj = self.session.cookies.copy()
-                    store_cks = ("scene", "idsrv", "idsrv.xsrf", "idsvr.clients", "idsvr.session", "idsvr.username")
-                    for cn in self.session.cookies.iterkeys():
-                        if cn not in store_cks:
-                            del cj[cn]
-
-                    logger.debug("Storing cookies: %r", cj)
-                    region.set("subscene_cookies2", cj)
-                    return
-        raise ProviderError("Something went wrong when trying to log in #1")
-
     def terminate(self):
         logger.info("Closing session")
         self.session.close()
 
     def _create_filters(self, languages):
         self.filters = dict(HearingImpaired="2")
-        acc_filters = self.filters.copy()
         if self.only_foreign:
             self.filters["ForeignOnly"] = "True"
-            acc_filters["ForeignOnly"] = self.filters["ForeignOnly"].lower()
             logger.info("Only searching for foreign/forced subtitles")
 
-        selected_ids = []
-        for l in languages:
-            lid = language_ids.get(l.basename, language_ids.get(l.alpha3, None))
-            if lid:
-                selected_ids.append(str(lid))
-
-        acc_filters["SelectedIds"] = selected_ids
-        self.filters["LanguageFilter"] = ",".join(acc_filters["SelectedIds"])
-
-        last_filters = region.get("subscene_filters")
-        if last_filters != acc_filters:
-            region.set("subscene_filters", acc_filters)
-            logger.debug("Setting account filters to %r", acc_filters)
-            self.session.post("https://u.subscene.com/filter", acc_filters, allow_redirects=False)
+        self.filters["LanguageFilter"] = ",".join((str(language_ids[l.alpha3]) for l in languages
+                                                   if l.alpha3 in language_ids))
 
         logger.debug("Filter created: '%s'" % self.filters)
 
@@ -264,11 +176,7 @@ class SubsceneProvider(Provider, ProviderSubtitleArchiveMixin):
     def parse_results(self, video, film):
         subtitles = []
         for s in film.subtitles:
-            try:
-                subtitle = SubsceneSubtitle.from_api(s)
-            except NotImplementedError, e:
-                logger.info(e)
-                continue
+            subtitle = SubsceneSubtitle.from_api(s)
             subtitle.asked_for_release_group = video.release_group
             if isinstance(video, Episode):
                 subtitle.asked_for_episode = video.episode
@@ -281,16 +189,10 @@ class SubsceneProvider(Provider, ProviderSubtitleArchiveMixin):
 
         return subtitles
 
-    def do_search(self, *args, **kwargs):
-        try:
-            return search(*args, **kwargs)
-        except requests.HTTPError:
-            region.delete("subscene_cookies2")
-
     def query(self, video):
-        # vfn = get_video_filename(video)
+        vfn = get_video_filename(video)
         subtitles = []
-        # logger.debug(u"Searching for: %s", vfn)
+        #logger.debug(u"Searching for: %s", vfn)
         # film = search(vfn, session=self.session)
         #
         # if film and film.subtitles:
@@ -299,17 +201,16 @@ class SubsceneProvider(Provider, ProviderSubtitleArchiveMixin):
         # else:
         #     logger.debug('No release results found')
 
-        # time.sleep(self.search_throttle)
+        #time.sleep(self.search_throttle)
 
         # re-search for episodes without explicit release name
         if isinstance(video, Episode):
-            titles = list(set([video.series] + video.alternative_series))[:2]
-            # term = u"%s S%02iE%02i" % (video.series, video.season, video.episode)
-            more_than_one = len(titles) > 1
-            for series in titles:
+            #term = u"%s S%02iE%02i" % (video.series, video.season, video.episode)
+            more_than_one = len([video.series] + video.alternative_series) > 1
+            for series in [video.series] + video.alternative_series:
                 term = u"%s - %s Season" % (series, p.number_to_words("%sth" % video.season).capitalize())
                 logger.debug('Searching for alternative results: %s', term)
-                film = self.do_search(term, session=self.session, release=False, throttle=self.search_throttle)
+                film = search(term, session=self.session, release=False, throttle=self.search_throttle)
                 if film and film.subtitles:
                     logger.debug('Alternative results found: %s', len(film.subtitles))
                     subtitles += self.parse_results(video, film)
@@ -317,27 +218,26 @@ class SubsceneProvider(Provider, ProviderSubtitleArchiveMixin):
                     logger.debug('No alternative results found')
 
                 # packs
-                # if video.season_fully_aired:
-                #     term = u"%s S%02i" % (series, video.season)
-                #     logger.debug('Searching for packs: %s', term)
-                #     time.sleep(self.search_throttle)
-                #     film = search(term, session=self.session, throttle=self.search_throttle)
-                #     if film and film.subtitles:
-                #         logger.debug('Pack results found: %s', len(film.subtitles))
-                #         subtitles += self.parse_results(video, film)
-                #     else:
-                #         logger.debug('No pack results found')
-                # else:
-                #     logger.debug("Not searching for packs, because the season hasn't fully aired")
+                if video.season_fully_aired:
+                    term = u"%s S%02i" % (series, video.season)
+                    logger.debug('Searching for packs: %s', term)
+                    time.sleep(self.search_throttle)
+                    film = search(term, session=self.session, throttle=self.search_throttle)
+                    if film and film.subtitles:
+                        logger.debug('Pack results found: %s', len(film.subtitles))
+                        subtitles += self.parse_results(video, film)
+                    else:
+                        logger.debug('No pack results found')
+                else:
+                    logger.debug("Not searching for packs, because the season hasn't fully aired")
                 if more_than_one:
                     time.sleep(self.search_throttle)
         else:
-            titles = list(set([video.title] + video.alternative_titles))[:2]
-            more_than_one = len(titles) > 1
-            for title in titles:
-                logger.debug('Searching for movie results: %r', title)
-                film = self.do_search(title, year=video.year, session=self.session, limit_to=None, release=False,
-                                      throttle=self.search_throttle)
+            more_than_one = len([video.title] + video.alternative_titles) > 1
+            for title in [video.title] + video.alternative_titles:
+                logger.debug('Searching for movie results: %s', title)
+                film = search(title, year=video.year, session=self.session, limit_to=None, release=False,
+                              throttle=self.search_throttle)
                 if film and film.subtitles:
                     subtitles += self.parse_results(video, film)
                 if more_than_one:

Contents/Libraries/Shared/subliminal_patch/providers/titlovi.py

@@ -2,35 +2,42 @@
 
 import io
 import logging
+import math
 import re
-from datetime import datetime
-import dateutil.parser
+import time
 
 import rarfile
 
+from bs4 import BeautifulSoup
 from zipfile import ZipFile, is_zipfile
 from rarfile import RarFile, is_rarfile
 from babelfish import language_converters, Script
-from requests import RequestException, codes as request_codes
+from requests import RequestException
 from guessit import guessit
 from subliminal_patch.http import RetryingCFSession
 from subliminal_patch.providers import Provider
 from subliminal_patch.providers.mixins import ProviderSubtitleArchiveMixin
 from subliminal_patch.subtitle import Subtitle
 from subliminal_patch.utils import sanitize, fix_inconsistent_naming as _fix_inconsistent_naming
-from subliminal.exceptions import ProviderError, AuthenticationError, ConfigurationError
+from subliminal.exceptions import ProviderError
 from subliminal.score import get_equivalent_release_groups
 from subliminal.utils import sanitize_release_group
 from subliminal.subtitle import guess_matches
 from subliminal.video import Episode, Movie
 from subliminal.subtitle import fix_line_ending
-
+from subliminal_patch.pitcher import pitchers, load_verification, store_verification
 from subzero.language import Language
-from dogpile.cache.api import NO_VALUE
-from subliminal.cache import region
+
+from random import randint
+from .utils import FIRST_THOUSAND_OR_SO_USER_AGENTS as AGENT_LIST
 
 # parsing regex definitions
 title_re = re.compile(r'(?P<title>(?:.+(?= [Aa][Kk][Aa] ))|.+)(?:(?:.+)(?P<altitle>(?<= [Aa][Kk][Aa] ).+))?')
+lang_re = re.compile(r'(?<=flags/)(?P<lang>.{2})(?:.)(?P<script>c?)(?:.+)')
+season_re = re.compile(r'Sezona (?P<season>\d+)')
+episode_re = re.compile(r'Epizoda (?P<episode>\d+)')
+year_re = re.compile(r'(?P<year>\d+)')
+fps_re = re.compile(r'fps: (?P<fps>.+)')
 
 
 def fix_inconsistent_naming(title):
@@ -44,7 +51,6 @@ def fix_inconsistent_naming(title):
     return _fix_inconsistent_naming(title, {"DC's Legends of Tomorrow": "Legends of Tomorrow",
                                             "Marvel's Jessica Jones": "Jessica Jones"})
 
-
 logger = logging.getLogger(__name__)
 
 # Configure :mod:`rarfile` to use the same path separator as :mod:`zipfile`
@@ -56,9 +62,9 @@ language_converters.register('titlovi = subliminal_patch.converters.titlovi:Titl
 class TitloviSubtitle(Subtitle):
     provider_name = 'titlovi'
 
-    def __init__(self, language, download_link, sid, releases, title, alt_title=None, season=None,
-                 episode=None, year=None, rating=None, download_count=None, asked_for_release_group=None, asked_for_episode=None):
-        super(TitloviSubtitle, self).__init__(language)
+    def __init__(self, language, page_link, download_link, sid, releases, title, alt_title=None, season=None,
+                 episode=None, year=None, fps=None, asked_for_release_group=None, asked_for_episode=None):
+        super(TitloviSubtitle, self).__init__(language, page_link=page_link)
         self.sid = sid
         self.releases = self.release_info = releases
         self.title = title
@@ -67,21 +73,11 @@ class TitloviSubtitle(Subtitle):
         self.episode = episode
         self.year = year
         self.download_link = download_link
-        self.rating = rating
-        self.download_count = download_count
+        self.fps = fps
         self.matches = None
         self.asked_for_release_group = asked_for_release_group
         self.asked_for_episode = asked_for_episode
 
-    def __repr__(self):
-        if self.season and self.episode:
-            return '<%s "%s (%r)" s%.2de%.2d [%s:%s] ID:%r R:%.2f D:%r>' % (
-                self.__class__.__name__, self.title, self.year, self.season, self.episode, self.language, self._guessed_encoding, self.sid,
-                self.rating, self.download_count)
-        else:
-            return '<%s "%s (%r)" [%s:%s] ID:%r R:%.2f D:%r>' % (
-                self.__class__.__name__, self.title, self.year,  self.language, self._guessed_encoding, self.sid, self.rating, self.download_count)
-
     @property
     def id(self):
         return self.sid
@@ -138,62 +134,20 @@ class TitloviSubtitle(Subtitle):
 class TitloviProvider(Provider, ProviderSubtitleArchiveMixin):
     subtitle_class = TitloviSubtitle
     languages = {Language.fromtitlovi(l) for l in language_converters['titlovi'].codes} | {Language.fromietf('sr-Latn')}
-    api_url = 'https://kodi.titlovi.com/api/subtitles'
-    api_gettoken_url = api_url + '/gettoken'
-    api_search_url = api_url + '/search'
-
-    def __init__(self, username=None, password=None):
-        if not all((username, password)):
-            raise ConfigurationError('Username and password must be specified')
-
-        self.username = username
-        self.password = password
-
-        self.session = None
-
-        self.user_id = None
-        self.login_token = None
-        self.token_exp = None
+    server_url = 'https://titlovi.com'
+    search_url = server_url + '/titlovi/?'
+    download_url = server_url + '/download/?type=1&mediaid='
 
     def initialize(self):
         self.session = RetryingCFSession()
         #load_verification("titlovi", self.session)
 
-        token = region.get("titlovi_token")
-        if token is not NO_VALUE:
-            self.user_id, self.login_token, self.token_exp = token
-            if datetime.now() > self.token_exp:
-                logger.debug('Token expired')
-                self.log_in()
-            else:
-                logger.debug('Use cached token')
-        else:
-            logger.debug('Token not found in cache')
-            self.log_in()
-
-    def log_in(self):
-        login_params = dict(username=self.username, password=self.password, json=True)
-        try:
-            response = self.session.post(self.api_gettoken_url, params=login_params)
-            if response.status_code == request_codes.ok:
-                resp_json = response.json()
-                self.login_token = resp_json.get('Token')
-                self.user_id = resp_json.get('UserId')
-                self.token_exp = dateutil.parser.parse(resp_json.get('ExpirationDate'))
-
-                region.set("titlovi_token", [self.user_id, self.login_token, self.token_exp])
-                logger.debug('New token obtained')
-
-            elif response.status_code == request_codes.unauthorized:
-                raise AuthenticationError('Login failed')
-
-        except RequestException as e:
-            logger.error(e)
     def terminate(self):
         self.session.close()
 
-    def query(self, languages, title, season=None, episode=None, year=None, imdb_id=None, video=None):
-        search_params = dict()
+    def query(self, languages, title, season=None, episode=None, year=None, video=None):
+        items_per_page = 10
+        current_page = 1
 
         used_languages = languages
         lang_strings = [str(lang) for lang in used_languages]
@@ -208,73 +162,135 @@ class TitloviProvider(Provider, ProviderSubtitleArchiveMixin):
         langs = '|'.join(map(str, [l.titlovi for l in used_languages]))
 
         # set query params
-        search_params['query'] = title
-        search_params['lang'] = langs
+        params = {'prijevod': title, 'jezik': langs}
         is_episode = False
         if season and episode:
             is_episode = True
-            search_params['season'] = season
-            search_params['episode'] = episode
-        #if year:
-        #    search_params['year'] = year
-        if imdb_id:
-            search_params['imdbID'] = imdb_id
+            params['s'] = season
+            params['e'] = episode
+        if year:
+            params['g'] = year
 
         # loop through paginated results
-        logger.info('Searching subtitles %r', search_params)
+        logger.info('Searching subtitles %r', params)
         subtitles = []
-        query_results = []
 
-        try:
-            search_params['token'] = self.login_token
-            search_params['userid'] = self.user_id
-            search_params['json'] = True
-
-            response = self.session.get(self.api_search_url, params=search_params)
-            resp_json = response.json()
-            if resp_json['SubtitleResults']:
-                query_results.extend(resp_json['SubtitleResults'])
-
-
-        except Exception as e:
-            logger.error(e)
-
-        for sub in query_results:
-
-            # title and alternate title
-            match = title_re.search(sub.get('Title'))
-            if match:
-                _title = match.group('title')
-                alt_title = match.group('altitle')
+        while True:
+            # query the server
+            try:
+                r = self.session.get(self.search_url, params=params, timeout=10)
+                r.raise_for_status()
+            except RequestException as e:
+                logger.exception('RequestException %s', e)
+                break
             else:
-                continue
+                try:
+                    soup = BeautifulSoup(r.content, 'lxml')
 
-            # handle movies and series separately
-            if is_episode:
-                subtitle = self.subtitle_class(Language.fromtitlovi(sub.get('Lang')), sub.get('Link'), sub.get('Id'), sub.get('Release'), _title,
-                                               alt_title=alt_title, season=sub.get('Season'), episode=sub.get('Episode'),
-                                               year=sub.get('Year'), rating=sub.get('Rating'),
-                                               download_count=sub.get('DownloadCount'),
-                                               asked_for_release_group=video.release_group,
-                                               asked_for_episode=episode)
-            else:
-                subtitle = self.subtitle_class(Language.fromtitlovi(sub.get('Lang')), sub.get('Link'), sub.get('Id'), sub.get('Release'), _title,
-                                               alt_title=alt_title, year=sub.get('Year'), rating=sub.get('Rating'),
-                                               download_count=sub.get('DownloadCount'),
-                                               asked_for_release_group=video.release_group)
-            logger.debug('Found subtitle %r', subtitle)
+                    # number of results
+                    result_count = int(soup.select_one('.results_count b').string)
+                except:
+                    result_count = None
 
-            # prime our matches so we can use the values later
-            subtitle.get_matches(video)
+                # exit if no results
+                if not result_count:
+                    if not subtitles:
+                        logger.debug('No subtitles found')
+                    else:
+                        logger.debug("No more subtitles found")
+                    break
 
-            # add found subtitles
-            subtitles.append(subtitle)
+                # number of pages with results
+                pages = int(math.ceil(result_count / float(items_per_page)))
+
+                # get current page
+                if 'pg' in params:
+                    current_page = int(params['pg'])
+
+                try:
+                    sublist = soup.select('section.titlovi > ul.titlovi > li.subtitleContainer.canEdit')
+                    for sub in sublist:
+                        # subtitle id
+                        sid = sub.find(attrs={'data-id': True}).attrs['data-id']
+                        # get download link
+                        download_link = self.download_url + sid
+                        # title and alternate title
+                        match = title_re.search(sub.a.string)
+                        if match:
+                            _title = match.group('title')
+                            alt_title = match.group('altitle')
+                        else:
+                            continue
+
+                        # page link
+                        page_link = self.server_url + sub.a.attrs['href']
+                        # subtitle language
+                        _lang = sub.select_one('.lang')
+                        match = lang_re.search(_lang.attrs.get('src', _lang.attrs.get('cfsrc', '')))
+                        if match:
+                            try:
+                                # decode language
+                                lang = Language.fromtitlovi(match.group('lang')+match.group('script'))
+                            except ValueError:
+                                continue
+
+                        # relase year or series start year
+                        match = year_re.search(sub.find(attrs={'data-id': True}).parent.i.string)
+                        if match:
+                            r_year = int(match.group('year'))
+                        # fps
+                        match = fps_re.search(sub.select_one('.fps').string)
+                        if match:
+                            fps = match.group('fps')
+                        # releases
+                        releases = str(sub.select_one('.fps').parent.contents[0].string)
+
+                        # handle movies and series separately
+                        if is_episode:
+                            # season and episode info
+                            sxe = sub.select_one('.s0xe0y').string
+                            r_season = None
+                            r_episode = None
+                            if sxe:
+                                match = season_re.search(sxe)
+                                if match:
+                                    r_season = int(match.group('season'))
+                                match = episode_re.search(sxe)
+                                if match:
+                                    r_episode = int(match.group('episode'))
+
+                            subtitle = self.subtitle_class(lang, page_link, download_link, sid, releases, _title,
+                                                           alt_title=alt_title, season=r_season, episode=r_episode,
+                                                           year=r_year, fps=fps,
+                                                           asked_for_release_group=video.release_group,
+                                                           asked_for_episode=episode)
+                        else:
+                            subtitle = self.subtitle_class(lang, page_link, download_link, sid, releases, _title,
+                                                           alt_title=alt_title, year=r_year, fps=fps,
+                                                           asked_for_release_group=video.release_group)
+                        logger.debug('Found subtitle %r', subtitle)
+
+                        # prime our matches so we can use the values later
+                        subtitle.get_matches(video)
+
+                        # add found subtitles
+                        subtitles.append(subtitle)
+
+                finally:
+                    soup.decompose()
+
+                # stop on last page
+                if current_page >= pages:
+                    break
+
+                # increment current page
+                params['pg'] = current_page + 1
+                logger.debug('Getting page %d', params['pg'])
 
         return subtitles
 
     def list_subtitles(self, video, languages):
         season = episode = None
-
         if isinstance(video, Episode):
             title = video.series
             season = video.season
@@ -284,7 +300,6 @@ class TitloviProvider(Provider, ProviderSubtitleArchiveMixin):
 
         return [s for s in
                 self.query(languages, fix_inconsistent_naming(title), season=season, episode=episode, year=video.year,
-                           imdb_id=video.imdb_id,
                            video=video)]
 
     def download_subtitle(self, subtitle):
@@ -322,12 +337,10 @@ class TitloviProvider(Provider, ProviderSubtitleArchiveMixin):
         sub_to_extract = None
 
         for sub_name in subs_in_archive:
-            _sub_name = sub_name.lower()
-
-            if not ('.cyr' in _sub_name or '.cir' in _sub_name or 'cyr)' in _sub_name):
+            if not ('.cyr' in sub_name or '.cir' in sub_name):
                 sr_lat_subs.append(sub_name)
 
-            if ('.cyr' in sub_name or '.cir' in _sub_name) and not '.lat' in _sub_name.lower():
+            if ('.cyr' in sub_name or '.cir' in sub_name) and not '.lat' in sub_name:
                 sr_cyr_subs.append(sub_name)
 
         if subtitle.language == 'sr':

Contents/Libraries/Shared/subliminal_patch/subtitle.py

@@ -117,14 +117,14 @@ class Subtitle(Subtitle_):
 
         logger.info('Guessing encoding for language %s', self.language)
 
-        encodings = ['utf-8']
+        encodings = ['utf-8', 'utf-16']
 
         # add language-specific encodings
         # http://scratchpad.wikia.com/wiki/Character_Encoding_Recommendation_for_Languages
 
         if self.language.alpha3 == 'zho':
             encodings.extend(['cp936', 'gb2312', 'gbk', 'gb18030', 'hz', 'iso2022_jp_2', 'cp950', 'gb18030', 'big5',
-                              'big5hkscs', 'utf-16'])
+                              'big5hkscs'])
         elif self.language.alpha3 == 'jpn':
             encodings.extend(['shift-jis', 'cp932', 'euc_jp', 'iso2022_jp', 'iso2022_jp_1', 'iso2022_jp_2',
                               'iso2022_jp_2004', 'iso2022_jp_3', 'iso2022_jp_ext', ])
@@ -133,7 +133,7 @@ class Subtitle(Subtitle_):
 
         # arabian/farsi
         elif self.language.alpha3 in ('ara', 'fas', 'per'):
-            encodings.extend(['windows-1256', 'utf-16'])
+            encodings.append('windows-1256')
         elif self.language.alpha3 == 'heb':
             encodings.extend(['windows-1255', 'iso-8859-8'])
         elif self.language.alpha3 == 'tur':
@@ -251,7 +251,8 @@ class Subtitle(Subtitle_):
                 subs = pysubs2.SSAFile.from_string(text, fps=self.plex_media_fps)
 
             unicontent = self.pysubs2_to_unicode(subs)
-            self.content = unicontent.encode(self._guessed_encoding)
+            self.content = unicontent.encode("utf-8")
+            self._guessed_encoding = "utf-8"
         except:
             logger.exception("Couldn't convert subtitle %s to .srt format: %s", self, traceback.format_exc())
             return False
@@ -319,8 +320,7 @@ class Subtitle(Subtitle_):
         :return: string 
         """
         if not self.mods:
-            return fix_text(self.content.decode(encoding=self._guessed_encoding), **ftfy_defaults).encode(
-                encoding=self._guessed_encoding)
+            return fix_text(self.content.decode("utf-8"), **ftfy_defaults).encode(encoding="utf-8")
 
         submods = SubtitleModifications(debug=debug)
         if submods.load(content=self.text, language=self.language):
@@ -329,7 +329,7 @@ class Subtitle(Subtitle_):
             self.mods = submods.mods_used
 
             content = fix_text(self.pysubs2_to_unicode(submods.f, format=format), **ftfy_defaults)\
-                .encode(encoding=self._guessed_encoding)
+                .encode(encoding="utf-8")
             submods.f = None
             del submods
             return content

Contents/Libraries/Shared/submod_test.py

@@ -21,7 +21,7 @@ if debug:
     logging.basicConfig(level=logging.DEBUG)
 
 #sub = Subtitle(Language.fromietf("eng:forced"), mods=["common", "remove_HI", "OCR_fixes", "fix_uppercase", "shift_offset(ms=-500)", "shift_offset(ms=500)", "shift_offset(s=2,ms=800)"])
-sub = Subtitle(Language.fromietf("eng"), mods=["common", "remove_HI", "OCR_fixes", "fix_uppercase", "shift_offset(ms=0,s=1)"])
+sub = Subtitle(Language.fromietf("eng:forced"), mods=["common", "remove_HI", "OCR_fixes", "fix_uppercase", "shift_offset(ms=0,s=1)"])
 sub.content = open(fn).read()
 sub.normalize()
 content = sub.get_modified_content(debug=True)

Contents/Libraries/Shared/subscene_api/subscene.py

@@ -30,7 +30,6 @@ import enum
 import sys
 import requests
 import time
-import logging
 
 is_PY2 = sys.version_info[0] < 3
 if is_PY2:
@@ -40,13 +39,8 @@ else:
     from contextlib import suppress
     from urllib2.request import Request, urlopen
 
-from dogpile.cache.api import NO_VALUE
-from subliminal.cache import region
 from bs4 import BeautifulSoup, NavigableString
 
-
-logger = logging.getLogger(__name__)
-
 # constants
 HEADERS = {
 }
@@ -56,21 +50,14 @@ DEFAULT_USER_AGENT = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWeb"\
                      "Kit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.47 Safari/537.36"
 
 
-ENDPOINT_RE = re.compile(ur'(?uis)<form.+?action="/subtitles/(.+)">.*?<input type="text"')
-
-
-class NewEndpoint(Exception):
-    pass
-
-
 # utils
-def soup_for(url, data=None, session=None, user_agent=DEFAULT_USER_AGENT):
+def soup_for(url, session=None, user_agent=DEFAULT_USER_AGENT):
     url = re.sub("\s", "+", url)
     if not session:
         r = Request(url, data=None, headers=dict(HEADERS, **{"User-Agent": user_agent}))
         html = urlopen(r).read().decode("utf-8")
     else:
-        ret = session.post(url, data=data)
+        ret = session.get(url)
         ret.raise_for_status()
         html = ret.text
     return BeautifulSoup(html, "html.parser")
@@ -125,7 +112,7 @@ class Subtitle(object):
         subtitles = []
 
         for row in rows:
-            if row.td.a is not None and row.td.get("class", ["lazy"])[0] != "empty":
+            if row.td.a is not None:
                 subtitles.append(cls.from_row(row))
 
         return subtitles
@@ -255,42 +242,29 @@ def get_first_film(soup, section, year=None, session=None):
                 url = SITE_DOMAIN + t.div.a.get("href")
                 break
         if not url:
-            # fallback to non-year results
-            logger.info("Falling back to non-year results as year wasn't found (%s)", year)
-            url = SITE_DOMAIN + tag.findNext("ul").find("li").div.a.get("href")
+            return
 
     return Film.from_url(url, session=session)
 
 
-def find_endpoint(session, content=None):
-    endpoint = region.get("subscene_endpoint2")
-    if endpoint is NO_VALUE:
-        if not content:
-            content = session.get(SITE_DOMAIN).text
-
-        m = ENDPOINT_RE.search(content)
-        if m:
-            endpoint = m.group(1).strip()
-            logger.debug("Switching main endpoint to %s", endpoint)
-            region.set("subscene_endpoint2", endpoint)
-    return endpoint
-
-
 def search(term, release=True, session=None, year=None, limit_to=SearchTypes.Exact, throttle=0):
     # note to subscene: if you actually start to randomize the endpoint, we'll have to query your server even more
-
+    endpoints = ["searching", "search", "srch", "find"]
     if release:
-        endpoint = "release"
-    else:
-        endpoint = find_endpoint(session)
-        time.sleep(throttle)
+        endpoints = ["release"]
 
-    if not endpoint:
-        logger.error("Couldn't find endpoint, exiting")
-        return
-
-    soup = soup_for("%s/subtitles/%s" % (SITE_DOMAIN, endpoint), data={"query": term},
-                    session=session)
+    soup = None
+    for endpoint in endpoints:
+        try:
+            soup = soup_for("%s/subtitles/%s?q=%s" % (SITE_DOMAIN, endpoint, term),
+                            session=session)
+        except requests.HTTPError, e:
+            if e.response.status_code == 404:
+                time.sleep(throttle)
+                # fixme: detect endpoint from html
+                continue
+            raise
+        break
 
     if soup:
         if "Subtitle search by" in str(soup):

Contents/Libraries/Shared/subzero/language.py

@@ -1,6 +1,5 @@
 # coding=utf-8
 import types
-import re
 
 from babelfish.exceptions import LanguageError
 from babelfish import Language as Language_, basestr
@@ -9,25 +8,6 @@ repl_map = {
     "dk": "da",
     "nld": "nl",
     "english": "en",
-    "alb": "sq",
-    "arm": "hy",
-    "baq": "eu",
-    "bur": "my",
-    "chi": "zh",
-    "cze": "cs",
-    "dut": "nl",
-    "fre": "fr",
-    "geo": "ka",
-    "ger": "de",
-    "gre": "el",
-    "ice": "is",
-    "mac": "mk",
-    "mao": "mi",
-    "may": "ms",
-    "per": "fa",
-    "rum": "ro",
-    "slo": "sk",
-    "tib": "bo",
 }
 
 
@@ -135,16 +115,3 @@ class Language(Language_):
             return Language(*Language_.fromietf(s).__getstate__())
 
         return Language(*Language_.fromalpha3b(s).__getstate__())
-
-
-IETF_MATCH = ".+\.([^-.]+)(?:-[A-Za-z]+)?$"
-ENDSWITH_LANGUAGECODE_RE = re.compile("\.([^-.]{2,3})(?:-[A-Za-z]{2,})?$")
-
-
-def match_ietf_language(s, ietf=False):
-    language_match = re.match(".+\.([^\.]+)$" if not ietf
-                              else IETF_MATCH, s)
-    if language_match and len(language_match.groups()) == 1:
-        language = language_match.groups()[0]
-        return language
-    return s

Contents/Libraries/Shared/subzero/lib/dict.py

@@ -107,12 +107,6 @@ class Dicked(object):
         for key, value in entries.iteritems():
             self.__dict__[key] = (Dicked(**value) if isinstance(value, dict) else value)
 
-    def has(self, key):
-        return self._entries is not None and key in self._entries
-
-    def get(self, key, default=None):
-        return self._entries.get(key, default) if self._entries else default
-
     def __repr__(self):
         return str(self)
 

Contents/Libraries/Shared/subzero/modification/dictionaries/data.py

@@ -42,7 +42,7 @@ data = {'bos': {'BeginLines': {'data': OrderedDict(),
                       'pattern': u"(?um)(?:\\,\\ sin|\\ mothen|\\ can\\'t\\_|\\ openiL|\\ of\\\ufb02|pshycol|\\ i\\.\\.\\.|\\ L\\.)$"},
          'PartialLines': {'data': OrderedDict([(u' /be ', u' I be '), (u" aren '1'", u" aren't"), (u" aren'tyou", u" aren't you"), (u" doesn '1'", u" doesn't"), (u" fr/eno'", u' friend'), (u" fr/eno'.", u' friend.'), (u" haven 'z' ", u" haven't "), (u" haven 'z'.", u" haven't."), (u' I ha ve ', u' I have '), (u" I']I ", u" I'll "), (u' L am', u' I am'), (u' L can', u' I can'), (u" L don't ", u" I don't "), (u' L hate ', u' I hate '), (u' L have ', u' I have '), (u' L like ', u' I like '), (u' L will', u' I will'), (u' L would', u' I would'), (u" L'll ", u" I'll "), (u" L've ", u" I've "), (u' m y family', u' my family'), (u" 's ", u"'s "), (u" shou/dn '1 ", u" shouldn't "), (u" won 'z' ", u" won't "), (u" won 'z'.", u" won't."), (u" wou/c/n 'z' ", u" wouldn't "), (u" wou/c/n 'z'.", u" wouldn't."), (u" wou/dn 'z' ", u" wouldn't "), (u" wou/dn 'z'.", u" wouldn't."), (u'/ did', u'I did'), (u'/ have ', u'I have '), (u'/ just ', u'I just '), (u'/ loved ', u'I loved '), (u'/ need', u'I need'), (u'|was11', u'I was 11'), (u'at Hrst', u'at first'), (u"B ullshiz'", u'Bullshit'), (u'big lunk', u'love you'), (u"can 't", u"can't"), (u"can' t ", u"can't "), (u"can 't ", u"can't "), (u'CHA TTERING', u'CHATTERING'), (u'come 0n', u'come on'), (u'Come 0n', u'Come on'), (u"couldn 't", u"couldn't"), (u"couldn' t ", u"couldn't "), (u"couldn 't ", u"couldn't "), (u"Destin y's", u"Destiny's"), (u"didn 't", u"didn't"), (u"didn' t ", u"didn't "), (u"didn 't ", u"didn't "), (u"Doesn '1'", u"Doesn't"), (u"doesn '1' ", u"doesn't "), (u"doesn '1\u2018 ", u"doesn't "), (u"doesn 't", u"doesn't"), (u"doesn'1' ", u"doesn't "), (u"doesn'1\u2018 ", u"doesn't "), (u"don '1' ", u"don't "), (u"don '1\u2018 ", u"don't "), (u"don '2' ", u"don't "), (u" aren '2'", u" aren't"), (u"aren '2' ", u"aren't "), (u"don '2\u2018 ", u"don't "), (u"don 't", u"don't"), (u"Don' t ", u"Don't "), (u"Don 't ", u"Don't "), (u"don'1' ", u"don't "), (u"don'1\u2018 ", u"don't "), (u"there '5 ", u"there's "), (u'E very', u'Every'), (u'get 0n', u'get on'), (u'go 0n', u'go on'), (u'Go 0n', u'Go on'), (u"H3993' birthday", u'Happy birthday'), (u"hadn 't", u"hadn't"), (u"he 's", u"he's"), (u"He 's", u"He's"), (u'He y', u'Hey'), (u'he)/', u'hey'), (u'He)/', u'Hey'), (u'HEA VY', u'HEAVY'), (u'Henry ll', u'Henry II'), (u'Henry lll', u'Henry III'), (u'Henry Vlll', u'Henry VIII'), (u'Henry Vll', u'Henry VII'), (u'Henry Vl', u'Henry VI'), (u'Hold 0n', u'Hold on'), (u'I am. ls', u'I am. Is'), (u'I d0', u'I do'), (u"I 'm", u"I'm"), (u"I 'rn ", u"I'm "), (u"I 've", u"I've"), (u'I0 ve her', u'love her'), (u'I0 ve you', u'love you'), (u"I02'", u'lot'), (u"I'm sony", u"I'm sorry"), (u"isn' t ", u"isn't "), (u"isn 't ", u"isn't "), (u'K)/le', u'Kyle'), (u'L ook', u'Look'), (u'let me 90', u'let me go'), (u'Let me 90', u'Let me go'), (u"let's 90", u"let's go"), (u"Let's 90", u"Let's go"), (u'lfl had', u'If I had'), (u'lova you', u'love you'), (u'Lova you', u'love you'), (u'lovo you', u'love you'), (u'Lovo you', u'love you'), (u'ls anyone', u'Is anyone'), (u'ls he', u'Is he'), (u'-ls he', u'- Is he'), (u'ls it', u'Is it'), (u'-ls it', u'- Is it'), (u'ls she', u'Is she'), (u'-ls she', u'- Is she'), (u'ls that', u'Is that'), (u'-ls that', u'- Is that'), (u'ls this', u'Is this'), (u'-ls this', u'- Is this'), (u'Maze] tov', u'Mazel tov'), (u"N02' ", u'Not '), (u' of 0ur ', u' of our '), (u' ot mine ', u' of mine '), (u'PLA YING', u'PLAYING'), (u'REPEA TING ', u'REPEATING '), (u'Sa y', u'Say'), (u"she 's", u"she's"), (u"She 's", u"She's"), (u"shouldn 't", u"shouldn't"), (u'sta y', u'stay'), (u'Sta y', u'Stay'), (u'SWO rd', u'Sword'), (u'taka care', u'take care'), (u'Taka care', u'Take care'), (u'the Hrst', u'the first'), (u'toc late', u'too late'), (u'uf me', u'of me'), (u'uf our', u'of our'), (u'wa y', u'way'), (u'Wal-I\\/Iart', u'Wal-Mart'), (u"wasn '1' ", u"wasn't "), (u"Wasn '1' ", u"Wasn't "), (u"wasn '1\u2018 ", u"wasn't "), (u"Wasn '1\u2018 ", u"Wasn't "), (u"wasn 't", u"wasn't"), (u"Wasn 't", u"Wasn't"), (u"we 've", u"we've"), (u"We 've", u"We've"), (u"wem' off", u'went off'), (u"weren 't", u"weren't"), (u"who 's", u"who's"), (u"won 't", u"won't"), (u'would ha ve', u'would have '), (u"wouldn 't", u"wouldn't"), (u"Wouldn 't", u"Wouldn't"), (u'y()u', u'you'), (u'you QUYS', u'you guys'), (u"you' re ", u"you're "), (u"you 're ", u"you're "), (u"you 've", u"you've"), (u"You 've", u"You've"), (u"you' ve ", u"you've "), (u"you 've ", u"you've "), (u'aftera while', u'after a while'), (u'Aftera while', u'After a while'), (u'THUN DERCLAPS', u'THUNDERCLAPS'), (u'(BUZZI N G)', u'(BUZZING)'), (u'[BUZZI N G]', u'[BUZZING]'), (u'(G RU NTING', u'(GRUNTING'), (u'[G RU NTING', u'[GRUNTING'), (u'(G ROWLING', u'(GROWLING'), (u'[G ROWLING', u'[GROWLING'), (u' WAI LS)', u'WAILS)'), (u' WAI LS]', u'WAILS]'), (u'(scu RRYING)', u'(SCURRYING)'), (u'[scu RRYING]', u'[SCURRYING]'), (u'(GRUNT5)', u'(GRUNTS)'), (u'[GRUNT5]', u'[GRUNTS]'), (u'NARRA TOR:', u'NARRATOR:'), (u'(GROAN ING', u'(GROANING'), (u'[GROAN ING'Line truncated
                           'pattern': u"(?um)(?:(?<=\\s)|(?<=^)|(?<=\\b))(?:\\ \\/be\\ |\\ aren\\ \\'1\\'|\\ aren\\'tyou|\\ doesn\\ \\'1\\'|\\ fr\\/eno\\'|\\ fr\\/eno\\'\\.|\\ haven\\ \\'z\\'\\ |\\ haven\\ \\'z\\'\\.|\\ I\\ ha\\ ve\\ |\\ I\\'\\]I\\ |\\ L\\ am|\\ L\\ can|\\ L\\ don\\'t\\ |\\ L\\ hate\\ |\\ L\\ have\\ |\\ L\\ like\\ |\\ L\\ will|\\ L\\ would|\\ L\\'ll\\ |\\ L\\'ve\\ |\\ m\\ y\\ family|\\ \\'s\\ |\\ shou\\/dn\\ \\'1\\ |\\ won\\ \\'z\\'\\ |\\ won\\ \\'z\\'\\.|\\ wou\\/c\\/n\\ \\'z\\'\\ |\\ wou\\/c\\/n\\ \\'z\\'\\.|\\ wou\\/dn\\ \\'z\\'\\ |\\ wou\\/dn\\ \\'z\\'\\.|\\/\\ did|\\/\\ have\\ |\\/\\ just\\ |\\/\\ loved\\ |\\/\\ need|\\|was11|at\\ Hrst|B\\ ullshiz\\'|big\\ lunk|can\\ \\'t|can\\'\\ t\\ |can\\ \\'t\\ |CHA\\ TTERING|come\\ 0n|Come\\ 0n|couldn\\ \\'t|couldn\\'\\ t\\ |couldn\\ \\'t\\ |Destin\\ y\\'s|didn\\ \\'t|didn\\'\\ t\\ |didn\\ \\'t\\ |Doesn\\ \\'1\\'|doesn\\ \\'1\\'\\ |doesn\\ \\'1\\\u2018\\ |doesn\\ \\'t|doesn\\'1\\'\\ |doesn\\'1\\\u2018\\ |don\\ \\'1\\'\\ |don\\ \\'1\\\u2018\\ |don\\ \\'2\\'\\ |\\ aren\\ \\'2\\'|aren\\ \\'2\\'\\ |don\\ \\'2\\\u2018\\ |don\\ \\'t|Don\\'\\ t\\ |Don\\ \\'t\\ |don\\'1\\'\\ |don\\'1\\\u2018\\ |there\\ \\'5\\ |E\\ very|get\\ 0n|go\\ 0n|Go\\ 0n|H3993\\'\\ birthday|hadn\\ \\'t|he\\ \\'s|He\\ \\'s|He\\ y|he\\)\\/|He\\)\\/|HEA\\ VY|Henry\\ ll|Henry\\ lll|Henry\\ Vlll|Henry\\ Vll|Henry\\ Vl|Hold\\ 0n|I\\ am\\.\\ ls|I\\ d0|I\\ \\'m|I\\ \\'rn\\ |I\\ \\'ve|I0\\ ve\\ her|I0\\ ve\\ you|I02\\'|I\\'m\\ sony|isn\\'\\ t\\ |isn\\ \\'t\\ |K\\)\\/le|L\\ ook|let\\ me\\ 90|Let\\ me\\ 90|let\\'s\\ 90|Let\\'s\\ 90|lfl\\ had|lova\\ you|Lova\\ you|lovo\\ you|Lovo\\ you|ls\\ anyone|ls\\ he|\\-ls\\ he|ls\\ it|\\-ls\\ it|ls\\ she|\\-ls\\ she|ls\\ that|\\-ls\\ that|ls\\ this|\\-ls\\ this|Maze\\]\\ tov|N02\\'\\ |\\ of\\ 0ur\\ |\\ ot\\ mine\\ |PLA\\ YING|REPEA\\ TING\\ |Sa\\ y|she\\ \\'s|She\\ \\'s|shouldn\\ \\'t|sta\\ y|Sta\\ y|SWO\\ rd|taka\\ care|Taka\\ care|the\\ Hrst|toc\\ late|uf\\ me|uf\\ our|wa\\ y|Wal\\-I\\\\\\/Iart|wasn\\ \\'1\\'\\ |Wasn\\ \\'1\\'\\ |wasn\\ \\'1\\\u2018\\ |Wasn\\ \\'1\\\u2018\\ |wasn\\ \\'t|Wasn\\ \\'t|we\\ \\'ve|We\\ \\'ve|wem\\'\\ off|weren\\ \\'t|who\\ \\'s|won\\ \\'t|would\\ ha\\ ve|wouldn\\ \\'t|Wouldn\\ \\'t|y\\(\\)u|you\\ QUYS|you\\'\\ re\\ |you\\ \\'re\\ |you\\ \\'ve|You\\ \\'ve|you\\'\\ ve\\ |you\\ \\'ve\\ |aftera\\ while|Aftera\\ while|THUN\\ DERCLAPS|\\(BUZZI\\ N\\ G\\)|\\[BUZZI\\ N\\ G\\]|\\(G\\ RU\\ NTING|\\[G\\ RU\\ NTING|\\(G\\ ROWLING|\\[G\\ ROWLING|\\ WAI\\ LS\\)|\\ WAI\\ LS\\]|\\(scu\\ RRYING\\)|\\[scu\\ RRYING\\]|\\(GRUNT5\\)|\\[GRUNT5\\]|NARRA\\ TOR\\:|\\(GROAN\\ ING|\\[GROAN\\ ING|GROAN\\ ING\\)|GROAN\\ ING\\]|\\(LAUGH\\ ING|\\[LAUGH\\ ING|LAUGH\\ ING\\)|LAUGH\\ ING\\]|\\(BU\\ BBLING|\\[BU\\ BBLING|BU\\ BBLING\\)|BU\\ BBLING\\]|\\(SH\\ USHING|\\[SH\\ USHING|SH\\ USHING\\)|SH\\ USHING\\]|\\(CH\\ ILDREN|\\[CH\\ ILDREN|CH\\ ILDREN\\)|CH\\ ILDREN\\]|\\(MURMU\\ RING|\\[MURMU\\ RING|MURMU\\ RING\\)|MURMU\\ RING\\]|\\(GU\\ N\\ |\\[GU\\ N\\ |GU\\ N\\)|GU\\ N\\]|CH\\ ILDREN\\:|STU\\ DENTS\\:|\\(WH\\ ISTLE|\\[WH\\ ISTLE|WH\\ ISTLE\\)|WH\\ ISTLE\\]|U\\ LU\\ LATING|AU\\ DIENCE\\:|HA\\ WAIIAN|\\(ARTH\\ UR|\\[ARTH\\ UR|ARTH\\ UR\\)|ARTH\\ UR\\]|J\\ EREMY\\:|\\(ELEVA\\ TOR|\\[ELEVA\\ TOR|ELEVA\\ TOR\\)|ELEVA\\ TOR\\]|CONTIN\\ U\\ ES|WIN\\ D\\ HOWLING|telis\\ me|Telis\\ me|\\.\\ Ls\\ |\\!\\ Ls\\ |\\?\\ Ls\\ |\\.\\ Lt\\ |\\!\\ Lt\\ |\\?\\ Lt\\ |SQMEWH\\ ERE\\ ELSE|\\ I\\,m\\ |\\ I\\,ve\\ |\\ you\\,re\\ |\\ you\\,ll\\ |\\ doesn\\,t\\ |\\ let\\,s\\ |\\ he\\,s\\ |\\ it\\,s\\ |\\ can\\,t\\ |\\ Can\\,t\\ |\\ don\\,t\\ |\\ Don\\,t\\ |wouldn\\ \\'tyou|\\ lgot\\ it|\\ you\\,ve\\ |\\ I\\ ve\\ |\\ I\\ ii\\ |\\ I\\ m\\ |\\ why\\ d\\ |\\ couldn\\ t\\ |\\ that\\ s\\ |\\ i\\.\\.\\.\\ |L\\ don\\'t|L\\ won\\'t|L\\ should|L\\ had|L\\ happen|L\\ wasn\\'t|H\\ i|L\\ didn\\'t|L\\ do|L\\ could|L\\ will|L\\ suggest|L\\ reckon|L\\ am|L\\ couldn\\'t|L\\ might|L\\ would|L\\ was|L\\ know|L\\ think|L\\ haven\\'t|L\\ have\\ |L\\ want|L\\ can|L\\ love|L\\ like)(?:(?=\\s)|(?=$)|(?=\\b))"},
-         'PartialWordsAlways': {'data': OrderedDict([(u'\xa4', u'o'), (u'lVI', u'M'), (u'IVl', u'M'), (u'lVl', u'M'), (u'I\\/I', u'M'), (u'l\\/I', u'M'), (u'I\\/l', u'M'), (u'l\\/l', u'M'), (u'IVIa', u'Ma'), (u'IVIe', u'Me'), (u'IVIi', u'Mi'), (u'IVIo', u'Mo'), (u'IVIu', u'Mu'), (u'IVIy', u'My'), (u' l ', u' I '), (u'l/an', u'lian'), (u'\xb0x\xb0', u'%'), (u'\xc3\xc2s', u"'s"), (u'at/on', u'ation'), (u'lljust', u'll just'), (u"'sjust", u"'s just"), (u'";', u"'s"), (u'compiete', u'complete'), (u' L ', u' I '), (u'a/ion', u'ation'), (u'\xc2s', u"'s"), (u"'tjust", u"'t just")]),
+         'PartialWordsAlways': {'data': OrderedDict([(u'\xa4', u'o'), (u'lVI', u'M'), (u'IVl', u'M'), (u'lVl', u'M'), (u'I\\/I', u'M'), (u'l\\/I', u'M'), (u'I\\/l', u'M'), (u'l\\/l', u'M'), (u'IVIa', u'Ma'), (u'IVIe', u'Me'), (u'IVIi', u'Mi'), (u'IVIo', u'Mo'), (u'IVIu', u'Mu'), (u'IVIy', u'My'), (u' l ', u' I '), (u'l/an', u'lian'), (u'\xb0x\xb0', u'%'), (u'\xc3\xc2s', u"'s"), (u'at/on', u'ation'), (u'lljust', u'll just'), (u"'sjust", u"'s just"), (u'compiete', u'complete'), (u' L ', u' I '), (u'a/ion', u'ation'), (u'\xc2s', u"'s"), (u"'tjust", u"'t just")]),
                                 'pattern': None},
          'WholeLines': {'data': OrderedDict([(u'H ey.', u'Hey.'), (u'He)\u2019-', u'Hey.'), (u'N0.', u'No.'), (u'-N0.', u'-No.'), (u'Noll', u'No!!'), (u'(G ROANS)', u'(GROANS)'), (u'[G ROANS]', u'[GROANS]'), (u'(M EOWS)', u'(MEOWS)'), (u'[M EOWS]', u'[MEOWS]'), (u'Uaughs]', u'[laughs]'), (u'[chitte rs]', u'[chitters]'), (u'Hil\u2018 it!', u'Hit it!'), (u'<i>Hil\u2018 it!</i>', u'<i>Hit it!</i>'), (u'ISIGHS]', u'[SIGHS]')]),
                         'pattern': None},

Contents/Libraries/Shared/subzero/modification/dictionaries/make_data.py

@@ -36,7 +36,6 @@ SZ_FIX_DATA = {
             u" l ": u" I ",
             u"'sjust": u"'s just",
             u"'tjust": u"'t just",
-            u"\";": u"'s",
         },
         "WholeWords": {
             u"I'11": u"I'll",

Contents/Libraries/Shared/subzero/modification/main.py

@@ -6,7 +6,7 @@ import pysubs2
 import logging
 import time
 
-from mods import EMPTY_TAG_PROCESSOR, EmptyEntryError
+from mods import EMPTY_TAG_PROCESSOR, EmptyEntryError, FullContentRep
 from registry import registry
 from subzero.language import Language
 
@@ -257,7 +257,16 @@ class SubtitleModifications(object):
                 mod.modify(None, debug=self.debug, parent=self, **args)
 
     def apply_line_mods(self, new_entries, mods):
-        for index, entry in enumerate(self.f, 1):
+        index = 1
+        entries = self.f[:]
+        entry_count = len(entries)
+
+        while 1:
+            if index > entry_count - 1:
+                break
+
+            entry = entries[index]
+
             applied_mods = []
             lines = []
 
@@ -265,116 +274,110 @@ class SubtitleModifications(object):
             start_tags = []
             end_tags = []
 
-            t = entry.text.strip()
-            if not t:
+            text = entry.text.replace(ur"\N", "\n").strip()
+            if not text:
                 if self.debug:
                     logger.debug(u"Skipping empty line: %s", index)
+                index += 1
                 continue
 
-            skip_entry = False
-            for line in t.split(ur"\N"):
-                # don't bother the mods with surrounding tags
-                old_line = line
-                line = line.strip()
-                skip_line = False
-                line_count += 1
+            try:
+                for line in text.split("\n"):
+                    # don't bother the mods with surrounding tags
+                    old_line = line
+                    line = line.strip()
+                    skip_line = False
+                    line_count += 1
 
-                if not line:
-                    continue
+                    if not line:
+                        continue
 
-                # clean {\X0} tags before processing
-                # fixme: handle nested tags?
-                start_tag = u""
-                end_tag = u""
-                if line.startswith(self.font_style_tag_start):
-                    start_tag = line[:5]
-                    line = line[5:]
-                if line[-5:-3] == self.font_style_tag_start:
-                    end_tag = line[-5:]
-                    line = line[:-5]
+                    # clean {\X0} tags before processing
+                    # fixme: handle nested tags?
+                    start_tag = u""
+                    end_tag = u""
+                    if line.startswith(self.font_style_tag_start):
+                        start_tag = line[:5]
+                        line = line[5:]
+                    if line[-5:-3] == self.font_style_tag_start:
+                        end_tag = line[-5:]
+                        line = line[:-5]
 
-                last_procs_mods = []
+                    last_procs_mods = []
 
-                # fixme: this double loop is ugly
-                for order, identifier, args in mods:
-                    mod = self.initialized_mods[identifier]
+                    # fixme: this double loop is ugly
+                    for order, identifier, args in mods:
+                        mod = self.initialized_mods[identifier]
 
-                    try:
-                        line = mod.modify(line.strip(), entry=entry.text, debug=self.debug, parent=self, index=index,
+                        line = mod.modify(line.strip(), entry=text, debug=self.debug, parent=self, index=index,
                                           **args)
-                    except EmptyEntryError:
-                        if self.debug:
-                            logger.debug(u"%d: %s: %r -> ''", index, identifier, entry.text)
-                        skip_entry = True
-                        break
 
-                    if not line:
-                        if self.debug:
-                            logger.debug(u"%d: %s: %r -> ''", index, identifier, old_line)
-                        skip_line = True
-                        break
+                        if not line:
+                            if self.debug:
+                                logger.debug(u"%d: %s: %r -> ''", index, identifier, old_line)
+                            skip_line = True
+                            break
 
-                    applied_mods.append(identifier)
-                    if mod.last_processors:
-                        last_procs_mods.append([identifier, args])
+                        applied_mods.append(identifier)
+                        if mod.last_processors:
+                            last_procs_mods.append([identifier, args])
 
-                if skip_entry:
-                    lines = []
-                    break
+                    if skip_line:
+                        continue
 
-                if skip_line:
-                    continue
+                    for identifier, args in last_procs_mods:
+                        mod = self.initialized_mods[identifier]
 
-                for identifier, args in last_procs_mods:
-                    mod = self.initialized_mods[identifier]
-
-                    try:
-                        line = mod.modify(line.strip(), entry=entry.text, debug=self.debug, parent=self, index=index,
+                        line = mod.modify(line.strip(), entry=text, debug=self.debug, parent=self, index=index,
                                           procs=["last_process"], **args)
-                    except EmptyEntryError:
+
+                        if not line:
+                            if self.debug:
+                                logger.debug(u"%d: %s: %r -> ''", index, identifier, old_line)
+                            skip_line = True
+                            break
+
+                    if skip_line:
+                        continue
+
+                    if start_tag:
+                        start_tags.append(start_tag)
+
+                    if end_tag:
+                        end_tags.append(end_tag)
+
+                    # append new line and clean possibly newly added empty tags
+                    cleaned_line = EMPTY_TAG_PROCESSOR.process(start_tag + line + end_tag, debug=self.debug).strip()
+                    if cleaned_line:
+                        # we may have a single closing tag, if so, try appending it to the previous line
+                        if len(cleaned_line) == 5 and cleaned_line.startswith("{\\") and cleaned_line.endswith("0}"):
+                            if lines:
+                                prev_line = lines.pop()
+                                lines.append(prev_line + cleaned_line)
+                                continue
+
+                        lines.append(cleaned_line)
+                    else:
                         if self.debug:
-                            logger.debug(u"%d: %s: %r -> ''", index, identifier, entry.text)
-                        skip_entry = True
-                        break
+                            logger.debug(u"%d: Ditching now empty line (%r)", index, line)
 
-                    if not line:
-                        if self.debug:
-                            logger.debug(u"%d: %s: %r -> ''", index, identifier, old_line)
-                        skip_line = True
-                        break
-
-                if skip_entry:
-                    lines = []
-                    break
-
-                if skip_line:
+                if not lines:
+                    # don't bother logging when the entry only had one line
+                    if self.debug and line_count > 1:
+                        logger.debug(u"%d: %r -> ''", index, text)
+                    index += 1
                     continue
+            except EmptyEntryError, e:
+                if self.debug:
+                    logger.debug(u"%d: %s: %r -> ''", index, e.mod.identifier, e.entry)
+                index += 1
+                continue
 
-                if start_tag:
-                    start_tags.append(start_tag)
-
-                if end_tag:
-                    end_tags.append(end_tag)
-
-                # append new line and clean possibly newly added empty tags
-                cleaned_line = EMPTY_TAG_PROCESSOR.process(start_tag + line + end_tag, debug=self.debug).strip()
-                if cleaned_line:
-                    # we may have a single closing tag, if so, try appending it to the previous line
-                    if len(cleaned_line) == 5 and cleaned_line.startswith("{\\") and cleaned_line.endswith("0}"):
-                        if lines:
-                            prev_line = lines.pop()
-                            lines.append(prev_line + cleaned_line)
-                            continue
-
-                    lines.append(cleaned_line)
-                else:
-                    if self.debug:
-                        logger.debug(u"%d: Ditching now empty line (%r)", index, line)
-
-            if not lines:
-                # don't bother logging when the entry only had one line
-                if self.debug and line_count > 1:
-                    logger.debug(u"%d: %r -> ''", index, entry.text)
+            except FullContentRep, e:
+                if self.debug:
+                    logger.debug(u"%d: %s: %r -> %r", index, e.mod.identifier, text, e.new_content)
+                new_entries.append(e.new_content.replace("\n", ur"\N"))
+                index += 1
                 continue
 
             new_text = ur"\N".join(lines)
@@ -403,6 +406,8 @@ class SubtitleModifications(object):
                 entry.text = new_text
 
             new_entries.append(entry)
+            index += 1
+
 
 SubMod = SubtitleModifications
 

Contents/Libraries/Shared/subzero/modification/mods/__init__.py

@@ -47,7 +47,7 @@ class SubtitleModification(object):
                 continue
 
             old_content = new_content
-            new_content = processor.process(new_content, debug=debug, **kwargs)
+            new_content = processor.process(new_content, debug=debug, mod=self, **kwargs)
             if not new_content:
                 if debug:
                     logger.debug("Processor returned empty line: %s", processor.name)
@@ -107,9 +107,22 @@ empty_line_post_processors = [
 ]
 
 
-class EmptyEntryError(Exception):
+class ModEvent(Exception):
+    def __init__(self, *args, **kwargs):
+        self.mod = kwargs.pop("mod", None)
+        self.entry = kwargs.pop("entry", None)
+        super(ModEvent, self).__init__(*args, **kwargs)
+
+
+class EmptyEntryError(ModEvent):
     pass
 
 
-class EmptyLineError(Exception):
+class EmptyLineError(ModEvent):
     pass
+
+
+class FullContentRep(ModEvent):
+    def __init__(self, *args, **kwargs):
+        self.new_content = kwargs.pop("new_content", None)
+        super(FullContentRep, self).__init__(*args, **kwargs)

Contents/Libraries/Shared/subzero/modification/mods/hearing_impaired.py

@@ -1,7 +1,8 @@
 # coding=utf-8
 import re
 
-from subzero.modification.mods import SubtitleTextModification, empty_line_post_processors, EmptyEntryError, TAG
+from subzero.modification.mods import SubtitleTextModification, empty_line_post_processors, EmptyEntryError, TAG, \
+    FullContentRep
 from subzero.modification.processors.re_processor import NReProcessor
 from subzero.modification import registry
 
@@ -10,9 +11,11 @@ class FullBracketEntryProcessor(NReProcessor):
     def process(self, content, debug=False, **kwargs):
         entry = kwargs.get("entry")
         if entry:
-            rep_content = super(FullBracketEntryProcessor, self).process(entry, debug=debug, **kwargs)
-            if not rep_content.strip():
-                raise EmptyEntryError()
+            rep_content = super(FullBracketEntryProcessor, self).process(entry, debug=debug, **kwargs).strip()
+            if not rep_content:
+                raise EmptyEntryError(mod=self.mod, entry=entry)
+            if content != rep_content:
+                raise FullContentRep(new_content=rep_content, mod=self.mod, entry=entry)
         return content
 
 
@@ -49,8 +52,8 @@ class HearingImpaired(SubtitleTextModification):
         NReProcessor(re.compile(ur'(?sux)-?%(t)s[([][^([)\]]+?(?=[A-zÀ-ž"\'.]{3,})[^([)\]]+[)\]][\s:]*%(t)s' %
                                 {"t": TAG}), "", name="HI_brackets"),
 
-        #NReProcessor(re.compile(ur'(?sux)-?%(t)s[([]%(t)s(?=[A-zÀ-ž"\'.]{3,})[^([)\]]+%(t)s$' % {"t": TAG}),
-        #             "", name="HI_bracket_open_start"),
+        FullBracketEntryProcessor(re.compile(ur'(?sux)-?%(t)s[([]%(t)s(?=[A-zÀ-ž"\'.]{3,})[^([)\]]+%(t)s$' % {"t": TAG}),
+                     "", name="HI_bracket_open_start"),
 
         #NReProcessor(re.compile(ur'(?sux)-?%(t)s(?=[A-zÀ-ž"\'.]{3,})[^([)\]]+[)\]][\s:]*%(t)s' % {"t": TAG}), "",
         #             name="HI_bracket_open_end"),

Contents/Libraries/Shared/subzero/modification/processors/__init__.py

@@ -7,12 +7,14 @@ class Processor(object):
     """
     name = None
     parent = None
+    mod = None
     supported = None
     enabled = True
 
-    def __init__(self, name=None, parent=None, supported=None):
+    def __init__(self, name=None, parent=None, mod=None, supported=None):
         self.name = name
         self.parent = parent
+        self.mod = mod
         self.supported = supported if supported else lambda parent: True
 
     @property
@@ -20,6 +22,8 @@ class Processor(object):
         return self.name
 
     def process(self, content, debug=False, **kwargs):
+        if not self.mod:
+            self.mod = kwargs.get("mod", None)
         return content
 
     def __repr__(self):

Contents/Libraries/Shared/subzero/modification/processors/re_processor.py

@@ -14,12 +14,13 @@ class ReProcessor(Processor):
     pattern = None
     replace_with = None
 
-    def __init__(self, pattern, replace_with, name=None, supported=None):
-        super(ReProcessor, self).__init__(name=name, supported=supported)
+    def __init__(self, pattern, replace_with, name=None, supported=None, **kwargs):
+        super(ReProcessor, self).__init__(name=name, supported=supported, **kwargs)
         self.pattern = pattern
         self.replace_with = replace_with
 
     def process(self, content, debug=False, **kwargs):
+        super(ReProcessor, self).process(content, debug=debug, **kwargs)
         return self.pattern.sub(self.replace_with, content)
 
 

Contents/Libraries/Shared/subzero/video.py

@@ -17,8 +17,7 @@ def has_external_subtitle(part_id, stored_subs, language):
 
 
 def set_existing_languages(video, video_info, external_subtitles=False, embedded_subtitles=False, known_embedded=None,
-                           stored_subs=None, languages=None, only_one=False, known_metadata_subs=None,
-                           match_strictness="strict"):
+                           stored_subs=None, languages=None, only_one=False, known_metadata_subs=None):
     logger.debug(u"Determining existing subtitles for %s", video.name)
 
     external_langs_found = set()
@@ -28,8 +27,7 @@ def set_existing_languages(video, video_info, external_subtitles=False, embedded
         external_langs_found = known_metadata_subs
 
     external_langs_found.update(set(search_external_subtitles(video.name, languages=languages,
-                                                              only_one=only_one,
-                                                              match_strictness=match_strictness).values()))
+                                                              only_one=only_one).values()))
 
     # found external subtitles should be considered?
     if external_subtitles:
@@ -54,10 +52,10 @@ def set_existing_languages(video, video_info, external_subtitles=False, embedded
             video.subtitle_languages.add(language)
 
 
-def parse_video(fn, hints, skip_hashing=False, dry_run=False, providers=None, hash_from=None):
+def parse_video(fn, hints, skip_hashing=False, dry_run=False, providers=None):
     logger.debug("Parsing video: %s, hints: %s", os.path.basename(fn), hints)
     return scan_video(fn, hints=hints, dont_use_actual_file=dry_run, providers=providers,
-                      skip_hashing=skip_hashing, hash_from=hash_from)
+                      skip_hashing=skip_hashing)
 
 
 def refine_video(video, no_refining=False, refiner_settings=None):

Contents/advanced_settings.json.template

@@ -24,13 +24,6 @@ Don't expect support if you mess this up.
     "find_better_as_extracted_tv_score": 352,
     "find_better_as_extracted_movie_score": 82,
 
-    // SZ can use mediainfo if present to detect titles/forced state of MP4 MOV_TEXT, because the PMS currently doesn't
-    // set the title attribute
-    "dont_use_mediainfo_mp4": False,
-
-    // specific mediainfo binary path
-    "mediainfo_bin": null,
-
     "debug_i18n": false,
 
     // per-provider-config

README.md

@@ -14,17 +14,13 @@ Check out **[the Sub-Zero Wiki](https://github.com/pannal/Sub-Zero.bundle/wiki)*
 
 ---
 
-**[Kitana is now required to have a UI](https://github.com/pannal/Kitana)**
-
----
-
 **[The future of Sub-Zero](https://www.reddit.com/r/PleX/comments/9n9qjl/subzero_the_future/)**
 
 ---
 
 ## Helping development
 
-If you like this, buy me a beer: <br>[![Donate](https://www.paypalobjects.com/en_US/i/btn/btn_donate_LG.gif)](https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=G9VKR2B8PMNKG) <br>or become a Patreon starting at **1 $ / month** <br><a href="https://www.patreon.com/subzero_plex" target="_blank"><img src="https://i0.wp.com/tablecakes.com/wp-content/uploads/2018/08/become-a-patron-button.png" height="54" /></a> <br>or use the OpenSubtitles Sub-Zero affiliate link to become VIP <br>**10€/year, ad-free subs, 1000 subs/day, no-cache *VIP* server**<br><a href="http://v.ht/osvip" target="_blank"><img src="https://static.opensubtitles.org/gfx/logo.gif" height="50" /></a> 
+If you like this, buy me a beer: <br>[![Donate](https://www.paypalobjects.com/en_US/i/btn/btn_donate_LG.gif)](https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=G9VKR2B8PMNKG) <br>or become a Patreon starting at **1 $ / month** <br><a href="https://www.patreon.com/subzero_plex" target="_blank"><img src="http://www.wenspencer.com/wp-content/uploads/2017/02/patreon-button.png" height="42" /></a> <br>or use the OpenSubtitles Sub-Zero affiliate link to become VIP <br>**10€/year, ad-free subs, 1000 subs/day, no-cache *VIP* server**<br><a href="http://v.ht/osvip" target="_blank"><img src="https://static.opensubtitles.org/gfx/logo.gif" height="50" /></a> 
 
 If you register with an anti-captcha service and you decide to use [Anti-Captcha.com](http://getcaptchasolution.com/kkvviom7nh), you can use [this affiliate link](http://getcaptchasolution.com/kkvviom7nh) to help development.
 
@@ -95,30 +91,28 @@ the.vbm, mmgoodnow, Vertig0ne, thliu78, tattoomees, ostman, count_confucius, ehe
 ## Changelog
 
 
-2.6.5.3152
+2.6.5.3074
 
-subscene, addic7ed
+subscene, addic7ed and titlovi
 - either of those providers might impose a reCAPTCHA verification. In order to use those providers, please create an account at an AntiCaptcha service ([anti-captcha.com](http://getcaptchasolution.com/kkvviom7nh) or [deathbycaptcha.com](http://deathbycaptcha.com)), add funds, then supply your credentials/apikey in the configuration
 
 Changelog
-- core: don't fall back to default providers if none enabled
-- core: don't process any further if stream info is missing
-- core: support using mediainfo for retrieving MP4 MOV_TEXT subtitle stream titles (PMS bug)
-- core: fix embedded subtitle extraction in some cases (#681, #680)
-- core: scanning: add additional INFO logging for undetected languages
-- core: bazarr-backport: remove existing subtitle file, to support MergerFS
-- core: bazarr-backport: generic 10 minute throttling if uncaught exception occurs
-- providers: addic7ed: fix recaptcha solving; fix show ID retrieval (#681)
-- providers: addic7ed: add timeout on authentication error
-- providers: addic7ed: fix shows with dots in them (Mayans M.C.)
-- providers: addic7ed: fix detection of completed subtitle for non-english users (#686)
-- providers: addic7ed: add more timeouts in the login process
-- providers: argenteam: bazarr-backport: use new url; fixes
+- core: cf: bypass cf 95% of the time without captchas
+- core: fix breaking line endings of certain languages (chinese, UTF-16); fixes #646
+- core: update pysubs2 to 0.2.3
+
+
+2.6.5.3062
+
+Changelog
+- core: cf: optimize
+- core: http: don't query DNS with IPs. thanks @fgump (fixes sonarr/radarr)
+
 
 [older changes](CHANGELOG.md)
 
 
-Subtitles provided by [OpenSubtitles.org](http://www.opensubtitles.org/), [Podnapisi.NET](https://www.podnapisi.net/), [TVSubtitles.net](http://www.tvsubtitles.net/), [Addic7ed.com](http://www.addic7ed.com/), [Legendas TV](http://legendas.tv/), [Napi Projekt](http://www.napiprojekt.pl/), [Shooter](http://shooter.cn/), [Titlovi](http://titlovi.com), [aRGENTeaM](http://argenteam.net), [SubScene](https://subscene.com/), [Hosszupuska](http://hosszupuskasub.com/), [Napisy24](https://napisy24.pl/)
+Subtitles provided by [OpenSubtitles.org](http://www.opensubtitles.org/), [Podnapisi.NET](https://www.podnapisi.net/), [TVSubtitles.net](http://www.tvsubtitles.net/), [Addic7ed.com](http://www.addic7ed.com/), [Legendas TV](http://legendas.tv/), [Napi Projekt](http://www.napiprojekt.pl/), [Shooter](http://shooter.cn/), [Titlovi](http://titlovi.com), [aRGENTeaM](http://argenteam.net), [SubScene](https://subscene.com/), [Hosszupuska](http://hosszupuskasub.com/)
 
 [3rd party licenses](https://github.com/pannal/Sub-Zero.bundle/tree/master/Licenses)