core: add start delay option for slow PMSs by placing "delayed_start" inside Data folder

Fix issue scandir not returning the name of the file inside Docker

CHANGELOG.md

@@ -1,4 +1,97 @@
 
+2.6.4.2881
+- core: extract embedded: fix automatic extraction not actually writing the subtitles to disk under certain circumstances; fixes #598
+- core: sonarr/radarr: don't block the main thread while checking connectivity; fixes #597
+- providers: hosszupuska: fix inconsistent series naming (thanks @morpheus133)
+- providers: opensubtitles: add advanced setting to optionally not skip subtitles with wrong FPS; fixes #578
+
+
+2.6.4.2864
+- core: scanning: don't fail on metadata subtitles with bad language code; fixes #596
+- providers: legendastv, napiprojekt, subscenter, tvsubtitles: fix "No language to search for" issue; fixes #596
+- menu: fix "ignore list list"
+- menu: advanced: add skip next search all recently missing subtitles entry
+
+
+2.6.4.2859
+- core: fix thread.lock error (only affected the history menu, not the actual functionality)
+- core: fix audio-based conditional subtitle decision making; fixes #592
+- core: massively improve metadata subtitle storage
+- providers: opensubtitles: skip non-forced results when searching for forced
+- providers: podnapisi: skip non-forced results when searching for forced
+- submod: common: correctly pad music symbols on either side
+
+
+
+2.6.4.2834
+- core: add option to use custom (Google, Cloudflare) DNS to resolve provider hosts in problematic countries; fixes #547
+- core: add support for downloading subtitles only when the audio streams don't match (any?) configured languages; fixes #519
+- core: add support for an include list instead of an ignore list; add the option to disable SZ by default, then enable it per item/series/section (inverse ignore list)
+- core/menu/config: support forced/foreign subtitles independently
+- core: fallback for OSError on scandir, should fix #532
+- core: add config versioning/migration system
+- core: correctly force non-foreign-only-capable providers off; remove subscene from foreign-only capable providers
+- core: scanning: collect information about audio streams
+- core: use correct storage path when storing subtitle info, when only VTT is used
+- core: fix disabled channel mode
+- core/menu: extract embedded: add extracted embedded subtitles to history
+- core: embedded subtitle streams: don't try parsing the language if inexistant
+- core: subtitle: fix log call, fixes #569
+- core: download best subtitles: only use actually languages searched for
+- core: refiners: tvdb: warn instead of error when no matching series was found
+- core: scanning: re-add expected title to guessit for narrowing down the video title
+- core: resolve #583
+- core: archives: explicitly skip forced subtitles if not searched for, when picking from an archive
+- core: activities/auto-refresh: fix hybrid-plus for movies
+- core: don't disable plugin if all providers throttled; fix #585 #574
+- core: skip cleanup for ignored paths
+- core: update requests to 2.20.0 (fixes security issue)
+- core: update certifi to 2018.10.15
+- core: auto extract: don't overwrite local sub even if unknown to SZ
+- config: set autoclean leftover/unused to off by default
+- providers: opensubtitles: respect rate limit (40 hits/10s); should fix long throttling behaviour
+- providers: opensubtitles: handle bad/inexistant responses
+- providers: opensubtitles: log bad response data
+- providers: opensubtitles: treat empty response as ServiceUnavailable for now
+- providers: opensubtitles: log reason for ServiceUnavailable
+- providers: legendastv: match second title and imdb id
+- providers: titlovi: fix language handling (thanks @viking1304)
+- providers: titlovi: proper handling of archives with both cyrlic and latin subtitles (thanks @viking1304) 
+- providers: titlovi: allow direct subtitle downloads as fallback (when a subtitle, not an archive was returned)
+- providers: hosszupuska: implement site change (thanks @morpheus133)
+- providers: supersubtitles: add base properties to subtitle
+- providers: opensubtitles, podnapisi: fix foreign/forced handling
+- providers: subscene: use original/sceneName if possible
+- menu: fix plugin not responding when ignoring an item in certain menus; fixes #535
+- menu: select active subtitle: return to item details afterwards; correctly set current
+- menu: add item thumbnails to history and a couple of submenus
+- menu: history: use series thumbnail instead of episode screenshot
+- menu: add full soft include/exclude menu handling
+- menu: add support for separate forced and not-forced subtitles
+- menu: fix order of embedded subtitle streams in item detail
+- menu: support S00E00 and equivalent
+- submod: add option to fix only-uppercase subtitles and make them readable
+- submod: keep track of actually applied mods
+- submod: correctly merge mods of the same kind (offset)
+- submod: OCR: add dictionaries for bosnian and norwegian bokmal; update dicts for dan, eng, hrv, spa, srp, swe
+- submod: OCR/HI: skip certain processors for all-caps subs
+- submod: HI: only remove caps before colon if the colon is followed by whitespace or EOL; fixes #542
+- submod: HI: remove MAN:
+- submod: common: improve detection and normalization of quotes, apostrophes
+- submod: common: fix double quotes that are meant to be single quotes inside words
+- submod: common: normalize small hyphens to dash
+- submod: common: remove line only consisting of colon; remove empty colon at start of line
+- submod: common: add space after punctuation
+- submod: common: fix lowercase i for english language
+- submod: common: better fix for music symbols
+- submod: reverse_RTL: also reverse ":,'-" chars in CM_RTL_reverse (thanks @doopler)
+- submod: reverse_RTL: enable mod for arabic, farsi and persian besides hebrew
+- i18n: fix not used translation for recently added missing subtitles menu
+- i18n: fix spanish translation, fixes #543
+- i18n: Hungarian translation is incomplete
+
+
+
 2.5.7.2663
 - implement translations for the channel and the settings
 - i18n: German (myself)

Contents/Code/__init__.py

@@ -70,7 +70,7 @@ def Start():
     ValidatePrefs()
     Log.Debug(config.full_version)
 
-    if not config.permissions_ok:
+    if config.initialized and not config.permissions_ok:
         Log.Error("Insufficient permissions on library folders:")
         for title, path in config.missing_permissions:
             Log.Error("Insufficient permissions on library %s, folder: %s" % (title, path))
@@ -99,7 +99,7 @@ def update_local_media(videos, ignore_parts_cleanup=None):
         support.localmedia.find_subtitles(video["plex_part"], ignore_parts_cleanup=ignore_parts_cleanup)
 
 
-def agent_extract_embedded(video_part_map, history_storage=None):
+def agent_extract_embedded(video_part_map):
     try:
         subtitle_storage = get_subtitle_storage()
 
@@ -109,24 +109,29 @@ def agent_extract_embedded(video_part_map, history_storage=None):
         for scanned_video, part_info in video_part_map.iteritems():
             plexapi_item = scanned_video.plexapi_metadata["item"]
             stored_subs = subtitle_storage.load_or_new(plexapi_item)
+            valid_langs_in_media = audio_streams_match_languages(scanned_video, config.get_lang_list(ordered=True))
 
-            if audio_streams_match_languages(scanned_video, list(config.lang_list)):
+            if not config.lang_list.difference(valid_langs_in_media):
                 Log.Debug("Skipping embedded subtitle extraction for %s, audio streams are in correct language(s)",
                           plexapi_item.rating_key)
                 continue
 
             for plexapi_part in get_all_parts(plexapi_item):
                 item_count = item_count + 1
+                used_one_unknown_stream = False
                 for requested_language in config.lang_list:
                     embedded_subs = stored_subs.get_by_provider(plexapi_part.id, requested_language, "embedded")
                     current = stored_subs.get_any(plexapi_part.id, requested_language) or \
-                        requested_language in scanned_video.subtitle_languages
+                        requested_language in scanned_video.external_subtitle_languages
 
                     if not embedded_subs:
-                        stream_data = get_embedded_subtitle_streams(plexapi_part, requested_language=requested_language)
+                        stream_data = get_embedded_subtitle_streams(plexapi_part, requested_language=requested_language,
+                                                                    skip_unknown=used_one_unknown_stream)
 
                         if stream_data:
                             stream = stream_data[0]["stream"]
+                            if stream_data[0]["is_unknown"]:
+                                used_one_unknown_stream = True
 
                             to_extract.append(({scanned_video: part_info}, plexapi_part, str(stream.index),
                                                str(requested_language), not current))
@@ -139,7 +144,7 @@ def agent_extract_embedded(video_part_map, history_storage=None):
         if to_extract:
             Log.Info("Triggering extraction of %d embedded subtitles of %d items", len(to_extract), item_count)
             Thread.Create(multi_extract_embedded, stream_list=to_extract, refresh=True, with_mods=True,
-                          single_thread=not config.advanced.auto_extract_multithread, history_storage=history_storage)
+                          single_thread=not config.advanced.auto_extract_multithread)
     except:
         Log.Error("Something went wrong when auto-extracting subtitles, continuing: %s", traceback.format_exc())
 
@@ -176,7 +181,6 @@ class SubZeroAgent(object):
             return
 
         intent = get_intent()
-        history = get_history()
 
         item_ids = []
         try:
@@ -222,9 +226,9 @@ class SubZeroAgent(object):
             # auto extract embedded
             if config.embedded_auto_extract:
                 if config.plex_transcoder:
-                    agent_extract_embedded(scanned_video_part_map, history_storage=history)
+                    agent_extract_embedded(scanned_video_part_map)
                 else:
-                    Log.Warning("Plex Transcoder not found, can't auto extract")
+                    Log.Warn("Plex Transcoder not found, can't auto extract")
 
             # clear missing subtitles menu data
             if not scheduler.is_task_running("MissingSubtitles"):
@@ -276,6 +280,7 @@ class SubZeroAgent(object):
                     for video, video_subtitles in downloaded_subtitles.items():
                         # store item(s) in history
                         for subtitle in video_subtitles:
+                            history = get_history()
                             item_title = get_title_for_video_metadata(video.plexapi_metadata, add_section_title=False)
                             history.add(item_title, video.id, section_title=video.plexapi_metadata["section"],
                                         thumb=video.plexapi_metadata["super_thumb"],
@@ -291,9 +296,6 @@ class SubZeroAgent(object):
             # update the menu state
             set_refresh_menu_state(None)
 
-            history.destroy()
-            history = None
-
             # notify any running tasks about our finished update
             for item_id in item_ids:
                 #scheduler.signal("updated_metadata", item_id)

Contents/Code/interface/advanced.py

@@ -61,6 +61,10 @@ def AdvancedMenu(randomize=None, header=None, message=None):
         key=Callback(SkipFindBetterSubtitles, randomize=timestamp()),
         title=pad_title(_("Skip next find better subtitles (sets last run to now)")),
     ))
+    oc.add(DirectoryObject(
+        key=Callback(SkipRecentlyAddedMissing, randomize=timestamp()),
+        title=pad_title(_("Skip next find recently added with missing subtitles (sets last run to now)")),
+    ))
     oc.add(DirectoryObject(
         key=Callback(TriggerStorageMaintenance, randomize=timestamp()),
         title=pad_title(_("Trigger subtitle storage maintenance")),
@@ -207,6 +211,19 @@ def SkipFindBetterSubtitles(randomize=None):
     )
 
 
+@route(PREFIX + '/skipram')
+@debounce
+def SkipRecentlyAddedMissing(randomize=None):
+    task = scheduler.task("SearchAllRecentlyAddedMissing")
+    task.last_run = datetime.datetime.now()
+
+    return AdvancedMenu(
+        randomize=timestamp(),
+        header=_("Success"),
+        message=_("SearchAllRecentlyAddedMissing skipped")
+    )
+
+
 @route(PREFIX + '/triggermaintenance')
 @debounce
 def TriggerStorageMaintenance(randomize=None):

Contents/Code/interface/main.py

@@ -1,6 +1,8 @@
 # coding=utf-8
 
-from subzero.constants import PREFIX, TITLE, ART
+import time
+
+from subzero.constants import PREFIX, TITLE, ART, START_DELAY
 from support.config import config
 from support.helpers import pad_title, timestamp, df, display_language
 from support.scheduler import scheduler
@@ -27,45 +29,56 @@ def fatality(randomize=None, force_title=None, header=None, message=None, only_r
                          no_history=no_history,
                          replace_parent=replace_parent, no_cache=True)
 
-    # always re-check permissions
-    config.refresh_permissions_status()
+    if config.initialized:
+        # always re-check permissions
+        config.refresh_permissions_status()
 
-    # always re-check enabled sections
-    config.refresh_enabled_sections()
+        # always re-check enabled sections
+        config.refresh_enabled_sections()
 
-    if config.lock_menu and not config.pin_correct:
-        oc.add(DirectoryObject(
-            key=Callback(PinMenu, randomize=timestamp()),
-            title=pad_title(_("Enter PIN")),
-            summary=_("The owner has restricted the access to this menu. Please enter the correct pin"),
-        ))
-        return oc
-
-    if not config.permissions_ok and config.missing_permissions:
-        if not isinstance(config.missing_permissions, list):
+        if config.lock_menu and not config.pin_correct:
             oc.add(DirectoryObject(
-                key=Callback(fatality, randomize=timestamp()),
-                title=pad_title(_("Insufficient permissions")),
-                summary=config.missing_permissions,
+                key=Callback(PinMenu, randomize=timestamp()),
+                title=pad_title(_("Enter PIN")),
+                summary=_("The owner has restricted the access to this menu. Please enter the correct pin"),
             ))
-        else:
-            for title, path in config.missing_permissions:
+            return oc
+
+        if not config.permissions_ok and config.missing_permissions:
+            if not isinstance(config.missing_permissions, list):
                 oc.add(DirectoryObject(
                     key=Callback(fatality, randomize=timestamp()),
                     title=pad_title(_("Insufficient permissions")),
-                    summary=_("Insufficient permissions on library %(title)s, folder: %(path)s",
-                              title=title,
-                              path=path),
+                    summary=config.missing_permissions,
                 ))
-        return oc
+            else:
+                for title, path in config.missing_permissions:
+                    oc.add(DirectoryObject(
+                        key=Callback(fatality, randomize=timestamp()),
+                        title=pad_title(_("Insufficient permissions")),
+                        summary=_("Insufficient permissions on library %(title)s, folder: %(path)s",
+                                  title=title,
+                                  path=path),
+                    ))
+            return oc
 
-    if not config.enabled_sections:
-        oc.add(DirectoryObject(
-            key=Callback(fatality, randomize=timestamp()),
-            title=pad_title(_("I'm not enabled!")),
-            summary=_("Please enable me for some of your libraries in your server settings; currently I do nothing"),
-        ))
-        return oc
+        if not config.enabled_sections:
+            oc.add(DirectoryObject(
+                key=Callback(fatality, randomize=timestamp()),
+                title=pad_title(_("I'm not enabled!")),
+                summary=_("Please enable me for some of your libraries in your server settings; currently I do nothing"),
+            ))
+            return oc
+    else:
+        if config.delay_system_queries:
+            elapsed = int(START_DELAY - (time.time() - config.start_delay_elapsed))
+            oc.add(DirectoryObject(
+                key=Callback(fatality, randomize=timestamp()),
+                title=pad_title(_("Finalizing ..."
+                                  if elapsed <= 0 else "Initializing, please wait %s seconds ..." % elapsed)),
+                summary=_("Start is delayed by %s seconds to cope with a slow PMS" % int(START_DELAY)),
+            ))
+            return oc
 
     if not only_refresh:
         if Dict["current_refresh_state"]:

Contents/Code/interface/menu.py

@@ -276,6 +276,40 @@ def replace_item(obj, key, replace_value):
     return obj
 
 
+def check_connections():
+    # debug drone
+    Log.Debug("Checking connections ...")
+    log_buffer = []
+    try:
+        from subliminal_patch.refiners.drone import SonarrClient, RadarrClient
+        log_buffer.append(["----- Connections -----"])
+        for key, cls in [("sonarr", SonarrClient), ("radarr", RadarrClient)]:
+            if key in config.refiner_settings:
+                cname = key.capitalize()
+                try:
+                    status = cls(**config.refiner_settings[key]).status(timeout=5)
+                except HTTPError, e:
+                    if e.response.status_code == 401:
+                        log_buffer.append(("%s: NOT WORKING - BAD API KEY", cname))
+                    else:
+                        log_buffer.append(("%s: NOT WORKING - %s", cname, traceback.format_exc()))
+                except:
+                    log_buffer.append(("%s: NOT WORKING - %s", cname, traceback.format_exc()))
+                else:
+                    if status and status["version"]:
+                        log_buffer.append(("%s: OK - %s", cname, status["version"]))
+                    else:
+                        log_buffer.append(("%s: NOT WORKING - %s", cname))
+    except:
+        log_buffer.append(("Something went really wrong when evaluating Sonarr/Radarr: %s", traceback.format_exc()))
+    finally:
+        Core.log.setLevel(logging.DEBUG)
+        for entry in log_buffer:
+            Log.Debug(*entry)
+
+        Core.log.setLevel(logging.getLevelName(Prefs["log_level"]))
+
+
 @route(PREFIX + '/ValidatePrefs', enforce_route=True)
 def ValidatePrefs():
     Core.log.setLevel(logging.DEBUG)
@@ -362,30 +396,8 @@ def ValidatePrefs():
             "subtitles.save.filesystem", ]:
         Log.Debug("Pref.%s: %s", attr, Prefs[attr])
 
-    # debug drone
     if "sonarr" in config.refiner_settings or "radarr" in config.refiner_settings:
-        Log.Debug("----- Connections -----")
-        try:
-            from subliminal_patch.refiners.drone import SonarrClient, RadarrClient
-            for key, cls in [("sonarr", SonarrClient), ("radarr", RadarrClient)]:
-                if key in config.refiner_settings:
-                    cname = key.capitalize()
-                    try:
-                        status = cls(**config.refiner_settings[key]).status()
-                    except HTTPError, e:
-                        if e.response.status_code == 401:
-                            Log.Debug("%s: NOT WORKING - BAD API KEY", cname)
-                        else:
-                            Log.Debug("%s: NOT WORKING - %s", cname, traceback.format_exc())
-                    except:
-                        Log.Debug("%s: NOT WORKING - %s", cname, traceback.format_exc())
-                    else:
-                        if status and status["version"]:
-                            Log.Debug("%s: OK - %s", cname, status["version"])
-                        else:
-                            Log.Debug("%s: NOT WORKING - %s", cname)
-        except:
-            Log.Debug("Something went really wrong when evaluating Sonarr/Radarr: %s", traceback.format_exc())
+        Thread.Create(check_connections)
 
     # fixme: check existance of and os access of logs
     Log.Debug("----- Environment -----")

Contents/Code/interface/menu_helpers.py

@@ -169,7 +169,6 @@ def extract_embedded_sub(**kwargs):
     part = kwargs.pop("part", get_part(plex_item, part_id))
     scanned_videos = kwargs.pop("scanned_videos", None)
     extract_mode = kwargs.pop("extract_mode", "a")
-    history_storage = kwargs.pop("history_storage", None)
 
     any_successful = False
 
@@ -220,17 +219,12 @@ def extract_embedded_sub(**kwargs):
                     # add item to history
                     item_title = get_title_for_video_metadata(video.plexapi_metadata,
                                                               add_section_title=False, add_episode_title=True)
-                    if history_storage:
-                        history = history_storage
-                    else:
-                        history = get_history()
 
+                    history = get_history()
                     history.add(item_title, video.id, section_title=video.plexapi_metadata["section"],
                                 thumb=video.plexapi_metadata["super_thumb"],
                                 subtitle=subtitle, mode=extract_mode)
-
-                    if not history_storage:
-                        history.destroy()
+                    history.destroy()
 
                     any_successful = True
 

Contents/Code/support/config.py

@@ -7,6 +7,7 @@ import sys
 import rarfile
 import jstyleson
 import datetime
+import time
 
 import subliminal
 import subliminal_patch
@@ -22,7 +23,7 @@ from subliminal.cli import MutexLock
 from subzero.lib.io import FileIO, get_viable_encoding
 from subzero.lib.dict import Dicked
 from subzero.util import get_root_path
-from subzero.constants import PLUGIN_NAME, PLUGIN_IDENTIFIER, MOVIE, SHOW, MEDIA_TYPE_TO_STRING
+from subzero.constants import PLUGIN_NAME, PLUGIN_IDENTIFIER, MOVIE, SHOW, MEDIA_TYPE_TO_STRING, START_DELAY
 from subzero.prefs import get_user_prefs, update_user_prefs
 from dogpile.cache.region import register_backend as register_cache_backend
 from lib import Plex
@@ -79,7 +80,7 @@ PROVIDER_THROTTLE_MAP = {
 
 
 class Config(object):
-    config_version = 2
+    config_version = 3
     libraries_root = None
     plugin_info = ""
     version = None
@@ -148,10 +149,15 @@ class Config(object):
     unrar = None
     adv_cfg_path = None
     use_custom_dns = False
+    delay_system_queries = False
 
     store_recently_played_amount = 40
 
     initialized = False
+    system_queries_done = False
+    base_init_done = False
+    system_queries_timer = None
+    start_delay_elapsed = None
 
     def initialize(self):
         self.libraries_root = os.path.abspath(os.path.join(get_root_path(), ".."))
@@ -169,9 +175,12 @@ class Config(object):
         self.set_log_paths()
         self.app_support_path = Core.app_support_path
         self.data_path = getattr(Data, "_core").storage.data_path
+        self.delay_system_queries = os.path.isfile(os.path.join(self.data_path, "delayed_start"))
         self.data_items_path = os.path.join(self.data_path, "DataItems")
         self.universal_plex_token = self.get_universal_plex_token()
         self.plex_token = os.environ.get("PLEXTOKEN", self.universal_plex_token)
+        self.new_style_cache = cast_bool(Prefs['new_style_cache'])
+        self.pack_cache_dir = self.get_pack_cache_dir()
         try:
             self.migrate_prefs()
         except:
@@ -180,8 +189,6 @@ class Config(object):
         subzero.constants.DEFAULT_TIMEOUT = lib.DEFAULT_TIMEOUT = self.pms_request_timeout = \
             min(cast_int(Prefs['pms_request_timeout'], 15), 45)
         self.low_impact_mode = cast_bool(Prefs['low_impact_mode'])
-        self.new_style_cache = cast_bool(Prefs['new_style_cache'])
-        self.pack_cache_dir = self.get_pack_cache_dir()
         self.advanced = self.get_advanced_config()
         self.debug_i18n = self.advanced.debug_i18n
 
@@ -205,8 +212,25 @@ class Config(object):
         self.missing_permissions = []
         self.include_exclude_sz_files = cast_bool(Prefs["subtitles.include_exclude_fs"])
         self.include_exclude_paths = self.parse_include_exclude_paths()
-        self.enabled_sections = self.check_enabled_sections()
-        self.permissions_ok = self.check_permissions()
+
+        self.system_queries_done = False
+
+        def system_queries():
+            self.enabled_sections = self.check_enabled_sections()
+            self.permissions_ok = self.check_permissions()
+            self.system_queries_done = True
+            self.system_queries_timer = None
+            if self.base_init_done:
+                self.initialized = True
+
+        if self.delay_system_queries:
+            if not self.system_queries_timer or not self.system_queries_timer.is_alive():
+                Log.Info("Waiting %s seconds until querying the system endpoints of your PMS" % START_DELAY)
+                Thread.CreateTimer(START_DELAY, system_queries)
+                self.start_delay_elapsed = time.time()
+        else:
+            system_queries()
+
         self.notify_executable = self.check_notify_executable()
         self.remove_hi = cast_bool(Prefs['subtitles.remove_hi'])
         self.remove_tags = cast_bool(Prefs['subtitles.remove_tags'])
@@ -228,7 +252,11 @@ class Config(object):
         self.embedded_auto_extract = cast_bool(Prefs["subtitles.embedded.autoextract"])
         self.ietf_as_alpha3 = cast_bool(Prefs["subtitles.language.ietf_normalize"])
         self.use_custom_dns = cast_bool(Prefs['use_custom_dns'])
-        self.initialized = True
+
+        self.base_init_done = True
+
+        if self.system_queries_done:
+            self.initialized = True
 
     def migrate_prefs(self):
         config_version = 0 if "config_version" not in Dict else Dict["config_version"]
@@ -256,6 +284,9 @@ class Config(object):
 
                 if update_prefs:
                     update_user_prefs(update_prefs, Prefs, Log)
+            else:
+                Dict["config_version"] = self.config_version
+                Dict.Save()
 
     def migrate_prefs_to_1(self, user_prefs, **kwargs):
         update_prefs = {}
@@ -275,6 +306,15 @@ class Config(object):
 
         return update_prefs
 
+    def migrate_prefs_to_3(self, user_prefs, **kwargs):
+        if config.new_style_cache:
+            self.init_cache()
+            try:
+                subliminal.region.backend.clear()
+            except:
+                pass
+        return {}
+
     def init_libraries(self):
         try_executables = []
         custom_unrar = os.environ.get("SZ_UNRAR_TOOL")
@@ -319,7 +359,8 @@ class Config(object):
         if self.new_style_cache:
             subliminal.region.configure('subzero.cache.file', expiration_time=datetime.timedelta(days=30),
                                         arguments={'appname': "sz_cache",
-                                                   'app_cache_dir': self.data_path})
+                                                   'app_cache_dir': self.data_path},
+                                        replace_existing_backend=True)
             Log.Info("Using new style file based cache!")
             return
 
@@ -359,14 +400,15 @@ class Config(object):
             try:
                 subliminal.region.configure('dogpile.cache.dbm', expiration_time=datetime.timedelta(days=30),
                                             arguments={'filename': dbfn,
-                                                       'lock_factory': MutexLock})
+                                                       'lock_factory': MutexLock},
+                                            replace_existing_backend=True)
                 Log.Info("Using file based cache!")
                 return
             except:
                 self.dbm_supported = False
 
         Log.Warn("Not using file based cache!")
-        subliminal.region.configure('dogpile.cache.memory')
+        subliminal.region.configure('dogpile.cache.memory', replace_existing_backend=True)
 
     def sync_cache(self):
         if not self.new_style_cache:
@@ -633,7 +675,7 @@ class Config(object):
         return enabled_sections
 
     # Prepare a list of languages we want subs for
-    def get_lang_list(self, provider=None):
+    def get_lang_list(self, provider=None, ordered=False):
         # advanced settings
         if provider and self.advanced.providers and provider in self.advanced.providers:
             adv_languages = self.advanced.providers[provider].get("languages", None)
@@ -654,21 +696,21 @@ class Config(object):
                 if adv_out:
                     return adv_out
 
-        l = {Language.fromietf(Prefs["langPref1a"])}
+        l = [Language.fromietf(Prefs["langPref1a"])]
         lang_custom = Prefs["langPrefCustom"].strip()
 
         if Prefs['subtitles.only_one']:
-            return l
+            return set(l) if not ordered else l
 
         if Prefs["langPref2a"] != "None":
             try:
-                l.update({Language.fromietf(Prefs["langPref2a"])})
+                l.append(Language.fromietf(Prefs["langPref2a"]))
             except:
                 pass
 
         if Prefs["langPref3a"] != "None":
             try:
-                l.update({Language.fromietf(Prefs["langPref3a"])})
+                l.append(Language.fromietf(Prefs["langPref3a"]))
             except:
                 pass
 
@@ -682,12 +724,12 @@ class Config(object):
                         real_lang = Language.fromname(lang)
                     except:
                         continue
-                l.update({real_lang})
+                l.append(real_lang)
 
         if self.forced_also:
-            langs_to_force = []
             if Prefs["subtitles.when_forced"] == "Always":
-                langs_to_force = list(l)
+                for lang in list(l):
+                    l.append(Language.rebuild(lang, forced=True))
 
             else:
                 for (setting, index) in (("Only for Subtitle Language (1)", 0),
@@ -695,19 +737,21 @@ class Config(object):
                                          ("Only for Subtitle Language (3)", 2)):
                     if Prefs["subtitles.when_forced"] == setting:
                         try:
-                            langs_to_force.append(list(l)[index])
+                            l.append(Language.rebuild(list(l)[index], forced=True))
                             break
                         except:
                             pass
 
-            for lang in langs_to_force:
-                l.add(Language.rebuild(lang, forced=True))
-
         elif self.forced_only:
             for lang in l:
                 lang.forced = True
 
-        return l
+        if not self.normal_subs:
+            for lang in l[:]:
+                if not lang.forced:
+                    l.remove(lang)
+
+        return set(l) if not ordered else l
 
     lang_list = property(get_lang_list)
 
@@ -815,7 +859,10 @@ class Config(object):
 
     def get_provider_settings(self):
         os_use_https = self.advanced.providers.opensubtitles.use_https \
-            if self.advanced.providers.opensubtitles.use_https != None else True
+            if self.advanced.providers.opensubtitles.use_https is not None else True
+
+        os_skip_wrong_fps = self.advanced.providers.opensubtitles.skip_wrong_fps \
+            if self.advanced.providers.opensubtitles.skip_wrong_fps is not None else True
 
         provider_settings = {'addic7ed': {'username': Prefs['provider.addic7ed.username'],
                                           'password': Prefs['provider.addic7ed.password'],
@@ -829,6 +876,7 @@ class Config(object):
                                                'is_vip': cast_bool(Prefs['provider.opensubtitles.is_vip']),
                                                'use_ssl': os_use_https,
                                                'timeout': self.advanced.providers.opensubtitles.timeout or 15,
+                                               'skip_wrong_fps': os_skip_wrong_fps,
                                                },
                              'podnapisi': {
                                  'only_foreign': self.forced_only,
@@ -968,27 +1016,37 @@ class Config(object):
             self.activity_mode = "next_episode"
 
     def get_plex_transcoder(self):
+        paths = []
         base_path = os.environ.get("PLEX_MEDIA_SERVER_HOME", None)
-        if not base_path:
-            # fall back to bundled plugins path
-            bundle_path = os.environ.get("PLEXBUNDLEDPLUGINSPATH", None)
-            if bundle_path:
-                base_path = os.path.normpath(os.path.join(bundle_path, "..", ".."))
+        if base_path:
+            paths.append(base_path)
 
+        bundle_path = os.environ.get("PLEXBUNDLEDPLUGINSPATH", None)
+        if bundle_path:
+            paths.append(os.path.normpath(os.path.join(bundle_path, "..", "..")))
+
+        paths.append(self.app_support_path)
+
+        bns = []
         if sys.platform == "darwin":
-            fn = os.path.join(base_path, "MacOS", "Plex Transcoder")
+            bns.append(("MacOS", "Plex Transcoder"))
         elif mswindows:
-            fn = os.path.join(base_path, "plextranscoder.exe")
+            bns = [("plextranscoder.exe",), ("plex transcoder.exe",)]
         else:
-            fn = os.path.join(base_path, "Plex Transcoder")
+            bns.append(("Plex Transcoder",))
 
-        if os.path.isfile(fn):
-            return fn
+        for path in paths:
+            for bn in bns:
+                fn = os.path.join(path, *bn)
 
-        # look inside Resources folder as fallback, as well
-        fn = os.path.join(base_path, "Resources", "Plex Transcoder")
-        if os.path.isfile(fn):
-            return fn
+                if os.path.isfile(fn):
+                    return fn
+
+            # look inside Resources folder as fallback, as well
+            for vbn in ("Plex Transcoder", "plextranscoder.exe", "plex transcoder.exe"):
+                fn = os.path.join(path, "Resources", vbn)
+                if os.path.isfile(fn):
+                    return fn
 
     def parse_rename_mode(self):
         # fixme: exact_filenames should be determined via callback combined with info about the current video

Contents/Code/support/download.py

@@ -6,8 +6,7 @@ from subzero.language import Language
 import subliminal_patch as subliminal
 
 from support.config import config
-from support.helpers import cast_bool, audio_streams_match_languages
-from subtitlehelpers import get_subtitles_from_metadata
+from support.helpers import audio_streams_match_languages
 from subliminal_patch import compute_score
 from support.plex_media import get_blacklist_from_part_map
 from subzero.video import refine_video
@@ -15,9 +14,17 @@ from support.storage import get_pack_data, store_pack_data
 
 
 def get_missing_languages(video, part):
-    languages = set([Language.rebuild(l) for l in config.lang_list])
+    languages_list = config.get_lang_list(ordered=True)
+    languages = set(languages_list)
+    valid_langs_in_media = set()
 
-    if audio_streams_match_languages(video, list(config.lang_list)):
+    if Prefs["subtitles.when"] != "Always":
+        valid_langs_in_media = audio_streams_match_languages(video, languages_list)
+        languages = languages.difference(valid_langs_in_media)
+        if languages:
+            Log.Debug("Languages missing after taking the audio streams into account: %s" % languages)
+
+    if valid_langs_in_media and not languages:
         Log.Debug("Skipping subtitle search for %s, audio streams are in correct language(s)",
                   video)
         return set()
@@ -30,14 +37,6 @@ def get_missing_languages(video, part):
                 alpha3_map[language.alpha3] = language.country
                 language.country = None
 
-    if not Prefs['subtitles.save.filesystem']:
-        # scan for existing metadata subtitles
-        meta_subs = get_subtitles_from_metadata(part)
-        for language, subList in meta_subs.iteritems():
-            if subList:
-                video.subtitle_languages.add(language)
-                Log.Debug("Found metadata subtitle %s for %s", language, video)
-
     have_languages = video.subtitle_languages.copy()
     if config.ietf_as_alpha3:
         for language in have_languages:

Contents/Code/support/helpers.py

@@ -12,10 +12,12 @@ import subprocess
 import sys
 from collections import OrderedDict
 
+from babelfish.exceptions import LanguageError
+
 import chardet
 
 from bs4 import UnicodeDammit
-from subzero.language import Language
+from subzero.language import Language, language_from_stream
 from subzero.analytics import track_event
 
 mswindows = (sys.platform == "win32")
@@ -274,6 +276,8 @@ def get_item_hints(data):
                 "title": data["original_title"] or data["series"],
             }
         )
+    if hints["title"]:
+        hints["title"] = hints["title"].replace(":", "")
     return hints
 
 
@@ -386,42 +390,44 @@ def get_language_from_stream(lang_code):
         if lang and lang != "xx":
             # Log.Debug("Found language: %r", lang)
             return Language.fromietf(lang)
+        elif lang:
+            try:
+                return language_from_stream(lang)
+            except LanguageError:
+                pass
 
 
 def audio_streams_match_languages(video, languages):
-    if video.audio_languages:
-        decision = []
-
+    without_forced = filter(lambda x: not x.forced, languages)
+    if video.audio_languages and without_forced:
         if Prefs["subtitles.when"] == "Always":
-            decision.append(False)
+            return set()
 
         elif Prefs["subtitles.when"] == "When main audio stream is not Subtitle Language (1)":
-            if video.audio_languages[0] == languages[0]:
-                decision.append(True)
+            if video.audio_languages[0] == without_forced[0]:
+                return set(without_forced)
 
         elif Prefs["subtitles.when"] == "When any audio stream is not Subtitle Language (1)":
-            if languages[0] in video.audio_languages:
-                decision.append(True)
+            if without_forced[0] in video.audio_languages:
+                return set(without_forced)
 
         elif Prefs["subtitles.when"] == "When main audio stream is not any configured language":
-            if video.audio_languages[0] in languages:
-                decision.append(True)
+            if video.audio_languages[0] in without_forced:
+                return set(without_forced)
 
         elif Prefs["subtitles.when"] == "When any audio stream is not any configured language":
-            if set(video.audio_languages).intersection(set(languages)):
-                decision.append(True)
+            matching = set(video.audio_languages).intersection(set(without_forced))
+            if matching:
+                return set(without_forced)
 
-        if Prefs["subtitles.when_forced"] in [
-            "Always",
-            "Only for Subtitle Language (1)",
-            "Only for Subtitle Language (2)",
-            "Only for Subtitle Language (3)"
-        ]:
-            decision.append(False)
+        # if Prefs["subtitles.when_forced"] in [
+        #     "Always",
+        #     "Only for Subtitle Language (1)",
+        #     "Only for Subtitle Language (2)",
+        #     "Only for Subtitle Language (3)"
+        # ]:
 
-        return all(decision)
-
-    return False
+    return set()
 
 
 def get_language(lang_short):

Contents/Code/support/plex_media.py

@@ -174,9 +174,11 @@ def get_all_parts(plex_item):
     return parts
 
 
-def get_embedded_subtitle_streams(part, requested_language=None, skip_duplicate_unknown=True):
+def get_embedded_subtitle_streams(part, requested_language=None, skip_duplicate_unknown=True, skip_unknown=False):
     streams = []
+    streams_unknown = []
     has_unknown = False
+    found_requested_language = False
     for stream in part.streams:
         # subtitle stream
         if stream.stream_type == 3 and not stream.stream_key and stream.codec in TEXT_SUBTITLE_EXTS:
@@ -196,14 +198,19 @@ def get_embedded_subtitle_streams(part, requested_language=None, skip_duplicate_
                 language = Language.rebuild(list(config.lang_list)[0], forced=is_forced)
                 is_unknown = True
                 has_unknown = True
+                streams_unknown.append({"stream": stream, "is_unknown": is_unknown, "language": language,
+                                        "is_forced": is_forced})
 
-            if not requested_language or found_requested_language or has_unknown:
+            if not requested_language or found_requested_language:
                 streams.append({"stream": stream, "is_unknown": is_unknown, "language": language,
                                 "is_forced": is_forced})
 
                 if found_requested_language:
                     break
 
+    if streams_unknown and not found_requested_language and not skip_unknown:
+        streams = streams_unknown
+
     return streams
 
 

Contents/Code/support/scanning.py

@@ -7,12 +7,12 @@ from support.lib import Plex, get_intent
 from support.plex_media import get_stream_fps
 from support.storage import get_subtitle_storage
 from support.config import config, TEXT_SUBTITLE_EXTS
-
+from support.subtitlehelpers import get_subtitles_from_metadata
 from subzero.video import parse_video, set_existing_languages
 from subzero.language import language_from_stream, Language
 
 
-def scan_video(pms_video_info, ignore_all=False, hints=None, rating_key=None, providers=None, skip_hashing=False):
+def prepare_video(pms_video_info, ignore_all=False, hints=None, rating_key=None, providers=None, skip_hashing=False):
     """
     returnes a subliminal/guessit-refined parsed video
     :param pms_video_info:
@@ -29,7 +29,7 @@ def scan_video(pms_video_info, ignore_all=False, hints=None, rating_key=None, pr
     if ignore_all:
         Log.Debug("Force refresh intended.")
 
-    Log.Debug("Scanning video: %s, external_subtitles=%s, embedded_subtitles=%s" % (
+    Log.Debug("Detecting streams: %s, external_subtitles=%s, embedded_subtitles=%s" % (
         plex_part.file, external_subtitles, embedded_subtitles))
 
     known_embedded = []
@@ -86,7 +86,28 @@ def scan_video(pms_video_info, ignore_all=False, hints=None, rating_key=None, pr
     else:
         Log.Warn("Part %s missing of %s, not able to scan internal streams", plex_part.id, rating_key)
 
-    Log.Debug("Known embedded: %r", known_embedded)
+    # metadata subtitles
+    known_metadata_subs = set()
+    meta_subs = get_subtitles_from_metadata(plex_part)
+    for language, subList in meta_subs.iteritems():
+        try:
+            lang = Language.fromietf(Locale.Language.Match(language))
+        except LanguageError:
+            if config.treat_und_as_first:
+                lang = Language.rebuild(list(config.lang_list)[0])
+            else:
+                continue
+
+        if subList:
+            for key in subList:
+                if key.startswith("subzero_md_forced"):
+                    lang = Language.rebuild(lang, forced=True)
+
+                known_metadata_subs.add(lang)
+                Log.Debug("Found metadata subtitle %r:%s for %s", lang, key, plex_part.file)
+
+    Log.Debug("Known metadata subtitles: %r", known_metadata_subs)
+    Log.Debug("Known embedded subtitles: %r", known_embedded)
 
     subtitle_storage = get_subtitle_storage()
     stored_subs = subtitle_storage.load(rating_key)
@@ -106,7 +127,7 @@ def scan_video(pms_video_info, ignore_all=False, hints=None, rating_key=None, pr
             set_existing_languages(video, pms_video_info, external_subtitles=external_subtitles,
                                    embedded_subtitles=embedded_subtitles, known_embedded=known_embedded,
                                    stored_subs=stored_subs, languages=config.lang_list,
-                                   only_one=config.only_one)
+                                   only_one=config.only_one, known_metadata_subs=known_metadata_subs)
 
         # add video fps info
         video.fps = plex_part.fps
@@ -133,9 +154,9 @@ def scan_videos(videos, ignore_all=False, providers=None, skip_hashing=False):
         hints = helpers.get_item_hints(video)
         video["plex_part"].fps = get_stream_fps(video["plex_part"].streams)
         p = providers or config.get_providers(media_type="series" if video["type"] == "episode" else "movies")
-        scanned_video = scan_video(video, ignore_all=force_refresh or ignore_all, hints=hints,
-                                   rating_key=video["id"], providers=p,
-                                   skip_hashing=skip_hashing)
+        scanned_video = prepare_video(video, ignore_all=force_refresh or ignore_all, hints=hints,
+                                      rating_key=video["id"], providers=p,
+                                      skip_hashing=skip_hashing)
 
         if not scanned_video:
             continue

Contents/Code/support/storage.py

@@ -33,7 +33,7 @@ def store_subtitle_info(scanned_video_part_map, downloaded_subtitles, storage_ty
         video_id = str(video.id)
         plex_item = get_item(video_id)
         if not plex_item:
-            Log.Warning("Plex item not found: %s", video_id)
+            Log.Warn("Plex item not found: %s", video_id)
             continue
 
         metadata = video.plexapi_metadata
@@ -143,9 +143,22 @@ def save_subtitles_to_metadata(videos, subtitles):
             else:
                 mp = mediaPart
             pm = Proxy.Media(content, ext="srt", forced="1" if subtitle.language.forced else None)
+            new_key = "subzero_md" + ("_forced" if subtitle.language.forced else "")
             lang = Locale.Language.Match(subtitle.language.alpha2)
-            mp.subtitles[lang].validate_keys({})
-            mp.subtitles[lang]["subzero"] = pm
+
+            for key, proxy in getattr(mp.subtitles[lang], "_proxies").iteritems():
+                if not proxy or not len(proxy) >= 5:
+                    Log.Debug("Can't parse metadata: %s" % repr(proxy))
+                    continue
+                if proxy[0] == "Media":
+                    if not key.startswith("subzero_"):
+                        if key == "subzero":
+                            Log.Debug("Removing legacy metadata subtitle for %s", lang)
+                        del mp.subtitles[lang][key]
+                    Log.Debug("Existing metadata subtitle for %s: %s", lang, key)
+
+            Log.Debug("Adding metadata sub for %s: %s", lang, subtitle)
+            mp.subtitles[lang][new_key] = pm
     return True
 
 

Contents/Code/support/subtitlehelpers.py

@@ -174,10 +174,11 @@ class DefaultSubtitleHelper(SubtitleHelper):
 
         Log('Found subtitle file: ' + self.filename + ' language: ' + language + ' codec: ' + str(
             codec) + ' format: ' + str(format) + ' default: ' + default + ' forced: ' + forced)
-        part.subtitles[language][basename] = Proxy.LocalFile(self.filename, codec=codec, format=format, default=default,
+        key = ("subzero_ex" + "_forced" if forced else "") + basename
+        part.subtitles[language][key] = Proxy.LocalFile(self.filename, codec=codec, format=format, default=default,
                                                              forced=forced)
 
-        lang_sub_map[language] = [basename]
+        lang_sub_map[language] = [key]
         return lang_sub_map
 
 
@@ -194,9 +195,12 @@ def get_subtitles_from_metadata(part):
                 p_type = proxy[0]
 
                 if p_type == "Media":
+                    if not key.startswith("subzero"):
+                        continue
+
                     # metadata subtitle
-                    Log.Debug(u"Found metadata subtitle: %s, %s" % (language, repr(proxy)))
-                    subs[language] = [key]
+                    #Log.Debug(u"Found metadata subtitle: %s, %s, %s" % (language, key, repr(proxy)))
+                    subs[language].append(key)
     return subs
 
 

Contents/Info.plist

@@ -13,7 +13,7 @@
         <key>CFBundleSignature</key>
         <string>????</string>
         <key>CFBundleVersion</key>
-        <string>2.6.4.2834</string>
+        <string>2.6.4.2934</string>
         <key>PlexFrameworkVersion</key>
         <string>2</string>
         <key>PlexPluginClass</key>
@@ -23,7 +23,7 @@
         <key>PlexPluginConsoleLogging</key>
         <string>0</string>
         <key>PlexPluginDevMode</key>
-        <string>0</string>
+        <string>1</string>
          <key>PlexPluginCodePolicy</key>
             <!-- this allows channels to access some python methods which are otherwise blocked, as well as import external code libraries, and interact with the PMS HTTP API -->
             <string>Elevated</string>
@@ -32,7 +32,7 @@
 
 &lt;h1&gt;Sub-Zero for Plex&lt;/h1&gt;&lt;i&gt;Subtitles done right&lt;/i&gt;
 
-Version 2.6.4.2834
+Version 2.6.4.2934 DEV
 
 Originally based on @bramwalet's awesome &lt;a href=&quot;https://github.com/bramwalet/Subliminal.bundle&quot;&gt;Subliminal.bundle&lt;/a&gt;
 

Contents/Libraries/Shared/fcache/cache.py

@@ -5,6 +5,7 @@ import pickle
 import shutil
 import tempfile
 import traceback
+import hashlib
 
 import appdirs
 
@@ -89,7 +90,7 @@ class FileCache(MutableMapping):
     """
 
     def __init__(self, appname, flag='c', mode=0o666, keyencoding='utf-8',
-                 serialize=True, app_cache_dir=None):
+                 serialize=True, app_cache_dir=None, key_file_ext=".txt"):
         """Initialize a :class:`FileCache` object."""
         if not isinstance(flag, str):
             raise TypeError("flag must be str not '{}'".format(type(flag)))
@@ -130,6 +131,7 @@ class FileCache(MutableMapping):
         self._mode = mode
         self._keyencoding = keyencoding
         self._serialize = serialize
+        self.key_file_ext = key_file_ext
 
     def _parse_appname(self, appname):
         """Splits an appname into the appname and subcache components."""
@@ -188,6 +190,11 @@ class FileCache(MutableMapping):
             except:
                 logger.error("Couldn't write content from %r to cache file: %r: %s", ekey, filename,
                              traceback.format_exc())
+            try:
+                self.__write_to_file(filename + self.key_file_ext, ekey)
+            except:
+                logger.error("Couldn't write content from %r to cache file: %r: %s", ekey, filename,
+                             traceback.format_exc())
         self._buffer.clear()
         self._sync = False
 
@@ -196,8 +203,7 @@ class FileCache(MutableMapping):
         raise ValueError("invalid operation on closed cache")
 
     def _encode_key(self, key):
-        """Encode key using *hex_codec* for constructing a cache filename.
-
+        """
         Keys are implicitly converted to :class:`bytes` if passed as
         :class:`str`.
 
@@ -206,16 +212,15 @@ class FileCache(MutableMapping):
             key = key.encode(self._keyencoding)
         elif not isinstance(key, bytes):
             raise TypeError("key must be bytes or str")
-        return codecs.encode(key, 'hex_codec').decode(self._keyencoding)
+        return key.decode(self._keyencoding)
 
     def _decode_key(self, key):
-        """Decode key using hex_codec to retrieve the original key.
-
+        """
         Keys are returned as :class:`str` if serialization is enabled.
         Keys are returned as :class:`bytes` if serialization is disabled.
 
         """
-        bkey = codecs.decode(key.encode(self._keyencoding), 'hex_codec')
+        bkey = key.encode(self._keyencoding)
         return bkey.decode(self._keyencoding) if self._serialize else bkey
 
     def _dumps(self, value):
@@ -226,18 +231,24 @@ class FileCache(MutableMapping):
 
     def _key_to_filename(self, key):
         """Convert an encoded key to an absolute cache filename."""
-        return os.path.join(self.cache_dir, key)
+        if isinstance(key, unicode):
+            key = key.encode(self._keyencoding)
+        return os.path.join(self.cache_dir, hashlib.md5(key).hexdigest())
 
     def _filename_to_key(self, absfilename):
         """Convert an absolute cache filename to a key name."""
-        return os.path.split(absfilename)[1]
+        hkey_hdr_fn = absfilename + self.key_file_ext
+        if os.path.isfile(hkey_hdr_fn):
+            with open(hkey_hdr_fn, 'rb') as f:
+                key = f.read()
+                return key.decode(self._keyencoding) if self._serialize else key
 
     def _all_filenames(self, scandir_generic=True):
         """Return a list of absolute cache filenames"""
         _scandir = _scandir_generic if scandir_generic else scandir
         try:
             for entry in _scandir(self.cache_dir):
-                if entry.is_file(follow_symlinks=False):
+                if entry.is_file(follow_symlinks=False) and not entry.name.endswith(self.key_file_ext):
                     yield os.path.join(self.cache_dir, entry.name)
         except (FileNotFoundError, OSError):
             raise StopIteration
@@ -250,14 +261,17 @@ class FileCache(MutableMapping):
         else:
             return set(file_keys + list(self._buffer))
 
-    def _write_to_file(self, filename, bytesvalue):
+    def __write_to_file(self, filename, value):
         """Write bytesvalue to filename."""
         fh, tmp = tempfile.mkstemp()
         with os.fdopen(fh, self._flag) as f:
-            f.write(self._dumps(bytesvalue))
+            f.write(value)
         rename(tmp, filename)
         os.chmod(filename, self._mode)
 
+    def _write_to_file(self, filename, bytesvalue):
+        self.__write_to_file(filename, self._dumps(bytesvalue))
+
     def _read_from_file(self, filename):
         """Read data from filename."""
         try:
@@ -274,6 +288,7 @@ class FileCache(MutableMapping):
         else:
             filename = self._key_to_filename(ekey)
             self._write_to_file(filename, value)
+            self.__write_to_file(filename + self.key_file_ext, ekey)
 
     def __getitem__(self, key):
         ekey = self._encode_key(key)
@@ -283,8 +298,9 @@ class FileCache(MutableMapping):
             except KeyError:
                 pass
         filename = self._key_to_filename(ekey)
-        if filename not in self._all_filenames():
+        if not os.path.isfile(filename):
             raise KeyError(key)
+
         return self._read_from_file(filename)
 
     def __delitem__(self, key):
@@ -301,6 +317,11 @@ class FileCache(MutableMapping):
         except (IOError, OSError):
             pass
 
+        try:
+            os.remove(filename + self.key_file_ext)
+        except (IOError, OSError):
+            pass
+
     def __iter__(self):
         for key in self._all_keys():
             yield self._decode_key(key)
@@ -310,4 +331,10 @@ class FileCache(MutableMapping):
 
     def __contains__(self, key):
         ekey = self._encode_key(key)
-        return ekey in self._all_keys()
+        if not self._sync:
+            try:
+                return ekey in self._buffer
+            except KeyError:
+                pass
+        filename = self._key_to_filename(ekey)
+        return os.path.isfile(filename)

Contents/Libraries/Shared/pysubs2/ssastyle.py

@@ -56,7 +56,7 @@ class SSAStyle(object):
         self.encoding = 1 #: Charset
 
         for k, v in fields.items():
-            if k in self.FIELDS:
+            if k in self.FIELDS and v is not None:
                 setattr(self, k, v)
             else:
                 raise ValueError("SSAStyle has no field named %r" % k)

Contents/Libraries/Shared/pysubs2/substation.py

@@ -150,7 +150,14 @@ class SubstationFormat(FormatBase):
                 if format_ == "ass":
                     return ass_rgba_to_color(v)
                 else:
-                    return ssa_rgb_to_color(v)
+                    try:
+                        return ssa_rgb_to_color(v)
+                    except ValueError:
+                        try:
+                            return ass_rgba_to_color(v)
+                        except:
+                            return Color(255, 255, 255, 0)
+
             elif f in {"bold", "underline", "italic", "strikeout"}:
                 return v == "-1"
             elif f in {"borderstyle", "encoding", "marginl", "marginr", "marginv", "layer", "alphalevel"}:

Contents/Libraries/Shared/subliminal_patch/core.py

@@ -493,7 +493,7 @@ def scan_video(path, dont_use_actual_file=False, hints=None, providers=None, ski
         raise ValueError('%r is not a valid video extension' % os.path.splitext(path)[1])
 
     dirpath, filename = os.path.split(path)
-    logger.info('Scanning video %r in %r', filename, dirpath)
+    logger.info('Determining basic video properties for %r in %r', filename, dirpath)
 
     # hint guessit the filename itself and its 2 parent directories if we're an episode (most likely
     # Series name/Season/filename), else only one
@@ -514,7 +514,7 @@ def scan_video(path, dont_use_actual_file=False, hints=None, providers=None, ski
 
     # guess
     hints["single_value"] = True
-    if video_type == "movie":
+    if "title" in hints:
         hints["expected_title"] = [hints["title"]]
 
     guessed_result = guessit(guess_from, options=hints)
@@ -559,13 +559,16 @@ def _search_external_subtitles(path, languages=None, only_one=False, scandir_gen
     subtitles = {}
     _scandir = _scandir_generic if scandir_generic else scandir
     for entry in _scandir(dirpath):
+        if not entry.name and not scandir_generic:
+            logger.debug('Could not determine the name of the file, retrying with scandir_generic')
+            return _search_external_subtitles(path, languages, only_one, True)
         if not entry.is_file(follow_symlinks=False):
             continue
 
         p = entry.name
 
         # keep only valid subtitle filenames
-        if not p.startswith(fileroot) or not p.endswith(SUBTITLE_EXTENSIONS):
+        if not p.lower().startswith(fileroot.lower()) or not p.lower().endswith(SUBTITLE_EXTENSIONS):
             continue
 
         p_root, p_ext = os.path.splitext(p)
@@ -600,7 +603,7 @@ def _search_external_subtitles(path, languages=None, only_one=False, scandir_gen
                 logger.error('Cannot parse language code %r', language_code)
                 language = None
 
-        if not language and only_one:
+        elif not language_code and only_one:
             language = Language.rebuild(list(languages)[0], forced=forced)
 
         subtitles[p] = language

Contents/Libraries/Shared/subliminal_patch/http.py

@@ -31,10 +31,17 @@ custom_resolver.nameservers = ['8.8.8.8', '1.1.1.1']
 
 
 class CertifiSession(Session):
+    timeout = 10
+
     def __init__(self):
         super(CertifiSession, self).__init__()
         self.verify = pem_file
 
+    def request(self, *args, **kwargs):
+        if kwargs.get('timeout') is None:
+            kwargs['timeout'] = self.timeout
+        return super(CertifiSession, self).request(*args, **kwargs)
+
 
 class RetryingSession(CertifiSession):
     proxied_functions = ("get", "post")

Contents/Libraries/Shared/subliminal_patch/providers/addic7ed.py

@@ -84,32 +84,35 @@ class Addic7edProvider(_Addic7edProvider):
         # login
         if self.username and self.password:
             ccks = region.get("addic7ed_cookies", expiration_time=86400)
-            do_login = False
             if ccks != NO_VALUE:
-                self.session.cookies.update(ccks)
-                r = self.session.get(self.server_url + 'panel.php', allow_redirects=False, timeout=10)
-                if r.status_code == 302:
-                    logger.info('Addic7ed: Login expired')
-                    do_login = True
-                else:
-                    logger.info('Addic7ed: Reusing old login')
-                    self.logged_in = True
+                try:
+                    self.session.cookies._cookies.update(ccks)
+                    r = self.session.get(self.server_url + 'panel.php', allow_redirects=False, timeout=10)
+                    if r.status_code == 302:
+                        logger.info('Addic7ed: Login expired')
+                        region.delete("addic7ed_cookies")
+                    else:
+                        logger.info('Addic7ed: Reusing old login')
+                        self.logged_in = True
+                        return
+                except:
+                    pass
 
-            if do_login:
-                logger.info('Addic7ed: Logging in')
-                data = {'username': self.username, 'password': self.password, 'Submit': 'Log in'}
-                r = self.session.post(self.server_url + 'dologin.php', data, allow_redirects=False, timeout=10)
+            logger.info('Addic7ed: Logging in')
+            data = {'username': self.username, 'password': self.password, 'Submit': 'Log in'}
+            r = self.session.post(self.server_url + 'dologin.php', data, allow_redirects=False, timeout=10,
+                                  headers={"Referer": self.server_url + "login.php"})
 
-                if "relax, slow down" in r.content:
-                    raise TooManyRequests(self.username)
+            if "relax, slow down" in r.content:
+                raise TooManyRequests(self.username)
 
-                if r.status_code != 302:
-                    raise AuthenticationError(self.username)
+            if r.status_code != 302:
+                raise AuthenticationError(self.username)
 
-                region.set("addic7ed_cookies", r.cookies)
+            region.set("addic7ed_cookies", self.session.cookies._cookies)
 
-                logger.debug('Addic7ed: Logged in')
-                self.logged_in = True
+            logger.debug('Addic7ed: Logged in')
+            self.logged_in = True
 
 
     @region.cache_on_arguments(expiration_time=SHOW_EXPIRATION_TIME)

Contents/Libraries/Shared/subliminal_patch/providers/hosszupuska.py

@@ -33,8 +33,9 @@ def fix_inconsistent_naming(title):
     :rtype: str
 
     """
-    return _fix_inconsistent_naming(title, {"DC's Legends of Tomorrow": "Legends of Tomorrow",
-                                            "Marvel's Jessica Jones": "Jessica Jones"})
+    return _fix_inconsistent_naming(title, {"Stargate Origins": "Stargate: Origins",
+                                            "Marvel's Agents of S.H.I.E.L.D.": "Marvels+Agents+of+S.H.I.E.L.D",
+                                            "Mayans M.C.": "Mayans MC"}, True)
 
 
 logger = logging.getLogger(__name__)
@@ -89,7 +90,7 @@ class HosszupuskaSubtitle(Subtitle):
     def get_matches(self, video):
         matches = set()
         # series
-        if video.series and sanitize(self.series) == sanitize(video.series):
+        if video.series and ( sanitize(self.series) == sanitize(fix_inconsistent_naming(video.series)) or sanitize(self.series) == sanitize(video.series)):
             matches.add('series')
         # season
         if video.season and self.season == video.season:
@@ -150,7 +151,7 @@ class HosszupuskaProvider(Provider, ProviderSubtitleArchiveMixin):
         seasona = "%02d" % season
         episodea = "%02d" % episode
         series = fix_inconsistent_naming(series)
-        seriesa = series.replace(' ', '+').replace('\'', '')
+        seriesa = series.replace(' ', '+')
 
         # get the episode page
         logger.info('Getting the page for episode %s', episode)

Contents/Libraries/Shared/subliminal_patch/providers/legendastv.py

@@ -7,7 +7,8 @@ from subliminal.exceptions import ConfigurationError
 from subliminal.providers.legendastv import LegendasTVSubtitle as _LegendasTVSubtitle, \
     LegendasTVProvider as _LegendasTVProvider, Episode, Movie, guess_matches, guessit, sanitize, region, type_map, \
     raise_for_status, json, SHOW_EXPIRATION_TIME, title_re, season_re, datetime, pytz, NO_VALUE, releases_key, \
-    SUBTITLE_EXTENSIONS
+    SUBTITLE_EXTENSIONS, language_converters
+from subzero.language import Language
 
 logger = logging.getLogger(__name__)
 
@@ -63,6 +64,7 @@ class LegendasTVSubtitle(_LegendasTVSubtitle):
 
 
 class LegendasTVProvider(_LegendasTVProvider):
+    languages = {Language(*l) for l in language_converters['legendastv'].to_legendastv.keys()}
     subtitle_class = LegendasTVSubtitle
 
     def __init__(self, username=None, password=None):

Contents/Libraries/Shared/subliminal_patch/providers/mixins.py

@@ -148,7 +148,8 @@ class ProviderSubtitleArchiveMixin(object):
                         subs_fallback.append(sub_name)
 
         if not matching_sub and not subs_unsure and not subs_fallback:
-            raise ProviderError("None of expected subtitle found in archive")
+            logger.error("None of expected subtitle found in archive")
+            return
 
         elif subs_unsure:
             matching_sub = subs_unsure[0]

Contents/Libraries/Shared/subliminal_patch/providers/napiprojekt.py

@@ -3,6 +3,7 @@ import logging
 
 from subliminal.providers.napiprojekt import NapiProjektProvider as _NapiProjektProvider, \
     NapiProjektSubtitle as _NapiProjektSubtitle, get_subhash
+from subzero.language import Language
 
 logger = logging.getLogger(__name__)
 
@@ -18,6 +19,7 @@ class NapiProjektSubtitle(_NapiProjektSubtitle):
 
 
 class NapiProjektProvider(_NapiProjektProvider):
+    languages = {Language.fromalpha2(l) for l in ['pl']}
     subtitle_class = NapiProjektSubtitle
 
     def query(self, language, hash):

Contents/Libraries/Shared/subliminal_patch/providers/opensubtitles.py

@@ -163,12 +163,13 @@ class OpenSubtitlesProvider(ProviderRetryMixin, _OpenSubtitlesProvider):
         token = region.get("os_token", expiration_time=3600)
         if token is not NO_VALUE:
             try:
-                logger.debug('Trying previous token')
+                logger.debug('Trying previous token: %r', token[:10]+"X"*(len(token)-10))
                 checked(lambda: self.server.NoOperation(token))
                 self.token = token
-                logger.debug("Using previous login token: %s", self.token)
+                logger.debug("Using previous login token: %r", token[:10]+"X"*(len(token)-10))
                 return
             except:
+                logger.debug('Token not valid.')
                 pass
 
         try:
@@ -299,6 +300,9 @@ class OpenSubtitlesProvider(ProviderRetryMixin, _OpenSubtitlesProvider):
             elif also_foreign and foreign_parts_only:
                 language = Language.rebuild(language, forced=True)
 
+            if language not in languages:
+                continue
+
             query_parameters = _subtitle_item.get("QueryParameters")
 
             subtitle = self.subtitle_class(language, hearing_impaired, page_link, subtitle_id, matched_by,

Contents/Libraries/Shared/subliminal_patch/providers/podnapisi.py

@@ -5,7 +5,6 @@ import re
 import io
 
 from zipfile import ZipFile
-from lxml.etree import XMLSyntaxError
 
 from guessit import guessit
 from subliminal.subtitle import guess_matches
@@ -24,11 +23,28 @@ from subliminal import Episode
 from subliminal import Movie
 from subliminal.providers.podnapisi import PodnapisiProvider as _PodnapisiProvider, \
     PodnapisiSubtitle as _PodnapisiSubtitle
+from subliminal_patch.utils import sanitize, fix_inconsistent_naming as _fix_inconsistent_naming
 from subzero.language import Language
 
 logger = logging.getLogger(__name__)
 
 
+def fix_inconsistent_naming(title):
+    """Fix titles with inconsistent naming using dictionary and sanitize them.
+
+    :param str title: original title.
+    :return: new title.
+    :rtype: str
+
+    """
+    d = {}
+    nt = title.replace("Marvels", "").replace("Marvel's", "")
+    if nt != title:
+        d[title] = nt
+
+    return _fix_inconsistent_naming(title, d)
+
+
 class PodnapisiSubtitle(_PodnapisiSubtitle):
     provider_name = 'podnapisi'
     hearing_impaired_verifiable = True
@@ -53,8 +69,8 @@ class PodnapisiSubtitle(_PodnapisiSubtitle):
         # episode
         if isinstance(video, Episode):
             # series
-            if video.series and (sanitize(self.title) in (
-                    sanitize(name) for name in [video.series] + video.alternative_series)):
+            if video.series and (fix_inconsistent_naming(self.title) in (
+                    fix_inconsistent_naming(name) for name in [video.series] + video.alternative_series)):
                 matches.add('series')
             # year
             if video.original_series and self.year is None or video.year and video.year == self.year:
@@ -113,7 +129,7 @@ class PodnapisiProvider(_PodnapisiProvider, ProviderSubtitleArchiveMixin):
 
         season = episode = None
         if isinstance(video, Episode):
-            titles = [video.series] + video.alternative_series
+            titles = [fix_inconsistent_naming(title) for title in [video.series] + video.alternative_series]
             season = video.season
             episode = video.episode
         else:
@@ -158,7 +174,7 @@ class PodnapisiProvider(_PodnapisiProvider, ProviderSubtitleArchiveMixin):
             try:
                 content = self.session.get(self.server_url + 'search/old', params=params, timeout=10).content
                 xml = etree.fromstring(content)
-            except XMLSyntaxError:
+            except etree.ParseError:
                 logger.error("Wrong data returned: %r", content)
                 break
 
@@ -175,7 +191,7 @@ class PodnapisiProvider(_PodnapisiProvider, ProviderSubtitleArchiveMixin):
                 if pid in pids:
                     continue
 
-                language = Language.fromietf(subtitle_xml.find('language').text)
+                _language = Language.fromietf(subtitle_xml.find('language').text)
                 hearing_impaired = 'n' in (subtitle_xml.find('flags').text or '')
                 foreign = 'f' in (subtitle_xml.find('flags').text or '')
                 if only_foreign and not foreign:
@@ -185,7 +201,10 @@ class PodnapisiProvider(_PodnapisiProvider, ProviderSubtitleArchiveMixin):
                     continue
 
                 elif also_foreign and foreign:
-                    language = Language.rebuild(language, forced=True)
+                    _language = Language.rebuild(_language, forced=True)
+
+                if language != _language:
+                    continue
 
                 page_link = subtitle_xml.find('url').text
                 releases = []
@@ -198,12 +217,12 @@ class PodnapisiProvider(_PodnapisiProvider, ProviderSubtitleArchiveMixin):
                 r_year = int(subtitle_xml.find('year').text)
 
                 if is_episode:
-                    subtitle = self.subtitle_class(language, hearing_impaired, page_link, pid, releases, title,
+                    subtitle = self.subtitle_class(_language, hearing_impaired, page_link, pid, releases, title,
                                                    season=r_season, episode=r_episode, year=r_year,
                                                    asked_for_release_group=video.release_group,
                                                    asked_for_episode=episode)
                 else:
-                    subtitle = self.subtitle_class(language, hearing_impaired, page_link, pid, releases, title,
+                    subtitle = self.subtitle_class(_language, hearing_impaired, page_link, pid, releases, title,
                                                    year=r_year, asked_for_release_group=video.release_group)
 
 

Contents/Libraries/Shared/subliminal_patch/providers/subscene.py

@@ -4,6 +4,7 @@ import io
 import logging
 import os
 import time
+import inflect
 
 from random import randint
 from zipfile import ZipFile
@@ -20,6 +21,8 @@ from subliminal_patch.converters.subscene import language_ids, supported_languag
 from subscene_api.subscene import search, Subtitle as APISubtitle
 from subzero.language import Language
 
+p = inflect.engine()
+
 
 language_converters.register('subscene = subliminal_patch.converters.subscene:SubsceneConverter')
 logger = logging.getLogger(__name__)
@@ -192,21 +195,27 @@ class SubsceneProvider(Provider, ProviderSubtitleArchiveMixin):
 
     def query(self, video):
         vfn = get_video_filename(video)
+        subtitles = []
         logger.debug(u"Searching for: %s", vfn)
         film = search(vfn, session=self.session)
-
-        subtitles = []
         if film and film.subtitles:
+            logger.debug('Release results found: %s', len(film.subtitles))
             subtitles = self.parse_results(video, film)
+        else:
+            logger.debug('No release results found')
 
         # re-search for episodes without explicit release name
         if isinstance(video, Episode):
-            term = u"%s S%02iE%02i" % (video.series, video.season, video.episode)
+            #term = u"%s S%02iE%02i" % (video.series, video.season, video.episode)
+            term = u"%s - %s Season" % (video.series, p.number_to_words("%sth" % video.season).capitalize())
             time.sleep(self.search_throttle)
             logger.debug('Searching for alternative results: %s', term)
-            film = search(term, session=self.session)
+            film = search(term, session=self.session, release=False)
             if film and film.subtitles:
+                logger.debug('Alternative results found: %s', len(film.subtitles))
                 subtitles += self.parse_results(video, film)
+            else:
+                logger.debug('No alternative results found')
 
             # packs
             if video.season_fully_aired:
@@ -215,9 +224,17 @@ class SubsceneProvider(Provider, ProviderSubtitleArchiveMixin):
                 time.sleep(self.search_throttle)
                 film = search(term, session=self.session)
                 if film and film.subtitles:
+                    logger.debug('Pack results found: %s', len(film.subtitles))
                     subtitles += self.parse_results(video, film)
+                else:
+                    logger.debug('No pack results found')
             else:
                 logger.debug("Not searching for packs, because the season hasn't fully aired")
+        else:
+            logger.debug('Searching for movie results: %s', video.title)
+            film = search(video.title, year=video.year, session=self.session, limit_to=None, release=False)
+            if film and film.subtitles:
+                subtitles += self.parse_results(video, film)
 
         logger.info("%s subtitles found" % len(subtitles))
         return subtitles

Contents/Libraries/Shared/subliminal_patch/providers/subscenter.py

@@ -2,6 +2,7 @@
 
 from subliminal.providers.subscenter import SubsCenterProvider as _SubsCenterProvider, \
     SubsCenterSubtitle as _SubsCenterSubtitle
+from subzero.language import Language
 
 
 class SubsCenterSubtitle(_SubsCenterSubtitle):
@@ -21,6 +22,7 @@ class SubsCenterSubtitle(_SubsCenterSubtitle):
 
 
 class SubsCenterProvider(_SubsCenterProvider):
+    languages = {Language.fromalpha2(l) for l in ['he']}
     subtitle_class = SubsCenterSubtitle
     hearing_impaired_verifiable = True
     server_url = 'http://www.subscenter.info/he/'

Contents/Libraries/Shared/subliminal_patch/providers/titlovi.py

@@ -11,7 +11,7 @@ from bs4 import BeautifulSoup
 from zipfile import ZipFile, is_zipfile
 from rarfile import RarFile, is_rarfile
 from babelfish import language_converters, Script
-from requests import Session
+from requests import Session, RequestException
 from guessit import guessit
 from subliminal_patch.providers import Provider
 from subliminal_patch.providers.mixins import ProviderSubtitleArchiveMixin
@@ -25,6 +25,9 @@ from subliminal.video import Episode, Movie
 from subliminal.subtitle import fix_line_ending
 from subzero.language import Language
 
+from random import randint
+from .utils import FIRST_THOUSAND_OR_SO_USER_AGENTS as AGENT_LIST
+
 # parsing regex definitions
 title_re = re.compile(r'(?P<title>(?:.+(?= [Aa][Kk][Aa] ))|.+)(?:(?:.+)(?P<altitle>(?<= [Aa][Kk][Aa] ).+))?')
 lang_re = re.compile(r'(?<=flags/)(?P<lang>.{2})(?:.)(?P<script>c?)(?:.+)')
@@ -134,8 +137,8 @@ class TitloviProvider(Provider, ProviderSubtitleArchiveMixin):
 
     def initialize(self):
         self.session = Session()
-        self.session.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ' \
-                                             '(KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36'
+        logger.debug("Using random user agents")
+        self.session.headers['User-Agent'] = AGENT_LIST[randint(0, len(AGENT_LIST) - 1)]
         logger.debug('User-Agent set to %s', self.session.headers['User-Agent'])
         self.session.headers['Referer'] = self.server_url
         logger.debug('Referer set to %s', self.session.headers['Referer'])
@@ -178,7 +181,11 @@ class TitloviProvider(Provider, ProviderSubtitleArchiveMixin):
             try:
                 r = self.session.get(self.search_url, params=params, timeout=10)
                 r.raise_for_status()
+            except RequestException as e:
+                logger.exception('RequestException %s', e)
+                break
 
+            try:
                 soup = BeautifulSoup(r.content, 'lxml')
 
                 # number of results
@@ -202,7 +209,7 @@ class TitloviProvider(Provider, ProviderSubtitleArchiveMixin):
                 current_page = int(params['pg'])
 
             try:
-                sublist = soup.select('section.titlovi > ul.titlovi > li')
+                sublist = soup.select('section.titlovi > ul.titlovi > li.subtitleContainer.canEdit')
                 for sub in sublist:
                     # subtitle id
                     sid = sub.find(attrs={'data-id': True}).attrs['data-id']

Contents/Libraries/Shared/subliminal_patch/providers/tvsubtitles.py

@@ -21,6 +21,10 @@ class TVsubtitlesSubtitle(_TVsubtitlesSubtitle):
 
 
 class TVsubtitlesProvider(_TVsubtitlesProvider):
+    languages = {Language('por', 'BR')} | {Language(l) for l in [
+        'ara', 'bul', 'ces', 'dan', 'deu', 'ell', 'eng', 'fin', 'fra', 'hun', 'ita', 'jpn', 'kor', 'nld', 'pol', 'por',
+        'ron', 'rus', 'spa', 'swe', 'tur', 'ukr', 'zho'
+    ]}
     subtitle_class = TVsubtitlesSubtitle
 
     @region.cache_on_arguments(expiration_time=SHOW_EXPIRATION_TIME)

Contents/Libraries/Shared/subliminal_patch/refiners/drone.py

@@ -63,12 +63,12 @@ class DroneAPIClient(object):
             out[key] = quote(value)
         return out
 
-    def get(self, endpoint, **params):
+    def get(self, endpoint, requests_kwargs=None, **params):
         url = urljoin(self.api_url, endpoint)
         params = self.build_params(params)
 
         # perform the request
-        r = self.session.get(url, params=params)
+        r = self.session.get(url, params=params, **(requests_kwargs or {}))
         r.raise_for_status()
 
         # get the response as json
@@ -79,8 +79,8 @@ class DroneAPIClient(object):
             return j
         return []
 
-    def status(self):
-        return self.get("system/status")
+    def status(self, **kwargs):
+        return self.get("system/status", requests_kwargs=kwargs)
 
     def update_video(self, video, scene_name):
         """

Contents/Libraries/Shared/subliminal_patch/subtitle.py

@@ -44,11 +44,13 @@ class Subtitle(Subtitle_):
 
     pack_data = None
     _guessed_encoding = None
+    _is_valid = False
 
     def __init__(self, language, hearing_impaired=False, page_link=None, encoding=None, mods=None):
         super(Subtitle, self).__init__(language, hearing_impaired=hearing_impaired, page_link=page_link,
                                        encoding=encoding)
         self.mods = mods
+        self._is_valid = False
 
     def __repr__(self):
         return '<%s %r [%s:%s]>' % (
@@ -212,6 +214,9 @@ class Subtitle(Subtitle_):
         :rtype: bool
 
         """
+        if self._is_valid:
+            return True
+
         text = self.text
         if not text:
             return False
@@ -222,6 +227,7 @@ class Subtitle(Subtitle_):
         except Exception:
             logger.error("PySRT-parsing failed, trying pysubs2")
         else:
+            self._is_valid = True
             return True
 
         # something else, try to return srt
@@ -247,6 +253,7 @@ class Subtitle(Subtitle_):
             logger.exception("Couldn't convert subtitle %s to .srt format: %s", self, traceback.format_exc())
             return False
 
+        self._is_valid = True
         return True
 
     @classmethod

Contents/Libraries/Shared/subliminal_patch/utils.py

@@ -35,11 +35,12 @@ def sanitize(string, ignore_characters=None, default_characters={'-', ':', '(',
     return string.strip().lower()
 
 
-def fix_inconsistent_naming(title, inconsistent_titles_dict=None):
+def fix_inconsistent_naming(title, inconsistent_titles_dict=None, no_sanitize=False):
     """Fix titles with inconsistent naming using dictionary and sanitize them.
 
     :param str title: original title.
     :param dict inconsistent_titles_dict: dictionary of titles with inconsistent naming.
+    :param bool no_sanitize: indication to not sanitize title.
     :return: new title.
     :rtype: str
 
@@ -54,5 +55,8 @@ def fix_inconsistent_naming(title, inconsistent_titles_dict=None):
         pattern = re.compile('|'.join(re.escape(key) for key in inconsistent_titles_dict.keys()))
         title = pattern.sub(lambda x: inconsistent_titles_dict[x.group()], title)
 
+    if no_sanitize:
+        return title
+    else:
+        return sanitize(title)
     # return fixed and sanitized title
-    return sanitize(title)

Contents/Libraries/Shared/subliminal_patch/video.py

@@ -12,6 +12,7 @@ class Video(Video_):
     hints = None
     season_fully_aired = None
     audio_languages = None
+    external_subtitle_languages = None
 
     def __init__(self, name, format=None, release_group=None, resolution=None, video_codec=None, audio_codec=None,
                  imdb_id=None, hashes=None, size=None, subtitle_languages=None, audio_languages=None):
@@ -22,3 +23,4 @@ class Video(Video_):
         self.plexapi_metadata = {}
         self.hints = {}
         self.audio_languages = audio_languages or set()
+        self.external_subtitle_languages = set()

Contents/Libraries/Shared/subscene_api/subscene.py

@@ -25,6 +25,7 @@ this script that does the job by parsing the website"s pages.
 
 # imports
 import re
+
 import enum
 import sys
 
@@ -36,7 +37,7 @@ else:
     from contextlib import suppress
     from urllib2.request import Request, urlopen
 
-from bs4 import BeautifulSoup
+from bs4 import BeautifulSoup, NavigableString
 
 # constants
 HEADERS = {
@@ -175,8 +176,12 @@ class Film(object):
 
         content = soup.find("div", "subtitles")
         header = content.find("div", "box clearfix")
+        cover = None
 
-        cover = header.find("div", "poster").img.get("src")
+        try:
+            cover = header.find("div", "poster").img.get("src")
+        except AttributeError:
+            pass
 
         title = header.find("div", "header").h2.text[:-12].strip()
 
@@ -207,7 +212,7 @@ def section_exists(soup, section):
     return False
 
 
-def get_first_film(soup, section, session=None):
+def get_first_film(soup, section, year=None, session=None):
     tag_part = SectionsParts[section]
     tag = None
 
@@ -220,12 +225,26 @@ def get_first_film(soup, section, session=None):
     if not tag:
         return
 
-    url = SITE_DOMAIN + tag.findNext("ul").find("li").div.a.get("href")
+    url = None
+
+    if not year:
+        url = SITE_DOMAIN + tag.findNext("ul").find("li").div.a.get("href")
+    else:
+        for t in tag.findNext("ul").findAll("li"):
+            if isinstance(t, NavigableString) or not t.div:
+                continue
+
+            if str(year) in t.div.a.string:
+                url = SITE_DOMAIN + t.div.a.get("href")
+                break
+        if not url:
+            return
+
     return Film.from_url(url, session=session)
 
 
-def search(term, session=None, limit_to=SearchTypes.Exact):
-    soup = soup_for("%s/subtitles/title?q=%s" % (SITE_DOMAIN, term), session=session)
+def search(term, release=True, session=None, year=None, limit_to=SearchTypes.Exact):
+    soup = soup_for("%s/subtitles/%s?q=%s" % (SITE_DOMAIN, "release" if release else "title", term), session=session)
 
     if "Subtitle search by" in str(soup):
         rows = soup.find("table").tbody.find_all("tr")
@@ -234,7 +253,7 @@ def search(term, session=None, limit_to=SearchTypes.Exact):
 
     for junk, search_type in SearchTypes.__members__.items():
         if section_exists(soup, search_type):
-            return get_first_film(soup, search_type)
+            return get_first_film(soup, search_type, year=year, session=session)
 
         if limit_to == search_type:
             return

Contents/Libraries/Shared/subzero/constants.py

@@ -15,6 +15,7 @@ ICON     = 'icon-default.jpg'
 ICON_SUB = 'icon-sub.jpg'
 
 DEFAULT_TIMEOUT = 15
+START_DELAY = 30.0
 
 
 # media types as on https://github.com/Arcanemagus/plex-api/wiki/MediaTypes

Contents/Libraries/Shared/subzero/language.py

@@ -1,9 +1,9 @@
 # coding=utf-8
+import types
+
 from babelfish.exceptions import LanguageError
-
 from babelfish import Language as Language_, basestr
 
-
 repl_map = {
     "dk": "da",
     "nld": "nl",
@@ -34,7 +34,12 @@ def wrap_forced(f):
         cls = args[0]
         args = args[1:]
         s = args.pop(0)
-        base, forced = s.split(":") if ":" in s else (s, False)
+        forced = None
+        if isinstance(s, types.StringTypes):
+            base, forced = s.split(":") if ":" in s else (s, False)
+        else:
+            base = s
+
         instance = f(cls, base, *args, **kwargs)
         if isinstance(instance, Language):
             instance.forced = forced == "forced"

Contents/Libraries/Shared/subzero/modification/dictionaries/make_data.py

@@ -117,10 +117,6 @@ SZ_FIX_DATA = {
 }
 
 SZ_FIX_DATA_GLOBAL = {
-    "PartialWordsAlways": {
-        u"¶¶": u"♫",
-        u"¶": u"♪"
-    }
 }
 
 if __name__ == "__main__":

Contents/Libraries/Shared/subzero/modification/mods/common.py

@@ -28,16 +28,18 @@ class CommonFixes(SubtitleTextModification):
         NReProcessor(re.compile(r'(?u)(\w|\b|\s|^)(-\s?-{1,2})'), ur"\1—", name="CM_multidash"),
 
         # line = _/-/\s
-        NReProcessor(re.compile(r'(?u)(^\W*[-_.:]+\W*$)'), "", name="CM_non_word_only"),
+        NReProcessor(re.compile(r'(?u)(^\W*[-_.:>~]+\W*$)'), "", name="CM_non_word_only"),
+
+        # remove >>
+        NReProcessor(re.compile(r'(?u)^\s?>>\s*'), "", name="CM_leading_crocodiles"),
 
         # line = : text
         NReProcessor(re.compile(r'(?u)(^\W*:\s*(?=\w+))'), "", name="CM_empty_colon_start"),
 
-        # multi space
-        NReProcessor(re.compile(r'(?u)(\s{2,})'), " ", name="CM_multi_space"),
-
         # fix music symbols
-        NReProcessor(re.compile(ur'(?u)(?:^[-\s]*[*#¶]+(?![^\s\-*#¶]))|(?:[*#¶]+\s*$)'), u"♪", name="CM_music_symbols"),
+        NReProcessor(re.compile(ur'(?u)(^[-\s>~]*[*#¶]+\s*)|(\s*[*#¶]+\s*$)'),
+                     lambda x: u"♪ " if x.group(1) else u" ♪",
+                     name="CM_music_symbols"),
 
         # '' = "
         NReProcessor(re.compile(ur'(?u)([\'’ʼ❜‘‛][\'’ʼ❜‘‛]+)'), u'"', name="CM_double_apostrophe"),
@@ -86,9 +88,6 @@ class CommonFixes(SubtitleTextModification):
 
         # space before ending doublequote?
 
-        # remove >>
-        NReProcessor(re.compile(r'(?u)^\s?>>\s*'), "", name="CM_leading_crocodiles"),
-
         # replace uppercase I with lowercase L in words
         NReProcessor(re.compile(ur'(?u)([a-zà-ž]+)(I+)'),
                      lambda match: ur'%s%s' % (match.group(1), "l" * len(match.group(2))),

Contents/Libraries/Shared/subzero/modification/mods/hearing_impaired.py

@@ -29,6 +29,22 @@ class HearingImpaired(SubtitleTextModification):
         FullBracketEntryProcessor(re.compile(ur'(?sux)^-?%(t)s[([].+(?=[^)\]]{3,}).+[)\]]%(t)s$' % {"t": TAG}),
                                   "", name="HI_brackets_full"),
 
+        # uppercase text before colon (at least 3 uppercase chars); at start or after a sentence,
+        # possibly with a dash in front; ignore anything ending with a quote
+        NReProcessor(re.compile(ur'(?u)(?:(?<=^)|(?<=[.\-!?\"\']))([\s\->~]*(?=[A-ZÀ-Ž&+]\s*[A-ZÀ-Ž&+]\s*[A-ZÀ-Ž&+])'
+                                ur'[A-zÀ-ž-_0-9\s\"\'&+()\[\],:]+:(?![\"\'’ʼ❜‘‛”“‟„])(?:\s+|$))(?![0-9])'), "",
+                     name="HI_before_colon_caps"),
+
+        # any text before colon (at least 3 chars); at start or after a sentence,
+        # possibly with a dash in front; try not breaking actual sentences with a colon at the end by not matching if
+        # a space is inside the text; ignore anything ending with a quote
+        NReProcessor(re.compile(ur'(?u)(?:(?<=^)|(?<=[.\-!?\"]))([\s\->~]*((?=[A-zÀ-ž&+]\s*[A-zÀ-ž&+]\s*[A-zÀ-ž&+])'
+                                ur'[A-zÀ-ž-_0-9\s\"\'&+()\[\]]+:)(?![\"’ʼ❜‘‛”“‟„])\s*)(?![0-9])'),
+                     lambda match:
+                     match.group(1) if (match.group(2).count(" ") > 0 or match.group(1).count("-") > 0)
+                     else "" if not match.group(1).startswith(" ") else " ",
+                     name="HI_before_colon_noncaps"),
+
         # brackets (only remove if at least 3 chars in brackets)
         NReProcessor(re.compile(ur'(?sux)-?%(t)s[([][^([)\]]+?(?=[A-zÀ-ž"\'.]{3,})[^([)\]]+[)\]][\s:]*%(t)s' %
                                 {"t": TAG}), "", name="HI_brackets"),
@@ -46,21 +62,6 @@ class HearingImpaired(SubtitleTextModification):
         #NReProcessor(re.compile(ur'(?u)(\b|^)([\s-]*(?=[A-zÀ-ž-_0-9"\']{3,})[A-zÀ-ž-_0-9"\']+:\s*)'), "",
         #             name="HI_before_colon"),
 
-        # uppercase text before colon (at least 3 uppercase chars); at start or after a sentence,
-        # possibly with a dash in front; ignore anything ending with a quote
-        NReProcessor(re.compile(ur'(?u)(?:(?<=^)|(?<=[.\-!?\"\']))([\s-]*(?=[A-ZÀ-Ž&+]\s*[A-ZÀ-Ž&+]\s*[A-ZÀ-Ž&+])'
-                                ur'[A-ZÀ-Ž-_0-9\s\"\'&+]+:(?![\"\'’ʼ❜‘‛”“‟„])(?:\s+|$))(?![0-9])'), "",
-                     name="HI_before_colon_caps"),
-
-        # any text before colon (at least 3 chars); at start or after a sentence,
-        # possibly with a dash in front; try not breaking actual sentences with a colon at the end by not matching if
-        # a space is inside the text; ignore anything ending with a quote
-        NReProcessor(re.compile(ur'(?u)(?:(?<=^)|(?<=[.\-!?\"]))([\s-]*((?=[A-zÀ-ž&+]\s*[A-zÀ-ž&+]\s*[A-zÀ-ž&+])'
-                                ur'[A-zÀ-ž-_0-9\s\"\'&+]+:)(?![\"’ʼ❜‘‛”“‟„])\s*)(?![0-9])'),
-                     lambda match:
-                     match.group(1) if (match.group(2).count(" ") > 0 or match.group(1).count("-") > 0)
-                     else "" if not match.group(1).startswith(" ") else " ",
-                     name="HI_before_colon_noncaps"),
 
         # text in brackets at start, after optional dash, before colon or at end of line
         # fixme: may be too aggressive

Contents/Libraries/Shared/subzero/video.py

@@ -17,17 +17,23 @@ def has_external_subtitle(part_id, stored_subs, language):
 
 
 def set_existing_languages(video, video_info, external_subtitles=False, embedded_subtitles=False, known_embedded=None,
-                           stored_subs=None, languages=None, only_one=False):
+                           stored_subs=None, languages=None, only_one=False, known_metadata_subs=None):
     logger.debug(u"Determining existing subtitles for %s", video.name)
 
+    external_langs_found = set()
     # scan for external subtitles
-    external_langs_found = set(search_external_subtitles(video.name, languages=languages,
-                                                         only_one=only_one).values())
+    if known_metadata_subs:
+        # existing metadata subtitles
+        external_langs_found = known_metadata_subs
+
+    external_langs_found.update(set(search_external_subtitles(video.name, languages=languages,
+                                                              only_one=only_one).values()))
 
     # found external subtitles should be considered?
     if external_subtitles:
         # |= is update, thanks plex
         video.subtitle_languages.update(external_langs_found)
+        video.external_subtitle_languages.update(external_langs_found)
 
     else:
         # did we already download subtitles for this?

Contents/Libraries/Shared/test.srt

@@ -26,7 +26,7 @@ this <font color="#000000">is a"subtitle" test</font> "with a"text before colons
 Mah numbar is wrong: 1 91 7
 :
 : Peter is funny!
-¶ You're singing in the rain #
+¶You're singing in the rain#
 
 5
 00:00:17,225 --> 00:00:19,684

Contents/Strings/en.json

@@ -403,12 +403,12 @@
     "forced": "forced",
     "%s in %s (%s, score: %s), %s": "%s in %s (%s, score: %s), %s",
     "Extract embedded subtitle streams": "Extract embedded subtitle streams",
-    "Display %(incl_excl_list_name)s (%(count)d)": "Display %(incl_excl_list_name)s list (%(count)d)",
+    "Display %(incl_excl_list_name)s (%(count)d)": "Display %(incl_excl_list_name)s (%(count)d)",
     "include list": "include list",
     "ignore list": "ignore list",
     "Include list": "Include list",
     "Ignore list": "Ignore list",
-    "Show the current %(incl_excl_list_name)s (mainly used for the automatic tasks)": "Show the current %(incl_excl_list_name)s list (mainly used for the automatic tasks)",
+    "Show the current %(incl_excl_list_name)s (mainly used for the automatic tasks)": "Show the current %(incl_excl_list_name)s (mainly used for the automatic tasks)",
     "Didn't change the %(incl_excl_list_name)s": "Didn't change the %(incl_excl_list_name)s",
     "%(title)s added to the include list": "%(title)s added to the include list",
     "%(title)s removed from the include list": "%(title)s removed from the include list",

Contents/advanced_settings.json.template

@@ -44,6 +44,9 @@ Don't expect support if you mess this up.
             "enabled_for": ["series", "movies"],
             "use_https": true,
             "timeout": 15,
+            // skip subtitles with a mismatched FPS value; might lead to more results when disabled
+            // but also to more false-positives
+            "skip_wrong_fps": true,
         },
         "podnapisi": {
             "enabled_for": ["series", "movies"],

Licenses/inflect/LICENSE

@@ -0,0 +1,7 @@
+Copyright Jason R. Coombs
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

README.md

@@ -77,83 +77,29 @@ Ever had broken music icons in a subtitle? Nordic characters like `Å` which tur
 
 ## Installation
 Simply go to the Plex Plugins in your Plex Media Server, search for Sub-Zero and install it.
-For further help or manual installation, [please go to the wiki](https://github.com/pannal/Sub-Zero.bundle/wiki).
+For further help or manual installation, [please go to the wiki](https://github.com/pannal/Sub-Zero.bundle/wiki/Installation).
 
-## Big thanks to the beta testing team (in no particular order)!
-the.vbm, mmgoodnow, Vertig0ne, thliu78, tattoomees, ostman, count_confucius, 
-eherberg, tywilliams_88, Swanny, Jippo, Joost1991 / joost, Marik, Jon, AmbyDK, 
-Clay, mmgoodnow, Abenlog, michael, smikwily, shoghicp, Zuikkis, Isilorn, 
-Jacob K, Ninjouz, chopeta, fvb, Jose
+## Big thanks to the beta/i18n testing team (in no particular order)!
+the.vbm, mmgoodnow, Vertig0ne, thliu78, tattoomees, ostman, count_confucius, eherberg, tywilliams_88, Swanny, Jippo, Joost1991 / joost, Marik, Jon, AmbyDK, Clay, Abenlog, michael, smikwily, shoghicp, Zuikkis, Isilorn, Jacob K, Ninjouz, chopeta, fvb, Uthman, Claus Møller, Semi Doludizgin, Rafael, sugarman402, Morpheus1333, Yamil.llanos, Notorius28 
 
 ## Changelog
 
-2.6.4.2834
-- core: add option to use custom (Google, Cloudflare) DNS to resolve provider hosts in problematic countries; fixes #547
-- core: add support for downloading subtitles only when the audio streams don't match (any?) configured languages; fixes #519
-- core: add support for an include list instead of an ignore list; add the option to disable SZ by default, then enable it per item/series/section (inverse ignore list)
-- core/menu/config: support forced/foreign subtitles independently
-- core: fallback for OSError on scandir, should fix #532
-- core: add config versioning/migration system
-- core: correctly force non-foreign-only-capable providers off; remove subscene from foreign-only capable providers
-- core: scanning: collect information about audio streams
-- core: use correct storage path when storing subtitle info, when only VTT is used
-- core: fix disabled channel mode
-- core/menu: extract embedded: add extracted embedded subtitles to history
-- core: embedded subtitle streams: don't try parsing the language if inexistant
-- core: subtitle: fix log call, fixes #569
-- core: download best subtitles: only use actually languages searched for
-- core: refiners: tvdb: warn instead of error when no matching series was found
-- core: scanning: re-add expected title to guessit for narrowing down the video title
-- core: resolve #583
-- core: archives: explicitly skip forced subtitles if not searched for, when picking from an archive
-- core: activities/auto-refresh: fix hybrid-plus for movies
-- core: don't disable plugin if all providers throttled; fix #585 #574
-- core: skip cleanup for ignored paths
-- core: update requests to 2.20.0 (fixes security issue)
-- core: update certifi to 2018.10.15
-- core: auto extract: don't overwrite local sub even if unknown to SZ
-- config: set autoclean leftover/unused to off by default
-- providers: opensubtitles: respect rate limit (40 hits/10s); should fix long throttling behaviour
-- providers: opensubtitles: handle bad/inexistant responses
-- providers: opensubtitles: log bad response data
-- providers: opensubtitles: treat empty response as ServiceUnavailable for now
-- providers: opensubtitles: log reason for ServiceUnavailable
-- providers: legendastv: match second title and imdb id
-- providers: titlovi: fix language handling (thanks @viking1304)
-- providers: titlovi: proper handling of archives with both cyrlic and latin subtitles (thanks @viking1304) 
-- providers: titlovi: allow direct subtitle downloads as fallback (when a subtitle, not an archive was returned)
-- providers: hosszupuska: implement site change (thanks @morpheus133)
-- providers: supersubtitles: add base properties to subtitle
-- providers: opensubtitles, podnapisi: fix foreign/forced handling
-- providers: subscene: use original/sceneName if possible
-- menu: fix plugin not responding when ignoring an item in certain menus; fixes #535
-- menu: select active subtitle: return to item details afterwards; correctly set current
-- menu: add item thumbnails to history and a couple of submenus
-- menu: history: use series thumbnail instead of episode screenshot
-- menu: add full soft include/exclude menu handling
-- menu: add support for separate forced and not-forced subtitles
-- menu: fix order of embedded subtitle streams in item detail
-- menu: support S00E00 and equivalent
-- submod: add option to fix only-uppercase subtitles and make them readable
-- submod: keep track of actually applied mods
-- submod: correctly merge mods of the same kind (offset)
-- submod: OCR: add dictionaries for bosnian and norwegian bokmal; update dicts for dan, eng, hrv, spa, srp, swe
-- submod: OCR/HI: skip certain processors for all-caps subs
-- submod: HI: only remove caps before colon if the colon is followed by whitespace or EOL; fixes #542
-- submod: HI: remove MAN:
-- submod: common: improve detection and normalization of quotes, apostrophes
-- submod: common: fix double quotes that are meant to be single quotes inside words
-- submod: common: normalize small hyphens to dash
-- submod: common: remove line only consisting of colon; remove empty colon at start of line
-- submod: common: add space after punctuation
-- submod: common: fix lowercase i for english language
-- submod: common: better fix for music symbols
-- submod: reverse_RTL: also reverse ":,'-" chars in CM_RTL_reverse (thanks @doopler)
-- submod: reverse_RTL: enable mod for arabic, farsi and persian besides hebrew
-- i18n: fix not used translation for recently added missing subtitles menu
-- i18n: fix spanish translation, fixes #543
-- i18n: Hungarian translation is incomplete
-
+2.6.4.2911
+- core: improve file cache (windows especially); use fixed-length cache filenames; fixes #600
+- core: don't log "Checking connections ..." when sonarr/radarr not activated
+- core: make logging for scanning/parsing/preparing videos more clear
+- core: extract embedded: continue searching for embbedded subs after undefined language is found (thanks @jippo015)
+- compat: core: don't assume hints["title"] exists
+- compat: providers: podnapisi: loosen lxml requirement
+- providers: addic7ed: fix not using user credentials; fixes #605
+- providers: titlovi: fix provider (thanks @viking1304)
+- providers: subscene: fix provider
+- providers: opensubtitles: improve token logging
+- providers: podnapisi: fix searching for Marvel series
+- submod: HI: correctly remove uppercase at start of a sentence when lead by a crocodile (>>)
+- submod: HI: correctly remove lowercase inside brackets when HI matched as well
+- submod: HI: remove multiple HI_before_colon_caps before one colon
+- submod: common: also match music symbols after a crocodile; move crocodile removal up the chain