submod retry; WIP

CHANGELOG.md

@@ -1,191 +1,3 @@
-2.6.5.3280
-
-temporarily enable OpenSubtitles.com instead of OpenSubtitles.org.
-You need to have an account there and an API consumer configured. Enter your API key in settings.
-
-This is barely tested but should work for basic usage.
-
-THIS PLUGIN IS DEPRECATED, PLEASE USE BAZARR!
-
-Changelog
-- cheaply backport opensubtitlescom from bazarr
-
-
-2.6.5.3268
-subscene, addic7ed
-- either of those providers might impose a reCAPTCHA verification. In order to use those providers, please create an account at an AntiCaptcha service ([anti-captcha.com](http://getcaptchasolution.com/kkvviom7nh) or [deathbycaptcha.com](http://deathbycaptcha.com)), add funds, then supply your credentials/apikey in the configuration
-
-Changelog
-- clarify README
-- core: fix custom folder handling; #761
-- core: providers: screwzira: move to ktuvit and wizdom
-- core: add option to not download subtitles for certain audio languages existing; and/or no audio stream; fix #756
-- core: delay item refreshes after refresh call (default: 5 seconds; exposed in advanced settings)
-- menu: allow extraction of embedded subtitles for whole tv shows
-
-
-2.6.5.3247
-
-subscene, addic7ed
-
-either of those providers might impose a reCAPTCHA verification. In order to use those providers, please create an account at an AntiCaptcha service (anti-captcha.com or deathbycaptcha.com), add funds, then supply your credentials/apikey in the configuration
-Changelog
-core: fix for tv.plex.agents.movie not populating its media types
-core: tasks: findBetterSubtitles: increase minimum score for better subtitles for movies with extracted embedded subs from 82 to 112
-
-
-2.6.5.3241
-
-subscene, addic7ed
-
-either of those providers might impose a reCAPTCHA verification. In order to use those providers, please create an account at an AntiCaptcha service (anti-captcha.com or deathbycaptcha.com), add funds, then supply your credentials/apikey in the configuration
-Changelog
-
-core: add support for new Plex Movie agent
-core: remove py3 compat breaking unnecessary change
-core: skip drawing tags for SRT
-core: advanced_settings: refiners: drone: add custom pem_file support; fixes #735
-providers: core: set DownloadLimitPerDayExceeded timeout to 4 hours (was one day);
-providers: addic7ed: limit downloads per day; add vip setting
-providers: addic7ed: properly compare last_dl, add last_reset tracking info to log #723
-providers: addic7ed: properly implement limits
-submod: HI: remove more music tags
-submod: common CM_punctuation_space2: detect AND don't try changing domain/url/host when fixing punctuation
-
-
-2.6.5.3223
-
-subscene, addic7ed
-
-either of those providers might impose a reCAPTCHA verification. In order to use those providers, please create an account at an AntiCaptcha service (anti-captcha.com or deathbycaptcha.com), add funds, then supply your credentials/apikey in the configuration
-Changelog
-
-core: scoring: reorder subtitles based on second non-hash-score if main hash score is the same; morpheus65535/bazarr#821
-providers: bsplayer: verify hash; clean up
-
-
-2.6.5.3217
-
-subscene, addic7ed
-- either of those providers might impose a reCAPTCHA verification. In order to use those providers, please create an account at an AntiCaptcha service ([anti-captcha.com](http://getcaptchasolution.com/kkvviom7nh) or [deathbycaptcha.com](http://deathbycaptcha.com)), add funds, then supply your credentials/apikey in the configuration
-
-Changelog
-- core: also extract (missing) embedded subtitles when SearchAllRecentlyAddedMissing is running
-- core: core: clarify detecting streams (in logs)
-- core: UnRAR: set binary to executable, even if not checked out from git; might fix #693
-- core: bazarr-backport: morpheus65535/bazarr#703: use proper language code detection instead of a wild guess; should fix bad existing subtitle detection
-- core: bazarr-backport: morpheus65535/bazarr#660: fix BOM encoding stuff
-- core: bazarr-backport: morpheus65535/bazarr#656 further generalize formats; skip release group match if format match failed
-- core: fix stream detection when using mediainfo (#711)
-- config/core: make periodic SZ-internal subtitle maintenance interval configurable
-- providers: add BSPlayer Subtitles
-- providers: add ScrewZira (Hebrew)
-
-2.6.5.3183
-
-
-subscene, addic7ed
-- either of those providers might impose a reCAPTCHA verification. In order to use those providers, please create an account at an AntiCaptcha service ([anti-captcha.com](http://getcaptchasolution.com/kkvviom7nh) or [deathbycaptcha.com](http://deathbycaptcha.com)), add funds, then supply your credentials/apikey in the configuration
-
-Changelog
-- core: don't fall back to default providers if none enabled
-- core: don't process any further if stream info is missing
-- core: support using mediainfo for retrieving MP4 MOV_TEXT subtitle stream titles (PMS bug)
-- core: fix embedded subtitle extraction in some cases (#681, #680)
-- core: scanning: add additional INFO logging for undetected languages
-- core: bazarr-backport: remove existing subtitle file, to support MergerFS
-- core: bazarr-backport: generic 10 minute throttling if uncaught exception occurs
-- providers: addic7ed: fix recaptcha solving; fix show ID retrieval (#681)
-- providers: addic7ed: add timeout on authentication error
-- providers: addic7ed: fix shows with dots in them (Mayans M.C.)
-- providers: addic7ed: fix detection of completed subtitle for non-english users (#686)
-- providers: addic7ed: add more timeouts in the login process
-- providers: argenteam: bazarr-backport: use new url; fixes
-
-
-2.6.5.3152
-
-subscene, addic7ed
-- either of those providers might impose a reCAPTCHA verification. In order to use those providers, please create an account at an AntiCaptcha service ([anti-captcha.com](http://getcaptchasolution.com/kkvviom7nh) or [deathbycaptcha.com](http://deathbycaptcha.com)), add funds, then supply your credentials/apikey in the configuration
-
-Changelog
-- core: fix core issue possibly impacting results on OpenSubtitles in certain conditions
-- core: fix default values of opensubtitles-skip-wrong-fps, use_https; fix #676
-- core: fix for determining whether to search under certain circumstances; fixes #666
-- core: #664 fix missing language processing of multiple videos refreshed at once
-- core: #661 fix match strictness when determining preexisting external subtitles
-- providers: titlovi: New implementation of Titlovi using API (thanks @viking1304)
-
-
-2.6.5.3124
-
-subscene, addic7ed and titlovi
-- either of those providers might impose a reCAPTCHA verification. In order to use those providers, please create an account at an AntiCaptcha service ([anti-captcha.com](http://getcaptchasolution.com/kkvviom7nh) or [deathbycaptcha.com](http://deathbycaptcha.com)), add funds, then supply your credentials/apikey in the configuration
-
-Changelog
-- core: http: fallback to default DNS when normal resolving fails; fixes #657
-- core: extract embedded/menu: fix detection of unknown streams; don't use unknown streams if a known language was previously found
-- core: language: use replacement map from bazarr
-- providers: titlovi: fix matching
-- providers: subscene: fix unknown language code error when "empty" result is returned
-- providers: subscene: add support for pt-BR (based on Diaoul/subliminal@b22cf08)
-- providers: subscene: explicitly set account filters for languages
-- providers: subscene: limit alternative searches to 3; set throttle to 8
-- providers: subscene: move login/cookies to initialization sequence
-- submod: generic: en: fix ";='s
-
-
-2.6.5.3109
-
-subscene, addic7ed and titlovi
-- either of those providers might impose a reCAPTCHA verification. In order to use those providers, please create an account at an AntiCaptcha service ([anti-captcha.com](http://getcaptchasolution.com/kkvviom7nh) or [deathbycaptcha.com](http://deathbycaptcha.com)), add funds, then supply your credentials/apikey in the configuration
-
-Changelog
-- providers: add Napisy24 (polish)
-- providers: subscene: reduce provider load by possibly half
-- providers: subscene: support logging in (username/password are now required)
-- providers: subscene: fallback to non year results if none found with year
-
-
-2.6.5.3099
-
-subscene, addic7ed and titlovi
-- either of those providers might impose a reCAPTCHA verification. In order to use those providers, please create an account at an AntiCaptcha service ([anti-captcha.com](http://getcaptchasolution.com/kkvviom7nh) or [deathbycaptcha.com](http://deathbycaptcha.com)), add funds, then supply your credentials/apikey in the configuration
-
-Changelog
-- core: allow system DNS again by putting "system" as the DNS
-- providers: subscene: fix again (subscene, contact us please, so we can end this)
-
-
-2.6.5.3092
-
-subscene, addic7ed and titlovi
-- either of those providers might impose a reCAPTCHA verification. In order to use those providers, please create an account at an AntiCaptcha service ([anti-captcha.com](http://getcaptchasolution.com/kkvviom7nh) or [deathbycaptcha.com](http://deathbycaptcha.com)), add funds, then supply your credentials/apikey in the configuration
-
-Changelog
-- providers: subscene: fix endpoint (hopefully for longer now)
-- providers: subscene: don't search for season packs (broken for now; relieves 50% of server load on provider)
-- providers: subscene: don't calculate video fn for now
-- providers: argenteam: backport fixes from bazarr
-- subtitle: try decoding with utf-16 by default as well (zho/farsi)
-- submod: HI: remove music tags by default
-- core: compat (bazarr): add env var SZ_KEEP_ENCODING to keep encoding of subtitles
-
-
-2.6.5.3074
-
-Changelog
-- core: cf: bypass cf 95% of the time without captchas
-- core: fix breaking line endings of certain languages (chinese, UTF-16); fixes #646
-- core: update pysubs2 to 0.2.3
-
-
-2.6.5.3062
-
-Changelog
-- core: cf: optimize
-- core: http: don't query DNS with IPs. thanks @fgump (fixes sonarr/radarr)
-
 
 2.6.5.3041
 

Contents/Code/__init__.py

@@ -21,21 +21,20 @@ import support
 import interface
 sys.modules["interface"] = interface
 
-from subzero.constants import OS_PLEX_USERAGENT
+from subzero.constants import OS_PLEX_USERAGENT, PERSONAL_MEDIA_IDENTIFIER
 from interface.menu import *
 from support.plex_media import media_to_videos, get_media_item_ids
-from support.extract import agent_extract_embedded
 from support.scanning import scan_videos
-from support.storage import save_subtitles, store_subtitle_info
+from support.storage import save_subtitles, store_subtitle_info, get_subtitle_storage
 from support.items import is_wanted
 from support.config import config
 from support.lib import get_intent
-from support.helpers import track_usage, get_title_for_video_metadata, get_identifier, cast_bool
+from support.helpers import track_usage, get_title_for_video_metadata, get_identifier, cast_bool, \
+    audio_streams_match_languages
 from support.history import get_history
 from support.data import dispatch_migrate
 from support.activities import activity
 from support.download import download_best_subtitles
-from support.localmedia import find_subtitles
 
 
 def Start():
@@ -97,7 +96,57 @@ def Start():
 
 def update_local_media(videos, ignore_parts_cleanup=None):
     for video in videos:
-        find_subtitles(video["plex_part"], ignore_parts_cleanup=ignore_parts_cleanup)
+        support.localmedia.find_subtitles(video["plex_part"], ignore_parts_cleanup=ignore_parts_cleanup)
+
+
+def agent_extract_embedded(video_part_map):
+    try:
+        subtitle_storage = get_subtitle_storage()
+
+        to_extract = []
+        item_count = 0
+
+        for scanned_video, part_info in video_part_map.iteritems():
+            plexapi_item = scanned_video.plexapi_metadata["item"]
+            stored_subs = subtitle_storage.load_or_new(plexapi_item)
+            valid_langs_in_media = audio_streams_match_languages(scanned_video, config.get_lang_list(ordered=True))
+
+            if not config.lang_list.difference(valid_langs_in_media):
+                Log.Debug("Skipping embedded subtitle extraction for %s, audio streams are in correct language(s)",
+                          plexapi_item.rating_key)
+                continue
+
+            for plexapi_part in get_all_parts(plexapi_item):
+                item_count = item_count + 1
+                used_one_unknown_stream = False
+                for requested_language in config.lang_list:
+                    embedded_subs = stored_subs.get_by_provider(plexapi_part.id, requested_language, "embedded")
+                    current = stored_subs.get_any(plexapi_part.id, requested_language) or \
+                        requested_language in scanned_video.external_subtitle_languages
+
+                    if not embedded_subs:
+                        stream_data = get_embedded_subtitle_streams(plexapi_part, requested_language=requested_language,
+                                                                    skip_unknown=used_one_unknown_stream)
+
+                        if stream_data:
+                            stream = stream_data[0]["stream"]
+                            if stream_data[0]["is_unknown"]:
+                                used_one_unknown_stream = True
+
+                            to_extract.append(({scanned_video: part_info}, plexapi_part, str(stream.index),
+                                               str(requested_language), not current))
+
+                            if not cast_bool(Prefs["subtitles.search_after_autoextract"]):
+                                scanned_video.subtitle_languages.update({requested_language})
+                    else:
+                        Log.Debug("Skipping embedded subtitle extraction for %s, already got %r from %s",
+                                  plexapi_item.rating_key, requested_language, embedded_subs[0].id)
+        if to_extract:
+            Log.Info("Triggering extraction of %d embedded subtitles of %d items", len(to_extract), item_count)
+            Thread.Create(multi_extract_embedded, stream_list=to_extract, refresh=True, with_mods=True,
+                          single_thread=not config.advanced.auto_extract_multithread)
+    except:
+        Log.Error("Something went wrong when auto-extracting subtitles, continuing: %s", traceback.format_exc())
 
 
 class SubZeroAgent(object):
@@ -262,8 +311,7 @@ class SubZeroAgent(object):
 
 
 class SubZeroSubtitlesAgentMovies(SubZeroAgent, Agent.Movies):
-    contributes_to = ['com.plexapp.agents.imdb', 'com.plexapp.agents.xbmcnfo', 'com.plexapp.agents.themoviedb',
-                      'com.plexapp.agents.hama',  'tv.plex.agents.movie']
+    contributes_to = ['com.plexapp.agents.imdb', 'com.plexapp.agents.xbmcnfo', 'com.plexapp.agents.themoviedb', 'com.plexapp.agents.hama']
     score_prefs_key = "subtitles.search.minimumMovieScore2"
     agent_type_verbose = "Movies"
 

Contents/Code/interface/item_details.py

@@ -7,16 +7,14 @@ from subzero.language import Language
 
 from sub_mod import SubtitleModificationsMenu
 from menu_helpers import debounce, SubFolderObjectContainer, default_thumb, add_incl_excl_options, get_item_task_data, \
-    set_refresh_menu_state, route
-from support.extract import extract_embedded_sub
+    set_refresh_menu_state, route, extract_embedded_sub
 
 from refresh_item import RefreshItem
 from subzero.constants import PREFIX
 from support.config import config, TEXT_SUBTITLE_EXTS
-from support.helpers import timestamp, df, get_language, display_language, get_language_from_stream
+from support.helpers import timestamp, df, get_language, display_language, get_language_from_stream, is_stream_forced
 from support.items import get_item_kind_from_rating_key, get_item, get_current_sub, get_item_title, save_stored_sub
-from support.plex_media import get_plex_metadata, get_part, get_embedded_subtitle_streams, is_stream_forced, \
-    update_stream_info
+from support.plex_media import get_plex_metadata, get_part, get_embedded_subtitle_streams
 from support.scanning import scan_videos
 from support.scheduler import scheduler
 from support.storage import get_subtitle_storage
@@ -120,8 +118,6 @@ def ItemDetailsMenu(rating_key, title=None, base_title=None, item_title=None, ra
             if not os.path.exists(part.file):
                 continue
 
-            update_stream_info(part)
-
             part_id = str(part.id)
             part_index += 1
 
@@ -674,28 +670,29 @@ def ListEmbeddedSubsForItemMenu(**kwargs):
             stream = stream_data["stream"]
             is_forced = stream_data["is_forced"]
 
-            oc.add(DirectoryObject(
-                key=Callback(TriggerExtractEmbeddedSubForItemMenu, randomize=timestamp(),
-                             stream_index=str(stream.index), language=language, with_mods=True, **kwargs),
-                title=_(u"Extract stream %(stream_index)s, %(language)s%(unknown_state)s%(forced_state)s"
-                        u"%(stream_title)s with default mods",
-                        stream_index=stream.index,
-                        language=display_language(language),
-                        unknown_state=_(" (unknown)") if is_unknown else "",
-                        forced_state=_(" (forced)") if is_forced else "",
-                        stream_title=" (\"%s\")" % stream.title if stream.title else ""),
-            ))
-            oc.add(DirectoryObject(
-                key=Callback(TriggerExtractEmbeddedSubForItemMenu, randomize=timestamp(),
-                             stream_index=str(stream.index), language=language, **kwargs),
-                title=_(u"Extract stream %(stream_index)s, %(language)s%(unknown_state)s%(forced_state)s"
-                        u"%(stream_title)s",
-                        stream_index=stream.index,
-                        language=display_language(language),
-                        unknown_state=_(" (unknown)") if is_unknown else "",
-                        forced_state=_(" (forced)") if is_forced else "",
-                        stream_title=" (\"%s\")" % stream.title if stream.title else ""),
-            ))
+            if language:
+                oc.add(DirectoryObject(
+                    key=Callback(TriggerExtractEmbeddedSubForItemMenu, randomize=timestamp(),
+                                 stream_index=str(stream.index), language=language, with_mods=True, **kwargs),
+                    title=_(u"Extract stream %(stream_index)s, %(language)s%(unknown_state)s%(forced_state)s"
+                            u"%(stream_title)s with default mods",
+                            stream_index=stream.index,
+                            language=display_language(language),
+                            unknown_state=_(" (unknown)") if is_unknown else "",
+                            forced_state=_(" (forced)") if is_forced else "",
+                            stream_title=" (\"%s\")" % stream.title if stream.title else ""),
+                ))
+                oc.add(DirectoryObject(
+                    key=Callback(TriggerExtractEmbeddedSubForItemMenu, randomize=timestamp(),
+                                 stream_index=str(stream.index), language=language, **kwargs),
+                    title=_(u"Extract stream %(stream_index)s, %(language)s%(unknown_state)s%(forced_state)s"
+                            u"%(stream_title)s",
+                            stream_index=stream.index,
+                            language=display_language(language),
+                            unknown_state=_(" (unknown)") if is_unknown else "",
+                            forced_state=_(" (forced)") if is_forced else "",
+                            stream_title=" (\"%s\")" % stream.title if stream.title else ""),
+                ))
     return oc
 
 

Contents/Code/interface/menu.py

@@ -12,16 +12,19 @@ from requests import HTTPError
 from item_details import ItemDetailsMenu
 from refresh_item import RefreshItem
 from menu_helpers import add_incl_excl_options, dig_tree, set_refresh_menu_state, \
-    default_thumb, debounce, ObjectContainer, SubFolderObjectContainer, route
+    default_thumb, debounce, ObjectContainer, SubFolderObjectContainer, route, \
+    extract_embedded_sub
 from main import fatality, InclExclMenu
 from advanced import DispatchRestart
 from subzero.constants import ART, PREFIX, DEPENDENCY_MODULE_NAMES
-from support.extract import season_extract_embedded
+from support.plex_media import get_all_parts, get_embedded_subtitle_streams
 from support.scheduler import scheduler
 from support.config import config
 from support.helpers import timestamp, df, display_language
 from support.ignore import get_decision_list
-from support.items import get_all_items, get_items_info, get_item_kind_from_rating_key, get_item, get_item_title
+from support.items import get_all_items, get_items_info, get_item_kind_from_rating_key, get_item, MI_KEY, \
+    get_item_title, get_item_thumb
+from support.storage import get_subtitle_storage
 from support.i18n import _
 
 # init GUI
@@ -108,51 +111,28 @@ def MetadataMenu(rating_key, title=None, base_title=None, display_items=False, p
             add_incl_excl_options(oc, current_kind, title=sub_title, rating_key=rating_key, callback_menu=InclExclMenu)
 
         # mass-extract embedded
-        if config.plex_transcoder:
-            if current_kind == "season":
-                for lang in config.lang_list:
-                    oc.add(DirectoryObject(
-                        key=Callback(SeasonExtractEmbedded, rating_key=rating_key, language=lang,
-                                     base_title=show.section.title, display_items=display_items, item_title=item_title,
-                                     title=title,
-                                     previous_item_type=previous_item_type, with_mods=True,
-                                     previous_rating_key=previous_rating_key, randomize=timestamp()),
-                        title=_(u"Extract missing %(language)s embedded subtitles", language=display_language(lang)),
-                        summary=_("Extracts the not yet extracted embedded subtitles of all episodes for the current "
-                                  "season with all configured default modifications")
-                    ))
-                    oc.add(DirectoryObject(
-                        key=Callback(SeasonExtractEmbedded, rating_key=rating_key, language=lang,
-                                     base_title=show.section.title, display_items=display_items, item_title=item_title,
-                                     title=title, force=True,
-                                     previous_item_type=previous_item_type, with_mods=True,
-                                     previous_rating_key=previous_rating_key, randomize=timestamp()),
-                        title=_(u"Extract and activate %(language)s embedded subtitles", language=display_language(lang)),
-                        summary=_("Extracts embedded subtitles of all episodes for the current season "
-                                  "with all configured default modifications")
-                    ))
-            elif current_kind == "series":
-                for lang in config.lang_list:
-                    oc.add(DirectoryObject(
-                        key=Callback(SeasonExtractEmbedded, rating_key=rating_key, language=lang,
-                                     base_title=title, display_items=display_items, item_title=item_title,
-                                     title=title, mode="series",
-                                     previous_item_type=previous_item_type, with_mods=True,
-                                     previous_rating_key=previous_rating_key, randomize=timestamp()),
-                        title=_(u"Extract missing %(language)s embedded subtitles", language=display_language(lang)),
-                        summary=_("Extracts the not yet extracted embedded subtitles of all episodes for the current "
-                                  "series with all configured default modifications")
-                    ))
-                    oc.add(DirectoryObject(
-                        key=Callback(SeasonExtractEmbedded, rating_key=rating_key, language=lang,
-                                     base_title=title, display_items=display_items, item_title=item_title,
-                                     title=title, force=True, mode="series",
-                                     previous_item_type=previous_item_type, with_mods=True,
-                                     previous_rating_key=previous_rating_key, randomize=timestamp()),
-                        title=_(u"Extract and activate %(language)s embedded subtitles", language=display_language(lang)),
-                        summary=_("Extracts embedded subtitles of all episodes for the current series "
-                                  "with all configured default modifications")
-                    ))
+        if current_kind == "season" and config.plex_transcoder:
+            for lang in config.lang_list:
+                oc.add(DirectoryObject(
+                    key=Callback(SeasonExtractEmbedded, rating_key=rating_key, language=lang,
+                                 base_title=show.section.title, display_items=display_items, item_title=item_title,
+                                 title=title,
+                                 previous_item_type=previous_item_type, with_mods=True,
+                                 previous_rating_key=previous_rating_key, randomize=timestamp()),
+                    title=_(u"Extract missing %(language)s embedded subtitles", language=display_language(lang)),
+                    summary=_("Extracts the not yet extracted embedded subtitles of all episodes for the current "
+                              "season with all configured default modifications")
+                ))
+                oc.add(DirectoryObject(
+                    key=Callback(SeasonExtractEmbedded, rating_key=rating_key, language=lang,
+                                 base_title=show.section.title, display_items=display_items, item_title=item_title,
+                                 title=title, force=True,
+                                 previous_item_type=previous_item_type, with_mods=True,
+                                 previous_rating_key=previous_rating_key, randomize=timestamp()),
+                    title=_(u"Extract and activate %(language)s embedded subtitles", language=display_language(lang)),
+                    summary=_("Extracts embedded subtitles of all episodes for the current season "
+                              "with all configured default modifications")
+                ))
 
         # add refresh
         oc.add(DirectoryObject(
@@ -183,10 +163,9 @@ def SeasonExtractEmbedded(**kwargs):
     item_title = kwargs.pop("item_title")
     title = kwargs.pop("title")
     force = kwargs.pop("force", False)
-    mode = kwargs.pop("mode", "season")
 
     Thread.Create(season_extract_embedded, **{"rating_key": rating_key, "requested_language": requested_language,
-                                              "with_mods": with_mods, "force": force, "mode": mode})
+                                              "with_mods": with_mods, "force": force})
 
     kwargs["header"] = _("Success")
     kwargs["message"] = _(u"Extracting of embedded subtitles for %s triggered", title)
@@ -195,6 +174,53 @@ def SeasonExtractEmbedded(**kwargs):
     return MetadataMenu(randomize=timestamp(), title=item_title, **kwargs)
 
 
+def multi_extract_embedded(stream_list, refresh=False, with_mods=False, single_thread=True, extract_mode="a",
+                           history_storage=None):
+    def execute():
+        for video_part_map, plexapi_part, stream_index, language, set_current in stream_list:
+            plexapi_item = video_part_map.keys()[0].plexapi_metadata["item"]
+
+            extract_embedded_sub(rating_key=plexapi_item.rating_key, part_id=plexapi_part.id,
+                                 plex_item=plexapi_item, part=plexapi_part, scanned_videos=video_part_map,
+                                 stream_index=stream_index, set_current=set_current,
+                                 language=language, with_mods=with_mods, refresh=refresh, extract_mode=extract_mode,
+                                 history_storage=history_storage)
+
+    if single_thread:
+        with Thread.Lock(key="extract_embedded"):
+            execute()
+    else:
+        execute()
+
+
+def season_extract_embedded(rating_key, requested_language, with_mods=False, force=False):
+    # get stored subtitle info for item id
+    subtitle_storage = get_subtitle_storage()
+
+    try:
+        for data in get_all_items(key="children", value=rating_key, base="library/metadata"):
+            item = get_item(data[MI_KEY])
+            if item:
+                stored_subs = subtitle_storage.load_or_new(item)
+                for part in get_all_parts(item):
+                    embedded_subs = stored_subs.get_by_provider(part.id, requested_language, "embedded")
+                    current = stored_subs.get_any(part.id, requested_language)
+                    if not embedded_subs or force:
+                        stream_data = get_embedded_subtitle_streams(part, requested_language=requested_language)
+                        if stream_data:
+                            stream = stream_data[0]["stream"]
+
+                            set_current = not current or force
+                            refresh = not current
+
+                            extract_embedded_sub(rating_key=item.rating_key, part_id=part.id,
+                                                 stream_index=str(stream.index), set_current=set_current,
+                                                 refresh=refresh, language=requested_language, with_mods=with_mods,
+                                                 extract_mode="m")
+    finally:
+        subtitle_storage.destroy()
+
+
 @route(PREFIX + '/ignore_list')
 def IgnoreListMenu():
     ref_list = get_decision_list()
@@ -342,7 +368,7 @@ def ValidatePrefs():
             "subtitle_destination_folder", "include", "include_exclude_paths", "include_exclude_sz_files",
             "new_style_cache", "dbm_supported", "lang_list", "providers", "normal_subs", "forced_only", "forced_also",
             "plex_transcoder", "refiner_settings", "unrar", "adv_cfg_path", "use_custom_dns",
-            "has_anticaptcha", "anticaptcha_cls", "mediainfo_bin"]:
+            "has_anticaptcha", "anticaptcha_cls"]:
 
         value = getattr(config, attr)
         if isinstance(value, dict):

Contents/Code/interface/menu_helpers.py

@@ -1,16 +1,29 @@
 # coding=utf-8
+import traceback
 import types
 import datetime
+import subprocess
+import os
+import operator
 
-from func import enable_channel_wrapper, route_wrapper
+from func import enable_channel_wrapper, route_wrapper, register_route_function
+from subzero.lib.io import get_viable_encoding
+from subzero.language import Language
 from support.i18n import is_localized_string, _
-from support.items import get_item_thumb
-from support.helpers import get_video_display_title, pad_title
+from support.items import get_kind, get_item_thumb, get_item, get_item_kind_from_item, refresh_item
+from support.helpers import get_video_display_title, pad_title, display_language, quote_args, is_stream_forced, \
+    get_title_for_video_metadata, mswindows
+from support.history import get_history
 from support.ignore import get_decision_list
 from support.lib import get_intent
 from support.config import config
 from subzero.constants import ICON_SUB, ICON
+from support.plex_media import get_part, get_plex_metadata
 from support.scheduler import scheduler
+from support.scanning import scan_videos
+from support.storage import save_subtitles
+
+from subliminal_patch.subtitle import ModifiedSubtitle
 
 default_thumb = R(ICON_SUB)
 main_icon = ICON if not config.is_development else "icon-dev.jpg"
@@ -143,6 +156,89 @@ def debounce(func):
     return func
 
 
+def extract_embedded_sub(**kwargs):
+    rating_key = kwargs["rating_key"]
+    part_id = kwargs.pop("part_id")
+    stream_index = kwargs.pop("stream_index")
+    with_mods = kwargs.pop("with_mods", False)
+    language = Language.fromietf(kwargs.pop("language"))
+    refresh = kwargs.pop("refresh", True)
+    set_current = kwargs.pop("set_current", True)
+
+    plex_item = kwargs.pop("plex_item", get_item(rating_key))
+    item_type = get_item_kind_from_item(plex_item)
+    part = kwargs.pop("part", get_part(plex_item, part_id))
+    scanned_videos = kwargs.pop("scanned_videos", None)
+    extract_mode = kwargs.pop("extract_mode", "a")
+
+    any_successful = False
+
+    if part:
+        if not scanned_videos:
+            metadata = get_plex_metadata(rating_key, part_id, item_type, plex_item=plex_item)
+            scanned_videos = scan_videos([metadata], ignore_all=True, skip_hashing=True)
+
+        for stream in part.streams:
+            # subtitle stream
+            if str(stream.index) == stream_index:
+                is_forced = is_stream_forced(stream)
+                bn = os.path.basename(part.file)
+
+                set_refresh_menu_state(_(u"Extracting subtitle %(stream_index)s of %(filename)s",
+                                         stream_index=stream_index,
+                                         filename=bn))
+                Log.Info(u"Extracting stream %s (%s) of %s", stream_index, str(language), bn)
+
+                out_codec = stream.codec if stream.codec != "mov_text" else "srt"
+
+                args = [
+                    config.plex_transcoder, "-i", part.file, "-map", "0:%s" % stream_index, "-f", out_codec, "-"
+                ]
+
+                cmdline = quote_args(args)
+                Log.Debug(u"Calling: %s", cmdline)
+                if mswindows:
+                    Log.Debug("MSWindows: Fixing encoding")
+                    cmdline = cmdline.encode("mbcs")
+
+                output = None
+                try:
+                    output = subprocess.check_output(cmdline, stderr=subprocess.PIPE, shell=True)
+                except:
+                    Log.Error("Extraction failed: %s", traceback.format_exc())
+
+                if output:
+                    subtitle = ModifiedSubtitle(language, mods=config.default_mods if with_mods else None)
+                    subtitle.content = output
+                    subtitle.provider_name = "embedded"
+                    subtitle.id = "stream_%s" % stream_index
+                    subtitle.score = 0
+                    subtitle.set_encoding("utf-8")
+
+                    # fixme: speedup video; only video.name is needed
+                    video = scanned_videos.keys()[0]
+                    save_successful = save_subtitles(scanned_videos, {video: [subtitle]}, mode="m",
+                                                     set_current=set_current)
+                    set_refresh_menu_state(None)
+
+                    if save_successful and refresh:
+                        refresh_item(rating_key)
+
+                    # add item to history
+                    item_title = get_title_for_video_metadata(video.plexapi_metadata,
+                                                              add_section_title=False, add_episode_title=True)
+
+                    history = get_history()
+                    history.add(item_title, video.id, section_title=video.plexapi_metadata["section"],
+                                thumb=video.plexapi_metadata["super_thumb"],
+                                subtitle=subtitle, mode=extract_mode)
+                    history.destroy()
+
+                    any_successful = True
+
+    return any_successful
+
+
 class SZObjectContainer(ObjectContainer):
     def __init__(self, *args, **kwargs):
         skip_pin_lock = kwargs.pop("skip_pin_lock", False)

Contents/Code/support/__init__.py

@@ -19,10 +19,6 @@ sys.modules["support.i18n"] = i18n
 
 helpers._ = i18n._
 
-import history
-
-sys.modules["support.history"] = history
-
 import plex_media
 sys.modules["support.plex_media"] = plex_media
 
@@ -53,10 +49,6 @@ import missing_subtitles
 
 sys.modules["support.missing_subtitles"] = missing_subtitles
 
-import extract
-
-sys.modules["support.extract"] = extract
-
 import tasks
 
 sys.modules["support.tasks"] = tasks
@@ -65,6 +57,10 @@ import ignore
 
 sys.modules["support.ignore"] = ignore
 
+import history
+
+sys.modules["support.history"] = history
+
 import data
 
 sys.modules["support.data"] = data

Contents/Code/support/config.py

@@ -8,17 +8,12 @@ import sys
 import rarfile
 import jstyleson
 import datetime
-import stat
-import traceback
-import socket
-import requests
 
 import subliminal
 import subliminal_patch
 import subzero.constants
 import lib
-from subliminal.exceptions import ServiceUnavailable, DownloadLimitExceeded, AuthenticationError, \
-    DownloadLimitPerDayExceeded
+from subliminal.exceptions import ServiceUnavailable, DownloadLimitExceeded, AuthenticationError
 from subliminal_patch.core import is_windows_special_path
 from whichdb import whichdb
 
@@ -27,7 +22,6 @@ from subzero.language import Language
 from subliminal.cli import MutexLock
 from subzero.lib.io import FileIO, get_viable_encoding
 from subzero.lib.dict import Dicked
-from subzero.lib.which import find_executable
 from subzero.util import get_root_path
 from subzero.constants import PLUGIN_NAME, PLUGIN_IDENTIFIER, MOVIE, SHOW, MEDIA_TYPE_TO_STRING
 from subzero.prefs import get_user_prefs, update_user_prefs
@@ -64,36 +58,23 @@ def int_or_default(s, default):
         return default
 
 
-VALID_THROTTLE_EXCEPTIONS = (TooManyRequests, DownloadLimitExceeded, DownloadLimitPerDayExceeded,
-                             ServiceUnavailable, APIThrottled, requests.Timeout, requests.ReadTimeout, socket.timeout)
-
-def_timeout = (datetime.timedelta(minutes=20), "20 minutes")
+VALID_THROTTLE_EXCEPTIONS = (TooManyRequests, DownloadLimitExceeded, ServiceUnavailable, APIThrottled)
 
 PROVIDER_THROTTLE_MAP = {
     "default": {
         TooManyRequests: (datetime.timedelta(hours=1), "1 hour"),
         DownloadLimitExceeded: (datetime.timedelta(hours=3), "3 hours"),
-        DownloadLimitPerDayExceeded: (datetime.timedelta(hours=4), "4 hours"),
         ServiceUnavailable: (datetime.timedelta(minutes=20), "20 minutes"),
         APIThrottled: (datetime.timedelta(minutes=10), "10 minutes"),
-        AuthenticationError: (datetime.timedelta(hours=2), "2 hours"),
-        requests.Timeout: def_timeout,
-        socket.timeout: def_timeout,
-        requests.ReadTimeout: def_timeout,
     },
     "opensubtitles": {
         TooManyRequests: (datetime.timedelta(hours=3), "3 hours"),
         DownloadLimitExceeded: (datetime.timedelta(hours=6), "6 hours"),
         APIThrottled: (datetime.timedelta(seconds=15), "15 seconds"),
     },
-    "opensubtitlescom": {
-        TooManyRequests: (datetime.timedelta(minutes=1), "1 minute"),
-        DownloadLimitExceeded: (datetime.timedelta(hours=24), "24 hours"),
-    },
     "addic7ed": {
         DownloadLimitExceeded: (datetime.timedelta(hours=3), "3 hours"),
         TooManyRequests: (datetime.timedelta(minutes=5), "5 minutes"),
-        AuthenticationError: (datetime.timedelta(hours=24), "24 hours"),
     }
 }
 
@@ -164,8 +145,6 @@ class Config(object):
     exact_filenames = False
     only_one = False
     any_language_is_enough = False
-    ignore_for_audio = False
-    ignore_subs_for_empty_audio = False
     embedded_auto_extract = False
     ietf_as_alpha3 = False
     unrar = None
@@ -174,7 +153,6 @@ class Config(object):
     anticaptcha_token = None
     anticaptcha_cls = None
     has_anticaptcha = False
-    mediainfo_bin = None
 
     store_recently_played_amount = 40
 
@@ -258,12 +236,9 @@ class Config(object):
         self.plex_transcoder = self.get_plex_transcoder()
         self.only_one = cast_bool(Prefs['subtitles.only_one'])
         self.any_language_is_enough = Prefs['subtitles.any_language_is_enough']
-        self.ignore_for_audio = self.ignore_subs_for_audio()
         self.embedded_auto_extract = cast_bool(Prefs["subtitles.embedded.autoextract"])
         self.ietf_as_alpha3 = cast_bool(Prefs["subtitles.language.ietf_normalize"])
         self.use_custom_dns = self.parse_custom_dns()
-        if not self.advanced.dont_use_mediainfo_mp4:
-            self.mediainfo_bin = self.advanced.mediainfo_bin or find_executable("mediainfo")
         self.initialized = True
 
     def migrate_prefs(self):
@@ -348,19 +323,6 @@ class Config(object):
         for exe in try_executables:
             rarfile.UNRAR_TOOL = exe
             rarfile.ORIG_UNRAR_TOOL = exe
-            if os.path.isfile(exe) and not os.access(exe, os.X_OK):
-                st = os.stat(exe)
-                try:
-                    Log.Debug("setting generic executable permissions for %s", exe)
-                    # fixme: too broad?
-                    os.chmod(exe, st.st_mode | stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH)
-                except:
-                    Log.Debug("failed setting generic executable permissions for %s: %s", exe, traceback.format_exc())
-                    try:
-                        Log.Debug("setting executable permissions for %s", exe)
-                        os.chmod(exe, st.st_mode | stat.S_IEXEC)
-                    except:
-                        Log.Debug("failed setting executable permissions for %s: %s", exe, traceback.format_exc())
             try:
                 rarfile.custom_check([rarfile.UNRAR_TOOL], True)
             except:
@@ -675,34 +637,17 @@ class Config(object):
         enabled_for_primary_agents = {"movie": [], "show": []}
         enabled_sections = {}
 
-        legacy_agents = {
-            "com.plexapp.agents.thetvdb": [SHOW],
-            "com.plexapp.agents.thetvdbdvdorder": [SHOW],
-            "com.plexapp.agents.hama": [SHOW, MOVIE],
-            "com.plexapp.agents.themoviedb": [SHOW, MOVIE],
-            "com.plexapp.agents.imdb": [SHOW, MOVIE],
-        }
-
         # find which agents we're enabled for
         for agent in Plex.agents():
-            #if not agent.primary:
-            #    continue
-            if agent.identifier not in legacy_agents:
+            if not agent.primary:
                 continue
 
-            #media_types = [t.media_type for t in list(agent.media_types)]
-            media_types = legacy_agents[agent.identifier] + []
-
-            # the new movie agent doesn't populate its media types, workaround
-            if not media_types and agent.identifier == "tv.plex.agents.movie":
-                media_types = [MOVIE]
-
-            for media_type in media_types:
-                if media_type in (MOVIE, SHOW):
-                    related_agents = Plex.primary_agent(agent.identifier, media_type)
+            for t in list(agent.media_types):
+                if t.media_type in (MOVIE, SHOW):
+                    related_agents = Plex.primary_agent(agent.identifier, t.media_type)
                     for a in related_agents:
                         if a.identifier == PLUGIN_IDENTIFIER and a.enabled:
-                            enabled_for_primary_agents[MEDIA_TYPE_TO_STRING[media_type]].append(agent.identifier)
+                            enabled_for_primary_agents[MEDIA_TYPE_TO_STRING[t.media_type]].append(agent.identifier)
 
         # find the libraries that use them
         for library in self.sections:
@@ -712,22 +657,6 @@ class Config(object):
         Log.Debug(u"I'm enabled for: %s" % [lib.title for key, lib in enabled_sections.iteritems()])
         return enabled_sections
 
-    def lang_str_to_list(self, s, l):
-        if len(s) and s != "None":
-            for lang in s.split(u","):
-                lang = lang.strip()
-                if lang == "NULL":
-                    l.append(lang)
-                    continue
-                try:
-                    real_lang = Language.fromietf(lang)
-                except:
-                    try:
-                        real_lang = Language.fromname(lang)
-                    except:
-                        continue
-                l.append(real_lang)
-
     # Prepare a list of languages we want subs for
     def get_lang_list(self, provider=None, ordered=False):
         # advanced settings
@@ -768,9 +697,17 @@ class Config(object):
             except:
                 pass
 
-        self.lang_str_to_list(lang_custom, l)
-        if "NULL" in l:
-            l.remove("NULL")
+        if len(lang_custom) and lang_custom != "None":
+            for lang in lang_custom.split(u","):
+                lang = lang.strip()
+                try:
+                    real_lang = Language.fromietf(lang)
+                except:
+                    try:
+                        real_lang = Language.fromname(lang)
+                    except:
+                        continue
+                l.append(real_lang)
 
         if self.forced_also:
             if Prefs["subtitles.when_forced"] == "Always":
@@ -801,22 +738,12 @@ class Config(object):
 
     lang_list = property(get_lang_list)
 
-    def ignore_subs_for_audio(self):
-        c = Prefs['subtitles.ignore_for_audio'].strip()
-        l = []
-
-        self.lang_str_to_list(c, l)
-        if "NULL" in l:
-            l.remove("NULL")
-            self.ignore_subs_for_empty_audio = True
-        return l
-
     def get_subtitle_destination_folder(self):
         if not Prefs["subtitles.save.filesystem"]:
             return
 
-        fld_custom = Prefs["subtitles.save.subFolder.Custom"].strip() if \
-            Prefs["subtitles.save.subFolder.Custom"] else None
+        fld_custom = Prefs["subtitles.save.subFolder.Custom"].strip() if cast_bool(
+            Prefs["subtitles.save.subFolder.Custom"]) else None
         return fld_custom or (
             Prefs["subtitles.save.subFolder"] if Prefs["subtitles.save.subFolder"] != "current folder" else None)
 
@@ -831,10 +758,9 @@ class Config(object):
 
     @property
     def providers_by_prefs(self):
-        return {'opensubtitlescom': cast_bool(Prefs['provider.opensubtitles.enabled']),
+        return {'opensubtitles': cast_bool(Prefs['provider.opensubtitles.enabled']),
                      # 'thesubdb': Prefs['provider.thesubdb.enabled'],
                      'podnapisi': cast_bool(Prefs['provider.podnapisi.enabled']),
-                     'napisy24': cast_bool(Prefs['provider.napisy24.enabled']),
                      'titlovi': cast_bool(Prefs['provider.titlovi.enabled']),
                      'addic7ed': cast_bool(Prefs['provider.addic7ed.enabled']) and self.has_anticaptcha,
                      'tvsubtitles': cast_bool(Prefs['provider.tvsubtitles.enabled']),
@@ -847,10 +773,7 @@ class Config(object):
                      'argenteam': cast_bool(Prefs['provider.argenteam.enabled']),
                      'subscenter': False,
                      'assrt': cast_bool(Prefs['provider.assrt.enabled']),
-                     'bsplayer': cast_bool(Prefs['provider.bsplayer.enabled']),
-                     'ktuvit': cast_bool(Prefs['provider.ktuvit.enabled']),
-                     'wizdom': cast_bool(Prefs['provider.wizdom.enabled']),
-                }
+                     }
 
     @property
     def providers_enabled(self):
@@ -878,10 +801,6 @@ class Config(object):
             providers["argenteam"] = False
             providers["assrt"] = False
             providers["subscene"] = False
-            providers["napisy24"] = False
-            providers["bsplayer"] = False
-            providers["ktuvit"] = False
-            providers["wizdom"] = False
             providers_forced_off = dict(providers)
 
         if not self.unrar and providers["legendastv"]:
@@ -922,51 +841,36 @@ class Config(object):
     providers = property(get_providers)
 
     def get_provider_settings(self):
-        os_use_https = self.advanced.providers.opensubtitles.get("use_https", True)
-        os_skip_wrong_fps = self.advanced.providers.opensubtitles.get("skip_wrong_fps", True)
+        os_use_https = self.advanced.providers.opensubtitles.use_https \
+            if self.advanced.providers.opensubtitles.use_https is not None else True
+
+        os_skip_wrong_fps = self.advanced.providers.opensubtitles.skip_wrong_fps \
+            if self.advanced.providers.opensubtitles.skip_wrong_fps is not None else True
 
         provider_settings = {'addic7ed': {'username': Prefs['provider.addic7ed.username'],
                                           'password': Prefs['provider.addic7ed.password'],
-                                          'is_vip': cast_bool(Prefs['provider.addic7ed.is_vip']),
                                           },
-                             'opensubtitlescom': {'username': Prefs['provider.opensubtitles.username'],
+                             'opensubtitles': {'username': Prefs['provider.opensubtitles.username'],
                                                'password': Prefs['provider.opensubtitles.password'],
-                                               #'use_tag_search': self.exact_filenames,
-                                               #'only_foreign': self.forced_only,
-                                               #'also_foreign': self.forced_also,
-                                               #'is_vip': cast_bool(Prefs['provider.opensubtitles.is_vip']),
-                                               #'use_ssl': os_use_https,
-                                               #'timeout': self.advanced.providers.opensubtitles.timeout or 15,
-                                               #'skip_wrong_fps': os_skip_wrong_fps,
-                                               'use_hash': cast_bool(Prefs['provider.opensubtitles.use_hash']),
-                                               'include_ai_translated': True,
-                                               'api_key': Prefs['provider.opensubtitles.api_key'],
+                                               'use_tag_search': self.exact_filenames,
+                                               'only_foreign': self.forced_only,
+                                               'also_foreign': self.forced_also,
+                                               'is_vip': cast_bool(Prefs['provider.opensubtitles.is_vip']),
+                                               'use_ssl': os_use_https,
+                                               'timeout': self.advanced.providers.opensubtitles.timeout or 15,
+                                               'skip_wrong_fps': os_skip_wrong_fps,
                                                },
                              'podnapisi': {
                                  'only_foreign': self.forced_only,
                                  'also_foreign': self.forced_also,
                              },
-                             'titlovi': {
-                                 'username': Prefs['provider.titlovi.username'],
-                                 'password': Prefs['provider.titlovi.password'],
-                             },
-                             'napisy24': {
-                                 'username': Prefs['provider.napisy24.username'],
-                                 'password': Prefs['provider.napisy24.password'],
-                             },
                              'subscene': {
                                  'only_foreign': self.forced_only,
-                                 'username': Prefs['provider.subscene.username'],
-                                 'password': Prefs['provider.subscene.password'],
                              },
                              'legendastv': {'username': Prefs['provider.legendastv.username'],
                                             'password': Prefs['provider.legendastv.password'],
                                             },
-                             'assrt': {'token': Prefs['provider.assrt.token'], },
-                             'ktuvit': {
-                                 'username': Prefs['provider.ktuvit.username'],
-                                 'password': Prefs['provider.ktuvit.password'],
-                             },
+                             'assrt': {'token': Prefs['provider.assrt.token'], }
                              }
 
         return provider_settings
@@ -990,10 +894,10 @@ class Config(object):
         throttle_data = PROVIDER_THROTTLE_MAP.get(name, PROVIDER_THROTTLE_MAP["default"]).get(cls, None) or \
             PROVIDER_THROTTLE_MAP["default"].get(cls, None)
 
-        if throttle_data:
-            throttle_delta, throttle_description = throttle_data
-        else:
-            throttle_delta, throttle_description = datetime.timedelta(minutes=10), "10 minutes"
+        if not throttle_data:
+            return
+
+        throttle_delta, throttle_description = throttle_data
 
         if "provider_throttle" not in Dict:
             Dict["provider_throttle"] = {}
@@ -1180,12 +1084,11 @@ class Config(object):
     def parse_custom_dns(self):
         custom_dns = Prefs['use_custom_dns2'].strip()
         os.environ["dns_resolvers"] = ""
-
-        if custom_dns and custom_dns != "system":
+        if custom_dns:
             ips = filter(lambda x: x, [d.strip() for d in custom_dns.split(",")])
             if ips:
                 os.environ["dns_resolvers"] = json.dumps(ips)
-            return os.environ["dns_resolvers"]
+                return os.environ["dns_resolvers"]
 
     def init_subliminal_patches(self):
         # configure custom subtitle destination folders for scanning pre-existing subs

Contents/Code/support/download.py

@@ -16,7 +16,6 @@ from support.storage import get_pack_data, store_pack_data
 def get_missing_languages(video, part):
     languages_list = config.get_lang_list(ordered=True)
     languages = set(languages_list)
-    ignore_langs = set(config.ignore_for_audio)
     valid_langs_in_media = set()
 
     if Prefs["subtitles.when"] != "Always":
@@ -30,27 +29,18 @@ def get_missing_languages(video, part):
                   video)
         return set()
 
-    if config.ignore_subs_for_empty_audio and not video.audio_languages:
-        Log.Debug("Skipping subtitle search for %s, ignoring as no audio stream exists", video)
-        return set()
-
-    inter = ignore_langs.intersection(video.audio_languages)
-    if ignore_langs and inter:
-        Log.Debug("Skipping subtitle search for %s, ignoring due to existing audio streams: %s", video, inter)
-        return set()
-
     # should we treat IETF as alpha3? (ditch the country part)
     alpha3_map = {}
     if config.ietf_as_alpha3:
         for language in languages:
-            if language and language.country:
+            if language.country:
                 alpha3_map[language.alpha3] = language.country
                 language.country = None
 
     have_languages = video.subtitle_languages.copy()
     if config.ietf_as_alpha3:
         for language in have_languages:
-            if language and language.country:
+            if language.country:
                 alpha3_map[language.alpha3] = language.country
                 language.country = None
 
@@ -63,14 +53,14 @@ def get_missing_languages(video, part):
                 filter(lambda l: not l.forced, video.subtitle_languages)
             if langs:
                 Log.Debug("We have at least one subtitle for any configured language.")
-                return set()
+                return False
 
         elif "External subtitle" in config.any_language_is_enough:
-            langs = video.external_subtitle_languages if not not_in_forced else \
+            langs = video.subtitle_languages if not not_in_forced else \
                 filter(lambda l: not l.forced, video.external_subtitle_languages)
             if langs:
                 Log.Debug("We have at least one external subtitle for any configured language.")
-                return set()
+                return False
 
     # all languages are found if we either really have subs for all languages or we only want to have exactly one language
     # and we've only found one (the case for a selected language, Prefs['subtitles.only_one'] (one found sub matches any language))
@@ -80,7 +70,7 @@ def get_missing_languages(video, part):
             Log.Debug('Only one language was requested, and we\'ve got a subtitle for %s', video)
         else:
             Log.Debug('All languages %r exist for %s', languages, video)
-        return set()
+        return False
 
     # re-add country codes to the missing languages, in case we've removed them above
     if config.ietf_as_alpha3:
@@ -116,22 +106,21 @@ def language_hook(provider):
 def download_best_subtitles(video_part_map, min_score=0, throttle_time=None, providers=None):
     hearing_impaired = Prefs['subtitles.search.hearingImpaired']
     languages = set([Language.rebuild(l) for l in config.lang_list])
+    missing_languages = []
     if not languages:
         return
 
     use_videos = []
-    missing_languages = set()
     for video, part in video_part_map.iteritems():
         if not video.ignore_all:
-            p_missing_languages = get_missing_languages(video, part)
+            missing_languages = get_missing_languages(video, part)
         else:
-            p_missing_languages = languages
+            missing_languages = languages
 
-        if p_missing_languages:
-            Log.Info(u"%s has missing languages: %s", os.path.basename(video.name), p_missing_languages)
+        if missing_languages:
+            Log.Info(u"%s has missing languages: %s", os.path.basename(video.name), missing_languages)
             refine_video(video, refiner_settings=config.refiner_settings)
             use_videos.append(video)
-            missing_languages.update(p_missing_languages)
 
     # prepare blacklist
     blacklist = get_blacklist_from_part_map(video_part_map, languages)

Contents/Code/support/extract.py

@@ -1,214 +0,0 @@
-# coding=utf-8
-import os
-import subprocess
-import traceback
-
-from support.helpers import quote_args, mswindows, get_title_for_video_metadata, cast_bool, \
-    audio_streams_match_languages
-from support.i18n import _
-from support.items import get_item_kind_from_item, refresh_item, get_all_items, get_item, MI_KEY
-from support.storage import get_subtitle_storage, save_subtitles
-from support.config import config
-from support.history import get_history
-from support.plex_media import get_all_parts, get_embedded_subtitle_streams, get_part, get_plex_metadata, \
-    update_stream_info, is_stream_forced
-from support.scanning import scan_videos
-from subzero.language import Language
-from subliminal_patch.subtitle import ModifiedSubtitle
-
-
-def agent_extract_embedded(video_part_map, set_as_existing=False):
-    try:
-        subtitle_storage = get_subtitle_storage()
-
-        to_extract = []
-        item_count = 0
-
-        threads = []
-
-        for scanned_video, part_info in video_part_map.iteritems():
-            plexapi_item = scanned_video.plexapi_metadata["item"]
-            stored_subs = subtitle_storage.load_or_new(plexapi_item)
-            valid_langs_in_media = \
-                audio_streams_match_languages(scanned_video, config.get_lang_list(ordered=True))
-
-            if not config.lang_list.difference(valid_langs_in_media):
-                Log.Debug("Skipping embedded subtitle extraction for %s, audio streams are in correct language(s)",
-                          plexapi_item.rating_key)
-                continue
-
-            for plexapi_part in get_all_parts(plexapi_item):
-                item_count = item_count + 1
-                used_one_unknown_stream = False
-                used_one_known_stream = False
-                for requested_language in config.lang_list:
-                    skip_unknown = used_one_unknown_stream or used_one_known_stream
-                    embedded_subs = stored_subs.get_by_provider(plexapi_part.id, requested_language, "embedded")
-                    current = stored_subs.get_any(plexapi_part.id, requested_language) or \
-                        requested_language in scanned_video.external_subtitle_languages
-
-                    if not embedded_subs:
-                        stream_data = get_embedded_subtitle_streams(plexapi_part, requested_language=requested_language,
-                                                                    skip_unknown=skip_unknown)
-
-                        if stream_data and stream_data[0]["language"]:
-                            stream = stream_data[0]["stream"]
-                            if stream_data[0]["is_unknown"]:
-                                used_one_unknown_stream = True
-                            else:
-                                used_one_known_stream = True
-
-                            to_extract.append(({scanned_video: part_info}, plexapi_part, str(stream.index),
-                                               str(requested_language), not current))
-
-                            if not cast_bool(Prefs["subtitles.search_after_autoextract"]) or set_as_existing:
-                                scanned_video.subtitle_languages.update({requested_language})
-                    else:
-                        Log.Debug("Skipping embedded subtitle extraction for %s, already got %r from %s",
-                                  plexapi_item.rating_key, requested_language, embedded_subs[0].id)
-        if to_extract:
-            Log.Info("Triggering extraction of %d embedded subtitles of %d items", len(to_extract), item_count)
-            threads.append(Thread.Create(multi_extract_embedded, stream_list=to_extract, refresh=True, with_mods=True,
-                                         single_thread=not config.advanced.auto_extract_multithread))
-            return threads
-    except:
-        Log.Error("Something went wrong when auto-extracting subtitles, continuing: %s", traceback.format_exc())
-
-
-def multi_extract_embedded(stream_list, refresh=False, with_mods=False, single_thread=True, extract_mode="a",
-                           history_storage=None):
-    def execute():
-        for video_part_map, plexapi_part, stream_index, language, set_current in stream_list:
-            plexapi_item = video_part_map.keys()[0].plexapi_metadata["item"]
-
-            extract_embedded_sub(rating_key=plexapi_item.rating_key, part_id=plexapi_part.id,
-                                 plex_item=plexapi_item, part=plexapi_part, scanned_videos=video_part_map,
-                                 stream_index=stream_index, set_current=set_current,
-                                 language=language, with_mods=with_mods, refresh=refresh, extract_mode=extract_mode,
-                                 history_storage=history_storage)
-
-    if single_thread:
-        with Thread.Lock(key="extract_embedded"):
-            execute()
-    else:
-        execute()
-
-
-def season_extract_embedded(rating_key, requested_language, with_mods=False, force=False, mode="season"):
-    # get stored subtitle info for item id
-    subtitle_storage = get_subtitle_storage()
-
-    try:
-        rating_keys = [rating_key]
-        if mode == "series":
-            seasons = get_all_items(key="children", value=rating_key, base="library/metadata")
-            rating_keys = [season[MI_KEY] for season in seasons]
-
-        for rating_key in rating_keys:
-            for data in get_all_items(key="children", value=rating_key, base="library/metadata"):
-                item = get_item(data[MI_KEY])
-                if item:
-                    stored_subs = subtitle_storage.load_or_new(item)
-                    for part in get_all_parts(item):
-                        embedded_subs = stored_subs.get_by_provider(part.id, requested_language, "embedded")
-                        current = stored_subs.get_any(part.id, requested_language)
-                        if not embedded_subs or force:
-                            stream_data = get_embedded_subtitle_streams(part, requested_language=requested_language)
-                            if stream_data:
-                                stream = stream_data[0]["stream"]
-
-                                set_current = not current or force
-                                refresh = not current
-
-                                extract_embedded_sub(rating_key=item.rating_key, part_id=part.id,
-                                                     stream_index=str(stream.index), set_current=set_current,
-                                                     refresh=refresh, language=requested_language, with_mods=with_mods,
-                                                     extract_mode="m")
-    finally:
-        subtitle_storage.destroy()
-
-
-def extract_embedded_sub(**kwargs):
-    rating_key = kwargs["rating_key"]
-    part_id = kwargs.pop("part_id")
-    stream_index = kwargs.pop("stream_index")
-    with_mods = kwargs.pop("with_mods", False)
-    language = Language.fromietf(kwargs.pop("language"))
-    refresh = kwargs.pop("refresh", True)
-    set_current = kwargs.pop("set_current", True)
-
-    plex_item = kwargs.pop("plex_item", get_item(rating_key))
-    item_type = get_item_kind_from_item(plex_item)
-    part = kwargs.pop("part", get_part(plex_item, part_id))
-    scanned_videos = kwargs.pop("scanned_videos", None)
-    extract_mode = kwargs.pop("extract_mode", "a")
-
-    any_successful = False
-
-    from interface.menu_helpers import set_refresh_menu_state
-
-    if part:
-        if not scanned_videos:
-            metadata = get_plex_metadata(rating_key, part_id, item_type, plex_item=plex_item)
-            scanned_videos = scan_videos([metadata], ignore_all=True, skip_hashing=True)
-
-        update_stream_info(part)
-        for stream in part.streams:
-            # subtitle stream
-            if str(stream.index) == stream_index:
-                is_forced = is_stream_forced(stream)
-                bn = os.path.basename(part.file)
-
-                set_refresh_menu_state(_(u"Extracting subtitle %(stream_index)s of %(filename)s",
-                                         stream_index=stream_index,
-                                         filename=bn))
-                Log.Info(u"Extracting stream %s (%s) of %s", stream_index, str(language), bn)
-
-                out_codec = stream.codec if stream.codec != "mov_text" else "srt"
-
-                args = [
-                    config.plex_transcoder, "-i", part.file, "-map", "0:%s" % stream_index, "-f", out_codec, "-"
-                ]
-
-                cmdline = quote_args(args)
-                Log.Debug(u"Calling: %s", cmdline)
-                if mswindows:
-                    Log.Debug("MSWindows: Fixing encoding")
-                    cmdline = cmdline.encode("mbcs")
-
-                output = None
-                try:
-                    output = subprocess.check_output(cmdline, stderr=subprocess.PIPE, shell=True)
-                except:
-                    Log.Error("Extraction failed: %s", traceback.format_exc())
-
-                if output:
-                    subtitle = ModifiedSubtitle(language, mods=config.default_mods if with_mods else None)
-                    subtitle.content = output
-                    subtitle.provider_name = "embedded"
-                    subtitle.id = "stream_%s" % stream_index
-                    subtitle.score = 0
-                    subtitle.set_encoding("utf-8")
-
-                    # fixme: speedup video; only video.name is needed
-                    video = scanned_videos.keys()[0]
-                    save_successful = save_subtitles(scanned_videos, {video: [subtitle]}, mode="m",
-                                                     set_current=set_current)
-                    set_refresh_menu_state(None)
-
-                    if save_successful and refresh:
-                        refresh_item(rating_key)
-
-                    # add item to history
-                    item_title = get_title_for_video_metadata(video.plexapi_metadata,
-                                                              add_section_title=False, add_episode_title=True)
-
-                    history = get_history()
-                    history.add(item_title, video.id, section_title=video.plexapi_metadata["section"],
-                                thumb=video.plexapi_metadata["super_thumb"],
-                                subtitle=subtitle, mode=extract_mode)
-                    history.destroy()
-
-                    any_successful = True
-
-    return any_successful

Contents/Code/support/helpers.py

@@ -394,7 +394,7 @@ def get_language_from_stream(lang_code):
             return Language.fromietf(lang)
         elif lang:
             try:
-                return language_from_stream(lang_code)
+                return language_from_stream(lang)
             except LanguageError:
                 pass
 
@@ -437,10 +437,17 @@ def get_language(lang_short):
 
 
 def display_language(l):
-    if not l:
-        return "Unknown"
     return _(str(l.basename).lower()) + ((u" (%s)" % _("forced")) if l.forced else "")
 
 
+def is_stream_forced(stream):
+    stream_title = getattr(stream, "title", "") or ""
+    forced = getattr(stream, "forced", False)
+    if not forced and stream_title and "forced" in stream_title.strip().lower():
+        forced = True
+
+    return forced
+
+
 class PartUnknownException(Exception):
     pass

Contents/Code/support/items.py

@@ -349,13 +349,11 @@ def refresh_item(rating_key, force=False, timeout=8000, refresh_kind=None, paren
         refresh = [item.rating_key for item in list(Plex["library/metadata"].children(int(rating_key)))]
 
     multiple = len(refresh) > 1
-    wait = config.advanced.get("refresh_after_called", 5)
-    Thread.Sleep(wait)
     for key in refresh:
         Log.Info("%s item %s", "Refreshing" if not force else "Forced-refreshing", key)
         Plex["library/metadata"].refresh(key)
         if multiple:
-            Thread.Sleep(wait)
+            Thread.Sleep(10.0)
 
 
 def get_current_sub(rating_key, part_id, language, plex_item=None):

Contents/Code/support/localmedia.py

@@ -7,7 +7,6 @@ import helpers
 import subtitlehelpers
 
 from config import config as sz_config
-from subzero.language import ENDSWITH_LANGUAGECODE_RE
 
 
 SECONDARY_TAGS = ['forced', 'normal', 'default', 'embedded', 'embedded-forced', 'custom', 'hi', 'cc', 'sdh']
@@ -126,7 +125,7 @@ def find_subtitles(part, ignore_parts_cleanup=None):
                             root = split_tag[0]
 
                         # get associated media file name without language
-                        sub_fn = ENDSWITH_LANGUAGECODE_RE.sub("", root)
+                        sub_fn = subtitlehelpers.ENDSWITH_LANGUAGECODE_RE.sub("", root)
 
                         # subtitle basename and basename without possible language tag  not found in collected
                         # media files? kill.

Contents/Code/support/missing_subtitles.py

@@ -7,8 +7,7 @@ import os
 from babelfish import LanguageReverseError
 
 from support.config import config, TEXT_SUBTITLE_EXTS
-from support.helpers import get_plex_item_display_title, cast_bool, get_language_from_stream
-from support.plex_media import is_stream_forced, update_stream_info
+from support.helpers import get_plex_item_display_title, cast_bool, get_language_from_stream, is_stream_forced
 from support.items import get_item
 from support.lib import Plex
 from support.storage import get_subtitle_storage
@@ -36,7 +35,7 @@ def item_discover_missing_subs(rating_key, kind="show", added_at=None, section_t
     for media in item.media:
         existing_subs = {"internal": [], "external": [], "own_external": [], "count": 0}
         for part in media.parts:
-            update_stream_info(part)
+
             # did we already download an external subtitle before?
             if subtitle_target_dir and stored_subs:
                 for language in languages_set:

Contents/Code/support/plex_media.py

@@ -1,7 +1,6 @@
 # coding=utf-8
 
 import os
-import subprocess
 
 import helpers
 from items import get_item
@@ -27,9 +26,6 @@ tvdb_guid_identifier = "com.plexapp.agents.thetvdb://"
 
 
 def get_plexapi_stream_info(plex_item, part_id=None):
-    if not plex_item:
-        return
-
     d = {"stream": {}}
     data = d["stream"]
 
@@ -104,9 +100,6 @@ def media_to_videos(media, kind="series"):
                 plex_episode = get_item(ep.id)
                 stream_info = get_plexapi_stream_info(plex_episode)
 
-                if not stream_info:
-                    continue
-
                 for item in media.seasons[season].episodes[episode].items:
                     for part in item.parts:
                         videos.append(
@@ -128,24 +121,22 @@ def media_to_videos(media, kind="series"):
                         )
     else:
         stream_info = get_plexapi_stream_info(plex_item)
-
-        if stream_info:
-            imdb_id = None
-            if imdb_guid_identifier in media.guid:
-                imdb_id = media.guid[len(imdb_guid_identifier):].split("?")[0]
-            for item in media.items:
-                for part in item.parts:
-                    videos.append(
-                        get_metadata_dict(plex_item, part, dict(stream_info, **{"plex_part": part, "type": "movie",
-                                                                                 "title": media.title, "id": media.id,
-                                                                                 "super_thumb": plex_item.thumb,
-                                                                                 "series_id": None, "year": year,
-                                                                                 "season_id": None, "imdb_id": imdb_id,
-                                                                                 "original_title": original_title,
-                                                                                 "series_tvdb_id": None, "tvdb_id": None,
-                                                                                 "section": plex_item.section.title})
-                                          )
-                    )
+        imdb_id = None
+        if imdb_guid_identifier in media.guid:
+            imdb_id = media.guid[len(imdb_guid_identifier):].split("?")[0]
+        for item in media.items:
+            for part in item.parts:
+                videos.append(
+                    get_metadata_dict(plex_item, part, dict(stream_info, **{"plex_part": part, "type": "movie",
+                                                                             "title": media.title, "id": media.id,
+                                                                             "super_thumb": plex_item.thumb,
+                                                                             "series_id": None, "year": year,
+                                                                             "season_id": None, "imdb_id": imdb_id,
+                                                                             "original_title": original_title,
+                                                                             "series_tvdb_id": None, "tvdb_id": None,
+                                                                             "section": plex_item.section.title})
+                                      )
+                )
     return videos
 
 
@@ -183,99 +174,42 @@ def get_all_parts(plex_item):
     return parts
 
 
-def update_stream_info(part):
-    try:
-        return update_stream_info_(part)
-    except:
-        Log.Exception("Getting Mediainfo failed for: %s", part.file)
-
-
-def update_stream_info_(part):
-    if config.mediainfo_bin and part.container == "mp4":
-        cmdline = '%s --Inform="Text;-%%ID%%_%%Title%%" %s' % (config.mediainfo_bin, helpers.quote(part.file))
-        result = subprocess.check_output(cmdline, stderr=subprocess.PIPE, shell=True)
-        if result:
-            try:
-                stream_titles = {}
-                for pair in result[1:].split("-"):
-                    sid, title = pair.split("_")
-                    stream_titles[int(sid.strip())] = title.strip()
-            except:
-                pass
-            else:
-                filled = []
-                for stream in part.streams:
-                    if stream.index is None:
-                        Log.Debug("Found stream with no index: %r", stream)
-
-                    index = stream.index+1 if stream.index is not None else 1
-                    if index in stream_titles:
-                        stream.title = stream_titles[index]
-                        filled.append(index-1)
-                if filled:
-                    Log.Debug("Filled missing MP4 stream title info for streams: %s", filled)
-
-
-def is_stream_forced(stream):
-    stream_title = getattr(stream, "title", "") or ""
-    forced = getattr(stream, "forced", False)
-    if not forced and stream_title and "forced" in stream_title.strip().lower():
-        forced = True
-
-    return forced
-
-
 def get_embedded_subtitle_streams(part, requested_language=None, skip_duplicate_unknown=True, skip_unknown=False):
     streams = []
     streams_unknown = []
-    all_streams = []
     has_unknown = False
     found_requested_language = False
-    update_stream_info(part)
     for stream in part.streams:
         # subtitle stream
         if stream.stream_type == 3 and not stream.stream_key and stream.codec in TEXT_SUBTITLE_EXTS:
-            is_forced = is_stream_forced(stream)
+            is_forced = helpers.is_stream_forced(stream)
             language = helpers.get_language_from_stream(stream.language_code)
             if language:
                 language = Language.rebuild(language, forced=is_forced)
 
             is_unknown = False
             found_requested_language = requested_language and requested_language == language
-            stream_data = None
 
-            if not language:
+            if not language and config.treat_und_as_first:
                 # only consider first unknown subtitle stream
-                if config.treat_und_as_first:
-                    if has_unknown and skip_duplicate_unknown:
-                        Log.Debug("skipping duplicate unknown")
-                        continue
+                if has_unknown and skip_duplicate_unknown:
+                    continue
 
-                    language = Language.rebuild(list(config.lang_list)[0], forced=is_forced)
-                else:
-                    language = None
+                language = Language.rebuild(list(config.lang_list)[0], forced=is_forced)
                 is_unknown = True
                 has_unknown = True
-                stream_data = {"stream": stream, "is_unknown": is_unknown, "language": language,
-                               "is_forced": is_forced}
-                streams_unknown.append(stream_data)
+                streams_unknown.append({"stream": stream, "is_unknown": is_unknown, "language": language,
+                                        "is_forced": is_forced})
 
             if not requested_language or found_requested_language:
-                stream_data = {"stream": stream, "is_unknown": is_unknown, "language": language,
-                               "is_forced": is_forced}
-                streams.append(stream_data)
+                streams.append({"stream": stream, "is_unknown": is_unknown, "language": language,
+                                "is_forced": is_forced})
 
                 if found_requested_language:
                     break
 
-            if stream_data:
-                all_streams.append(stream_data)
-
-    if requested_language:
-        if streams_unknown and not found_requested_language and not skip_unknown:
-            streams = streams_unknown
-    else:
-        streams = all_streams
+    if streams_unknown and not found_requested_language and not skip_unknown:
+        streams = streams_unknown
 
     return streams
 
@@ -311,9 +245,6 @@ def get_plex_metadata(rating_key, part_id, item_type, plex_item=None):
 
     stream_info = get_plexapi_stream_info(plex_item, part_id)
 
-    if not stream_info:
-        return
-
     # get normalized metadata
     # fixme: duplicated logic of media_to_videos
     if item_type == "episode":
@@ -435,4 +366,3 @@ class PMSMediaProxy(object):
 
             m = m.children[0]
         return parts
-

Contents/Code/support/scanning.py

@@ -4,7 +4,7 @@ import helpers
 from babelfish.exceptions import LanguageError
 
 from support.lib import Plex, get_intent
-from support.plex_media import get_stream_fps, is_stream_forced, update_stream_info
+from support.plex_media import get_stream_fps
 from support.storage import get_subtitle_storage
 from support.config import config, TEXT_SUBTITLE_EXTS
 from support.subtitlehelpers import get_subtitles_from_metadata
@@ -29,7 +29,7 @@ def prepare_video(pms_video_info, ignore_all=False, hints=None, rating_key=None,
     if ignore_all:
         Log.Debug("Force refresh intended.")
 
-    Log.Debug("Detecting streams: %s, account_for_external_subtitles=%s, account_for_embedded_subtitles=%s" % (
+    Log.Debug("Detecting streams: %s, external_subtitles=%s, embedded_subtitles=%s" % (
         plex_part.file, external_subtitles, embedded_subtitles))
 
     known_embedded = []
@@ -46,25 +46,23 @@ def prepare_video(pms_video_info, ignore_all=False, hints=None, rating_key=None,
     # fixme: skip the whole scanning process if known_embedded == wanted languages?
     audio_languages = []
     if plexpy_part:
-        update_stream_info(plexpy_part)
         for stream in plexpy_part.streams:
             if stream.stream_type == 2:
                 lang = None
                 try:
                     lang = language_from_stream(stream.language_code)
                 except LanguageError:
-                    Log.Info("Couldn't detect embedded audio stream language: %s", stream.language_code)
+                    Log.Debug("Couldn't detect embedded audio stream language: %s", stream.language_code)
 
                 # treat unknown language as lang1?
                 if not lang and config.treat_und_as_first:
                     lang = Language.rebuild(list(config.lang_list)[0])
-                    Log.Info("Assuming language %s for audio stream: %s", lang, getattr(stream, "index", None))
 
                 audio_languages.append(lang)
 
             # subtitle stream
             elif stream.stream_type == 3 and embedded_subtitles:
-                is_forced = is_stream_forced(stream)
+                is_forced = helpers.is_stream_forced(stream)
 
                 if ((config.forced_only or config.forced_also) and is_forced) or not is_forced:
                     # embedded subtitle
@@ -75,13 +73,11 @@ def prepare_video(pms_video_info, ignore_all=False, hints=None, rating_key=None,
                             try:
                                 lang = language_from_stream(stream.language_code)
                             except LanguageError:
-                                Log.Info("Couldn't detect embedded subtitle stream language: %s", stream.language_code)
+                                Log.Debug("Couldn't detect embedded subtitle stream language: %s", stream.language_code)
 
                             # treat unknown language as lang1?
                             if not lang and config.treat_und_as_first:
                                 lang = Language.rebuild(list(config.lang_list)[0])
-                                Log.Info("Assuming language %s for subtitle stream: %s", lang,
-                                         getattr(stream, "index", None))
 
                             if lang:
                                 if is_forced:
@@ -131,8 +127,7 @@ def prepare_video(pms_video_info, ignore_all=False, hints=None, rating_key=None,
             set_existing_languages(video, pms_video_info, external_subtitles=external_subtitles,
                                    embedded_subtitles=embedded_subtitles, known_embedded=known_embedded,
                                    stored_subs=stored_subs, languages=config.lang_list,
-                                   only_one=config.only_one, known_metadata_subs=known_metadata_subs,
-                                   match_strictness=config.ext_match_strictness)
+                                   only_one=config.only_one, known_metadata_subs=known_metadata_subs)
 
         # add video fps info
         video.fps = plex_part.fps

Contents/Code/support/subtitlehelpers.py

@@ -5,7 +5,6 @@ import helpers
 
 from config import config, SUBTITLE_EXTS, TEXT_SUBTITLE_EXTS
 from bs4 import UnicodeDammit
-from subzero.language import match_ietf_language
 
 
 class SubtitleHelper(object):
@@ -86,6 +85,19 @@ class VobSubSubtitleHelper(SubtitleHelper):
 #####################################################################################################################
 
 
+IETF_MATCH = ".+\.([^-.]+)(?:-[A-Za-z]+)?$"
+ENDSWITH_LANGUAGECODE_RE = re.compile("\.([^-.]{2,3})(?:-[A-Za-z]{2,})?$")
+
+
+def match_ietf_language(s):
+    language_match = re.match(".+\.([^\.]+)$" if not helpers.cast_bool(Prefs["subtitles.language.ietf_display"])
+                              else IETF_MATCH, s)
+    if language_match and len(language_match.groups()) == 1:
+        language = language_match.groups()[0]
+        return language
+    return s
+
+
 class DefaultSubtitleHelper(SubtitleHelper):
     @classmethod
     def is_helper_for(cls, filename):
@@ -121,7 +133,7 @@ class DefaultSubtitleHelper(SubtitleHelper):
         # Attempt to extract the language from the filename (e.g. Avatar (2009).eng)
         # IETF support thanks to
         # https://github.com/hpsbranco/LocalMedia.bundle/commit/4fad9aefedece78a1fa96401304351347f644369
-        lang_part = match_ietf_language(file, ietf=helpers.cast_bool(Prefs["subtitles.language.ietf_display"]))
+        lang_part = match_ietf_language(file)
         if lang_part != file:
             language = Locale.Language.Match(lang_part)
         elif config.only_one:

Contents/Code/support/tasks.py

@@ -19,7 +19,6 @@ from support.config import config
 from support.items import get_recent_items, get_item, is_wanted, get_item_title
 from support.helpers import track_usage, get_title_for_video_metadata, cast_bool, PartUnknownException
 from support.plex_media import get_plex_metadata
-from support.extract import agent_extract_embedded
 from support.scanning import scan_videos
 from support.i18n import _
 from download import download_best_subtitles, pre_download_hook, post_download_hook, language_hook
@@ -128,8 +127,8 @@ class SubtitleListingMixin(object):
         config.init_subliminal_patches()
 
         provider_settings = config.provider_settings
-        #if not skip_wrong_fps:
-        #    provider_settings["opensubtitlescom"]["skip_wrong_fps"] = False
+        if not skip_wrong_fps:
+            provider_settings["opensubtitles"]["skip_wrong_fps"] = False
 
         if item_type == "episode":
             min_score = 240
@@ -171,15 +170,12 @@ class SubtitleListingMixin(object):
                     else:
                         s.wrong_season_ep = True
 
-            orig_matches = matches.copy()
-            score, score_without_hash = compute_score(matches, s, video, hearing_impaired=use_hearing_impaired)
-
             unsorted_subtitles.append(
-                (s, score, score_without_hash, matches, orig_matches))
-        scored_subtitles = sorted(unsorted_subtitles, key=operator.itemgetter(1, 2), reverse=True)
+                (s, compute_score(matches, s, video, hearing_impaired=use_hearing_impaired), matches))
+        scored_subtitles = sorted(unsorted_subtitles, key=operator.itemgetter(1), reverse=True)
 
         subtitles = []
-        for subtitle, score, score_without_hash, matches, orig_matches in scored_subtitles:
+        for subtitle, score, matches in scored_subtitles:
             # check score
             if score < min_score and not subtitle.wrong_series:
                 Log.Info(u'%s: Score %d is below min_score (%d)', self.name, score, min_score)
@@ -453,17 +449,6 @@ class SearchAllRecentlyAddedMissing(Task):
                     Log.Debug(u"%s: Looking for missing subtitles: %s", self.name, get_item_title(plex_item))
                     scanned_parts = scan_videos([metadata], providers=providers)
 
-                    # auto extract embedded
-                    if config.embedded_auto_extract:
-                        if config.plex_transcoder:
-                            ts = agent_extract_embedded(scanned_parts, set_as_existing=True)
-                            if ts:
-                                Log.Debug("Waiting for %i extraction threads to finish" % len(ts))
-                                for t in ts:
-                                    t.join()
-                        else:
-                            Log.Warn("Plex Transcoder not found, can't auto extract")
-
                     downloaded_subtitles = download_best_subtitles(scanned_parts, min_score=min_score,
                                                                    providers=providers)
                     hit_providers = downloaded_subtitles is not None
@@ -670,7 +655,7 @@ class FindBetterSubtitles(DownloadSubtitleMixin, SubtitleListingMixin, Task):
         min_score_series = int(Prefs["subtitles.search.minimumTVScore2"].strip())
         min_score_movies = int(Prefs["subtitles.search.minimumMovieScore2"].strip())
         min_score_extracted_series = config.advanced.find_better_as_extracted_tv_score or 352
-        min_score_extracted_movies = config.advanced.find_better_as_extracted_movie_score or 112
+        min_score_extracted_movies = config.advanced.find_better_as_extracted_movie_score or 82
         overwrite_manually_modified = cast_bool(
             Prefs["scheduler.tasks.FindBetterSubtitles.overwrite_manually_modified"])
         overwrite_manually_selected = cast_bool(

Contents/DefaultPrefs.json

@@ -192,12 +192,6 @@
     ],
     "default": "Always"
   },
-  {
-    "id": "subtitles.ignore_for_audio",
-    "label": "Don't download subtitles for Audio languages (use ISO-639-1 codes; comma-separated; NULL=no audio)",
-    "type": "text",
-    "default": "None"
-  },
   {
     "id": "subtitles.when_forced",
     "label": "Download foreign/forced subtitles",
@@ -294,7 +288,7 @@
   },
   {
     "id": "anticaptcha.service",
-    "label": "AntiCaptcha-Service (needs paid account; enables Addic7ed)",
+    "label": "AntiCaptcha-Service (needs paid account; enables Addic7ed, titlovi)",
     "type": "enum",
     "values": [
       "none",
@@ -311,7 +305,7 @@
   },
   {
     "id": "provider.opensubtitles.enabled",
-    "label": "Provider: Enable OpenSubtitles.com",
+    "label": "Provider: Enable OpenSubtitles",
     "type": "bool",
     "default": "true"
   },
@@ -330,16 +324,10 @@
     "secure": "true"
   },
   {
-    "id": "provider.opensubtitles.use_hash",
-    "label": "OpenSubtitles hash?",
+    "id": "provider.opensubtitles.is_vip",
+    "label": "OpenSubtitles VIP? (ad-free subs, 1000 subs/day, no-cache VIP server: http://v.ht/osvip)",
     "type": "bool",
-    "default": "true"
-  },
-  {
-    "id": "provider.opensubtitles.api_key",
-    "label": "OpenSubtitles APIKey",
-    "type": "text",
-    "default": ""
+    "default": "false"
   },
   {
     "id": "provider.podnapisi.enabled",
@@ -347,26 +335,6 @@
     "type": "bool",
     "default": "true"
   },
-  {
-    "id": "provider.napisy24.enabled",
-    "label": "Provider: Enable Napisy24 (pl)",
-    "type": "bool",
-    "default": "false"
-  },
-  {
-    "id": "provider.napisy24.username",
-    "label": "Napisy24 Username",
-    "type": "text",
-    "default": ""
-  },
-  {
-    "id": "provider.napisy24.password",
-    "label": "Napisy24 Password",
-    "type": "text",
-    "option": "hidden",
-    "default": "",
-    "secure": "true"
-  },
   {
     "id": "provider.addic7ed.enabled",
     "label": "Provider: Enable Addic7ed (needs AntiCaptcha)",
@@ -387,12 +355,6 @@
     "default": "",
     "secure": "true"
   },
-  {
-    "id": "provider.addic7ed.is_vip",
-    "label": "Addic7ed VIP? (80 vs 40 downloads per day)",
-    "type": "bool",
-    "default": "false"
-  },
   {
     "id": "provider.addic7ed.boost_by2",
     "label": "Addic7ed: boost score (if requirements met)",
@@ -427,24 +389,10 @@
   },
   {
     "id": "provider.titlovi.enabled",
-    "label": "Provider: Enable Titlovi.com (User and Password required)",
+    "label": "Provider: Enable Titlovi.com (might need AntiCaptcha)",
     "type": "bool",
     "default": "true"
   },
-  {
-    "id": "provider.titlovi.username",
-    "label": "Titlovi Username",
-    "type": "text",
-    "default": ""
-  },
-  {
-    "id": "provider.titlovi.password",
-    "label": "Titlovi Password",
-    "type": "text",
-    "option": "hidden",
-    "default": "",
-    "secure": "true"
-  },
   {
     "id": "provider.legendastv.enabled",
     "label": "Provider: Enable Legendas TV (mostly pt-BR; UNRAR NEEDED)",
@@ -483,20 +431,6 @@
     "type": "bool",
     "default": "true"
   },
-  {
-    "id": "provider.subscene.username",
-    "label": "SubScene Username",
-    "type": "text",
-    "default": ""
-  },
-  {
-    "id": "provider.subscene.password",
-    "label": "SubScene Password",
-    "type": "text",
-    "option": "hidden",
-    "default": "",
-    "secure": "true"
-  },
   {
     "id": "provider.supersubtitles.enabled",
     "label": "Provider: Enable feliratok.info (Hungarian)",
@@ -527,38 +461,6 @@
     "type": "text",
     "default": ""
   },
-  {
-    "id": "provider.bsplayer.enabled",
-    "label": "Provider: Enable BSPlayer Subtitles",
-    "type": "bool",
-    "default": "true"
-  },
-  {
-    "id": "provider.wizdom.enabled",
-    "label": "Provider: Enable WizdomSubs (Hebrew)",
-    "type": "bool",
-    "default": "true"
-  },
- {
-    "id": "provider.ktuvit.enabled",
-    "label": "Provider: Enable Ktuvit (Hebrew)",
-    "type": "bool",
-    "default": "true"
-  },
-  {
-    "id": "provider.ktuvit.username",
-    "label": "Ktuvit Username",
-    "type": "text",
-    "default": ""
-  },
-  {
-    "id": "provider.ktuvit.password",
-    "label": "Ktuvit Password",
-    "type": "text",
-    "option": "hidden",
-    "secure": "true",
-    "default": ""
-  },
   {
     "id": "providers.multithreading",
     "label": "Search enabled providers simultaneously (multithreading)",
@@ -844,29 +746,6 @@
     "type": "bool",
     "default": "false"
   },
-  {
-    "id": "scheduler.tasks.SubtitleStorageMaintenance.frequency",
-    "label": "Scheduler: Periodically run subtitle storage maintenance (SZ internal)",
-    "type": "enum",
-    "values": [
-      "never",
-      "every 6 hours",
-      "every 12 hours",
-      "every 24 hours",
-      "every 1 days",
-      "every 2 days",
-      "every 3 days",
-      "every 4 days",
-      "every 1 weeks",
-      "every 2 weeks",
-      "every 3 weeks",
-      "every 4 weeks",
-      "every 5 weeks",
-      "every 6 weeks",
-      "every 12 weeks"
-    ],
-    "default": "every 1 weeks"
-  },
   {
     "id": "history_size",
     "label": "History: amount of items to store historical data for",
@@ -982,7 +861,7 @@
   },
   {
     "id": "use_custom_dns2",
-    "label": "Use custom DNS (IPs, comma-separated, set to 'system' for system DNS. Default: Google/CF)",
+    "label": "Use custom DNS (IPs, comma-separated, leave empty for system DNS. Default: Google/CF)",
     "type": "text",
     "default": "1.1.1.1, 8.8.8.8"
   },

Contents/Info.plist

@@ -13,7 +13,7 @@
         <key>CFBundleSignature</key>
         <string>????</string>
         <key>CFBundleVersion</key>
-        <string>2.6.5.3280</string>
+        <string>2.6.5.3074</string>
         <key>PlexFrameworkVersion</key>
         <string>2</string>
         <key>PlexPluginClass</key>
@@ -23,7 +23,7 @@
         <key>PlexPluginConsoleLogging</key>
         <string>0</string>
         <key>PlexPluginDevMode</key>
-        <string>0</string>
+        <string>1</string>
          <key>PlexPluginCodePolicy</key>
             <!-- this allows channels to access some python methods which are otherwise blocked, as well as import external code libraries, and interact with the PMS HTTP API -->
             <string>Elevated</string>
@@ -32,7 +32,7 @@
 
 &lt;h1&gt;Sub-Zero for Plex&lt;/h1&gt;&lt;i&gt;Subtitles done right&lt;/i&gt;
 
-Version 2.6.5.3280
+Version 2.6.5.3074 DEV
 
 Originally based on @bramwalet's awesome &lt;a href=&quot;https://github.com/bramwalet/Subliminal.bundle&quot;&gt;Subliminal.bundle&lt;/a&gt;
 

Contents/Libraries/Shared/backports/__init__.py

@@ -1 +0,0 @@
-__path__ = __import__('pkgutil').extend_path(__path__, __name__)

Contents/Libraries/Shared/backports/functools_lru_cache.py

@@ -1,196 +0,0 @@
-from __future__ import absolute_import
-
-import functools
-from collections import namedtuple
-from threading import RLock
-
-_CacheInfo = namedtuple("CacheInfo", ["hits", "misses", "maxsize", "currsize"])
-
-
-@functools.wraps(functools.update_wrapper)
-def update_wrapper(
-    wrapper,
-    wrapped,
-    assigned=functools.WRAPPER_ASSIGNMENTS,
-    updated=functools.WRAPPER_UPDATES,
-):
-    """
-    Patch two bugs in functools.update_wrapper.
-    """
-    # workaround for http://bugs.python.org/issue3445
-    assigned = tuple(attr for attr in assigned if hasattr(wrapped, attr))
-    wrapper = functools.update_wrapper(wrapper, wrapped, assigned, updated)
-    # workaround for https://bugs.python.org/issue17482
-    wrapper.__wrapped__ = wrapped
-    return wrapper
-
-
-class _HashedSeq(list):
-    __slots__ = 'hashvalue'
-
-    def __init__(self, tup, hash=hash):
-        self[:] = tup
-        self.hashvalue = hash(tup)
-
-    def __hash__(self):
-        return self.hashvalue
-
-
-def _make_key(
-    args,
-    kwds,
-    typed,
-    kwd_mark=(object(),),
-    fasttypes=set([int, str, frozenset, type(None)]),
-    sorted=sorted,
-    tuple=tuple,
-    type=type,
-    len=len,
-):
-    'Make a cache key from optionally typed positional and keyword arguments'
-    key = args
-    if kwds:
-        sorted_items = sorted(kwds.items())
-        key += kwd_mark
-        for item in sorted_items:
-            key += item
-    if typed:
-        key += tuple(type(v) for v in args)
-        if kwds:
-            key += tuple(type(v) for k, v in sorted_items)
-    elif len(key) == 1 and type(key[0]) in fasttypes:
-        return key[0]
-    return _HashedSeq(key)
-
-
-def lru_cache(maxsize=100, typed=False):
-    """Least-recently-used cache decorator.
-
-    If *maxsize* is set to None, the LRU features are disabled and the cache
-    can grow without bound.
-
-    If *typed* is True, arguments of different types will be cached separately.
-    For example, f(3.0) and f(3) will be treated as distinct calls with
-    distinct results.
-
-    Arguments to the cached function must be hashable.
-
-    View the cache statistics named tuple (hits, misses, maxsize, currsize) with
-    f.cache_info().  Clear the cache and statistics with f.cache_clear().
-    Access the underlying function with f.__wrapped__.
-
-    See:  http://en.wikipedia.org/wiki/Cache_algorithms#Least_Recently_Used
-
-    """
-
-    # Users should only access the lru_cache through its public API:
-    #       cache_info, cache_clear, and f.__wrapped__
-    # The internals of the lru_cache are encapsulated for thread safety and
-    # to allow the implementation to change (including a possible C version).
-
-    def decorating_function(user_function):
-
-        cache = dict()
-        stats = [0, 0]  # make statistics updateable non-locally
-        HITS, MISSES = 0, 1  # names for the stats fields
-        make_key = _make_key
-        cache_get = cache.get  # bound method to lookup key or return None
-        _len = len  # localize the global len() function
-        lock = RLock()  # because linkedlist updates aren't threadsafe
-        root = []  # root of the circular doubly linked list
-        root[:] = [root, root, None, None]  # initialize by pointing to self
-        nonlocal_root = [root]  # make updateable non-locally
-        PREV, NEXT, KEY, RESULT = 0, 1, 2, 3  # names for the link fields
-
-        if maxsize == 0:
-
-            def wrapper(*args, **kwds):
-                # no caching, just do a statistics update after a successful call
-                result = user_function(*args, **kwds)
-                stats[MISSES] += 1
-                return result
-
-        elif maxsize is None:
-
-            def wrapper(*args, **kwds):
-                # simple caching without ordering or size limit
-                key = make_key(args, kwds, typed)
-                result = cache_get(
-                    key, root
-                )  # root used here as a unique not-found sentinel
-                if result is not root:
-                    stats[HITS] += 1
-                    return result
-                result = user_function(*args, **kwds)
-                cache[key] = result
-                stats[MISSES] += 1
-                return result
-
-        else:
-
-            def wrapper(*args, **kwds):
-                # size limited caching that tracks accesses by recency
-                key = make_key(args, kwds, typed) if kwds or typed else args
-                with lock:
-                    link = cache_get(key)
-                    if link is not None:
-                        # record recent use of the key by moving it
-                        # to the front of the list
-                        root, = nonlocal_root
-                        link_prev, link_next, key, result = link
-                        link_prev[NEXT] = link_next
-                        link_next[PREV] = link_prev
-                        last = root[PREV]
-                        last[NEXT] = root[PREV] = link
-                        link[PREV] = last
-                        link[NEXT] = root
-                        stats[HITS] += 1
-                        return result
-                result = user_function(*args, **kwds)
-                with lock:
-                    root, = nonlocal_root
-                    if key in cache:
-                        # getting here means that this same key was added to the
-                        # cache while the lock was released.  since the link
-                        # update is already done, we need only return the
-                        # computed result and update the count of misses.
-                        pass
-                    elif _len(cache) >= maxsize:
-                        # use the old root to store the new key and result
-                        oldroot = root
-                        oldroot[KEY] = key
-                        oldroot[RESULT] = result
-                        # empty the oldest link and make it the new root
-                        root = nonlocal_root[0] = oldroot[NEXT]
-                        oldkey = root[KEY]
-                        root[KEY] = root[RESULT] = None
-                        # now update the cache dictionary for the new links
-                        del cache[oldkey]
-                        cache[key] = oldroot
-                    else:
-                        # put result in a new link at the front of the list
-                        last = root[PREV]
-                        link = [last, root, key, result]
-                        last[NEXT] = root[PREV] = cache[key] = link
-                    stats[MISSES] += 1
-                return result
-
-        def cache_info():
-            """Report cache statistics"""
-            with lock:
-                return _CacheInfo(stats[HITS], stats[MISSES], maxsize, len(cache))
-
-        def cache_clear():
-            """Clear the cache and cache statistics"""
-            with lock:
-                cache.clear()
-                root = nonlocal_root[0]
-                root[:] = [root, root, None, None]
-                stats[:] = [0, 0]
-
-        wrapper.__wrapped__ = user_function
-        wrapper.cache_info = cache_info
-        wrapper.cache_clear = cache_clear
-        return update_wrapper(wrapper, user_function)
-
-    return decorating_function

Contents/Libraries/Shared/certifi/__init__.py

@@ -1,3 +1,3 @@
-from .core import contents, where
+from .core import where
 
-__version__ = "2020.06.20"
+__version__ = "2019.03.09"

Contents/Libraries/Shared/certifi/__main__.py

@@ -1,12 +1,2 @@
-import argparse
-
-from certifi import contents, where
-
-parser = argparse.ArgumentParser()
-parser.add_argument("-c", "--contents", action="store_true")
-args = parser.parse_args()
-
-if args.contents:
-    print(contents())
-else:
-    print(where())
+from certifi import where
+print(where())

Contents/Libraries/Shared/certifi/cacert.pem

@@ -58,6 +58,38 @@ AfvDbbnvRG15RjF+Cv6pgsH/76tuIMRQyV+dTZsXjAzlAcmgQWpzU/qlULRuJQ/7
 TBj0/VLZjmmx6BEP3ojY+x1J96relc8geMJgEtslQIxq/H5COEBkEveegeGTLg==
 -----END CERTIFICATE-----
 
+# Issuer: CN=VeriSign Class 3 Public Primary Certification Authority - G3 O=VeriSign, Inc. OU=VeriSign Trust Network/(c) 1999 VeriSign, Inc. - For authorized use only
+# Subject: CN=VeriSign Class 3 Public Primary Certification Authority - G3 O=VeriSign, Inc. OU=VeriSign Trust Network/(c) 1999 VeriSign, Inc. - For authorized use only
+# Label: "Verisign Class 3 Public Primary Certification Authority - G3"
+# Serial: 206684696279472310254277870180966723415
+# MD5 Fingerprint: cd:68:b6:a7:c7:c4:ce:75:e0:1d:4f:57:44:61:92:09
+# SHA1 Fingerprint: 13:2d:0d:45:53:4b:69:97:cd:b2:d5:c3:39:e2:55:76:60:9b:5c:c6
+# SHA256 Fingerprint: eb:04:cf:5e:b1:f3:9a:fa:76:2f:2b:b1:20:f2:96:cb:a5:20:c1:b9:7d:b1:58:95:65:b8:1c:b9:a1:7b:72:44
+-----BEGIN CERTIFICATE-----
+MIIEGjCCAwICEQCbfgZJoz5iudXukEhxKe9XMA0GCSqGSIb3DQEBBQUAMIHKMQsw
+CQYDVQQGEwJVUzEXMBUGA1UEChMOVmVyaVNpZ24sIEluYy4xHzAdBgNVBAsTFlZl
+cmlTaWduIFRydXN0IE5ldHdvcmsxOjA4BgNVBAsTMShjKSAxOTk5IFZlcmlTaWdu
+LCBJbmMuIC0gRm9yIGF1dGhvcml6ZWQgdXNlIG9ubHkxRTBDBgNVBAMTPFZlcmlT
+aWduIENsYXNzIDMgUHVibGljIFByaW1hcnkgQ2VydGlmaWNhdGlvbiBBdXRob3Jp
+dHkgLSBHMzAeFw05OTEwMDEwMDAwMDBaFw0zNjA3MTYyMzU5NTlaMIHKMQswCQYD
+VQQGEwJVUzEXMBUGA1UEChMOVmVyaVNpZ24sIEluYy4xHzAdBgNVBAsTFlZlcmlT
+aWduIFRydXN0IE5ldHdvcmsxOjA4BgNVBAsTMShjKSAxOTk5IFZlcmlTaWduLCBJ
+bmMuIC0gRm9yIGF1dGhvcml6ZWQgdXNlIG9ubHkxRTBDBgNVBAMTPFZlcmlTaWdu
+IENsYXNzIDMgUHVibGljIFByaW1hcnkgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkg
+LSBHMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMu6nFL8eB8aHm8b
+N3O9+MlrlBIwT/A2R/XQkQr1F8ilYcEWQE37imGQ5XYgwREGfassbqb1EUGO+i2t
+KmFZpGcmTNDovFJbcCAEWNF6yaRpvIMXZK0Fi7zQWM6NjPXr8EJJC52XJ2cybuGu
+kxUccLwgTS8Y3pKI6GyFVxEa6X7jJhFUokWWVYPKMIno3Nij7SqAP395ZVc+FSBm
+CC+Vk7+qRy+oRpfwEuL+wgorUeZ25rdGt+INpsyow0xZVYnm6FNcHOqd8GIWC6fJ
+Xwzw3sJ2zq/3avL6QaaiMxTJ5Xpj055iN9WFZZ4O5lMkdBteHRJTW8cs54NJOxWu
+imi5V5cCAwEAATANBgkqhkiG9w0BAQUFAAOCAQEAERSWwauSCPc/L8my/uRan2Te
+2yFPhpk0djZX3dAVL8WtfxUfN2JzPtTnX84XA9s1+ivbrmAJXx5fj267Cz3qWhMe
+DGBvtcC1IyIuBwvLqXTLR7sdwdela8wv0kL9Sd2nic9TutoAWii/gt/4uhMdUIaC
+/Y4wjylGsB49Ndo4YhYYSq3mtlFs3q9i6wHQHiT+eo8SGhJouPtmmRQURVyu565p
+F4ErWjfJXir0xuKhXFSbplQAz/DxwceYMBo7Nhbbo27q/a2ywtrvAkcTisDxszGt
+TxzhT5yvDwyd93gN2PQ1VoDat20Xj50egWTh/sVFuq1ruQp6Tk9LhO5L8X3dEQ==
+-----END CERTIFICATE-----
+
 # Issuer: CN=Entrust.net Certification Authority (2048) O=Entrust.net OU=www.entrust.net/CPS_2048 incorp. by ref. (limits liab.)/(c) 1999 Entrust.net Limited
 # Subject: CN=Entrust.net Certification Authority (2048) O=Entrust.net OU=www.entrust.net/CPS_2048 incorp. by ref. (limits liab.)/(c) 1999 Entrust.net Limited
 # Label: "Entrust.net Premium 2048 Secure Server CA"
@@ -120,6 +152,39 @@ ksLi4xaNmjICq44Y3ekQEe5+NauQrz4wlHrQMz2nZQ/1/I6eYs9HRCwBXbsdtTLS
 R9I4LtD+gdwyah617jzV/OeBHRnDJELqYzmp
 -----END CERTIFICATE-----
 
+# Issuer: CN=AddTrust External CA Root O=AddTrust AB OU=AddTrust External TTP Network
+# Subject: CN=AddTrust External CA Root O=AddTrust AB OU=AddTrust External TTP Network
+# Label: "AddTrust External Root"
+# Serial: 1
+# MD5 Fingerprint: 1d:35:54:04:85:78:b0:3f:42:42:4d:bf:20:73:0a:3f
+# SHA1 Fingerprint: 02:fa:f3:e2:91:43:54:68:60:78:57:69:4d:f5:e4:5b:68:85:18:68
+# SHA256 Fingerprint: 68:7f:a4:51:38:22:78:ff:f0:c8:b1:1f:8d:43:d5:76:67:1c:6e:b2:bc:ea:b4:13:fb:83:d9:65:d0:6d:2f:f2
+-----BEGIN CERTIFICATE-----
+MIIENjCCAx6gAwIBAgIBATANBgkqhkiG9w0BAQUFADBvMQswCQYDVQQGEwJTRTEU
+MBIGA1UEChMLQWRkVHJ1c3QgQUIxJjAkBgNVBAsTHUFkZFRydXN0IEV4dGVybmFs
+IFRUUCBOZXR3b3JrMSIwIAYDVQQDExlBZGRUcnVzdCBFeHRlcm5hbCBDQSBSb290
+MB4XDTAwMDUzMDEwNDgzOFoXDTIwMDUzMDEwNDgzOFowbzELMAkGA1UEBhMCU0Ux
+FDASBgNVBAoTC0FkZFRydXN0IEFCMSYwJAYDVQQLEx1BZGRUcnVzdCBFeHRlcm5h
+bCBUVFAgTmV0d29yazEiMCAGA1UEAxMZQWRkVHJ1c3QgRXh0ZXJuYWwgQ0EgUm9v
+dDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALf3GjPm8gAELTngTlvt
+H7xsD821+iO2zt6bETOXpClMfZOfvUq8k+0DGuOPz+VtUFrWlymUWoCwSXrbLpX9
+uMq/NzgtHj6RQa1wVsfwTz/oMp50ysiQVOnGXw94nZpAPA6sYapeFI+eh6FqUNzX
+mk6vBbOmcZSccbNQYArHE504B4YCqOmoaSYYkKtMsE8jqzpPhNjfzp/haW+710LX
+a0Tkx63ubUFfclpxCDezeWWkWaCUN/cALw3CknLa0Dhy2xSoRcRdKn23tNbE7qzN
+E0S3ySvdQwAl+mG5aWpYIxG3pzOPVnVZ9c0p10a3CitlttNCbxWyuHv77+ldU9U0
+WicCAwEAAaOB3DCB2TAdBgNVHQ4EFgQUrb2YejS0Jvf6xCZU7wO94CTLVBowCwYD
+VR0PBAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wgZkGA1UdIwSBkTCBjoAUrb2YejS0
+Jvf6xCZU7wO94CTLVBqhc6RxMG8xCzAJBgNVBAYTAlNFMRQwEgYDVQQKEwtBZGRU
+cnVzdCBBQjEmMCQGA1UECxMdQWRkVHJ1c3QgRXh0ZXJuYWwgVFRQIE5ldHdvcmsx
+IjAgBgNVBAMTGUFkZFRydXN0IEV4dGVybmFsIENBIFJvb3SCAQEwDQYJKoZIhvcN
+AQEFBQADggEBALCb4IUlwtYj4g+WBpKdQZic2YR5gdkeWxQHIzZlj7DYd7usQWxH
+YINRsPkyPef89iYTx4AWpb9a/IfPeHmJIZriTAcKhjW88t5RxNKWt9x+Tu5w/Rw5
+6wwCURQtjr0W4MHfRnXnJK3s9EK0hZNwEGe6nQY1ShjTK3rMUUKhemPR5ruhxSvC
+Nr4TDea9Y355e6cJDUCrat2PisP29owaQgVR1EX1n6diIWgVIEM8med8vSTYqZEX
+c4g/VhsxOBi0cQ+azcgOno4uG+GMmIPLHzHxREzGBHNJdmAPx/i9F4BrLunMTA5a
+mnkPIAou1Z5jJh5VkpTYghdae9C8x49OhgQ=
+-----END CERTIFICATE-----
+
 # Issuer: CN=Entrust Root Certification Authority O=Entrust, Inc. OU=www.entrust.net/CPS is incorporated by reference/(c) 2006 Entrust, Inc.
 # Subject: CN=Entrust Root Certification Authority O=Entrust, Inc. OU=www.entrust.net/CPS is incorporated by reference/(c) 2006 Entrust, Inc.
 # Label: "Entrust Root Certification Authority"
@@ -706,6 +771,36 @@ vEsXCS+0yx5DaMkHJ8HSXPfqIbloEpw8nL+e/IBcm2PN7EeqJSdnoDfzAIJ9VNep
 +OkuE6N36B9K
 -----END CERTIFICATE-----
 
+# Issuer: CN=Class 2 Primary CA O=Certplus
+# Subject: CN=Class 2 Primary CA O=Certplus
+# Label: "Certplus Class 2 Primary CA"
+# Serial: 177770208045934040241468760488327595043
+# MD5 Fingerprint: 88:2c:8c:52:b8:a2:3c:f3:f7:bb:03:ea:ae:ac:42:0b
+# SHA1 Fingerprint: 74:20:74:41:72:9c:dd:92:ec:79:31:d8:23:10:8d:c2:81:92:e2:bb
+# SHA256 Fingerprint: 0f:99:3c:8a:ef:97:ba:af:56:87:14:0e:d5:9a:d1:82:1b:b4:af:ac:f0:aa:9a:58:b5:d5:7a:33:8a:3a:fb:cb
+-----BEGIN CERTIFICATE-----
+MIIDkjCCAnqgAwIBAgIRAIW9S/PY2uNp9pTXX8OlRCMwDQYJKoZIhvcNAQEFBQAw
+PTELMAkGA1UEBhMCRlIxETAPBgNVBAoTCENlcnRwbHVzMRswGQYDVQQDExJDbGFz
+cyAyIFByaW1hcnkgQ0EwHhcNOTkwNzA3MTcwNTAwWhcNMTkwNzA2MjM1OTU5WjA9
+MQswCQYDVQQGEwJGUjERMA8GA1UEChMIQ2VydHBsdXMxGzAZBgNVBAMTEkNsYXNz
+IDIgUHJpbWFyeSBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBANxQ
+ltAS+DXSCHh6tlJw/W/uz7kRy1134ezpfgSN1sxvc0NXYKwzCkTsA18cgCSR5aiR
+VhKC9+Ar9NuuYS6JEI1rbLqzAr3VNsVINyPi8Fo3UjMXEuLRYE2+L0ER4/YXJQyL
+kcAbmXuZVg2v7tK8R1fjeUl7NIknJITesezpWE7+Tt9avkGtrAjFGA7v0lPubNCd
+EgETjdyAYveVqUSISnFOYFWe2yMZeVYHDD9jC1yw4r5+FfyUM1hBOHTE4Y+L3yas
+H7WLO7dDWWuwJKZtkIvEcupdM5i3y95ee++U8Rs+yskhwcWYAqqi9lt3m/V+llU0
+HGdpwPFC40es/CgcZlUCAwEAAaOBjDCBiTAPBgNVHRMECDAGAQH/AgEKMAsGA1Ud
+DwQEAwIBBjAdBgNVHQ4EFgQU43Mt38sOKAze3bOkynm4jrvoMIkwEQYJYIZIAYb4
+QgEBBAQDAgEGMDcGA1UdHwQwMC4wLKAqoCiGJmh0dHA6Ly93d3cuY2VydHBsdXMu
+Y29tL0NSTC9jbGFzczIuY3JsMA0GCSqGSIb3DQEBBQUAA4IBAQCnVM+IRBnL39R/
+AN9WM2K191EBkOvDP9GIROkkXe/nFL0gt5o8AP5tn9uQ3Nf0YtaLcF3n5QRIqWh8
+yfFC82x/xXp8HVGIutIKPidd3i1RTtMTZGnkLuPT55sJmabglZvOGtd/vjzOUrMR
+FcEPF80Du5wlFbqidon8BvEY0JNLDnyCt6X09l/+7UCmnYR0ObncHoUW2ikbhiMA
+ybuJfm6AiB4vFLQDJKgybwOaRywwvlbGp0ICcBvqQNi6BQNwB6SW//1IMwrh3KWB
+kJtN3X3n57LNXMhqlfil9o3EXXgIvnsG1knPGTZQIy4I5p4FTUcY1Rbpsda2ENW7
+l7+ijrRU
+-----END CERTIFICATE-----
+
 # Issuer: CN=DST Root CA X3 O=Digital Signature Trust Co.
 # Subject: CN=DST Root CA X3 O=Digital Signature Trust Co.
 # Label: "DST Root CA X3"
@@ -1124,6 +1219,36 @@ t0QmwCbAr1UwnjvVNioZBPRcHv/PLLf/0P2HQBHVESO7SMAhqaQoLf0V+LBOK/Qw
 WyH8EZE0vkHve52Xdf+XlcCWWC/qu0bXu+TZLg==
 -----END CERTIFICATE-----
 
+# Issuer: CN=Deutsche Telekom Root CA 2 O=Deutsche Telekom AG OU=T-TeleSec Trust Center
+# Subject: CN=Deutsche Telekom Root CA 2 O=Deutsche Telekom AG OU=T-TeleSec Trust Center
+# Label: "Deutsche Telekom Root CA 2"
+# Serial: 38
+# MD5 Fingerprint: 74:01:4a:91:b1:08:c4:58:ce:47:cd:f0:dd:11:53:08
+# SHA1 Fingerprint: 85:a4:08:c0:9c:19:3e:5d:51:58:7d:cd:d6:13:30:fd:8c:de:37:bf
+# SHA256 Fingerprint: b6:19:1a:50:d0:c3:97:7f:7d:a9:9b:cd:aa:c8:6a:22:7d:ae:b9:67:9e:c7:0b:a3:b0:c9:d9:22:71:c1:70:d3
+-----BEGIN CERTIFICATE-----
+MIIDnzCCAoegAwIBAgIBJjANBgkqhkiG9w0BAQUFADBxMQswCQYDVQQGEwJERTEc
+MBoGA1UEChMTRGV1dHNjaGUgVGVsZWtvbSBBRzEfMB0GA1UECxMWVC1UZWxlU2Vj
+IFRydXN0IENlbnRlcjEjMCEGA1UEAxMaRGV1dHNjaGUgVGVsZWtvbSBSb290IENB
+IDIwHhcNOTkwNzA5MTIxMTAwWhcNMTkwNzA5MjM1OTAwWjBxMQswCQYDVQQGEwJE
+RTEcMBoGA1UEChMTRGV1dHNjaGUgVGVsZWtvbSBBRzEfMB0GA1UECxMWVC1UZWxl
+U2VjIFRydXN0IENlbnRlcjEjMCEGA1UEAxMaRGV1dHNjaGUgVGVsZWtvbSBSb290
+IENBIDIwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCrC6M14IspFLEU
+ha88EOQ5bzVdSq7d6mGNlUn0b2SjGmBmpKlAIoTZ1KXleJMOaAGtuU1cOs7TuKhC
+QN/Po7qCWWqSG6wcmtoIKyUn+WkjR/Hg6yx6m/UTAtB+NHzCnjwAWav12gz1Mjwr
+rFDa1sPeg5TKqAyZMg4ISFZbavva4VhYAUlfckE8FQYBjl2tqriTtM2e66foai1S
+NNs671x1Udrb8zH57nGYMsRUFUQM+ZtV7a3fGAigo4aKSe5TBY8ZTNXeWHmb0moc
+QqvF1afPaA+W5OFhmHZhyJF81j4A4pFQh+GdCuatl9Idxjp9y7zaAzTVjlsB9WoH
+txa2bkp/AgMBAAGjQjBAMB0GA1UdDgQWBBQxw3kbuvVT1xfgiXotF2wKsyudMzAP
+BgNVHRMECDAGAQH/AgEFMA4GA1UdDwEB/wQEAwIBBjANBgkqhkiG9w0BAQUFAAOC
+AQEAlGRZrTlk5ynrE/5aw4sTV8gEJPB0d8Bg42f76Ymmg7+Wgnxu1MM9756Abrsp
+tJh6sTtU6zkXR34ajgv8HzFZMQSyzhfzLMdiNlXiItiJVbSYSKpk+tYcNthEeFpa
+IzpXl/V6ME+un2pMSyuOoAPjPuCp1NJ70rOo4nI8rZ7/gFnkm0W09juwzTkZmDLl
+6iFhkOQxIY40sfcvNUqFENrnijchvllj4PKFiDFT1FQUhXB59C4Gdyd1Lx+4ivn+
+xbrYNuSD7Odlt79jWvNGr4GUN9RBjNYj1h7P9WgbRGOiWrqnNVmh5XAFmw4jV5mU
+Cm26OWMohpLzGITY+9HPBVZkVw==
+-----END CERTIFICATE-----
+
 # Issuer: CN=Cybertrust Global Root O=Cybertrust, Inc
 # Subject: CN=Cybertrust Global Root O=Cybertrust, Inc
 # Label: "Cybertrust Global Root"
@@ -1434,6 +1559,47 @@ uLjbvrW5KfnaNwUASZQDhETnv0Mxz3WLJdH0pmT1kvarBes96aULNmLazAZfNou2
 XjG4Kvte9nHfRCaexOYNkbQudZWAUWpLMKawYqGT8ZvYzsRjdT9ZR7E=
 -----END CERTIFICATE-----
 
+# Issuer: CN=Staat der Nederlanden Root CA - G2 O=Staat der Nederlanden
+# Subject: CN=Staat der Nederlanden Root CA - G2 O=Staat der Nederlanden
+# Label: "Staat der Nederlanden Root CA - G2"
+# Serial: 10000012
+# MD5 Fingerprint: 7c:a5:0f:f8:5b:9a:7d:6d:30:ae:54:5a:e3:42:a2:8a
+# SHA1 Fingerprint: 59:af:82:79:91:86:c7:b4:75:07:cb:cf:03:57:46:eb:04:dd:b7:16
+# SHA256 Fingerprint: 66:8c:83:94:7d:a6:3b:72:4b:ec:e1:74:3c:31:a0:e6:ae:d0:db:8e:c5:b3:1b:e3:77:bb:78:4f:91:b6:71:6f
+-----BEGIN CERTIFICATE-----
+MIIFyjCCA7KgAwIBAgIEAJiWjDANBgkqhkiG9w0BAQsFADBaMQswCQYDVQQGEwJO
+TDEeMBwGA1UECgwVU3RhYXQgZGVyIE5lZGVybGFuZGVuMSswKQYDVQQDDCJTdGFh
+dCBkZXIgTmVkZXJsYW5kZW4gUm9vdCBDQSAtIEcyMB4XDTA4MDMyNjExMTgxN1oX
+DTIwMDMyNTExMDMxMFowWjELMAkGA1UEBhMCTkwxHjAcBgNVBAoMFVN0YWF0IGRl
+ciBOZWRlcmxhbmRlbjErMCkGA1UEAwwiU3RhYXQgZGVyIE5lZGVybGFuZGVuIFJv
+b3QgQ0EgLSBHMjCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAMVZ5291
+qj5LnLW4rJ4L5PnZyqtdj7U5EILXr1HgO+EASGrP2uEGQxGZqhQlEq0i6ABtQ8Sp
+uOUfiUtnvWFI7/3S4GCI5bkYYCjDdyutsDeqN95kWSpGV+RLufg3fNU254DBtvPU
+Z5uW6M7XxgpT0GtJlvOjCwV3SPcl5XCsMBQgJeN/dVrlSPhOewMHBPqCYYdu8DvE
+pMfQ9XQ+pV0aCPKbJdL2rAQmPlU6Yiile7Iwr/g3wtG61jj99O9JMDeZJiFIhQGp
+5Rbn3JBV3w/oOM2ZNyFPXfUib2rFEhZgF1XyZWampzCROME4HYYEhLoaJXhena/M
+UGDWE4dS7WMfbWV9whUYdMrhfmQpjHLYFhN9C0lK8SgbIHRrxT3dsKpICT0ugpTN
+GmXZK4iambwYfp/ufWZ8Pr2UuIHOzZgweMFvZ9C+X+Bo7d7iscksWXiSqt8rYGPy
+5V6548r6f1CGPqI0GAwJaCgRHOThuVw+R7oyPxjMW4T182t0xHJ04eOLoEq9jWYv
+6q012iDTiIJh8BIitrzQ1aTsr1SIJSQ8p22xcik/Plemf1WvbibG/ufMQFxRRIEK
+eN5KzlW/HdXZt1bv8Hb/C3m1r737qWmRRpdogBQ2HbN/uymYNqUg+oJgYjOk7Na6
+B6duxc8UpufWkjTYgfX8HV2qXB72o007uPc5AgMBAAGjgZcwgZQwDwYDVR0TAQH/
+BAUwAwEB/zBSBgNVHSAESzBJMEcGBFUdIAAwPzA9BggrBgEFBQcCARYxaHR0cDov
+L3d3dy5wa2lvdmVyaGVpZC5ubC9wb2xpY2llcy9yb290LXBvbGljeS1HMjAOBgNV
+HQ8BAf8EBAMCAQYwHQYDVR0OBBYEFJFoMocVHYnitfGsNig0jQt8YojrMA0GCSqG
+SIb3DQEBCwUAA4ICAQCoQUpnKpKBglBu4dfYszk78wIVCVBR7y29JHuIhjv5tLyS
+CZa59sCrI2AGeYwRTlHSeYAz+51IvuxBQ4EffkdAHOV6CMqqi3WtFMTC6GY8ggen
+5ieCWxjmD27ZUD6KQhgpxrRW/FYQoAUXvQwjf/ST7ZwaUb7dRUG/kSS0H4zpX897
+IZmflZ85OkYcbPnNe5yQzSipx6lVu6xiNGI1E0sUOlWDuYaNkqbG9AclVMwWVxJK
+gnjIFNkXgiYtXSAfea7+1HAWFpWD2DU5/1JddRwWxRNVz0fMdWVSSt7wsKfkCpYL
++63C4iWEst3kvX5ZbJvw8NjnyvLplzh+ib7M+zkXYT9y2zqR2GUBGR2tUKRXCnxL
+vJxxcypFURmFzI79R6d0lR2o0a9OF7FpJsKqeFdbxU2n5Z4FF5TKsl+gSRiNNOkm
+bEgeqmiSBeGCc1qb3AdbCG19ndeNIdn8FCCqwkXfP+cAslHkwvgFuXkajDTznlvk
+N1trSt8sV4pAWja63XVECDdCcAz+3F4hoKOKwJCcaNpQ5kUQR3i2TtJlycM33+FC
+Y7BXN0Ute4qcvwXqZVUz9zkQxSgqIXobisQk+T8VyJoVIPVVYpbtbZNQvOSqeK3Z
+ywplh6ZmwcSBo3c6WB4L7oOLnR7SUqTMHW+wmG2UMbX4cQrcufx9MmDm66+KAQ==
+-----END CERTIFICATE-----
+
 # Issuer: CN=Hongkong Post Root CA 1 O=Hongkong Post
 # Subject: CN=Hongkong Post Root CA 1 O=Hongkong Post
 # Label: "Hongkong Post Root CA 1"
@@ -2034,45 +2200,6 @@ t/2jioSgrGK+KwmHNPBqAbubKVY8/gA3zyNs8U6qtnRGEmyR7jTV7JqR50S+kDFy
 SjnRBUkLp7Y3gaVdjKozXoEofKd9J+sAro03
 -----END CERTIFICATE-----
 
-# Issuer: CN=EC-ACC O=Agencia Catalana de Certificacio (NIF Q-0801176-I) OU=Serveis Publics de Certificacio/Vegeu https://www.catcert.net/verarrel (c)03/Jerarquia Entitats de Certificacio Catalanes
-# Subject: CN=EC-ACC O=Agencia Catalana de Certificacio (NIF Q-0801176-I) OU=Serveis Publics de Certificacio/Vegeu https://www.catcert.net/verarrel (c)03/Jerarquia Entitats de Certificacio Catalanes
-# Label: "EC-ACC"
-# Serial: -23701579247955709139626555126524820479
-# MD5 Fingerprint: eb:f5:9d:29:0d:61:f9:42:1f:7c:c2:ba:6d:e3:15:09
-# SHA1 Fingerprint: 28:90:3a:63:5b:52:80:fa:e6:77:4c:0b:6d:a7:d6:ba:a6:4a:f2:e8
-# SHA256 Fingerprint: 88:49:7f:01:60:2f:31:54:24:6a:e2:8c:4d:5a:ef:10:f1:d8:7e:bb:76:62:6f:4a:e0:b7:f9:5b:a7:96:87:99
------BEGIN CERTIFICATE-----
-MIIFVjCCBD6gAwIBAgIQ7is969Qh3hSoYqwE893EATANBgkqhkiG9w0BAQUFADCB
-8zELMAkGA1UEBhMCRVMxOzA5BgNVBAoTMkFnZW5jaWEgQ2F0YWxhbmEgZGUgQ2Vy
-dGlmaWNhY2lvIChOSUYgUS0wODAxMTc2LUkpMSgwJgYDVQQLEx9TZXJ2ZWlzIFB1
-YmxpY3MgZGUgQ2VydGlmaWNhY2lvMTUwMwYDVQQLEyxWZWdldSBodHRwczovL3d3
-dy5jYXRjZXJ0Lm5ldC92ZXJhcnJlbCAoYykwMzE1MDMGA1UECxMsSmVyYXJxdWlh
-IEVudGl0YXRzIGRlIENlcnRpZmljYWNpbyBDYXRhbGFuZXMxDzANBgNVBAMTBkVD
-LUFDQzAeFw0wMzAxMDcyMzAwMDBaFw0zMTAxMDcyMjU5NTlaMIHzMQswCQYDVQQG
-EwJFUzE7MDkGA1UEChMyQWdlbmNpYSBDYXRhbGFuYSBkZSBDZXJ0aWZpY2FjaW8g
-KE5JRiBRLTA4MDExNzYtSSkxKDAmBgNVBAsTH1NlcnZlaXMgUHVibGljcyBkZSBD
-ZXJ0aWZpY2FjaW8xNTAzBgNVBAsTLFZlZ2V1IGh0dHBzOi8vd3d3LmNhdGNlcnQu
-bmV0L3ZlcmFycmVsIChjKTAzMTUwMwYDVQQLEyxKZXJhcnF1aWEgRW50aXRhdHMg
-ZGUgQ2VydGlmaWNhY2lvIENhdGFsYW5lczEPMA0GA1UEAxMGRUMtQUNDMIIBIjAN
-BgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAsyLHT+KXQpWIR4NA9h0X84NzJB5R
-85iKw5K4/0CQBXCHYMkAqbWUZRkiFRfCQ2xmRJoNBD45b6VLeqpjt4pEndljkYRm
-4CgPukLjbo73FCeTae6RDqNfDrHrZqJyTxIThmV6PttPB/SnCWDaOkKZx7J/sxaV
-HMf5NLWUhdWZXqBIoH7nF2W4onW4HvPlQn2v7fOKSGRdghST2MDk/7NQcvJ29rNd
-QlB50JQ+awwAvthrDk4q7D7SzIKiGGUzE3eeml0aE9jD2z3Il3rucO2n5nzbcc8t
-lGLfbdb1OL4/pYUKGbio2Al1QnDE6u/LDsg0qBIimAy4E5S2S+zw0JDnJwIDAQAB
-o4HjMIHgMB0GA1UdEQQWMBSBEmVjX2FjY0BjYXRjZXJ0Lm5ldDAPBgNVHRMBAf8E
-BTADAQH/MA4GA1UdDwEB/wQEAwIBBjAdBgNVHQ4EFgQUoMOLRKo3pUW/l4Ba0fF4
-opvpXY0wfwYDVR0gBHgwdjB0BgsrBgEEAfV4AQMBCjBlMCwGCCsGAQUFBwIBFiBo
-dHRwczovL3d3dy5jYXRjZXJ0Lm5ldC92ZXJhcnJlbDA1BggrBgEFBQcCAjApGidW
-ZWdldSBodHRwczovL3d3dy5jYXRjZXJ0Lm5ldC92ZXJhcnJlbCAwDQYJKoZIhvcN
-AQEFBQADggEBAKBIW4IB9k1IuDlVNZyAelOZ1Vr/sXE7zDkJlF7W2u++AVtd0x7Y
-/X1PzaBB4DSTv8vihpw3kpBWHNzrKQXlxJ7HNd+KDM3FIUPpqojlNcAZQmNaAl6k
-SBg6hW/cnbw/nZzBh7h6YQjpdwt/cKt63dmXLGQehb+8dJahw3oS7AwaboMMPOhy
-Rp/7SNVel+axofjk70YllJyJ22k4vuxcDlbHZVHlUIiIv0LVKz3l+bqeLrPK9HOS
-Agu+TGbrIP65y7WZf+a2E/rKS03Z7lNGBjvGTq2TWoF+bCpLagVFjPIhpDGQh2xl
-nJ2lYJU6Un/10asIbvPuW/mIPX64b24D5EI=
------END CERTIFICATE-----
-
 # Issuer: CN=Hellenic Academic and Research Institutions RootCA 2011 O=Hellenic Academic and Research Institutions Cert. Authority
 # Subject: CN=Hellenic Academic and Research Institutions RootCA 2011 O=Hellenic Academic and Research Institutions Cert. Authority
 # Label: "Hellenic Academic and Research Institutions RootCA 2011"
@@ -3326,6 +3453,46 @@ AAoACxGV2lZFA4gKn2fQ1XmxqI1AbQ3CekD6819kR5LLU7m7Wc5P/dAVUwHY3+vZ
 5nbv0CO7O6l5s9UCKc2Jo5YPSjXnTkLAdc0Hz+Ys63su
 -----END CERTIFICATE-----
 
+# Issuer: CN=Certinomis - Root CA O=Certinomis OU=0002 433998903
+# Subject: CN=Certinomis - Root CA O=Certinomis OU=0002 433998903
+# Label: "Certinomis - Root CA"
+# Serial: 1
+# MD5 Fingerprint: 14:0a:fd:8d:a8:28:b5:38:69:db:56:7e:61:22:03:3f
+# SHA1 Fingerprint: 9d:70:bb:01:a5:a4:a0:18:11:2e:f7:1c:01:b9:32:c5:34:e7:88:a8
+# SHA256 Fingerprint: 2a:99:f5:bc:11:74:b7:3c:bb:1d:62:08:84:e0:1c:34:e5:1c:cb:39:78:da:12:5f:0e:33:26:88:83:bf:41:58
+-----BEGIN CERTIFICATE-----
+MIIFkjCCA3qgAwIBAgIBATANBgkqhkiG9w0BAQsFADBaMQswCQYDVQQGEwJGUjET
+MBEGA1UEChMKQ2VydGlub21pczEXMBUGA1UECxMOMDAwMiA0MzM5OTg5MDMxHTAb
+BgNVBAMTFENlcnRpbm9taXMgLSBSb290IENBMB4XDTEzMTAyMTA5MTcxOFoXDTMz
+MTAyMTA5MTcxOFowWjELMAkGA1UEBhMCRlIxEzARBgNVBAoTCkNlcnRpbm9taXMx
+FzAVBgNVBAsTDjAwMDIgNDMzOTk4OTAzMR0wGwYDVQQDExRDZXJ0aW5vbWlzIC0g
+Um9vdCBDQTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBANTMCQosP5L2
+fxSeC5yaah1AMGT9qt8OHgZbn1CF6s2Nq0Nn3rD6foCWnoR4kkjW4znuzuRZWJfl
+LieY6pOod5tK8O90gC3rMB+12ceAnGInkYjwSond3IjmFPnVAy//ldu9n+ws+hQV
+WZUKxkd8aRi5pwP5ynapz8dvtF4F/u7BUrJ1Mofs7SlmO/NKFoL21prbcpjp3vDF
+TKWrteoB4owuZH9kb/2jJZOLyKIOSY008B/sWEUuNKqEUL3nskoTuLAPrjhdsKkb
+5nPJWqHZZkCqqU2mNAKthH6yI8H7KsZn9DS2sJVqM09xRLWtwHkziOC/7aOgFLSc
+CbAK42C++PhmiM1b8XcF4LVzbsF9Ri6OSyemzTUK/eVNfaoqoynHWmgE6OXWk6Ri
+wsXm9E/G+Z8ajYJJGYrKWUM66A0ywfRMEwNvbqY/kXPLynNvEiCL7sCCeN5LLsJJ
+wx3tFvYk9CcbXFcx3FXuqB5vbKziRcxXV4p1VxngtViZSTYxPDMBbRZKzbgqg4SG
+m/lg0h9tkQPTYKbVPZrdd5A9NaSfD171UkRpucC63M9933zZxKyGIjK8e2uR73r4
+F2iw4lNVYC2vPsKD2NkJK/DAZNuHi5HMkesE/Xa0lZrmFAYb1TQdvtj/dBxThZng
+WVJKYe2InmtJiUZ+IFrZ50rlau7SZRFDAgMBAAGjYzBhMA4GA1UdDwEB/wQEAwIB
+BjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBTvkUz1pcMw6C8I6tNxIqSSaHh0
+2TAfBgNVHSMEGDAWgBTvkUz1pcMw6C8I6tNxIqSSaHh02TANBgkqhkiG9w0BAQsF
+AAOCAgEAfj1U2iJdGlg+O1QnurrMyOMaauo++RLrVl89UM7g6kgmJs95Vn6RHJk/
+0KGRHCwPT5iVWVO90CLYiF2cN/z7ZMF4jIuaYAnq1fohX9B0ZedQxb8uuQsLrbWw
+F6YSjNRieOpWauwK0kDDPAUwPk2Ut59KA9N9J0u2/kTO+hkzGm2kQtHdzMjI1xZS
+g081lLMSVX3l4kLr5JyTCcBMWwerx20RoFAXlCOotQqSD7J6wWAsOMwaplv/8gzj
+qh8c3LigkyfeY+N/IZ865Z764BNqdeuWXGKRlI5nU7aJ+BIJy29SWwNyhlCVCNSN
+h4YVH5Uk2KRvms6knZtt0rJ2BobGVgjF6wnaNsIbW0G+YSrjcOa4pvi2WsS9Iff/
+ql+hbHY5ZtbqTFXhADObE5hjyW/QASAJN1LnDE8+zbz1X5YnpyACleAu6AdBBR8V
+btaw5BngDwKTACdyxYvRVB9dSsNAl35VpnzBMwQUAR1JIGkLGZOdblgi90AMRgwj
+Y/M50n92Uaf0yKHxDHYiI0ZSKS3io0EHVmmY0gUJvGnHWmHNj4FgFU2A3ZDifcRQ
+8ow7bkrHxuaAKzyBvBGAFhAn1/DNP3nMcyrDflOR1m749fPH0FFNjkulW+YZFzvW
+gQncItzujrnEj1PhZ7szuIgVRs/taTX/dQ1G885x4cVrhkIGuUE=
+-----END CERTIFICATE-----
+
 # Issuer: CN=OISTE WISeKey Global Root GB CA O=WISeKey OU=OISTE Foundation Endorsed
 # Subject: CN=OISTE WISeKey Global Root GB CA O=WISeKey OU=OISTE Foundation Endorsed
 # Label: "OISTE WISeKey Global Root GB CA"
@@ -3682,6 +3849,47 @@ CkcO8DdZEv8tmZQoTipPNU0zWgIxAOp1AE47xDqUEpHJWEadIRNyp4iciuRMStuW
 1KyLa2tJElMzrdfkviT8tQp21KW8EA==
 -----END CERTIFICATE-----
 
+# Issuer: CN=LuxTrust Global Root 2 O=LuxTrust S.A.
+# Subject: CN=LuxTrust Global Root 2 O=LuxTrust S.A.
+# Label: "LuxTrust Global Root 2"
+# Serial: 59914338225734147123941058376788110305822489521
+# MD5 Fingerprint: b2:e1:09:00:61:af:f7:f1:91:6f:c4:ad:8d:5e:3b:7c
+# SHA1 Fingerprint: 1e:0e:56:19:0a:d1:8b:25:98:b2:04:44:ff:66:8a:04:17:99:5f:3f
+# SHA256 Fingerprint: 54:45:5f:71:29:c2:0b:14:47:c4:18:f9:97:16:8f:24:c5:8f:c5:02:3b:f5:da:5b:e2:eb:6e:1d:d8:90:2e:d5
+-----BEGIN CERTIFICATE-----
+MIIFwzCCA6ugAwIBAgIUCn6m30tEntpqJIWe5rgV0xZ/u7EwDQYJKoZIhvcNAQEL
+BQAwRjELMAkGA1UEBhMCTFUxFjAUBgNVBAoMDUx1eFRydXN0IFMuQS4xHzAdBgNV
+BAMMFkx1eFRydXN0IEdsb2JhbCBSb290IDIwHhcNMTUwMzA1MTMyMTU3WhcNMzUw
+MzA1MTMyMTU3WjBGMQswCQYDVQQGEwJMVTEWMBQGA1UECgwNTHV4VHJ1c3QgUy5B
+LjEfMB0GA1UEAwwWTHV4VHJ1c3QgR2xvYmFsIFJvb3QgMjCCAiIwDQYJKoZIhvcN
+AQEBBQADggIPADCCAgoCggIBANeFl78RmOnwYoNMPIf5U2o3C/IPPIfOb9wmKb3F
+ibrJgz337spbxm1Jc7TJRqMbNBM/wYlFV/TZsfs2ZUv7COJIcRHIbjuend+JZTem
+hfY7RBi2xjcwYkSSl2l9QjAk5A0MiWtj3sXh306pFGxT4GHO9hcvHTy95iJMHZP1
+EMShduxq3sVs35a0VkBCwGKSMKEtFZSg0iAGCW5qbeXrt77U8PEVfIvmTroTzEsn
+Xpk8F12PgX8zPU/TPxvsXD/wPEx1bvKm1Z3aLQdjAsZy6ZS8TEmVT4hSyNvoaYL4
+zDRbIvCGp4m9SAptZoFtyMhk+wHh9OHe2Z7d21vUKpkmFRseTJIpgp7VkoGSQXAZ
+96Tlk0u8d2cx3Rz9MXANF5kM+Qw5GSoXtTBxVdUPrljhPS80m8+f9niFwpN6cj5m
+j5wWEWCPnolvZ77gR1o7DJpni89Gxq44o/KnvObWhWszJHAiS8sIm7vI+AIpHb4g
+DEa/a4ebsypmQjVGbKq6rfmYe+lQVRQxv7HaLe2ArWgk+2mr2HETMOZns4dA/Yl+
+8kPREd8vZS9kzl8UubG/Mb2HeFpZZYiq/FkySIbWTLkpS5XTdvN3JW1CHDiDTf2j
+X5t/Lax5Gw5CMZdjpPuKadUiDTSQMC6otOBttpSsvItO13D8xTiOZCXhTTmQzsmH
+hFhxAgMBAAGjgagwgaUwDwYDVR0TAQH/BAUwAwEB/zBCBgNVHSAEOzA5MDcGByuB
+KwEBAQowLDAqBggrBgEFBQcCARYeaHR0cHM6Ly9yZXBvc2l0b3J5Lmx1eHRydXN0
+Lmx1MA4GA1UdDwEB/wQEAwIBBjAfBgNVHSMEGDAWgBT/GCh2+UgFLKGu8SsbK7JT
++Et8szAdBgNVHQ4EFgQU/xgodvlIBSyhrvErGyuyU/hLfLMwDQYJKoZIhvcNAQEL
+BQADggIBAGoZFO1uecEsh9QNcH7X9njJCwROxLHOk3D+sFTAMs2ZMGQXvw/l4jP9
+BzZAcg4atmpZ1gDlaCDdLnINH2pkMSCEfUmmWjfrRcmF9dTHF5kH5ptV5AzoqbTO
+jFu1EVzPig4N1qx3gf4ynCSecs5U89BvolbW7MM3LGVYvlcAGvI1+ut7MV3CwRI9
+loGIlonBWVx65n9wNOeD4rHh4bhY79SV5GCc8JaXcozrhAIuZY+kt9J/Z93I055c
+qqmkoCUUBpvsT34tC38ddfEz2O3OuHVtPlu5mB0xDVbYQw8wkbIEa91WvpWAVWe+
+2M2D2RjuLg+GLZKecBPs3lHJQ3gCpU3I+V/EkVhGFndadKpAvAefMLmx9xIX3eP/
+JEAdemrRTxgKqpAd60Ae36EeRJIQmvKN4dFLRp7oRUKX6kWZ8+xm1QL68qZKJKre
+zrnK+T+Tb/mjuuqlPpmt/f97mfVl7vBZKGfXkJWkE4SphMHozs51k2MavDzq1WQf
+LSoSOcbDWjLtR5EWDrw4wVDej8oqkDQc7kGUnF4ZLvhFSZl0kbAEb+MEWrGrKqv+
+x9CWttrhSmQGbmBNvUJO/3jaJMobtNeWOWyu8Q6qp31IiyBMz2TWuJdGsE7RKlY6
+oJO9r4Ak4Ap+58rVyuiFVdw2KuGUaJPHZnJED4AhMmwlxyOAgwrr
+-----END CERTIFICATE-----
+
 # Issuer: CN=TUBITAK Kamu SM SSL Kok Sertifikasi - Surum 1 O=Turkiye Bilimsel ve Teknolojik Arastirma Kurumu - TUBITAK OU=Kamu Sertifikasyon Merkezi - Kamu SM
 # Subject: CN=TUBITAK Kamu SM SSL Kok Sertifikasi - Surum 1 O=Turkiye Bilimsel ve Teknolojik Arastirma Kurumu - TUBITAK OU=Kamu Sertifikasyon Merkezi - Kamu SM
 # Label: "TUBITAK Kamu SM SSL Kok Sertifikasi - Surum 1"
@@ -4448,173 +4656,3 @@ L5/ndtFhKvshuzHQqp9HpLIiyhY6UFfEW0NnxWViA0kB60PZ2Pierc+xYw5F9KBa
 LJstxabArahH9CdMOA0uG0k7UvToiIMrVCjU8jVStDKDYmlkDJGcn5fqdBb9HxEG
 mpv0
 -----END CERTIFICATE-----
-
-# Issuer: CN=Entrust Root Certification Authority - G4 O=Entrust, Inc. OU=See www.entrust.net/legal-terms/(c) 2015 Entrust, Inc. - for authorized use only
-# Subject: CN=Entrust Root Certification Authority - G4 O=Entrust, Inc. OU=See www.entrust.net/legal-terms/(c) 2015 Entrust, Inc. - for authorized use only
-# Label: "Entrust Root Certification Authority - G4"
-# Serial: 289383649854506086828220374796556676440
-# MD5 Fingerprint: 89:53:f1:83:23:b7:7c:8e:05:f1:8c:71:38:4e:1f:88
-# SHA1 Fingerprint: 14:88:4e:86:26:37:b0:26:af:59:62:5c:40:77:ec:35:29:ba:96:01
-# SHA256 Fingerprint: db:35:17:d1:f6:73:2a:2d:5a:b9:7c:53:3e:c7:07:79:ee:32:70:a6:2f:b4:ac:42:38:37:24:60:e6:f0:1e:88
------BEGIN CERTIFICATE-----
-MIIGSzCCBDOgAwIBAgIRANm1Q3+vqTkPAAAAAFVlrVgwDQYJKoZIhvcNAQELBQAw
-gb4xCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1FbnRydXN0LCBJbmMuMSgwJgYDVQQL
-Ex9TZWUgd3d3LmVudHJ1c3QubmV0L2xlZ2FsLXRlcm1zMTkwNwYDVQQLEzAoYykg
-MjAxNSBFbnRydXN0LCBJbmMuIC0gZm9yIGF1dGhvcml6ZWQgdXNlIG9ubHkxMjAw
-BgNVBAMTKUVudHJ1c3QgUm9vdCBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eSAtIEc0
-MB4XDTE1MDUyNzExMTExNloXDTM3MTIyNzExNDExNlowgb4xCzAJBgNVBAYTAlVT
-MRYwFAYDVQQKEw1FbnRydXN0LCBJbmMuMSgwJgYDVQQLEx9TZWUgd3d3LmVudHJ1
-c3QubmV0L2xlZ2FsLXRlcm1zMTkwNwYDVQQLEzAoYykgMjAxNSBFbnRydXN0LCBJ
-bmMuIC0gZm9yIGF1dGhvcml6ZWQgdXNlIG9ubHkxMjAwBgNVBAMTKUVudHJ1c3Qg
-Um9vdCBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eSAtIEc0MIICIjANBgkqhkiG9w0B
-AQEFAAOCAg8AMIICCgKCAgEAsewsQu7i0TD/pZJH4i3DumSXbcr3DbVZwbPLqGgZ
-2K+EbTBwXX7zLtJTmeH+H17ZSK9dE43b/2MzTdMAArzE+NEGCJR5WIoV3imz/f3E
-T+iq4qA7ec2/a0My3dl0ELn39GjUu9CH1apLiipvKgS1sqbHoHrmSKvS0VnM1n4j
-5pds8ELl3FFLFUHtSUrJ3hCX1nbB76W1NhSXNdh4IjVS70O92yfbYVaCNNzLiGAM
-C1rlLAHGVK/XqsEQe9IFWrhAnoanw5CGAlZSCXqc0ieCU0plUmr1POeo8pyvi73T
-DtTUXm6Hnmo9RR3RXRv06QqsYJn7ibT/mCzPfB3pAqoEmh643IhuJbNsZvc8kPNX
-wbMv9W3y+8qh+CmdRouzavbmZwe+LGcKKh9asj5XxNMhIWNlUpEbsZmOeX7m640A
-2Vqq6nPopIICR5b+W45UYaPrL0swsIsjdXJ8ITzI9vF01Bx7owVV7rtNOzK+mndm
-nqxpkCIHH2E6lr7lmk/MBTwoWdPBDFSoWWG9yHJM6Nyfh3+9nEg2XpWjDrk4JFX8
-dWbrAuMINClKxuMrLzOg2qOGpRKX/YAr2hRC45K9PvJdXmd0LhyIRyk0X+IyqJwl
-N4y6mACXi0mWHv0liqzc2thddG5msP9E36EYxr5ILzeUePiVSj9/E15dWf10hkNj
-c0kCAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYwHQYD
-VR0OBBYEFJ84xFYjwznooHFs6FRM5Og6sb9nMA0GCSqGSIb3DQEBCwUAA4ICAQAS
-5UKme4sPDORGpbZgQIeMJX6tuGguW8ZAdjwD+MlZ9POrYs4QjbRaZIxowLByQzTS
-Gwv2LFPSypBLhmb8qoMi9IsabyZIrHZ3CL/FmFz0Jomee8O5ZDIBf9PD3Vht7LGr
-hFV0d4QEJ1JrhkzO3bll/9bGXp+aEJlLdWr+aumXIOTkdnrG0CSqkM0gkLpHZPt/
-B7NTeLUKYvJzQ85BK4FqLoUWlFPUa19yIqtRLULVAJyZv967lDtX/Zr1hstWO1uI
-AeV8KEsD+UmDfLJ/fOPtjqF/YFOOVZ1QNBIPt5d7bIdKROf1beyAN/BYGW5KaHbw
-H5Lk6rWS02FREAutp9lfx1/cH6NcjKF+m7ee01ZvZl4HliDtC3T7Zk6LERXpgUl+
-b7DUUH8i119lAg2m9IUe2K4GS0qn0jFmwvjO5QimpAKWRGhXxNUzzxkvFMSUHHuk
-2fCfDrGA4tGeEWSpiBE6doLlYsKA2KSD7ZPvfC+QsDJMlhVoSFLUmQjAJOgc47Ol
-IQ6SwJAfzyBfyjs4x7dtOvPmRLgOMWuIjnDrnBdSqEGULoe256YSxXXfW8AKbnuk
-5F6G+TaU33fD6Q3AOfF5u0aOq0NZJ7cguyPpVkAh7DE9ZapD8j3fcEThuk0mEDuY
-n/PIjhs4ViFqUZPTkcpG2om3PVODLAgfi49T3f+sHw==
------END CERTIFICATE-----
-
-# Issuer: CN=Microsoft ECC Root Certificate Authority 2017 O=Microsoft Corporation
-# Subject: CN=Microsoft ECC Root Certificate Authority 2017 O=Microsoft Corporation
-# Label: "Microsoft ECC Root Certificate Authority 2017"
-# Serial: 136839042543790627607696632466672567020
-# MD5 Fingerprint: dd:a1:03:e6:4a:93:10:d1:bf:f0:19:42:cb:fe:ed:67
-# SHA1 Fingerprint: 99:9a:64:c3:7f:f4:7d:9f:ab:95:f1:47:69:89:14:60:ee:c4:c3:c5
-# SHA256 Fingerprint: 35:8d:f3:9d:76:4a:f9:e1:b7:66:e9:c9:72:df:35:2e:e1:5c:fa:c2:27:af:6a:d1:d7:0e:8e:4a:6e:dc:ba:02
------BEGIN CERTIFICATE-----
-MIICWTCCAd+gAwIBAgIQZvI9r4fei7FK6gxXMQHC7DAKBggqhkjOPQQDAzBlMQsw
-CQYDVQQGEwJVUzEeMBwGA1UEChMVTWljcm9zb2Z0IENvcnBvcmF0aW9uMTYwNAYD
-VQQDEy1NaWNyb3NvZnQgRUNDIFJvb3QgQ2VydGlmaWNhdGUgQXV0aG9yaXR5IDIw
-MTcwHhcNMTkxMjE4MjMwNjQ1WhcNNDIwNzE4MjMxNjA0WjBlMQswCQYDVQQGEwJV
-UzEeMBwGA1UEChMVTWljcm9zb2Z0IENvcnBvcmF0aW9uMTYwNAYDVQQDEy1NaWNy
-b3NvZnQgRUNDIFJvb3QgQ2VydGlmaWNhdGUgQXV0aG9yaXR5IDIwMTcwdjAQBgcq
-hkjOPQIBBgUrgQQAIgNiAATUvD0CQnVBEyPNgASGAlEvaqiBYgtlzPbKnR5vSmZR
-ogPZnZH6thaxjG7efM3beaYvzrvOcS/lpaso7GMEZpn4+vKTEAXhgShC48Zo9OYb
-hGBKia/teQ87zvH2RPUBeMCjVDBSMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMBAf8E
-BTADAQH/MB0GA1UdDgQWBBTIy5lycFIM+Oa+sgRXKSrPQhDtNTAQBgkrBgEEAYI3
-FQEEAwIBADAKBggqhkjOPQQDAwNoADBlAjBY8k3qDPlfXu5gKcs68tvWMoQZP3zV
-L8KxzJOuULsJMsbG7X7JNpQS5GiFBqIb0C8CMQCZ6Ra0DvpWSNSkMBaReNtUjGUB
-iudQZsIxtzm6uBoiB078a1QWIP8rtedMDE2mT3M=
------END CERTIFICATE-----
-
-# Issuer: CN=Microsoft RSA Root Certificate Authority 2017 O=Microsoft Corporation
-# Subject: CN=Microsoft RSA Root Certificate Authority 2017 O=Microsoft Corporation
-# Label: "Microsoft RSA Root Certificate Authority 2017"
-# Serial: 40975477897264996090493496164228220339
-# MD5 Fingerprint: 10:ff:00:ff:cf:c9:f8:c7:7a:c0:ee:35:8e:c9:0f:47
-# SHA1 Fingerprint: 73:a5:e6:4a:3b:ff:83:16:ff:0e:dc:cc:61:8a:90:6e:4e:ae:4d:74
-# SHA256 Fingerprint: c7:41:f7:0f:4b:2a:8d:88:bf:2e:71:c1:41:22:ef:53:ef:10:eb:a0:cf:a5:e6:4c:fa:20:f4:18:85:30:73:e0
------BEGIN CERTIFICATE-----
-MIIFqDCCA5CgAwIBAgIQHtOXCV/YtLNHcB6qvn9FszANBgkqhkiG9w0BAQwFADBl
-MQswCQYDVQQGEwJVUzEeMBwGA1UEChMVTWljcm9zb2Z0IENvcnBvcmF0aW9uMTYw
-NAYDVQQDEy1NaWNyb3NvZnQgUlNBIFJvb3QgQ2VydGlmaWNhdGUgQXV0aG9yaXR5
-IDIwMTcwHhcNMTkxMjE4MjI1MTIyWhcNNDIwNzE4MjMwMDIzWjBlMQswCQYDVQQG
-EwJVUzEeMBwGA1UEChMVTWljcm9zb2Z0IENvcnBvcmF0aW9uMTYwNAYDVQQDEy1N
-aWNyb3NvZnQgUlNBIFJvb3QgQ2VydGlmaWNhdGUgQXV0aG9yaXR5IDIwMTcwggIi
-MA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQDKW76UM4wplZEWCpW9R2LBifOZ
-Nt9GkMml7Xhqb0eRaPgnZ1AzHaGm++DlQ6OEAlcBXZxIQIJTELy/xztokLaCLeX0
-ZdDMbRnMlfl7rEqUrQ7eS0MdhweSE5CAg2Q1OQT85elss7YfUJQ4ZVBcF0a5toW1
-HLUX6NZFndiyJrDKxHBKrmCk3bPZ7Pw71VdyvD/IybLeS2v4I2wDwAW9lcfNcztm
-gGTjGqwu+UcF8ga2m3P1eDNbx6H7JyqhtJqRjJHTOoI+dkC0zVJhUXAoP8XFWvLJ
-jEm7FFtNyP9nTUwSlq31/niol4fX/V4ggNyhSyL71Imtus5Hl0dVe49FyGcohJUc
-aDDv70ngNXtk55iwlNpNhTs+VcQor1fznhPbRiefHqJeRIOkpcrVE7NLP8TjwuaG
-YaRSMLl6IE9vDzhTyzMMEyuP1pq9KsgtsRx9S1HKR9FIJ3Jdh+vVReZIZZ2vUpC6
-W6IYZVcSn2i51BVrlMRpIpj0M+Dt+VGOQVDJNE92kKz8OMHY4Xu54+OU4UZpyw4K
-UGsTuqwPN1q3ErWQgR5WrlcihtnJ0tHXUeOrO8ZV/R4O03QK0dqq6mm4lyiPSMQH
-+FJDOvTKVTUssKZqwJz58oHhEmrARdlns87/I6KJClTUFLkqqNfs+avNJVgyeY+Q
-W5g5xAgGwax/Dj0ApQIDAQABo1QwUjAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/
-BAUwAwEB/zAdBgNVHQ4EFgQUCctZf4aycI8awznjwNnpv7tNsiMwEAYJKwYBBAGC
-NxUBBAMCAQAwDQYJKoZIhvcNAQEMBQADggIBAKyvPl3CEZaJjqPnktaXFbgToqZC
-LgLNFgVZJ8og6Lq46BrsTaiXVq5lQ7GPAJtSzVXNUzltYkyLDVt8LkS/gxCP81OC
-gMNPOsduET/m4xaRhPtthH80dK2Jp86519efhGSSvpWhrQlTM93uCupKUY5vVau6
-tZRGrox/2KJQJWVggEbbMwSubLWYdFQl3JPk+ONVFT24bcMKpBLBaYVu32TxU5nh
-SnUgnZUP5NbcA/FZGOhHibJXWpS2qdgXKxdJ5XbLwVaZOjex/2kskZGT4d9Mozd2
-TaGf+G0eHdP67Pv0RR0Tbc/3WeUiJ3IrhvNXuzDtJE3cfVa7o7P4NHmJweDyAmH3
-pvwPuxwXC65B2Xy9J6P9LjrRk5Sxcx0ki69bIImtt2dmefU6xqaWM/5TkshGsRGR
-xpl/j8nWZjEgQRCHLQzWwa80mMpkg/sTV9HB8Dx6jKXB/ZUhoHHBk2dxEuqPiApp
-GWSZI1b7rCoucL5mxAyE7+WL85MB+GqQk2dLsmijtWKP6T+MejteD+eMuMZ87zf9
-dOLITzNy4ZQ5bb0Sr74MTnB8G2+NszKTc0QWbej09+CVgI+WXTik9KveCjCHk9hN
-AHFiRSdLOkKEW39lt2c0Ui2cFmuqqNh7o0JMcccMyj6D5KbvtwEwXlGjefVwaaZB
-RA+GsCyRxj3qrg+E
------END CERTIFICATE-----
-
-# Issuer: CN=e-Szigno Root CA 2017 O=Microsec Ltd.
-# Subject: CN=e-Szigno Root CA 2017 O=Microsec Ltd.
-# Label: "e-Szigno Root CA 2017"
-# Serial: 411379200276854331539784714
-# MD5 Fingerprint: de:1f:f6:9e:84:ae:a7:b4:21:ce:1e:58:7d:d1:84:98
-# SHA1 Fingerprint: 89:d4:83:03:4f:9e:9a:48:80:5f:72:37:d4:a9:a6:ef:cb:7c:1f:d1
-# SHA256 Fingerprint: be:b0:0b:30:83:9b:9b:c3:2c:32:e4:44:79:05:95:06:41:f2:64:21:b1:5e:d0:89:19:8b:51:8a:e2:ea:1b:99
------BEGIN CERTIFICATE-----
-MIICQDCCAeWgAwIBAgIMAVRI7yH9l1kN9QQKMAoGCCqGSM49BAMCMHExCzAJBgNV
-BAYTAkhVMREwDwYDVQQHDAhCdWRhcGVzdDEWMBQGA1UECgwNTWljcm9zZWMgTHRk
-LjEXMBUGA1UEYQwOVkFUSFUtMjM1ODQ0OTcxHjAcBgNVBAMMFWUtU3ppZ25vIFJv
-b3QgQ0EgMjAxNzAeFw0xNzA4MjIxMjA3MDZaFw00MjA4MjIxMjA3MDZaMHExCzAJ
-BgNVBAYTAkhVMREwDwYDVQQHDAhCdWRhcGVzdDEWMBQGA1UECgwNTWljcm9zZWMg
-THRkLjEXMBUGA1UEYQwOVkFUSFUtMjM1ODQ0OTcxHjAcBgNVBAMMFWUtU3ppZ25v
-IFJvb3QgQ0EgMjAxNzBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABJbcPYrYsHtv
-xie+RJCxs1YVe45DJH0ahFnuY2iyxl6H0BVIHqiQrb1TotreOpCmYF9oMrWGQd+H
-Wyx7xf58etqjYzBhMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMB0G
-A1UdDgQWBBSHERUI0arBeAyxr87GyZDvvzAEwDAfBgNVHSMEGDAWgBSHERUI0arB
-eAyxr87GyZDvvzAEwDAKBggqhkjOPQQDAgNJADBGAiEAtVfd14pVCzbhhkT61Nlo
-jbjcI4qKDdQvfepz7L9NbKgCIQDLpbQS+ue16M9+k/zzNY9vTlp8tLxOsvxyqltZ
-+efcMQ==
------END CERTIFICATE-----
-
-# Issuer: O=CERTSIGN SA OU=certSIGN ROOT CA G2
-# Subject: O=CERTSIGN SA OU=certSIGN ROOT CA G2
-# Label: "certSIGN Root CA G2"
-# Serial: 313609486401300475190
-# MD5 Fingerprint: 8c:f1:75:8a:c6:19:cf:94:b7:f7:65:20:87:c3:97:c7
-# SHA1 Fingerprint: 26:f9:93:b4:ed:3d:28:27:b0:b9:4b:a7:e9:15:1d:a3:8d:92:e5:32
-# SHA256 Fingerprint: 65:7c:fe:2f:a7:3f:aa:38:46:25:71:f3:32:a2:36:3a:46:fc:e7:02:09:51:71:07:02:cd:fb:b6:ee:da:33:05
------BEGIN CERTIFICATE-----
-MIIFRzCCAy+gAwIBAgIJEQA0tk7GNi02MA0GCSqGSIb3DQEBCwUAMEExCzAJBgNV
-BAYTAlJPMRQwEgYDVQQKEwtDRVJUU0lHTiBTQTEcMBoGA1UECxMTY2VydFNJR04g
-Uk9PVCBDQSBHMjAeFw0xNzAyMDYwOTI3MzVaFw00MjAyMDYwOTI3MzVaMEExCzAJ
-BgNVBAYTAlJPMRQwEgYDVQQKEwtDRVJUU0lHTiBTQTEcMBoGA1UECxMTY2VydFNJ
-R04gUk9PVCBDQSBHMjCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAMDF
-dRmRfUR0dIf+DjuW3NgBFszuY5HnC2/OOwppGnzC46+CjobXXo9X69MhWf05N0Iw
-vlDqtg+piNguLWkh59E3GE59kdUWX2tbAMI5Qw02hVK5U2UPHULlj88F0+7cDBrZ
-uIt4ImfkabBoxTzkbFpG583H+u/E7Eu9aqSs/cwoUe+StCmrqzWaTOTECMYmzPhp
-n+Sc8CnTXPnGFiWeI8MgwT0PPzhAsP6CRDiqWhqKa2NYOLQV07YRaXseVO6MGiKs
-cpc/I1mbySKEwQdPzH/iV8oScLumZfNpdWO9lfsbl83kqK/20U6o2YpxJM02PbyW
-xPFsqa7lzw1uKA2wDrXKUXt4FMMgL3/7FFXhEZn91QqhngLjYl/rNUssuHLoPj1P
-rCy7Lobio3aP5ZMqz6WryFyNSwb/EkaseMsUBzXgqd+L6a8VTxaJW732jcZZroiF
-DsGJ6x9nxUWO/203Nit4ZoORUSs9/1F3dmKh7Gc+PoGD4FapUB8fepmrY7+EF3fx
-DTvf95xhszWYijqy7DwaNz9+j5LP2RIUZNoQAhVB/0/E6xyjyfqZ90bp4RjZsbgy
-LcsUDFDYg2WD7rlcz8sFWkz6GZdr1l0T08JcVLwyc6B49fFtHsufpaafItzRUZ6C
-eWRgKRM+o/1Pcmqr4tTluCRVLERLiohEnMqE0yo7AgMBAAGjQjBAMA8GA1UdEwEB
-/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMB0GA1UdDgQWBBSCIS1mxteg4BXrzkwJ
-d8RgnlRuAzANBgkqhkiG9w0BAQsFAAOCAgEAYN4auOfyYILVAzOBywaK8SJJ6ejq
-kX/GM15oGQOGO0MBzwdw5AgeZYWR5hEit/UCI46uuR59H35s5r0l1ZUa8gWmr4UC
-b6741jH/JclKyMeKqdmfS0mbEVeZkkMR3rYzpMzXjWR91M08KCy0mpbqTfXERMQl
-qiCA2ClV9+BB/AYm/7k29UMUA2Z44RGx2iBfRgB4ACGlHgAoYXhvqAEBj500mv/0
-OJD7uNGzcgbJceaBxXntC6Z58hMLnPddDnskk7RI24Zf3lCGeOdA5jGokHZwYa+c
-NywRtYK3qq4kNFtyDGkNzVmf9nGvnAvRCjj5BiKDUyUM/FHE5r7iOZULJK2v0ZXk
-ltd0ZGtxTgI8qoXzIKNDOXZbbFD+mpwUHmUUihW9o4JFWklWatKcsWMy5WHgUyIO
-pwpJ6st+H6jiYoD2EEVSmAYY3qXNL3+q1Ok+CHLsIwMCPKaq2LxndD0UF/tUSxfj
-03k9bWtJySgOLnRQvwzZRjoQhsmnP+mg7H/rpXdYaXHmgwo38oZJar55CJD2AhZk
-PuXaTH4MNMn5X7azKFGnpyuqSfqNZSlO42sTp5SjLVFteAxEy9/eCG/Oo2Sr05WE
-1LlSVHJ7liXMvGnjSG4N0MedJ5qq+BOS3R7fY581qRY27Iy4g/Q9iY/NtBde17MX
-QRBdJ3NghVdJIgc=
------END CERTIFICATE-----

Contents/Libraries/Shared/certifi/core.py

@@ -4,57 +4,12 @@
 certifi.py
 ~~~~~~~~~~
 
-This module returns the installation location of cacert.pem or its contents.
+This module returns the installation location of cacert.pem.
 """
 import os
 
-try:
-    from importlib.resources import path as get_path, read_text
 
-    _CACERT_CTX = None
-    _CACERT_PATH = None
+def where():
+    f = os.path.dirname(__file__)
 
-    def where():
-        # This is slightly terrible, but we want to delay extracting the file
-        # in cases where we're inside of a zipimport situation until someone
-        # actually calls where(), but we don't want to re-extract the file
-        # on every call of where(), so we'll do it once then store it in a
-        # global variable.
-        global _CACERT_CTX
-        global _CACERT_PATH
-        if _CACERT_PATH is None:
-            # This is slightly janky, the importlib.resources API wants you to
-            # manage the cleanup of this file, so it doesn't actually return a
-            # path, it returns a context manager that will give you the path
-            # when you enter it and will do any cleanup when you leave it. In
-            # the common case of not needing a temporary file, it will just
-            # return the file system location and the __exit__() is a no-op.
-            #
-            # We also have to hold onto the actual context manager, because
-            # it will do the cleanup whenever it gets garbage collected, so
-            # we will also store that at the global level as well.
-            _CACERT_CTX = get_path("certifi", "cacert.pem")
-            _CACERT_PATH = str(_CACERT_CTX.__enter__())
-
-        return _CACERT_PATH
-
-
-except ImportError:
-    # This fallback will work for Python versions prior to 3.7 that lack the
-    # importlib.resources module but relies on the existing `where` function
-    # so won't address issues with environments like PyOxidizer that don't set
-    # __file__ on modules.
-    def read_text(_module, _path, encoding="ascii"):
-        with open(where(), "r", encoding=encoding) as data:
-            return data.read()
-
-    # If we don't have importlib.resources, then we will just do the old logic
-    # of assuming we're on the filesystem and munge the path directly.
-    def where():
-        f = os.path.dirname(__file__)
-
-        return os.path.join(f, "cacert.pem")
-
-
-def contents():
-    return read_text("certifi", "cacert.pem", encoding="ascii")
+    return os.path.join(f, 'cacert.pem')

Contents/Libraries/Shared/pbkdf2.py

@@ -1,297 +0,0 @@
-#!/usr/bin/python
-# -*- coding: ascii -*-
-###########################################################################
-# pbkdf2 - PKCS#5 v2.0 Password-Based Key Derivation
-#
-# Copyright (C) 2007-2011 Dwayne C. Litzenberger <dlitz@dlitz.net>
-#
-# Permission is hereby granted, free of charge, to any person obtaining
-# a copy of this software and associated documentation files (the
-# "Software"), to deal in the Software without restriction, including
-# without limitation the rights to use, copy, modify, merge, publish,
-# distribute, sublicense, and/or sell copies of the Software, and to
-# permit persons to whom the Software is furnished to do so, subject to
-# the following conditions:
-#
-# The above copyright notice and this permission notice shall be
-# included in all copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
-# LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
-# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
-# WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
-#
-# Country of origin: Canada
-#
-###########################################################################
-# Sample PBKDF2 usage:
-#   from Crypto.Cipher import AES
-#   from pbkdf2 import PBKDF2
-#   import os
-#
-#   salt = os.urandom(8)    # 64-bit salt
-#   key = PBKDF2("This passphrase is a secret.", salt).read(32) # 256-bit key
-#   iv = os.urandom(16)     # 128-bit IV
-#   cipher = AES.new(key, AES.MODE_CBC, iv)
-#     ...
-#
-# Sample crypt() usage:
-#   from pbkdf2 import crypt
-#   pwhash = crypt("secret")
-#   alleged_pw = raw_input("Enter password: ")
-#   if pwhash == crypt(alleged_pw, pwhash):
-#       print "Password good"
-#   else:
-#       print "Invalid password"
-#
-###########################################################################
-
-__version__ = "1.3"
-__all__ = ['PBKDF2', 'crypt']
-
-from struct import pack
-from random import randint
-import string
-import sys
-
-try:
-    # Use PyCrypto (if available).
-    from Crypto.Hash import HMAC, SHA as SHA1
-except ImportError:
-    # PyCrypto not available.  Use the Python standard library.
-    import hmac as HMAC
-    try:
-        from hashlib import sha1 as SHA1
-    except ImportError:
-        # hashlib not available.  Use the old sha module.
-        import sha as SHA1
-
-#
-# Python 2.1 thru 3.2 compatibility
-#
-
-if sys.version_info[0] == 2:
-    _0xffffffffL = long(1) << 32
-    def isunicode(s):
-        return isinstance(s, unicode)
-    def isbytes(s):
-        return isinstance(s, str)
-    def isinteger(n):
-        return isinstance(n, (int, long))
-    def b(s):
-        return s
-    def binxor(a, b):
-        return "".join([chr(ord(x) ^ ord(y)) for (x, y) in zip(a, b)])
-    def b64encode(data, chars="+/"):
-        tt = string.maketrans("+/", chars)
-        return data.encode('base64').replace("\n", "").translate(tt)
-    from binascii import b2a_hex
-else:
-    _0xffffffffL = 0xffffffff
-    def isunicode(s):
-        return isinstance(s, str)
-    def isbytes(s):
-        return isinstance(s, bytes)
-    def isinteger(n):
-        return isinstance(n, int)
-    def callable(obj):
-        return hasattr(obj, '__call__')
-    def b(s):
-       return s.encode("latin-1")
-    def binxor(a, b):
-        return bytes([x ^ y for (x, y) in zip(a, b)])
-    from base64 import b64encode as _b64encode
-    def b64encode(data, chars="+/"):
-        if isunicode(chars):
-            return _b64encode(data, chars.encode('utf-8')).decode('utf-8')
-        else:
-            return _b64encode(data, chars)
-    from binascii import b2a_hex as _b2a_hex
-    def b2a_hex(s):
-        return _b2a_hex(s).decode('us-ascii')
-    xrange = range
-
-class PBKDF2(object):
-    """PBKDF2.py : PKCS#5 v2.0 Password-Based Key Derivation
-
-    This implementation takes a passphrase and a salt (and optionally an
-    iteration count, a digest module, and a MAC module) and provides a
-    file-like object from which an arbitrarily-sized key can be read.
-
-    If the passphrase and/or salt are unicode objects, they are encoded as
-    UTF-8 before they are processed.
-
-    The idea behind PBKDF2 is to derive a cryptographic key from a
-    passphrase and a salt.
-
-    PBKDF2 may also be used as a strong salted password hash.  The
-    'crypt' function is provided for that purpose.
-
-    Remember: Keys generated using PBKDF2 are only as strong as the
-    passphrases they are derived from.
-    """
-
-    def __init__(self, passphrase, salt, iterations=1000,
-                 digestmodule=SHA1, macmodule=HMAC):
-        self.__macmodule = macmodule
-        self.__digestmodule = digestmodule
-        self._setup(passphrase, salt, iterations, self._pseudorandom)
-
-    def _pseudorandom(self, key, msg):
-        """Pseudorandom function.  e.g. HMAC-SHA1"""
-        return self.__macmodule.new(key=key, msg=msg,
-            digestmod=self.__digestmodule).digest()
-
-    def read(self, bytes):
-        """Read the specified number of key bytes."""
-        if self.closed:
-            raise ValueError("file-like object is closed")
-
-        size = len(self.__buf)
-        blocks = [self.__buf]
-        i = self.__blockNum
-        while size < bytes:
-            i += 1
-            if i > _0xffffffffL or i < 1:
-                # We could return "" here, but
-                raise OverflowError("derived key too long")
-            block = self.__f(i)
-            blocks.append(block)
-            size += len(block)
-        buf = b("").join(blocks)
-        retval = buf[:bytes]
-        self.__buf = buf[bytes:]
-        self.__blockNum = i
-        return retval
-
-    def __f(self, i):
-        # i must fit within 32 bits
-        assert 1 <= i <= _0xffffffffL
-        U = self.__prf(self.__passphrase, self.__salt + pack("!L", i))
-        result = U
-        for j in xrange(2, 1+self.__iterations):
-            U = self.__prf(self.__passphrase, U)
-            result = binxor(result, U)
-        return result
-
-    def hexread(self, octets):
-        """Read the specified number of octets. Return them as hexadecimal.
-
-        Note that len(obj.hexread(n)) == 2*n.
-        """
-        return b2a_hex(self.read(octets))
-
-    def _setup(self, passphrase, salt, iterations, prf):
-        # Sanity checks:
-
-        # passphrase and salt must be str or unicode (in the latter
-        # case, we convert to UTF-8)
-        if isunicode(passphrase):
-            passphrase = passphrase.encode("UTF-8")
-        elif not isbytes(passphrase):
-            raise TypeError("passphrase must be str or unicode")
-        if isunicode(salt):
-            salt = salt.encode("UTF-8")
-        elif not isbytes(salt):
-            raise TypeError("salt must be str or unicode")
-
-        # iterations must be an integer >= 1
-        if not isinteger(iterations):
-            raise TypeError("iterations must be an integer")
-        if iterations < 1:
-            raise ValueError("iterations must be at least 1")
-
-        # prf must be callable
-        if not callable(prf):
-            raise TypeError("prf must be callable")
-
-        self.__passphrase = passphrase
-        self.__salt = salt
-        self.__iterations = iterations
-        self.__prf = prf
-        self.__blockNum = 0
-        self.__buf = b("")
-        self.closed = False
-
-    def close(self):
-        """Close the stream."""
-        if not self.closed:
-            del self.__passphrase
-            del self.__salt
-            del self.__iterations
-            del self.__prf
-            del self.__blockNum
-            del self.__buf
-            self.closed = True
-
-def crypt(word, salt=None, iterations=None):
-    """PBKDF2-based unix crypt(3) replacement.
-
-    The number of iterations specified in the salt overrides the 'iterations'
-    parameter.
-
-    The effective hash length is 192 bits.
-    """
-
-    # Generate a (pseudo-)random salt if the user hasn't provided one.
-    if salt is None:
-        salt = _makesalt()
-
-    # salt must be a string or the us-ascii subset of unicode
-    if isunicode(salt):
-        salt = salt.encode('us-ascii').decode('us-ascii')
-    elif isbytes(salt):
-        salt = salt.decode('us-ascii')
-    else:
-        raise TypeError("salt must be a string")
-
-    # word must be a string or unicode (in the latter case, we convert to UTF-8)
-    if isunicode(word):
-        word = word.encode("UTF-8")
-    elif not isbytes(word):
-        raise TypeError("word must be a string or unicode")
-
-    # Try to extract the real salt and iteration count from the salt
-    if salt.startswith("$p5k2$"):
-        (iterations, salt, dummy) = salt.split("$")[2:5]
-        if iterations == "":
-            iterations = 400
-        else:
-            converted = int(iterations, 16)
-            if iterations != "%x" % converted:  # lowercase hex, minimum digits
-                raise ValueError("Invalid salt")
-            iterations = converted
-            if not (iterations >= 1):
-                raise ValueError("Invalid salt")
-
-    # Make sure the salt matches the allowed character set
-    allowed = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789./"
-    for ch in salt:
-        if ch not in allowed:
-            raise ValueError("Illegal character %r in salt" % (ch,))
-
-    if iterations is None or iterations == 400:
-        iterations = 400
-        salt = "$p5k2$$" + salt
-    else:
-        salt = "$p5k2$%x$%s" % (iterations, salt)
-    rawhash = PBKDF2(word, salt, iterations).read(24)
-    return salt + "$" + b64encode(rawhash, "./")
-
-# Add crypt as a static method of the PBKDF2 class
-# This makes it easier to do "from PBKDF2 import PBKDF2" and still use
-# crypt.
-PBKDF2.crypt = staticmethod(crypt)
-
-def _makesalt():
-    """Return a 48-bit pseudorandom salt for crypt().
-
-    This function is not suitable for generating cryptographic secrets.
-    """
-    binarysalt = b("").join([pack("@H", randint(0, 0xffff)) for i in range(3)])
-    return b64encode(binarysalt, "./")
-
-# vim:set ts=4 sw=4 sts=4 expandtab:

Contents/Libraries/Shared/provider_test.sh

@@ -43,8 +43,6 @@ python -c "import logging; logging.basicConfig(level=logging.DEBUG); logging.get
 # subscenter:list
 python -c "import logging; logging.basicConfig(level=logging.DEBUG); logging.getLogger('rebulk').setLevel(logging.WARNING); import subliminal_patch, subliminal; subliminal.region.configure('dogpile.cache.memory'); from subliminal_patch.core import SZProviderPool; from babelfish import Language; from subliminal.core import scan_video; print SZProviderPool(providers=['subscenter'], )['subscenter'].list_subtitles(scan_video('FULL_PATH'), languages=[Language('heb')])"
 
-# subscene:list
-python -c "import logging; logging.basicConfig(level=logging.DEBUG); logging.getLogger('rebulk').setLevel(logging.WARNING); import subliminal_patch, subliminal; subliminal.region.configure('dogpile.cache.memory'); from subliminal_patch.core import SZProviderPool; from subzero.language import Language; from subzero.video import parse_video; SZProviderPool(providers=['subscene'], provider_configs={'subscene': {'username': 'USERNAME', 'password': 'PASSWORD'}})['subscene'].list_subtitles(parse_video('FILENAME', {}, {'type': 'episode'}, dry_run=True), languages=[Language('eng')])"
 
 # refining
 python -c "import logging; logging.basicConfig(level=logging.DEBUG); logging.getLogger('rebulk').setLevel(logging.WARNING); import os; os.environ['U1pfT01EQl9LRVk'] = '789CF30DAC2C8B0AF433F5C9AD34290A712DF30D7135F12D0FB3E502006FDE081E'; import subliminal_patch, subliminal; subliminal.region.configure('dogpile.cache.memory'); from subzero.video import parse_video, refine_video; video = parse_video('FILE_NAME', {'type': 'episode'}, dry_run=True); print refine_video(video)"

Contents/Libraries/Shared/pyaes/__init__.py

@@ -1,53 +0,0 @@
-# The MIT License (MIT)
-#
-# Copyright (c) 2014 Richard Moore
-#
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is
-# furnished to do so, subject to the following conditions:
-#
-# The above copyright notice and this permission notice shall be included in
-# all copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
-# THE SOFTWARE.
-
-# This is a pure-Python implementation of the AES algorithm and AES common
-# modes of operation.
-
-# See: https://en.wikipedia.org/wiki/Advanced_Encryption_Standard
-# See: https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation
-
-
-# Supported key sizes:
-#   128-bit
-#   192-bit
-#   256-bit
-
-
-# Supported modes of operation:
-#   ECB - Electronic Codebook
-#   CBC - Cipher-Block Chaining
-#   CFB - Cipher Feedback
-#   OFB - Output Feedback
-#   CTR - Counter
-
-# See the README.md for API details and general information.
-
-# Also useful, PyCrypto, a crypto library implemented in C with Python bindings:
-# https://www.dlitz.net/software/pycrypto/
-
-
-VERSION = [1, 3, 0]
-
-from .aes import AES, AESModeOfOperationCTR, AESModeOfOperationCBC, AESModeOfOperationCFB, AESModeOfOperationECB, AESModeOfOperationOFB, AESModesOfOperation, Counter
-from .blockfeeder import decrypt_stream, Decrypter, encrypt_stream, Encrypter
-from .blockfeeder import PADDING_NONE, PADDING_DEFAULT

Contents/Libraries/Shared/pyaes/aes.py

@@ -1,589 +0,0 @@
-# The MIT License (MIT)
-#
-# Copyright (c) 2014 Richard Moore
-#
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is
-# furnished to do so, subject to the following conditions:
-#
-# The above copyright notice and this permission notice shall be included in
-# all copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
-# THE SOFTWARE.
-
-# This is a pure-Python implementation of the AES algorithm and AES common
-# modes of operation.
-
-# See: https://en.wikipedia.org/wiki/Advanced_Encryption_Standard
-
-# Honestly, the best description of the modes of operations are the wonderful
-# diagrams on Wikipedia. They explain in moments what my words could never
-# achieve. Hence the inline documentation here is sparer than I'd prefer.
-# See: https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation
-
-# Also useful, PyCrypto, a crypto library implemented in C with Python bindings:
-# https://www.dlitz.net/software/pycrypto/
-
-
-# Supported key sizes:
-#   128-bit
-#   192-bit
-#   256-bit
-
-
-# Supported modes of operation:
-#   ECB - Electronic Codebook
-#   CBC - Cipher-Block Chaining
-#   CFB - Cipher Feedback
-#   OFB - Output Feedback
-#   CTR - Counter
-
-
-# See the README.md for API details and general information.
-
-
-import copy
-import struct
-
-__all__ = ["AES", "AESModeOfOperationCTR", "AESModeOfOperationCBC", "AESModeOfOperationCFB",
-           "AESModeOfOperationECB", "AESModeOfOperationOFB", "AESModesOfOperation", "Counter"]
-
-
-def _compact_word(word):
-    return (word[0] << 24) | (word[1] << 16) | (word[2] << 8) | word[3]
-
-def _string_to_bytes(text):
-    return list(ord(c) for c in text)
-
-def _bytes_to_string(binary):
-    return "".join(chr(b) for b in binary)
-
-def _concat_list(a, b):
-    return a + b
-
-
-# Python 3 compatibility
-try:
-    xrange
-except Exception:
-    xrange = range
-
-    # Python 3 supports bytes, which is already an array of integers
-    def _string_to_bytes(text):
-        if isinstance(text, bytes):
-            return text
-        return [ord(c) for c in text]
-
-    # In Python 3, we return bytes
-    def _bytes_to_string(binary):
-        return bytes(binary)
-
-    # Python 3 cannot concatenate a list onto a bytes, so we bytes-ify it first
-    def _concat_list(a, b):
-        return a + bytes(b)
-
-
-# Based *largely* on the Rijndael implementation
-# See: http://csrc.nist.gov/publications/fips/fips197/fips-197.pdf
-class AES(object):
-    '''Encapsulates the AES block cipher.
-
-    You generally should not need this. Use the AESModeOfOperation classes
-    below instead.'''
-
-    # Number of rounds by keysize
-    number_of_rounds = {16: 10, 24: 12, 32: 14}
-
-    # Round constant words
-    rcon = [ 0x01, 0x02, 0x04, 0x08, 0x10, 0x20, 0x40, 0x80, 0x1b, 0x36, 0x6c, 0xd8, 0xab, 0x4d, 0x9a, 0x2f, 0x5e, 0xbc, 0x63, 0xc6, 0x97, 0x35, 0x6a, 0xd4, 0xb3, 0x7d, 0xfa, 0xef, 0xc5, 0x91 ]
-
-    # S-box and Inverse S-box (S is for Substitution)
-    S = [ 0x63, 0x7c, 0x77, 0x7b, 0xf2, 0x6b, 0x6f, 0xc5, 0x30, 0x01, 0x67, 0x2b, 0xfe, 0xd7, 0xab, 0x76, 0xca, 0x82, 0xc9, 0x7d, 0xfa, 0x59, 0x47, 0xf0, 0xad, 0xd4, 0xa2, 0xaf, 0x9c, 0xa4, 0x72, 0xc0, 0xb7, 0xfd, 0x93, 0x26, 0x36, 0x3f, 0xf7, 0xcc, 0x34, 0xa5, 0xe5, 0xf1, 0x71, 0xd8, 0x31, 0x15, 0x04, 0xc7, 0x23, 0xc3, 0x18, 0x96, 0x05, 0x9a, 0x07, 0x12, 0x80, 0xe2, 0xeb, 0x27, 0xb2, 0x75, 0x09, 0x83, 0x2c, 0x1a, 0x1b, 0x6e, 0x5a, 0xa0, 0x52, 0x3b, 0xd6, 0xb3, 0x29, 0xe3, 0x2f, 0x84, 0x53, 0xd1, 0x00, 0xed, 0x20, 0xfc, 0xb1, 0x5b, 0x6a, 0xcb, 0xbe, 0x39, 0x4a, 0x4c, 0x58, 0xcf, 0xd0, 0xef, 0xaa, 0xfb, 0x43, 0x4d, 0x33, 0x85, 0x45, 0xf9, 0x02, 0x7f, 0x50, 0x3c, 0x9f, 0xa8, 0x51, 0xa3, 0x40, 0x8f, 0x92, 0x9d, 0x38, 0xf5, 0xbc, 0xb6, 0xda, 0x21, 0x10, 0xff, 0xf3, 0xd2, 0xcd, 0x0c, 0x13, 0xec, 0x5f, 0x97, 0x44, 0x17, 0xc4, 0xa7, 0x7e, 0x3d, 0x64, 0x5d, 0x19, 0x73, 0x60, 0x81, 0x4f, 0xdc, 0x22, 0x2a, 0x90, 0x88, 0x46, 0xee, 0xb8, 0x14, 0xde, 0x5e, 0x0b, 0xdb, 0xe0, 0x32, 0x3a, 0x0a, 0x49, 0x06, 0x24, 0x5c, 0xc2, 0xd3, 0xac, 0x62, 0x91, 0x95, 0xe4, 0x79, 0xe7, 0xc8, 0x37, 0x6d, 0x8d, 0xd5, 0x4e, 0xa9, 0x6c, 0x56, 0xf4, 0xea, 0x65, 0x7a, 0xae, 0x08, 0xba, 0x78, 0x25, 0x2e, 0x1c, 0xa6, 0xb4, 0xc6, 0xe8, 0xdd, 0x74, 0x1f, 0x4b, 0xbd, 0x8b, 0x8a, 0x70, 0x3e, 0xb5, 0x66, 0x48, 0x03, 0xf6, 0x0e, 0x61, 0x35, 0x57, 0xb9, 0x86, 0xc1, 0x1d, 0x9e, 0xe1, 0xf8, 0x98, 0x11, 0x69, 0xd9, 0x8e, 0x94, 0x9b, 0x1e, 0x87, 0xe9, 0xce, 0x55, 0x28, 0xdf, 0x8c, 0xa1, 0x89, 0x0d, 0xbf, 0xe6, 0x42, 0x68, 0x41, 0x99, 0x2d, 0x0f, 0xb0, 0x54, 0xbb, 0x16 ]
-    Si =[ 0x52, 0x09, 0x6a, 0xd5, 0x30, 0x36, 0xa5, 0x38, 0xbf, 0x40, 0xa3, 0x9e, 0x81, 0xf3, 0xd7, 0xfb, 0x7c, 0xe3, 0x39, 0x82, 0x9b, 0x2f, 0xff, 0x87, 0x34, 0x8e, 0x43, 0x44, 0xc4, 0xde, 0xe9, 0xcb, 0x54, 0x7b, 0x94, 0x32, 0xa6, 0xc2, 0x23, 0x3d, 0xee, 0x4c, 0x95, 0x0b, 0x42, 0xfa, 0xc3, 0x4e, 0x08, 0x2e, 0xa1, 0x66, 0x28, 0xd9, 0x24, 0xb2, 0x76, 0x5b, 0xa2, 0x49, 0x6d, 0x8b, 0xd1, 0x25, 0x72, 0xf8, 0xf6, 0x64, 0x86, 0x68, 0x98, 0x16, 0xd4, 0xa4, 0x5c, 0xcc, 0x5d, 0x65, 0xb6, 0x92, 0x6c, 0x70, 0x48, 0x50, 0xfd, 0xed, 0xb9, 0xda, 0x5e, 0x15, 0x46, 0x57, 0xa7, 0x8d, 0x9d, 0x84, 0x90, 0xd8, 0xab, 0x00, 0x8c, 0xbc, 0xd3, 0x0a, 0xf7, 0xe4, 0x58, 0x05, 0xb8, 0xb3, 0x45, 0x06, 0xd0, 0x2c, 0x1e, 0x8f, 0xca, 0x3f, 0x0f, 0x02, 0xc1, 0xaf, 0xbd, 0x03, 0x01, 0x13, 0x8a, 0x6b, 0x3a, 0x91, 0x11, 0x41, 0x4f, 0x67, 0xdc, 0xea, 0x97, 0xf2, 0xcf, 0xce, 0xf0, 0xb4, 0xe6, 0x73, 0x96, 0xac, 0x74, 0x22, 0xe7, 0xad, 0x35, 0x85, 0xe2, 0xf9, 0x37, 0xe8, 0x1c, 0x75, 0xdf, 0x6e, 0x47, 0xf1, 0x1a, 0x71, 0x1d, 0x29, 0xc5, 0x89, 0x6f, 0xb7, 0x62, 0x0e, 0xaa, 0x18, 0xbe, 0x1b, 0xfc, 0x56, 0x3e, 0x4b, 0xc6, 0xd2, 0x79, 0x20, 0x9a, 0xdb, 0xc0, 0xfe, 0x78, 0xcd, 0x5a, 0xf4, 0x1f, 0xdd, 0xa8, 0x33, 0x88, 0x07, 0xc7, 0x31, 0xb1, 0x12, 0x10, 0x59, 0x27, 0x80, 0xec, 0x5f, 0x60, 0x51, 0x7f, 0xa9, 0x19, 0xb5, 0x4a, 0x0d, 0x2d, 0xe5, 0x7a, 0x9f, 0x93, 0xc9, 0x9c, 0xef, 0xa0, 0xe0, 0x3b, 0x4d, 0xae, 0x2a, 0xf5, 0xb0, 0xc8, 0xeb, 0xbb, 0x3c, 0x83, 0x53, 0x99, 0x61, 0x17, 0x2b, 0x04, 0x7e, 0xba, 0x77, 0xd6, 0x26, 0xe1, 0x69, 0x14, 0x63, 0x55, 0x21, 0x0c, 0x7d ] 
-
-    # Transformations for encryption
-    T1 = [ 0xc66363a5, 0xf87c7c84, 0xee777799, 0xf67b7b8d, 0xfff2f20d, 0xd66b6bbd, 0xde6f6fb1, 0x91c5c554, 0x60303050, 0x02010103, 0xce6767a9, 0x562b2b7d, 0xe7fefe19, 0xb5d7d762, 0x4dababe6, 0xec76769a, 0x8fcaca45, 0x1f82829d, 0x89c9c940, 0xfa7d7d87, 0xeffafa15, 0xb25959eb, 0x8e4747c9, 0xfbf0f00b, 0x41adadec, 0xb3d4d467, 0x5fa2a2fd, 0x45afafea, 0x239c9cbf, 0x53a4a4f7, 0xe4727296, 0x9bc0c05b, 0x75b7b7c2, 0xe1fdfd1c, 0x3d9393ae, 0x4c26266a, 0x6c36365a, 0x7e3f3f41, 0xf5f7f702, 0x83cccc4f, 0x6834345c, 0x51a5a5f4, 0xd1e5e534, 0xf9f1f108, 0xe2717193, 0xabd8d873, 0x62313153, 0x2a15153f, 0x0804040c, 0x95c7c752, 0x46232365, 0x9dc3c35e, 0x30181828, 0x379696a1, 0x0a05050f, 0x2f9a9ab5, 0x0e070709, 0x24121236, 0x1b80809b, 0xdfe2e23d, 0xcdebeb26, 0x4e272769, 0x7fb2b2cd, 0xea75759f, 0x1209091b, 0x1d83839e, 0x582c2c74, 0x341a1a2e, 0x361b1b2d, 0xdc6e6eb2, 0xb45a5aee, 0x5ba0a0fb, 0xa45252f6, 0x763b3b4d, 0xb7d6d661, 0x7db3b3ce, 0x5229297b, 0xdde3e33e, 0x5e2f2f71, 0x13848497, 0xa65353f5, 0xb9d1d168, 0x00000000, 0xc1eded2c, 0x40202060, 0xe3fcfc1f, 0x79b1b1c8, 0xb65b5bed, 0xd46a6abe, 0x8dcbcb46, 0x67bebed9, 0x7239394b, 0x944a4ade, 0x984c4cd4, 0xb05858e8, 0x85cfcf4a, 0xbbd0d06b, 0xc5efef2a, 0x4faaaae5, 0xedfbfb16, 0x864343c5, 0x9a4d4dd7, 0x66333355, 0x11858594, 0x8a4545cf, 0xe9f9f910, 0x04020206, 0xfe7f7f81, 0xa05050f0, 0x783c3c44, 0x259f9fba, 0x4ba8a8e3, 0xa25151f3, 0x5da3a3fe, 0x804040c0, 0x058f8f8a, 0x3f9292ad, 0x219d9dbc, 0x70383848, 0xf1f5f504, 0x63bcbcdf, 0x77b6b6c1, 0xafdada75, 0x42212163, 0x20101030, 0xe5ffff1a, 0xfdf3f30e, 0xbfd2d26d, 0x81cdcd4c, 0x180c0c14, 0x26131335, 0xc3ecec2f, 0xbe5f5fe1, 0x359797a2, 0x884444cc, 0x2e171739, 0x93c4c457, 0x55a7a7f2, 0xfc7e7e82, 0x7a3d3d47, 0xc86464ac, 0xba5d5de7, 0x3219192b, 0xe6737395, 0xc06060a0, 0x19818198, 0x9e4f4fd1, 0xa3dcdc7f, 0x44222266, 0x542a2a7e, 0x3b9090ab, 0x0b888883, 0x8c4646ca, 0xc7eeee29, 0x6bb8b8d3, 0x2814143c, 0xa7dede79, 0xbc5e5ee2, 0x160b0b1d, 0xaddbdb76, 0xdbe0e03b, 0x64323256, 0x743a3a4e, 0x140a0a1e, 0x924949db, 0x0c06060a, 0x4824246c, 0xb85c5ce4, 0x9fc2c25d, 0xbdd3d36e, 0x43acacef, 0xc46262a6, 0x399191a8, 0x319595a4, 0xd3e4e437, 0xf279798b, 0xd5e7e732, 0x8bc8c843, 0x6e373759, 0xda6d6db7, 0x018d8d8c, 0xb1d5d564, 0x9c4e4ed2, 0x49a9a9e0, 0xd86c6cb4, 0xac5656fa, 0xf3f4f407, 0xcfeaea25, 0xca6565af, 0xf47a7a8e, 0x47aeaee9, 0x10080818, 0x6fbabad5, 0xf0787888, 0x4a25256f, 0x5c2e2e72, 0x381c1c24, 0x57a6a6f1, 0x73b4b4c7, 0x97c6c651, 0xcbe8e823, 0xa1dddd7c, 0xe874749c, 0x3e1f1f21, 0x964b4bdd, 0x61bdbddc, 0x0d8b8b86, 0x0f8a8a85, 0xe0707090, 0x7c3e3e42, 0x71b5b5c4, 0xcc6666aa, 0x904848d8, 0x06030305, 0xf7f6f601, 0x1c0e0e12, 0xc26161a3, 0x6a35355f, 0xae5757f9, 0x69b9b9d0, 0x17868691, 0x99c1c158, 0x3a1d1d27, 0x279e9eb9, 0xd9e1e138, 0xebf8f813, 0x2b9898b3, 0x22111133, 0xd26969bb, 0xa9d9d970, 0x078e8e89, 0x339494a7, 0x2d9b9bb6, 0x3c1e1e22, 0x15878792, 0xc9e9e920, 0x87cece49, 0xaa5555ff, 0x50282878, 0xa5dfdf7a, 0x038c8c8f, 0x59a1a1f8, 0x09898980, 0x1a0d0d17, 0x65bfbfda, 0xd7e6e631, 0x844242c6, 0xd06868b8, 0x824141c3, 0x299999b0, 0x5a2d2d77, 0x1e0f0f11, 0x7bb0b0cb, 0xa85454fc, 0x6dbbbbd6, 0x2c16163a ]
-    T2 = [ 0xa5c66363, 0x84f87c7c, 0x99ee7777, 0x8df67b7b, 0x0dfff2f2, 0xbdd66b6b, 0xb1de6f6f, 0x5491c5c5, 0x50603030, 0x03020101, 0xa9ce6767, 0x7d562b2b, 0x19e7fefe, 0x62b5d7d7, 0xe64dabab, 0x9aec7676, 0x458fcaca, 0x9d1f8282, 0x4089c9c9, 0x87fa7d7d, 0x15effafa, 0xebb25959, 0xc98e4747, 0x0bfbf0f0, 0xec41adad, 0x67b3d4d4, 0xfd5fa2a2, 0xea45afaf, 0xbf239c9c, 0xf753a4a4, 0x96e47272, 0x5b9bc0c0, 0xc275b7b7, 0x1ce1fdfd, 0xae3d9393, 0x6a4c2626, 0x5a6c3636, 0x417e3f3f, 0x02f5f7f7, 0x4f83cccc, 0x5c683434, 0xf451a5a5, 0x34d1e5e5, 0x08f9f1f1, 0x93e27171, 0x73abd8d8, 0x53623131, 0x3f2a1515, 0x0c080404, 0x5295c7c7, 0x65462323, 0x5e9dc3c3, 0x28301818, 0xa1379696, 0x0f0a0505, 0xb52f9a9a, 0x090e0707, 0x36241212, 0x9b1b8080, 0x3ddfe2e2, 0x26cdebeb, 0x694e2727, 0xcd7fb2b2, 0x9fea7575, 0x1b120909, 0x9e1d8383, 0x74582c2c, 0x2e341a1a, 0x2d361b1b, 0xb2dc6e6e, 0xeeb45a5a, 0xfb5ba0a0, 0xf6a45252, 0x4d763b3b, 0x61b7d6d6, 0xce7db3b3, 0x7b522929, 0x3edde3e3, 0x715e2f2f, 0x97138484, 0xf5a65353, 0x68b9d1d1, 0x00000000, 0x2cc1eded, 0x60402020, 0x1fe3fcfc, 0xc879b1b1, 0xedb65b5b, 0xbed46a6a, 0x468dcbcb, 0xd967bebe, 0x4b723939, 0xde944a4a, 0xd4984c4c, 0xe8b05858, 0x4a85cfcf, 0x6bbbd0d0, 0x2ac5efef, 0xe54faaaa, 0x16edfbfb, 0xc5864343, 0xd79a4d4d, 0x55663333, 0x94118585, 0xcf8a4545, 0x10e9f9f9, 0x06040202, 0x81fe7f7f, 0xf0a05050, 0x44783c3c, 0xba259f9f, 0xe34ba8a8, 0xf3a25151, 0xfe5da3a3, 0xc0804040, 0x8a058f8f, 0xad3f9292, 0xbc219d9d, 0x48703838, 0x04f1f5f5, 0xdf63bcbc, 0xc177b6b6, 0x75afdada, 0x63422121, 0x30201010, 0x1ae5ffff, 0x0efdf3f3, 0x6dbfd2d2, 0x4c81cdcd, 0x14180c0c, 0x35261313, 0x2fc3ecec, 0xe1be5f5f, 0xa2359797, 0xcc884444, 0x392e1717, 0x5793c4c4, 0xf255a7a7, 0x82fc7e7e, 0x477a3d3d, 0xacc86464, 0xe7ba5d5d, 0x2b321919, 0x95e67373, 0xa0c06060, 0x98198181, 0xd19e4f4f, 0x7fa3dcdc, 0x66442222, 0x7e542a2a, 0xab3b9090, 0x830b8888, 0xca8c4646, 0x29c7eeee, 0xd36bb8b8, 0x3c281414, 0x79a7dede, 0xe2bc5e5e, 0x1d160b0b, 0x76addbdb, 0x3bdbe0e0, 0x56643232, 0x4e743a3a, 0x1e140a0a, 0xdb924949, 0x0a0c0606, 0x6c482424, 0xe4b85c5c, 0x5d9fc2c2, 0x6ebdd3d3, 0xef43acac, 0xa6c46262, 0xa8399191, 0xa4319595, 0x37d3e4e4, 0x8bf27979, 0x32d5e7e7, 0x438bc8c8, 0x596e3737, 0xb7da6d6d, 0x8c018d8d, 0x64b1d5d5, 0xd29c4e4e, 0xe049a9a9, 0xb4d86c6c, 0xfaac5656, 0x07f3f4f4, 0x25cfeaea, 0xafca6565, 0x8ef47a7a, 0xe947aeae, 0x18100808, 0xd56fbaba, 0x88f07878, 0x6f4a2525, 0x725c2e2e, 0x24381c1c, 0xf157a6a6, 0xc773b4b4, 0x5197c6c6, 0x23cbe8e8, 0x7ca1dddd, 0x9ce87474, 0x213e1f1f, 0xdd964b4b, 0xdc61bdbd, 0x860d8b8b, 0x850f8a8a, 0x90e07070, 0x427c3e3e, 0xc471b5b5, 0xaacc6666, 0xd8904848, 0x05060303, 0x01f7f6f6, 0x121c0e0e, 0xa3c26161, 0x5f6a3535, 0xf9ae5757, 0xd069b9b9, 0x91178686, 0x5899c1c1, 0x273a1d1d, 0xb9279e9e, 0x38d9e1e1, 0x13ebf8f8, 0xb32b9898, 0x33221111, 0xbbd26969, 0x70a9d9d9, 0x89078e8e, 0xa7339494, 0xb62d9b9b, 0x223c1e1e, 0x92158787, 0x20c9e9e9, 0x4987cece, 0xffaa5555, 0x78502828, 0x7aa5dfdf, 0x8f038c8c, 0xf859a1a1, 0x80098989, 0x171a0d0d, 0xda65bfbf, 0x31d7e6e6, 0xc6844242, 0xb8d06868, 0xc3824141, 0xb0299999, 0x775a2d2d, 0x111e0f0f, 0xcb7bb0b0, 0xfca85454, 0xd66dbbbb, 0x3a2c1616 ]
-    T3 = [ 0x63a5c663, 0x7c84f87c, 0x7799ee77, 0x7b8df67b, 0xf20dfff2, 0x6bbdd66b, 0x6fb1de6f, 0xc55491c5, 0x30506030, 0x01030201, 0x67a9ce67, 0x2b7d562b, 0xfe19e7fe, 0xd762b5d7, 0xabe64dab, 0x769aec76, 0xca458fca, 0x829d1f82, 0xc94089c9, 0x7d87fa7d, 0xfa15effa, 0x59ebb259, 0x47c98e47, 0xf00bfbf0, 0xadec41ad, 0xd467b3d4, 0xa2fd5fa2, 0xafea45af, 0x9cbf239c, 0xa4f753a4, 0x7296e472, 0xc05b9bc0, 0xb7c275b7, 0xfd1ce1fd, 0x93ae3d93, 0x266a4c26, 0x365a6c36, 0x3f417e3f, 0xf702f5f7, 0xcc4f83cc, 0x345c6834, 0xa5f451a5, 0xe534d1e5, 0xf108f9f1, 0x7193e271, 0xd873abd8, 0x31536231, 0x153f2a15, 0x040c0804, 0xc75295c7, 0x23654623, 0xc35e9dc3, 0x18283018, 0x96a13796, 0x050f0a05, 0x9ab52f9a, 0x07090e07, 0x12362412, 0x809b1b80, 0xe23ddfe2, 0xeb26cdeb, 0x27694e27, 0xb2cd7fb2, 0x759fea75, 0x091b1209, 0x839e1d83, 0x2c74582c, 0x1a2e341a, 0x1b2d361b, 0x6eb2dc6e, 0x5aeeb45a, 0xa0fb5ba0, 0x52f6a452, 0x3b4d763b, 0xd661b7d6, 0xb3ce7db3, 0x297b5229, 0xe33edde3, 0x2f715e2f, 0x84971384, 0x53f5a653, 0xd168b9d1, 0x00000000, 0xed2cc1ed, 0x20604020, 0xfc1fe3fc, 0xb1c879b1, 0x5bedb65b, 0x6abed46a, 0xcb468dcb, 0xbed967be, 0x394b7239, 0x4ade944a, 0x4cd4984c, 0x58e8b058, 0xcf4a85cf, 0xd06bbbd0, 0xef2ac5ef, 0xaae54faa, 0xfb16edfb, 0x43c58643, 0x4dd79a4d, 0x33556633, 0x85941185, 0x45cf8a45, 0xf910e9f9, 0x02060402, 0x7f81fe7f, 0x50f0a050, 0x3c44783c, 0x9fba259f, 0xa8e34ba8, 0x51f3a251, 0xa3fe5da3, 0x40c08040, 0x8f8a058f, 0x92ad3f92, 0x9dbc219d, 0x38487038, 0xf504f1f5, 0xbcdf63bc, 0xb6c177b6, 0xda75afda, 0x21634221, 0x10302010, 0xff1ae5ff, 0xf30efdf3, 0xd26dbfd2, 0xcd4c81cd, 0x0c14180c, 0x13352613, 0xec2fc3ec, 0x5fe1be5f, 0x97a23597, 0x44cc8844, 0x17392e17, 0xc45793c4, 0xa7f255a7, 0x7e82fc7e, 0x3d477a3d, 0x64acc864, 0x5de7ba5d, 0x192b3219, 0x7395e673, 0x60a0c060, 0x81981981, 0x4fd19e4f, 0xdc7fa3dc, 0x22664422, 0x2a7e542a, 0x90ab3b90, 0x88830b88, 0x46ca8c46, 0xee29c7ee, 0xb8d36bb8, 0x143c2814, 0xde79a7de, 0x5ee2bc5e, 0x0b1d160b, 0xdb76addb, 0xe03bdbe0, 0x32566432, 0x3a4e743a, 0x0a1e140a, 0x49db9249, 0x060a0c06, 0x246c4824, 0x5ce4b85c, 0xc25d9fc2, 0xd36ebdd3, 0xacef43ac, 0x62a6c462, 0x91a83991, 0x95a43195, 0xe437d3e4, 0x798bf279, 0xe732d5e7, 0xc8438bc8, 0x37596e37, 0x6db7da6d, 0x8d8c018d, 0xd564b1d5, 0x4ed29c4e, 0xa9e049a9, 0x6cb4d86c, 0x56faac56, 0xf407f3f4, 0xea25cfea, 0x65afca65, 0x7a8ef47a, 0xaee947ae, 0x08181008, 0xbad56fba, 0x7888f078, 0x256f4a25, 0x2e725c2e, 0x1c24381c, 0xa6f157a6, 0xb4c773b4, 0xc65197c6, 0xe823cbe8, 0xdd7ca1dd, 0x749ce874, 0x1f213e1f, 0x4bdd964b, 0xbddc61bd, 0x8b860d8b, 0x8a850f8a, 0x7090e070, 0x3e427c3e, 0xb5c471b5, 0x66aacc66, 0x48d89048, 0x03050603, 0xf601f7f6, 0x0e121c0e, 0x61a3c261, 0x355f6a35, 0x57f9ae57, 0xb9d069b9, 0x86911786, 0xc15899c1, 0x1d273a1d, 0x9eb9279e, 0xe138d9e1, 0xf813ebf8, 0x98b32b98, 0x11332211, 0x69bbd269, 0xd970a9d9, 0x8e89078e, 0x94a73394, 0x9bb62d9b, 0x1e223c1e, 0x87921587, 0xe920c9e9, 0xce4987ce, 0x55ffaa55, 0x28785028, 0xdf7aa5df, 0x8c8f038c, 0xa1f859a1, 0x89800989, 0x0d171a0d, 0xbfda65bf, 0xe631d7e6, 0x42c68442, 0x68b8d068, 0x41c38241, 0x99b02999, 0x2d775a2d, 0x0f111e0f, 0xb0cb7bb0, 0x54fca854, 0xbbd66dbb, 0x163a2c16 ]
-    T4 = [ 0x6363a5c6, 0x7c7c84f8, 0x777799ee, 0x7b7b8df6, 0xf2f20dff, 0x6b6bbdd6, 0x6f6fb1de, 0xc5c55491, 0x30305060, 0x01010302, 0x6767a9ce, 0x2b2b7d56, 0xfefe19e7, 0xd7d762b5, 0xababe64d, 0x76769aec, 0xcaca458f, 0x82829d1f, 0xc9c94089, 0x7d7d87fa, 0xfafa15ef, 0x5959ebb2, 0x4747c98e, 0xf0f00bfb, 0xadadec41, 0xd4d467b3, 0xa2a2fd5f, 0xafafea45, 0x9c9cbf23, 0xa4a4f753, 0x727296e4, 0xc0c05b9b, 0xb7b7c275, 0xfdfd1ce1, 0x9393ae3d, 0x26266a4c, 0x36365a6c, 0x3f3f417e, 0xf7f702f5, 0xcccc4f83, 0x34345c68, 0xa5a5f451, 0xe5e534d1, 0xf1f108f9, 0x717193e2, 0xd8d873ab, 0x31315362, 0x15153f2a, 0x04040c08, 0xc7c75295, 0x23236546, 0xc3c35e9d, 0x18182830, 0x9696a137, 0x05050f0a, 0x9a9ab52f, 0x0707090e, 0x12123624, 0x80809b1b, 0xe2e23ddf, 0xebeb26cd, 0x2727694e, 0xb2b2cd7f, 0x75759fea, 0x09091b12, 0x83839e1d, 0x2c2c7458, 0x1a1a2e34, 0x1b1b2d36, 0x6e6eb2dc, 0x5a5aeeb4, 0xa0a0fb5b, 0x5252f6a4, 0x3b3b4d76, 0xd6d661b7, 0xb3b3ce7d, 0x29297b52, 0xe3e33edd, 0x2f2f715e, 0x84849713, 0x5353f5a6, 0xd1d168b9, 0x00000000, 0xeded2cc1, 0x20206040, 0xfcfc1fe3, 0xb1b1c879, 0x5b5bedb6, 0x6a6abed4, 0xcbcb468d, 0xbebed967, 0x39394b72, 0x4a4ade94, 0x4c4cd498, 0x5858e8b0, 0xcfcf4a85, 0xd0d06bbb, 0xefef2ac5, 0xaaaae54f, 0xfbfb16ed, 0x4343c586, 0x4d4dd79a, 0x33335566, 0x85859411, 0x4545cf8a, 0xf9f910e9, 0x02020604, 0x7f7f81fe, 0x5050f0a0, 0x3c3c4478, 0x9f9fba25, 0xa8a8e34b, 0x5151f3a2, 0xa3a3fe5d, 0x4040c080, 0x8f8f8a05, 0x9292ad3f, 0x9d9dbc21, 0x38384870, 0xf5f504f1, 0xbcbcdf63, 0xb6b6c177, 0xdada75af, 0x21216342, 0x10103020, 0xffff1ae5, 0xf3f30efd, 0xd2d26dbf, 0xcdcd4c81, 0x0c0c1418, 0x13133526, 0xecec2fc3, 0x5f5fe1be, 0x9797a235, 0x4444cc88, 0x1717392e, 0xc4c45793, 0xa7a7f255, 0x7e7e82fc, 0x3d3d477a, 0x6464acc8, 0x5d5de7ba, 0x19192b32, 0x737395e6, 0x6060a0c0, 0x81819819, 0x4f4fd19e, 0xdcdc7fa3, 0x22226644, 0x2a2a7e54, 0x9090ab3b, 0x8888830b, 0x4646ca8c, 0xeeee29c7, 0xb8b8d36b, 0x14143c28, 0xdede79a7, 0x5e5ee2bc, 0x0b0b1d16, 0xdbdb76ad, 0xe0e03bdb, 0x32325664, 0x3a3a4e74, 0x0a0a1e14, 0x4949db92, 0x06060a0c, 0x24246c48, 0x5c5ce4b8, 0xc2c25d9f, 0xd3d36ebd, 0xacacef43, 0x6262a6c4, 0x9191a839, 0x9595a431, 0xe4e437d3, 0x79798bf2, 0xe7e732d5, 0xc8c8438b, 0x3737596e, 0x6d6db7da, 0x8d8d8c01, 0xd5d564b1, 0x4e4ed29c, 0xa9a9e049, 0x6c6cb4d8, 0x5656faac, 0xf4f407f3, 0xeaea25cf, 0x6565afca, 0x7a7a8ef4, 0xaeaee947, 0x08081810, 0xbabad56f, 0x787888f0, 0x25256f4a, 0x2e2e725c, 0x1c1c2438, 0xa6a6f157, 0xb4b4c773, 0xc6c65197, 0xe8e823cb, 0xdddd7ca1, 0x74749ce8, 0x1f1f213e, 0x4b4bdd96, 0xbdbddc61, 0x8b8b860d, 0x8a8a850f, 0x707090e0, 0x3e3e427c, 0xb5b5c471, 0x6666aacc, 0x4848d890, 0x03030506, 0xf6f601f7, 0x0e0e121c, 0x6161a3c2, 0x35355f6a, 0x5757f9ae, 0xb9b9d069, 0x86869117, 0xc1c15899, 0x1d1d273a, 0x9e9eb927, 0xe1e138d9, 0xf8f813eb, 0x9898b32b, 0x11113322, 0x6969bbd2, 0xd9d970a9, 0x8e8e8907, 0x9494a733, 0x9b9bb62d, 0x1e1e223c, 0x87879215, 0xe9e920c9, 0xcece4987, 0x5555ffaa, 0x28287850, 0xdfdf7aa5, 0x8c8c8f03, 0xa1a1f859, 0x89898009, 0x0d0d171a, 0xbfbfda65, 0xe6e631d7, 0x4242c684, 0x6868b8d0, 0x4141c382, 0x9999b029, 0x2d2d775a, 0x0f0f111e, 0xb0b0cb7b, 0x5454fca8, 0xbbbbd66d, 0x16163a2c ]
-
-    # Transformations for decryption
-    T5 = [ 0x51f4a750, 0x7e416553, 0x1a17a4c3, 0x3a275e96, 0x3bab6bcb, 0x1f9d45f1, 0xacfa58ab, 0x4be30393, 0x2030fa55, 0xad766df6, 0x88cc7691, 0xf5024c25, 0x4fe5d7fc, 0xc52acbd7, 0x26354480, 0xb562a38f, 0xdeb15a49, 0x25ba1b67, 0x45ea0e98, 0x5dfec0e1, 0xc32f7502, 0x814cf012, 0x8d4697a3, 0x6bd3f9c6, 0x038f5fe7, 0x15929c95, 0xbf6d7aeb, 0x955259da, 0xd4be832d, 0x587421d3, 0x49e06929, 0x8ec9c844, 0x75c2896a, 0xf48e7978, 0x99583e6b, 0x27b971dd, 0xbee14fb6, 0xf088ad17, 0xc920ac66, 0x7dce3ab4, 0x63df4a18, 0xe51a3182, 0x97513360, 0x62537f45, 0xb16477e0, 0xbb6bae84, 0xfe81a01c, 0xf9082b94, 0x70486858, 0x8f45fd19, 0x94de6c87, 0x527bf8b7, 0xab73d323, 0x724b02e2, 0xe31f8f57, 0x6655ab2a, 0xb2eb2807, 0x2fb5c203, 0x86c57b9a, 0xd33708a5, 0x302887f2, 0x23bfa5b2, 0x02036aba, 0xed16825c, 0x8acf1c2b, 0xa779b492, 0xf307f2f0, 0x4e69e2a1, 0x65daf4cd, 0x0605bed5, 0xd134621f, 0xc4a6fe8a, 0x342e539d, 0xa2f355a0, 0x058ae132, 0xa4f6eb75, 0x0b83ec39, 0x4060efaa, 0x5e719f06, 0xbd6e1051, 0x3e218af9, 0x96dd063d, 0xdd3e05ae, 0x4de6bd46, 0x91548db5, 0x71c45d05, 0x0406d46f, 0x605015ff, 0x1998fb24, 0xd6bde997, 0x894043cc, 0x67d99e77, 0xb0e842bd, 0x07898b88, 0xe7195b38, 0x79c8eedb, 0xa17c0a47, 0x7c420fe9, 0xf8841ec9, 0x00000000, 0x09808683, 0x322bed48, 0x1e1170ac, 0x6c5a724e, 0xfd0efffb, 0x0f853856, 0x3daed51e, 0x362d3927, 0x0a0fd964, 0x685ca621, 0x9b5b54d1, 0x24362e3a, 0x0c0a67b1, 0x9357e70f, 0xb4ee96d2, 0x1b9b919e, 0x80c0c54f, 0x61dc20a2, 0x5a774b69, 0x1c121a16, 0xe293ba0a, 0xc0a02ae5, 0x3c22e043, 0x121b171d, 0x0e090d0b, 0xf28bc7ad, 0x2db6a8b9, 0x141ea9c8, 0x57f11985, 0xaf75074c, 0xee99ddbb, 0xa37f60fd, 0xf701269f, 0x5c72f5bc, 0x44663bc5, 0x5bfb7e34, 0x8b432976, 0xcb23c6dc, 0xb6edfc68, 0xb8e4f163, 0xd731dcca, 0x42638510, 0x13972240, 0x84c61120, 0x854a247d, 0xd2bb3df8, 0xaef93211, 0xc729a16d, 0x1d9e2f4b, 0xdcb230f3, 0x0d8652ec, 0x77c1e3d0, 0x2bb3166c, 0xa970b999, 0x119448fa, 0x47e96422, 0xa8fc8cc4, 0xa0f03f1a, 0x567d2cd8, 0x223390ef, 0x87494ec7, 0xd938d1c1, 0x8ccaa2fe, 0x98d40b36, 0xa6f581cf, 0xa57ade28, 0xdab78e26, 0x3fadbfa4, 0x2c3a9de4, 0x5078920d, 0x6a5fcc9b, 0x547e4662, 0xf68d13c2, 0x90d8b8e8, 0x2e39f75e, 0x82c3aff5, 0x9f5d80be, 0x69d0937c, 0x6fd52da9, 0xcf2512b3, 0xc8ac993b, 0x10187da7, 0xe89c636e, 0xdb3bbb7b, 0xcd267809, 0x6e5918f4, 0xec9ab701, 0x834f9aa8, 0xe6956e65, 0xaaffe67e, 0x21bccf08, 0xef15e8e6, 0xbae79bd9, 0x4a6f36ce, 0xea9f09d4, 0x29b07cd6, 0x31a4b2af, 0x2a3f2331, 0xc6a59430, 0x35a266c0, 0x744ebc37, 0xfc82caa6, 0xe090d0b0, 0x33a7d815, 0xf104984a, 0x41ecdaf7, 0x7fcd500e, 0x1791f62f, 0x764dd68d, 0x43efb04d, 0xccaa4d54, 0xe49604df, 0x9ed1b5e3, 0x4c6a881b, 0xc12c1fb8, 0x4665517f, 0x9d5eea04, 0x018c355d, 0xfa877473, 0xfb0b412e, 0xb3671d5a, 0x92dbd252, 0xe9105633, 0x6dd64713, 0x9ad7618c, 0x37a10c7a, 0x59f8148e, 0xeb133c89, 0xcea927ee, 0xb761c935, 0xe11ce5ed, 0x7a47b13c, 0x9cd2df59, 0x55f2733f, 0x1814ce79, 0x73c737bf, 0x53f7cdea, 0x5ffdaa5b, 0xdf3d6f14, 0x7844db86, 0xcaaff381, 0xb968c43e, 0x3824342c, 0xc2a3405f, 0x161dc372, 0xbce2250c, 0x283c498b, 0xff0d9541, 0x39a80171, 0x080cb3de, 0xd8b4e49c, 0x6456c190, 0x7bcb8461, 0xd532b670, 0x486c5c74, 0xd0b85742 ]
-    T6 = [ 0x5051f4a7, 0x537e4165, 0xc31a17a4, 0x963a275e, 0xcb3bab6b, 0xf11f9d45, 0xabacfa58, 0x934be303, 0x552030fa, 0xf6ad766d, 0x9188cc76, 0x25f5024c, 0xfc4fe5d7, 0xd7c52acb, 0x80263544, 0x8fb562a3, 0x49deb15a, 0x6725ba1b, 0x9845ea0e, 0xe15dfec0, 0x02c32f75, 0x12814cf0, 0xa38d4697, 0xc66bd3f9, 0xe7038f5f, 0x9515929c, 0xebbf6d7a, 0xda955259, 0x2dd4be83, 0xd3587421, 0x2949e069, 0x448ec9c8, 0x6a75c289, 0x78f48e79, 0x6b99583e, 0xdd27b971, 0xb6bee14f, 0x17f088ad, 0x66c920ac, 0xb47dce3a, 0x1863df4a, 0x82e51a31, 0x60975133, 0x4562537f, 0xe0b16477, 0x84bb6bae, 0x1cfe81a0, 0x94f9082b, 0x58704868, 0x198f45fd, 0x8794de6c, 0xb7527bf8, 0x23ab73d3, 0xe2724b02, 0x57e31f8f, 0x2a6655ab, 0x07b2eb28, 0x032fb5c2, 0x9a86c57b, 0xa5d33708, 0xf2302887, 0xb223bfa5, 0xba02036a, 0x5ced1682, 0x2b8acf1c, 0x92a779b4, 0xf0f307f2, 0xa14e69e2, 0xcd65daf4, 0xd50605be, 0x1fd13462, 0x8ac4a6fe, 0x9d342e53, 0xa0a2f355, 0x32058ae1, 0x75a4f6eb, 0x390b83ec, 0xaa4060ef, 0x065e719f, 0x51bd6e10, 0xf93e218a, 0x3d96dd06, 0xaedd3e05, 0x464de6bd, 0xb591548d, 0x0571c45d, 0x6f0406d4, 0xff605015, 0x241998fb, 0x97d6bde9, 0xcc894043, 0x7767d99e, 0xbdb0e842, 0x8807898b, 0x38e7195b, 0xdb79c8ee, 0x47a17c0a, 0xe97c420f, 0xc9f8841e, 0x00000000, 0x83098086, 0x48322bed, 0xac1e1170, 0x4e6c5a72, 0xfbfd0eff, 0x560f8538, 0x1e3daed5, 0x27362d39, 0x640a0fd9, 0x21685ca6, 0xd19b5b54, 0x3a24362e, 0xb10c0a67, 0x0f9357e7, 0xd2b4ee96, 0x9e1b9b91, 0x4f80c0c5, 0xa261dc20, 0x695a774b, 0x161c121a, 0x0ae293ba, 0xe5c0a02a, 0x433c22e0, 0x1d121b17, 0x0b0e090d, 0xadf28bc7, 0xb92db6a8, 0xc8141ea9, 0x8557f119, 0x4caf7507, 0xbbee99dd, 0xfda37f60, 0x9ff70126, 0xbc5c72f5, 0xc544663b, 0x345bfb7e, 0x768b4329, 0xdccb23c6, 0x68b6edfc, 0x63b8e4f1, 0xcad731dc, 0x10426385, 0x40139722, 0x2084c611, 0x7d854a24, 0xf8d2bb3d, 0x11aef932, 0x6dc729a1, 0x4b1d9e2f, 0xf3dcb230, 0xec0d8652, 0xd077c1e3, 0x6c2bb316, 0x99a970b9, 0xfa119448, 0x2247e964, 0xc4a8fc8c, 0x1aa0f03f, 0xd8567d2c, 0xef223390, 0xc787494e, 0xc1d938d1, 0xfe8ccaa2, 0x3698d40b, 0xcfa6f581, 0x28a57ade, 0x26dab78e, 0xa43fadbf, 0xe42c3a9d, 0x0d507892, 0x9b6a5fcc, 0x62547e46, 0xc2f68d13, 0xe890d8b8, 0x5e2e39f7, 0xf582c3af, 0xbe9f5d80, 0x7c69d093, 0xa96fd52d, 0xb3cf2512, 0x3bc8ac99, 0xa710187d, 0x6ee89c63, 0x7bdb3bbb, 0x09cd2678, 0xf46e5918, 0x01ec9ab7, 0xa8834f9a, 0x65e6956e, 0x7eaaffe6, 0x0821bccf, 0xe6ef15e8, 0xd9bae79b, 0xce4a6f36, 0xd4ea9f09, 0xd629b07c, 0xaf31a4b2, 0x312a3f23, 0x30c6a594, 0xc035a266, 0x37744ebc, 0xa6fc82ca, 0xb0e090d0, 0x1533a7d8, 0x4af10498, 0xf741ecda, 0x0e7fcd50, 0x2f1791f6, 0x8d764dd6, 0x4d43efb0, 0x54ccaa4d, 0xdfe49604, 0xe39ed1b5, 0x1b4c6a88, 0xb8c12c1f, 0x7f466551, 0x049d5eea, 0x5d018c35, 0x73fa8774, 0x2efb0b41, 0x5ab3671d, 0x5292dbd2, 0x33e91056, 0x136dd647, 0x8c9ad761, 0x7a37a10c, 0x8e59f814, 0x89eb133c, 0xeecea927, 0x35b761c9, 0xede11ce5, 0x3c7a47b1, 0x599cd2df, 0x3f55f273, 0x791814ce, 0xbf73c737, 0xea53f7cd, 0x5b5ffdaa, 0x14df3d6f, 0x867844db, 0x81caaff3, 0x3eb968c4, 0x2c382434, 0x5fc2a340, 0x72161dc3, 0x0cbce225, 0x8b283c49, 0x41ff0d95, 0x7139a801, 0xde080cb3, 0x9cd8b4e4, 0x906456c1, 0x617bcb84, 0x70d532b6, 0x74486c5c, 0x42d0b857 ]
-    T7 = [ 0xa75051f4, 0x65537e41, 0xa4c31a17, 0x5e963a27, 0x6bcb3bab, 0x45f11f9d, 0x58abacfa, 0x03934be3, 0xfa552030, 0x6df6ad76, 0x769188cc, 0x4c25f502, 0xd7fc4fe5, 0xcbd7c52a, 0x44802635, 0xa38fb562, 0x5a49deb1, 0x1b6725ba, 0x0e9845ea, 0xc0e15dfe, 0x7502c32f, 0xf012814c, 0x97a38d46, 0xf9c66bd3, 0x5fe7038f, 0x9c951592, 0x7aebbf6d, 0x59da9552, 0x832dd4be, 0x21d35874, 0x692949e0, 0xc8448ec9, 0x896a75c2, 0x7978f48e, 0x3e6b9958, 0x71dd27b9, 0x4fb6bee1, 0xad17f088, 0xac66c920, 0x3ab47dce, 0x4a1863df, 0x3182e51a, 0x33609751, 0x7f456253, 0x77e0b164, 0xae84bb6b, 0xa01cfe81, 0x2b94f908, 0x68587048, 0xfd198f45, 0x6c8794de, 0xf8b7527b, 0xd323ab73, 0x02e2724b, 0x8f57e31f, 0xab2a6655, 0x2807b2eb, 0xc2032fb5, 0x7b9a86c5, 0x08a5d337, 0x87f23028, 0xa5b223bf, 0x6aba0203, 0x825ced16, 0x1c2b8acf, 0xb492a779, 0xf2f0f307, 0xe2a14e69, 0xf4cd65da, 0xbed50605, 0x621fd134, 0xfe8ac4a6, 0x539d342e, 0x55a0a2f3, 0xe132058a, 0xeb75a4f6, 0xec390b83, 0xefaa4060, 0x9f065e71, 0x1051bd6e, 0x8af93e21, 0x063d96dd, 0x05aedd3e, 0xbd464de6, 0x8db59154, 0x5d0571c4, 0xd46f0406, 0x15ff6050, 0xfb241998, 0xe997d6bd, 0x43cc8940, 0x9e7767d9, 0x42bdb0e8, 0x8b880789, 0x5b38e719, 0xeedb79c8, 0x0a47a17c, 0x0fe97c42, 0x1ec9f884, 0x00000000, 0x86830980, 0xed48322b, 0x70ac1e11, 0x724e6c5a, 0xfffbfd0e, 0x38560f85, 0xd51e3dae, 0x3927362d, 0xd9640a0f, 0xa621685c, 0x54d19b5b, 0x2e3a2436, 0x67b10c0a, 0xe70f9357, 0x96d2b4ee, 0x919e1b9b, 0xc54f80c0, 0x20a261dc, 0x4b695a77, 0x1a161c12, 0xba0ae293, 0x2ae5c0a0, 0xe0433c22, 0x171d121b, 0x0d0b0e09, 0xc7adf28b, 0xa8b92db6, 0xa9c8141e, 0x198557f1, 0x074caf75, 0xddbbee99, 0x60fda37f, 0x269ff701, 0xf5bc5c72, 0x3bc54466, 0x7e345bfb, 0x29768b43, 0xc6dccb23, 0xfc68b6ed, 0xf163b8e4, 0xdccad731, 0x85104263, 0x22401397, 0x112084c6, 0x247d854a, 0x3df8d2bb, 0x3211aef9, 0xa16dc729, 0x2f4b1d9e, 0x30f3dcb2, 0x52ec0d86, 0xe3d077c1, 0x166c2bb3, 0xb999a970, 0x48fa1194, 0x642247e9, 0x8cc4a8fc, 0x3f1aa0f0, 0x2cd8567d, 0x90ef2233, 0x4ec78749, 0xd1c1d938, 0xa2fe8cca, 0x0b3698d4, 0x81cfa6f5, 0xde28a57a, 0x8e26dab7, 0xbfa43fad, 0x9de42c3a, 0x920d5078, 0xcc9b6a5f, 0x4662547e, 0x13c2f68d, 0xb8e890d8, 0xf75e2e39, 0xaff582c3, 0x80be9f5d, 0x937c69d0, 0x2da96fd5, 0x12b3cf25, 0x993bc8ac, 0x7da71018, 0x636ee89c, 0xbb7bdb3b, 0x7809cd26, 0x18f46e59, 0xb701ec9a, 0x9aa8834f, 0x6e65e695, 0xe67eaaff, 0xcf0821bc, 0xe8e6ef15, 0x9bd9bae7, 0x36ce4a6f, 0x09d4ea9f, 0x7cd629b0, 0xb2af31a4, 0x23312a3f, 0x9430c6a5, 0x66c035a2, 0xbc37744e, 0xcaa6fc82, 0xd0b0e090, 0xd81533a7, 0x984af104, 0xdaf741ec, 0x500e7fcd, 0xf62f1791, 0xd68d764d, 0xb04d43ef, 0x4d54ccaa, 0x04dfe496, 0xb5e39ed1, 0x881b4c6a, 0x1fb8c12c, 0x517f4665, 0xea049d5e, 0x355d018c, 0x7473fa87, 0x412efb0b, 0x1d5ab367, 0xd25292db, 0x5633e910, 0x47136dd6, 0x618c9ad7, 0x0c7a37a1, 0x148e59f8, 0x3c89eb13, 0x27eecea9, 0xc935b761, 0xe5ede11c, 0xb13c7a47, 0xdf599cd2, 0x733f55f2, 0xce791814, 0x37bf73c7, 0xcdea53f7, 0xaa5b5ffd, 0x6f14df3d, 0xdb867844, 0xf381caaf, 0xc43eb968, 0x342c3824, 0x405fc2a3, 0xc372161d, 0x250cbce2, 0x498b283c, 0x9541ff0d, 0x017139a8, 0xb3de080c, 0xe49cd8b4, 0xc1906456, 0x84617bcb, 0xb670d532, 0x5c74486c, 0x5742d0b8 ]
-    T8 = [ 0xf4a75051, 0x4165537e, 0x17a4c31a, 0x275e963a, 0xab6bcb3b, 0x9d45f11f, 0xfa58abac, 0xe303934b, 0x30fa5520, 0x766df6ad, 0xcc769188, 0x024c25f5, 0xe5d7fc4f, 0x2acbd7c5, 0x35448026, 0x62a38fb5, 0xb15a49de, 0xba1b6725, 0xea0e9845, 0xfec0e15d, 0x2f7502c3, 0x4cf01281, 0x4697a38d, 0xd3f9c66b, 0x8f5fe703, 0x929c9515, 0x6d7aebbf, 0x5259da95, 0xbe832dd4, 0x7421d358, 0xe0692949, 0xc9c8448e, 0xc2896a75, 0x8e7978f4, 0x583e6b99, 0xb971dd27, 0xe14fb6be, 0x88ad17f0, 0x20ac66c9, 0xce3ab47d, 0xdf4a1863, 0x1a3182e5, 0x51336097, 0x537f4562, 0x6477e0b1, 0x6bae84bb, 0x81a01cfe, 0x082b94f9, 0x48685870, 0x45fd198f, 0xde6c8794, 0x7bf8b752, 0x73d323ab, 0x4b02e272, 0x1f8f57e3, 0x55ab2a66, 0xeb2807b2, 0xb5c2032f, 0xc57b9a86, 0x3708a5d3, 0x2887f230, 0xbfa5b223, 0x036aba02, 0x16825ced, 0xcf1c2b8a, 0x79b492a7, 0x07f2f0f3, 0x69e2a14e, 0xdaf4cd65, 0x05bed506, 0x34621fd1, 0xa6fe8ac4, 0x2e539d34, 0xf355a0a2, 0x8ae13205, 0xf6eb75a4, 0x83ec390b, 0x60efaa40, 0x719f065e, 0x6e1051bd, 0x218af93e, 0xdd063d96, 0x3e05aedd, 0xe6bd464d, 0x548db591, 0xc45d0571, 0x06d46f04, 0x5015ff60, 0x98fb2419, 0xbde997d6, 0x4043cc89, 0xd99e7767, 0xe842bdb0, 0x898b8807, 0x195b38e7, 0xc8eedb79, 0x7c0a47a1, 0x420fe97c, 0x841ec9f8, 0x00000000, 0x80868309, 0x2bed4832, 0x1170ac1e, 0x5a724e6c, 0x0efffbfd, 0x8538560f, 0xaed51e3d, 0x2d392736, 0x0fd9640a, 0x5ca62168, 0x5b54d19b, 0x362e3a24, 0x0a67b10c, 0x57e70f93, 0xee96d2b4, 0x9b919e1b, 0xc0c54f80, 0xdc20a261, 0x774b695a, 0x121a161c, 0x93ba0ae2, 0xa02ae5c0, 0x22e0433c, 0x1b171d12, 0x090d0b0e, 0x8bc7adf2, 0xb6a8b92d, 0x1ea9c814, 0xf1198557, 0x75074caf, 0x99ddbbee, 0x7f60fda3, 0x01269ff7, 0x72f5bc5c, 0x663bc544, 0xfb7e345b, 0x4329768b, 0x23c6dccb, 0xedfc68b6, 0xe4f163b8, 0x31dccad7, 0x63851042, 0x97224013, 0xc6112084, 0x4a247d85, 0xbb3df8d2, 0xf93211ae, 0x29a16dc7, 0x9e2f4b1d, 0xb230f3dc, 0x8652ec0d, 0xc1e3d077, 0xb3166c2b, 0x70b999a9, 0x9448fa11, 0xe9642247, 0xfc8cc4a8, 0xf03f1aa0, 0x7d2cd856, 0x3390ef22, 0x494ec787, 0x38d1c1d9, 0xcaa2fe8c, 0xd40b3698, 0xf581cfa6, 0x7ade28a5, 0xb78e26da, 0xadbfa43f, 0x3a9de42c, 0x78920d50, 0x5fcc9b6a, 0x7e466254, 0x8d13c2f6, 0xd8b8e890, 0x39f75e2e, 0xc3aff582, 0x5d80be9f, 0xd0937c69, 0xd52da96f, 0x2512b3cf, 0xac993bc8, 0x187da710, 0x9c636ee8, 0x3bbb7bdb, 0x267809cd, 0x5918f46e, 0x9ab701ec, 0x4f9aa883, 0x956e65e6, 0xffe67eaa, 0xbccf0821, 0x15e8e6ef, 0xe79bd9ba, 0x6f36ce4a, 0x9f09d4ea, 0xb07cd629, 0xa4b2af31, 0x3f23312a, 0xa59430c6, 0xa266c035, 0x4ebc3774, 0x82caa6fc, 0x90d0b0e0, 0xa7d81533, 0x04984af1, 0xecdaf741, 0xcd500e7f, 0x91f62f17, 0x4dd68d76, 0xefb04d43, 0xaa4d54cc, 0x9604dfe4, 0xd1b5e39e, 0x6a881b4c, 0x2c1fb8c1, 0x65517f46, 0x5eea049d, 0x8c355d01, 0x877473fa, 0x0b412efb, 0x671d5ab3, 0xdbd25292, 0x105633e9, 0xd647136d, 0xd7618c9a, 0xa10c7a37, 0xf8148e59, 0x133c89eb, 0xa927eece, 0x61c935b7, 0x1ce5ede1, 0x47b13c7a, 0xd2df599c, 0xf2733f55, 0x14ce7918, 0xc737bf73, 0xf7cdea53, 0xfdaa5b5f, 0x3d6f14df, 0x44db8678, 0xaff381ca, 0x68c43eb9, 0x24342c38, 0xa3405fc2, 0x1dc37216, 0xe2250cbc, 0x3c498b28, 0x0d9541ff, 0xa8017139, 0x0cb3de08, 0xb4e49cd8, 0x56c19064, 0xcb84617b, 0x32b670d5, 0x6c5c7448, 0xb85742d0 ]
-
-    # Transformations for decryption key expansion
-    U1 = [ 0x00000000, 0x0e090d0b, 0x1c121a16, 0x121b171d, 0x3824342c, 0x362d3927, 0x24362e3a, 0x2a3f2331, 0x70486858, 0x7e416553, 0x6c5a724e, 0x62537f45, 0x486c5c74, 0x4665517f, 0x547e4662, 0x5a774b69, 0xe090d0b0, 0xee99ddbb, 0xfc82caa6, 0xf28bc7ad, 0xd8b4e49c, 0xd6bde997, 0xc4a6fe8a, 0xcaaff381, 0x90d8b8e8, 0x9ed1b5e3, 0x8ccaa2fe, 0x82c3aff5, 0xa8fc8cc4, 0xa6f581cf, 0xb4ee96d2, 0xbae79bd9, 0xdb3bbb7b, 0xd532b670, 0xc729a16d, 0xc920ac66, 0xe31f8f57, 0xed16825c, 0xff0d9541, 0xf104984a, 0xab73d323, 0xa57ade28, 0xb761c935, 0xb968c43e, 0x9357e70f, 0x9d5eea04, 0x8f45fd19, 0x814cf012, 0x3bab6bcb, 0x35a266c0, 0x27b971dd, 0x29b07cd6, 0x038f5fe7, 0x0d8652ec, 0x1f9d45f1, 0x119448fa, 0x4be30393, 0x45ea0e98, 0x57f11985, 0x59f8148e, 0x73c737bf, 0x7dce3ab4, 0x6fd52da9, 0x61dc20a2, 0xad766df6, 0xa37f60fd, 0xb16477e0, 0xbf6d7aeb, 0x955259da, 0x9b5b54d1, 0x894043cc, 0x87494ec7, 0xdd3e05ae, 0xd33708a5, 0xc12c1fb8, 0xcf2512b3, 0xe51a3182, 0xeb133c89, 0xf9082b94, 0xf701269f, 0x4de6bd46, 0x43efb04d, 0x51f4a750, 0x5ffdaa5b, 0x75c2896a, 0x7bcb8461, 0x69d0937c, 0x67d99e77, 0x3daed51e, 0x33a7d815, 0x21bccf08, 0x2fb5c203, 0x058ae132, 0x0b83ec39, 0x1998fb24, 0x1791f62f, 0x764dd68d, 0x7844db86, 0x6a5fcc9b, 0x6456c190, 0x4e69e2a1, 0x4060efaa, 0x527bf8b7, 0x5c72f5bc, 0x0605bed5, 0x080cb3de, 0x1a17a4c3, 0x141ea9c8, 0x3e218af9, 0x302887f2, 0x223390ef, 0x2c3a9de4, 0x96dd063d, 0x98d40b36, 0x8acf1c2b, 0x84c61120, 0xaef93211, 0xa0f03f1a, 0xb2eb2807, 0xbce2250c, 0xe6956e65, 0xe89c636e, 0xfa877473, 0xf48e7978, 0xdeb15a49, 0xd0b85742, 0xc2a3405f, 0xccaa4d54, 0x41ecdaf7, 0x4fe5d7fc, 0x5dfec0e1, 0x53f7cdea, 0x79c8eedb, 0x77c1e3d0, 0x65daf4cd, 0x6bd3f9c6, 0x31a4b2af, 0x3fadbfa4, 0x2db6a8b9, 0x23bfa5b2, 0x09808683, 0x07898b88, 0x15929c95, 0x1b9b919e, 0xa17c0a47, 0xaf75074c, 0xbd6e1051, 0xb3671d5a, 0x99583e6b, 0x97513360, 0x854a247d, 0x8b432976, 0xd134621f, 0xdf3d6f14, 0xcd267809, 0xc32f7502, 0xe9105633, 0xe7195b38, 0xf5024c25, 0xfb0b412e, 0x9ad7618c, 0x94de6c87, 0x86c57b9a, 0x88cc7691, 0xa2f355a0, 0xacfa58ab, 0xbee14fb6, 0xb0e842bd, 0xea9f09d4, 0xe49604df, 0xf68d13c2, 0xf8841ec9, 0xd2bb3df8, 0xdcb230f3, 0xcea927ee, 0xc0a02ae5, 0x7a47b13c, 0x744ebc37, 0x6655ab2a, 0x685ca621, 0x42638510, 0x4c6a881b, 0x5e719f06, 0x5078920d, 0x0a0fd964, 0x0406d46f, 0x161dc372, 0x1814ce79, 0x322bed48, 0x3c22e043, 0x2e39f75e, 0x2030fa55, 0xec9ab701, 0xe293ba0a, 0xf088ad17, 0xfe81a01c, 0xd4be832d, 0xdab78e26, 0xc8ac993b, 0xc6a59430, 0x9cd2df59, 0x92dbd252, 0x80c0c54f, 0x8ec9c844, 0xa4f6eb75, 0xaaffe67e, 0xb8e4f163, 0xb6edfc68, 0x0c0a67b1, 0x02036aba, 0x10187da7, 0x1e1170ac, 0x342e539d, 0x3a275e96, 0x283c498b, 0x26354480, 0x7c420fe9, 0x724b02e2, 0x605015ff, 0x6e5918f4, 0x44663bc5, 0x4a6f36ce, 0x587421d3, 0x567d2cd8, 0x37a10c7a, 0x39a80171, 0x2bb3166c, 0x25ba1b67, 0x0f853856, 0x018c355d, 0x13972240, 0x1d9e2f4b, 0x47e96422, 0x49e06929, 0x5bfb7e34, 0x55f2733f, 0x7fcd500e, 0x71c45d05, 0x63df4a18, 0x6dd64713, 0xd731dcca, 0xd938d1c1, 0xcb23c6dc, 0xc52acbd7, 0xef15e8e6, 0xe11ce5ed, 0xf307f2f0, 0xfd0efffb, 0xa779b492, 0xa970b999, 0xbb6bae84, 0xb562a38f, 0x9f5d80be, 0x91548db5, 0x834f9aa8, 0x8d4697a3 ]
-    U2 = [ 0x00000000, 0x0b0e090d, 0x161c121a, 0x1d121b17, 0x2c382434, 0x27362d39, 0x3a24362e, 0x312a3f23, 0x58704868, 0x537e4165, 0x4e6c5a72, 0x4562537f, 0x74486c5c, 0x7f466551, 0x62547e46, 0x695a774b, 0xb0e090d0, 0xbbee99dd, 0xa6fc82ca, 0xadf28bc7, 0x9cd8b4e4, 0x97d6bde9, 0x8ac4a6fe, 0x81caaff3, 0xe890d8b8, 0xe39ed1b5, 0xfe8ccaa2, 0xf582c3af, 0xc4a8fc8c, 0xcfa6f581, 0xd2b4ee96, 0xd9bae79b, 0x7bdb3bbb, 0x70d532b6, 0x6dc729a1, 0x66c920ac, 0x57e31f8f, 0x5ced1682, 0x41ff0d95, 0x4af10498, 0x23ab73d3, 0x28a57ade, 0x35b761c9, 0x3eb968c4, 0x0f9357e7, 0x049d5eea, 0x198f45fd, 0x12814cf0, 0xcb3bab6b, 0xc035a266, 0xdd27b971, 0xd629b07c, 0xe7038f5f, 0xec0d8652, 0xf11f9d45, 0xfa119448, 0x934be303, 0x9845ea0e, 0x8557f119, 0x8e59f814, 0xbf73c737, 0xb47dce3a, 0xa96fd52d, 0xa261dc20, 0xf6ad766d, 0xfda37f60, 0xe0b16477, 0xebbf6d7a, 0xda955259, 0xd19b5b54, 0xcc894043, 0xc787494e, 0xaedd3e05, 0xa5d33708, 0xb8c12c1f, 0xb3cf2512, 0x82e51a31, 0x89eb133c, 0x94f9082b, 0x9ff70126, 0x464de6bd, 0x4d43efb0, 0x5051f4a7, 0x5b5ffdaa, 0x6a75c289, 0x617bcb84, 0x7c69d093, 0x7767d99e, 0x1e3daed5, 0x1533a7d8, 0x0821bccf, 0x032fb5c2, 0x32058ae1, 0x390b83ec, 0x241998fb, 0x2f1791f6, 0x8d764dd6, 0x867844db, 0x9b6a5fcc, 0x906456c1, 0xa14e69e2, 0xaa4060ef, 0xb7527bf8, 0xbc5c72f5, 0xd50605be, 0xde080cb3, 0xc31a17a4, 0xc8141ea9, 0xf93e218a, 0xf2302887, 0xef223390, 0xe42c3a9d, 0x3d96dd06, 0x3698d40b, 0x2b8acf1c, 0x2084c611, 0x11aef932, 0x1aa0f03f, 0x07b2eb28, 0x0cbce225, 0x65e6956e, 0x6ee89c63, 0x73fa8774, 0x78f48e79, 0x49deb15a, 0x42d0b857, 0x5fc2a340, 0x54ccaa4d, 0xf741ecda, 0xfc4fe5d7, 0xe15dfec0, 0xea53f7cd, 0xdb79c8ee, 0xd077c1e3, 0xcd65daf4, 0xc66bd3f9, 0xaf31a4b2, 0xa43fadbf, 0xb92db6a8, 0xb223bfa5, 0x83098086, 0x8807898b, 0x9515929c, 0x9e1b9b91, 0x47a17c0a, 0x4caf7507, 0x51bd6e10, 0x5ab3671d, 0x6b99583e, 0x60975133, 0x7d854a24, 0x768b4329, 0x1fd13462, 0x14df3d6f, 0x09cd2678, 0x02c32f75, 0x33e91056, 0x38e7195b, 0x25f5024c, 0x2efb0b41, 0x8c9ad761, 0x8794de6c, 0x9a86c57b, 0x9188cc76, 0xa0a2f355, 0xabacfa58, 0xb6bee14f, 0xbdb0e842, 0xd4ea9f09, 0xdfe49604, 0xc2f68d13, 0xc9f8841e, 0xf8d2bb3d, 0xf3dcb230, 0xeecea927, 0xe5c0a02a, 0x3c7a47b1, 0x37744ebc, 0x2a6655ab, 0x21685ca6, 0x10426385, 0x1b4c6a88, 0x065e719f, 0x0d507892, 0x640a0fd9, 0x6f0406d4, 0x72161dc3, 0x791814ce, 0x48322bed, 0x433c22e0, 0x5e2e39f7, 0x552030fa, 0x01ec9ab7, 0x0ae293ba, 0x17f088ad, 0x1cfe81a0, 0x2dd4be83, 0x26dab78e, 0x3bc8ac99, 0x30c6a594, 0x599cd2df, 0x5292dbd2, 0x4f80c0c5, 0x448ec9c8, 0x75a4f6eb, 0x7eaaffe6, 0x63b8e4f1, 0x68b6edfc, 0xb10c0a67, 0xba02036a, 0xa710187d, 0xac1e1170, 0x9d342e53, 0x963a275e, 0x8b283c49, 0x80263544, 0xe97c420f, 0xe2724b02, 0xff605015, 0xf46e5918, 0xc544663b, 0xce4a6f36, 0xd3587421, 0xd8567d2c, 0x7a37a10c, 0x7139a801, 0x6c2bb316, 0x6725ba1b, 0x560f8538, 0x5d018c35, 0x40139722, 0x4b1d9e2f, 0x2247e964, 0x2949e069, 0x345bfb7e, 0x3f55f273, 0x0e7fcd50, 0x0571c45d, 0x1863df4a, 0x136dd647, 0xcad731dc, 0xc1d938d1, 0xdccb23c6, 0xd7c52acb, 0xe6ef15e8, 0xede11ce5, 0xf0f307f2, 0xfbfd0eff, 0x92a779b4, 0x99a970b9, 0x84bb6bae, 0x8fb562a3, 0xbe9f5d80, 0xb591548d, 0xa8834f9a, 0xa38d4697 ]
-    U3 = [ 0x00000000, 0x0d0b0e09, 0x1a161c12, 0x171d121b, 0x342c3824, 0x3927362d, 0x2e3a2436, 0x23312a3f, 0x68587048, 0x65537e41, 0x724e6c5a, 0x7f456253, 0x5c74486c, 0x517f4665, 0x4662547e, 0x4b695a77, 0xd0b0e090, 0xddbbee99, 0xcaa6fc82, 0xc7adf28b, 0xe49cd8b4, 0xe997d6bd, 0xfe8ac4a6, 0xf381caaf, 0xb8e890d8, 0xb5e39ed1, 0xa2fe8cca, 0xaff582c3, 0x8cc4a8fc, 0x81cfa6f5, 0x96d2b4ee, 0x9bd9bae7, 0xbb7bdb3b, 0xb670d532, 0xa16dc729, 0xac66c920, 0x8f57e31f, 0x825ced16, 0x9541ff0d, 0x984af104, 0xd323ab73, 0xde28a57a, 0xc935b761, 0xc43eb968, 0xe70f9357, 0xea049d5e, 0xfd198f45, 0xf012814c, 0x6bcb3bab, 0x66c035a2, 0x71dd27b9, 0x7cd629b0, 0x5fe7038f, 0x52ec0d86, 0x45f11f9d, 0x48fa1194, 0x03934be3, 0x0e9845ea, 0x198557f1, 0x148e59f8, 0x37bf73c7, 0x3ab47dce, 0x2da96fd5, 0x20a261dc, 0x6df6ad76, 0x60fda37f, 0x77e0b164, 0x7aebbf6d, 0x59da9552, 0x54d19b5b, 0x43cc8940, 0x4ec78749, 0x05aedd3e, 0x08a5d337, 0x1fb8c12c, 0x12b3cf25, 0x3182e51a, 0x3c89eb13, 0x2b94f908, 0x269ff701, 0xbd464de6, 0xb04d43ef, 0xa75051f4, 0xaa5b5ffd, 0x896a75c2, 0x84617bcb, 0x937c69d0, 0x9e7767d9, 0xd51e3dae, 0xd81533a7, 0xcf0821bc, 0xc2032fb5, 0xe132058a, 0xec390b83, 0xfb241998, 0xf62f1791, 0xd68d764d, 0xdb867844, 0xcc9b6a5f, 0xc1906456, 0xe2a14e69, 0xefaa4060, 0xf8b7527b, 0xf5bc5c72, 0xbed50605, 0xb3de080c, 0xa4c31a17, 0xa9c8141e, 0x8af93e21, 0x87f23028, 0x90ef2233, 0x9de42c3a, 0x063d96dd, 0x0b3698d4, 0x1c2b8acf, 0x112084c6, 0x3211aef9, 0x3f1aa0f0, 0x2807b2eb, 0x250cbce2, 0x6e65e695, 0x636ee89c, 0x7473fa87, 0x7978f48e, 0x5a49deb1, 0x5742d0b8, 0x405fc2a3, 0x4d54ccaa, 0xdaf741ec, 0xd7fc4fe5, 0xc0e15dfe, 0xcdea53f7, 0xeedb79c8, 0xe3d077c1, 0xf4cd65da, 0xf9c66bd3, 0xb2af31a4, 0xbfa43fad, 0xa8b92db6, 0xa5b223bf, 0x86830980, 0x8b880789, 0x9c951592, 0x919e1b9b, 0x0a47a17c, 0x074caf75, 0x1051bd6e, 0x1d5ab367, 0x3e6b9958, 0x33609751, 0x247d854a, 0x29768b43, 0x621fd134, 0x6f14df3d, 0x7809cd26, 0x7502c32f, 0x5633e910, 0x5b38e719, 0x4c25f502, 0x412efb0b, 0x618c9ad7, 0x6c8794de, 0x7b9a86c5, 0x769188cc, 0x55a0a2f3, 0x58abacfa, 0x4fb6bee1, 0x42bdb0e8, 0x09d4ea9f, 0x04dfe496, 0x13c2f68d, 0x1ec9f884, 0x3df8d2bb, 0x30f3dcb2, 0x27eecea9, 0x2ae5c0a0, 0xb13c7a47, 0xbc37744e, 0xab2a6655, 0xa621685c, 0x85104263, 0x881b4c6a, 0x9f065e71, 0x920d5078, 0xd9640a0f, 0xd46f0406, 0xc372161d, 0xce791814, 0xed48322b, 0xe0433c22, 0xf75e2e39, 0xfa552030, 0xb701ec9a, 0xba0ae293, 0xad17f088, 0xa01cfe81, 0x832dd4be, 0x8e26dab7, 0x993bc8ac, 0x9430c6a5, 0xdf599cd2, 0xd25292db, 0xc54f80c0, 0xc8448ec9, 0xeb75a4f6, 0xe67eaaff, 0xf163b8e4, 0xfc68b6ed, 0x67b10c0a, 0x6aba0203, 0x7da71018, 0x70ac1e11, 0x539d342e, 0x5e963a27, 0x498b283c, 0x44802635, 0x0fe97c42, 0x02e2724b, 0x15ff6050, 0x18f46e59, 0x3bc54466, 0x36ce4a6f, 0x21d35874, 0x2cd8567d, 0x0c7a37a1, 0x017139a8, 0x166c2bb3, 0x1b6725ba, 0x38560f85, 0x355d018c, 0x22401397, 0x2f4b1d9e, 0x642247e9, 0x692949e0, 0x7e345bfb, 0x733f55f2, 0x500e7fcd, 0x5d0571c4, 0x4a1863df, 0x47136dd6, 0xdccad731, 0xd1c1d938, 0xc6dccb23, 0xcbd7c52a, 0xe8e6ef15, 0xe5ede11c, 0xf2f0f307, 0xfffbfd0e, 0xb492a779, 0xb999a970, 0xae84bb6b, 0xa38fb562, 0x80be9f5d, 0x8db59154, 0x9aa8834f, 0x97a38d46 ]
-    U4 = [ 0x00000000, 0x090d0b0e, 0x121a161c, 0x1b171d12, 0x24342c38, 0x2d392736, 0x362e3a24, 0x3f23312a, 0x48685870, 0x4165537e, 0x5a724e6c, 0x537f4562, 0x6c5c7448, 0x65517f46, 0x7e466254, 0x774b695a, 0x90d0b0e0, 0x99ddbbee, 0x82caa6fc, 0x8bc7adf2, 0xb4e49cd8, 0xbde997d6, 0xa6fe8ac4, 0xaff381ca, 0xd8b8e890, 0xd1b5e39e, 0xcaa2fe8c, 0xc3aff582, 0xfc8cc4a8, 0xf581cfa6, 0xee96d2b4, 0xe79bd9ba, 0x3bbb7bdb, 0x32b670d5, 0x29a16dc7, 0x20ac66c9, 0x1f8f57e3, 0x16825ced, 0x0d9541ff, 0x04984af1, 0x73d323ab, 0x7ade28a5, 0x61c935b7, 0x68c43eb9, 0x57e70f93, 0x5eea049d, 0x45fd198f, 0x4cf01281, 0xab6bcb3b, 0xa266c035, 0xb971dd27, 0xb07cd629, 0x8f5fe703, 0x8652ec0d, 0x9d45f11f, 0x9448fa11, 0xe303934b, 0xea0e9845, 0xf1198557, 0xf8148e59, 0xc737bf73, 0xce3ab47d, 0xd52da96f, 0xdc20a261, 0x766df6ad, 0x7f60fda3, 0x6477e0b1, 0x6d7aebbf, 0x5259da95, 0x5b54d19b, 0x4043cc89, 0x494ec787, 0x3e05aedd, 0x3708a5d3, 0x2c1fb8c1, 0x2512b3cf, 0x1a3182e5, 0x133c89eb, 0x082b94f9, 0x01269ff7, 0xe6bd464d, 0xefb04d43, 0xf4a75051, 0xfdaa5b5f, 0xc2896a75, 0xcb84617b, 0xd0937c69, 0xd99e7767, 0xaed51e3d, 0xa7d81533, 0xbccf0821, 0xb5c2032f, 0x8ae13205, 0x83ec390b, 0x98fb2419, 0x91f62f17, 0x4dd68d76, 0x44db8678, 0x5fcc9b6a, 0x56c19064, 0x69e2a14e, 0x60efaa40, 0x7bf8b752, 0x72f5bc5c, 0x05bed506, 0x0cb3de08, 0x17a4c31a, 0x1ea9c814, 0x218af93e, 0x2887f230, 0x3390ef22, 0x3a9de42c, 0xdd063d96, 0xd40b3698, 0xcf1c2b8a, 0xc6112084, 0xf93211ae, 0xf03f1aa0, 0xeb2807b2, 0xe2250cbc, 0x956e65e6, 0x9c636ee8, 0x877473fa, 0x8e7978f4, 0xb15a49de, 0xb85742d0, 0xa3405fc2, 0xaa4d54cc, 0xecdaf741, 0xe5d7fc4f, 0xfec0e15d, 0xf7cdea53, 0xc8eedb79, 0xc1e3d077, 0xdaf4cd65, 0xd3f9c66b, 0xa4b2af31, 0xadbfa43f, 0xb6a8b92d, 0xbfa5b223, 0x80868309, 0x898b8807, 0x929c9515, 0x9b919e1b, 0x7c0a47a1, 0x75074caf, 0x6e1051bd, 0x671d5ab3, 0x583e6b99, 0x51336097, 0x4a247d85, 0x4329768b, 0x34621fd1, 0x3d6f14df, 0x267809cd, 0x2f7502c3, 0x105633e9, 0x195b38e7, 0x024c25f5, 0x0b412efb, 0xd7618c9a, 0xde6c8794, 0xc57b9a86, 0xcc769188, 0xf355a0a2, 0xfa58abac, 0xe14fb6be, 0xe842bdb0, 0x9f09d4ea, 0x9604dfe4, 0x8d13c2f6, 0x841ec9f8, 0xbb3df8d2, 0xb230f3dc, 0xa927eece, 0xa02ae5c0, 0x47b13c7a, 0x4ebc3774, 0x55ab2a66, 0x5ca62168, 0x63851042, 0x6a881b4c, 0x719f065e, 0x78920d50, 0x0fd9640a, 0x06d46f04, 0x1dc37216, 0x14ce7918, 0x2bed4832, 0x22e0433c, 0x39f75e2e, 0x30fa5520, 0x9ab701ec, 0x93ba0ae2, 0x88ad17f0, 0x81a01cfe, 0xbe832dd4, 0xb78e26da, 0xac993bc8, 0xa59430c6, 0xd2df599c, 0xdbd25292, 0xc0c54f80, 0xc9c8448e, 0xf6eb75a4, 0xffe67eaa, 0xe4f163b8, 0xedfc68b6, 0x0a67b10c, 0x036aba02, 0x187da710, 0x1170ac1e, 0x2e539d34, 0x275e963a, 0x3c498b28, 0x35448026, 0x420fe97c, 0x4b02e272, 0x5015ff60, 0x5918f46e, 0x663bc544, 0x6f36ce4a, 0x7421d358, 0x7d2cd856, 0xa10c7a37, 0xa8017139, 0xb3166c2b, 0xba1b6725, 0x8538560f, 0x8c355d01, 0x97224013, 0x9e2f4b1d, 0xe9642247, 0xe0692949, 0xfb7e345b, 0xf2733f55, 0xcd500e7f, 0xc45d0571, 0xdf4a1863, 0xd647136d, 0x31dccad7, 0x38d1c1d9, 0x23c6dccb, 0x2acbd7c5, 0x15e8e6ef, 0x1ce5ede1, 0x07f2f0f3, 0x0efffbfd, 0x79b492a7, 0x70b999a9, 0x6bae84bb, 0x62a38fb5, 0x5d80be9f, 0x548db591, 0x4f9aa883, 0x4697a38d ]
-
-    def __init__(self, key):
-
-        if len(key) not in (16, 24, 32):
-            raise ValueError('Invalid key size')
-
-        rounds = self.number_of_rounds[len(key)]
-
-        # Encryption round keys
-        self._Ke = [[0] * 4 for i in xrange(rounds + 1)]
-
-        # Decryption round keys
-        self._Kd = [[0] * 4 for i in xrange(rounds + 1)]
-
-        round_key_count = (rounds + 1) * 4
-        KC = len(key) // 4
-
-        # Convert the key into ints
-        tk = [ struct.unpack('>i', key[i:i + 4])[0] for i in xrange(0, len(key), 4) ]
-
-        # Copy values into round key arrays
-        for i in xrange(0, KC):
-            self._Ke[i // 4][i % 4] = tk[i]
-            self._Kd[rounds - (i // 4)][i % 4] = tk[i]
-
-        # Key expansion (fips-197 section 5.2)
-        rconpointer = 0
-        t = KC
-        while t < round_key_count:
-
-            tt = tk[KC - 1]
-            tk[0] ^= ((self.S[(tt >> 16) & 0xFF] << 24) ^
-                      (self.S[(tt >>  8) & 0xFF] << 16) ^
-                      (self.S[ tt        & 0xFF] <<  8) ^
-                       self.S[(tt >> 24) & 0xFF]        ^
-                      (self.rcon[rconpointer] << 24))
-            rconpointer += 1
-
-            if KC != 8:
-                for i in xrange(1, KC):
-                    tk[i] ^= tk[i - 1]
-
-            # Key expansion for 256-bit keys is "slightly different" (fips-197)
-            else:
-                for i in xrange(1, KC // 2):
-                    tk[i] ^= tk[i - 1]
-                tt = tk[KC // 2 - 1]
-
-                tk[KC // 2] ^= (self.S[ tt        & 0xFF]        ^
-                               (self.S[(tt >>  8) & 0xFF] <<  8) ^
-                               (self.S[(tt >> 16) & 0xFF] << 16) ^
-                               (self.S[(tt >> 24) & 0xFF] << 24))
-
-                for i in xrange(KC // 2 + 1, KC):
-                    tk[i] ^= tk[i - 1]
-
-            # Copy values into round key arrays
-            j = 0
-            while j < KC and t < round_key_count:
-                self._Ke[t // 4][t % 4] = tk[j]
-                self._Kd[rounds - (t // 4)][t % 4] = tk[j]
-                j += 1
-                t += 1
-
-        # Inverse-Cipher-ify the decryption round key (fips-197 section 5.3)
-        for r in xrange(1, rounds):
-            for j in xrange(0, 4):
-                tt = self._Kd[r][j]
-                self._Kd[r][j] = (self.U1[(tt >> 24) & 0xFF] ^
-                                  self.U2[(tt >> 16) & 0xFF] ^
-                                  self.U3[(tt >>  8) & 0xFF] ^
-                                  self.U4[ tt        & 0xFF])
-
-    def encrypt(self, plaintext):
-        'Encrypt a block of plain text using the AES block cipher.'
-
-        if len(plaintext) != 16:
-            raise ValueError('wrong block length')
-
-        rounds = len(self._Ke) - 1
-        (s1, s2, s3) = [1, 2, 3]
-        a = [0, 0, 0, 0]
-
-        # Convert plaintext to (ints ^ key)
-        t = [(_compact_word(plaintext[4 * i:4 * i + 4]) ^ self._Ke[0][i]) for i in xrange(0, 4)]
-
-        # Apply round transforms
-        for r in xrange(1, rounds):
-            for i in xrange(0, 4):
-                a[i] = (self.T1[(t[ i          ] >> 24) & 0xFF] ^
-                        self.T2[(t[(i + s1) % 4] >> 16) & 0xFF] ^
-                        self.T3[(t[(i + s2) % 4] >>  8) & 0xFF] ^
-                        self.T4[ t[(i + s3) % 4]        & 0xFF] ^
-                        self._Ke[r][i])
-            t = copy.copy(a)
-
-        # The last round is special
-        result = [ ]
-        for i in xrange(0, 4):
-            tt = self._Ke[rounds][i]
-            result.append((self.S[(t[ i           ] >> 24) & 0xFF] ^ (tt >> 24)) & 0xFF)
-            result.append((self.S[(t[(i + s1) % 4] >> 16) & 0xFF] ^ (tt >> 16)) & 0xFF)
-            result.append((self.S[(t[(i + s2) % 4] >>  8) & 0xFF] ^ (tt >>  8)) & 0xFF)
-            result.append((self.S[ t[(i + s3) % 4]        & 0xFF] ^  tt       ) & 0xFF)
-
-        return result
-
-    def decrypt(self, ciphertext):
-        'Decrypt a block of cipher text using the AES block cipher.'
-
-        if len(ciphertext) != 16:
-            raise ValueError('wrong block length')
-
-        rounds = len(self._Kd) - 1
-        (s1, s2, s3) = [3, 2, 1]
-        a = [0, 0, 0, 0]
-
-        # Convert ciphertext to (ints ^ key)
-        t = [(_compact_word(ciphertext[4 * i:4 * i + 4]) ^ self._Kd[0][i]) for i in xrange(0, 4)]
-
-        # Apply round transforms
-        for r in xrange(1, rounds):
-            for i in xrange(0, 4):
-                a[i] = (self.T5[(t[ i          ] >> 24) & 0xFF] ^
-                        self.T6[(t[(i + s1) % 4] >> 16) & 0xFF] ^
-                        self.T7[(t[(i + s2) % 4] >>  8) & 0xFF] ^
-                        self.T8[ t[(i + s3) % 4]        & 0xFF] ^
-                        self._Kd[r][i])
-            t = copy.copy(a)
-
-        # The last round is special
-        result = [ ]
-        for i in xrange(0, 4):
-            tt = self._Kd[rounds][i]
-            result.append((self.Si[(t[ i           ] >> 24) & 0xFF] ^ (tt >> 24)) & 0xFF)
-            result.append((self.Si[(t[(i + s1) % 4] >> 16) & 0xFF] ^ (tt >> 16)) & 0xFF)
-            result.append((self.Si[(t[(i + s2) % 4] >>  8) & 0xFF] ^ (tt >>  8)) & 0xFF)
-            result.append((self.Si[ t[(i + s3) % 4]        & 0xFF] ^  tt       ) & 0xFF)
-
-        return result
-
-
-class Counter(object):
-    '''A counter object for the Counter (CTR) mode of operation.
-
-       To create a custom counter, you can usually just override the
-       increment method.'''
-
-    def __init__(self, initial_value = 1):
-
-        # Convert the value into an array of bytes long
-        self._counter = [ ((initial_value >> i) % 256) for i in xrange(128 - 8, -1, -8) ]
-
-    value = property(lambda s: s._counter)
-
-    def increment(self):
-        '''Increment the counter (overflow rolls back to 0).'''
-
-        for i in xrange(len(self._counter) - 1, -1, -1):
-            self._counter[i] += 1
-
-            if self._counter[i] < 256: break
-
-            # Carry the one
-            self._counter[i] = 0
-
-        # Overflow
-        else:
-            self._counter = [ 0 ] * len(self._counter)
-
-
-class AESBlockModeOfOperation(object):
-    '''Super-class for AES modes of operation that require blocks.'''
-    def __init__(self, key):
-        self._aes = AES(key)
-
-    def decrypt(self, ciphertext):
-        raise Exception('not implemented')
-
-    def encrypt(self, plaintext):
-        raise Exception('not implemented')
-
-
-class AESStreamModeOfOperation(AESBlockModeOfOperation):
-    '''Super-class for AES modes of operation that are stream-ciphers.'''
-
-class AESSegmentModeOfOperation(AESStreamModeOfOperation):
-    '''Super-class for AES modes of operation that segment data.'''
-
-    segment_bytes = 16
-
-
-
-class AESModeOfOperationECB(AESBlockModeOfOperation):
-    '''AES Electronic Codebook Mode of Operation.
-
-       o Block-cipher, so data must be padded to 16 byte boundaries
-
-   Security Notes:
-       o This mode is not recommended
-       o Any two identical blocks produce identical encrypted values,
-         exposing data patterns. (See the image of Tux on wikipedia)
-
-   Also see:
-       o https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Electronic_codebook_.28ECB.29
-       o See NIST SP800-38A (http://csrc.nist.gov/publications/nistpubs/800-38a/sp800-38a.pdf); section 6.1'''
-
-
-    name = "Electronic Codebook (ECB)"
-
-    def encrypt(self, plaintext):
-        if len(plaintext) != 16:
-            raise ValueError('plaintext block must be 16 bytes')
-
-        plaintext = _string_to_bytes(plaintext)
-        return _bytes_to_string(self._aes.encrypt(plaintext))
-
-    def decrypt(self, ciphertext):
-        if len(ciphertext) != 16:
-            raise ValueError('ciphertext block must be 16 bytes')
-
-        ciphertext = _string_to_bytes(ciphertext)
-        return _bytes_to_string(self._aes.decrypt(ciphertext))
-
-
-
-class AESModeOfOperationCBC(AESBlockModeOfOperation):
-    '''AES Cipher-Block Chaining Mode of Operation.
-
-       o The Initialization Vector (IV)
-       o Block-cipher, so data must be padded to 16 byte boundaries
-       o An incorrect initialization vector will only cause the first
-         block to be corrupt; all other blocks will be intact
-       o A corrupt bit in the cipher text will cause a block to be
-         corrupted, and the next block to be inverted, but all other
-         blocks will be intact.
-
-   Security Notes:
-       o This method (and CTR) ARE recommended.
-
-   Also see:
-       o https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Cipher-block_chaining_.28CBC.29
-       o See NIST SP800-38A (http://csrc.nist.gov/publications/nistpubs/800-38a/sp800-38a.pdf); section 6.2'''
-
-
-    name = "Cipher-Block Chaining (CBC)"
-
-    def __init__(self, key, iv = None):
-        if iv is None:
-            self._last_cipherblock = [ 0 ] * 16
-        elif len(iv) != 16:
-            raise ValueError('initialization vector must be 16 bytes')
-        else:
-            self._last_cipherblock = _string_to_bytes(iv)
-
-        AESBlockModeOfOperation.__init__(self, key)
-
-    def encrypt(self, plaintext):
-        if len(plaintext) != 16:
-            raise ValueError('plaintext block must be 16 bytes')
-
-        plaintext = _string_to_bytes(plaintext)
-        precipherblock = [ (p ^ l) for (p, l) in zip(plaintext, self._last_cipherblock) ]
-        self._last_cipherblock = self._aes.encrypt(precipherblock)
-
-        return _bytes_to_string(self._last_cipherblock)
-
-    def decrypt(self, ciphertext):
-        if len(ciphertext) != 16:
-            raise ValueError('ciphertext block must be 16 bytes')
-
-        cipherblock = _string_to_bytes(ciphertext)
-        plaintext = [ (p ^ l) for (p, l) in zip(self._aes.decrypt(cipherblock), self._last_cipherblock) ]
-        self._last_cipherblock = cipherblock
-
-        return _bytes_to_string(plaintext)
-
-
-
-class AESModeOfOperationCFB(AESSegmentModeOfOperation):
-    '''AES Cipher Feedback Mode of Operation.
-
-       o A stream-cipher, so input does not need to be padded to blocks,
-         but does need to be padded to segment_size
-
-    Also see:
-       o https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Cipher_feedback_.28CFB.29
-       o See NIST SP800-38A (http://csrc.nist.gov/publications/nistpubs/800-38a/sp800-38a.pdf); section 6.3'''
-
-
-    name = "Cipher Feedback (CFB)"
-
-    def __init__(self, key, iv, segment_size = 1):
-        if segment_size == 0: segment_size = 1
-
-        if iv is None:
-            self._shift_register = [ 0 ] * 16
-        elif len(iv) != 16:
-            raise ValueError('initialization vector must be 16 bytes')
-        else:
-          self._shift_register = _string_to_bytes(iv)
-
-        self._segment_bytes = segment_size
-
-        AESBlockModeOfOperation.__init__(self, key)
-
-    segment_bytes = property(lambda s: s._segment_bytes)
-
-    def encrypt(self, plaintext):
-        if len(plaintext) % self._segment_bytes != 0:
-            raise ValueError('plaintext block must be a multiple of segment_size')
-
-        plaintext = _string_to_bytes(plaintext)
-
-        # Break block into segments
-        encrypted = [ ]
-        for i in xrange(0, len(plaintext), self._segment_bytes):
-            plaintext_segment = plaintext[i: i + self._segment_bytes]
-            xor_segment = self._aes.encrypt(self._shift_register)[:len(plaintext_segment)]
-            cipher_segment = [ (p ^ x) for (p, x) in zip(plaintext_segment, xor_segment) ]
-
-            # Shift the top bits out and the ciphertext in
-            self._shift_register = _concat_list(self._shift_register[len(cipher_segment):], cipher_segment)
-
-            encrypted.extend(cipher_segment)
-
-        return _bytes_to_string(encrypted)
-
-    def decrypt(self, ciphertext):
-        if len(ciphertext) % self._segment_bytes != 0:
-            raise ValueError('ciphertext block must be a multiple of segment_size')
-
-        ciphertext = _string_to_bytes(ciphertext)
-
-        # Break block into segments
-        decrypted = [ ]
-        for i in xrange(0, len(ciphertext), self._segment_bytes):
-            cipher_segment = ciphertext[i: i + self._segment_bytes]
-            xor_segment = self._aes.encrypt(self._shift_register)[:len(cipher_segment)]
-            plaintext_segment = [ (p ^ x) for (p, x) in zip(cipher_segment, xor_segment) ]
-
-            # Shift the top bits out and the ciphertext in
-            self._shift_register = _concat_list(self._shift_register[len(cipher_segment):], cipher_segment)
-
-            decrypted.extend(plaintext_segment)
-
-        return _bytes_to_string(decrypted)
-
-
-
-class AESModeOfOperationOFB(AESStreamModeOfOperation):
-    '''AES Output Feedback Mode of Operation.
-
-       o A stream-cipher, so input does not need to be padded to blocks,
-         allowing arbitrary length data.
-       o A bit twiddled in the cipher text, twiddles the same bit in the
-         same bit in the plain text, which can be useful for error
-         correction techniques.
-
-    Also see:
-       o https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Output_feedback_.28OFB.29
-       o See NIST SP800-38A (http://csrc.nist.gov/publications/nistpubs/800-38a/sp800-38a.pdf); section 6.4'''
-
-
-    name = "Output Feedback (OFB)"
-
-    def __init__(self, key, iv = None):
-        if iv is None:
-            self._last_precipherblock = [ 0 ] * 16
-        elif len(iv) != 16:
-            raise ValueError('initialization vector must be 16 bytes')
-        else:
-          self._last_precipherblock = _string_to_bytes(iv)
-
-        self._remaining_block = [ ]
-
-        AESBlockModeOfOperation.__init__(self, key)
-
-    def encrypt(self, plaintext):
-        encrypted = [ ]
-        for p in _string_to_bytes(plaintext):
-            if len(self._remaining_block) == 0:
-                self._remaining_block = self._aes.encrypt(self._last_precipherblock)
-                self._last_precipherblock = [ ]
-            precipherbyte = self._remaining_block.pop(0)
-            self._last_precipherblock.append(precipherbyte)
-            cipherbyte = p ^ precipherbyte
-            encrypted.append(cipherbyte)
-
-        return _bytes_to_string(encrypted)
-
-    def decrypt(self, ciphertext):
-        # AES-OFB is symetric
-        return self.encrypt(ciphertext)
-
-
-
-class AESModeOfOperationCTR(AESStreamModeOfOperation):
-    '''AES Counter Mode of Operation.
-
-       o A stream-cipher, so input does not need to be padded to blocks,
-         allowing arbitrary length data.
-       o The counter must be the same size as the key size (ie. len(key))
-       o Each block independant of the other, so a corrupt byte will not
-         damage future blocks.
-       o Each block has a uniue counter value associated with it, which
-         contributes to the encrypted value, so no data patterns are
-         leaked.
-       o Also known as: Counter Mode (CM), Integer Counter Mode (ICM) and
-         Segmented Integer Counter (SIC
-
-   Security Notes:
-       o This method (and CBC) ARE recommended.
-       o Each message block is associated with a counter value which must be
-         unique for ALL messages with the same key. Otherwise security may be
-         compromised.
-
-    Also see:
-
-       o https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Counter_.28CTR.29
-       o See NIST SP800-38A (http://csrc.nist.gov/publications/nistpubs/800-38a/sp800-38a.pdf); section 6.5
-         and Appendix B for managing the initial counter'''
-
-
-    name = "Counter (CTR)"
-
-    def __init__(self, key, counter = None):
-        AESBlockModeOfOperation.__init__(self, key)
-
-        if counter is None:
-            counter = Counter()
-
-        self._counter = counter
-        self._remaining_counter = [ ]
-
-    def encrypt(self, plaintext):
-        while len(self._remaining_counter) < len(plaintext):
-            self._remaining_counter += self._aes.encrypt(self._counter.value)
-            self._counter.increment()
-
-        plaintext = _string_to_bytes(plaintext)
-
-        encrypted = [ (p ^ c) for (p, c) in zip(plaintext, self._remaining_counter) ]
-        self._remaining_counter = self._remaining_counter[len(encrypted):]
-
-        return _bytes_to_string(encrypted)
-
-    def decrypt(self, crypttext):
-        # AES-CTR is symetric
-        return self.encrypt(crypttext)
-
-
-# Simple lookup table for each mode
-AESModesOfOperation = dict(
-    ctr = AESModeOfOperationCTR,
-    cbc = AESModeOfOperationCBC,
-    cfb = AESModeOfOperationCFB,
-    ecb = AESModeOfOperationECB,
-    ofb = AESModeOfOperationOFB,
-)

Contents/Libraries/Shared/pyaes/blockfeeder.py

@@ -1,227 +0,0 @@
-# The MIT License (MIT)
-#
-# Copyright (c) 2014 Richard Moore
-#
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is
-# furnished to do so, subject to the following conditions:
-#
-# The above copyright notice and this permission notice shall be included in
-# all copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
-# THE SOFTWARE.
-
-
-from .aes import AESBlockModeOfOperation, AESSegmentModeOfOperation, AESStreamModeOfOperation
-from .util import append_PKCS7_padding, strip_PKCS7_padding, to_bufferable
-
-
-# First we inject three functions to each of the modes of operations
-#
-#    _can_consume(size)
-#       - Given a size, determine how many bytes could be consumed in
-#         a single call to either the decrypt or encrypt method
-#
-#    _final_encrypt(data, padding = PADDING_DEFAULT)
-#       - call and return encrypt on this (last) chunk of data,
-#         padding as necessary; this will always be at least 16
-#         bytes unless the total incoming input was less than 16
-#         bytes
-#
-#    _final_decrypt(data, padding = PADDING_DEFAULT)
-#       - same as _final_encrypt except for decrypt, for
-#         stripping off padding
-#
-
-PADDING_NONE       = 'none'
-PADDING_DEFAULT    = 'default'
-
-# @TODO: Ciphertext stealing and explicit PKCS#7
-# PADDING_CIPHERTEXT_STEALING
-# PADDING_PKCS7
-
-# ECB and CBC are block-only ciphers
-
-def _block_can_consume(self, size):
-    if size >= 16: return 16
-    return 0
-
-# After padding, we may have more than one block
-def _block_final_encrypt(self, data, padding = PADDING_DEFAULT):
-    if padding == PADDING_DEFAULT:
-        data = append_PKCS7_padding(data)
-
-    elif padding == PADDING_NONE:
-        if len(data) != 16:
-            raise Exception('invalid data length for final block')
-    else:
-        raise Exception('invalid padding option')
-
-    if len(data) == 32:
-        return self.encrypt(data[:16]) + self.encrypt(data[16:])
-
-    return self.encrypt(data)
-
-
-def _block_final_decrypt(self, data, padding = PADDING_DEFAULT):
-    if padding == PADDING_DEFAULT:
-        return strip_PKCS7_padding(self.decrypt(data))
-
-    if padding == PADDING_NONE:
-        if len(data) != 16:
-            raise Exception('invalid data length for final block')
-        return self.decrypt(data)
-
-    raise Exception('invalid padding option')
-
-AESBlockModeOfOperation._can_consume = _block_can_consume
-AESBlockModeOfOperation._final_encrypt = _block_final_encrypt
-AESBlockModeOfOperation._final_decrypt = _block_final_decrypt
-
-
-
-# CFB is a segment cipher
-
-def _segment_can_consume(self, size):
-    return self.segment_bytes * int(size // self.segment_bytes)
-
-# CFB can handle a non-segment-sized block at the end using the remaining cipherblock
-def _segment_final_encrypt(self, data, padding = PADDING_DEFAULT):
-    if padding != PADDING_DEFAULT:
-        raise Exception('invalid padding option')
-
-    faux_padding = (chr(0) * (self.segment_bytes - (len(data) % self.segment_bytes)))
-    padded = data + to_bufferable(faux_padding)
-    return self.encrypt(padded)[:len(data)]
-
-# CFB can handle a non-segment-sized block at the end using the remaining cipherblock
-def _segment_final_decrypt(self, data, padding = PADDING_DEFAULT):
-    if padding != PADDING_DEFAULT:
-        raise Exception('invalid padding option')
-
-    faux_padding = (chr(0) * (self.segment_bytes - (len(data) % self.segment_bytes)))
-    padded = data + to_bufferable(faux_padding)
-    return self.decrypt(padded)[:len(data)]
-
-AESSegmentModeOfOperation._can_consume = _segment_can_consume
-AESSegmentModeOfOperation._final_encrypt = _segment_final_encrypt
-AESSegmentModeOfOperation._final_decrypt = _segment_final_decrypt
-
-
-
-# OFB and CTR are stream ciphers
-
-def _stream_can_consume(self, size):
-    return size
-
-def _stream_final_encrypt(self, data, padding = PADDING_DEFAULT):
-    if padding not in [PADDING_NONE, PADDING_DEFAULT]:
-        raise Exception('invalid padding option')
-
-    return self.encrypt(data)
-
-def _stream_final_decrypt(self, data, padding = PADDING_DEFAULT):
-    if padding not in [PADDING_NONE, PADDING_DEFAULT]:
-        raise Exception('invalid padding option')
-
-    return self.decrypt(data)
-
-AESStreamModeOfOperation._can_consume = _stream_can_consume
-AESStreamModeOfOperation._final_encrypt = _stream_final_encrypt
-AESStreamModeOfOperation._final_decrypt = _stream_final_decrypt
-
-
-
-class BlockFeeder(object):
-    '''The super-class for objects to handle chunking a stream of bytes
-       into the appropriate block size for the underlying mode of operation
-       and applying (or stripping) padding, as necessary.'''
-
-    def __init__(self, mode, feed, final, padding = PADDING_DEFAULT):
-        self._mode = mode
-        self._feed = feed
-        self._final = final
-        self._buffer = to_bufferable("")
-        self._padding = padding
-
-    def feed(self, data = None):
-        '''Provide bytes to encrypt (or decrypt), returning any bytes
-           possible from this or any previous calls to feed.
-
-           Call with None or an empty string to flush the mode of
-           operation and return any final bytes; no further calls to
-           feed may be made.'''
-
-        if self._buffer is None:
-            raise ValueError('already finished feeder')
-
-        # Finalize; process the spare bytes we were keeping
-        if data is None:
-            result = self._final(self._buffer, self._padding)
-            self._buffer = None
-            return result
-
-        self._buffer += to_bufferable(data)
-
-        # We keep 16 bytes around so we can determine padding
-        result = to_bufferable('')
-        while len(self._buffer) > 16:
-            can_consume = self._mode._can_consume(len(self._buffer) - 16)
-            if can_consume == 0: break
-            result += self._feed(self._buffer[:can_consume])
-            self._buffer = self._buffer[can_consume:]
-
-        return result
-
-
-class Encrypter(BlockFeeder):
-    'Accepts bytes of plaintext and returns encrypted ciphertext.'
-
-    def __init__(self, mode, padding = PADDING_DEFAULT):
-        BlockFeeder.__init__(self, mode, mode.encrypt, mode._final_encrypt, padding)
-
-
-class Decrypter(BlockFeeder):
-    'Accepts bytes of ciphertext and returns decrypted plaintext.'
-
-    def __init__(self, mode, padding = PADDING_DEFAULT):
-        BlockFeeder.__init__(self, mode, mode.decrypt, mode._final_decrypt, padding)
-
-
-# 8kb blocks
-BLOCK_SIZE = (1 << 13)
-
-def _feed_stream(feeder, in_stream, out_stream, block_size = BLOCK_SIZE):
-    'Uses feeder to read and convert from in_stream and write to out_stream.'
-
-    while True:
-        chunk = in_stream.read(block_size)
-        if not chunk:
-            break
-        converted = feeder.feed(chunk)
-        out_stream.write(converted)
-    converted = feeder.feed()
-    out_stream.write(converted)
-
-
-def encrypt_stream(mode, in_stream, out_stream, block_size = BLOCK_SIZE, padding = PADDING_DEFAULT):
-    'Encrypts a stream of bytes from in_stream to out_stream using mode.'
-
-    encrypter = Encrypter(mode, padding = padding)
-    _feed_stream(encrypter, in_stream, out_stream, block_size)
-
-
-def decrypt_stream(mode, in_stream, out_stream, block_size = BLOCK_SIZE, padding = PADDING_DEFAULT):
-    'Decrypts a stream of bytes from in_stream to out_stream using mode.'
-
-    decrypter = Decrypter(mode, padding = padding)
-    _feed_stream(decrypter, in_stream, out_stream, block_size)

Contents/Libraries/Shared/pyaes/util.py

@@ -1,60 +0,0 @@
-# The MIT License (MIT)
-#
-# Copyright (c) 2014 Richard Moore
-#
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is
-# furnished to do so, subject to the following conditions:
-#
-# The above copyright notice and this permission notice shall be included in
-# all copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
-# THE SOFTWARE.
-
-# Why to_bufferable?
-# Python 3 is very different from Python 2.x when it comes to strings of text
-# and strings of bytes; in Python 3, strings of bytes do not exist, instead to
-# represent arbitrary binary data, we must use the "bytes" object. This method
-# ensures the object behaves as we need it to.
-
-def to_bufferable(binary):
-    return binary
-
-def _get_byte(c):
-    return ord(c)
-
-try:
-    xrange
-except:
-
-    def to_bufferable(binary):
-        if isinstance(binary, bytes):
-            return binary
-        return bytes(ord(b) for b in binary)
-
-    def _get_byte(c):
-        return c
-
-def append_PKCS7_padding(data):
-    pad = 16 - (len(data) % 16)
-    return data + to_bufferable(chr(pad) * pad)
-
-def strip_PKCS7_padding(data):
-    if len(data) % 16 != 0:
-        raise ValueError("invalid length")
-
-    pad = _get_byte(data[-1])
-
-    if pad > 16:
-        raise ValueError("invalid padding byte")
-
-    return data[:-pad]

Contents/Libraries/Shared/pysubs2/exceptions.py

@@ -12,6 +12,3 @@ class UnknownFormatIdentifierError(Pysubs2Error):
 
 class FormatAutodetectionError(Pysubs2Error):
     """Subtitle format is ambiguous or unknown."""
-
-class ContentNotUsable(Pysubs2Error):
-    """Current content not usable for specified format"""

Contents/Libraries/Shared/pysubs2/ssastyle.py

@@ -41,7 +41,6 @@ class SSAStyle(object):
         self.italic = False #: Italic
         self.underline = False #: Underline (ASS only)
         self.strikeout = False #: Strikeout (ASS only)
-        self.drawing = False #: Drawing (ASS only, see http://docs.aegisub.org/3.1/ASS_Tags/#drawing-tags
         self.scalex = 100.0 #: Horizontal scaling (ASS only)
         self.scaley = 100.0 #: Vertical scaling (ASS only)
         self.spacing = 0.0 #: Letter spacing (ASS only)

Contents/Libraries/Shared/pysubs2/subrip.py

@@ -5,7 +5,6 @@ from .formatbase import FormatBase
 from .ssaevent import SSAEvent
 from .ssastyle import SSAStyle
 from .substation import parse_tags
-from .exceptions import ContentNotUsable
 from .time import ms_to_times, make_time, TIMESTAMP, timestamp_to_ms
 
 #: Largest timestamp allowed in SubRip, ie. 99:59:59,999.
@@ -82,7 +81,6 @@ class SubripFormat(FormatBase):
                 if sty.italic: fragment = "<i>%s</i>" % fragment
                 if sty.underline: fragment = "<u>%s</u>" % fragment
                 if sty.strikeout: fragment = "<s>%s</s>" % fragment
-                if sty.drawing: raise ContentNotUsable
                 body.append(fragment)
 
             return re.sub("\n+", "\n", "".join(body).strip())
@@ -92,10 +90,7 @@ class SubripFormat(FormatBase):
         for i, line in enumerate(visible_lines, 1):
             start = ms_to_timestamp(line.start)
             end = ms_to_timestamp(line.end)
-            try:
-                text = prepare_text(line.text, subs.styles.get(line.style, SSAStyle.DEFAULT_STYLE))
-            except ContentNotUsable:
-                continue
+            text = prepare_text(line.text, subs.styles.get(line.style, SSAStyle.DEFAULT_STYLE))
 
             print("%d" % i, file=fp) # Python 2.7 compat
             print(start, "-->", end, file=fp)

Contents/Libraries/Shared/pysubs2/substation.py

@@ -110,7 +110,7 @@ def parse_tags(text, style=SSAStyle.DEFAULT_STYLE, styles={}):
     
     def apply_overrides(all_overrides):
         s = style.copy()
-        for tag in re.findall(r"\\[ibusp][0-9]|\\r[a-zA-Z_0-9 ]*", all_overrides):
+        for tag in re.findall(r"\\[ibus][10]|\\r[a-zA-Z_0-9 ]*", all_overrides):
             if tag == r"\r":
                 s = style.copy() # reset to original line style
             elif tag.startswith(r"\r"):
@@ -122,13 +122,6 @@ def parse_tags(text, style=SSAStyle.DEFAULT_STYLE, styles={}):
                 elif "b" in tag: s.bold = "1" in tag
                 elif "u" in tag: s.underline = "1" in tag
                 elif "s" in tag: s.strikeout = "1" in tag
-                elif "p" in tag:
-                    try:
-                        scale = int(tag[2:])
-                    except (ValueError, IndexError):
-                        continue
-
-                    s.drawing = scale > 0
         return s
     
     overrides = SSAEvent.OVERRIDE_SEQUENCE.findall(text)

Contents/Libraries/Shared/subliminal/exceptions.py

@@ -27,8 +27,3 @@ class ServiceUnavailable(ProviderError):
 class DownloadLimitExceeded(ProviderError):
     """Exception raised by providers when download limit is exceeded."""
     pass
-
-
-class DownloadLimitPerDayExceeded(ProviderError):
-    """Exception raised by providers when download limit is exceeded."""
-    pass

Contents/Libraries/Shared/subliminal/extensions.py

@@ -94,9 +94,7 @@ provider_manager = RegistrableExtensionManager('subliminal.providers', [
     'podnapisi = subliminal.providers.podnapisi:PodnapisiProvider',
     'shooter = subliminal.providers.shooter:ShooterProvider',
     'thesubdb = subliminal.providers.thesubdb:TheSubDBProvider',
-    'tvsubtitles = subliminal.providers.tvsubtitles:TVsubtitlesProvider',
-    'ktuvit = subliminal.providers.ktuvit:KtuvitProvider'
-    'wizdom = subliminal.providers.wizdom:WizdomProvider'
+    'tvsubtitles = subliminal.providers.tvsubtitles:TVsubtitlesProvider'
 ])
 
 #: Refiner manager

Contents/Libraries/Shared/subliminal_patch/converters/subscene.py

@@ -12,13 +12,6 @@ from_subscene = {
         'Malay': 'msa', 'Pashto': 'pus', 'Punjabi': 'pan', 'Swahili': 'swa'
 }
 
-from_subscene_with_country = {
-    'Brazillian Portuguese': ('por', 'BR')
-}
-
-to_subscene_with_country = {val: key for key, val in from_subscene_with_country.items()}
-
-
 to_subscene = {v: k for k, v in from_subscene.items()}
 
 exact_languages_alpha3 = [
@@ -41,12 +34,12 @@ language_ids = {
         'mkd': 48, 'mal': 64, 'mni': 65, 'mon': 72, 'pus': 67, 'pol': 31,
         'por': 32, 'pan': 66, 'rus': 34, 'srp': 35, 'sin': 58, 'slk': 36,
         'slv': 37, 'som': 70, 'tgl': 53, 'tam': 59, 'tel': 63, 'tha': 40,
-        'tur': 41, 'ukr': 56, 'urd': 42, 'yor': 71, 'pt-BR': 4
+        'tur': 41, 'ukr': 56, 'urd': 42, 'yor': 71
 }
 
 # TODO: specify codes for unspecified_languages
 unspecified_languages = [
-        'Big 5 code', 'Bulgarian/ English',
+        'Big 5 code', 'Brazillian Portuguese', 'Bulgarian/ English',
         'Chinese BG code', 'Dutch/ English', 'English/ German',
         'Hungarian/ English', 'Rohingya'
 ]
@@ -57,8 +50,6 @@ alpha3_of_code = {l.name: l.alpha3 for l in supported_languages}
 
 supported_languages.update({Language(l) for l in to_subscene})
 
-supported_languages.update({Language(lang, cr) for lang, cr in to_subscene_with_country})
-
 
 class SubsceneConverter(LanguageReverseConverter):
     codes = {l.name for l in supported_languages}
@@ -70,15 +61,9 @@ class SubsceneConverter(LanguageReverseConverter):
         if alpha3 in to_subscene:
             return to_subscene[alpha3]
 
-        if (alpha3, country) in to_subscene_with_country:
-            return to_subscene_with_country[(alpha3, country)]
-
         raise ConfigurationError('Unsupported language for subscene: %s, %s, %s' % (alpha3, country, script))
 
     def reverse(self, code):
-        if code in from_subscene_with_country:
-            return from_subscene_with_country[code]
-
         if code in from_subscene:
             return (from_subscene[code],)
 

Contents/Libraries/Shared/subliminal_patch/converters/titlovi.py

@@ -27,6 +27,16 @@ class TitloviConverter(LanguageReverseConverter):
                            }
         self.codes = set(self.from_titlovi.keys())
 
+        # temporary fix, should be removed as soon as API is used
+        self.lang_from_countrycode = {'ba': ('bos',),
+                                 'en': ('eng',),
+                                 'hr': ('hrv',),
+                                 'mk': ('mkd',),
+                                 'rs': ('srp',),
+                                 'rsc': ('srp', None, 'Cyrl'),
+                                 'si': ('slv',)
+                                 }
+
     def convert(self, alpha3, country=None, script=None):
         if (alpha3, country, script) in self.to_titlovi:
             return self.to_titlovi[(alpha3, country, script)]
@@ -39,5 +49,9 @@ class TitloviConverter(LanguageReverseConverter):
         if titlovi in self.from_titlovi:
             return self.from_titlovi[titlovi]
 
+        # temporary fix, should be removed as soon as API is used
+        if titlovi in self.lang_from_countrycode:
+            return self.lang_from_countrycode[titlovi]
+
         raise ConfigurationError('Unsupported language number for titlovi: %s' % titlovi)
 

Contents/Libraries/Shared/subliminal_patch/core.py

@@ -30,7 +30,7 @@ from subliminal.core import guessit, ProviderPool, io, is_windows_special_path,
     ThreadPoolExecutor, check_video
 from subliminal_patch.exceptions import TooManyRequests, APIThrottled
 
-from subzero.language import Language, ENDSWITH_LANGUAGECODE_RE, FULL_LANGUAGE_LIST
+from subzero.language import Language
 from scandir import scandir, scandir_generic as _scandir_generic
 
 logger = logging.getLogger(__name__)
@@ -62,7 +62,7 @@ class SZProviderPool(ProviderPool):
     def __init__(self, providers=None, provider_configs=None, blacklist=None, throttle_callback=None,
                  pre_download_hook=None, post_download_hook=None, language_hook=None):
         #: Name of providers to use
-        self.providers = providers
+        self.providers = providers or provider_registry.names()
 
         #: Provider configuration
         self.provider_configs = provider_configs or {}
@@ -108,12 +108,10 @@ class SZProviderPool(ProviderPool):
         try:
             logger.info('Terminating provider %s', name)
             self.initialized_providers[name].terminate()
-        except (requests.Timeout, socket.timeout) as e:
+        except (requests.Timeout, socket.timeout):
             logger.error('Provider %r timed out, improperly terminated', name)
-            self.throttle_callback(name, e)
-        except Exception as e:
+        except:
             logger.exception('Provider %r terminated unexpectedly', name)
-            self.throttle_callback(name, e)
 
         del self.initialized_providers[name]
 
@@ -185,13 +183,15 @@ class SZProviderPool(ProviderPool):
 
             return out
 
-        except (requests.Timeout, socket.timeout) as e:
-            logger.exception('Provider %r timed out', provider)
-            self.throttle_callback(provider, e)
+        except (requests.Timeout, socket.timeout):
+            logger.error('Provider %r timed out', provider)
 
-        except Exception as e:
-            logger.exception('Unexpected error in provider %r: %s', provider, traceback.format_exc())
+        except (TooManyRequests, DownloadLimitExceeded, ServiceUnavailable, APIThrottled), e:
             self.throttle_callback(provider, e)
+            return
+
+        except:
+            logger.exception('Unexpected error in provider %r: %s', provider, traceback.format_exc())
 
     def list_subtitles(self, video, languages):
         """List subtitles.
@@ -266,11 +266,10 @@ class SZProviderPool(ProviderPool):
                     requests.exceptions.ProxyError,
                     requests.exceptions.SSLError,
                     requests.Timeout,
-                    socket.timeout) as e:
-                logger.exception('Provider %r connection error', subtitle.provider_name)
-                self.throttle_callback(subtitle.provider_name, e)
+                    socket.timeout):
+                logger.error('Provider %r connection error', subtitle.provider_name)
 
-            except ResponseNotReady as e:
+            except ResponseNotReady:
                 logger.error('Provider %r response error, reinitializing', subtitle.provider_name)
                 try:
                     self[subtitle.provider_name].terminate()
@@ -278,17 +277,20 @@ class SZProviderPool(ProviderPool):
                 except:
                     logger.error('Provider %r reinitialization error: %s', subtitle.provider_name,
                                  traceback.format_exc())
-                    self.throttle_callback(subtitle.provider_name, e)
 
             except rarfile.BadRarFile:
                 logger.error('Malformed RAR file from provider %r, skipping subtitle.', subtitle.provider_name)
                 logger.debug("RAR Traceback: %s", traceback.format_exc())
                 return False
 
-            except Exception as e:
+            except (TooManyRequests, DownloadLimitExceeded, ServiceUnavailable, APIThrottled), e:
+                self.throttle_callback(subtitle.provider_name, e)
+                self.discarded_providers.add(subtitle.provider_name)
+                return False
+
+            except:
                 logger.exception('Unexpected error in provider %r, Traceback: %s', subtitle.provider_name,
                                  traceback.format_exc())
-                self.throttle_callback(subtitle.provider_name, e)
                 self.discarded_providers.add(subtitle.provider_name)
                 return False
 
@@ -307,8 +309,7 @@ class SZProviderPool(ProviderPool):
             logger.error('Invalid subtitle')
             return False
 
-        if not os.environ.get("SZ_KEEP_ENCODING", False):
-            subtitle.normalize()
+        subtitle.normalize()
 
         return True
 
@@ -359,16 +360,15 @@ class SZProviderPool(ProviderPool):
             orig_matches = matches.copy()
 
             logger.debug('%r: Found matches %r', s, matches)
-            score, score_without_hash = compute_score(matches, s, video, hearing_impaired=use_hearing_impaired)
             unsorted_subtitles.append(
-                (s, score, score_without_hash, matches, orig_matches))
+                (s, compute_score(matches, s, video, hearing_impaired=use_hearing_impaired), matches, orig_matches))
 
         # sort subtitles by score
-        scored_subtitles = sorted(unsorted_subtitles, key=operator.itemgetter(1, 2), reverse=True)
+        scored_subtitles = sorted(unsorted_subtitles, key=operator.itemgetter(1), reverse=True)
 
         # download best subtitles, falling back on the next on error
         downloaded_subtitles = []
-        for subtitle, score, score_without_hash, matches, orig_matches in scored_subtitles:
+        for subtitle, score, matches, orig_matches in scored_subtitles:
             # check score
             if score < min_score:
                 logger.info('%r: Score %d is below min_score (%d)', subtitle, score, min_score)
@@ -472,7 +472,7 @@ if is_windows_special_path:
     SZAsyncProviderPool = SZProviderPool
 
 
-def scan_video(path, dont_use_actual_file=False, hints=None, providers=None, skip_hashing=False, hash_from=None):
+def scan_video(path, dont_use_actual_file=False, hints=None, providers=None, skip_hashing=False):
     """Scan a video from a `path`.
 
     patch:
@@ -529,7 +529,7 @@ def scan_video(path, dont_use_actual_file=False, hints=None, providers=None, ski
     video.hints = hints
 
     # get possibly alternative title from the filename itself
-    alt_guess = guessit(filename, options={k: v for k, v in hints.items() if k not in ('expected_title', 'title')})
+    alt_guess = guessit(filename, options=hints)
     if "title" in alt_guess and alt_guess["title"] != guessed_result["title"]:
         if video_type == "episode":
             video.alternative_series.append(alt_guess["title"])
@@ -537,41 +537,28 @@ def scan_video(path, dont_use_actual_file=False, hints=None, providers=None, ski
             video.alternative_titles.append(alt_guess["title"])
         logger.debug("Adding alternative title: %s", alt_guess["title"])
 
-    if dont_use_actual_file and not hash_from:
+    if dont_use_actual_file:
         return video
 
     # size and hashes
     if not skip_hashing:
-        hash_path = hash_from or path
-        video.size = os.path.getsize(hash_path)
+        video.size = os.path.getsize(path)
         if video.size > 10485760:
             logger.debug('Size is %d', video.size)
-            osub_hash = None
-
-            if "bsplayer" in providers:
-                video.hashes['bsplayer'] = osub_hash = hash_opensubtitles(hash_path)
-
             if "opensubtitles" in providers:
-                video.hashes['opensubtitles'] = osub_hash = osub_hash or hash_opensubtitles(hash_path)
-
-            if "opensubtitlescom" in providers:
-                video.hashes['opensubtitlescom'] = osub_hash = osub_hash or hash_opensubtitles(hash_path)
+                video.hashes['opensubtitles'] = hash_opensubtitles(path)
 
             if "shooter" in providers:
-                video.hashes['shooter'] = hash_shooter(hash_path)
+                video.hashes['shooter'] = hash_shooter(path)
 
             if "thesubdb" in providers:
-                video.hashes['thesubdb'] = hash_thesubdb(hash_path)
+                video.hashes['thesubdb'] = hash_thesubdb(path)
 
             if "napiprojekt" in providers:
                 try:
-                    video.hashes['napiprojekt'] = hash_napiprojekt(hash_path)
+                    video.hashes['napiprojekt'] = hash_napiprojekt(path)
                 except MemoryError:
-                    logger.warning(u"Couldn't compute napiprojekt hash for %s", hash_path)
-
-            if "napisy24" in providers:
-                # Napisy24 uses the same hash as opensubtitles
-                video.hashes['napisy24'] = osub_hash or hash_opensubtitles(hash_path)
+                    logger.warning(u"Couldn't compute napiprojekt hash for %s", path)
 
             logger.debug('Computed hashes %r', video.hashes)
         else:
@@ -580,16 +567,14 @@ def scan_video(path, dont_use_actual_file=False, hints=None, providers=None, ski
     return video
 
 
-def _search_external_subtitles(path, languages=None, only_one=False, scandir_generic=False, match_strictness="strict"):
+def _search_external_subtitles(path, languages=None, only_one=False, scandir_generic=False):
     dirpath, filename = os.path.split(path)
     dirpath = dirpath or '.'
-    fn_no_ext, fileext = os.path.splitext(filename)
-    fn_no_ext_lower = fn_no_ext.lower()
+    fileroot, fileext = os.path.splitext(filename)
     subtitles = {}
     _scandir = _scandir_generic if scandir_generic else scandir
-
     for entry in _scandir(dirpath):
-        if (not entry.name or entry.name in ('\x0c', '$', ',', '\x7f')) and not scandir_generic:
+        if not entry.name and not scandir_generic:
             logger.debug('Could not determine the name of the file, retrying with scandir_generic')
             return _search_external_subtitles(path, languages, only_one, True)
         if not entry.is_file(follow_symlinks=False):
@@ -598,11 +583,9 @@ def _search_external_subtitles(path, languages=None, only_one=False, scandir_gen
         p = entry.name
 
         # keep only valid subtitle filenames
-        if not p.lower().endswith(SUBTITLE_EXTENSIONS):
+        if not p.lower().startswith(fileroot.lower()) or not p.lower().endswith(SUBTITLE_EXTENSIONS):
             continue
 
-        # not p.lower().startswith(fileroot.lower()) or not
-
         p_root, p_ext = os.path.splitext(p)
         if not INCLUDE_EXOTIC_SUBS and p_ext not in (".srt", ".ass", ".ssa", ".vtt"):
             continue
@@ -620,34 +603,22 @@ def _search_external_subtitles(path, languages=None, only_one=False, scandir_gen
         if adv_tag:
             forced = "forced" in adv_tag
 
-        # remove possible language code for matching
-        p_root_bare = ENDSWITH_LANGUAGECODE_RE.sub(
-            lambda m: "" if str(m.group(1)).lower() in FULL_LANGUAGE_LIST else m.group(0), p_root)
-
-        p_root_lower = p_root_bare.lower()
-
-        filename_matches = p_root_lower == fn_no_ext_lower
-        filename_contains = p_root_lower in fn_no_ext_lower
-
-        if not filename_matches:
-            if match_strictness == "strict" or (match_strictness == "loose" and not filename_contains):
-                continue
-
-        language = None
-
         # extract the potential language code
-        try:
-            language_code = p_root.rsplit(".", 1)[1].replace('_', '-')
+        language_code = p_root[len(fileroot):].replace('_', '-')[1:]
+
+        # default language is undefined
+        language = Language('und')
+
+        # attempt to parse
+        if language_code:
             try:
                 language = Language.fromietf(language_code)
                 language.forced = forced
-            except (ValueError, LanguageReverseError):
+            except ValueError:
                 logger.error('Cannot parse language code %r', language_code)
-                language_code = None
-        except IndexError:
-            language_code = None
+                language = None
 
-        if not language and not language_code and only_one:
+        elif not language_code and only_one:
             language = Language.rebuild(list(languages)[0], forced=forced)
 
         subtitles[p] = language
@@ -657,7 +628,7 @@ def _search_external_subtitles(path, languages=None, only_one=False, scandir_gen
     return subtitles
 
 
-def search_external_subtitles(path, languages=None, only_one=False, match_strictness="strict"):
+def search_external_subtitles(path, languages=None, only_one=False):
     """
     wrap original search_external_subtitles function to search multiple paths for one given video
     # todo: cleanup and merge with _search_external_subtitles
@@ -678,11 +649,10 @@ def search_external_subtitles(path, languages=None, only_one=False, match_strict
         if os.path.isdir(os.path.dirname(abspath)):
             try:
                 subtitles.update(_search_external_subtitles(abspath, languages=languages,
-                                                            only_one=only_one, match_strictness=match_strictness))
+                                                            only_one=only_one))
             except OSError:
                 subtitles.update(_search_external_subtitles(abspath, languages=languages,
-                                                            only_one=only_one, match_strictness=match_strictness,
-                                                            scandir_generic=True))
+                                                            only_one=only_one, scandir_generic=True))
     logger.debug("external subs: found %s", subtitles)
     return subtitles
 
@@ -875,9 +845,6 @@ def save_subtitles(file_path, subtitles, single=False, directory=None, chmod=Non
             logger.debug(u"Saving %r to %r", subtitle, subtitle_path)
             content = subtitle.get_modified_content(format=format, debug=debug_mods)
             if content:
-                if os.path.exists(subtitle_path):
-                    os.remove(subtitle_path)
-
                 with open(subtitle_path, 'w') as f:
                     f.write(content)
                 subtitle.storage_path = subtitle_path

Contents/Libraries/Shared/subliminal_patch/exceptions.py

@@ -9,8 +9,3 @@ class TooManyRequests(ProviderError):
 
 class APIThrottled(ProviderError):
     pass
-
-
-class ParseResponseError(ProviderError):
-    """Exception raised by providers when they are not able to parse the response."""
-    pass

Contents/Libraries/Shared/subliminal_patch/http.py

@@ -35,8 +35,12 @@ except ImportError:
 from subzero.lib.io import get_viable_encoding
 
 logger = logging.getLogger(__name__)
-pem_file = os.path.normpath(os.path.join(os.path.dirname(os.path.realpath(unicode(__file__, get_viable_encoding()))),
-                                         "..", certifi.where()))
+pem_file = os.path.normpath(os.path.join(os.path.dirname(os.path.realpath(unicode(__file__, get_viable_encoding()))), "..", certifi.where()))
+try:
+    default_ssl_context = ssl.create_default_context(cafile=pem_file)
+except AttributeError:
+    # < Python 2.7.9
+    default_ssl_context = None
 
 
 class TimeoutSession(requests.Session):
@@ -54,9 +58,9 @@ class TimeoutSession(requests.Session):
 
 
 class CertifiSession(TimeoutSession):
-    def __init__(self, verify=None):
+    def __init__(self):
         super(CertifiSession, self).__init__()
-        self.verify = verify or pem_file
+        self.verify = pem_file
 
 
 class NeedsCaptchaException(Exception):
@@ -237,20 +241,12 @@ class SubZeroRequestsTransport(xmlrpclib.SafeTransport):
     # change our user agent to reflect Requests
     user_agent = "Python XMLRPC with Requests (python-requests.org)"
     proxies = None
-    xm_ver = 1
-    session_var = "PHPSESSID"
 
     def __init__(self, use_https=True, verify=None, user_agent=None, timeout=10, *args, **kwargs):
         self.verify = pem_file if verify is None else verify
         self.use_https = use_https
         self.user_agent = user_agent if user_agent is not None else self.user_agent
         self.timeout = timeout
-        self.session = requests.Session()
-        self.session.headers['User-Agent'] = self.user_agent
-        # if 'requests' in self.session.headers['User-Agent']:
-        #     # Set a random User-Agent if no custom User-Agent has been set
-        #     self.session.headers = User_Agent(allow_brotli=False).headers
-
         proxy = os.environ.get('SZ_HTTP_PROXY')
         if proxy:
             self.proxies = {
@@ -264,40 +260,18 @@ class SubZeroRequestsTransport(xmlrpclib.SafeTransport):
         """
         Make an xmlrpc request.
         """
+        headers = {'User-Agent': self.user_agent}
         url = self._build_url(host, handler)
-        cache_key = "xm%s_%s" % (self.xm_ver, host)
-
-        old_sessvar = self.session.cookies.get(self.session_var, "")
-        if not old_sessvar:
-            data = region.get(cache_key)
-            if data is not NO_VALUE:
-                logger.debug("Trying to re-use headers/cookies for %s" % host)
-                self.session.cookies, self.session.headers = data
-                old_sessvar = self.session.cookies.get(self.session_var, "")
-
         try:
-            resp = self.session.post(url, data=request_body,
-                                     stream=True, timeout=self.timeout, proxies=self.proxies,
-                                     verify=self.verify)
-
-            if self.session_var in resp.cookies and resp.cookies[self.session_var] != old_sessvar:
-                logger.debug("Storing %s cookies" % host)
-                region.set(cache_key, [self.session.cookies, self.session.headers])
+            resp = requests.post(url, data=request_body, headers=headers,
+                                 stream=True, timeout=self.timeout, proxies=self.proxies,
+                                 verify=self.verify)
         except ValueError:
-            logger.debug("Wiping cookies/headers cache (VE) for %s" % host)
-            region.delete(cache_key)
             raise
         except Exception:
-            logger.debug("Wiping cookies/headers cache (EX) for %s" % host)
-            region.delete(cache_key)
             raise  # something went wrong
         else:
-            try:
-                resp.raise_for_status()
-            except requests.exceptions.HTTPError:
-                logger.debug("Wiping cookies/headers cache (RE) for %s" % host)
-                region.delete(cache_key)
-                raise
+            resp.raise_for_status()
 
             try:
                 if 'x-ratelimit-remaining' in resp.headers and int(resp.headers['x-ratelimit-remaining']) <= 2:
@@ -380,6 +354,7 @@ def patch_create_connection():
                         return _orig_create_connection((ip, port), *args, **kwargs)
                     except dns.exception.DNSException:
                         logger.warning("DNS: Couldn't resolve %s with DNS: %s", host, custom_resolver.nameservers)
+                        raise
 
         logger.debug("DNS: Falling back to default DNS or IP on %s", host)
         return _orig_create_connection((host, port), *args, **kwargs)

Contents/Libraries/Shared/subliminal_patch/providers/addic7ed.py

@@ -2,20 +2,15 @@
 import logging
 import re
 import datetime
-import types
-
 import subliminal
 import time
 
 from random import randint
-
-from dogpile.cache.api import NO_VALUE
 from requests import Session
 from subliminal.cache import region
-from subliminal.exceptions import DownloadLimitExceeded, AuthenticationError, ConfigurationError, \
-    DownloadLimitPerDayExceeded
+from subliminal.exceptions import DownloadLimitExceeded, AuthenticationError
 from subliminal.providers.addic7ed import Addic7edProvider as _Addic7edProvider, \
-    Addic7edSubtitle as _Addic7edSubtitle, ParserBeautifulSoup
+    Addic7edSubtitle as _Addic7edSubtitle, ParserBeautifulSoup, show_cells_re
 from subliminal.subtitle import fix_line_ending
 from subliminal_patch.utils import sanitize
 from subliminal_patch.exceptions import TooManyRequests
@@ -24,8 +19,6 @@ from subzero.language import Language
 
 logger = logging.getLogger(__name__)
 
-show_cells_re = re.compile(b'<td class="(?:version|vr)">.*?</td>', re.DOTALL)
-
 #: Series header parsing regex
 series_year_re = re.compile(r'^(?P<series>[ \w\'.:(),*&!?-]+?)(?: \((?P<year>\d{4})\))?$')
 
@@ -67,22 +60,16 @@ class Addic7edProvider(_Addic7edProvider):
         'slk', 'slv', 'spa', 'sqi', 'srp', 'swe', 'tha', 'tur', 'ukr', 'vie', 'zho'
     ]} | {Language.fromietf(l) for l in ["sr-Latn", "sr-Cyrl"]}
 
-    vip = False
     USE_ADDICTED_RANDOM_AGENTS = False
     hearing_impaired_verifiable = True
     subtitle_class = Addic7edSubtitle
     server_url = 'https://www.addic7ed.com/'
 
     sanitize_characters = {'-', ':', '(', ')', '.', '/'}
-    last_show_ids_fetch_key = "addic7ed_last_id_fetch"
 
-    def __init__(self, username=None, password=None, use_random_agents=False, is_vip=False):
+    def __init__(self, username=None, password=None, use_random_agents=False):
         super(Addic7edProvider, self).__init__(username=username, password=password)
         self.USE_ADDICTED_RANDOM_AGENTS = use_random_agents
-        self.vip = is_vip
-
-        if not all((username, password)):
-            raise ConfigurationError('Username and password must be specified')
 
     def initialize(self):
         self.session = Session()
@@ -114,18 +101,13 @@ class Addic7edProvider(_Addic7edProvider):
                     'remember': 'true'}
 
             tries = 0
-            while tries <= 3:
-                tries += 1
+            while tries < 3:
                 r = self.session.get(self.server_url + 'login.php', timeout=10, headers={"Referer": self.server_url})
-                if "g-recaptcha" in r.content or "grecaptcha" in r.content:
+                if "grecaptcha" in r.content:
                     logger.info('Addic7ed: Solving captcha. This might take a couple of minutes, but should only '
                                 'happen once every so often')
 
-                    for g, s in (("g-recaptcha-response", r'g-recaptcha.+?data-sitekey=\"(.+?)\"'),
-                                 ("recaptcha_response", r'grecaptcha.execute\(\'(.+?)\',')):
-                        site_key = re.search(s, r.content).group(1)
-                        if site_key:
-                            break
+                    site_key = re.search(r'grecaptcha.execute\(\'(.+?)\',', r.content).group(1)
                     if not site_key:
                         logger.error("Addic7ed: Captcha site-key not found!")
                         return
@@ -137,31 +119,23 @@ class Addic7edProvider(_Addic7edProvider):
 
                     result = pitcher.throw()
                     if not result:
-                        if tries >= 3:
-                            raise Exception("Addic7ed: Couldn't solve captcha!")
-                        logger.info("Addic7ed: Couldn't solve captcha! Retrying")
-                        time.sleep(4)
-                        continue
+                        raise Exception("Addic7ed: Couldn't solve captcha!")
 
-                    data[g] = result
+                    data["recaptcha_response"] = result
 
-                time.sleep(1)
                 r = self.session.post(self.server_url + 'dologin.php', data, allow_redirects=False, timeout=10,
                                       headers={"Referer": self.server_url + "login.php"})
 
                 if "relax, slow down" in r.content:
                     raise TooManyRequests(self.username)
 
-                if "Wrong password" in r.content or "doesn't exist" in r.content:
-                    raise AuthenticationError(self.username)
-
                 if r.status_code != 302:
-                    if tries >= 3:
-                        logger.error("Addic7ed: Something went wrong when logging in")
-                        raise AuthenticationError(self.username)
-                    logger.info("Addic7ed: Something went wrong when logging in; retrying")
-                    time.sleep(4)
-                    continue
+                    if "User <b></b> doesn't exist" in r.content and tries <= 2:
+                        logger.info("Addic7ed: Error, trying again. (%s/%s)", tries+1, 3)
+                        tries += 1
+                        continue
+
+                    raise AuthenticationError(self.username)
                 break
 
             store_verification("addic7ed", self.session)
@@ -169,12 +143,10 @@ class Addic7edProvider(_Addic7edProvider):
             logger.debug('Addic7ed: Logged in')
             self.logged_in = True
 
-            time.sleep(2)
-
     def terminate(self):
         self.session.close()
 
-    def get_show_id(self, series, year=None, country_code=None, ignore_cache=False):
+    def get_show_id(self, series, year=None, country_code=None):
         """Get the best matching show id for `series`, `year` and `country_code`.
 
         First search in the result of :meth:`_get_show_ids` and fallback on a search with :meth:`_search_show_id`.
@@ -186,45 +158,32 @@ class Addic7edProvider(_Addic7edProvider):
         :type country_code: str
         :return: the show id, if found.
         :rtype: int
+
         """
-        show_id = None
-        ids_to_look_for = {sanitize(series).lower(), sanitize(series.replace(".", "")).lower()}
+        series_sanitized = sanitize(series).lower()
         show_ids = self._get_show_ids()
-        if ignore_cache or not show_ids:
-            show_ids = self._get_show_ids.refresh(self)
+        show_id = None
 
-        logger.debug("Trying show ids: %s", ids_to_look_for)
-        for series_sanitized in ids_to_look_for:
-            # attempt with country
-            if not show_id and country_code:
-                logger.debug('Getting show id with country')
-                show_id = show_ids.get('%s %s' % (series_sanitized, country_code.lower()))
+        # attempt with country
+        if not show_id and country_code:
+            logger.debug('Getting show id with country')
+            show_id = show_ids.get('%s %s' % (series_sanitized, country_code.lower()))
 
-            # attempt with year
-            if not show_id and year:
-                logger.debug('Getting show id with year')
-                show_id = show_ids.get('%s %d' % (series_sanitized, year))
+        # attempt with year
+        if not show_id and year:
+            logger.debug('Getting show id with year')
+            show_id = show_ids.get('%s %d' % (series_sanitized, year))
 
-            # attempt clean
-            if not show_id:
-                logger.debug('Getting show id')
-                show_id = show_ids.get(series_sanitized)
+        # attempt clean
+        if not show_id:
+            logger.debug('Getting show id')
+            show_id = show_ids.get(series_sanitized)
 
-                if not show_id:
-                    now = datetime.datetime.now()
-                    last_fetch = region.get(self.last_show_ids_fetch_key)
-
-                    # re-fetch show ids once per day if any show ID not found
-                    if not ignore_cache and last_fetch != NO_VALUE and last_fetch + datetime.timedelta(days=1) < now:
-                        logger.info("Show id not found; re-fetching show ids")
-                        return self.get_show_id(series, year=year, country_code=country_code, ignore_cache=True)
-                    logger.debug("Not refreshing show ids, as the last fetch has been too recent")
-
-            # search as last resort
-            # broken right now
-            # if not show_id:
-            #     logger.warning('Series %s not found in show ids', series)
-            #     show_id = self._search_show_id(series)
+        # search as last resort
+        # broken right now
+        # if not show_id:
+        #     logger.warning('Series %s not found in show ids', series)
+        #     show_id = self._search_show_id(series)
 
         return show_id
 
@@ -238,8 +197,6 @@ class Addic7edProvider(_Addic7edProvider):
         """
         # get the show page
         logger.info('Getting show ids')
-        region.set(self.last_show_ids_fetch_key, datetime.datetime.now())
-
         r = self.session.get(self.server_url + 'shows.php', timeout=10)
         r.raise_for_status()
 
@@ -248,15 +205,14 @@ class Addic7edProvider(_Addic7edProvider):
         # Assuming the site's markup is bad, and stripping it down to only contain what's needed.
         show_cells = re.findall(show_cells_re, r.content)
         if show_cells:
-            soup = ParserBeautifulSoup(b''.join(show_cells).decode('utf-8', 'ignore'), ['lxml', 'html.parser'])
+            soup = ParserBeautifulSoup(b''.join(show_cells), ['lxml', 'html.parser'])
         else:
             # If RegEx fails, fall back to original r.content and use 'html.parser'
             soup = ParserBeautifulSoup(r.content, ['html.parser'])
 
         # populate the show ids
         show_ids = {}
-        shows = soup.select('td > h3 > a[href^="/show/"]')
-        for show in shows:
+        for show in soup.select('td > h3 > a[href^="/show/"]'):
             show_clean = sanitize(show.text, default_characters=self.sanitize_characters)
             try:
                 show_id = int(show['href'][6:])
@@ -274,9 +230,6 @@ class Addic7edProvider(_Addic7edProvider):
 
         logger.debug('Found %d show ids', len(show_ids))
 
-        if not show_ids:
-            raise Exception("Addic7ed: No show IDs found!")
-
         return show_ids
 
     @region.cache_on_arguments(expiration_time=SHOW_EXPIRATION_TIME)
@@ -376,7 +329,7 @@ class Addic7edProvider(_Addic7edProvider):
 
             # ignore incomplete subtitles
             status = cells[5].text
-            if "%" in status:
+            if status != 'Completed':
                 logger.debug('Ignoring subtitle with status %s', status)
                 continue
 
@@ -402,27 +355,6 @@ class Addic7edProvider(_Addic7edProvider):
         return subtitles
 
     def download_subtitle(self, subtitle):
-        last_dls = region.get("addic7ed_dls")
-        now = datetime.datetime.now()
-        one_day = datetime.timedelta(hours=24)
-
-        def raise_limit():
-            logger.info("Addic7ed: Downloads per day exceeded (%s)", cap)
-            raise DownloadLimitPerDayExceeded
-
-        if not isinstance(last_dls, types.ListType):
-            last_dls = []
-        else:
-            # filter all non-expired DLs
-            last_dls = filter(lambda t: t + one_day > now, last_dls)
-            region.set("addic7ed_dls", last_dls)
-
-        cap = self.vip and 80 or 40
-        amount = len(last_dls)
-
-        if amount >= cap:
-            raise_limit()
-
         # download the subtitle
         r = self.session.get(self.server_url + subtitle.download_link, headers={'Referer': subtitle.page_link},
                              timeout=10)
@@ -434,7 +366,7 @@ class Addic7edProvider(_Addic7edProvider):
         if not r.content:
             # Provider wrongful return a status of 304 Not Modified with an empty content
             # raise_for_status won't raise exception for that status code
-            logger.error('Addic7ed: Unable to download subtitle. No data returned from provider')
+            logger.error('Unable to download subtitle. No data returned from provider')
             return
 
         # detect download limit exceeded
@@ -442,10 +374,3 @@ class Addic7edProvider(_Addic7edProvider):
             raise DownloadLimitExceeded
 
         subtitle.content = fix_line_ending(r.content)
-        last_dls.append(datetime.datetime.now())
-        region.set("addic7ed_dls", last_dls)
-        logger.info("Addic7ed: Used %s/%s downloads", amount + 1, cap)
-
-        if amount + 1 >= cap:
-            raise_limit()
-

Contents/Libraries/Shared/subliminal_patch/providers/argenteam.py

@@ -23,10 +23,9 @@ class ArgenteamSubtitle(Subtitle):
     hearing_impaired_verifiable = False
     _release_info = None
 
-    def __init__(self, language, page_link, download_link, movie_kind, title, season, episode, year, release, version, source,
+    def __init__(self, language, download_link, movie_kind, title, season, episode, year, release, version, source,
                  video_codec, tvdb_id, imdb_id, asked_for_episode=None, asked_for_release_group=None, *args, **kwargs):
-        super(ArgenteamSubtitle, self).__init__(language, page_link=page_link, *args, **kwargs)
-        self.page_link = page_link
+        super(ArgenteamSubtitle, self).__init__(language, download_link, *args, **kwargs)
         self.download_link = download_link
         self.movie_kind = movie_kind
         self.title = title
@@ -136,8 +135,7 @@ class ArgenteamProvider(Provider, ProviderSubtitleArchiveMixin):
     provider_name = 'argenteam'
     languages = {Language.fromalpha2(l) for l in ['es']}
     video_types = (Episode, Movie)
-    BASE_URL = "https://argenteam.net/"
-    API_URL = BASE_URL + "api/v1/"
+    API_URL = "http://argenteam.net/api/v1/"
     subtitle_class = ArgenteamSubtitle
     hearing_impaired_verifiable = False
     language_list = list(languages)
@@ -242,15 +240,12 @@ class ArgenteamProvider(Provider, ProviderSubtitleArchiveMixin):
 
             for r in content['releases']:
                 for s in r['subtitles']:
-                    movie_kind = "episode" if is_episode else "movie"
-                    page_link = self.BASE_URL + movie_kind + "/" + str(aid)
-                    # use https and new domain
-                    download_link = s['uri'].replace('http://www.argenteam.net/', self.BASE_URL)
-                    sub = ArgenteamSubtitle(language, page_link, download_link, movie_kind, returned_title,
+                    sub = ArgenteamSubtitle(language, s['uri'], "episode" if is_episode else "movie", returned_title,
                                             season, episode, year, r.get('team'), r.get('tags'),
                                             r.get('source'), r.get('codec'), content.get("tvdb"), imdb_id,
                                             asked_for_release_group=video.release_group,
-                                            asked_for_episode=episode)
+                                            asked_for_episode=episode
+                                            )
                     subtitles.append(sub)
 
             if has_multiple_ids:

Contents/Libraries/Shared/subliminal_patch/providers/bsplayer.py

@@ -1,213 +0,0 @@
-# -*- coding: utf-8 -*-
-from __future__ import absolute_import
-import logging
-import io
-import os
-
-from requests import Session
-from guessit import guessit
-from subliminal_patch.providers import Provider
-from subliminal_patch.subtitle import Subtitle
-from subliminal.utils import sanitize_release_group
-from subliminal.subtitle import guess_matches
-from subzero.language import Language
-
-import gzip
-import random
-from time import sleep
-from xml.etree import ElementTree
-
-logger = logging.getLogger(__name__)
-
-
-class BSPlayerSubtitle(Subtitle):
-    """BSPlayer Subtitle."""
-    provider_name = 'bsplayer'
-    hash_verifiable = True
-
-    def __init__(self, language, filename, subtype, video, link):
-        super(BSPlayerSubtitle, self).__init__(language)
-        self.language = language
-        self.filename = filename
-        self.page_link = link
-        self.subtype = subtype
-        self.video = video
-
-    @property
-    def id(self):
-        return self.page_link
-
-    @property
-    def release_info(self):
-        return self.filename
-
-    def get_matches(self, video):
-        matches = set()
-        matches |= guess_matches(video, guessit(self.filename))
-        matches.add('hash')
-
-        return matches
-
-
-class BSPlayerProvider(Provider):
-    """BSPlayer Provider."""
-    languages = {Language('por', 'BR')} | {Language(l) for l in [
-        'ara', 'bul', 'ces', 'dan', 'deu', 'ell', 'eng', 'fin', 'fra', 'hun', 'ita', 'jpn', 'kor', 'nld', 'pol', 'por',
-        'ron', 'rus', 'spa', 'swe', 'tur', 'ukr', 'zho'
-    ]}
-    SEARCH_THROTTLE = 8
-    hash_verifiable = True
-
-    # batantly based on kodi's bsplayer plugin
-    # also took from BSPlayer-Subtitles-Downloader
-    def __init__(self):
-        self.initialize()
-
-    def initialize(self):
-        self.session = Session()
-        self.search_url = self.get_sub_domain()
-        self.token = None
-        self.login()
-
-    def terminate(self):
-        self.session.close()
-        self.logout()
-
-    def api_request(self, func_name='logIn', params='', tries=5):
-        headers = {
-            'User-Agent': 'BSPlayer/2.x (1022.12360)',
-            'Content-Type': 'text/xml; charset=utf-8',
-            'Connection': 'close',
-            'SOAPAction': '"http://api.bsplayer-subtitles.com/v1.php#{func_name}"'.format(func_name=func_name)
-        }
-        data = (
-            '<?xml version="1.0" encoding="UTF-8"?>\n'
-            '<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" '
-            'xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" '
-            'xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" '
-            'xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:ns1="{search_url}">'
-            '<SOAP-ENV:Body SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">'
-            '<ns1:{func_name}>{params}</ns1:{func_name}></SOAP-ENV:Body></SOAP-ENV:Envelope>'
-        ).format(search_url=self.search_url, func_name=func_name, params=params)
-        logger.info('Sending request: %s.' % func_name)
-        for i in iter(range(tries)):
-            try:
-                self.session.headers.update(headers.items())
-                res = self.session.post(self.search_url, data)
-                return ElementTree.fromstring(res.text)
-
-            except Exception as ex:
-                logger.info("ERROR: %s." % ex)
-                if func_name == 'logIn':
-                    self.search_url = self.get_sub_domain()
-
-                sleep(1)
-        logger.info('ERROR: Too many tries (%d)...' % tries)
-        raise Exception('Too many tries...')
-
-    def login(self):
-        # If already logged in
-        if self.token:
-            return True
-
-        root = self.api_request(
-            func_name='logIn',
-            params=('<username></username>'
-                    '<password></password>'
-                    '<AppID>BSPlayer v2.67</AppID>')
-        )
-        res = root.find('.//return')
-        if res.find('status').text == 'OK':
-            self.token = res.find('data').text
-            logger.info("Logged In Successfully.")
-            return True
-        return False
-
-    def logout(self):
-        # If already logged out / not logged in
-        if not self.token:
-            return True
-
-        root = self.api_request(
-            func_name='logOut',
-            params='<handle>{token}</handle>'.format(token=self.token)
-        )
-        res = root.find('.//return')
-        self.token = None
-        if res.find('status').text == 'OK':
-            logger.info("Logged Out Successfully.")
-            return True
-        return False
-
-    def query(self, video, video_hash, language):
-        if not self.login():
-            return []
-
-        if isinstance(language, (tuple, list, set)):
-            # language_ids = ",".join(language)
-            # language_ids = 'spa'
-            language_ids = ','.join(sorted(l.opensubtitles for l in language))
-
-        if video.imdb_id is None:
-            imdbId = '*'
-        else:
-            imdbId = video.imdb_id
-        sleep(self.SEARCH_THROTTLE)
-        root = self.api_request(
-            func_name='searchSubtitles',
-            params=(
-                '<handle>{token}</handle>'
-                '<movieHash>{movie_hash}</movieHash>'
-                '<movieSize>{movie_size}</movieSize>'
-                '<languageId>{language_ids}</languageId>'
-                '<imdbId>{imdbId}</imdbId>'
-            ).format(token=self.token, movie_hash=video_hash,
-                     movie_size=video.size, language_ids=language_ids, imdbId=imdbId)
-        )
-        res = root.find('.//return/result')
-        if res.find('status').text != 'OK':
-            return []
-
-        items = root.findall('.//return/data/item')
-        subtitles = []
-        if items:
-            logger.info("Subtitles Found.")
-            for item in items:
-                subID = item.find('subID').text
-                subDownloadLink = item.find('subDownloadLink').text
-                subLang = Language.fromopensubtitles(item.find('subLang').text)
-                subName = item.find('subName').text
-                subFormat = item.find('subFormat').text
-                subtitles.append(
-                    BSPlayerSubtitle(subLang, subName, subFormat, video, subDownloadLink)
-                )
-        return subtitles
-
-    def list_subtitles(self, video, languages):
-        return self.query(video, video.hashes['bsplayer'], languages)
-
-    def get_sub_domain(self):
-        # s1-9, s101-109
-        SUB_DOMAINS = ['s1', 's2', 's3', 's4', 's5', 's6', 's7', 's8', 's9',
-                       's101', 's102', 's103', 's104', 's105', 's106', 's107', 's108', 's109']
-        API_URL_TEMPLATE = "http://{sub_domain}.api.bsplayer-subtitles.com/v1.php"
-        sub_domains_end = len(SUB_DOMAINS) - 1
-        return API_URL_TEMPLATE.format(sub_domain=SUB_DOMAINS[random.randint(0, sub_domains_end)])
-
-    def download_subtitle(self, subtitle):
-        session = Session()
-        _addheaders = {
-            'User-Agent': 'Mozilla/4.0 (compatible; Synapse)'
-        }
-        session.headers.update(_addheaders)
-        res = session.get(subtitle.page_link)
-        if res:
-            if res.text == '500':
-                raise ValueError('Error 500 on server')
-
-            with gzip.GzipFile(fileobj=io.BytesIO(res.content)) as gf:
-                subtitle.content = gf.read()
-                subtitle.normalize()
-
-            return subtitle
-        raise ValueError('Problems conecting to the server')

Contents/Libraries/Shared/subliminal_patch/providers/ktuvit.py

@@ -1,417 +0,0 @@
-# -*- coding: utf-8 -*-
-import re
-from requests import Session
-from requests.cookies import RequestsCookieJar
-import json
-import logging
-from subzero.language import Language
-from bs4 import BeautifulSoup
-from guessit import guessit
-
-from subliminal_patch.providers import Provider
-from subliminal.providers import Episode, Movie
-from subliminal_patch.utils import sanitize
-from subliminal_patch.subtitle import Subtitle, guess_matches
-from subliminal.subtitle import fix_line_ending
-from subliminal.exceptions import ConfigurationError, AuthenticationError
-
-from string import hexdigits
-from collections import defaultdict
-import pbkdf2
-from base64 import b64decode, b64encode
-from hashlib import sha256
-import pyaes
-
-__author__ = "Dor Nizar"
-
-logger = logging.getLogger(__name__)
-
-
-class KtuvitSubtitle(Subtitle):
-    provider_name = 'ktuvit'
-
-    def __init__(self, language, title_id, subtitle_id, series, season, episode, release, year):
-        super(KtuvitSubtitle, self).__init__(language, subtitle_id)
-        self.title_id = title_id
-        self.subtitle_id = subtitle_id
-        self.series = series
-        self.season = season
-        self.episode = episode
-        self.release = release
-        self.year = year
-
-    def get_matches(self, video):
-        matches = set()
-        logger.debug("[{}]\n{}".format(self.__class__.__name__, self.__dict__))
-
-        # episode
-        if isinstance(video, Episode):
-            # series
-            if video.series and sanitize(self.series) == sanitize(video.series):
-                matches.add('series')
-            # season
-            if video.season and self.season == video.season:
-                matches.add('season')
-            # episode
-            if video.episode and self.episode == video.episode:
-                matches.add('episode')
-            # guess
-            matches |= guess_matches(video, guessit(self.release, {'type': 'episode'}))
-        # movie
-        elif isinstance(video, Movie):
-            # title
-            if video.title and (sanitize(self.series) in (
-                    sanitize(name) for name in [video.title] + video.alternative_titles)):
-                matches.add('title')
-            # year
-            if video.year and self.year == video.year:
-                matches.add('year')
-            # guess
-            matches |= guess_matches(video, guessit(self.release, {'type': 'movie'}))
-
-        logger.debug("Ktuvit subtitle criteria match:\n{}".format(matches))
-        return matches
-
-    @property
-    def id(self):
-        return self.subtitle_id
-
-
-class KtuvitProvider(Provider):
-    subtitle_class = KtuvitSubtitle
-    languages = {Language.fromalpha2(l) for l in ['he']}
-    URL_SERVER = 'https://www.ktuvit.me/'
-
-    URI_LOGIN = 'Login.aspx'
-    URI_LOGIN_POST = 'Services/MembershipService.svc/Login'
-    URI_SEARCH_TITLE = 'Services/ContentProvider.svc/GetSearchForecast'
-    URI_SEARCH_SERIES_SUBTITLE = 'Services/GetModuleAjax.ashx'
-    URI_SEARCH_MOVIE_SUBTITLE = "MovieInfo.aspx"
-    URI_REQ_SUBTITLE_ID = "Services/ContentProvider.svc/RequestSubtitleDownload"
-    URI_DOWNLOAD_SUBTITLE = "Services/DownloadFile.ashx"
-
-    def __init__(self, username=None, password=None):
-        if not all((username, password)):
-            raise ConfigurationError('Username and password must be specified')
-
-        self.session = None
-        self.username = username
-        self.password = password
-        self.encrypted_password = None
-        self.salt = None
-
-    def encrypt_password(self):
-        logger.debug("password: {}".format(self.password))
-        encrypted_password = KtuvitEncryptor(self.username, self.password, self.salt).encrypt()
-        if not encrypted_password:
-            logger.error("Could not encrypt password")
-            return False
-        self.encrypted_password = encrypted_password
-        return True
-
-    def get_encryption_salt(self):
-        p = re.compile(r"var encryptionSalt = '(.*)';")
-        r = self.session.get(self.URL_SERVER + self.URI_LOGIN)
-        r.raise_for_status()
-
-        logger.debug("Searching for encryptionSalt...")
-        script_list = [i for i in BeautifulSoup(r.content, 'html.parser').select('div#navbar script') if i.contents]
-        for item in script_list:
-            search_salt = p.search(item.contents[0])
-            if search_salt:
-                self.salt = search_salt.group(1)
-                return True
-        logger.error("Could not get encryptionSalt")
-        return False
-
-    def login(self):
-        if not self.get_encryption_salt():
-            return False
-        if not self.encrypt_password():
-            return False
-        data = {
-            "request": {
-                "Email": self.username,
-                "Password": self.encrypted_password
-            }
-        }
-        logger.debug("Trying to log in using: {}".format(json.dumps(data)))
-        r = self.session.post(self.URL_SERVER + self.URI_LOGIN_POST, json=data)
-        r_result = r.json()
-        login_results = ""
-        if 'd' in r_result:
-            try:
-                login_results = json.loads(r_result['d'])
-            except ValueError:
-                logger.error("Could not process JSON from login response")
-                return False
-
-        if 'IsSuccess' not in login_results:
-            logger.error("Login response is different than expected")
-            return False
-
-        if not login_results['IsSuccess']:
-            logger.error("Wrong username or password!")
-            raise AuthenticationError('Wrong username or password!')
-            return False
-
-        if not r.cookies or type(r.cookies) is not RequestsCookieJar:
-            logger.error("Could not get the cookie response of the login")
-            return False
-
-        if 'Login' not in r.cookies.keys():
-            logger.error("Could not found login cookie!")
-            return False
-
-        logger.info("Connected successfully to Ktuvit!")
-        return True
-
-    def initialize(self):
-        logger.debug("Ktuvit initialize")
-        self.session = Session()
-        self.session.headers[
-            'User-Agent'] = 'Mozilla/5.0 (Windows NT 10.0; ' \
-                            'Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36'
-
-        if not self.login():
-            return False
-
-    def terminate(self):
-        logger.debug("Ktuvit terminate")
-        self.session.close()
-
-    def _search_series(self, title):
-        logger.debug("Searching '{}'".format(title))
-        title_request = {
-            "request": {
-                "SearchString": title,
-                "SearchType": "Film"
-            }
-        }
-        r = self.session.post(self.URL_SERVER + self.URI_SEARCH_TITLE, json=title_request, allow_redirects=False,
-                              timeout=10)
-        r.raise_for_status()
-        series_found = r.json()
-        if 'd' in series_found:
-            try:
-                series_found = json.loads(series_found['d'])
-            except ValueError:
-                series_found = None
-        if 'Items' in series_found:
-            return series_found['Items']
-        return []
-
-    def _search_subtitles(self, title_id, season=None, episode=None):
-        if season and episode:
-            params = {
-                'moduleName': 'SubtitlesList',
-                'SeriesID': title_id,
-                'Season': season,
-                'Episode': episode
-            }
-            r = self.session.get(url=self.URL_SERVER + self.URI_SEARCH_SERIES_SUBTITLE, params=params)
-        else:
-            params = {
-                'ID': title_id,
-            }
-            r = self.session.get(url=self.URL_SERVER + self.URI_SEARCH_MOVIE_SUBTITLE, params=params)
-
-        r.raise_for_status()
-        results = r.content
-        if not results:
-            return []
-        subtitles = BeautifulSoup(results, 'html.parser').select('a.fa')
-        logger.debug("[BS4] Elements found:\n{}".format(subtitles))
-        subtitle_list = []
-        for i in subtitles:
-            subtitle_id = i.attrs['data-subtitle-id']
-            release = i.findParent().findParent().text.strip().split('\n')[0]
-            subtitle_list.append((subtitle_id, release))
-
-        return subtitle_list  # [(Subtitle ID, name), (....)]
-
-    def _req_download_identifier(self, title_id, subtitle_id):
-        logger.debug("Request subtitle identifier for: title id: {}, subtitle id: {}".format(title_id, subtitle_id))
-        data = {
-            'request': {
-                'FilmID': title_id,
-                'SubtitleID': subtitle_id,
-                'FontSize': 0,
-                'FontColor': "",
-                'PredefinedLayout': -1
-            }
-        }
-
-        r = self.session.post(self.URL_SERVER + self.URI_REQ_SUBTITLE_ID, json=data, allow_redirects=False,
-                              timeout=10)
-        r.raise_for_status()
-        try:
-            r = json.loads(r.json()['d'])
-        except ValueError:
-            r = {}
-
-        if 'DownloadIdentifier' not in r:
-            logger.error("Download Identifier not found")
-            return None
-        return r['DownloadIdentifier']
-
-    def _download_subtitles(self, download_id):
-        logger.debug("Downloading subtitles by download identifier - {}".format(download_id))
-        data = {'DownloadIdentifier': download_id}
-        r = self.session.get(self.URL_SERVER + self.URI_DOWNLOAD_SUBTITLE, params=data,
-                             timeout=10)
-        r.raise_for_status()
-        if not r.content:
-            logger.debug("Download subtitle failed")
-            return None
-
-        logger.debug("Download subtitle success")
-        return r.content
-
-    def query(self, title, season=None, episode=None, year=None):
-        subtitles = []
-        titles = self._search_series(title)
-        if season and episode:
-            logger.debug("Searching for:\nTitle: {}\nSeason: {}\nEpisode: {}\nYear: {}".format(title, season,
-                                                                                               episode, year))
-        else:
-            logger.debug("Searching for:\nTitle: {}\nYear: {}\n".format(title, year))
-        for title in titles:
-            logger.debug("Title Candidate: {}".format(title))
-            title_id = title['ID']
-            if season and episode:
-                result = self._search_subtitles(title_id, season, episode)
-            else:
-                result = self._search_subtitles(title_id)
-
-            if not result:
-                continue
-
-            for subtitle_id, release in result:
-                subtitles.append(self.subtitle_class(next(iter(self.languages)), title_id, subtitle_id,
-                                                     title['EngName'], season, episode, release, year))
-
-        if subtitles:
-            logger.debug("Found Subtitle Candidates: {}".format(subtitles))
-        return subtitles
-
-    def list_subtitles(self, video, languages):
-        season = episode = year = title = None
-
-        if isinstance(video, Episode):
-            logger.info("list_subtitles Series: {}, season: {}, episode: {}".format(video.series,
-                                                                                    video.season,
-                                                                                    video.episode))
-            title = video.series
-            season = video.season
-            episode = video.episode
-        elif isinstance(video, Movie):
-            logger.info("list_subtitles Movie: {}, year: {}".format(video.title, video.year))
-            title = video.title
-            year = video.year
-
-        return [s for s in self.query(title, season, episode, year) if s.language in languages]
-
-    def download_subtitle(self, subtitle):
-        # type: (KtuvitSubtitle) -> None
-
-        logger.info('Downloading subtitle from Ktuvit: %r', subtitle)
-        download_id = self._req_download_identifier(subtitle.title_id, subtitle.subtitle_id)
-        if not download_id:
-            logger.debug('Unable to retrieve download identifier')
-            return None
-
-        content = self._download_subtitles(download_id)
-        if not content:
-            logger.debug('Unable to download subtitle')
-            return None
-
-        subtitle.content = fix_line_ending(content)
-
-
-class KtuvitEncryptor:
-    def __init__(self, username, password, salt):
-        if not all((username, password, salt)):
-            raise Exception("Encryptor did not get all required arguments")
-
-        self.encrypted_password = None
-        self.username = username
-        self.password = password
-        self.salt = salt
-
-    @staticmethod
-    def rshift(val, n):
-        return (val % 0x100000000) >> n
-
-    @staticmethod
-    def js_parseint(s, rad=10):
-        digits = ''
-        for c in str(s).strip():
-            if c not in hexdigits:
-                break
-            digits += c
-
-        return int(digits, rad) if digits else 0
-
-    @staticmethod
-    def to_signed32(n):
-        n = n & 0xffffffff
-        return n | (-(n & 0x80000000))
-
-    def stringify(self, words, length):
-        sigbytes = int(length / 2) + int((length % 2) > 0)
-        hex_chars = list()
-
-        for i in xrange(0, sigbytes):
-            bite = self.rshift(words[self.rshift(i, 2)], (24 - (i % 4) * 8)) & 0xff
-            hex_chars.append(format(self.rshift(bite, 4), 'x'))
-            hex_chars.append(format(bite & 0x0f, 'x'))
-        return ''.join(hex_chars)
-
-    def cryptojs_hexparse(self, s):
-        words = defaultdict(int)
-        for i in range(0, len(s), 2):
-            tmp1 = (self.js_parseint(s[i:i + 2], 16))
-            tmp2 = (24 - (i % 8) * 4)
-            tmp3 = self.to_signed32(tmp1 << tmp2)
-            words[self.rshift(i, 3)] |= tmp3
-        return self.stringify(words, len(s))
-
-    def cryptojs_pad_iv(self, iv):
-        return str.ljust(self.cryptojs_hexparse(iv), 32, '0').decode('hex')
-
-    @staticmethod
-    def pbkdf2_encrypt(key, salt):
-        a = pbkdf2.PBKDF2(salt, key, 3000)
-        return a.read(16)
-
-    @staticmethod
-    def pad(m):
-        return m + chr(16 - len(m) % 16) * (16 - len(m) % 16)
-
-    def aes_encrypt(self, msg, key, iv):
-        if len(iv) != 16:
-            logger.error("iv (Len: {}) - {} is not 16 length".format(len(iv), iv))
-            return False
-        msg = self.pad(msg)
-
-        aes = pyaes.AESModeOfOperationCBC(key, iv=iv[:16])
-        return b64encode(aes.encrypt(msg))
-
-    def encrypt(self):
-        if not self.salt:
-            logger.error("No salt was instantiated!")
-            return False
-        msg = self.password.encode('utf-8')
-        iv = self.cryptojs_pad_iv(self.username)
-        key = self.pbkdf2_encrypt(self.username, self.salt.encode('utf-8'))
-
-        cipher = self.aes_encrypt(msg, key, iv)
-        if not cipher:
-            return False
-
-        hash_sha256 = sha256(b64decode(cipher))
-        self.encrypted_password = b64encode(hash_sha256.digest())
-        logger.debug("Encrypted password: {}".format(self.encrypted_password))
-        logger.debug("Original password: {}".format(self.password))
-        return self.encrypted_password

Contents/Libraries/Shared/subliminal_patch/providers/napisy24.py

@@ -1,124 +0,0 @@
-import logging
-import os
-from io import BytesIO
-from zipfile import ZipFile
-
-from requests import Session
-
-from subliminal_patch.subtitle import Subtitle
-from subliminal_patch.providers import Provider
-from subliminal import __short_version__
-from subliminal.exceptions import AuthenticationError, ConfigurationError
-from subliminal.subtitle import fix_line_ending
-from subzero.language import Language
-
-logger = logging.getLogger(__name__)
-
-
-class Napisy24Subtitle(Subtitle):
-    '''Napisy24 Subtitle.'''
-    provider_name = 'napisy24'
-
-    def __init__(self, language, hash, imdb_id, napis_id):
-        super(Napisy24Subtitle, self).__init__(language)
-        self.hash = hash
-        self.imdb_id = imdb_id
-        self.napis_id = napis_id
-
-    @property
-    def id(self):
-        return self.hash
-
-    def get_matches(self, video):
-        matches = set()
-
-        # hash
-        if 'napisy24' in video.hashes and video.hashes['napisy24'] == self.hash:
-            matches.add('hash')
-
-        # imdb_id
-        if video.imdb_id and self.imdb_id == video.imdb_id:
-            matches.add('imdb_id')
-
-        return matches
-
-
-class Napisy24Provider(Provider):
-    '''Napisy24 Provider.'''
-    languages = {Language(l) for l in ['pol']}
-    required_hash = 'napisy24'
-    api_url = 'http://napisy24.pl/run/CheckSubAgent.php'
-
-    def __init__(self, username=None, password=None):
-        if all((username, password)):
-            self.username = username
-            self.password = password
-        else:
-            self.username = 'subliminal'
-            self.password = 'lanimilbus'
-
-        self.session = None
-
-    def initialize(self):
-        self.session = Session()
-        self.session.headers['User-Agent'] = 'Subliminal/%s' % __short_version__
-        self.session.headers['Content-Type'] = 'application/x-www-form-urlencoded'
-
-    def terminate(self):
-        self.session.close()
-
-    def query(self, language, size, name, hash):
-        params = {
-            'postAction': 'CheckSub',
-            'ua': self.username,
-            'ap': self.password,
-            'fs': size,
-            'fh': hash,
-            'fn': os.path.basename(name),
-            'n24pref': 1
-        }
-
-        response = self.session.post(self.api_url, data=params, timeout=10)
-        response.raise_for_status()
-
-        response_content = response.content.split(b'||', 1)
-        n24_data = response_content[0].decode()
-
-        if n24_data[:2] != 'OK':
-            if n24_data[:11] == 'login error':
-                raise AuthenticationError('Login failed')
-            logger.error('Unknown response: %s', response.content)
-            return None
-
-        n24_status = n24_data[:4]
-        if n24_status == 'OK-0':
-            logger.info('No subtitles found')
-            return None
-
-        subtitle_info = dict(p.split(':', 1) for p in n24_data.split('|')[1:])
-        logger.debug('Subtitle info: %s', subtitle_info)
-
-        if n24_status == 'OK-1':
-            logger.info('No subtitles found but got video info')
-            return None
-        elif n24_status == 'OK-2':
-            logger.info('Found subtitles')
-        elif n24_status == 'OK-3':
-            logger.info('Found subtitles but not from Napisy24 database')
-            return None
-        
-        subtitle_content = response_content[1]
-
-        subtitle = Napisy24Subtitle(language, hash, 'tt%s' % subtitle_info['imdb'].zfill(7), subtitle_info['napisId'])
-        with ZipFile(BytesIO(subtitle_content)) as zf:
-            subtitle.content = fix_line_ending(zf.open(zf.namelist()[0]).read())
-
-        return subtitle
-
-    def list_subtitles(self, video, languages):
-        subtitles = [self.query(l, video.size, video.name, video.hashes['napisy24']) for l in languages]
-        return [s for s in subtitles if s is not None]
-
-    def download_subtitle(self, subtitle):
-        # there is no download step, content is already filled from listing subtitles
-        pass

Contents/Libraries/Shared/subliminal_patch/providers/opensubtitles.py

@@ -105,7 +105,7 @@ class OpenSubtitlesProvider(ProviderRetryMixin, _OpenSubtitlesProvider):
 
     def __init__(self, username=None, password=None, use_tag_search=False, only_foreign=False, also_foreign=False,
                  skip_wrong_fps=True, is_vip=False, use_ssl=True, timeout=15):
-        if not all((username, password)):
+        if any((username, password)) and not all((username, password)):
             raise ConfigurationError('Username and password must be specified')
 
         self.username = username or ''
@@ -154,7 +154,6 @@ class OpenSubtitlesProvider(ProviderRetryMixin, _OpenSubtitlesProvider):
         logger.debug('Logged in with token %r', self.token[:10]+"X"*(len(self.token)-10))
 
         region.set("os_token", self.token)
-        time.sleep(1)
 
     def use_token_or_login(self, func):
         if not self.token:
@@ -163,7 +162,6 @@ class OpenSubtitlesProvider(ProviderRetryMixin, _OpenSubtitlesProvider):
         try:
             return func()
         except Unauthorized:
-            logger.debug("Token not valid, logging in again")
             self.log_in()
             return func()
 
@@ -199,11 +197,16 @@ class OpenSubtitlesProvider(ProviderRetryMixin, _OpenSubtitlesProvider):
                     return
 
             logger.error("Login failed, please check your credentials")
-            raise
                 
     def terminate(self):
+        if self.token:
+            try:
+                checked(lambda: self.server.LogOut(self.token))
+            except:
+                logger.error("Logout failed: %s", traceback.format_exc())
+
         self.server = None
-        #self.token = None
+        self.token = None
 
     def list_subtitles(self, video, languages):
         """

Contents/Libraries/Shared/subliminal_patch/providers/opensubtitlescom.py

@@ -1,579 +0,0 @@
-# -*- coding: utf-8 -*-
-import logging
-import os
-import time
-import datetime
-import json
-import types
-
-from requests import Session, ConnectionError, Timeout, ReadTimeout, RequestException
-from simplejson import JSONDecodeError
-from subzero.language import Language
-
-from babelfish import language_converters
-from subliminal import Episode, Movie
-from subliminal.score import get_equivalent_release_groups
-from subliminal.utils import sanitize_release_group, sanitize
-from subliminal_patch.exceptions import TooManyRequests, APIThrottled
-from subliminal.exceptions import DownloadLimitExceeded, AuthenticationError, ConfigurationError, ServiceUnavailable, \
-    ProviderError
-from .mixins import ProviderRetryMixin
-from subliminal_patch.subtitle import Subtitle
-from subliminal.subtitle import fix_line_ending, SUBTITLE_EXTENSIONS
-from subliminal_patch.providers import Provider
-from subliminal_patch.subtitle import guess_matches
-from subliminal_patch.utils import fix_inconsistent_naming
-from subliminal.cache import region
-from dogpile.cache.api import NO_VALUE
-from guessit import guessit
-
-logger = logging.getLogger(__name__)
-
-SHOW_EXPIRATION_TIME = datetime.timedelta(weeks=1).total_seconds()
-TOKEN_EXPIRATION_TIME = datetime.timedelta(hours=12).total_seconds()
-
-retry_amount = 3
-
-
-def fix_tv_naming(title):
-    """Fix TV show titles with inconsistent naming using dictionary, but do not sanitize them.
-
-    :param str title: original title.
-    :return: new title.
-    :rtype: str
-
-    """
-    return fix_inconsistent_naming(title, {"Superman & Lois": "Superman and Lois",
-                                           }, True)
-
-
-def fix_movie_naming(title):
-    return fix_inconsistent_naming(title, {
-    }, True)
-
-
-custom_languages = {
-    'pt': 'pt-PT',
-    'zh': 'zh-CN',
-}
-
-
-def to_opensubtitlescom(lang):
-    if lang in custom_languages.keys():
-        return custom_languages[lang]
-    else:
-        return lang
-
-
-def from_opensubtitlescom(lang):
-    from_custom_languages = {v: k for k, v in custom_languages.items()}
-    if lang in from_custom_languages.keys():
-        return from_custom_languages[lang]
-    else:
-        return lang
-
-
-class OpenSubtitlesComSubtitle(Subtitle):
-    provider_name = 'opensubtitlescom'
-    hash_verifiable = True
-    hearing_impaired_verifiable = True
-
-    def __init__(self, language, forced, hearing_impaired, page_link, file_id, releases, uploader, title, year,
-                 hash_matched, file_hash=None, season=None, episode=None, imdb_match=False):
-        super(OpenSubtitlesComSubtitle, self).__init__(language, hearing_impaired, page_link)
-        language = Language.rebuild(language, hi=hearing_impaired, forced=forced)
-
-        self.title = title
-        self.year = year
-        self.season = season
-        self.episode = episode
-        self.releases = releases
-        self.release_info = releases
-        self.language = language
-        self.hearing_impaired = hearing_impaired
-        self.forced = forced
-        self.file_id = file_id
-        self.page_link = page_link
-        self.download_link = None
-        self.uploader = uploader
-        self.matches = None
-        self.hash = file_hash
-        self.encoding = 'utf-8'
-        self.hash_matched = hash_matched
-        self.imdb_match = imdb_match
-
-    @property
-    def id(self):
-        return self.file_id
-
-    def get_matches(self, video):
-        matches = set()
-        type_ = "movie" if isinstance(video, Movie) else "episode"
-
-        # handle movies and series separately
-        if type_ == "episode":
-            # series
-            matches.add('series')
-            # season
-            if video.season == self.season:
-                matches.add('season')
-            # episode
-            if video.episode == self.episode:
-                matches.add('episode')
-            # imdb
-            if self.imdb_match:
-                matches.add('series_imdb_id')
-        else:
-            # title
-            matches.add('title')
-            # imdb
-            if self.imdb_match:
-                matches.add('imdb_id')
-
-        # rest is same for both groups
-
-        # year
-        if video.year == self.year:
-            matches.add('year')
-
-        # release_group
-        if (video.release_group and self.releases and
-                any(r in sanitize_release_group(self.releases)
-                    for r in get_equivalent_release_groups(sanitize_release_group(video.release_group)))):
-            matches.add('release_group')
-
-        if self.hash_matched:
-            matches.add('hash')
-
-        # other properties
-        matches |= guess_matches(video, guessit(self.releases, {"type": type_}))
-
-        self.matches = matches
-
-        return matches
-
-
-class OpenSubtitlesComProvider(ProviderRetryMixin, Provider):
-    """OpenSubtitlesCom Provider"""
-    server_url = 'https://api.opensubtitles.com/api/v1/'
-
-    languages = {Language.fromopensubtitles(lang) for lang in language_converters['szopensubtitles'].codes}
-    languages.update(set(Language.rebuild(lang, forced=True) for lang in languages))
-    languages.update(set(Language.rebuild(l, hi=True) for l in languages))
-
-    video_types = (Episode, Movie)
-
-    def __init__(self, username=None, password=None, use_hash=True, include_ai_translated=False, api_key=None):
-        if not all((username, password)):
-            raise ConfigurationError('Username and password must be specified')
-
-        if not api_key:
-            raise ConfigurationError('Api_key must be specified')
-
-        if not all((username, password)):
-            raise ConfigurationError('Username and password must be specified')
-
-        self.session = Session()
-        self.session.headers = {'User-Agent': os.environ.get("SZ_USER_AGENT", "Sub-Zero/2"),
-                                'Api-Key': api_key,
-                                'Content-Type': 'application/json'}
-        self.token = None
-        self.username = username
-        self.password = password
-        self.video = None
-        self.use_hash = use_hash
-        self.include_ai_translated = include_ai_translated
-        self._started = None
-
-    def initialize(self):
-        self._started = time.time()
-
-        if region.get("oscom_token", expiration_time=TOKEN_EXPIRATION_TIME) is NO_VALUE:
-            logger.debug("No cached token, we'll try to login again.")
-            self.login()
-        else:
-            self.token = region.get("oscom_token", expiration_time=TOKEN_EXPIRATION_TIME)
-
-    def terminate(self):
-        self.session.close()
-
-    def ping(self):
-        return self._started and (time.time() - self._started) < TOKEN_EXPIRATION_TIME
-
-    def login(self, is_retry=False):
-        r = self.checked(
-            lambda: self.session.post(self.server_url + 'login',
-                                      json={"username": self.username, "password": self.password},
-                                      allow_redirects=False,
-                                      timeout=30),
-            is_retry=is_retry)
-
-        try:
-            self.token = r.json()['token']
-        except (ValueError, JSONDecodeError):
-            log_request_response(r)
-            raise ProviderError("Cannot get token from provider login response")
-        else:
-            log_request_response(r, non_standard=False)
-            region.set("oscom_token", self.token)
-
-    @staticmethod
-    def sanitize_external_ids(external_id):
-        if isinstance(external_id, types.StringTypes):
-            external_id = external_id.lower().lstrip('tt').lstrip('0')
-        sanitized_id = external_id[:-1].lstrip('0') + external_id[-1]
-        return int(sanitized_id)
-
-    @region.cache_on_arguments(expiration_time=SHOW_EXPIRATION_TIME)
-    def search_titles(self, title):
-        title_id = None
-
-        parameters = {'query': title.lower()}
-        logging.debug('Searching using this title: %s' % title)
-
-        results = self.retry(
-            lambda: self.checked(
-                lambda: self.session.get(self.server_url + 'features', params=parameters, timeout=30),
-                validate_json=True,
-                json_key_name='data'
-            ),
-            amount=retry_amount
-        )
-
-        # deserialize results
-        results_dict = results.json()['data']
-
-        # loop over results
-        for result in results_dict:
-            if 'title' in result['attributes']:
-                if isinstance(self.video, Episode):
-                    if fix_tv_naming(title).lower() == result['attributes']['title'].lower() and \
-                            (not self.video.year or self.video.year == int(result['attributes']['year'])):
-                        title_id = result['id']
-                        break
-                else:
-                    if fix_movie_naming(title).lower() == result['attributes']['title'].lower() and \
-                            (not self.video.year or self.video.year == int(result['attributes']['year'])):
-                        title_id = result['id']
-                        break
-            else:
-                continue
-
-        if title_id:
-            logging.debug('Found this title ID: %s' % title_id)
-            return self.sanitize_external_ids(title_id)
-
-        if not title_id:
-            logger.debug('No match found for %s' % title)
-
-    def query(self, languages, video):
-        self.video = video
-        if self.use_hash:
-            file_hash = self.video.hashes.get('opensubtitlescom')
-            logging.debug('Searching using this hash: %s' % hash)
-        else:
-            file_hash = None
-
-        if isinstance(self.video, Episode):
-            title = self.video.series
-        else:
-            title = self.video.title
-
-        imdb_id = None
-        if isinstance(self.video, Episode) and self.video.series_imdb_id:
-            imdb_id = self.sanitize_external_ids(self.video.series_imdb_id)
-        elif isinstance(self.video, Movie) and self.video.imdb_id:
-            imdb_id = self.sanitize_external_ids(self.video.imdb_id)
-
-        title_id = None
-        if not imdb_id:
-            title_id = self.search_titles(title)
-            if not title_id:
-                return []
-
-        # be sure to remove duplicates using list(set())
-        langs_list = sorted(list(set([to_opensubtitlescom(lang.basename).lower() for lang in languages])))
-
-        langs = ','.join(langs_list)
-        logging.debug('Searching for those languages: %s' % langs)
-
-        # query the server
-        if isinstance(self.video, Episode):
-            res = self.retry(
-                lambda: self.checked(
-                    lambda: self.session.get(self.server_url + 'subtitles',
-                                             params=(('ai_translated', 'exclude' if not self.include_ai_translated
-                                                     else 'include'),
-                                                     ('episode_number', self.video.episode),
-                                                     ('imdb_id', imdb_id if not title_id else None),
-                                                     ('languages', langs),
-                                                     ('moviehash', file_hash),
-                                                     ('parent_feature_id', title_id if title_id else None),
-                                                     ('season_number', self.video.season)),
-                                             timeout=30),
-                    validate_json=True,
-                    json_key_name='data'
-                ),
-                amount=retry_amount
-            )
-        else:
-            res = self.retry(
-                lambda: self.checked(
-                    lambda: self.session.get(self.server_url + 'subtitles',
-                                             params=(('ai_translated', 'exclude' if not self.include_ai_translated
-                                                     else 'include'),
-                                                     ('id', title_id if title_id else None),
-                                                     ('imdb_id', imdb_id if not title_id else None),
-                                                     ('languages', langs),
-                                                     ('moviehash', file_hash)),
-                                             timeout=30),
-                    validate_json=True,
-                    json_key_name='data'
-                ),
-                amount=retry_amount
-            )
-
-        subtitles = []
-
-        result = res.json()
-
-        # filter out forced subtitles or not depending on the required languages
-        if all([lang.forced for lang in languages]):  # only forced
-            result['data'] = [x for x in result['data'] if x['attributes']['foreign_parts_only']]
-        elif any([lang.forced for lang in languages]):  # also forced
-            pass
-        else:  # not forced
-            result['data'] = [x for x in result['data'] if not x['attributes']['foreign_parts_only']]
-
-        logging.debug("Query returned %s subtitles" % len(result['data']))
-
-        if len(result['data']):
-            for item in result['data']:
-                # ignore AI translated subtitles
-                if 'ai_translated' in item['attributes'] and item['attributes']['ai_translated']:
-                    logging.debug("Skipping AI translated subtitles")
-                    continue
-
-                # ignore machine translated subtitles
-                if 'machine_translated' in item['attributes'] and item['attributes']['machine_translated']:
-                    logging.debug("Skipping machine translated subtitles")
-                    continue
-
-                if 'season_number' in item['attributes']['feature_details']:
-                    season_number = item['attributes']['feature_details']['season_number']
-                else:
-                    season_number = None
-
-                if 'episode_number' in item['attributes']['feature_details']:
-                    episode_number = item['attributes']['feature_details']['episode_number']
-                else:
-                    episode_number = None
-
-                if 'moviehash_match' in item['attributes']:
-                    moviehash_match = item['attributes']['moviehash_match']
-                else:
-                    moviehash_match = False
-
-                try:
-                    year = int(item['attributes']['feature_details']['year'])
-                except TypeError:
-                    year = item['attributes']['feature_details']['year']
-
-                if len(item['attributes']['files']):
-                    subtitle = OpenSubtitlesComSubtitle(
-                        language=Language.fromietf(from_opensubtitlescom(item['attributes']['language'])),
-                        forced=item['attributes']['foreign_parts_only'],
-                        hearing_impaired=item['attributes']['hearing_impaired'],
-                        page_link=item['attributes']['url'],
-                        file_id=item['attributes']['files'][0]['file_id'],
-                        releases=item['attributes']['release'],
-                        uploader=item['attributes']['uploader']['name'],
-                        title=item['attributes']['feature_details']['movie_name'],
-                        year=year,
-                        season=season_number,
-                        episode=episode_number,
-                        hash_matched=moviehash_match,
-                        imdb_match=True if imdb_id else False
-                    )
-                    subtitle.get_matches(self.video)
-                    subtitles.append(subtitle)
-
-        return subtitles
-
-    def list_subtitles(self, video, languages):
-        return self.query(languages, video)
-
-    def download_subtitle(self, subtitle):
-        logger.info('Downloading subtitle %r', subtitle)
-
-        headers = {'Accept': 'application/json', 'Content-Type': 'application/json',
-                   'Authorization': 'Bearer ' + self.token}
-        res = self.retry(
-            lambda: self.checked(
-                lambda: self.session.post(self.server_url + 'download',
-                                          json={'file_id': subtitle.file_id, 'sub_format': 'srt'},
-                                          headers=headers,
-                                          timeout=30),
-                validate_json=True,
-                json_key_name='link'
-            ),
-            amount=retry_amount
-        )
-
-        download_data = res.json()
-        subtitle.download_link = download_data['link']
-
-        r = self.retry(
-            lambda: self.checked(
-                lambda: self.session.get(subtitle.download_link, timeout=30),
-                validate_content=True
-            ),
-            amount=retry_amount
-        )
-
-        if not r:
-            logger.debug('Could not download subtitle from %s' % subtitle.download_link)
-            subtitle.content = None
-            return
-        else:
-            subtitle_content = r.content
-            subtitle.content = fix_line_ending(subtitle_content)
-
-    @staticmethod
-    def reset_token():
-        logging.debug('Authentication failed: clearing cache and attempting to login.')
-        region.delete("oscom_token")
-        return
-
-    def checked(self, fn, raise_api_limit=False, validate_json=False, json_key_name=None, validate_content=False,
-                is_retry=False):
-        """Run :fn: and check the response status before returning it.
-
-        :param fn: the function to make an API call to OpenSubtitles.com.
-        :param raise_api_limit: if True we wait a little bit longer before running the call again.
-        :param validate_json: test if response is valid json.
-        :param json_key_name: test if returned json contain a specific key.
-        :param validate_content: test if response have a content (used with download).
-        :param is_retry: prevent additional retries with login endpoint.
-        :return: the response.
-
-        """
-        response = None
-        try:
-            try:
-                response = fn()
-            except APIThrottled:
-                if not raise_api_limit:
-                    logger.info("API request limit hit, waiting and trying again once.")
-                    time.sleep(15)
-                    return self.checked(fn, raise_api_limit=True)
-                raise
-            except (ConnectionError, Timeout, ReadTimeout):
-                raise ServiceUnavailable('Unknown Error, empty response: {}: {}'.format(response.status_code, response))
-            except Exception:
-                logging.exception('Unhandled exception raised.')
-                raise ProviderError('Unhandled exception raised. Check log.')
-            else:
-                status_code = response.status_code
-        except Exception:
-            status_code = None
-        else:
-            if status_code == 400:
-                try:
-                    json_response = response.json()
-                    message = json_response['message']
-                except JSONDecodeError:
-                    raise ProviderError('Invalid JSON returned by provider')
-                else:
-                    log_request_response(response)
-                    raise ConfigurationError(message)
-            elif status_code == 401:
-                log_request_response(response)
-                self.reset_token()
-                if is_retry:
-                    raise AuthenticationError('Login failed')
-                else:
-                    time.sleep(1)
-                    self.login(is_retry=True)
-                    self.checked(fn, raise_api_limit=raise_api_limit, validate_json=validate_json,
-                                 json_key_name=json_key_name, validate_content=validate_content, is_retry=True)
-            elif status_code == 403:
-                log_request_response(response)
-                raise ProviderError("Bazarr API key seems to be in problem")
-            elif status_code == 406:
-                try:
-                    json_response = response.json()
-                    download_count = json_response['requests']
-                    remaining_download = json_response['remaining']
-                    quota_reset_time = json_response['reset_time']
-                except JSONDecodeError:
-                    raise ProviderError('Invalid JSON returned by provider')
-                else:
-                    log_request_response(response)
-                    raise DownloadLimitExceeded("Daily download limit reached. {} subtitles have been "
-                                                "downloaded and {} remaining subtitles can be "
-                                                "downloaded. Quota will be reset in {}.".format(download_count, remaining_download, quota_reset_time))
-            elif status_code == 410:
-                log_request_response(response)
-                raise ProviderError("Download as expired")
-            elif status_code == 429:
-                log_request_response(response)
-                raise TooManyRequests()
-            elif status_code == 500:
-                logging.debug("Server side exception raised while downloading from opensubtitles.com website. They "
-                              "should mitigate this soon.")
-                return None
-            elif status_code == 502:
-                # this one should deal with Bad Gateway issue on their side.
-                raise APIThrottled()
-            elif 500 <= status_code <= 599:
-                raise ProviderError(response.reason)
-
-            if status_code != 200:
-                log_request_response(response)
-                raise ProviderError('Bad status code: %s' % response.status_code)
-
-            if validate_json:
-                try:
-                    json_test = response.json()
-                except JSONDecodeError:
-                    raise ProviderError('Invalid JSON returned by provider')
-                else:
-                    if json_key_name not in json_test:
-                        raise ProviderError('Invalid JSON returned by provider: no %s key in returned json.' % json_key_name)
-
-            if validate_content:
-                if not hasattr(response, 'content'):
-                    logging.error('Download link returned no content attribute.')
-                    return False
-                elif not response.content:
-                    logging.error('This download link returned empty content: %s' % response.url)
-                    return False
-
-        return response
-
-
-def log_request_response(response, non_standard=True):
-    redacted_request_headers = response.request.headers
-    if 'Authorization' in redacted_request_headers and isinstance(redacted_request_headers['Authorization'], str):
-        redacted_request_headers['Authorization'] = redacted_request_headers['Authorization'][:-8]+8*'x'
-
-    redacted_request_body = json.loads(response.request.body)
-    if 'password' in redacted_request_body:
-        redacted_request_body['password'] = 'redacted'
-
-    redacted_response_body = json.loads(response.text)
-    if 'token' in redacted_response_body and isinstance(redacted_response_body['token'], str):
-        redacted_response_body['token'] = redacted_response_body['token'][:-8] + 8 * 'x'
-
-    if non_standard:
-        logging.debug("opensubtitlescom returned a non standard response. Logging request/response for debugging "
-                      "purpose.")
-    else:
-        logging.debug("opensubtitlescom returned a standard response. Logging request/response for debugging purpose.")
-    logging.debug("Request URL: %s" % response.request.url)
-    logging.debug("Request Headers: %s" % redacted_request_headers)
-    logging.debug("Request Body: %s" % json.dumps(redacted_request_body))
-    logging.debug("Response Status Code: %s" % {response.status_code})
-    logging.debug("Response Headers: %s" % response.headers)
-    logging.debug("Response Body: %s" % json.dumps(redacted_response_body))

Contents/Libraries/Shared/subliminal_patch/providers/subscene.py

@@ -4,34 +4,24 @@ import io
 import logging
 import os
 import time
-import traceback
-
-import requests
-
 import inflect
-import re
-import json
-import HTMLParser
-import urlparse
 
 from zipfile import ZipFile
 from babelfish import language_converters
 from guessit import guessit
-from dogpile.cache.api import NO_VALUE
 from subliminal import Episode, ProviderError
-from subliminal.exceptions import ConfigurationError, ServiceUnavailable
 from subliminal.utils import sanitize_release_group
-from subliminal.cache import region
 from subliminal_patch.http import RetryingCFSession
 from subliminal_patch.providers import Provider
 from subliminal_patch.providers.mixins import ProviderSubtitleArchiveMixin
 from subliminal_patch.subtitle import Subtitle, guess_matches
 from subliminal_patch.converters.subscene import language_ids, supported_languages
-from subscene_api.subscene import search, Subtitle as APISubtitle, SITE_DOMAIN
+from subscene_api.subscene import search, Subtitle as APISubtitle
 from subzero.language import Language
 
 p = inflect.engine()
 
+
 language_converters.register('subscene = subliminal_patch.converters.subscene:SubsceneConverter')
 logger = logging.getLogger(__name__)
 
@@ -122,106 +112,28 @@ class SubsceneProvider(Provider, ProviderSubtitleArchiveMixin):
     skip_wrong_fps = False
     hearing_impaired_verifiable = True
     only_foreign = False
-    username = None
-    password = None
 
-    search_throttle = 8  # seconds
-
-    def __init__(self, only_foreign=False, username=None, password=None):
-        if not all((username, password)):
-            raise ConfigurationError('Username and password must be specified')
+    search_throttle = 2  # seconds
 
+    def __init__(self, only_foreign=False):
         self.only_foreign = only_foreign
-        self.username = username
-        self.password = password
 
     def initialize(self):
         logger.info("Creating session")
         self.session = RetryingCFSession()
 
-        prev_cookies = region.get("subscene_cookies2")
-        if prev_cookies != NO_VALUE:
-            logger.debug("Re-using old subscene cookies: %r", prev_cookies)
-            self.session.cookies.update(prev_cookies)
-
-        else:
-            logger.debug("Logging in")
-            self.login()
-
-    def login(self):
-        r = self.session.get("https://subscene.com/account/login")
-        if "Server Error" in r.content:
-            logger.error("Login unavailable; Maintenance?")
-            raise ServiceUnavailable("Login unavailable; Maintenance?")
-
-        match = re.search(r"<script id='modelJson' type='application/json'>\s*(.+)\s*</script>", r.content)
-
-        if match:
-            h = HTMLParser.HTMLParser()
-            data = json.loads(h.unescape(match.group(1)))
-            login_url = urlparse.urljoin(data["siteUrl"], data["loginUrl"])
-            time.sleep(1.0)
-
-            r = self.session.post(login_url,
-                                  {
-                                      "username": self.username,
-                                      "password": self.password,
-                                      data["antiForgery"]["name"]: data["antiForgery"]["value"]
-                                  })
-            pep_content = re.search(r"<form method=\"post\" action=\"https://subscene\.com/\">"
-                                    r".+name=\"id_token\".+?value=\"(?P<id_token>.+?)\".*?"
-                                    r"access_token\".+?value=\"(?P<access_token>.+?)\".+?"
-                                    r"token_type.+?value=\"(?P<token_type>.+?)\".+?"
-                                    r"expires_in.+?value=\"(?P<expires_in>.+?)\".+?"
-                                    r"scope.+?value=\"(?P<scope>.+?)\".+?"
-                                    r"state.+?value=\"(?P<state>.+?)\".+?"
-                                    r"session_state.+?value=\"(?P<session_state>.+?)\"",
-                                    r.content, re.MULTILINE | re.DOTALL)
-
-            if pep_content:
-                r = self.session.post(SITE_DOMAIN, pep_content.groupdict())
-                try:
-                    r.raise_for_status()
-                except Exception:
-                    raise ProviderError("Something went wrong when trying to log in: %s", traceback.format_exc())
-                else:
-                    cj = self.session.cookies.copy()
-                    store_cks = ("scene", "idsrv", "idsrv.xsrf", "idsvr.clients", "idsvr.session", "idsvr.username")
-                    for cn in self.session.cookies.iterkeys():
-                        if cn not in store_cks:
-                            del cj[cn]
-
-                    logger.debug("Storing cookies: %r", cj)
-                    region.set("subscene_cookies2", cj)
-                    return
-        raise ProviderError("Something went wrong when trying to log in #1")
-
     def terminate(self):
         logger.info("Closing session")
         self.session.close()
 
     def _create_filters(self, languages):
         self.filters = dict(HearingImpaired="2")
-        acc_filters = self.filters.copy()
         if self.only_foreign:
             self.filters["ForeignOnly"] = "True"
-            acc_filters["ForeignOnly"] = self.filters["ForeignOnly"].lower()
             logger.info("Only searching for foreign/forced subtitles")
 
-        selected_ids = []
-        for l in languages:
-            lid = language_ids.get(l.basename, language_ids.get(l.alpha3, None))
-            if lid:
-                selected_ids.append(str(lid))
-
-        acc_filters["SelectedIds"] = selected_ids
-        self.filters["LanguageFilter"] = ",".join(acc_filters["SelectedIds"])
-
-        last_filters = region.get("subscene_filters")
-        if last_filters != acc_filters:
-            region.set("subscene_filters", acc_filters)
-            logger.debug("Setting account filters to %r", acc_filters)
-            self.session.post("https://u.subscene.com/filter", acc_filters, allow_redirects=False)
+        self.filters["LanguageFilter"] = ",".join((str(language_ids[l.alpha3]) for l in languages
+                                                   if l.alpha3 in language_ids))
 
         logger.debug("Filter created: '%s'" % self.filters)
 
@@ -264,11 +176,7 @@ class SubsceneProvider(Provider, ProviderSubtitleArchiveMixin):
     def parse_results(self, video, film):
         subtitles = []
         for s in film.subtitles:
-            try:
-                subtitle = SubsceneSubtitle.from_api(s)
-            except NotImplementedError, e:
-                logger.info(e)
-                continue
+            subtitle = SubsceneSubtitle.from_api(s)
             subtitle.asked_for_release_group = video.release_group
             if isinstance(video, Episode):
                 subtitle.asked_for_episode = video.episode
@@ -281,16 +189,10 @@ class SubsceneProvider(Provider, ProviderSubtitleArchiveMixin):
 
         return subtitles
 
-    def do_search(self, *args, **kwargs):
-        try:
-            return search(*args, **kwargs)
-        except requests.HTTPError:
-            region.delete("subscene_cookies2")
-
     def query(self, video):
-        # vfn = get_video_filename(video)
+        vfn = get_video_filename(video)
         subtitles = []
-        # logger.debug(u"Searching for: %s", vfn)
+        #logger.debug(u"Searching for: %s", vfn)
         # film = search(vfn, session=self.session)
         #
         # if film and film.subtitles:
@@ -299,17 +201,16 @@ class SubsceneProvider(Provider, ProviderSubtitleArchiveMixin):
         # else:
         #     logger.debug('No release results found')
 
-        # time.sleep(self.search_throttle)
+        #time.sleep(self.search_throttle)
 
         # re-search for episodes without explicit release name
         if isinstance(video, Episode):
-            titles = list(set([video.series] + video.alternative_series))[:2]
-            # term = u"%s S%02iE%02i" % (video.series, video.season, video.episode)
-            more_than_one = len(titles) > 1
-            for series in titles:
+            #term = u"%s S%02iE%02i" % (video.series, video.season, video.episode)
+            more_than_one = len([video.series] + video.alternative_series) > 1
+            for series in [video.series] + video.alternative_series:
                 term = u"%s - %s Season" % (series, p.number_to_words("%sth" % video.season).capitalize())
                 logger.debug('Searching for alternative results: %s', term)
-                film = self.do_search(term, session=self.session, release=False, throttle=self.search_throttle)
+                film = search(term, session=self.session, release=False, throttle=self.search_throttle)
                 if film and film.subtitles:
                     logger.debug('Alternative results found: %s', len(film.subtitles))
                     subtitles += self.parse_results(video, film)
@@ -317,27 +218,26 @@ class SubsceneProvider(Provider, ProviderSubtitleArchiveMixin):
                     logger.debug('No alternative results found')
 
                 # packs
-                # if video.season_fully_aired:
-                #     term = u"%s S%02i" % (series, video.season)
-                #     logger.debug('Searching for packs: %s', term)
-                #     time.sleep(self.search_throttle)
-                #     film = search(term, session=self.session, throttle=self.search_throttle)
-                #     if film and film.subtitles:
-                #         logger.debug('Pack results found: %s', len(film.subtitles))
-                #         subtitles += self.parse_results(video, film)
-                #     else:
-                #         logger.debug('No pack results found')
-                # else:
-                #     logger.debug("Not searching for packs, because the season hasn't fully aired")
+                if video.season_fully_aired:
+                    term = u"%s S%02i" % (series, video.season)
+                    logger.debug('Searching for packs: %s', term)
+                    time.sleep(self.search_throttle)
+                    film = search(term, session=self.session, throttle=self.search_throttle)
+                    if film and film.subtitles:
+                        logger.debug('Pack results found: %s', len(film.subtitles))
+                        subtitles += self.parse_results(video, film)
+                    else:
+                        logger.debug('No pack results found')
+                else:
+                    logger.debug("Not searching for packs, because the season hasn't fully aired")
                 if more_than_one:
                     time.sleep(self.search_throttle)
         else:
-            titles = list(set([video.title] + video.alternative_titles))[:2]
-            more_than_one = len(titles) > 1
-            for title in titles:
-                logger.debug('Searching for movie results: %r', title)
-                film = self.do_search(title, year=video.year, session=self.session, limit_to=None, release=False,
-                                      throttle=self.search_throttle)
+            more_than_one = len([video.title] + video.alternative_titles) > 1
+            for title in [video.title] + video.alternative_titles:
+                logger.debug('Searching for movie results: %s', title)
+                film = search(title, year=video.year, session=self.session, limit_to=None, release=False,
+                              throttle=self.search_throttle)
                 if film and film.subtitles:
                     subtitles += self.parse_results(video, film)
                 if more_than_one:

Contents/Libraries/Shared/subliminal_patch/providers/supersubtitles.py

@@ -143,7 +143,7 @@ class SuperSubtitlesProvider(Provider, ProviderSubtitleArchiveMixin):
     ]}
     video_types = (Episode, Movie)
     # https://www.feliratok.info/?search=&soriSorszam=&nyelv=&sorozatnev=The+Flash+%282014%29&sid=3212&complexsearch=true&knyelv=0&evad=4&epizod1=1&cimke=0&minoseg=0&rlsr=0&tab=all
-    server_url = 'https://www.feliratok.eu/'
+    server_url = 'https://www.feliratok.info/'
     subtitle_class = SuperSubtitlesSubtitle
     hearing_impaired_verifiable = False
     multi_result_throttle = 2  # seconds

Contents/Libraries/Shared/subliminal_patch/providers/titlovi.py

@@ -2,35 +2,42 @@
 
 import io
 import logging
+import math
 import re
-from datetime import datetime
-import dateutil.parser
+import time
 
 import rarfile
 
+from bs4 import BeautifulSoup
 from zipfile import ZipFile, is_zipfile
 from rarfile import RarFile, is_rarfile
 from babelfish import language_converters, Script
-from requests import RequestException, codes as request_codes
+from requests import RequestException
 from guessit import guessit
 from subliminal_patch.http import RetryingCFSession
 from subliminal_patch.providers import Provider
 from subliminal_patch.providers.mixins import ProviderSubtitleArchiveMixin
 from subliminal_patch.subtitle import Subtitle
 from subliminal_patch.utils import sanitize, fix_inconsistent_naming as _fix_inconsistent_naming
-from subliminal.exceptions import ProviderError, AuthenticationError, ConfigurationError
+from subliminal.exceptions import ProviderError
 from subliminal.score import get_equivalent_release_groups
 from subliminal.utils import sanitize_release_group
 from subliminal.subtitle import guess_matches
 from subliminal.video import Episode, Movie
 from subliminal.subtitle import fix_line_ending
-
+from subliminal_patch.pitcher import pitchers, load_verification, store_verification
 from subzero.language import Language
-from dogpile.cache.api import NO_VALUE
-from subliminal.cache import region
+
+from random import randint
+from .utils import FIRST_THOUSAND_OR_SO_USER_AGENTS as AGENT_LIST
 
 # parsing regex definitions
 title_re = re.compile(r'(?P<title>(?:.+(?= [Aa][Kk][Aa] ))|.+)(?:(?:.+)(?P<altitle>(?<= [Aa][Kk][Aa] ).+))?')
+lang_re = re.compile(r'(?<=flags/)(?P<lang>.{2})(?:.)(?P<script>c?)(?:.+)')
+season_re = re.compile(r'Sezona (?P<season>\d+)')
+episode_re = re.compile(r'Epizoda (?P<episode>\d+)')
+year_re = re.compile(r'(?P<year>\d+)')
+fps_re = re.compile(r'fps: (?P<fps>.+)')
 
 
 def fix_inconsistent_naming(title):
@@ -44,7 +51,6 @@ def fix_inconsistent_naming(title):
     return _fix_inconsistent_naming(title, {"DC's Legends of Tomorrow": "Legends of Tomorrow",
                                             "Marvel's Jessica Jones": "Jessica Jones"})
 
-
 logger = logging.getLogger(__name__)
 
 # Configure :mod:`rarfile` to use the same path separator as :mod:`zipfile`
@@ -56,9 +62,9 @@ language_converters.register('titlovi = subliminal_patch.converters.titlovi:Titl
 class TitloviSubtitle(Subtitle):
     provider_name = 'titlovi'
 
-    def __init__(self, language, download_link, sid, releases, title, alt_title=None, season=None,
-                 episode=None, year=None, rating=None, download_count=None, asked_for_release_group=None, asked_for_episode=None):
-        super(TitloviSubtitle, self).__init__(language)
+    def __init__(self, language, page_link, download_link, sid, releases, title, alt_title=None, season=None,
+                 episode=None, year=None, fps=None, asked_for_release_group=None, asked_for_episode=None):
+        super(TitloviSubtitle, self).__init__(language, page_link=page_link)
         self.sid = sid
         self.releases = self.release_info = releases
         self.title = title
@@ -67,21 +73,11 @@ class TitloviSubtitle(Subtitle):
         self.episode = episode
         self.year = year
         self.download_link = download_link
-        self.rating = rating
-        self.download_count = download_count
+        self.fps = fps
         self.matches = None
         self.asked_for_release_group = asked_for_release_group
         self.asked_for_episode = asked_for_episode
 
-    def __repr__(self):
-        if self.season and self.episode:
-            return '<%s "%s (%r)" s%.2de%.2d [%s:%s] ID:%r R:%.2f D:%r>' % (
-                self.__class__.__name__, self.title, self.year, self.season, self.episode, self.language, self._guessed_encoding, self.sid,
-                self.rating, self.download_count)
-        else:
-            return '<%s "%s (%r)" [%s:%s] ID:%r R:%.2f D:%r>' % (
-                self.__class__.__name__, self.title, self.year,  self.language, self._guessed_encoding, self.sid, self.rating, self.download_count)
-
     @property
     def id(self):
         return self.sid
@@ -138,62 +134,20 @@ class TitloviSubtitle(Subtitle):
 class TitloviProvider(Provider, ProviderSubtitleArchiveMixin):
     subtitle_class = TitloviSubtitle
     languages = {Language.fromtitlovi(l) for l in language_converters['titlovi'].codes} | {Language.fromietf('sr-Latn')}
-    api_url = 'https://kodi.titlovi.com/api/subtitles'
-    api_gettoken_url = api_url + '/gettoken'
-    api_search_url = api_url + '/search'
-
-    def __init__(self, username=None, password=None):
-        if not all((username, password)):
-            raise ConfigurationError('Username and password must be specified')
-
-        self.username = username
-        self.password = password
-
-        self.session = None
-
-        self.user_id = None
-        self.login_token = None
-        self.token_exp = None
+    server_url = 'https://titlovi.com'
+    search_url = server_url + '/titlovi/?'
+    download_url = server_url + '/download/?type=1&mediaid='
 
     def initialize(self):
         self.session = RetryingCFSession()
         #load_verification("titlovi", self.session)
 
-        token = region.get("titlovi_token")
-        if token is not NO_VALUE:
-            self.user_id, self.login_token, self.token_exp = token
-            if datetime.now() > self.token_exp:
-                logger.debug('Token expired')
-                self.log_in()
-            else:
-                logger.debug('Use cached token')
-        else:
-            logger.debug('Token not found in cache')
-            self.log_in()
-
-    def log_in(self):
-        login_params = dict(username=self.username, password=self.password, json=True)
-        try:
-            response = self.session.post(self.api_gettoken_url, params=login_params)
-            if response.status_code == request_codes.ok:
-                resp_json = response.json()
-                self.login_token = resp_json.get('Token')
-                self.user_id = resp_json.get('UserId')
-                self.token_exp = dateutil.parser.parse(resp_json.get('ExpirationDate'))
-
-                region.set("titlovi_token", [self.user_id, self.login_token, self.token_exp])
-                logger.debug('New token obtained')
-
-            elif response.status_code == request_codes.unauthorized:
-                raise AuthenticationError('Login failed')
-
-        except RequestException as e:
-            logger.error(e)
     def terminate(self):
         self.session.close()
 
-    def query(self, languages, title, season=None, episode=None, year=None, imdb_id=None, video=None):
-        search_params = dict()
+    def query(self, languages, title, season=None, episode=None, year=None, video=None):
+        items_per_page = 10
+        current_page = 1
 
         used_languages = languages
         lang_strings = [str(lang) for lang in used_languages]
@@ -208,73 +162,135 @@ class TitloviProvider(Provider, ProviderSubtitleArchiveMixin):
         langs = '|'.join(map(str, [l.titlovi for l in used_languages]))
 
         # set query params
-        search_params['query'] = title
-        search_params['lang'] = langs
+        params = {'prijevod': title, 'jezik': langs}
         is_episode = False
         if season and episode:
             is_episode = True
-            search_params['season'] = season
-            search_params['episode'] = episode
-        #if year:
-        #    search_params['year'] = year
-        if imdb_id:
-            search_params['imdbID'] = imdb_id
+            params['s'] = season
+            params['e'] = episode
+        if year:
+            params['g'] = year
 
         # loop through paginated results
-        logger.info('Searching subtitles %r', search_params)
+        logger.info('Searching subtitles %r', params)
         subtitles = []
-        query_results = []
 
-        try:
-            search_params['token'] = self.login_token
-            search_params['userid'] = self.user_id
-            search_params['json'] = True
-
-            response = self.session.get(self.api_search_url, params=search_params)
-            resp_json = response.json()
-            if resp_json['SubtitleResults']:
-                query_results.extend(resp_json['SubtitleResults'])
-
-
-        except Exception as e:
-            logger.error(e)
-
-        for sub in query_results:
-
-            # title and alternate title
-            match = title_re.search(sub.get('Title'))
-            if match:
-                _title = match.group('title')
-                alt_title = match.group('altitle')
+        while True:
+            # query the server
+            try:
+                r = self.session.get(self.search_url, params=params, timeout=10)
+                r.raise_for_status()
+            except RequestException as e:
+                logger.exception('RequestException %s', e)
+                break
             else:
-                continue
+                try:
+                    soup = BeautifulSoup(r.content, 'lxml')
 
-            # handle movies and series separately
-            if is_episode:
-                subtitle = self.subtitle_class(Language.fromtitlovi(sub.get('Lang')), sub.get('Link'), sub.get('Id'), sub.get('Release'), _title,
-                                               alt_title=alt_title, season=sub.get('Season'), episode=sub.get('Episode'),
-                                               year=sub.get('Year'), rating=sub.get('Rating'),
-                                               download_count=sub.get('DownloadCount'),
-                                               asked_for_release_group=video.release_group,
-                                               asked_for_episode=episode)
-            else:
-                subtitle = self.subtitle_class(Language.fromtitlovi(sub.get('Lang')), sub.get('Link'), sub.get('Id'), sub.get('Release'), _title,
-                                               alt_title=alt_title, year=sub.get('Year'), rating=sub.get('Rating'),
-                                               download_count=sub.get('DownloadCount'),
-                                               asked_for_release_group=video.release_group)
-            logger.debug('Found subtitle %r', subtitle)
+                    # number of results
+                    result_count = int(soup.select_one('.results_count b').string)
+                except:
+                    result_count = None
 
-            # prime our matches so we can use the values later
-            subtitle.get_matches(video)
+                # exit if no results
+                if not result_count:
+                    if not subtitles:
+                        logger.debug('No subtitles found')
+                    else:
+                        logger.debug("No more subtitles found")
+                    break
 
-            # add found subtitles
-            subtitles.append(subtitle)
+                # number of pages with results
+                pages = int(math.ceil(result_count / float(items_per_page)))
+
+                # get current page
+                if 'pg' in params:
+                    current_page = int(params['pg'])
+
+                try:
+                    sublist = soup.select('section.titlovi > ul.titlovi > li.subtitleContainer.canEdit')
+                    for sub in sublist:
+                        # subtitle id
+                        sid = sub.find(attrs={'data-id': True}).attrs['data-id']
+                        # get download link
+                        download_link = self.download_url + sid
+                        # title and alternate title
+                        match = title_re.search(sub.a.string)
+                        if match:
+                            _title = match.group('title')
+                            alt_title = match.group('altitle')
+                        else:
+                            continue
+
+                        # page link
+                        page_link = self.server_url + sub.a.attrs['href']
+                        # subtitle language
+                        _lang = sub.select_one('.lang')
+                        match = lang_re.search(_lang.attrs.get('src', _lang.attrs.get('cfsrc', '')))
+                        if match:
+                            try:
+                                # decode language
+                                lang = Language.fromtitlovi(match.group('lang')+match.group('script'))
+                            except ValueError:
+                                continue
+
+                        # relase year or series start year
+                        match = year_re.search(sub.find(attrs={'data-id': True}).parent.i.string)
+                        if match:
+                            r_year = int(match.group('year'))
+                        # fps
+                        match = fps_re.search(sub.select_one('.fps').string)
+                        if match:
+                            fps = match.group('fps')
+                        # releases
+                        releases = str(sub.select_one('.fps').parent.contents[0].string)
+
+                        # handle movies and series separately
+                        if is_episode:
+                            # season and episode info
+                            sxe = sub.select_one('.s0xe0y').string
+                            r_season = None
+                            r_episode = None
+                            if sxe:
+                                match = season_re.search(sxe)
+                                if match:
+                                    r_season = int(match.group('season'))
+                                match = episode_re.search(sxe)
+                                if match:
+                                    r_episode = int(match.group('episode'))
+
+                            subtitle = self.subtitle_class(lang, page_link, download_link, sid, releases, _title,
+                                                           alt_title=alt_title, season=r_season, episode=r_episode,
+                                                           year=r_year, fps=fps,
+                                                           asked_for_release_group=video.release_group,
+                                                           asked_for_episode=episode)
+                        else:
+                            subtitle = self.subtitle_class(lang, page_link, download_link, sid, releases, _title,
+                                                           alt_title=alt_title, year=r_year, fps=fps,
+                                                           asked_for_release_group=video.release_group)
+                        logger.debug('Found subtitle %r', subtitle)
+
+                        # prime our matches so we can use the values later
+                        subtitle.get_matches(video)
+
+                        # add found subtitles
+                        subtitles.append(subtitle)
+
+                finally:
+                    soup.decompose()
+
+                # stop on last page
+                if current_page >= pages:
+                    break
+
+                # increment current page
+                params['pg'] = current_page + 1
+                logger.debug('Getting page %d', params['pg'])
 
         return subtitles
 
     def list_subtitles(self, video, languages):
         season = episode = None
-
         if isinstance(video, Episode):
             title = video.series
             season = video.season
@@ -284,7 +300,6 @@ class TitloviProvider(Provider, ProviderSubtitleArchiveMixin):
 
         return [s for s in
                 self.query(languages, fix_inconsistent_naming(title), season=season, episode=episode, year=video.year,
-                           imdb_id=video.imdb_id,
                            video=video)]
 
     def download_subtitle(self, subtitle):
@@ -322,12 +337,10 @@ class TitloviProvider(Provider, ProviderSubtitleArchiveMixin):
         sub_to_extract = None
 
         for sub_name in subs_in_archive:
-            _sub_name = sub_name.lower()
-
-            if not ('.cyr' in _sub_name or '.cir' in _sub_name or 'cyr)' in _sub_name):
+            if not ('.cyr' in sub_name or '.cir' in sub_name):
                 sr_lat_subs.append(sub_name)
 
-            if ('.cyr' in sub_name or '.cir' in _sub_name) and not '.lat' in _sub_name.lower():
+            if ('.cyr' in sub_name or '.cir' in sub_name) and not '.lat' in sub_name:
                 sr_cyr_subs.append(sub_name)
 
         if subtitle.language == 'sr':

Contents/Libraries/Shared/subliminal_patch/providers/wizdom.py

@@ -1,235 +0,0 @@
-# -*- coding: utf-8 -*-
-from requests import Session
-import logging
-from subzero.language import Language
-from guessit import guessit
-
-from subliminal_patch.providers import Provider
-from subliminal.providers import Episode, Movie
-from subliminal_patch.utils import sanitize
-from subliminal_patch.subtitle import Subtitle, guess_matches
-from subliminal.subtitle import fix_line_ending
-
-from io import BytesIO
-from zipfile import ZipFile
-
-__author__ = "Dor Nizar"
-
-logger = logging.getLogger(__name__)
-
-
-class WizdomSubtitle(Subtitle):
-    provider_name = 'wizdom'
-
-    def __init__(self, language, title_id, subtitle_id, series, season, episode, release, year, page_link):
-        super(WizdomSubtitle, self).__init__(language, subtitle_id)
-        self.title_id = title_id
-        self.subtitle_id = subtitle_id
-        self.series = series
-        self.season = season
-        self.episode = episode
-        self.release = release
-        self.year = year
-        self.page_link = page_link
-
-    def get_matches(self, video):
-        matches = set()
-        logger.debug("[{}]\n{}".format(self.__class__.__name__, self.__dict__))
-
-        # episode
-        if isinstance(video, Episode):
-            # series
-            if video.series and sanitize(self.series) == sanitize(video.series):
-                matches.add('series')
-            # season
-            if video.season and self.season == video.season:
-                matches.add('season')
-            # episode
-            if video.episode and self.episode == video.episode:
-                matches.add('episode')
-            # guess
-            matches |= guess_matches(video, guessit(self.release, {'type': 'episode'}))
-        # movie
-        elif isinstance(video, Movie):
-            # title
-            if video.title and (sanitize(self.series) in (
-                    sanitize(name) for name in [video.title] + video.alternative_titles)):
-                matches.add('title')
-            # year
-            if video.year and self.year == video.year:
-                matches.add('year')
-            # guess
-            matches |= guess_matches(video, guessit(self.release, {'type': 'movie'}))
-
-        logger.debug("Wizdom subtitle criteria match:\n{}".format(matches))
-        return matches
-
-    @property
-    def id(self):
-        return self.subtitle_id
-
-
-class WizdomProvider(Provider):
-    subtitle_class = WizdomSubtitle
-    languages = {Language.fromalpha2(lng) for lng in ['he']}
-    URL_JSON_SERVER = 'https://json.wizdom.xyz/'
-    URL_DOWNLOAD_SERVER = 'https://zip.wizdom.xyz/'
-    URL_WIZDOM_PAGELINK = 'https://wizdom.xyz/{}/{}'
-
-    URL_SEARCH = URL_JSON_SERVER + "search.php?search={}"
-    URL_INFO = URL_JSON_SERVER + "{}.json"
-    URL_DOWNLOAD_SUBTITLE = URL_DOWNLOAD_SERVER + "{}.zip"
-
-    def __init__(self):
-        self.session = None
-
-    def initialize(self):
-        logger.info("Wizdom initialize")
-        self.session = Session()
-        self.session.headers[
-            'User-Agent'] = 'Mozilla/5.0 (Windows NT 10.0; ' \
-                            'Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36'
-        return True
-
-    def terminate(self):
-        logger.info("Wizdom terminate")
-        self.session.close()
-
-    def _search_series(self, title, movie=False):
-        logger.debug("Searching '{}'".format(title))
-        r = self.session.get(self.URL_SEARCH.format(title))
-        if not r.ok:
-            logger.error("Bad response from server while searching: '{}']".format(title))
-            return []
-        try:
-            found = r.json()
-        except ValueError:
-            logger.error("Could not extract JSON from response")
-            return []
-
-        if movie:
-            series_list = [x for x in found if 'type' in x and x['type'] == 'movie']
-        else:
-            series_list = [x for x in found if 'type' in x and x['type'] == 'tv']
-
-        logger.debug("Found the following titles: {}".format([x['title_en'] for x in series_list if 'title_en' in x]))
-        return series_list
-
-    def _search_subtitles(self, title_id, season=None, episode=None):
-        r = self.session.get(self.URL_INFO.format(title_id))
-
-        if not r.ok:
-            logger.error("Bad response from server while searching subtitles [title_id={}]".format(title_id))
-            return []
-        if not r.content:
-            return []
-        try:
-            results = r.json()
-        except ValueError:
-            return []
-
-        if 'subs' not in results:
-            return []
-
-        if season and episode:
-            s_season = str(season)
-            s_episode = str(episode)
-            if s_season in results['subs'] and s_episode in results['subs'][s_season]:
-                return results['subs'][s_season][s_episode]
-        else:
-            return results['subs']
-        return []
-
-    def query(self, title, season=None, episode=None, year=None):
-        subtitles = []
-        if season and episode:
-            logger.debug("Searching for:\nTitle: {}\nSeason: {}\nEpisode: {}\nYear: {}".format(title, season,
-                                                                                               episode, year))
-            titles = self._search_series(title)
-        else:
-            logger.debug("Searching for:\nTitle: {}\nYear: {}\n".format(title, year))
-            titles = self._search_series(title, movie=True)
-
-        for title in titles:
-            title_name = title['title_en']
-            title_id = title['imdb']
-            if season and episode:
-                results = self._search_subtitles(title_id, season, episode)
-                page_link = self.URL_WIZDOM_PAGELINK.format("tv", title_id)
-            else:
-                results = self._search_subtitles(title_id)
-                page_link = self.URL_WIZDOM_PAGELINK.format("movie", title_id)
-
-            if not results:
-                logger.info("No subtitles found for: {}".format(title_name))
-                continue
-
-            for result in results:
-                subtitle_id, release = result['id'], result['version']
-                subtitles.append(self.subtitle_class(next(iter(self.languages)), title_id, subtitle_id,
-                                                     title_name, season, episode, release, year, page_link))
-
-        if subtitles:
-            logger.debug("Found Subtitle Candidates: {}".format([x.release for x in subtitles]))
-        return subtitles
-
-    def list_subtitles(self, video, languages):
-        season = episode = year = title = None
-
-        if isinstance(video, Episode):
-            logger.info("list_subtitles Series: {}, season: {}, episode: {}".format(video.series,
-                                                                                    video.season,
-                                                                                    video.episode))
-            title = video.series
-            season = video.season
-            if video.episode == 0:
-                episode = 1
-            else:
-                episode = video.episode
-        elif isinstance(video, Movie):
-            logger.info("list_subtitles Movie: {}, year: {}".format(video.title, video.year))
-            title = video.title
-            year = video.year
-
-        return [s for s in self.query(title, season, episode, year) if s.language in languages]
-
-    def _download_subtitles(self, subtitle_id):
-        logger.debug("Downloading subtitle id - {}".format(subtitle_id))
-        r = self.session.get(self.URL_DOWNLOAD_SUBTITLE.format(subtitle_id))
-
-        if not r.ok:
-            logger.error("Bad response from server while downloding zip [id={}]".format(subtitle_id))
-            return None
-        if not r.content:
-            logger.error("Unable to download zip [id={}]".format(subtitle_id))
-            return None
-
-        return r.content
-
-    def download_subtitle(self, subtitle):
-        # type: (WizdomSubtitle) -> None
-
-        logger.info('Downloading subtitle from WizdomSubs: %r', subtitle)
-
-        content = None
-        zip_content = self._download_subtitles(subtitle.subtitle_id)
-        if not zip_content:
-            return None
-        if not zip_content[:2] == "PK":
-            logger.warning("Response did not contain zip file")
-            return None
-
-        zfile = ZipFile(BytesIO(zip_content))
-        if not len(zfile.namelist()):
-            logger.warning("Response did not contain files inside the zip archive")
-            return None
-
-        sub_files = [x for x in zfile.namelist() if x.endswith(('.srt', '.idx', '.sub'))]
-        if sub_files:
-            content = zfile.open(sub_files[0]).read()
-
-        if not content:
-            logger.warning("File inside zip is empty")
-            return None
-        subtitle.content = fix_line_ending(content)
-        logger.info("Downloaded {} successfuly!".format(subtitle))

Contents/Libraries/Shared/subliminal_patch/refiners/drone.py

@@ -19,11 +19,11 @@ class DroneAPIClient(object):
     _fill_attrs = None
 
     def __init__(self, version=1, session=None, headers=None, timeout=10, base_url=None, api_key=None,
-                 ssl_no_verify=False, pem_file=None):
+                 ssl_no_verify=False):
         headers = dict(headers or {}, **{"X-Api-Key": api_key})
 
         #: Session for the requests
-        self.session = session or CertifiSession(verify=pem_file)
+        self.session = session or CertifiSession()
         if ssl_no_verify:
             self.session.verify = False
 

Contents/Libraries/Shared/subliminal_patch/refiners/tvdb.py

@@ -87,10 +87,7 @@ def refine(video, **kwargs):
         # parse series year
         series_year = None
         if result['firstAired']:
-            try:
-                series_year = datetime.datetime.strptime(result['firstAired'], '%Y-%m-%d').year
-            except ValueError:
-                continue
+            series_year = datetime.datetime.strptime(result['firstAired'], '%Y-%m-%d').year
 
         # discard mismatches on year
         if video.year and series_year and video.year != series_year:

Contents/Libraries/Shared/subliminal_patch/score.py

@@ -60,8 +60,6 @@ def compute_score(matches, subtitle, video, hearing_impaired=None):
     episode_hash_valid_if = {"series", "season", "episode", "format"}
     movie_hash_valid_if = {"video_codec", "format"}
 
-    orig_matches = matches.copy()
-
     # on hash match, discard everything else
     if subtitle.hash_verifiable:
         if 'hash' in matches:
@@ -85,47 +83,41 @@ def compute_score(matches, subtitle, video, hearing_impaired=None):
         matches &= {'hash'}
 
     # handle equivalent matches
-    eq_matches = set()
     if is_episode:
         if 'title' in matches:
             logger.debug('Adding title match equivalent')
-            eq_matches.add('episode')
+            matches.add('episode')
         if 'series_imdb_id' in matches:
             logger.debug('Adding series_imdb_id match equivalent')
-            eq_matches |= {'series', 'year'}
+            matches |= {'series', 'year'}
         if 'imdb_id' in matches:
             logger.debug('Adding imdb_id match equivalents')
-            eq_matches |= {'series', 'year', 'season', 'episode'}
+            matches |= {'series', 'year', 'season', 'episode'}
         if 'tvdb_id' in matches:
             logger.debug('Adding tvdb_id match equivalents')
-            eq_matches |= {'series', 'year', 'season', 'episode', 'title'}
+            matches |= {'series', 'year', 'season', 'episode', 'title'}
         if 'series_tvdb_id' in matches:
             logger.debug('Adding series_tvdb_id match equivalents')
-            eq_matches |= {'series', 'year'}
+            matches |= {'series', 'year'}
 
         # specials
         if video.is_special and 'title' in matches and 'series' in matches \
                 and 'year' in matches:
             logger.debug('Adding special title match equivalent')
-            eq_matches |= {'season', 'episode'}
+            matches |= {'season', 'episode'}
 
     elif is_movie:
         if 'imdb_id' in matches:
             logger.debug('Adding imdb_id match equivalents')
-            eq_matches |= {'title', 'year'}
-
-    matches |= eq_matches
+            matches |= {'title', 'year'}
 
     # handle hearing impaired
     if hearing_impaired is not None and subtitle.hearing_impaired == hearing_impaired:
         logger.debug('Matched hearing_impaired')
         matches.add('hearing_impaired')
-        orig_matches.add('hearing_impaired')
 
     # compute the score
     score = sum((scores.get(match, 0) for match in matches))
     logger.info('%r: Computed score %r with final matches %r', subtitle, score, matches)
 
-    score_without_hash = sum((scores.get(match, 0) for match in orig_matches | eq_matches if match != "hash"))
-
-    return score, score_without_hash
+    return score

Contents/Libraries/Shared/subliminal_patch/subtitle.py

@@ -19,15 +19,6 @@ from subliminal import Subtitle as Subtitle_
 from subliminal.subtitle import Episode, Movie, sanitize_release_group, get_equivalent_release_groups
 from subliminal_patch.utils import sanitize
 from ftfy import fix_text
-from codecs import BOM_UTF8, BOM_UTF16_BE, BOM_UTF16_LE, BOM_UTF32_BE, BOM_UTF32_LE
-
-BOMS = (
-    (BOM_UTF8, "UTF-8"),
-    (BOM_UTF32_BE, "UTF-32-BE"),
-    (BOM_UTF32_LE, "UTF-32-LE"),
-    (BOM_UTF16_BE, "UTF-16-BE"),
-    (BOM_UTF16_LE, "UTF-16-LE"),
-)
 
 logger = logging.getLogger(__name__)
 
@@ -114,9 +105,6 @@ class Subtitle(Subtitle_):
         # normalize line endings
         self.content = self.content.replace("\r\n", "\n").replace('\r', '\n')
 
-    def _check_bom(self, data):
-        return [encoding for bom, encoding in BOMS if data.startswith(bom)]
-
     def guess_encoding(self):
         """Guess encoding using the language, falling back on chardet.
 
@@ -129,19 +117,14 @@ class Subtitle(Subtitle_):
 
         logger.info('Guessing encoding for language %s', self.language)
 
-        encodings = ['utf-8']
-
-        # check UTF BOMs
-        bom_encodings = self._check_bom(self.content)
-        if bom_encodings:
-            encodings = list(set(enc.lower() for enc in bom_encodings + encodings))
+        encodings = ['utf-8', 'utf-16']
 
         # add language-specific encodings
         # http://scratchpad.wikia.com/wiki/Character_Encoding_Recommendation_for_Languages
 
         if self.language.alpha3 == 'zho':
             encodings.extend(['cp936', 'gb2312', 'gbk', 'gb18030', 'hz', 'iso2022_jp_2', 'cp950', 'gb18030', 'big5',
-                              'big5hkscs', 'utf-16'])
+                              'big5hkscs'])
         elif self.language.alpha3 == 'jpn':
             encodings.extend(['shift-jis', 'cp932', 'euc_jp', 'iso2022_jp', 'iso2022_jp_1', 'iso2022_jp_2',
                               'iso2022_jp_2004', 'iso2022_jp_3', 'iso2022_jp_ext', ])
@@ -150,7 +133,7 @@ class Subtitle(Subtitle_):
 
         # arabian/farsi
         elif self.language.alpha3 in ('ara', 'fas', 'per'):
-            encodings.extend(['windows-1256', 'utf-16'])
+            encodings.append('windows-1256')
         elif self.language.alpha3 == 'heb':
             encodings.extend(['windows-1255', 'iso-8859-8'])
         elif self.language.alpha3 == 'tur':
@@ -268,7 +251,8 @@ class Subtitle(Subtitle_):
                 subs = pysubs2.SSAFile.from_string(text, fps=self.plex_media_fps)
 
             unicontent = self.pysubs2_to_unicode(subs)
-            self.content = unicontent.encode(self._guessed_encoding)
+            self.content = unicontent.encode("utf-8")
+            self._guessed_encoding = "utf-8"
         except:
             logger.exception("Couldn't convert subtitle %s to .srt format: %s", self, traceback.format_exc())
             return False
@@ -278,12 +262,6 @@ class Subtitle(Subtitle_):
 
     @classmethod
     def pysubs2_to_unicode(cls, sub, format="srt"):
-        """
-        this is a modified version of pysubs2.SubripFormat.to_file with special handling for drawing tags in ASS
-        :param sub:
-        :param format:
-        :return:
-        """
         def ms_to_timestamp(ms, mssep=","):
             """Convert ms to 'HH:MM:SS,mmm'"""
             # XXX throw on overflow/underflow?
@@ -295,12 +273,9 @@ class Subtitle(Subtitle_):
         def prepare_text(text, style):
             body = []
             for fragment, sty in parse_tags(text, style, sub.styles):
-                fragment = fragment.replace(r"\h", u" ")
-                fragment = fragment.replace(r"\n", u"\n")
-                fragment = fragment.replace(r"\N", u"\n")
-                if sty.drawing:
-                    raise pysubs2.ContentNotUsable
-
+                fragment = fragment.replace(ur"\h", u" ")
+                fragment = fragment.replace(ur"\n", u"\n")
+                fragment = fragment.replace(ur"\N", u"\n")
                 if format == "srt":
                     if sty.italic:
                         fragment = u"<i>%s</i>" % fragment
@@ -332,10 +307,7 @@ class Subtitle(Subtitle_):
         for i, line in enumerate(visible_lines, 1):
             start = ms_to_timestamp(line.start, mssep=mssep)
             end = ms_to_timestamp(line.end, mssep=mssep)
-            try:
-                text = prepare_text(line.text, sub.styles.get(line.style, SSAStyle.DEFAULT_STYLE))
-            except pysubs2.ContentNotUsable:
-                continue
+            text = prepare_text(line.text, sub.styles.get(line.style, SSAStyle.DEFAULT_STYLE))
 
             out.append(u"%d\n" % i)
             out.append(u"%s --> %s\n" % (start, end))
@@ -348,8 +320,7 @@ class Subtitle(Subtitle_):
         :return: string 
         """
         if not self.mods:
-            return fix_text(self.content.decode(encoding=self._guessed_encoding), **ftfy_defaults).encode(
-                encoding=self._guessed_encoding)
+            return fix_text(self.content.decode("utf-8"), **ftfy_defaults).encode(encoding="utf-8")
 
         submods = SubtitleModifications(debug=debug)
         if submods.load(content=self.text, language=self.language):
@@ -358,7 +329,7 @@ class Subtitle(Subtitle_):
             self.mods = submods.mods_used
 
             content = fix_text(self.pysubs2_to_unicode(submods.f, format=format), **ftfy_defaults)\
-                .encode(encoding=self._guessed_encoding)
+                .encode(encoding="utf-8")
             submods.f = None
             del submods
             return content
@@ -369,15 +340,6 @@ class ModifiedSubtitle(Subtitle):
     id = None
 
 
-MERGED_FORMATS = {
-    "TV": ("HDTV", "SDTV", "AHDTV", "UHDTV"),
-    "Air": ("SATRip", "DVB", "PPV"),
-    "Disk": ("DVD", "HD-DVD", "BluRay")
-}
-
-MERGED_FORMATS_REV = dict((v.lower(), k.lower()) for k in MERGED_FORMATS for v in MERGED_FORMATS[k])
-
-
 def guess_matches(video, guess, partial=False):
     """Get matches between a `video` and a `guess`.
 
@@ -460,25 +422,21 @@ def guess_matches(video, guess, partial=False):
             formats = [formats]
 
         if video.format:
-            video_format = video.format.lower()
-            _video_gen_format = MERGED_FORMATS_REV.get(video_format)
-            if _video_gen_format:
-                logger.debug("Treating %s as %s the same", video_format, _video_gen_format)
+            video_format = video.format
+            if video_format in ("HDTV", "SDTV", "TV"):
+                video_format = "TV"
+                logger.debug("Treating HDTV/SDTV the same")
 
             for frmt in formats:
-                _guess_gen_frmt = MERGED_FORMATS_REV.get(frmt.lower())
+                if frmt in ("HDTV", "SDTV"):
+                    frmt = "TV"
 
-                if _guess_gen_frmt == _video_gen_format:
+                if frmt.lower() == video_format.lower():
                     matches.add('format')
                     break
-        if "release_group" in matches and "format" not in matches:
-            logger.info("Release group matched but format didn't. Remnoving release group match.")
-            matches.remove("release_group")
-
     # video_codec
     if video.video_codec and 'video_codec' in guess and guess['video_codec'] == video.video_codec:
         matches.add('video_codec')
-
     # audio_codec
     if video.audio_codec and 'audio_codec' in guess and guess['audio_codec'] == video.audio_codec:
         matches.add('audio_codec')

Contents/Libraries/Shared/submod_test.py

@@ -21,10 +21,9 @@ if debug:
     logging.basicConfig(level=logging.DEBUG)
 
 #sub = Subtitle(Language.fromietf("eng:forced"), mods=["common", "remove_HI", "OCR_fixes", "fix_uppercase", "shift_offset(ms=-500)", "shift_offset(ms=500)", "shift_offset(s=2,ms=800)"])
-sub = Subtitle(Language.fromietf("eng"), mods=["common", "remove_HI", "OCR_fixes", "fix_uppercase", "shift_offset(ms=0,s=1)"])
+sub = Subtitle(Language.fromietf("eng:forced"), mods=["common", "remove_HI", "OCR_fixes", "fix_uppercase", "shift_offset(ms=0,s=1)"])
 sub.content = open(fn).read()
 sub.normalize()
-sub.is_valid()
 content = sub.get_modified_content(debug=True)
 
 #submod = SubMod(debug=debug)

Contents/Libraries/Shared/subscene_api/subscene.py

@@ -30,7 +30,6 @@ import enum
 import sys
 import requests
 import time
-import logging
 
 is_PY2 = sys.version_info[0] < 3
 if is_PY2:
@@ -40,13 +39,8 @@ else:
     from contextlib import suppress
     from urllib2.request import Request, urlopen
 
-from dogpile.cache.api import NO_VALUE
-from subliminal.cache import region
 from bs4 import BeautifulSoup, NavigableString
 
-
-logger = logging.getLogger(__name__)
-
 # constants
 HEADERS = {
 }
@@ -56,21 +50,14 @@ DEFAULT_USER_AGENT = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWeb"\
                      "Kit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.47 Safari/537.36"
 
 
-ENDPOINT_RE = re.compile(ur'(?uis)<form.+?action="/subtitles/(.+)">.*?<input type="text"')
-
-
-class NewEndpoint(Exception):
-    pass
-
-
 # utils
-def soup_for(url, data=None, session=None, user_agent=DEFAULT_USER_AGENT):
+def soup_for(url, session=None, user_agent=DEFAULT_USER_AGENT):
     url = re.sub("\s", "+", url)
     if not session:
         r = Request(url, data=None, headers=dict(HEADERS, **{"User-Agent": user_agent}))
         html = urlopen(r).read().decode("utf-8")
     else:
-        ret = session.post(url, data=data)
+        ret = session.get(url)
         ret.raise_for_status()
         html = ret.text
     return BeautifulSoup(html, "html.parser")
@@ -125,7 +112,7 @@ class Subtitle(object):
         subtitles = []
 
         for row in rows:
-            if row.td.a is not None and row.td.get("class", ["lazy"])[0] != "empty":
+            if row.td.a is not None:
                 subtitles.append(cls.from_row(row))
 
         return subtitles
@@ -255,42 +242,29 @@ def get_first_film(soup, section, year=None, session=None):
                 url = SITE_DOMAIN + t.div.a.get("href")
                 break
         if not url:
-            # fallback to non-year results
-            logger.info("Falling back to non-year results as year wasn't found (%s)", year)
-            url = SITE_DOMAIN + tag.findNext("ul").find("li").div.a.get("href")
+            return
 
     return Film.from_url(url, session=session)
 
 
-def find_endpoint(session, content=None):
-    endpoint = region.get("subscene_endpoint2")
-    if endpoint is NO_VALUE:
-        if not content:
-            content = session.get(SITE_DOMAIN).text
-
-        m = ENDPOINT_RE.search(content)
-        if m:
-            endpoint = m.group(1).strip()
-            logger.debug("Switching main endpoint to %s", endpoint)
-            region.set("subscene_endpoint2", endpoint)
-    return endpoint
-
-
 def search(term, release=True, session=None, year=None, limit_to=SearchTypes.Exact, throttle=0):
     # note to subscene: if you actually start to randomize the endpoint, we'll have to query your server even more
-
+    endpoints = ["searching", "search", "srch", "find"]
     if release:
-        endpoint = "release"
-    else:
-        endpoint = find_endpoint(session)
-        time.sleep(throttle)
+        endpoints = ["release"]
 
-    if not endpoint:
-        logger.error("Couldn't find endpoint, exiting")
-        return
-
-    soup = soup_for("%s/subtitles/%s" % (SITE_DOMAIN, endpoint), data={"query": term},
-                    session=session)
+    soup = None
+    for endpoint in endpoints:
+        try:
+            soup = soup_for("%s/subtitles/%s?q=%s" % (SITE_DOMAIN, endpoint, term),
+                            session=session)
+        except requests.HTTPError, e:
+            if e.response.status_code == 404:
+                time.sleep(throttle)
+                # fixme: detect endpoint from html
+                continue
+            raise
+        break
 
     if soup:
         if "Subtitle search by" in str(soup):

Contents/Libraries/Shared/subzero/language.py

@@ -1,47 +1,15 @@
 # coding=utf-8
-from __future__ import absolute_import
 import types
-import re
 
 from babelfish.exceptions import LanguageError
-from babelfish import Language as Language_, basestr, LANGUAGE_MATRIX
-from six.moves import zip
+from babelfish import Language as Language_, basestr
 
 repl_map = {
     "dk": "da",
     "nld": "nl",
     "english": "en",
-    "alb": "sq",
-    "arm": "hy",
-    "baq": "eu",
-    "bur": "my",
-    "chi": "zh",
-    "cze": "cs",
-    "dut": "nl",
-    "fre": "fr",
-    "geo": "ka",
-    "ger": "de",
-    "gre": "el",
-    "ice": "is",
-    "mac": "mk",
-    "mao": "mi",
-    "may": "ms",
-    "per": "fa",
-    "rum": "ro",
-    "slo": "sk",
-    "tib": "bo",
 }
 
-CUSTOM_LIST = ["chs", "sc", "zhs", "hans", "gb", u"简", u"双语",
-               "cht", "tc", "zht", "hant", "big5", u"繁", u"雙語",
-               "spl", "ea", "pob", "pb"]
-
-ALPHA2_LIST = list(set(filter(lambda x: x, map(lambda x: x.alpha2, LANGUAGE_MATRIX)))) + list(repl_map.values())
-ALPHA3b_LIST = list(set(filter(lambda x: x, map(lambda x: x.alpha3, LANGUAGE_MATRIX)))) + \
-               list(set(filter(lambda x: len(x) == 3, list(repl_map.keys()))))
-FULL_LANGUAGE_LIST = ALPHA2_LIST + ALPHA3b_LIST
-FULL_LANGUAGE_LIST.extend(CUSTOM_LIST)
-
 
 def language_from_stream(l):
     if not l:
@@ -67,8 +35,7 @@ def wrap_forced(f):
         args = args[1:]
         s = args.pop(0)
         forced = None
-        hi = None
-        if isinstance(s, (str,)):
+        if isinstance(s, types.StringTypes):
             base, forced = s.split(":") if ":" in s else (s, False)
         else:
             base = s
@@ -76,7 +43,6 @@ def wrap_forced(f):
         instance = f(cls, base, *args, **kwargs)
         if isinstance(instance, Language):
             instance.forced = forced == "forced"
-            instance.hi = hi == "hi"
         return instance
 
     return inner
@@ -84,21 +50,16 @@ def wrap_forced(f):
 
 class Language(Language_):
     forced = False
-    hi = False
 
-    def __init__(self, language, country=None, script=None, unknown=None, forced=False, hi=False):
+    def __init__(self, language, country=None, script=None, unknown=None, forced=False):
         self.forced = forced
-        self.hi = hi
         super(Language, self).__init__(language, country=country, script=script, unknown=unknown)
 
     def __getstate__(self):
-        return self.alpha3, self.country, self.script, self.hi, self.forced
+        return self.alpha3, self.country, self.script, self.forced
 
-    def __setstate__(self, forced):
-        self.alpha3, self.country, self.script, self.hi, self.forced = forced
-
-    def __hash__(self):
-        return hash(str(self))
+    def __setstate__(self, state):
+        self.alpha3, self.country, self.script, self.forced = state
 
     def __eq__(self, other):
         if isinstance(other, basestr):
@@ -108,16 +69,11 @@ class Language(Language_):
         return (self.alpha3 == other.alpha3 and
                 self.country == other.country and
                 self.script == other.script and
-                bool(self.forced) == bool(other.forced) and
-                bool(self.hi) == bool(other.hi))
+                bool(self.forced) == bool(other.forced))
 
     def __str__(self):
         return super(Language, self).__str__() + (":forced" if self.forced else "")
 
-    def __repr__(self):
-        info = ";".join("{}={}".format(k, v) for k, v in vars(self).items() if v)
-        return "<{}: {}>".format(self.__class__.__name__, info)
-
     @property
     def basename(self):
         return super(Language, self).__str__()
@@ -126,15 +82,14 @@ class Language(Language_):
         ret = super(Language, self).__getattr__(name)
         if isinstance(ret, Language):
             ret.forced = self.forced
-            ret.hi = self.hi
         return ret
 
     @classmethod
     def rebuild(cls, instance, **replkw):
         state = instance.__getstate__()
-        attrs = ("country", "script", "hi", "forced")
+        attrs = ("country", "script", "forced")
         language = state[0]
-        kwa = dict(list(zip(attrs, state[1:])))
+        kwa = dict(zip(attrs, state[1:]))
         kwa.update(replkw)
         return cls(language, **kwa)
 
@@ -160,16 +115,3 @@ class Language(Language_):
             return Language(*Language_.fromietf(s).__getstate__())
 
         return Language(*Language_.fromalpha3b(s).__getstate__())
-
-
-IETF_MATCH = ".+\.([^-.]+)(?:-[A-Za-z]+)?$"
-ENDSWITH_LANGUAGECODE_RE = re.compile("\.([^-.]{2,3})(?:-[A-Za-z]{2,})?$")
-
-
-def match_ietf_language(s, ietf=False):
-    language_match = re.match(".+\.([^\.]+)$" if not ietf
-                              else IETF_MATCH, s)
-    if language_match and len(language_match.groups()) == 1:
-        language = language_match.groups()[0]
-        return language
-    return s

Contents/Libraries/Shared/subzero/lib/dict.py

@@ -107,12 +107,6 @@ class Dicked(object):
         for key, value in entries.iteritems():
             self.__dict__[key] = (Dicked(**value) if isinstance(value, dict) else value)
 
-    def has(self, key):
-        return self._entries is not None and key in self._entries
-
-    def get(self, key, default=None):
-        return self._entries.get(key, default) if self._entries else default
-
     def __repr__(self):
         return str(self)
 

Contents/Libraries/Shared/subzero/modification/dictionaries/make_data.py

@@ -36,7 +36,6 @@ SZ_FIX_DATA = {
             u" l ": u" I ",
             u"'sjust": u"'s just",
             u"'tjust": u"'t just",
-            u"\";": u"'s",
         },
         "WholeWords": {
             u"I'11": u"I'll",

Contents/Libraries/Shared/subzero/modification/exc.py

@@ -1,7 +0,0 @@
-# coding=utf-8
-class EmptyEntryError(Exception):
-    pass
-
-
-class EmptyLineError(Exception):
-    pass

Contents/Libraries/Shared/subzero/modification/main.py

@@ -6,8 +6,7 @@ import pysubs2
 import logging
 import time
 
-from mods import EMPTY_TAG_PROCESSOR
-from exc import EmptyEntryError
+from mods import EMPTY_TAG_PROCESSOR, EmptyEntryError, FullContentRep
 from registry import registry
 from subzero.language import Language
 
@@ -258,7 +257,16 @@ class SubtitleModifications(object):
                 mod.modify(None, debug=self.debug, parent=self, **args)
 
     def apply_line_mods(self, new_entries, mods):
-        for index, entry in enumerate(self.f, 1):
+        index = 1
+        entries = self.f[:]
+        entry_count = len(entries)
+
+        while 1:
+            if index > entry_count - 1:
+                break
+
+            entry = entries[index]
+
             applied_mods = []
             lines = []
 
@@ -266,116 +274,110 @@ class SubtitleModifications(object):
             start_tags = []
             end_tags = []
 
-            t = entry.text.strip()
-            if not t:
+            text = entry.text.replace(ur"\N", "\n").strip()
+            if not text:
                 if self.debug:
                     logger.debug(u"Skipping empty line: %s", index)
+                index += 1
                 continue
 
-            skip_entry = False
-            for line in t.split(ur"\N"):
-                # don't bother the mods with surrounding tags
-                old_line = line
-                line = line.strip()
-                skip_line = False
-                line_count += 1
+            try:
+                for line in text.split("\n"):
+                    # don't bother the mods with surrounding tags
+                    old_line = line
+                    line = line.strip()
+                    skip_line = False
+                    line_count += 1
 
-                if not line:
-                    continue
+                    if not line:
+                        continue
 
-                # clean {\X0} tags before processing
-                # fixme: handle nested tags?
-                start_tag = u""
-                end_tag = u""
-                if line.startswith(self.font_style_tag_start):
-                    start_tag = line[:5]
-                    line = line[5:]
-                if line[-5:-3] == self.font_style_tag_start:
-                    end_tag = line[-5:]
-                    line = line[:-5]
+                    # clean {\X0} tags before processing
+                    # fixme: handle nested tags?
+                    start_tag = u""
+                    end_tag = u""
+                    if line.startswith(self.font_style_tag_start):
+                        start_tag = line[:5]
+                        line = line[5:]
+                    if line[-5:-3] == self.font_style_tag_start:
+                        end_tag = line[-5:]
+                        line = line[:-5]
 
-                last_procs_mods = []
+                    last_procs_mods = []
 
-                # fixme: this double loop is ugly
-                for order, identifier, args in mods:
-                    mod = self.initialized_mods[identifier]
+                    # fixme: this double loop is ugly
+                    for order, identifier, args in mods:
+                        mod = self.initialized_mods[identifier]
 
-                    try:
-                        line = mod.modify(line.strip(), entry=t, debug=self.debug, parent=self, index=index,
+                        line = mod.modify(line.strip(), entry=text, debug=self.debug, parent=self, index=index,
                                           **args)
-                    except EmptyEntryError:
-                        if self.debug:
-                            logger.debug(u"%d: %s: %r -> ''", index, identifier, t)
-                        skip_entry = True
-                        break
 
-                    if not line:
-                        if self.debug:
-                            logger.debug(u"%d: %s: %r -> ''", index, identifier, old_line)
-                        skip_line = True
-                        break
+                        if not line:
+                            if self.debug:
+                                logger.debug(u"%d: %s: %r -> ''", index, identifier, old_line)
+                            skip_line = True
+                            break
 
-                    applied_mods.append(identifier)
-                    if mod.last_processors:
-                        last_procs_mods.append([identifier, args])
+                        applied_mods.append(identifier)
+                        if mod.last_processors:
+                            last_procs_mods.append([identifier, args])
 
-                if skip_entry:
-                    lines = []
-                    break
+                    if skip_line:
+                        continue
 
-                if skip_line:
-                    continue
+                    for identifier, args in last_procs_mods:
+                        mod = self.initialized_mods[identifier]
 
-                for identifier, args in last_procs_mods:
-                    mod = self.initialized_mods[identifier]
-
-                    try:
-                        line = mod.modify(line.strip(), entry=t, debug=self.debug, parent=self, index=index,
+                        line = mod.modify(line.strip(), entry=text, debug=self.debug, parent=self, index=index,
                                           procs=["last_process"], **args)
-                    except EmptyEntryError:
+
+                        if not line:
+                            if self.debug:
+                                logger.debug(u"%d: %s: %r -> ''", index, identifier, old_line)
+                            skip_line = True
+                            break
+
+                    if skip_line:
+                        continue
+
+                    if start_tag:
+                        start_tags.append(start_tag)
+
+                    if end_tag:
+                        end_tags.append(end_tag)
+
+                    # append new line and clean possibly newly added empty tags
+                    cleaned_line = EMPTY_TAG_PROCESSOR.process(start_tag + line + end_tag, debug=self.debug).strip()
+                    if cleaned_line:
+                        # we may have a single closing tag, if so, try appending it to the previous line
+                        if len(cleaned_line) == 5 and cleaned_line.startswith("{\\") and cleaned_line.endswith("0}"):
+                            if lines:
+                                prev_line = lines.pop()
+                                lines.append(prev_line + cleaned_line)
+                                continue
+
+                        lines.append(cleaned_line)
+                    else:
                         if self.debug:
-                            logger.debug(u"%d: %s: %r -> ''", index, identifier, t)
-                        skip_entry = True
-                        break
+                            logger.debug(u"%d: Ditching now empty line (%r)", index, line)
 
-                    if not line:
-                        if self.debug:
-                            logger.debug(u"%d: %s: %r -> ''", index, identifier, old_line)
-                        skip_line = True
-                        break
-
-                if skip_entry:
-                    lines = []
-                    break
-
-                if skip_line:
+                if not lines:
+                    # don't bother logging when the entry only had one line
+                    if self.debug and line_count > 1:
+                        logger.debug(u"%d: %r -> ''", index, text)
+                    index += 1
                     continue
+            except EmptyEntryError, e:
+                if self.debug:
+                    logger.debug(u"%d: %s: %r -> ''", index, e.mod.identifier, e.entry)
+                index += 1
+                continue
 
-                if start_tag:
-                    start_tags.append(start_tag)
-
-                if end_tag:
-                    end_tags.append(end_tag)
-
-                # append new line and clean possibly newly added empty tags
-                cleaned_line = EMPTY_TAG_PROCESSOR.process(start_tag + line + end_tag, debug=self.debug).strip()
-                if cleaned_line:
-                    # we may have a single closing tag, if so, try appending it to the previous line
-                    if len(cleaned_line) == 5 and cleaned_line.startswith("{\\") and cleaned_line.endswith("0}"):
-                        if lines:
-                            prev_line = lines.pop()
-                            lines.append(prev_line + cleaned_line)
-                            continue
-
-                    lines.append(cleaned_line)
-                else:
-                    if self.debug:
-                        logger.debug(u"%d: Ditching now empty line (%r)", index, line)
-
-            if not lines:
-                # don't bother logging when the entry only had one line
-                if self.debug and line_count > 1:
-                    logger.debug(u"%d: %r -> ''", index, entry.text)
+            except FullContentRep, e:
+                if self.debug:
+                    logger.debug(u"%d: %s: %r -> %r", index, e.mod.identifier, text, e.new_content)
+                new_entries.append(e.new_content.replace("\n", ur"\N"))
+                index += 1
                 continue
 
             new_text = ur"\N".join(lines)
@@ -404,6 +406,8 @@ class SubtitleModifications(object):
                 entry.text = new_text
 
             new_entries.append(entry)
+            index += 1
+
 
 SubMod = SubtitleModifications
 

Contents/Libraries/Shared/subzero/modification/mods/__init__.py

@@ -47,7 +47,7 @@ class SubtitleModification(object):
                 continue
 
             old_content = new_content
-            new_content = processor.process(new_content, debug=debug, **kwargs)
+            new_content = processor.process(new_content, debug=debug, mod=self, **kwargs)
             if not new_content:
                 if debug:
                     logger.debug("Processor returned empty line: %s", processor.name)
@@ -107,3 +107,22 @@ empty_line_post_processors = [
 ]
 
 
+class ModEvent(Exception):
+    def __init__(self, *args, **kwargs):
+        self.mod = kwargs.pop("mod", None)
+        self.entry = kwargs.pop("entry", None)
+        super(ModEvent, self).__init__(*args, **kwargs)
+
+
+class EmptyEntryError(ModEvent):
+    pass
+
+
+class EmptyLineError(ModEvent):
+    pass
+
+
+class FullContentRep(ModEvent):
+    def __init__(self, *args, **kwargs):
+        self.new_content = kwargs.pop("new_content", None)
+        super(FullContentRep, self).__init__(*args, **kwargs)

Contents/Libraries/Shared/subzero/modification/mods/common.py

@@ -7,7 +7,6 @@ from subzero.modification.mods import SubtitleTextModification, empty_line_post_
 from subzero.modification.processors import FuncProcessor
 from subzero.modification.processors.re_processor import NReProcessor
 from subzero.modification import registry
-from tld import get_tld
 
 
 ENGLISH = Language("eng")
@@ -29,7 +28,7 @@ class CommonFixes(SubtitleTextModification):
         NReProcessor(re.compile(r'(?u)(\w|\b|\s|^)(-\s?-{1,2})'), ur"\1—", name="CM_multidash"),
 
         # line = _/-/\s
-        NReProcessor(re.compile(r'(?u)(^\W*[-_.:<>~"\']+\W*$)'), "", name="CM_non_word_only"),
+        NReProcessor(re.compile(r'(?u)(^\W*[-_.:>~]+\W*$)'), "", name="<CM_non_word_only"),
 
         # remove >>
         NReProcessor(re.compile(r'(?u)^\s?>>\s*'), "", name="CM_leading_crocodiles"),
@@ -114,9 +113,7 @@ class CommonFixes(SubtitleTextModification):
         NReProcessor(re.compile(r'(?u)(?:(?<=^)|(?<=\w)) +([!?.,](?![!?.,]| \.))'), r"\1", name="CM_punctuation_space"),
 
         # add space after punctuation
-        NReProcessor(re.compile(r'(?u)(([^\s]*)([!?.,:])([A-zÀ-ž]{2,}))'),
-                     lambda match: u"%s%s %s" % (match.group(2), match.group(3), match.group(4)) if not get_tld(match.group(1), fail_silently=True, fix_protocol=True) else match.group(1),
-                     name="CM_punctuation_space2"),
+        NReProcessor(re.compile(r'(?u)([!?.,:])([A-zÀ-ž]{2,})'), r"\1 \2", name="CM_punctuation_space2"),
 
         # fix lowercase I in english
         NReProcessor(re.compile(r'(?u)(\b)i(\b)'), r"\1I\2", name="CM_EN_lowercase_i",
@@ -181,14 +178,6 @@ class FixUppercase(SubtitleModification):
             entry.plaintext = self.capitalize(entry.plaintext)
 
 
-"""
-subsync
-
-subsync --cli --offline --overwrite --window-size=600 --max-point-dist=2.0 --min-points-no=20 --min-word-prob=0.3 --min-word-len=5 --min-correlation=0.9999 --min-words-sim=0.6 --out-time-offset=-0.08 sync --out SUBTITLE -s SUBTITLE --sub-lang=eng --sub-enc=utf-8 -r REF_FILE --ref-stream-by-type=audio --ref-stream-by-lang=eng
-
-"""
-
-
 registry.register(CommonFixes)
 registry.register(RemoveTags)
 registry.register(ReverseRTL)

Contents/Libraries/Shared/subzero/modification/mods/hearing_impaired.py

@@ -1,8 +1,8 @@
 # coding=utf-8
 import re
 
-from subzero.modification.mods import SubtitleTextModification, empty_line_post_processors, TAG
-from subzero.modification.exc import EmptyEntryError
+from subzero.modification.mods import SubtitleTextModification, empty_line_post_processors, EmptyEntryError, TAG, \
+    FullContentRep
 from subzero.modification.processors.re_processor import NReProcessor
 from subzero.modification import registry
 
@@ -11,9 +11,11 @@ class FullBracketEntryProcessor(NReProcessor):
     def process(self, content, debug=False, **kwargs):
         entry = kwargs.get("entry")
         if entry:
-            rep_content = super(FullBracketEntryProcessor, self).process(entry, debug=debug, **kwargs)
-            if not rep_content.strip():
-                raise EmptyEntryError()
+            rep_content = super(FullBracketEntryProcessor, self).process(entry, debug=debug, **kwargs).strip()
+            if not rep_content:
+                raise EmptyEntryError(mod=self.mod, entry=entry)
+            if content != rep_content:
+                raise FullContentRep(new_content=rep_content, mod=self.mod, entry=entry)
         return content
 
 
@@ -47,11 +49,11 @@ class HearingImpaired(SubtitleTextModification):
                      name="HI_before_colon_noncaps"),
 
         # brackets (only remove if at least 3 chars in brackets)
-        NReProcessor(re.compile(ur'(?sux)-?%(t)s["\']*[([][^([)\]]+?(?=[A-zÀ-ž"\'.]{3,})[^([)\]]+[)\]]["\']*[\s:]*%(t)s' %
+        NReProcessor(re.compile(ur'(?sux)-?%(t)s[([][^([)\]]+?(?=[A-zÀ-ž"\'.]{3,})[^([)\]]+[)\]][\s:]*%(t)s' %
                                 {"t": TAG}), "", name="HI_brackets"),
 
-        #NReProcessor(re.compile(ur'(?sux)-?%(t)s[([]%(t)s(?=[A-zÀ-ž"\'.]{3,})[^([)\]]+%(t)s$' % {"t": TAG}),
-        #             "", name="HI_bracket_open_start"),
+        FullBracketEntryProcessor(re.compile(ur'(?sux)-?%(t)s[([]%(t)s(?=[A-zÀ-ž"\'.]{3,})[^([)\]]+%(t)s$' % {"t": TAG}),
+                     "", name="HI_bracket_open_start"),
 
         #NReProcessor(re.compile(ur'(?sux)-?%(t)s(?=[A-zÀ-ž"\'.]{3,})[^([)\]]+[)\]][\s:]*%(t)s' % {"t": TAG}), "",
         #             name="HI_bracket_open_end"),
@@ -91,8 +93,8 @@ class HearingImpaired(SubtitleTextModification):
                      "", name="HI_music_symbols_only"),
 
         # remove music entries
-        NReProcessor(re.compile(ur'(?ums)(^[-\s>~]*[*#¶♫♪]+\s*.+|.+\s*[*#¶♫♪]+\s*$)'),
-                     "", name="HI_music", entry=True),
+        NReProcessor(re.compile(ur'(?ums)(^[-\s>~]*[♫♪]+\s*.+|.+\s*[♫♪]+\s*$)'),
+                     "", name="HI_music"),
     ]
 
 

Contents/Libraries/Shared/subzero/modification/processors/__init__.py

@@ -7,12 +7,14 @@ class Processor(object):
     """
     name = None
     parent = None
+    mod = None
     supported = None
     enabled = True
 
-    def __init__(self, name=None, parent=None, supported=None, **kwargs):
+    def __init__(self, name=None, parent=None, mod=None, supported=None):
         self.name = name
         self.parent = parent
+        self.mod = mod
         self.supported = supported if supported else lambda parent: True
 
     @property
@@ -20,6 +22,8 @@ class Processor(object):
         return self.name
 
     def process(self, content, debug=False, **kwargs):
+        if not self.mod:
+            self.mod = kwargs.get("mod", None)
         return content
 
     def __repr__(self):
@@ -35,7 +39,7 @@ class Processor(object):
 class FuncProcessor(Processor):
     func = None
 
-    def __init__(self, func, name=None, parent=None, supported=None, **kwargs):
+    def __init__(self, func, name=None, parent=None, supported=None):
         super(FuncProcessor, self).__init__(name=name, supported=supported)
         self.func = func
 

Contents/Libraries/Shared/subzero/modification/processors/re_processor.py

@@ -2,7 +2,6 @@
 import re
 import logging
 
-from subzero.modification.exc import EmptyEntryError
 from subzero.modification.processors import Processor
 
 logger = logging.getLogger(__name__)
@@ -15,22 +14,14 @@ class ReProcessor(Processor):
     pattern = None
     replace_with = None
 
-    def __init__(self, pattern, replace_with, name=None, supported=None, entry=False, **kwargs):
-        super(ReProcessor, self).__init__(name=name, supported=supported)
+    def __init__(self, pattern, replace_with, name=None, supported=None, **kwargs):
+        super(ReProcessor, self).__init__(name=name, supported=supported, **kwargs)
         self.pattern = pattern
         self.replace_with = replace_with
-        self.use_entry = entry
 
-    def process(self, content, debug=False, entry=None, **kwargs):
-        if not self.use_entry:
-            return self.pattern.sub(self.replace_with, content)
-
-        ret = self.pattern.sub(self.replace_with, entry)
-        if not ret:
-            raise EmptyEntryError()
-        elif ret != entry:
-            return ret
-        return content
+    def process(self, content, debug=False, **kwargs):
+        super(ReProcessor, self).process(content, debug=debug, **kwargs)
+        return self.pattern.sub(self.replace_with, content)
 
 
 class NReProcessor(ReProcessor):
@@ -46,7 +37,7 @@ class MultipleWordReProcessor(ReProcessor):
     }
     replaces found key in pattern with the corresponding value in data
     """
-    def __init__(self, snr_dict, name=None, parent=None, supported=None, **kwargs):
+    def __init__(self, snr_dict, name=None, parent=None, supported=None):
         super(ReProcessor, self).__init__(name=name, supported=supported)
         self.snr_dict = snr_dict
 

Contents/Libraries/Shared/subzero/modification/processors/string_processor.py

@@ -12,7 +12,7 @@ class StringProcessor(Processor):
     String replacement processor base
     """
 
-    def __init__(self, search, replace, name=None, parent=None, supported=None, **kwargs):
+    def __init__(self, search, replace, name=None, parent=None, supported=None):
         super(StringProcessor, self).__init__(name=name, supported=supported)
         self.search = search
         self.replace = replace
@@ -31,7 +31,7 @@ class MultipleLineProcessor(Processor):
         "data": {"old_value": "new_value"}
     }
     """
-    def __init__(self, snr_dict, name=None, parent=None, supported=None, **kwargs):
+    def __init__(self, snr_dict, name=None, parent=None, supported=None):
         super(MultipleLineProcessor, self).__init__(name=name, supported=supported)
         self.snr_dict = snr_dict
 

Contents/Libraries/Shared/subzero/video.py

@@ -17,8 +17,7 @@ def has_external_subtitle(part_id, stored_subs, language):
 
 
 def set_existing_languages(video, video_info, external_subtitles=False, embedded_subtitles=False, known_embedded=None,
-                           stored_subs=None, languages=None, only_one=False, known_metadata_subs=None,
-                           match_strictness="strict"):
+                           stored_subs=None, languages=None, only_one=False, known_metadata_subs=None):
     logger.debug(u"Determining existing subtitles for %s", video.name)
 
     external_langs_found = set()
@@ -28,8 +27,7 @@ def set_existing_languages(video, video_info, external_subtitles=False, embedded
         external_langs_found = known_metadata_subs
 
     external_langs_found.update(set(search_external_subtitles(video.name, languages=languages,
-                                                              only_one=only_one,
-                                                              match_strictness=match_strictness).values()))
+                                                              only_one=only_one).values()))
 
     # found external subtitles should be considered?
     if external_subtitles:
@@ -54,10 +52,10 @@ def set_existing_languages(video, video_info, external_subtitles=False, embedded
             video.subtitle_languages.add(language)
 
 
-def parse_video(fn, hints, skip_hashing=False, dry_run=False, providers=None, hash_from=None):
+def parse_video(fn, hints, skip_hashing=False, dry_run=False, providers=None):
     logger.debug("Parsing video: %s, hints: %s", os.path.basename(fn), hints)
     return scan_video(fn, hints=hints, dont_use_actual_file=dry_run, providers=providers,
-                      skip_hashing=skip_hashing, hash_from=hash_from)
+                      skip_hashing=skip_hashing)
 
 
 def refine_video(video, no_refining=False, refiner_settings=None):

Contents/Libraries/Shared/test.srt

@@ -19,8 +19,6 @@ I can't keep running. L can't!
 <b>i don't know. Some kind of wrong "1 00" number---
 of signal, drawing the Tardis off.... course.</b>
 # I'm singing in the rain
-www.website.com
-www.nowebsite.badlol
 
 4
 00:00:16,099 --> 00:00:17,224

Contents/Libraries/Shared/tld/__init__.py

@@ -1,14 +0,0 @@
-# -*- coding: utf-8 -*-
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-from __future__ import unicode_literals
-from .utils import get_fld, get_tld, get_tld_names, is_tld, parse_tld, Result, update_tld_names
-__title__ = u'tld'
-__version__ = u'0.11.10'
-__author__ = u'Artur Barseghyan'
-__copyright__ = u'2013-2019 Artur Barseghyan'
-__license__ = u'MPL-1.1 OR GPL-2.0-only OR LGPL-2.1-or-later'
-__all__ = (u'get_fld', u'get_tld', u'get_tld_names', u'is_tld',
-           u'parse_tld', u'Result', u'update_tld_names')

Contents/Libraries/Shared/tld/base.py

@@ -1,57 +0,0 @@
-# -*- coding: utf-8 -*-
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-from __future__ import unicode_literals
-from six import with_metaclass as _py_backwards_six_withmetaclass
-from codecs import open as codecs_open
-try:
-    from urllib.request import urlopen
-except ImportError:
-    from six.moves.urllib.request import urlopen as urlopen
-from .exceptions import TldIOError, TldImproperlyConfigured
-from .helpers import project_dir
-from .registry import Registry
-__author__ = u'Artur Barseghyan'
-__copyright__ = u'2013-2019 Artur Barseghyan'
-__license__ = u'MPL-1.1 OR GPL-2.0-only OR LGPL-2.1-or-later'
-__all__ = (u'BaseTLDSourceParser',)
-
-
-class BaseTLDSourceParser(
-        _py_backwards_six_withmetaclass(Registry, *[object])):
-    u'Base TLD source parser.'
-    uid = None
-    source_url = None
-    local_path = None
-
-    @classmethod
-    def validate(cls):
-        u'Constructor.'
-        if (not cls.uid):
-            raise TldImproperlyConfigured(
-                u'The `uid` property of the TLD source parser shall be defined.')
-
-    @classmethod
-    def get_tld_names(cls, fail_silently=False, retry_count=0):
-        u'Get tld names.\n\n        :param fail_silently:\n        :param retry_count:\n        :return:\n        '
-        cls.validate()
-        raise NotImplementedError(
-            u'Your TLD source parser shall implement `get_tld_names` method.')
-
-    @classmethod
-    def update_tld_names(cls, fail_silently=False):
-        u'Update the local copy of the TLD file.\n\n        :param fail_silently:\n        :return:\n        '
-        try:
-            remote_file = urlopen(cls.source_url)
-            local_file = codecs_open(project_dir(
-                cls.local_path), u'wb', encoding='utf8')
-            local_file.write(remote_file.read().decode(u'utf8'))
-            local_file.close()
-            remote_file.close()
-        except Exception as err:
-            if fail_silently:
-                return False
-            raise TldIOError(err)
-        return True

Contents/Libraries/Shared/tld/conf.py

@@ -1,45 +0,0 @@
-# -*- coding: utf-8 -*-
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-from __future__ import unicode_literals
-from typing import Any
-from . import defaults
-__author__ = u'Artur Barseghyan'
-__copyright__ = u'2013-2019 Artur Barseghyan'
-__license__ = u'MPL-1.1 OR GPL-2.0-only OR LGPL-2.1-or-later'
-__all__ = (u'get_setting', u'reset_settings', u'set_setting', u'settings')
-
-
-class Settings(object):
-    u'Settings registry.'
-
-    def __init__(self):
-        self._settings = {
-
-        }
-        self._settings_get = self._settings.get
-
-    def set(self, name, value):
-        u'\n        Override default settings.\n\n        :param str name:\n        :param mixed value:\n        '
-        self._settings[name] = value
-
-    def get(self, name, default=None):
-        u'\n        Gets a variable from local settings.\n\n        :param str name:\n        :param mixed default: Default value.\n        :return mixed:\n        '
-        if (name in self._settings):
-            return self._settings_get(name, default)
-        elif hasattr(defaults, name):
-            return getattr(defaults, name, default)
-        return default
-
-    def reset(self):
-        u'Reset settings.'
-        for name in defaults.__all__:
-            self.set(name, getattr(defaults, name))
-
-
-settings = Settings()
-get_setting = settings.get
-set_setting = settings.set
-reset_settings = settings.reset

Contents/Libraries/Shared/tld/defaults.py

@@ -1,13 +0,0 @@
-# -*- coding: utf-8 -*-
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-from __future__ import unicode_literals
-from os.path import dirname
-__author__ = u'Artur Barseghyan'
-__copyright__ = u'2013-2019 Artur Barseghyan'
-__license__ = u'MPL-1.1 OR GPL-2.0-only OR LGPL-2.1-or-later'
-__all__ = (u'DEBUG', u'NAMES_LOCAL_PATH_PARENT')
-NAMES_LOCAL_PATH_PARENT = dirname(__file__)
-DEBUG = False

Contents/Libraries/Shared/tld/exceptions.py

@@ -1,49 +0,0 @@
-# -*- coding: utf-8 -*-
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-from __future__ import unicode_literals
-from .conf import get_setting
-__author__ = u'Artur Barseghyan'
-__copyright__ = u'2013-2019 Artur Barseghyan'
-__license__ = u'MPL-1.1 OR GPL-2.0-only OR LGPL-2.1-or-later'
-__all__ = (u'TldBadUrl', u'TldDomainNotFound',
-           u'TldImproperlyConfigured', u'TldIOError')
-
-
-class TldIOError(IOError):
-    u'TldIOError.\n\n    Supposed to be thrown when problems with reading/writing occur.\n    '
-
-    def __init__(self, msg=None):
-        tld_names_local_path = get_setting(u'NAMES_LOCAL_PATH')
-        if (msg is None):
-            msg = (u"Can't read from or write to the %s file!" %
-                   tld_names_local_path)
-        super(TldIOError, self).__init__(msg)
-
-
-class TldDomainNotFound(ValueError):
-    u"TldDomainNotFound.\n\n    Supposed to be thrown when domain name is not found (didn't match) the\n    local TLD policy.\n    "
-
-    def __init__(self, domain_name):
-        super(TldDomainNotFound, self).__init__(
-            (u"Domain %s didn't match any existing TLD name!" % domain_name))
-
-
-class TldBadUrl(ValueError):
-    u'TldBadUrl.\n\n    Supposed to be thrown when bad URL is given.\n    '
-
-    def __init__(self, url):
-        super(TldBadUrl, self).__init__((u'Is not a valid URL %s!' % url))
-
-
-class TldImproperlyConfigured(Exception):
-    u'TldImproperlyConfigured.\n\n    Supposed to be thrown when code is improperly configured. Typical use-case\n    is when user tries to use `get_tld` function with both `search_public` and\n    `search_private` set to False.\n    '
-
-    def __init__(self, msg=None):
-        if (msg is None):
-            msg = u'Improperly configured.'
-        else:
-            msg = (u'Improperly configured. %s' % msg)
-        super(TldImproperlyConfigured, self).__init__(msg)

Contents/Libraries/Shared/tld/helpers.py

@@ -1,21 +0,0 @@
-# -*- coding: utf-8 -*-
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-from __future__ import unicode_literals
-from os.path import abspath, join
-from .conf import get_setting
-__author__ = u'Artur Barseghyan'
-__copyright__ = u'2013-2019 Artur Barseghyan'
-__license__ = u'MPL-1.1 OR GPL-2.0-only OR LGPL-2.1-or-later'
-__all__ = (u'project_dir', u'PROJECT_DIR')
-
-
-def project_dir(base):
-    u'Project dir.'
-    tld_names_local_path_parent = get_setting(u'NAMES_LOCAL_PATH_PARENT')
-    return abspath(join(tld_names_local_path_parent, base).replace(u'\\', u'/'))
-
-
-PROJECT_DIR = project_dir

Contents/Libraries/Shared/tld/parsers/__init__.py

@@ -1,5 +0,0 @@
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-from __future__ import unicode_literals

Contents/Libraries/Shared/tld/parsers/mozilla.py

@@ -1,5 +0,0 @@
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-from __future__ import unicode_literals

Contents/Libraries/Shared/tld/registry.py

@@ -1,41 +0,0 @@
-# -*- coding: utf-8 -*-
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-from __future__ import unicode_literals
-from typing import Type, Dict
-__author__ = u'Artur Barseghyan'
-__copyright__ = u'2013-2019 Artur Barseghyan'
-__license__ = u'MPL-1.1 OR GPL-2.0-only OR LGPL-2.1-or-later'
-__all__ = (u'Registry',)
-
-
-class Registry(type):
-    REGISTRY = {
-
-    }
-
-    def __new__(cls, name, bases, attrs):
-        new_cls = type.__new__(cls, name, bases, attrs)
-        if getattr(new_cls, u'_uid', None):
-            cls.REGISTRY[new_cls._uid] = new_cls
-        return new_cls
-
-    @property
-    def _uid(cls):
-        return getattr(cls, 'uid', cls.__name__)
-
-    @classmethod
-    def reset(cls):
-        cls.REGISTRY = {
-
-        }
-
-    @classmethod
-    def get(cls, key, default=None):
-        return cls.REGISTRY.get(key, default)
-
-    @classmethod
-    def items(cls):
-        return cls.REGISTRY.items()

Contents/Libraries/Shared/tld/result.py

@@ -1,57 +0,0 @@
-# -*- coding: utf-8 -*-
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-from __future__ import unicode_literals
-from typing import Dict
-try:
-    from urllib.parse import SplitResult
-except ImportError:
-    from six.moves.urllib_parse import SplitResult
-__author__ = u'Artur Barseghyan'
-__copyright__ = u'2013-2019 Artur Barseghyan'
-__license__ = u'MPL-1.1 OR GPL-2.0-only OR LGPL-2.1-or-later'
-__all__ = (u'Result',)
-
-
-class Result(object):
-    u'Container.'
-    __slots__ = (u'subdomain', u'domain', u'tld', u'__fld', u'parsed_url')
-
-    def __init__(self, tld, domain, subdomain, parsed_url):
-        self.tld = tld
-        self.domain = (domain if (domain != u'') else tld)
-        self.subdomain = subdomain
-        self.parsed_url = parsed_url
-        if domain:
-            self.__fld = u''.join(
-                [u'{}'.format(self.domain), u'.', u'{}'.format(self.tld)])
-        else:
-            self.__fld = self.tld
-
-    @property
-    def extension(self):
-        u'Alias of ``tld``.\n\n        :return str:\n        '
-        return self.tld
-    suffix = extension
-
-    @property
-    def fld(self):
-        u'First level domain.\n\n        :return:\n        :rtype: str\n        '
-        return self.__fld
-
-    def __str__(self):
-        return self.tld
-    __repr__ = __str__
-
-    @property
-    def __dict__(self):
-        u'Mimic __dict__ functionality.\n\n        :return:\n        :rtype: dict\n        '
-        return {
-            u'tld': self.tld,
-            u'domain': self.domain,
-            u'subdomain': self.subdomain,
-            u'fld': self.fld,
-            u'parsed_url': self.parsed_url,
-        }

Contents/Libraries/Shared/tld/tests/__init__.py

@@ -1,11 +0,0 @@
-# -*- coding: utf-8 -*-
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-from __future__ import unicode_literals
-import unittest
-from .test_core import *
-from .test_commands import *
-if (__name__ == u'__main__'):
-    unittest.main()

Contents/Libraries/Shared/tld/tests/base.py

@@ -1,65 +0,0 @@
-# -*- coding: utf-8 -*-
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-from __future__ import unicode_literals
-from backports.functools_lru_cache import lru_cache
-import logging
-import socket
-__author__ = u'Artur Barseghyan'
-__copyright__ = u'2013-2019 Artur Barseghyan'
-__license__ = u'MPL-1.1 OR GPL-2.0-only OR LGPL-2.1-or-later'
-__all__ = (u'internet_available_only', u'log_info')
-LOG_INFO = True
-LOGGER = logging.getLogger(__name__)
-
-
-def log_info(func):
-    u'Log some useful info.'
-    if (not LOG_INFO):
-        return func
-
-    def inner(self, *args, **kwargs):
-        u'Inner.'
-        result = func(*([self] + list(args)), **kwargs)
-        LOGGER.debug(u'\n\n%s', func.__name__)
-        LOGGER.debug(u'============================')
-        if func.__doc__:
-            LOGGER.debug(u'""" %s """', func.__doc__.strip())
-        LOGGER.debug(u'----------------------------')
-        if (result is not None):
-            LOGGER.debug(result)
-        LOGGER.debug(u'\n++++++++++++++++++++++++++++')
-        return result
-    return inner
-
-
-@lru_cache(maxsize=32)
-def is_internet_available(host='8.8.8.8', port=53, timeout=3):
-    u'Check if internet is available.\n\n    Host: 8.8.8.8 (google-public-dns-a.google.com)\n    OpenPort: 53/tcp\n    Service: domain (DNS/TCP)\n    '
-    try:
-        socket.setdefaulttimeout(timeout)
-        socket.socket(socket.AF_INET, socket.SOCK_STREAM).connect((host, port))
-        return True
-    except socket.error as ex:
-        print(ex)
-        return False
-
-
-def internet_available_only(func):
-
-    def inner(self, *args, **kwargs):
-        u'Inner.'
-        if (not is_internet_available()):
-            LOGGER.debug(u'\n\n%s', func.__name__)
-            LOGGER.debug(u'============================')
-            if func.__doc__:
-                LOGGER.debug(u'""" %s """', func.__doc__.strip())
-            LOGGER.debug(u'----------------------------')
-            LOGGER.debug(u'Skipping because no Internet connection available.')
-            LOGGER.debug(u'\n++++++++++++++++++++++++++++')
-            return None
-        result = func(*([self] + list(args)), **kwargs)
-        return result
-    return inner

Contents/Libraries/Shared/tld/tests/test_commands.py

@@ -1,43 +0,0 @@
-# -*- coding: utf-8 -*-
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-from __future__ import unicode_literals
-import logging
-import unittest
-import subprocess
-from .base import log_info, internet_available_only
-__author__ = u'Artur Barseghyan'
-__copyright__ = u'2013-2019 Artur Barseghyan'
-__license__ = u'GPL 2.0/LGPL 2.1'
-__all__ = (u'TestCommands',)
-LOGGER = logging.getLogger(__name__)
-
-
-class TestCommands(unittest.TestCase):
-    u'Tld commands tests.'
-
-    def setUp(self):
-        u'Set up.'
-
-    @internet_available_only
-    @log_info
-    def test_1_update_tld_names_command(self):
-        u'Test updating the tld names (re-fetch mozilla source).'
-        res = subprocess.check_output([u'update-tld-names']).strip()
-        self.assertEqual(res, b'')
-        return res
-
-    @internet_available_only
-    @log_info
-    def test_1_update_tld_names_mozilla_command(self):
-        u'Test updating the tld names (re-fetch mozilla source).'
-        res = subprocess.check_output(
-            [u'update-tld-names', u'mozilla']).strip()
-        self.assertEqual(res, b'')
-        return res
-
-
-if (__name__ == u'__main__'):
-    unittest.main()

Contents/Libraries/Shared/tld/tests/test_core.py

@@ -1,708 +0,0 @@
-# -*- coding: utf-8 -*-
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-
-import copy
-import logging
-from os.path import abspath, join
-import unittest
-from tempfile import gettempdir
-from typing import Type
-try:
-    from urllib.parse import urlsplit
-except ImportError:
-    from six.moves.urllib_parse import urlsplit
-from faker import Faker
-from .. import defaults
-from ..base import BaseTLDSourceParser
-from ..conf import get_setting, reset_settings, set_setting
-from ..exceptions import TldBadUrl, TldDomainNotFound, TldImproperlyConfigured, TldIOError
-from ..helpers import project_dir
-from ..registry import Registry
-from ..utils import get_fld, get_tld, get_tld_names, get_tld_names_container, is_tld, MozillaTLDSourceParser, BaseMozillaTLDSourceParser, parse_tld, reset_tld_names, update_tld_names, update_tld_names_cli
-from .base import internet_available_only, log_info
-__author__ = u'Artur Barseghyan'
-__copyright__ = u'2013-2019 Artur Barseghyan'
-__license__ = u'MPL-1.1 OR GPL-2.0-only OR LGPL-2.1-or-later'
-__all__ = (u'TestCore',)
-LOGGER = logging.getLogger(__name__)
-
-
-class TestCore(unittest.TestCase):
-    u'Core tld functionality tests.'
-
-    @classmethod
-    def setUpClass(cls):
-        cls.faker = Faker()
-        cls.temp_dir = gettempdir()
-
-    def setUp(self):
-        u'Set up.'
-        self.good_patterns = [{
-            u'url': u'http://www.google.co.uk',
-            u'fld': u'google.co.uk',
-            u'subdomain': u'www',
-            u'domain': u'google',
-            u'suffix': u'co.uk',
-            u'tld': u'co.uk',
-            u'kwargs': {
-                u'fail_silently': True,
-            },
-        }, {
-            u'url': u'http://www.v2.google.co.uk',
-            u'fld': u'google.co.uk',
-            u'subdomain': u'www.v2',
-            u'domain': u'google',
-            u'suffix': u'co.uk',
-            u'tld': u'co.uk',
-            u'kwargs': {
-                u'fail_silently': True,
-            },
-        }, {
-            u'url': u'http://хром.гугл.рф',
-            u'fld': u'гугл.рф',
-            u'subdomain': u'хром',
-            u'domain': u'гугл',
-            u'suffix': u'рф',
-            u'tld': u'рф',
-            u'kwargs': {
-                u'fail_silently': True,
-            },
-        }, {
-            u'url': u'http://www.google.co.uk:8001/lorem-ipsum/',
-            u'fld': u'google.co.uk',
-            u'subdomain': u'www',
-            u'domain': u'google',
-            u'suffix': u'co.uk',
-            u'tld': u'co.uk',
-            u'kwargs': {
-                u'fail_silently': True,
-            },
-        }, {
-            u'url': u'http://www.me.cloudfront.net',
-            u'fld': u'me.cloudfront.net',
-            u'subdomain': u'www',
-            u'domain': u'me',
-            u'suffix': u'cloudfront.net',
-            u'tld': u'cloudfront.net',
-            u'kwargs': {
-                u'fail_silently': True,
-            },
-        }, {
-            u'url': u'http://www.v2.forum.tech.google.co.uk:8001/lorem-ipsum/',
-            u'fld': u'google.co.uk',
-            u'subdomain': u'www.v2.forum.tech',
-            u'domain': u'google',
-            u'suffix': u'co.uk',
-            u'tld': u'co.uk',
-            u'kwargs': {
-                u'fail_silently': True,
-            },
-        }, {
-            u'url': u'https://pantheon.io/',
-            u'fld': u'pantheon.io',
-            u'subdomain': u'',
-            u'domain': u'pantheon',
-            u'suffix': u'io',
-            u'tld': u'io',
-            u'kwargs': {
-                u'fail_silently': True,
-            },
-        }, {
-            u'url': u'v2.www.google.com',
-            u'fld': u'google.com',
-            u'subdomain': u'v2.www',
-            u'domain': u'google',
-            u'suffix': u'com',
-            u'tld': u'com',
-            u'kwargs': {
-                u'fail_silently': True,
-                u'fix_protocol': True,
-            },
-        }, {
-            u'url': u'//v2.www.google.com',
-            u'fld': u'google.com',
-            u'subdomain': u'v2.www',
-            u'domain': u'google',
-            u'suffix': u'com',
-            u'tld': u'com',
-            u'kwargs': {
-                u'fail_silently': True,
-                u'fix_protocol': True,
-            },
-        }, {
-            u'url': u'http://foo@bar.com',
-            u'fld': u'bar.com',
-            u'subdomain': u'',
-            u'domain': u'bar',
-            u'suffix': u'com',
-            u'tld': u'com',
-            u'kwargs': {
-                u'fail_silently': True,
-            },
-        }, {
-            u'url': u'http://user:foo@bar.com',
-            u'fld': u'bar.com',
-            u'subdomain': u'',
-            u'domain': u'bar',
-            u'suffix': u'com',
-            u'tld': u'com',
-            u'kwargs': {
-                u'fail_silently': True,
-            },
-        }, {
-            u'url': u'https://faguoren.xn--fiqs8s',
-            u'fld': u'faguoren.xn--fiqs8s',
-            u'subdomain': u'',
-            u'domain': u'faguoren',
-            u'suffix': u'xn--fiqs8s',
-            u'tld': u'xn--fiqs8s',
-            u'kwargs': {
-                u'fail_silently': True,
-            },
-        }, {
-            u'url': u'blogs.lemonde.paris',
-            u'fld': u'lemonde.paris',
-            u'subdomain': u'blogs',
-            u'domain': u'lemonde',
-            u'suffix': u'paris',
-            u'tld': u'paris',
-            u'kwargs': {
-                u'fail_silently': True,
-                u'fix_protocol': True,
-            },
-        }, {
-            u'url': u'axel.brighton.ac.uk',
-            u'fld': u'brighton.ac.uk',
-            u'subdomain': u'axel',
-            u'domain': u'brighton',
-            u'suffix': u'ac.uk',
-            u'tld': u'ac.uk',
-            u'kwargs': {
-                u'fail_silently': True,
-                u'fix_protocol': True,
-            },
-        }, {
-            u'url': u'm.fr.blogspot.com.au',
-            u'fld': u'fr.blogspot.com.au',
-            u'subdomain': u'm',
-            u'domain': u'fr',
-            u'suffix': u'blogspot.com.au',
-            u'tld': u'blogspot.com.au',
-            u'kwargs': {
-                u'fail_silently': True,
-                u'fix_protocol': True,
-            },
-        }, {
-            u'url': u'help.www.福岡.jp',
-            u'fld': u'www.福岡.jp',
-            u'subdomain': u'help',
-            u'domain': u'www',
-            u'suffix': u'福岡.jp',
-            u'tld': u'福岡.jp',
-            u'kwargs': {
-                u'fail_silently': True,
-                u'fix_protocol': True,
-            },
-        }, {
-            u'url': u'syria.arabic.variant.سوريا',
-            u'fld': u'variant.سوريا',
-            u'subdomain': u'syria.arabic',
-            u'domain': u'variant',
-            u'suffix': u'سوريا',
-            u'tld': u'سوريا',
-            u'kwargs': {
-                u'fail_silently': True,
-                u'fix_protocol': True,
-            },
-        }, {
-            u'url': u'http://www.help.kawasaki.jp',
-            u'fld': u'www.help.kawasaki.jp',
-            u'subdomain': u'',
-            u'domain': u'www',
-            u'suffix': u'help.kawasaki.jp',
-            u'tld': u'help.kawasaki.jp',
-            u'kwargs': {
-                u'fail_silently': True,
-            },
-        }, {
-            u'url': u'http://www.city.kawasaki.jp',
-            u'fld': u'city.kawasaki.jp',
-            u'subdomain': u'www',
-            u'domain': u'city',
-            u'suffix': u'kawasaki.jp',
-            u'tld': u'kawasaki.jp',
-            u'kwargs': {
-                u'fail_silently': True,
-            },
-        }, {
-            u'url': u'http://fedoraproject.org',
-            u'fld': u'fedoraproject.org',
-            u'subdomain': u'',
-            u'domain': u'fedoraproject',
-            u'suffix': u'org',
-            u'tld': u'org',
-            u'kwargs': {
-                u'fail_silently': True,
-            },
-        }, {
-            u'url': u'http://www.cloud.fedoraproject.org',
-            u'fld': u'www.cloud.fedoraproject.org',
-            u'subdomain': u'',
-            u'domain': u'www',
-            u'suffix': u'cloud.fedoraproject.org',
-            u'tld': u'cloud.fedoraproject.org',
-            u'kwargs': {
-                u'fail_silently': True,
-            },
-        }, {
-            u'url': u'https://www.john.app.os.fedoraproject.org',
-            u'fld': u'john.app.os.fedoraproject.org',
-            u'subdomain': u'www',
-            u'domain': u'john',
-            u'suffix': u'app.os.fedoraproject.org',
-            u'tld': u'app.os.fedoraproject.org',
-            u'kwargs': {
-                u'fail_silently': True,
-            },
-        }, {
-            u'url': u'ftp://www.xn--mxail5aa.xn--11b4c3d',
-            u'fld': u'xn--mxail5aa.xn--11b4c3d',
-            u'subdomain': u'www',
-            u'domain': u'xn--mxail5aa',
-            u'suffix': u'xn--11b4c3d',
-            u'tld': u'xn--11b4c3d',
-            u'kwargs': {
-                u'fail_silently': True,
-            },
-        }, {
-            u'url': u'http://cloud.fedoraproject.org',
-            u'fld': u'cloud.fedoraproject.org',
-            u'subdomain': u'',
-            u'domain': u'cloud.fedoraproject.org',
-            u'suffix': u'cloud.fedoraproject.org',
-            u'tld': u'cloud.fedoraproject.org',
-            u'kwargs': {
-                u'fail_silently': True,
-            },
-        }, {
-            u'url': u'github.io',
-            u'fld': u'github.io',
-            u'subdomain': u'',
-            u'domain': u'github.io',
-            u'suffix': u'github.io',
-            u'tld': u'github.io',
-            u'kwargs': {
-                u'fail_silently': True,
-                u'fix_protocol': True,
-            },
-        }, {
-            u'url': urlsplit(u'http://lemonde.fr/article.html'),
-            u'fld': u'lemonde.fr',
-            u'subdomain': u'',
-            u'domain': u'lemonde',
-            u'suffix': u'fr',
-            u'tld': u'fr',
-            u'kwargs': {
-                u'fail_silently': True,
-            },
-        }]
-        self.bad_patterns = {
-            u'v2.www.google.com': {
-                u'exception': TldBadUrl,
-            },
-            u'/index.php?a=1&b=2': {
-                u'exception': TldBadUrl,
-            },
-            u'http://www.tld.doesnotexist': {
-                u'exception': TldDomainNotFound,
-            },
-            u'https://2001:0db8:0000:85a3:0000:0000:ac1f:8001': {
-                u'exception': TldDomainNotFound,
-            },
-            u'http://192.169.1.1': {
-                u'exception': TldDomainNotFound,
-            },
-            u'http://localhost:8080': {
-                u'exception': TldDomainNotFound,
-            },
-            u'https://localhost': {
-                u'exception': TldDomainNotFound,
-            },
-            u'https://localhost2': {
-                u'exception': TldImproperlyConfigured,
-                u'kwargs': {
-                    u'search_public': False,
-                    u'search_private': False,
-                },
-            },
-        }
-        self.invalid_tlds = {u'v2.www.google.com', u'tld.doesnotexist',
-                             u'2001:0db8:0000:85a3:0000:0000:ac1f', u'192.169.1.1', 'localhost', u'google.com'}
-        self.tld_names_local_path_custom = project_dir(
-            join(u'tests', u'res', u'effective_tld_names_custom.dat.txt'))
-        self.good_patterns_custom_parser = [{
-            u'url': u'http://www.foreverchild',
-            u'fld': u'www.foreverchild',
-            u'subdomain': u'',
-            u'domain': u'www',
-            u'suffix': u'foreverchild',
-            u'tld': u'foreverchild',
-            u'kwargs': {
-                u'fail_silently': True,
-            },
-        }, {
-            u'url': u'http://www.v2.foreverchild',
-            u'fld': u'v2.foreverchild',
-            u'subdomain': u'www',
-            u'domain': u'v2',
-            u'suffix': u'foreverchild',
-            u'tld': u'foreverchild',
-            u'kwargs': {
-                u'fail_silently': True,
-            },
-        }]
-        reset_settings()
-
-    def tearDown(self):
-        u'Tear down.'
-        reset_settings()
-        Registry.reset()
-
-    @property
-    def good_url(self):
-        return self.good_patterns[0][u'url']
-
-    @property
-    def bad_url(self):
-        return list(self.bad_patterns.keys())[0]
-
-    def get_custom_parser_class(self, uid='custom_mozilla', source_url=None, local_path='tests/res/effective_tld_names_custom.dat.txt'):
-        parser_class = type('CustomMozillaTLDSourceParser', (BaseMozillaTLDSourceParser,), {
-            'uid': uid,
-            'source_url': source_url,
-            'local_path': local_path,
-        })
-        return parser_class
-
-    @log_info
-    def test_0_tld_names_loaded(self):
-        u'Test if tld names are loaded.'
-        get_fld(u'http://www.google.co.uk')
-        from ..utils import tld_names
-        res = (len(tld_names) > 0)
-        self.assertTrue(res)
-        return res
-
-    @internet_available_only
-    @log_info
-    def test_1_update_tld_names(self):
-        u'Test updating the tld names (re-fetch mozilla source).'
-        res = update_tld_names(fail_silently=False)
-        self.assertTrue(res)
-        return res
-
-    @log_info
-    def test_2_fld_good_patterns_pass(self):
-        u'Test good URL patterns.'
-        res = []
-        for data in self.good_patterns:
-            _res = get_fld(data[u'url'], **data[u'kwargs'])
-            self.assertEqual(_res, data[u'fld'])
-            res.append(_res)
-        return res
-
-    @log_info
-    def test_3_fld_bad_patterns_pass(self):
-        u'Test bad URL patterns.'
-        res = []
-        for (url, params) in self.bad_patterns.items():
-            _res = get_fld(url, fail_silently=True)
-            self.assertEqual(_res, None)
-            res.append(_res)
-        return res
-
-    @log_info
-    def test_4_override_settings(self):
-        u'Testing settings override.'
-
-        def override_settings():
-            u'Override settings.'
-            return get_setting(u'DEBUG')
-        self.assertEqual(defaults.DEBUG, override_settings())
-        set_setting(u'DEBUG', True)
-        self.assertEqual(True, override_settings())
-        return override_settings()
-
-    @log_info
-    def test_5_tld_good_patterns_pass_parsed_object(self):
-        u'Test good URL patterns.'
-        res = []
-        for data in self.good_patterns:
-            kwargs = copy.copy(data[u'kwargs'])
-            kwargs.update({
-                u'as_object': True,
-            })
-            _res = get_tld(data[u'url'], **kwargs)
-            self.assertEqual(_res.tld, data[u'tld'])
-            self.assertEqual(_res.subdomain, data[u'subdomain'])
-            self.assertEqual(_res.domain, data[u'domain'])
-            self.assertEqual(_res.suffix, data[u'suffix'])
-            self.assertEqual(_res.fld, data[u'fld'])
-            self.assertEqual(unicode(_res).encode(u'utf8'),
-                             data[u'tld'].encode(u'utf8'))
-            self.assertEqual(_res.__dict__, {
-                u'tld': _res.tld,
-                u'domain': _res.domain,
-                u'subdomain': _res.subdomain,
-                u'fld': _res.fld,
-                u'parsed_url': _res.parsed_url,
-            })
-            res.append(_res)
-        return res
-
-    @log_info
-    def test_6_override_full_names_path(self):
-        default = project_dir(u'dummy.txt')
-        override_base = u'/tmp/test'
-        set_setting(u'NAMES_LOCAL_PATH_PARENT', override_base)
-        modified = project_dir(u'dummy.txt')
-        self.assertNotEqual(default, modified)
-        self.assertEqual(modified, abspath(u'/tmp/test/dummy.txt'))
-
-    @log_info
-    def test_7_public_private(self):
-        res = get_fld(u'http://silly.cc.ua',
-                      fail_silently=True, search_private=False)
-        self.assertEqual(res, None)
-        res = get_fld(u'http://silly.cc.ua',
-                      fail_silently=True, search_private=True)
-        self.assertEqual(res, u'silly.cc.ua')
-        res = get_fld(u'mercy.compute.amazonaws.com',
-                      fail_silently=True, search_private=False, fix_protocol=True)
-        self.assertEqual(res, None)
-        res = get_fld(u'http://whatever.com',
-                      fail_silently=True, search_public=False)
-        self.assertEqual(res, None)
-
-    @log_info
-    def test_8_fld_bad_patterns_exceptions(self):
-        u'Test exceptions.'
-        res = []
-        for (url, params) in self.bad_patterns.items():
-            kwargs = (params[u'kwargs'] if (u'kwargs' in params) else {
-
-            })
-            kwargs.update({
-                u'fail_silently': False,
-            })
-            with self.assertRaises(params[u'exception']):
-                _res = get_fld(url, **kwargs)
-                res.append(_res)
-        return res
-
-    @log_info
-    def test_9_tld_good_patterns_pass(self):
-        u'Test `get_tld` good URL patterns.'
-        res = []
-        for data in self.good_patterns:
-            _res = get_tld(data[u'url'], **data[u'kwargs'])
-            self.assertEqual(_res, data[u'tld'])
-            res.append(_res)
-        return res
-
-    @log_info
-    def test_10_tld_bad_patterns_pass(self):
-        u'Test `get_tld` bad URL patterns.'
-        res = []
-        for (url, params) in self.bad_patterns.items():
-            _res = get_tld(url, fail_silently=True)
-            self.assertEqual(_res, None)
-            res.append(_res)
-        return res
-
-    @log_info
-    def test_11_parse_tld_good_patterns(self):
-        u'Test `parse_tld` good URL patterns.'
-        res = []
-        for data in self.good_patterns:
-            _res = parse_tld(data[u'url'], **data[u'kwargs'])
-            self.assertEqual(
-                _res, (data[u'tld'], data[u'domain'], data[u'subdomain']))
-            res.append(_res)
-        return res
-
-    @log_info
-    def test_12_is_tld_good_patterns(self):
-        u'Test `is_tld` good URL patterns.'
-        for data in self.good_patterns:
-            self.assertTrue(is_tld(data[u'tld']))
-
-    @log_info
-    def test_13_is_tld_bad_patterns(self):
-        u'Test `is_tld` bad URL patterns.'
-        for _tld in self.invalid_tlds:
-            self.assertFalse(is_tld(_tld))
-
-    @log_info
-    def test_14_fail_update_tld_names(self):
-        u'Test fail `update_tld_names`.'
-        parser_class = self.get_custom_parser_class(
-            uid='custom_mozilla_2', source_url='i-do-not-exist')
-        with self.assertRaises(TldIOError):
-            update_tld_names(fail_silently=False, parser_uid=parser_class.uid)
-        self.assertFalse(update_tld_names(
-            fail_silently=True, parser_uid=parser_class.uid))
-
-    @log_info
-    def test_15_fail_get_fld_wrong_kwargs(self):
-        u'Test fail `get_fld` with wrong kwargs.'
-        with self.assertRaises(TldImproperlyConfigured):
-            get_fld(self.good_url, as_object=True)
-
-    @log_info
-    def test_16_fail_parse_tld(self):
-        u'Test fail `parse_tld`.\n\n        Assert raise TldIOError on wrong `NAMES_SOURCE_URL` for `parse_tld`.\n        '
-        parser_class = self.get_custom_parser_class(
-            source_url='i-do-not-exist')
-        parsed_tld = parse_tld(
-            self.bad_url, fail_silently=False, parser_class=parser_class)
-        self.assertEqual(parsed_tld, (None, None, None))
-
-    @log_info
-    def test_17_get_tld_names_and_reset_tld_names(self):
-        u'Test fail `get_tld_names` and repair using `reset_tld_names`.'
-        tmp_filename = join(gettempdir(), u''.join(
-            [u'{}'.format(self.faker.uuid4()), u'.dat.txt']))
-        parser_class = self.get_custom_parser_class(
-            source_url='i-do-not-exist', local_path=tmp_filename)
-        reset_tld_names()
-        if True:
-            with self.assertRaises(TldIOError):
-                get_tld_names(fail_silently=False, parser_class=parser_class)
-        tmp_filename = join(gettempdir(), u''.join(
-            [u'{}'.format(self.faker.uuid4()), u'.dat.txt']))
-        parser_class_2 = self.get_custom_parser_class(
-            source_url='i-do-not-exist-2', local_path=tmp_filename)
-        reset_tld_names()
-        if True:
-            self.assertIsNone(get_tld_names(
-                fail_silently=True, parser_class=parser_class_2))
-
-    @internet_available_only
-    @log_info
-    def test_18_update_tld_names_cli(self):
-        u'Test the return code of the CLI version of `update_tld_names`.'
-        reset_tld_names()
-        res = update_tld_names_cli()
-        self.assertEqual(res, 0)
-
-    @log_info
-    def test_19_parse_tld_custom_tld_names_good_patterns(self):
-        u'Test `parse_tld` good URL patterns for custom tld names.'
-        res = []
-        for data in self.good_patterns_custom_parser:
-            kwargs = copy.copy(data[u'kwargs'])
-            kwargs.update({
-                u'parser_class': self.get_custom_parser_class(),
-            })
-            _res = parse_tld(data[u'url'], **kwargs)
-            self.assertEqual(
-                _res, (data[u'tld'], data[u'domain'], data[u'subdomain']))
-            res.append(_res)
-        return res
-
-    @log_info
-    def test_20_tld_custom_tld_names_good_patterns_pass_parsed_object(self):
-        u'Test `get_tld` good URL patterns for custom tld names.'
-        res = []
-        for data in self.good_patterns_custom_parser:
-            kwargs = copy.copy(data[u'kwargs'])
-            kwargs.update({
-                u'as_object': True,
-                u'parser_class': self.get_custom_parser_class(),
-            })
-            _res = get_tld(data[u'url'], **kwargs)
-            self.assertEqual(_res.tld, data[u'tld'])
-            self.assertEqual(_res.subdomain, data[u'subdomain'])
-            self.assertEqual(_res.domain, data[u'domain'])
-            self.assertEqual(_res.suffix, data[u'suffix'])
-            self.assertEqual(_res.fld, data[u'fld'])
-            self.assertEqual(unicode(_res).encode(u'utf8'),
-                             data[u'tld'].encode(u'utf8'))
-            self.assertEqual(_res.__dict__, {
-                u'tld': _res.tld,
-                u'domain': _res.domain,
-                u'subdomain': _res.subdomain,
-                u'fld': _res.fld,
-                u'parsed_url': _res.parsed_url,
-            })
-            res.append(_res)
-        return res
-
-    @log_info
-    def test_21_reset_tld_names_for_custom_parser(self):
-        u'Test `reset_tld_names` for `tld_names_local_path`.'
-        res = []
-        parser_class = self.get_custom_parser_class()
-        for data in self.good_patterns_custom_parser:
-            kwargs = copy.copy(data[u'kwargs'])
-            kwargs.update({
-                u'as_object': True,
-                u'parser_class': self.get_custom_parser_class(),
-            })
-            _res = get_tld(data[u'url'], **kwargs)
-            self.assertEqual(_res.tld, data[u'tld'])
-            self.assertEqual(_res.subdomain, data[u'subdomain'])
-            self.assertEqual(_res.domain, data[u'domain'])
-            self.assertEqual(_res.suffix, data[u'suffix'])
-            self.assertEqual(_res.fld, data[u'fld'])
-            self.assertEqual(unicode(_res).encode(u'utf8'),
-                             data[u'tld'].encode(u'utf8'))
-            self.assertEqual(_res.__dict__, {
-                u'tld': _res.tld,
-                u'domain': _res.domain,
-                u'subdomain': _res.subdomain,
-                u'fld': _res.fld,
-                u'parsed_url': _res.parsed_url,
-            })
-            res.append(_res)
-        tld_names = get_tld_names_container()
-        self.assertIn(parser_class.local_path, tld_names)
-        reset_tld_names(parser_class.local_path)
-        self.assertNotIn(parser_class.local_path, tld_names)
-        return res
-
-    @log_info
-    def test_22_fail_define_custom_parser_class_without_uid(self):
-        u'Test fail define custom parser class without `uid`.'
-
-        class CustomParser(BaseTLDSourceParser):
-            pass
-
-        class AnotherCustomParser(BaseTLDSourceParser):
-            uid = u'another-custom-parser'
-        with self.assertRaises(TldImproperlyConfigured):
-            CustomParser.get_tld_names()
-        with self.assertRaises(NotImplementedError):
-            AnotherCustomParser.get_tld_names()
-
-    @log_info
-    def test_23_len_trie_nodes(self):
-        u'Test len of the trie nodes.'
-        get_tld(u'http://delusionalinsanity.com')
-        tld_names = get_tld_names_container()
-        self.assertGreater(
-            len(tld_names[MozillaTLDSourceParser.local_path]), 0)
-
-    @log_info
-    def test_24_get_tld_names_no_arguments(self):
-        u'Test len of the trie nodes.'
-        tld_names = get_tld_names()
-        self.assertGreater(len(tld_names), 0)
-
-
-if (__name__ == u'__main__'):
-    unittest.main()

Contents/Libraries/Shared/tld/trie.py

@@ -1,54 +0,0 @@
-# -*- coding: utf-8 -*-
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-from __future__ import unicode_literals
-__author__ = u'Artur Barseghyan'
-__copyright__ = u'2013-2019 Artur Barseghyan'
-__license__ = u'MPL-1.1 OR GPL-2.0-only OR LGPL-2.1-or-later'
-__all__ = (u'Trie', u'TrieNode')
-
-
-class TrieNode(object):
-    u'Class representing a single Trie node.'
-    __slots__ = (u'children', u'exception', u'leaf', u'private')
-
-    def __init__(self):
-        self.children = None
-        self.exception = None
-        self.leaf = False
-        self.private = False
-
-
-class Trie(object):
-    u'An adhoc Trie data structure to store tlds in reverse notation order.'
-
-    def __init__(self):
-        self.root = TrieNode()
-        self.__nodes = 0
-
-    def __len__(self):
-        return self.__nodes
-
-    def add(self, tld, private=False):
-        node = self.root
-        for part in reversed(tld.split(u'.')):
-            if part.startswith(u'!'):
-                node.exception = part[1:]
-                break
-            if (node.children is None):
-                node.children = {
-
-                }
-                child = TrieNode()
-            else:
-                child = node.children.get(part)
-                if (child is None):
-                    child = TrieNode()
-            node.children[part] = child
-            node = child
-        node.leaf = True
-        if private:
-            node.private = True
-        self.__nodes += 1

Contents/Libraries/Shared/tld/utils.py

@@ -1,271 +0,0 @@
-# -*- coding: utf-8 -*-
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-from __future__ import unicode_literals
-from __future__ import unicode_literals
-import argparse
-from codecs import open as codecs_open
-from backports.functools_lru_cache import lru_cache
-from os.path import isabs
-import sys
-from typing import Dict, Type, Union, Tuple, List
-try:
-    from urllib.parse import urlsplit, SplitResult
-except ImportError:
-    from six.moves.urllib_parse import urlsplit, SplitResult
-from .base import BaseTLDSourceParser
-from .exceptions import TldBadUrl, TldDomainNotFound, TldImproperlyConfigured, TldIOError
-from .helpers import project_dir
-from .trie import Trie
-from .registry import Registry
-from .result import Result
-__author__ = u'Artur Barseghyan'
-__copyright__ = u'2013-2019 Artur Barseghyan'
-__license__ = u'MPL-1.1 OR GPL-2.0-only OR LGPL-2.1-or-later'
-__all__ = (u'BaseMozillaTLDSourceParser', u'get_fld', u'get_tld', u'get_tld_names', u'get_tld_names_container', u'is_tld', u'MozillaTLDSourceParser', u'parse_tld',
-           u'pop_tld_names_container', u'process_url', u'reset_tld_names', u'Result', u'update_tld_names', u'update_tld_names_cli', u'update_tld_names_container')
-tld_names = {
-
-}
-
-
-def get_tld_names_container():
-    u'Get container of all tld names.\n\n    :return:\n    :rtype dict:\n    '
-    global tld_names
-    return tld_names
-
-
-def update_tld_names_container(tld_names_local_path, trie_obj):
-    u'Update TLD Names container item.\n\n    :param tld_names_local_path:\n    :param trie_obj:\n    :return:\n    '
-    global tld_names
-    tld_names.update({
-        tld_names_local_path: trie_obj,
-    })
-
-
-def pop_tld_names_container(tld_names_local_path):
-    u'Remove TLD names container item.\n\n    :param tld_names_local_path:\n    :return:\n    '
-    global tld_names
-    tld_names.pop(tld_names_local_path, None)
-
-
-@lru_cache(maxsize=128, typed=True)
-def update_tld_names(fail_silently=False, parser_uid=None):
-    u'Update TLD names.\n\n    :param fail_silently:\n    :param parser_uid:\n    :return:\n    '
-    results = []
-    results_append = results.append
-    if parser_uid:
-        parser_cls = Registry.get(parser_uid, None)
-        if (parser_cls and parser_cls.source_url):
-            results_append(parser_cls.update_tld_names(
-                fail_silently=fail_silently))
-    else:
-        for (parser_uid, parser_cls) in Registry.items():
-            if (parser_cls and parser_cls.source_url):
-                results_append(parser_cls.update_tld_names(
-                    fail_silently=fail_silently))
-    return all(results)
-
-
-def update_tld_names_cli():
-    u'CLI wrapper for update_tld_names.\n\n    Since update_tld_names returns True on success, we need to negate the\n    result to match CLI semantics.\n    '
-    parser = argparse.ArgumentParser(description='Update TLD names')
-    parser.add_argument(u'parser_uid', nargs='?', default=None,
-                        help='UID of the parser to update TLD names for.')
-    parser.add_argument(u'--fail-silently', dest='fail_silently',
-                        default=False, action='store_true', help='Fail silently')
-    args = parser.parse_args(sys.argv[1:])
-    parser_uid = args.parser_uid
-    fail_silently = args.fail_silently
-    return int((not update_tld_names(parser_uid=parser_uid, fail_silently=fail_silently)))
-
-
-def get_tld_names(fail_silently=False, retry_count=0, parser_class=None):
-    u'Build the ``tlds`` list if empty. Recursive.\n\n    :param fail_silently: If set to True, no exceptions are raised and None\n        is returned on failure.\n    :param retry_count: If greater than 1, we raise an exception in order\n        to avoid infinite loops.\n    :param parser_class:\n    :type fail_silently: bool\n    :type retry_count: int\n    :type parser_class: BaseTLDSourceParser\n    :return: List of TLD names\n    :rtype: obj:`tld.utils.Trie`\n    '
-    if (not parser_class):
-        parser_class = MozillaTLDSourceParser
-    return parser_class.get_tld_names(fail_silently=fail_silently, retry_count=retry_count)
-
-
-class BaseMozillaTLDSourceParser(BaseTLDSourceParser):
-
-    @classmethod
-    def get_tld_names(cls, fail_silently=False, retry_count=0):
-        u'Parse.\n\n        :param fail_silently:\n        :param retry_count:\n        :return:\n        '
-        if (retry_count > 1):
-            if fail_silently:
-                return None
-            else:
-                raise TldIOError
-        global tld_names
-        _tld_names = tld_names
-        if ((cls.local_path in _tld_names) and (_tld_names[cls.local_path] is not None)):
-            return _tld_names
-        local_file = None
-        try:
-            if isabs(cls.local_path):
-                local_path = cls.local_path
-            else:
-                local_path = project_dir(cls.local_path)
-            local_file = codecs_open(local_path, u'r', encoding='utf8')
-            trie = Trie()
-            trie_add = trie.add
-            private_section = False
-            for line in local_file:
-                if (u'===BEGIN PRIVATE DOMAINS===' in line):
-                    private_section = True
-                if (u'// xn--' in line):
-                    line = line.split()[1]
-                if (line[0] in (u'/', u'\n')):
-                    continue
-                trie_add(u''.join([u'{}'.format(line.strip())]),
-                         private=private_section)
-            update_tld_names_container(cls.local_path, trie)
-            local_file.close()
-        except IOError as err:
-            cls.update_tld_names(fail_silently=fail_silently)
-            retry_count += 1
-            return cls.get_tld_names(fail_silently=fail_silently, retry_count=retry_count)
-        except Exception as err:
-            if fail_silently:
-                return None
-            else:
-                raise err
-        finally:
-            try:
-                local_file.close()
-            except Exception:
-                pass
-        return _tld_names
-
-
-class MozillaTLDSourceParser(BaseMozillaTLDSourceParser):
-    u'Mozilla TLD source.'
-    uid = u'mozilla'
-    source_url = u'http://mxr.mozilla.org/mozilla/source/netwerk/dns/src/effective_tld_names.dat?raw=1'
-    local_path = u'res/effective_tld_names.dat.txt'
-
-
-def process_url(url, fail_silently=False, fix_protocol=False, search_public=True, search_private=True, parser_class=MozillaTLDSourceParser):
-    u'Process URL.\n\n    :param parser_class:\n    :param url:\n    :param fail_silently:\n    :param fix_protocol:\n    :param search_public:\n    :param search_private:\n    :return:\n    '
-    if (not (search_public or search_private)):
-        raise TldImproperlyConfigured(
-            u'Either `search_public` or `search_private` (or both) shall be set to True.')
-    _tld_names = get_tld_names(
-        fail_silently=fail_silently, parser_class=parser_class)
-    if (not isinstance(url, SplitResult)):
-        url = url.lower()
-        if (fix_protocol and (not url.startswith((u'//', u'http://', u'https://')))):
-            url = u''.join([u'https://', u'{}'.format(url)])
-        parsed_url = urlsplit(url)
-    else:
-        parsed_url = url
-    domain_name = parsed_url.hostname
-    if (not domain_name):
-        if fail_silently:
-            return (None, None, parsed_url)
-        else:
-            raise TldBadUrl(url=url)
-    domain_parts = domain_name.split(u'.')
-    tld_names_local_path = parser_class.local_path
-    node = _tld_names[tld_names_local_path].root
-    current_length = 0
-    tld_length = 0
-    match = None
-    len_domain_parts = len(domain_parts)
-    for i in reversed(range(len_domain_parts)):
-        part = domain_parts[i]
-        if (node.children is None):
-            break
-        if (part == node.exception):
-            break
-        child = node.children.get(part)
-        if (child is None):
-            child = node.children.get(u'*')
-        if (child is None):
-            break
-        current_length += 1
-        node = child
-        if node.leaf:
-            tld_length = current_length
-            match = node
-    if ((match is None) or (not match.leaf) or ((not search_public) and (not match.private)) or ((not search_private) and match.private)):
-        if fail_silently:
-            return (None, None, parsed_url)
-        else:
-            raise TldDomainNotFound(domain_name=domain_name)
-    if (len_domain_parts == tld_length):
-        non_zero_i = (- 1)
-    else:
-        non_zero_i = max(1, (len_domain_parts - tld_length))
-    return (domain_parts, non_zero_i, parsed_url)
-
-
-def get_fld(url, fail_silently=False, fix_protocol=False, search_public=True, search_private=True, parser_class=MozillaTLDSourceParser, **kwargs):
-    u"Extract the first level domain.\n\n    Extract the top level domain based on the mozilla's effective TLD names\n    dat file. Returns a string. May throw ``TldBadUrl`` or\n    ``TldDomainNotFound`` exceptions if there's bad URL provided or no TLD\n    match found respectively.\n\n    :param url: URL to get top level domain from.\n    :param fail_silently: If set to True, no exceptions are raised and None\n        is returned on failure.\n    :param fix_protocol: If set to True, missing or wrong protocol is\n        ignored (https is appended instead).\n    :param search_public: If set to True, search in public domains.\n    :param search_private: If set to True, search in private domains.\n    :param parser_class:\n    :type url: str\n    :type fail_silently: bool\n    :type fix_protocol: bool\n    :type search_public: bool\n    :type search_private: bool\n    :return: String with top level domain (if ``as_object`` argument\n        is set to False) or a ``tld.utils.Result`` object (if ``as_object``\n        argument is set to True); returns None on failure.\n    :rtype: str\n    "
-    if (u'as_object' in kwargs):
-        raise TldImproperlyConfigured(
-            u'`as_object` argument is deprecated for `get_fld`. Use `get_tld` instead.')
-    (domain_parts, non_zero_i, parsed_url) = process_url(url=url, fail_silently=fail_silently,
-                                                         fix_protocol=fix_protocol, search_public=search_public, search_private=search_private, parser_class=parser_class)
-    if (domain_parts is None):
-        return None
-    if (non_zero_i < 0):
-        return parsed_url.hostname
-    return u'.'.join(domain_parts[(non_zero_i - 1):])
-
-
-def get_tld(url, fail_silently=False, as_object=False, fix_protocol=False, search_public=True, search_private=True, parser_class=MozillaTLDSourceParser):
-    u"Extract the top level domain.\n\n    Extract the top level domain based on the mozilla's effective TLD names\n    dat file. Returns a string. May throw ``TldBadUrl`` or\n    ``TldDomainNotFound`` exceptions if there's bad URL provided or no TLD\n    match found respectively.\n\n    :param url: URL to get top level domain from.\n    :param fail_silently: If set to True, no exceptions are raised and None\n        is returned on failure.\n    :param as_object: If set to True, ``tld.utils.Result`` object is returned,\n        ``domain``, ``suffix`` and ``tld`` properties.\n    :param fix_protocol: If set to True, missing or wrong protocol is\n        ignored (https is appended instead).\n    :param search_public: If set to True, search in public domains.\n    :param search_private: If set to True, search in private domains.\n    :param parser_class:\n    :type url: str\n    :type fail_silently: bool\n    :type as_object: bool\n    :type fix_protocol: bool\n    :type search_public: bool\n    :type search_private: bool\n    :return: String with top level domain (if ``as_object`` argument\n        is set to False) or a ``tld.utils.Result`` object (if ``as_object``\n        argument is set to True); returns None on failure.\n    :rtype: str\n    "
-    (domain_parts, non_zero_i, parsed_url) = process_url(url=url, fail_silently=fail_silently,
-                                                         fix_protocol=fix_protocol, search_public=search_public, search_private=search_private, parser_class=parser_class)
-    if (domain_parts is None):
-        return None
-    if (not as_object):
-        if (non_zero_i < 0):
-            return parsed_url.hostname
-        return u'.'.join(domain_parts[non_zero_i:])
-    if (non_zero_i < 0):
-        subdomain = u''
-        domain = u''
-        _tld = parsed_url.hostname
-    else:
-        subdomain = u'.'.join(domain_parts[:(non_zero_i - 1)])
-        domain = u'.'.join(domain_parts[(non_zero_i - 1):non_zero_i])
-        _tld = u'.'.join(domain_parts[non_zero_i:])
-    return Result(subdomain=subdomain, domain=domain, tld=_tld, parsed_url=parsed_url)
-
-
-def parse_tld(url, fail_silently=False, fix_protocol=False, search_public=True, search_private=True, parser_class=MozillaTLDSourceParser):
-    u'Parse TLD into parts.\n\n    :param url:\n    :param fail_silently:\n    :param fix_protocol:\n    :param search_public:\n    :param search_private:\n    :param parser_class:\n    :return:\n    :rtype: tuple\n    '
-    try:
-        obj = get_tld(url, fail_silently=fail_silently, as_object=True, fix_protocol=fix_protocol,
-                      search_public=search_public, search_private=search_private, parser_class=parser_class)
-        _tld = obj.tld
-        domain = obj.domain
-        subdomain = obj.subdomain
-    except (TldBadUrl, TldDomainNotFound, TldImproperlyConfigured, TldIOError):
-        _tld = None
-        domain = None
-        subdomain = None
-    return (_tld, domain, subdomain)
-
-
-def is_tld(value, search_public=True, search_private=True, parser_class=MozillaTLDSourceParser):
-    u'Check if given URL is tld.\n\n    :param value: URL to get top level domain from.\n    :param search_public: If set to True, search in public domains.\n    :param search_private: If set to True, search in private domains.\n    :param parser_class:\n    :type value: str\n    :type search_public: bool\n    :type search_private: bool\n    :return:\n    :rtype: bool\n    '
-    _tld = get_tld(url=value, fail_silently=True, fix_protocol=True,
-                   search_public=search_public, search_private=search_private, parser_class=parser_class)
-    return (value == _tld)
-
-
-def reset_tld_names(tld_names_local_path=None):
-    u'Reset the ``tld_names`` to empty value.\n\n    If ``tld_names_local_path`` is given, removes specified\n    entry from ``tld_names`` instead.\n\n    :param tld_names_local_path:\n    :type tld_names_local_path: str\n    :return:\n    '
-    if tld_names_local_path:
-        pop_tld_names_container(tld_names_local_path)
-    else:
-        global tld_names
-        tld_names = {
-
-        }

Contents/advanced_settings.json.template

@@ -22,20 +22,10 @@ Don't expect support if you mess this up.
     // when the find better subtitles task finds a subtitle for an item that has an active extracted embedded subtitle
     // set, what should be the minimum score that subtitle has to have in order to be considered better?
     "find_better_as_extracted_tv_score": 352,
-    "find_better_as_extracted_movie_score": 112,
-
-    // SZ can use mediainfo if present to detect titles/forced state of MP4 MOV_TEXT, because the PMS currently doesn't
-    // set the title attribute
-    "dont_use_mediainfo_mp4": False,
-
-    // specific mediainfo binary path
-    "mediainfo_bin": null,
+    "find_better_as_extracted_movie_score": 82,
 
     "debug_i18n": false,
 
-    // refresh item after plugin refresh has been called (default: 5)
-    "refresh_after_called": 5,
-
     // per-provider-config
     "providers": {
         // enabled_for: specifies which media type to enable the provider for. does not override global/throttled
@@ -88,12 +78,9 @@ Don't expect support if you mess this up.
         "sonarr": {
             // don't verify HTTPS certificates? Set to True for self-signed certificates
             "ssl_no_verify": false,
-            // custom path to certificate pem file
-            "pem_file": None,
         },
         "radarr": {
             "ssl_no_verify": false,
-            "pem_file": None,
         },
     }
 }

README.md

@@ -1,7 +1,7 @@
 # Sub-Zero for Plex
 [![](https://img.shields.io/github/release/pannal/Sub-Zero.bundle.svg?style=flat&label=stable)](https://github.com/pannal/Sub-Zero.bundle/releases/latest) 
 [![master](https://img.shields.io/badge/master-stable-green.svg?maxAge=2592000)]()
-[![Maintenance](https://img.shields.io/maintenance/yes/2023.svg)]()
+[![Maintenance](https://img.shields.io/maintenance/yes/2019.svg)]()
 [![Slack Status](https://szslack.fragstore.net/badge.svg)](https://szslack.fragstore.net)
 [![FOSSA Status](https://app.fossa.io/api/projects/git%2Bgithub.com%2Fpannal%2FSub-Zero.bundle.svg?type=shield)](https://app.fossa.io/projects/git%2Bgithub.com%2Fpannal%2FSub-Zero.bundle?ref=badge_shield)
 
@@ -12,14 +12,6 @@ Check out **[the Sub-Zero Wiki](https://github.com/pannal/Sub-Zero.bundle/wiki)*
 
 <br />
 
-**DEPRECATED, USE BAZARR**
-## Legacy maintenance mode
-This addon will not be developed any further. It still works and arguably is still the best for managing subtitles when using Plex. As long as Plex Inc. supports agents, Sub-Zero will be maintained to work with the latest PMS version.
-
----
-
-**[Kitana is now required to have a UI](https://github.com/pannal/Kitana)**
-
 ---
 
 **[The future of Sub-Zero](https://www.reddit.com/r/PleX/comments/9n9qjl/subzero_the_future/)**
@@ -28,7 +20,7 @@ This addon will not be developed any further. It still works and arguably is sti
 
 ## Helping development
 
-If you like this, buy me a beer: <br>[![Donate](https://www.paypalobjects.com/en_US/i/btn/btn_donate_LG.gif)](https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=G9VKR2B8PMNKG) <br>or become a Patreon starting at **1 $ / month** <br><a href="https://www.patreon.com/subzero_plex" target="_blank"><img src="https://images.squarespace-cdn.com/content/v1/56c7831bf8baf3ae17ce9259/1561826418792-IJOMFASTOR6CW80N5W0Z/ke17ZwdGBToddI8pDm48kEycOuEejcFJqqLot0yQ4VVZw-zPPgdn4jUwVcJE1ZvWEtT5uBSRWt4vQZAgTJucoTqqXjS3CfNDSuuf31e0tVFrfGYmOrPFZFUXr1UxW4wA0PgPOjs31URy2JeWL9DdYhur-lC0WofN0YB1wFg-ZW0/footer-patreon.png?format=500w" height="54" /></a>
+If you like this, buy me a beer: <br>[![Donate](https://www.paypalobjects.com/en_US/i/btn/btn_donate_LG.gif)](https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=G9VKR2B8PMNKG) <br>or become a Patreon starting at **1 $ / month** <br><a href="https://www.patreon.com/subzero_plex" target="_blank"><img src="http://www.wenspencer.com/wp-content/uploads/2017/02/patreon-button.png" height="42" /></a> <br>or use the OpenSubtitles Sub-Zero affiliate link to become VIP <br>**10€/year, ad-free subs, 1000 subs/day, no-cache *VIP* server**<br><a href="http://v.ht/osvip" target="_blank"><img src="https://static.opensubtitles.org/gfx/logo.gif" height="50" /></a> 
 
 If you register with an anti-captcha service and you decide to use [Anti-Captcha.com](http://getcaptchasolution.com/kkvviom7nh), you can use [this affiliate link](http://getcaptchasolution.com/kkvviom7nh) to help development.
 
@@ -90,48 +82,37 @@ In addition to that Sub-Zero also fixes problems introduced by the subtitle crea
 Ever had broken music icons in a subtitle? Nordic characters like `Å` which turned into `Ã¥`? Not anymore.
 
 ## Installation
-[Please go to the wiki](https://github.com/pannal/Sub-Zero.bundle/wiki/Installation).
+Simply go to the Plex Plugins in your Plex Media Server, search for Sub-Zero and install it.
+For further help or manual installation, [please go to the wiki](https://github.com/pannal/Sub-Zero.bundle/wiki/Installation).
 
 ## Big thanks to the beta/i18n testing team (in no particular order)!
 the.vbm, mmgoodnow, Vertig0ne, thliu78, tattoomees, ostman, count_confucius, eherberg, tywilliams_88, Swanny, Jippo, Joost1991 / joost, Marik, Jon, AmbyDK, Clay, Abenlog, michael, smikwily, shoghicp, Zuikkis, Isilorn, Jacob K, Ninjouz, chopeta, fvb, Uthman, Claus Møller, Semi Doludizgin, Rafael, sugarman402, Morpheus1333, Yamil.llanos, Notorius28 
 
 ## Changelog
 
-2.6.5.3280
 
-temporarily enable OpenSubtitles.com instead of OpenSubtitles.org.
-You need to have an account there and an API consumer configured. Enter your API key in settings.
+2.6.5.3074
 
-This is barely tested but should work for basic usage.
-
-THIS PLUGIN IS DEPRECATED, PLEASE USE BAZARR!
-
-Changelog
-- cheaply backport opensubtitlescom from bazarr
-
-
-2.6.5.3277
-- core: fix enabled library/agents detection (Plex removed certain features)
-  fix Plex agent integration; Plex Inc removed certain attributes; SZ is now limited to thetvdb, thetvdbdvdorder, hama, themoviedb, imdb)
-
-
-
-2.6.5.3268
-subscene, addic7ed
+subscene, addic7ed and titlovi
 - either of those providers might impose a reCAPTCHA verification. In order to use those providers, please create an account at an AntiCaptcha service ([anti-captcha.com](http://getcaptchasolution.com/kkvviom7nh) or [deathbycaptcha.com](http://deathbycaptcha.com)), add funds, then supply your credentials/apikey in the configuration
 
 Changelog
-- clarify README
-- core: fix custom folder handling; #761
-- core: providers: screwzira: move to ktuvit and wizdom
-- core: add option to not download subtitles for certain audio languages existing; and/or no audio stream; fix #756
-- core: delay item refreshes after refresh call (default: 5 seconds; exposed in advanced settings)
-- menu: allow extraction of embedded subtitles for whole tv shows
+- core: cf: bypass cf 95% of the time without captchas
+- core: fix breaking line endings of certain languages (chinese, UTF-16); fixes #646
+- core: update pysubs2 to 0.2.3
+
+
+2.6.5.3062
+
+Changelog
+- core: cf: optimize
+- core: http: don't query DNS with IPs. thanks @fgump (fixes sonarr/radarr)
+
 
 [older changes](CHANGELOG.md)
 
 
-Subtitles provided by [OpenSubtitles.org](http://www.opensubtitles.org/), [Podnapisi.NET](https://www.podnapisi.net/), [TVSubtitles.net](http://www.tvsubtitles.net/), [Addic7ed.com](http://www.addic7ed.com/), [Legendas TV](http://legendas.tv/), [Napi Projekt](http://www.napiprojekt.pl/), [Shooter](http://shooter.cn/), [Titlovi](http://titlovi.com), [aRGENTeaM](http://argenteam.net), [SubScene](https://subscene.com/), [Hosszupuska](http://hosszupuskasub.com/), [Napisy24](https://napisy24.pl/)
+Subtitles provided by [OpenSubtitles.org](http://www.opensubtitles.org/), [Podnapisi.NET](https://www.podnapisi.net/), [TVSubtitles.net](http://www.tvsubtitles.net/), [Addic7ed.com](http://www.addic7ed.com/), [Legendas TV](http://legendas.tv/), [Napi Projekt](http://www.napiprojekt.pl/), [Shooter](http://shooter.cn/), [Titlovi](http://titlovi.com), [aRGENTeaM](http://argenteam.net), [SubScene](https://subscene.com/), [Hosszupuska](http://hosszupuskasub.com/)
 
 [3rd party licenses](https://github.com/pannal/Sub-Zero.bundle/tree/master/Licenses)