Update README.md

Change supersubtitles server url

CHANGELOG.md

@@ -1,5 +1,89 @@
+2.6.5.3280
+
+temporarily enable OpenSubtitles.com instead of OpenSubtitles.org.
+You need to have an account there and an API consumer configured. Enter your API key in settings.
+
+This is barely tested but should work for basic usage.
+
+THIS PLUGIN IS DEPRECATED, PLEASE USE BAZARR!
+
+Changelog
+- cheaply backport opensubtitlescom from bazarr
+
+
+2.6.5.3268
+subscene, addic7ed
+- either of those providers might impose a reCAPTCHA verification. In order to use those providers, please create an account at an AntiCaptcha service ([anti-captcha.com](http://getcaptchasolution.com/kkvviom7nh) or [deathbycaptcha.com](http://deathbycaptcha.com)), add funds, then supply your credentials/apikey in the configuration
+
+Changelog
+- clarify README
+- core: fix custom folder handling; #761
+- core: providers: screwzira: move to ktuvit and wizdom
+- core: add option to not download subtitles for certain audio languages existing; and/or no audio stream; fix #756
+- core: delay item refreshes after refresh call (default: 5 seconds; exposed in advanced settings)
+- menu: allow extraction of embedded subtitles for whole tv shows
+
+
+2.6.5.3247
+
+subscene, addic7ed
+
+either of those providers might impose a reCAPTCHA verification. In order to use those providers, please create an account at an AntiCaptcha service (anti-captcha.com or deathbycaptcha.com), add funds, then supply your credentials/apikey in the configuration
+Changelog
+core: fix for tv.plex.agents.movie not populating its media types
+core: tasks: findBetterSubtitles: increase minimum score for better subtitles for movies with extracted embedded subs from 82 to 112
+
+
+2.6.5.3241
+
+subscene, addic7ed
+
+either of those providers might impose a reCAPTCHA verification. In order to use those providers, please create an account at an AntiCaptcha service (anti-captcha.com or deathbycaptcha.com), add funds, then supply your credentials/apikey in the configuration
+Changelog
+
+core: add support for new Plex Movie agent
+core: remove py3 compat breaking unnecessary change
+core: skip drawing tags for SRT
+core: advanced_settings: refiners: drone: add custom pem_file support; fixes #735
+providers: core: set DownloadLimitPerDayExceeded timeout to 4 hours (was one day);
+providers: addic7ed: limit downloads per day; add vip setting
+providers: addic7ed: properly compare last_dl, add last_reset tracking info to log #723
+providers: addic7ed: properly implement limits
+submod: HI: remove more music tags
+submod: common CM_punctuation_space2: detect AND don't try changing domain/url/host when fixing punctuation
+
+
+2.6.5.3223
+
+subscene, addic7ed
+
+either of those providers might impose a reCAPTCHA verification. In order to use those providers, please create an account at an AntiCaptcha service (anti-captcha.com or deathbycaptcha.com), add funds, then supply your credentials/apikey in the configuration
+Changelog
+
+core: scoring: reorder subtitles based on second non-hash-score if main hash score is the same; morpheus65535/bazarr#821
+providers: bsplayer: verify hash; clean up
+
+
+2.6.5.3217
+
+subscene, addic7ed
+- either of those providers might impose a reCAPTCHA verification. In order to use those providers, please create an account at an AntiCaptcha service ([anti-captcha.com](http://getcaptchasolution.com/kkvviom7nh) or [deathbycaptcha.com](http://deathbycaptcha.com)), add funds, then supply your credentials/apikey in the configuration
+
+Changelog
+- core: also extract (missing) embedded subtitles when SearchAllRecentlyAddedMissing is running
+- core: core: clarify detecting streams (in logs)
+- core: UnRAR: set binary to executable, even if not checked out from git; might fix #693
+- core: bazarr-backport: morpheus65535/bazarr#703: use proper language code detection instead of a wild guess; should fix bad existing subtitle detection
+- core: bazarr-backport: morpheus65535/bazarr#660: fix BOM encoding stuff
+- core: bazarr-backport: morpheus65535/bazarr#656 further generalize formats; skip release group match if format match failed
+- core: fix stream detection when using mediainfo (#711)
+- config/core: make periodic SZ-internal subtitle maintenance interval configurable
+- providers: add BSPlayer Subtitles
+- providers: add ScrewZira (Hebrew)
+
 2.6.5.3183
 
+
 subscene, addic7ed
 - either of those providers might impose a reCAPTCHA verification. In order to use those providers, please create an account at an AntiCaptcha service ([anti-captcha.com](http://getcaptchasolution.com/kkvviom7nh) or [deathbycaptcha.com](http://deathbycaptcha.com)), add funds, then supply your credentials/apikey in the configuration
 

Contents/Code/__init__.py

@@ -262,7 +262,8 @@ class SubZeroAgent(object):
 
 
 class SubZeroSubtitlesAgentMovies(SubZeroAgent, Agent.Movies):
-    contributes_to = ['com.plexapp.agents.imdb', 'com.plexapp.agents.xbmcnfo', 'com.plexapp.agents.themoviedb', 'com.plexapp.agents.hama']
+    contributes_to = ['com.plexapp.agents.imdb', 'com.plexapp.agents.xbmcnfo', 'com.plexapp.agents.themoviedb',
+                      'com.plexapp.agents.hama',  'tv.plex.agents.movie']
     score_prefs_key = "subtitles.search.minimumMovieScore2"
     agent_type_verbose = "Movies"
 

Contents/Code/interface/menu.py

@@ -108,28 +108,51 @@ def MetadataMenu(rating_key, title=None, base_title=None, display_items=False, p
             add_incl_excl_options(oc, current_kind, title=sub_title, rating_key=rating_key, callback_menu=InclExclMenu)
 
         # mass-extract embedded
-        if current_kind == "season" and config.plex_transcoder:
-            for lang in config.lang_list:
-                oc.add(DirectoryObject(
-                    key=Callback(SeasonExtractEmbedded, rating_key=rating_key, language=lang,
-                                 base_title=show.section.title, display_items=display_items, item_title=item_title,
-                                 title=title,
-                                 previous_item_type=previous_item_type, with_mods=True,
-                                 previous_rating_key=previous_rating_key, randomize=timestamp()),
-                    title=_(u"Extract missing %(language)s embedded subtitles", language=display_language(lang)),
-                    summary=_("Extracts the not yet extracted embedded subtitles of all episodes for the current "
-                              "season with all configured default modifications")
-                ))
-                oc.add(DirectoryObject(
-                    key=Callback(SeasonExtractEmbedded, rating_key=rating_key, language=lang,
-                                 base_title=show.section.title, display_items=display_items, item_title=item_title,
-                                 title=title, force=True,
-                                 previous_item_type=previous_item_type, with_mods=True,
-                                 previous_rating_key=previous_rating_key, randomize=timestamp()),
-                    title=_(u"Extract and activate %(language)s embedded subtitles", language=display_language(lang)),
-                    summary=_("Extracts embedded subtitles of all episodes for the current season "
-                              "with all configured default modifications")
-                ))
+        if config.plex_transcoder:
+            if current_kind == "season":
+                for lang in config.lang_list:
+                    oc.add(DirectoryObject(
+                        key=Callback(SeasonExtractEmbedded, rating_key=rating_key, language=lang,
+                                     base_title=show.section.title, display_items=display_items, item_title=item_title,
+                                     title=title,
+                                     previous_item_type=previous_item_type, with_mods=True,
+                                     previous_rating_key=previous_rating_key, randomize=timestamp()),
+                        title=_(u"Extract missing %(language)s embedded subtitles", language=display_language(lang)),
+                        summary=_("Extracts the not yet extracted embedded subtitles of all episodes for the current "
+                                  "season with all configured default modifications")
+                    ))
+                    oc.add(DirectoryObject(
+                        key=Callback(SeasonExtractEmbedded, rating_key=rating_key, language=lang,
+                                     base_title=show.section.title, display_items=display_items, item_title=item_title,
+                                     title=title, force=True,
+                                     previous_item_type=previous_item_type, with_mods=True,
+                                     previous_rating_key=previous_rating_key, randomize=timestamp()),
+                        title=_(u"Extract and activate %(language)s embedded subtitles", language=display_language(lang)),
+                        summary=_("Extracts embedded subtitles of all episodes for the current season "
+                                  "with all configured default modifications")
+                    ))
+            elif current_kind == "series":
+                for lang in config.lang_list:
+                    oc.add(DirectoryObject(
+                        key=Callback(SeasonExtractEmbedded, rating_key=rating_key, language=lang,
+                                     base_title=title, display_items=display_items, item_title=item_title,
+                                     title=title, mode="series",
+                                     previous_item_type=previous_item_type, with_mods=True,
+                                     previous_rating_key=previous_rating_key, randomize=timestamp()),
+                        title=_(u"Extract missing %(language)s embedded subtitles", language=display_language(lang)),
+                        summary=_("Extracts the not yet extracted embedded subtitles of all episodes for the current "
+                                  "series with all configured default modifications")
+                    ))
+                    oc.add(DirectoryObject(
+                        key=Callback(SeasonExtractEmbedded, rating_key=rating_key, language=lang,
+                                     base_title=title, display_items=display_items, item_title=item_title,
+                                     title=title, force=True, mode="series",
+                                     previous_item_type=previous_item_type, with_mods=True,
+                                     previous_rating_key=previous_rating_key, randomize=timestamp()),
+                        title=_(u"Extract and activate %(language)s embedded subtitles", language=display_language(lang)),
+                        summary=_("Extracts embedded subtitles of all episodes for the current series "
+                                  "with all configured default modifications")
+                    ))
 
         # add refresh
         oc.add(DirectoryObject(
@@ -160,9 +183,10 @@ def SeasonExtractEmbedded(**kwargs):
     item_title = kwargs.pop("item_title")
     title = kwargs.pop("title")
     force = kwargs.pop("force", False)
+    mode = kwargs.pop("mode", "season")
 
     Thread.Create(season_extract_embedded, **{"rating_key": rating_key, "requested_language": requested_language,
-                                              "with_mods": with_mods, "force": force})
+                                              "with_mods": with_mods, "force": force, "mode": mode})
 
     kwargs["header"] = _("Success")
     kwargs["message"] = _(u"Extracting of embedded subtitles for %s triggered", title)

Contents/Code/support/config.py

@@ -10,12 +10,15 @@ import jstyleson
 import datetime
 import stat
 import traceback
+import socket
+import requests
 
 import subliminal
 import subliminal_patch
 import subzero.constants
 import lib
-from subliminal.exceptions import ServiceUnavailable, DownloadLimitExceeded, AuthenticationError
+from subliminal.exceptions import ServiceUnavailable, DownloadLimitExceeded, AuthenticationError, \
+    DownloadLimitPerDayExceeded
 from subliminal_patch.core import is_windows_special_path
 from whichdb import whichdb
 
@@ -61,21 +64,32 @@ def int_or_default(s, default):
         return default
 
 
-VALID_THROTTLE_EXCEPTIONS = (TooManyRequests, DownloadLimitExceeded, ServiceUnavailable, APIThrottled)
+VALID_THROTTLE_EXCEPTIONS = (TooManyRequests, DownloadLimitExceeded, DownloadLimitPerDayExceeded,
+                             ServiceUnavailable, APIThrottled, requests.Timeout, requests.ReadTimeout, socket.timeout)
+
+def_timeout = (datetime.timedelta(minutes=20), "20 minutes")
 
 PROVIDER_THROTTLE_MAP = {
     "default": {
         TooManyRequests: (datetime.timedelta(hours=1), "1 hour"),
         DownloadLimitExceeded: (datetime.timedelta(hours=3), "3 hours"),
+        DownloadLimitPerDayExceeded: (datetime.timedelta(hours=4), "4 hours"),
         ServiceUnavailable: (datetime.timedelta(minutes=20), "20 minutes"),
         APIThrottled: (datetime.timedelta(minutes=10), "10 minutes"),
         AuthenticationError: (datetime.timedelta(hours=2), "2 hours"),
+        requests.Timeout: def_timeout,
+        socket.timeout: def_timeout,
+        requests.ReadTimeout: def_timeout,
     },
     "opensubtitles": {
         TooManyRequests: (datetime.timedelta(hours=3), "3 hours"),
         DownloadLimitExceeded: (datetime.timedelta(hours=6), "6 hours"),
         APIThrottled: (datetime.timedelta(seconds=15), "15 seconds"),
     },
+    "opensubtitlescom": {
+        TooManyRequests: (datetime.timedelta(minutes=1), "1 minute"),
+        DownloadLimitExceeded: (datetime.timedelta(hours=24), "24 hours"),
+    },
     "addic7ed": {
         DownloadLimitExceeded: (datetime.timedelta(hours=3), "3 hours"),
         TooManyRequests: (datetime.timedelta(minutes=5), "5 minutes"),
@@ -150,6 +164,8 @@ class Config(object):
     exact_filenames = False
     only_one = False
     any_language_is_enough = False
+    ignore_for_audio = False
+    ignore_subs_for_empty_audio = False
     embedded_auto_extract = False
     ietf_as_alpha3 = False
     unrar = None
@@ -242,6 +258,7 @@ class Config(object):
         self.plex_transcoder = self.get_plex_transcoder()
         self.only_one = cast_bool(Prefs['subtitles.only_one'])
         self.any_language_is_enough = Prefs['subtitles.any_language_is_enough']
+        self.ignore_for_audio = self.ignore_subs_for_audio()
         self.embedded_auto_extract = cast_bool(Prefs["subtitles.embedded.autoextract"])
         self.ietf_as_alpha3 = cast_bool(Prefs["subtitles.language.ietf_normalize"])
         self.use_custom_dns = self.parse_custom_dns()
@@ -658,17 +675,34 @@ class Config(object):
         enabled_for_primary_agents = {"movie": [], "show": []}
         enabled_sections = {}
 
+        legacy_agents = {
+            "com.plexapp.agents.thetvdb": [SHOW],
+            "com.plexapp.agents.thetvdbdvdorder": [SHOW],
+            "com.plexapp.agents.hama": [SHOW, MOVIE],
+            "com.plexapp.agents.themoviedb": [SHOW, MOVIE],
+            "com.plexapp.agents.imdb": [SHOW, MOVIE],
+        }
+
         # find which agents we're enabled for
         for agent in Plex.agents():
-            if not agent.primary:
+            #if not agent.primary:
+            #    continue
+            if agent.identifier not in legacy_agents:
                 continue
 
-            for t in list(agent.media_types):
-                if t.media_type in (MOVIE, SHOW):
-                    related_agents = Plex.primary_agent(agent.identifier, t.media_type)
+            #media_types = [t.media_type for t in list(agent.media_types)]
+            media_types = legacy_agents[agent.identifier] + []
+
+            # the new movie agent doesn't populate its media types, workaround
+            if not media_types and agent.identifier == "tv.plex.agents.movie":
+                media_types = [MOVIE]
+
+            for media_type in media_types:
+                if media_type in (MOVIE, SHOW):
+                    related_agents = Plex.primary_agent(agent.identifier, media_type)
                     for a in related_agents:
                         if a.identifier == PLUGIN_IDENTIFIER and a.enabled:
-                            enabled_for_primary_agents[MEDIA_TYPE_TO_STRING[t.media_type]].append(agent.identifier)
+                            enabled_for_primary_agents[MEDIA_TYPE_TO_STRING[media_type]].append(agent.identifier)
 
         # find the libraries that use them
         for library in self.sections:
@@ -678,6 +712,22 @@ class Config(object):
         Log.Debug(u"I'm enabled for: %s" % [lib.title for key, lib in enabled_sections.iteritems()])
         return enabled_sections
 
+    def lang_str_to_list(self, s, l):
+        if len(s) and s != "None":
+            for lang in s.split(u","):
+                lang = lang.strip()
+                if lang == "NULL":
+                    l.append(lang)
+                    continue
+                try:
+                    real_lang = Language.fromietf(lang)
+                except:
+                    try:
+                        real_lang = Language.fromname(lang)
+                    except:
+                        continue
+                l.append(real_lang)
+
     # Prepare a list of languages we want subs for
     def get_lang_list(self, provider=None, ordered=False):
         # advanced settings
@@ -718,17 +768,9 @@ class Config(object):
             except:
                 pass
 
-        if len(lang_custom) and lang_custom != "None":
-            for lang in lang_custom.split(u","):
-                lang = lang.strip()
-                try:
-                    real_lang = Language.fromietf(lang)
-                except:
-                    try:
-                        real_lang = Language.fromname(lang)
-                    except:
-                        continue
-                l.append(real_lang)
+        self.lang_str_to_list(lang_custom, l)
+        if "NULL" in l:
+            l.remove("NULL")
 
         if self.forced_also:
             if Prefs["subtitles.when_forced"] == "Always":
@@ -759,12 +801,22 @@ class Config(object):
 
     lang_list = property(get_lang_list)
 
+    def ignore_subs_for_audio(self):
+        c = Prefs['subtitles.ignore_for_audio'].strip()
+        l = []
+
+        self.lang_str_to_list(c, l)
+        if "NULL" in l:
+            l.remove("NULL")
+            self.ignore_subs_for_empty_audio = True
+        return l
+
     def get_subtitle_destination_folder(self):
         if not Prefs["subtitles.save.filesystem"]:
             return
 
-        fld_custom = Prefs["subtitles.save.subFolder.Custom"].strip() if cast_bool(
-            Prefs["subtitles.save.subFolder.Custom"]) else None
+        fld_custom = Prefs["subtitles.save.subFolder.Custom"].strip() if \
+            Prefs["subtitles.save.subFolder.Custom"] else None
         return fld_custom or (
             Prefs["subtitles.save.subFolder"] if Prefs["subtitles.save.subFolder"] != "current folder" else None)
 
@@ -779,7 +831,7 @@ class Config(object):
 
     @property
     def providers_by_prefs(self):
-        return {'opensubtitles': cast_bool(Prefs['provider.opensubtitles.enabled']),
+        return {'opensubtitlescom': cast_bool(Prefs['provider.opensubtitles.enabled']),
                      # 'thesubdb': Prefs['provider.thesubdb.enabled'],
                      'podnapisi': cast_bool(Prefs['provider.podnapisi.enabled']),
                      'napisy24': cast_bool(Prefs['provider.napisy24.enabled']),
@@ -796,7 +848,8 @@ class Config(object):
                      'subscenter': False,
                      'assrt': cast_bool(Prefs['provider.assrt.enabled']),
                      'bsplayer': cast_bool(Prefs['provider.bsplayer.enabled']),
-                     'screwzira': cast_bool(Prefs['provider.screwzira.enabled']),
+                     'ktuvit': cast_bool(Prefs['provider.ktuvit.enabled']),
+                     'wizdom': cast_bool(Prefs['provider.wizdom.enabled']),
                 }
 
     @property
@@ -827,7 +880,8 @@ class Config(object):
             providers["subscene"] = False
             providers["napisy24"] = False
             providers["bsplayer"] = False
-            providers["screwzira"] = False
+            providers["ktuvit"] = False
+            providers["wizdom"] = False
             providers_forced_off = dict(providers)
 
         if not self.unrar and providers["legendastv"]:
@@ -873,16 +927,20 @@ class Config(object):
 
         provider_settings = {'addic7ed': {'username': Prefs['provider.addic7ed.username'],
                                           'password': Prefs['provider.addic7ed.password'],
+                                          'is_vip': cast_bool(Prefs['provider.addic7ed.is_vip']),
                                           },
-                             'opensubtitles': {'username': Prefs['provider.opensubtitles.username'],
+                             'opensubtitlescom': {'username': Prefs['provider.opensubtitles.username'],
                                                'password': Prefs['provider.opensubtitles.password'],
-                                               'use_tag_search': self.exact_filenames,
-                                               'only_foreign': self.forced_only,
-                                               'also_foreign': self.forced_also,
-                                               'is_vip': cast_bool(Prefs['provider.opensubtitles.is_vip']),
-                                               'use_ssl': os_use_https,
-                                               'timeout': self.advanced.providers.opensubtitles.timeout or 15,
-                                               'skip_wrong_fps': os_skip_wrong_fps,
+                                               #'use_tag_search': self.exact_filenames,
+                                               #'only_foreign': self.forced_only,
+                                               #'also_foreign': self.forced_also,
+                                               #'is_vip': cast_bool(Prefs['provider.opensubtitles.is_vip']),
+                                               #'use_ssl': os_use_https,
+                                               #'timeout': self.advanced.providers.opensubtitles.timeout or 15,
+                                               #'skip_wrong_fps': os_skip_wrong_fps,
+                                               'use_hash': cast_bool(Prefs['provider.opensubtitles.use_hash']),
+                                               'include_ai_translated': True,
+                                               'api_key': Prefs['provider.opensubtitles.api_key'],
                                                },
                              'podnapisi': {
                                  'only_foreign': self.forced_only,
@@ -904,7 +962,11 @@ class Config(object):
                              'legendastv': {'username': Prefs['provider.legendastv.username'],
                                             'password': Prefs['provider.legendastv.password'],
                                             },
-                             'assrt': {'token': Prefs['provider.assrt.token'], }
+                             'assrt': {'token': Prefs['provider.assrt.token'], },
+                             'ktuvit': {
+                                 'username': Prefs['provider.ktuvit.username'],
+                                 'password': Prefs['provider.ktuvit.password'],
+                             },
                              }
 
         return provider_settings

Contents/Code/support/download.py

@@ -16,6 +16,7 @@ from support.storage import get_pack_data, store_pack_data
 def get_missing_languages(video, part):
     languages_list = config.get_lang_list(ordered=True)
     languages = set(languages_list)
+    ignore_langs = set(config.ignore_for_audio)
     valid_langs_in_media = set()
 
     if Prefs["subtitles.when"] != "Always":
@@ -29,6 +30,15 @@ def get_missing_languages(video, part):
                   video)
         return set()
 
+    if config.ignore_subs_for_empty_audio and not video.audio_languages:
+        Log.Debug("Skipping subtitle search for %s, ignoring as no audio stream exists", video)
+        return set()
+
+    inter = ignore_langs.intersection(video.audio_languages)
+    if ignore_langs and inter:
+        Log.Debug("Skipping subtitle search for %s, ignoring due to existing audio streams: %s", video, inter)
+        return set()
+
     # should we treat IETF as alpha3? (ditch the country part)
     alpha3_map = {}
     if config.ietf_as_alpha3:

Contents/Code/support/extract.py

@@ -94,30 +94,36 @@ def multi_extract_embedded(stream_list, refresh=False, with_mods=False, single_t
         execute()
 
 
-def season_extract_embedded(rating_key, requested_language, with_mods=False, force=False):
+def season_extract_embedded(rating_key, requested_language, with_mods=False, force=False, mode="season"):
     # get stored subtitle info for item id
     subtitle_storage = get_subtitle_storage()
 
     try:
-        for data in get_all_items(key="children", value=rating_key, base="library/metadata"):
-            item = get_item(data[MI_KEY])
-            if item:
-                stored_subs = subtitle_storage.load_or_new(item)
-                for part in get_all_parts(item):
-                    embedded_subs = stored_subs.get_by_provider(part.id, requested_language, "embedded")
-                    current = stored_subs.get_any(part.id, requested_language)
-                    if not embedded_subs or force:
-                        stream_data = get_embedded_subtitle_streams(part, requested_language=requested_language)
-                        if stream_data:
-                            stream = stream_data[0]["stream"]
+        rating_keys = [rating_key]
+        if mode == "series":
+            seasons = get_all_items(key="children", value=rating_key, base="library/metadata")
+            rating_keys = [season[MI_KEY] for season in seasons]
 
-                            set_current = not current or force
-                            refresh = not current
+        for rating_key in rating_keys:
+            for data in get_all_items(key="children", value=rating_key, base="library/metadata"):
+                item = get_item(data[MI_KEY])
+                if item:
+                    stored_subs = subtitle_storage.load_or_new(item)
+                    for part in get_all_parts(item):
+                        embedded_subs = stored_subs.get_by_provider(part.id, requested_language, "embedded")
+                        current = stored_subs.get_any(part.id, requested_language)
+                        if not embedded_subs or force:
+                            stream_data = get_embedded_subtitle_streams(part, requested_language=requested_language)
+                            if stream_data:
+                                stream = stream_data[0]["stream"]
 
-                            extract_embedded_sub(rating_key=item.rating_key, part_id=part.id,
-                                                 stream_index=str(stream.index), set_current=set_current,
-                                                 refresh=refresh, language=requested_language, with_mods=with_mods,
-                                                 extract_mode="m")
+                                set_current = not current or force
+                                refresh = not current
+
+                                extract_embedded_sub(rating_key=item.rating_key, part_id=part.id,
+                                                     stream_index=str(stream.index), set_current=set_current,
+                                                     refresh=refresh, language=requested_language, with_mods=with_mods,
+                                                     extract_mode="m")
     finally:
         subtitle_storage.destroy()
 

Contents/Code/support/items.py

@@ -349,11 +349,13 @@ def refresh_item(rating_key, force=False, timeout=8000, refresh_kind=None, paren
         refresh = [item.rating_key for item in list(Plex["library/metadata"].children(int(rating_key)))]
 
     multiple = len(refresh) > 1
+    wait = config.advanced.get("refresh_after_called", 5)
+    Thread.Sleep(wait)
     for key in refresh:
         Log.Info("%s item %s", "Refreshing" if not force else "Forced-refreshing", key)
         Plex["library/metadata"].refresh(key)
         if multiple:
-            Thread.Sleep(10.0)
+            Thread.Sleep(wait)
 
 
 def get_current_sub(rating_key, part_id, language, plex_item=None):

Contents/Code/support/tasks.py

@@ -128,8 +128,8 @@ class SubtitleListingMixin(object):
         config.init_subliminal_patches()
 
         provider_settings = config.provider_settings
-        if not skip_wrong_fps:
-            provider_settings["opensubtitles"]["skip_wrong_fps"] = False
+        #if not skip_wrong_fps:
+        #    provider_settings["opensubtitlescom"]["skip_wrong_fps"] = False
 
         if item_type == "episode":
             min_score = 240
@@ -171,12 +171,15 @@ class SubtitleListingMixin(object):
                     else:
                         s.wrong_season_ep = True
 
+            orig_matches = matches.copy()
+            score, score_without_hash = compute_score(matches, s, video, hearing_impaired=use_hearing_impaired)
+
             unsorted_subtitles.append(
-                (s, compute_score(matches, s, video, hearing_impaired=use_hearing_impaired), matches))
-        scored_subtitles = sorted(unsorted_subtitles, key=operator.itemgetter(1), reverse=True)
+                (s, score, score_without_hash, matches, orig_matches))
+        scored_subtitles = sorted(unsorted_subtitles, key=operator.itemgetter(1, 2), reverse=True)
 
         subtitles = []
-        for subtitle, score, matches in scored_subtitles:
+        for subtitle, score, score_without_hash, matches, orig_matches in scored_subtitles:
             # check score
             if score < min_score and not subtitle.wrong_series:
                 Log.Info(u'%s: Score %d is below min_score (%d)', self.name, score, min_score)
@@ -667,7 +670,7 @@ class FindBetterSubtitles(DownloadSubtitleMixin, SubtitleListingMixin, Task):
         min_score_series = int(Prefs["subtitles.search.minimumTVScore2"].strip())
         min_score_movies = int(Prefs["subtitles.search.minimumMovieScore2"].strip())
         min_score_extracted_series = config.advanced.find_better_as_extracted_tv_score or 352
-        min_score_extracted_movies = config.advanced.find_better_as_extracted_movie_score or 82
+        min_score_extracted_movies = config.advanced.find_better_as_extracted_movie_score or 112
         overwrite_manually_modified = cast_bool(
             Prefs["scheduler.tasks.FindBetterSubtitles.overwrite_manually_modified"])
         overwrite_manually_selected = cast_bool(

Contents/DefaultPrefs.json

@@ -192,6 +192,12 @@
     ],
     "default": "Always"
   },
+  {
+    "id": "subtitles.ignore_for_audio",
+    "label": "Don't download subtitles for Audio languages (use ISO-639-1 codes; comma-separated; NULL=no audio)",
+    "type": "text",
+    "default": "None"
+  },
   {
     "id": "subtitles.when_forced",
     "label": "Download foreign/forced subtitles",
@@ -305,7 +311,7 @@
   },
   {
     "id": "provider.opensubtitles.enabled",
-    "label": "Provider: Enable OpenSubtitles",
+    "label": "Provider: Enable OpenSubtitles.com",
     "type": "bool",
     "default": "true"
   },
@@ -324,10 +330,16 @@
     "secure": "true"
   },
   {
-    "id": "provider.opensubtitles.is_vip",
-    "label": "OpenSubtitles VIP? (ad-free subs, 1000 subs/day, no-cache VIP server: http://v.ht/osvip)",
+    "id": "provider.opensubtitles.use_hash",
+    "label": "OpenSubtitles hash?",
     "type": "bool",
-    "default": "false"
+    "default": "true"
+  },
+  {
+    "id": "provider.opensubtitles.api_key",
+    "label": "OpenSubtitles APIKey",
+    "type": "text",
+    "default": ""
   },
   {
     "id": "provider.podnapisi.enabled",
@@ -375,6 +387,12 @@
     "default": "",
     "secure": "true"
   },
+  {
+    "id": "provider.addic7ed.is_vip",
+    "label": "Addic7ed VIP? (80 vs 40 downloads per day)",
+    "type": "bool",
+    "default": "false"
+  },
   {
     "id": "provider.addic7ed.boost_by2",
     "label": "Addic7ed: boost score (if requirements met)",
@@ -516,10 +534,30 @@
     "default": "true"
   },
   {
-    "id": "provider.screwzira.enabled",
-    "label": "Provider: Enable ScrewZira (Hebrew)",
+    "id": "provider.wizdom.enabled",
+    "label": "Provider: Enable WizdomSubs (Hebrew)",
     "type": "bool",
-    "default": "false"
+    "default": "true"
+  },
+ {
+    "id": "provider.ktuvit.enabled",
+    "label": "Provider: Enable Ktuvit (Hebrew)",
+    "type": "bool",
+    "default": "true"
+  },
+  {
+    "id": "provider.ktuvit.username",
+    "label": "Ktuvit Username",
+    "type": "text",
+    "default": ""
+  },
+  {
+    "id": "provider.ktuvit.password",
+    "label": "Ktuvit Password",
+    "type": "text",
+    "option": "hidden",
+    "secure": "true",
+    "default": ""
   },
   {
     "id": "providers.multithreading",

Contents/Info.plist

@@ -13,7 +13,7 @@
         <key>CFBundleSignature</key>
         <string>????</string>
         <key>CFBundleVersion</key>
-        <string>2.6.5.3217</string>
+        <string>2.6.5.3280</string>
         <key>PlexFrameworkVersion</key>
         <string>2</string>
         <key>PlexPluginClass</key>
@@ -32,7 +32,7 @@
 
 &lt;h1&gt;Sub-Zero for Plex&lt;/h1&gt;&lt;i&gt;Subtitles done right&lt;/i&gt;
 
-Version 2.6.5.3217
+Version 2.6.5.3280
 
 Originally based on @bramwalet's awesome &lt;a href=&quot;https://github.com/bramwalet/Subliminal.bundle&quot;&gt;Subliminal.bundle&lt;/a&gt;
 

Contents/Libraries/Shared/backports/__init__.py

@@ -0,0 +1 @@
+__path__ = __import__('pkgutil').extend_path(__path__, __name__)

Contents/Libraries/Shared/backports/functools_lru_cache.py

@@ -0,0 +1,196 @@
+from __future__ import absolute_import
+
+import functools
+from collections import namedtuple
+from threading import RLock
+
+_CacheInfo = namedtuple("CacheInfo", ["hits", "misses", "maxsize", "currsize"])
+
+
+@functools.wraps(functools.update_wrapper)
+def update_wrapper(
+    wrapper,
+    wrapped,
+    assigned=functools.WRAPPER_ASSIGNMENTS,
+    updated=functools.WRAPPER_UPDATES,
+):
+    """
+    Patch two bugs in functools.update_wrapper.
+    """
+    # workaround for http://bugs.python.org/issue3445
+    assigned = tuple(attr for attr in assigned if hasattr(wrapped, attr))
+    wrapper = functools.update_wrapper(wrapper, wrapped, assigned, updated)
+    # workaround for https://bugs.python.org/issue17482
+    wrapper.__wrapped__ = wrapped
+    return wrapper
+
+
+class _HashedSeq(list):
+    __slots__ = 'hashvalue'
+
+    def __init__(self, tup, hash=hash):
+        self[:] = tup
+        self.hashvalue = hash(tup)
+
+    def __hash__(self):
+        return self.hashvalue
+
+
+def _make_key(
+    args,
+    kwds,
+    typed,
+    kwd_mark=(object(),),
+    fasttypes=set([int, str, frozenset, type(None)]),
+    sorted=sorted,
+    tuple=tuple,
+    type=type,
+    len=len,
+):
+    'Make a cache key from optionally typed positional and keyword arguments'
+    key = args
+    if kwds:
+        sorted_items = sorted(kwds.items())
+        key += kwd_mark
+        for item in sorted_items:
+            key += item
+    if typed:
+        key += tuple(type(v) for v in args)
+        if kwds:
+            key += tuple(type(v) for k, v in sorted_items)
+    elif len(key) == 1 and type(key[0]) in fasttypes:
+        return key[0]
+    return _HashedSeq(key)
+
+
+def lru_cache(maxsize=100, typed=False):
+    """Least-recently-used cache decorator.
+
+    If *maxsize* is set to None, the LRU features are disabled and the cache
+    can grow without bound.
+
+    If *typed* is True, arguments of different types will be cached separately.
+    For example, f(3.0) and f(3) will be treated as distinct calls with
+    distinct results.
+
+    Arguments to the cached function must be hashable.
+
+    View the cache statistics named tuple (hits, misses, maxsize, currsize) with
+    f.cache_info().  Clear the cache and statistics with f.cache_clear().
+    Access the underlying function with f.__wrapped__.
+
+    See:  http://en.wikipedia.org/wiki/Cache_algorithms#Least_Recently_Used
+
+    """
+
+    # Users should only access the lru_cache through its public API:
+    #       cache_info, cache_clear, and f.__wrapped__
+    # The internals of the lru_cache are encapsulated for thread safety and
+    # to allow the implementation to change (including a possible C version).
+
+    def decorating_function(user_function):
+
+        cache = dict()
+        stats = [0, 0]  # make statistics updateable non-locally
+        HITS, MISSES = 0, 1  # names for the stats fields
+        make_key = _make_key
+        cache_get = cache.get  # bound method to lookup key or return None
+        _len = len  # localize the global len() function
+        lock = RLock()  # because linkedlist updates aren't threadsafe
+        root = []  # root of the circular doubly linked list
+        root[:] = [root, root, None, None]  # initialize by pointing to self
+        nonlocal_root = [root]  # make updateable non-locally
+        PREV, NEXT, KEY, RESULT = 0, 1, 2, 3  # names for the link fields
+
+        if maxsize == 0:
+
+            def wrapper(*args, **kwds):
+                # no caching, just do a statistics update after a successful call
+                result = user_function(*args, **kwds)
+                stats[MISSES] += 1
+                return result
+
+        elif maxsize is None:
+
+            def wrapper(*args, **kwds):
+                # simple caching without ordering or size limit
+                key = make_key(args, kwds, typed)
+                result = cache_get(
+                    key, root
+                )  # root used here as a unique not-found sentinel
+                if result is not root:
+                    stats[HITS] += 1
+                    return result
+                result = user_function(*args, **kwds)
+                cache[key] = result
+                stats[MISSES] += 1
+                return result
+
+        else:
+
+            def wrapper(*args, **kwds):
+                # size limited caching that tracks accesses by recency
+                key = make_key(args, kwds, typed) if kwds or typed else args
+                with lock:
+                    link = cache_get(key)
+                    if link is not None:
+                        # record recent use of the key by moving it
+                        # to the front of the list
+                        root, = nonlocal_root
+                        link_prev, link_next, key, result = link
+                        link_prev[NEXT] = link_next
+                        link_next[PREV] = link_prev
+                        last = root[PREV]
+                        last[NEXT] = root[PREV] = link
+                        link[PREV] = last
+                        link[NEXT] = root
+                        stats[HITS] += 1
+                        return result
+                result = user_function(*args, **kwds)
+                with lock:
+                    root, = nonlocal_root
+                    if key in cache:
+                        # getting here means that this same key was added to the
+                        # cache while the lock was released.  since the link
+                        # update is already done, we need only return the
+                        # computed result and update the count of misses.
+                        pass
+                    elif _len(cache) >= maxsize:
+                        # use the old root to store the new key and result
+                        oldroot = root
+                        oldroot[KEY] = key
+                        oldroot[RESULT] = result
+                        # empty the oldest link and make it the new root
+                        root = nonlocal_root[0] = oldroot[NEXT]
+                        oldkey = root[KEY]
+                        root[KEY] = root[RESULT] = None
+                        # now update the cache dictionary for the new links
+                        del cache[oldkey]
+                        cache[key] = oldroot
+                    else:
+                        # put result in a new link at the front of the list
+                        last = root[PREV]
+                        link = [last, root, key, result]
+                        last[NEXT] = root[PREV] = cache[key] = link
+                    stats[MISSES] += 1
+                return result
+
+        def cache_info():
+            """Report cache statistics"""
+            with lock:
+                return _CacheInfo(stats[HITS], stats[MISSES], maxsize, len(cache))
+
+        def cache_clear():
+            """Clear the cache and cache statistics"""
+            with lock:
+                cache.clear()
+                root = nonlocal_root[0]
+                root[:] = [root, root, None, None]
+                stats[:] = [0, 0]
+
+        wrapper.__wrapped__ = user_function
+        wrapper.cache_info = cache_info
+        wrapper.cache_clear = cache_clear
+        return update_wrapper(wrapper, user_function)
+
+    return decorating_function

Contents/Libraries/Shared/certifi/__init__.py

@@ -1,3 +1,3 @@
-from .core import where
+from .core import contents, where
 
-__version__ = "2019.03.09"
+__version__ = "2020.06.20"

Contents/Libraries/Shared/certifi/__main__.py

@@ -1,2 +1,12 @@
-from certifi import where
-print(where())
+import argparse
+
+from certifi import contents, where
+
+parser = argparse.ArgumentParser()
+parser.add_argument("-c", "--contents", action="store_true")
+args = parser.parse_args()
+
+if args.contents:
+    print(contents())
+else:
+    print(where())

Contents/Libraries/Shared/certifi/cacert.pem

@@ -58,38 +58,6 @@ AfvDbbnvRG15RjF+Cv6pgsH/76tuIMRQyV+dTZsXjAzlAcmgQWpzU/qlULRuJQ/7
 TBj0/VLZjmmx6BEP3ojY+x1J96relc8geMJgEtslQIxq/H5COEBkEveegeGTLg==
 -----END CERTIFICATE-----
 
-# Issuer: CN=VeriSign Class 3 Public Primary Certification Authority - G3 O=VeriSign, Inc. OU=VeriSign Trust Network/(c) 1999 VeriSign, Inc. - For authorized use only
-# Subject: CN=VeriSign Class 3 Public Primary Certification Authority - G3 O=VeriSign, Inc. OU=VeriSign Trust Network/(c) 1999 VeriSign, Inc. - For authorized use only
-# Label: "Verisign Class 3 Public Primary Certification Authority - G3"
-# Serial: 206684696279472310254277870180966723415
-# MD5 Fingerprint: cd:68:b6:a7:c7:c4:ce:75:e0:1d:4f:57:44:61:92:09
-# SHA1 Fingerprint: 13:2d:0d:45:53:4b:69:97:cd:b2:d5:c3:39:e2:55:76:60:9b:5c:c6
-# SHA256 Fingerprint: eb:04:cf:5e:b1:f3:9a:fa:76:2f:2b:b1:20:f2:96:cb:a5:20:c1:b9:7d:b1:58:95:65:b8:1c:b9:a1:7b:72:44
------BEGIN CERTIFICATE-----
-MIIEGjCCAwICEQCbfgZJoz5iudXukEhxKe9XMA0GCSqGSIb3DQEBBQUAMIHKMQsw
-CQYDVQQGEwJVUzEXMBUGA1UEChMOVmVyaVNpZ24sIEluYy4xHzAdBgNVBAsTFlZl
-cmlTaWduIFRydXN0IE5ldHdvcmsxOjA4BgNVBAsTMShjKSAxOTk5IFZlcmlTaWdu
-LCBJbmMuIC0gRm9yIGF1dGhvcml6ZWQgdXNlIG9ubHkxRTBDBgNVBAMTPFZlcmlT
-aWduIENsYXNzIDMgUHVibGljIFByaW1hcnkgQ2VydGlmaWNhdGlvbiBBdXRob3Jp
-dHkgLSBHMzAeFw05OTEwMDEwMDAwMDBaFw0zNjA3MTYyMzU5NTlaMIHKMQswCQYD
-VQQGEwJVUzEXMBUGA1UEChMOVmVyaVNpZ24sIEluYy4xHzAdBgNVBAsTFlZlcmlT
-aWduIFRydXN0IE5ldHdvcmsxOjA4BgNVBAsTMShjKSAxOTk5IFZlcmlTaWduLCBJ
-bmMuIC0gRm9yIGF1dGhvcml6ZWQgdXNlIG9ubHkxRTBDBgNVBAMTPFZlcmlTaWdu
-IENsYXNzIDMgUHVibGljIFByaW1hcnkgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkg
-LSBHMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMu6nFL8eB8aHm8b
-N3O9+MlrlBIwT/A2R/XQkQr1F8ilYcEWQE37imGQ5XYgwREGfassbqb1EUGO+i2t
-KmFZpGcmTNDovFJbcCAEWNF6yaRpvIMXZK0Fi7zQWM6NjPXr8EJJC52XJ2cybuGu
-kxUccLwgTS8Y3pKI6GyFVxEa6X7jJhFUokWWVYPKMIno3Nij7SqAP395ZVc+FSBm
-CC+Vk7+qRy+oRpfwEuL+wgorUeZ25rdGt+INpsyow0xZVYnm6FNcHOqd8GIWC6fJ
-Xwzw3sJ2zq/3avL6QaaiMxTJ5Xpj055iN9WFZZ4O5lMkdBteHRJTW8cs54NJOxWu
-imi5V5cCAwEAATANBgkqhkiG9w0BAQUFAAOCAQEAERSWwauSCPc/L8my/uRan2Te
-2yFPhpk0djZX3dAVL8WtfxUfN2JzPtTnX84XA9s1+ivbrmAJXx5fj267Cz3qWhMe
-DGBvtcC1IyIuBwvLqXTLR7sdwdela8wv0kL9Sd2nic9TutoAWii/gt/4uhMdUIaC
-/Y4wjylGsB49Ndo4YhYYSq3mtlFs3q9i6wHQHiT+eo8SGhJouPtmmRQURVyu565p
-F4ErWjfJXir0xuKhXFSbplQAz/DxwceYMBo7Nhbbo27q/a2ywtrvAkcTisDxszGt
-TxzhT5yvDwyd93gN2PQ1VoDat20Xj50egWTh/sVFuq1ruQp6Tk9LhO5L8X3dEQ==
------END CERTIFICATE-----
-
 # Issuer: CN=Entrust.net Certification Authority (2048) O=Entrust.net OU=www.entrust.net/CPS_2048 incorp. by ref. (limits liab.)/(c) 1999 Entrust.net Limited
 # Subject: CN=Entrust.net Certification Authority (2048) O=Entrust.net OU=www.entrust.net/CPS_2048 incorp. by ref. (limits liab.)/(c) 1999 Entrust.net Limited
 # Label: "Entrust.net Premium 2048 Secure Server CA"
@@ -152,39 +120,6 @@ ksLi4xaNmjICq44Y3ekQEe5+NauQrz4wlHrQMz2nZQ/1/I6eYs9HRCwBXbsdtTLS
 R9I4LtD+gdwyah617jzV/OeBHRnDJELqYzmp
 -----END CERTIFICATE-----
 
-# Issuer: CN=AddTrust External CA Root O=AddTrust AB OU=AddTrust External TTP Network
-# Subject: CN=AddTrust External CA Root O=AddTrust AB OU=AddTrust External TTP Network
-# Label: "AddTrust External Root"
-# Serial: 1
-# MD5 Fingerprint: 1d:35:54:04:85:78:b0:3f:42:42:4d:bf:20:73:0a:3f
-# SHA1 Fingerprint: 02:fa:f3:e2:91:43:54:68:60:78:57:69:4d:f5:e4:5b:68:85:18:68
-# SHA256 Fingerprint: 68:7f:a4:51:38:22:78:ff:f0:c8:b1:1f:8d:43:d5:76:67:1c:6e:b2:bc:ea:b4:13:fb:83:d9:65:d0:6d:2f:f2
------BEGIN CERTIFICATE-----
-MIIENjCCAx6gAwIBAgIBATANBgkqhkiG9w0BAQUFADBvMQswCQYDVQQGEwJTRTEU
-MBIGA1UEChMLQWRkVHJ1c3QgQUIxJjAkBgNVBAsTHUFkZFRydXN0IEV4dGVybmFs
-IFRUUCBOZXR3b3JrMSIwIAYDVQQDExlBZGRUcnVzdCBFeHRlcm5hbCBDQSBSb290
-MB4XDTAwMDUzMDEwNDgzOFoXDTIwMDUzMDEwNDgzOFowbzELMAkGA1UEBhMCU0Ux
-FDASBgNVBAoTC0FkZFRydXN0IEFCMSYwJAYDVQQLEx1BZGRUcnVzdCBFeHRlcm5h
-bCBUVFAgTmV0d29yazEiMCAGA1UEAxMZQWRkVHJ1c3QgRXh0ZXJuYWwgQ0EgUm9v
-dDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALf3GjPm8gAELTngTlvt
-H7xsD821+iO2zt6bETOXpClMfZOfvUq8k+0DGuOPz+VtUFrWlymUWoCwSXrbLpX9
-uMq/NzgtHj6RQa1wVsfwTz/oMp50ysiQVOnGXw94nZpAPA6sYapeFI+eh6FqUNzX
-mk6vBbOmcZSccbNQYArHE504B4YCqOmoaSYYkKtMsE8jqzpPhNjfzp/haW+710LX
-a0Tkx63ubUFfclpxCDezeWWkWaCUN/cALw3CknLa0Dhy2xSoRcRdKn23tNbE7qzN
-E0S3ySvdQwAl+mG5aWpYIxG3pzOPVnVZ9c0p10a3CitlttNCbxWyuHv77+ldU9U0
-WicCAwEAAaOB3DCB2TAdBgNVHQ4EFgQUrb2YejS0Jvf6xCZU7wO94CTLVBowCwYD
-VR0PBAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wgZkGA1UdIwSBkTCBjoAUrb2YejS0
-Jvf6xCZU7wO94CTLVBqhc6RxMG8xCzAJBgNVBAYTAlNFMRQwEgYDVQQKEwtBZGRU
-cnVzdCBBQjEmMCQGA1UECxMdQWRkVHJ1c3QgRXh0ZXJuYWwgVFRQIE5ldHdvcmsx
-IjAgBgNVBAMTGUFkZFRydXN0IEV4dGVybmFsIENBIFJvb3SCAQEwDQYJKoZIhvcN
-AQEFBQADggEBALCb4IUlwtYj4g+WBpKdQZic2YR5gdkeWxQHIzZlj7DYd7usQWxH
-YINRsPkyPef89iYTx4AWpb9a/IfPeHmJIZriTAcKhjW88t5RxNKWt9x+Tu5w/Rw5
-6wwCURQtjr0W4MHfRnXnJK3s9EK0hZNwEGe6nQY1ShjTK3rMUUKhemPR5ruhxSvC
-Nr4TDea9Y355e6cJDUCrat2PisP29owaQgVR1EX1n6diIWgVIEM8med8vSTYqZEX
-c4g/VhsxOBi0cQ+azcgOno4uG+GMmIPLHzHxREzGBHNJdmAPx/i9F4BrLunMTA5a
-mnkPIAou1Z5jJh5VkpTYghdae9C8x49OhgQ=
------END CERTIFICATE-----
-
 # Issuer: CN=Entrust Root Certification Authority O=Entrust, Inc. OU=www.entrust.net/CPS is incorporated by reference/(c) 2006 Entrust, Inc.
 # Subject: CN=Entrust Root Certification Authority O=Entrust, Inc. OU=www.entrust.net/CPS is incorporated by reference/(c) 2006 Entrust, Inc.
 # Label: "Entrust Root Certification Authority"
@@ -771,36 +706,6 @@ vEsXCS+0yx5DaMkHJ8HSXPfqIbloEpw8nL+e/IBcm2PN7EeqJSdnoDfzAIJ9VNep
 +OkuE6N36B9K
 -----END CERTIFICATE-----
 
-# Issuer: CN=Class 2 Primary CA O=Certplus
-# Subject: CN=Class 2 Primary CA O=Certplus
-# Label: "Certplus Class 2 Primary CA"
-# Serial: 177770208045934040241468760488327595043
-# MD5 Fingerprint: 88:2c:8c:52:b8:a2:3c:f3:f7:bb:03:ea:ae:ac:42:0b
-# SHA1 Fingerprint: 74:20:74:41:72:9c:dd:92:ec:79:31:d8:23:10:8d:c2:81:92:e2:bb
-# SHA256 Fingerprint: 0f:99:3c:8a:ef:97:ba:af:56:87:14:0e:d5:9a:d1:82:1b:b4:af:ac:f0:aa:9a:58:b5:d5:7a:33:8a:3a:fb:cb
------BEGIN CERTIFICATE-----
-MIIDkjCCAnqgAwIBAgIRAIW9S/PY2uNp9pTXX8OlRCMwDQYJKoZIhvcNAQEFBQAw
-PTELMAkGA1UEBhMCRlIxETAPBgNVBAoTCENlcnRwbHVzMRswGQYDVQQDExJDbGFz
-cyAyIFByaW1hcnkgQ0EwHhcNOTkwNzA3MTcwNTAwWhcNMTkwNzA2MjM1OTU5WjA9
-MQswCQYDVQQGEwJGUjERMA8GA1UEChMIQ2VydHBsdXMxGzAZBgNVBAMTEkNsYXNz
-IDIgUHJpbWFyeSBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBANxQ
-ltAS+DXSCHh6tlJw/W/uz7kRy1134ezpfgSN1sxvc0NXYKwzCkTsA18cgCSR5aiR
-VhKC9+Ar9NuuYS6JEI1rbLqzAr3VNsVINyPi8Fo3UjMXEuLRYE2+L0ER4/YXJQyL
-kcAbmXuZVg2v7tK8R1fjeUl7NIknJITesezpWE7+Tt9avkGtrAjFGA7v0lPubNCd
-EgETjdyAYveVqUSISnFOYFWe2yMZeVYHDD9jC1yw4r5+FfyUM1hBOHTE4Y+L3yas
-H7WLO7dDWWuwJKZtkIvEcupdM5i3y95ee++U8Rs+yskhwcWYAqqi9lt3m/V+llU0
-HGdpwPFC40es/CgcZlUCAwEAAaOBjDCBiTAPBgNVHRMECDAGAQH/AgEKMAsGA1Ud
-DwQEAwIBBjAdBgNVHQ4EFgQU43Mt38sOKAze3bOkynm4jrvoMIkwEQYJYIZIAYb4
-QgEBBAQDAgEGMDcGA1UdHwQwMC4wLKAqoCiGJmh0dHA6Ly93d3cuY2VydHBsdXMu
-Y29tL0NSTC9jbGFzczIuY3JsMA0GCSqGSIb3DQEBBQUAA4IBAQCnVM+IRBnL39R/
-AN9WM2K191EBkOvDP9GIROkkXe/nFL0gt5o8AP5tn9uQ3Nf0YtaLcF3n5QRIqWh8
-yfFC82x/xXp8HVGIutIKPidd3i1RTtMTZGnkLuPT55sJmabglZvOGtd/vjzOUrMR
-FcEPF80Du5wlFbqidon8BvEY0JNLDnyCt6X09l/+7UCmnYR0ObncHoUW2ikbhiMA
-ybuJfm6AiB4vFLQDJKgybwOaRywwvlbGp0ICcBvqQNi6BQNwB6SW//1IMwrh3KWB
-kJtN3X3n57LNXMhqlfil9o3EXXgIvnsG1knPGTZQIy4I5p4FTUcY1Rbpsda2ENW7
-l7+ijrRU
------END CERTIFICATE-----
-
 # Issuer: CN=DST Root CA X3 O=Digital Signature Trust Co.
 # Subject: CN=DST Root CA X3 O=Digital Signature Trust Co.
 # Label: "DST Root CA X3"
@@ -1219,36 +1124,6 @@ t0QmwCbAr1UwnjvVNioZBPRcHv/PLLf/0P2HQBHVESO7SMAhqaQoLf0V+LBOK/Qw
 WyH8EZE0vkHve52Xdf+XlcCWWC/qu0bXu+TZLg==
 -----END CERTIFICATE-----
 
-# Issuer: CN=Deutsche Telekom Root CA 2 O=Deutsche Telekom AG OU=T-TeleSec Trust Center
-# Subject: CN=Deutsche Telekom Root CA 2 O=Deutsche Telekom AG OU=T-TeleSec Trust Center
-# Label: "Deutsche Telekom Root CA 2"
-# Serial: 38
-# MD5 Fingerprint: 74:01:4a:91:b1:08:c4:58:ce:47:cd:f0:dd:11:53:08
-# SHA1 Fingerprint: 85:a4:08:c0:9c:19:3e:5d:51:58:7d:cd:d6:13:30:fd:8c:de:37:bf
-# SHA256 Fingerprint: b6:19:1a:50:d0:c3:97:7f:7d:a9:9b:cd:aa:c8:6a:22:7d:ae:b9:67:9e:c7:0b:a3:b0:c9:d9:22:71:c1:70:d3
------BEGIN CERTIFICATE-----
-MIIDnzCCAoegAwIBAgIBJjANBgkqhkiG9w0BAQUFADBxMQswCQYDVQQGEwJERTEc
-MBoGA1UEChMTRGV1dHNjaGUgVGVsZWtvbSBBRzEfMB0GA1UECxMWVC1UZWxlU2Vj
-IFRydXN0IENlbnRlcjEjMCEGA1UEAxMaRGV1dHNjaGUgVGVsZWtvbSBSb290IENB
-IDIwHhcNOTkwNzA5MTIxMTAwWhcNMTkwNzA5MjM1OTAwWjBxMQswCQYDVQQGEwJE
-RTEcMBoGA1UEChMTRGV1dHNjaGUgVGVsZWtvbSBBRzEfMB0GA1UECxMWVC1UZWxl
-U2VjIFRydXN0IENlbnRlcjEjMCEGA1UEAxMaRGV1dHNjaGUgVGVsZWtvbSBSb290
-IENBIDIwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCrC6M14IspFLEU
-ha88EOQ5bzVdSq7d6mGNlUn0b2SjGmBmpKlAIoTZ1KXleJMOaAGtuU1cOs7TuKhC
-QN/Po7qCWWqSG6wcmtoIKyUn+WkjR/Hg6yx6m/UTAtB+NHzCnjwAWav12gz1Mjwr
-rFDa1sPeg5TKqAyZMg4ISFZbavva4VhYAUlfckE8FQYBjl2tqriTtM2e66foai1S
-NNs671x1Udrb8zH57nGYMsRUFUQM+ZtV7a3fGAigo4aKSe5TBY8ZTNXeWHmb0moc
-QqvF1afPaA+W5OFhmHZhyJF81j4A4pFQh+GdCuatl9Idxjp9y7zaAzTVjlsB9WoH
-txa2bkp/AgMBAAGjQjBAMB0GA1UdDgQWBBQxw3kbuvVT1xfgiXotF2wKsyudMzAP
-BgNVHRMECDAGAQH/AgEFMA4GA1UdDwEB/wQEAwIBBjANBgkqhkiG9w0BAQUFAAOC
-AQEAlGRZrTlk5ynrE/5aw4sTV8gEJPB0d8Bg42f76Ymmg7+Wgnxu1MM9756Abrsp
-tJh6sTtU6zkXR34ajgv8HzFZMQSyzhfzLMdiNlXiItiJVbSYSKpk+tYcNthEeFpa
-IzpXl/V6ME+un2pMSyuOoAPjPuCp1NJ70rOo4nI8rZ7/gFnkm0W09juwzTkZmDLl
-6iFhkOQxIY40sfcvNUqFENrnijchvllj4PKFiDFT1FQUhXB59C4Gdyd1Lx+4ivn+
-xbrYNuSD7Odlt79jWvNGr4GUN9RBjNYj1h7P9WgbRGOiWrqnNVmh5XAFmw4jV5mU
-Cm26OWMohpLzGITY+9HPBVZkVw==
------END CERTIFICATE-----
-
 # Issuer: CN=Cybertrust Global Root O=Cybertrust, Inc
 # Subject: CN=Cybertrust Global Root O=Cybertrust, Inc
 # Label: "Cybertrust Global Root"
@@ -1559,47 +1434,6 @@ uLjbvrW5KfnaNwUASZQDhETnv0Mxz3WLJdH0pmT1kvarBes96aULNmLazAZfNou2
 XjG4Kvte9nHfRCaexOYNkbQudZWAUWpLMKawYqGT8ZvYzsRjdT9ZR7E=
 -----END CERTIFICATE-----
 
-# Issuer: CN=Staat der Nederlanden Root CA - G2 O=Staat der Nederlanden
-# Subject: CN=Staat der Nederlanden Root CA - G2 O=Staat der Nederlanden
-# Label: "Staat der Nederlanden Root CA - G2"
-# Serial: 10000012
-# MD5 Fingerprint: 7c:a5:0f:f8:5b:9a:7d:6d:30:ae:54:5a:e3:42:a2:8a
-# SHA1 Fingerprint: 59:af:82:79:91:86:c7:b4:75:07:cb:cf:03:57:46:eb:04:dd:b7:16
-# SHA256 Fingerprint: 66:8c:83:94:7d:a6:3b:72:4b:ec:e1:74:3c:31:a0:e6:ae:d0:db:8e:c5:b3:1b:e3:77:bb:78:4f:91:b6:71:6f
------BEGIN CERTIFICATE-----
-MIIFyjCCA7KgAwIBAgIEAJiWjDANBgkqhkiG9w0BAQsFADBaMQswCQYDVQQGEwJO
-TDEeMBwGA1UECgwVU3RhYXQgZGVyIE5lZGVybGFuZGVuMSswKQYDVQQDDCJTdGFh
-dCBkZXIgTmVkZXJsYW5kZW4gUm9vdCBDQSAtIEcyMB4XDTA4MDMyNjExMTgxN1oX
-DTIwMDMyNTExMDMxMFowWjELMAkGA1UEBhMCTkwxHjAcBgNVBAoMFVN0YWF0IGRl
-ciBOZWRlcmxhbmRlbjErMCkGA1UEAwwiU3RhYXQgZGVyIE5lZGVybGFuZGVuIFJv
-b3QgQ0EgLSBHMjCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAMVZ5291
-qj5LnLW4rJ4L5PnZyqtdj7U5EILXr1HgO+EASGrP2uEGQxGZqhQlEq0i6ABtQ8Sp
-uOUfiUtnvWFI7/3S4GCI5bkYYCjDdyutsDeqN95kWSpGV+RLufg3fNU254DBtvPU
-Z5uW6M7XxgpT0GtJlvOjCwV3SPcl5XCsMBQgJeN/dVrlSPhOewMHBPqCYYdu8DvE
-pMfQ9XQ+pV0aCPKbJdL2rAQmPlU6Yiile7Iwr/g3wtG61jj99O9JMDeZJiFIhQGp
-5Rbn3JBV3w/oOM2ZNyFPXfUib2rFEhZgF1XyZWampzCROME4HYYEhLoaJXhena/M
-UGDWE4dS7WMfbWV9whUYdMrhfmQpjHLYFhN9C0lK8SgbIHRrxT3dsKpICT0ugpTN
-GmXZK4iambwYfp/ufWZ8Pr2UuIHOzZgweMFvZ9C+X+Bo7d7iscksWXiSqt8rYGPy
-5V6548r6f1CGPqI0GAwJaCgRHOThuVw+R7oyPxjMW4T182t0xHJ04eOLoEq9jWYv
-6q012iDTiIJh8BIitrzQ1aTsr1SIJSQ8p22xcik/Plemf1WvbibG/ufMQFxRRIEK
-eN5KzlW/HdXZt1bv8Hb/C3m1r737qWmRRpdogBQ2HbN/uymYNqUg+oJgYjOk7Na6
-B6duxc8UpufWkjTYgfX8HV2qXB72o007uPc5AgMBAAGjgZcwgZQwDwYDVR0TAQH/
-BAUwAwEB/zBSBgNVHSAESzBJMEcGBFUdIAAwPzA9BggrBgEFBQcCARYxaHR0cDov
-L3d3dy5wa2lvdmVyaGVpZC5ubC9wb2xpY2llcy9yb290LXBvbGljeS1HMjAOBgNV
-HQ8BAf8EBAMCAQYwHQYDVR0OBBYEFJFoMocVHYnitfGsNig0jQt8YojrMA0GCSqG
-SIb3DQEBCwUAA4ICAQCoQUpnKpKBglBu4dfYszk78wIVCVBR7y29JHuIhjv5tLyS
-CZa59sCrI2AGeYwRTlHSeYAz+51IvuxBQ4EffkdAHOV6CMqqi3WtFMTC6GY8ggen
-5ieCWxjmD27ZUD6KQhgpxrRW/FYQoAUXvQwjf/ST7ZwaUb7dRUG/kSS0H4zpX897
-IZmflZ85OkYcbPnNe5yQzSipx6lVu6xiNGI1E0sUOlWDuYaNkqbG9AclVMwWVxJK
-gnjIFNkXgiYtXSAfea7+1HAWFpWD2DU5/1JddRwWxRNVz0fMdWVSSt7wsKfkCpYL
-+63C4iWEst3kvX5ZbJvw8NjnyvLplzh+ib7M+zkXYT9y2zqR2GUBGR2tUKRXCnxL
-vJxxcypFURmFzI79R6d0lR2o0a9OF7FpJsKqeFdbxU2n5Z4FF5TKsl+gSRiNNOkm
-bEgeqmiSBeGCc1qb3AdbCG19ndeNIdn8FCCqwkXfP+cAslHkwvgFuXkajDTznlvk
-N1trSt8sV4pAWja63XVECDdCcAz+3F4hoKOKwJCcaNpQ5kUQR3i2TtJlycM33+FC
-Y7BXN0Ute4qcvwXqZVUz9zkQxSgqIXobisQk+T8VyJoVIPVVYpbtbZNQvOSqeK3Z
-ywplh6ZmwcSBo3c6WB4L7oOLnR7SUqTMHW+wmG2UMbX4cQrcufx9MmDm66+KAQ==
------END CERTIFICATE-----
-
 # Issuer: CN=Hongkong Post Root CA 1 O=Hongkong Post
 # Subject: CN=Hongkong Post Root CA 1 O=Hongkong Post
 # Label: "Hongkong Post Root CA 1"
@@ -2200,6 +2034,45 @@ t/2jioSgrGK+KwmHNPBqAbubKVY8/gA3zyNs8U6qtnRGEmyR7jTV7JqR50S+kDFy
 SjnRBUkLp7Y3gaVdjKozXoEofKd9J+sAro03
 -----END CERTIFICATE-----
 
+# Issuer: CN=EC-ACC O=Agencia Catalana de Certificacio (NIF Q-0801176-I) OU=Serveis Publics de Certificacio/Vegeu https://www.catcert.net/verarrel (c)03/Jerarquia Entitats de Certificacio Catalanes
+# Subject: CN=EC-ACC O=Agencia Catalana de Certificacio (NIF Q-0801176-I) OU=Serveis Publics de Certificacio/Vegeu https://www.catcert.net/verarrel (c)03/Jerarquia Entitats de Certificacio Catalanes
+# Label: "EC-ACC"
+# Serial: -23701579247955709139626555126524820479
+# MD5 Fingerprint: eb:f5:9d:29:0d:61:f9:42:1f:7c:c2:ba:6d:e3:15:09
+# SHA1 Fingerprint: 28:90:3a:63:5b:52:80:fa:e6:77:4c:0b:6d:a7:d6:ba:a6:4a:f2:e8
+# SHA256 Fingerprint: 88:49:7f:01:60:2f:31:54:24:6a:e2:8c:4d:5a:ef:10:f1:d8:7e:bb:76:62:6f:4a:e0:b7:f9:5b:a7:96:87:99
+-----BEGIN CERTIFICATE-----
+MIIFVjCCBD6gAwIBAgIQ7is969Qh3hSoYqwE893EATANBgkqhkiG9w0BAQUFADCB
+8zELMAkGA1UEBhMCRVMxOzA5BgNVBAoTMkFnZW5jaWEgQ2F0YWxhbmEgZGUgQ2Vy
+dGlmaWNhY2lvIChOSUYgUS0wODAxMTc2LUkpMSgwJgYDVQQLEx9TZXJ2ZWlzIFB1
+YmxpY3MgZGUgQ2VydGlmaWNhY2lvMTUwMwYDVQQLEyxWZWdldSBodHRwczovL3d3
+dy5jYXRjZXJ0Lm5ldC92ZXJhcnJlbCAoYykwMzE1MDMGA1UECxMsSmVyYXJxdWlh
+IEVudGl0YXRzIGRlIENlcnRpZmljYWNpbyBDYXRhbGFuZXMxDzANBgNVBAMTBkVD
+LUFDQzAeFw0wMzAxMDcyMzAwMDBaFw0zMTAxMDcyMjU5NTlaMIHzMQswCQYDVQQG
+EwJFUzE7MDkGA1UEChMyQWdlbmNpYSBDYXRhbGFuYSBkZSBDZXJ0aWZpY2FjaW8g
+KE5JRiBRLTA4MDExNzYtSSkxKDAmBgNVBAsTH1NlcnZlaXMgUHVibGljcyBkZSBD
+ZXJ0aWZpY2FjaW8xNTAzBgNVBAsTLFZlZ2V1IGh0dHBzOi8vd3d3LmNhdGNlcnQu
+bmV0L3ZlcmFycmVsIChjKTAzMTUwMwYDVQQLEyxKZXJhcnF1aWEgRW50aXRhdHMg
+ZGUgQ2VydGlmaWNhY2lvIENhdGFsYW5lczEPMA0GA1UEAxMGRUMtQUNDMIIBIjAN
+BgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAsyLHT+KXQpWIR4NA9h0X84NzJB5R
+85iKw5K4/0CQBXCHYMkAqbWUZRkiFRfCQ2xmRJoNBD45b6VLeqpjt4pEndljkYRm
+4CgPukLjbo73FCeTae6RDqNfDrHrZqJyTxIThmV6PttPB/SnCWDaOkKZx7J/sxaV
+HMf5NLWUhdWZXqBIoH7nF2W4onW4HvPlQn2v7fOKSGRdghST2MDk/7NQcvJ29rNd
+QlB50JQ+awwAvthrDk4q7D7SzIKiGGUzE3eeml0aE9jD2z3Il3rucO2n5nzbcc8t
+lGLfbdb1OL4/pYUKGbio2Al1QnDE6u/LDsg0qBIimAy4E5S2S+zw0JDnJwIDAQAB
+o4HjMIHgMB0GA1UdEQQWMBSBEmVjX2FjY0BjYXRjZXJ0Lm5ldDAPBgNVHRMBAf8E
+BTADAQH/MA4GA1UdDwEB/wQEAwIBBjAdBgNVHQ4EFgQUoMOLRKo3pUW/l4Ba0fF4
+opvpXY0wfwYDVR0gBHgwdjB0BgsrBgEEAfV4AQMBCjBlMCwGCCsGAQUFBwIBFiBo
+dHRwczovL3d3dy5jYXRjZXJ0Lm5ldC92ZXJhcnJlbDA1BggrBgEFBQcCAjApGidW
+ZWdldSBodHRwczovL3d3dy5jYXRjZXJ0Lm5ldC92ZXJhcnJlbCAwDQYJKoZIhvcN
+AQEFBQADggEBAKBIW4IB9k1IuDlVNZyAelOZ1Vr/sXE7zDkJlF7W2u++AVtd0x7Y
+/X1PzaBB4DSTv8vihpw3kpBWHNzrKQXlxJ7HNd+KDM3FIUPpqojlNcAZQmNaAl6k
+SBg6hW/cnbw/nZzBh7h6YQjpdwt/cKt63dmXLGQehb+8dJahw3oS7AwaboMMPOhy
+Rp/7SNVel+axofjk70YllJyJ22k4vuxcDlbHZVHlUIiIv0LVKz3l+bqeLrPK9HOS
+Agu+TGbrIP65y7WZf+a2E/rKS03Z7lNGBjvGTq2TWoF+bCpLagVFjPIhpDGQh2xl
+nJ2lYJU6Un/10asIbvPuW/mIPX64b24D5EI=
+-----END CERTIFICATE-----
+
 # Issuer: CN=Hellenic Academic and Research Institutions RootCA 2011 O=Hellenic Academic and Research Institutions Cert. Authority
 # Subject: CN=Hellenic Academic and Research Institutions RootCA 2011 O=Hellenic Academic and Research Institutions Cert. Authority
 # Label: "Hellenic Academic and Research Institutions RootCA 2011"
@@ -3453,46 +3326,6 @@ AAoACxGV2lZFA4gKn2fQ1XmxqI1AbQ3CekD6819kR5LLU7m7Wc5P/dAVUwHY3+vZ
 5nbv0CO7O6l5s9UCKc2Jo5YPSjXnTkLAdc0Hz+Ys63su
 -----END CERTIFICATE-----
 
-# Issuer: CN=Certinomis - Root CA O=Certinomis OU=0002 433998903
-# Subject: CN=Certinomis - Root CA O=Certinomis OU=0002 433998903
-# Label: "Certinomis - Root CA"
-# Serial: 1
-# MD5 Fingerprint: 14:0a:fd:8d:a8:28:b5:38:69:db:56:7e:61:22:03:3f
-# SHA1 Fingerprint: 9d:70:bb:01:a5:a4:a0:18:11:2e:f7:1c:01:b9:32:c5:34:e7:88:a8
-# SHA256 Fingerprint: 2a:99:f5:bc:11:74:b7:3c:bb:1d:62:08:84:e0:1c:34:e5:1c:cb:39:78:da:12:5f:0e:33:26:88:83:bf:41:58
------BEGIN CERTIFICATE-----
-MIIFkjCCA3qgAwIBAgIBATANBgkqhkiG9w0BAQsFADBaMQswCQYDVQQGEwJGUjET
-MBEGA1UEChMKQ2VydGlub21pczEXMBUGA1UECxMOMDAwMiA0MzM5OTg5MDMxHTAb
-BgNVBAMTFENlcnRpbm9taXMgLSBSb290IENBMB4XDTEzMTAyMTA5MTcxOFoXDTMz
-MTAyMTA5MTcxOFowWjELMAkGA1UEBhMCRlIxEzARBgNVBAoTCkNlcnRpbm9taXMx
-FzAVBgNVBAsTDjAwMDIgNDMzOTk4OTAzMR0wGwYDVQQDExRDZXJ0aW5vbWlzIC0g
-Um9vdCBDQTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBANTMCQosP5L2
-fxSeC5yaah1AMGT9qt8OHgZbn1CF6s2Nq0Nn3rD6foCWnoR4kkjW4znuzuRZWJfl
-LieY6pOod5tK8O90gC3rMB+12ceAnGInkYjwSond3IjmFPnVAy//ldu9n+ws+hQV
-WZUKxkd8aRi5pwP5ynapz8dvtF4F/u7BUrJ1Mofs7SlmO/NKFoL21prbcpjp3vDF
-TKWrteoB4owuZH9kb/2jJZOLyKIOSY008B/sWEUuNKqEUL3nskoTuLAPrjhdsKkb
-5nPJWqHZZkCqqU2mNAKthH6yI8H7KsZn9DS2sJVqM09xRLWtwHkziOC/7aOgFLSc
-CbAK42C++PhmiM1b8XcF4LVzbsF9Ri6OSyemzTUK/eVNfaoqoynHWmgE6OXWk6Ri
-wsXm9E/G+Z8ajYJJGYrKWUM66A0ywfRMEwNvbqY/kXPLynNvEiCL7sCCeN5LLsJJ
-wx3tFvYk9CcbXFcx3FXuqB5vbKziRcxXV4p1VxngtViZSTYxPDMBbRZKzbgqg4SG
-m/lg0h9tkQPTYKbVPZrdd5A9NaSfD171UkRpucC63M9933zZxKyGIjK8e2uR73r4
-F2iw4lNVYC2vPsKD2NkJK/DAZNuHi5HMkesE/Xa0lZrmFAYb1TQdvtj/dBxThZng
-WVJKYe2InmtJiUZ+IFrZ50rlau7SZRFDAgMBAAGjYzBhMA4GA1UdDwEB/wQEAwIB
-BjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBTvkUz1pcMw6C8I6tNxIqSSaHh0
-2TAfBgNVHSMEGDAWgBTvkUz1pcMw6C8I6tNxIqSSaHh02TANBgkqhkiG9w0BAQsF
-AAOCAgEAfj1U2iJdGlg+O1QnurrMyOMaauo++RLrVl89UM7g6kgmJs95Vn6RHJk/
-0KGRHCwPT5iVWVO90CLYiF2cN/z7ZMF4jIuaYAnq1fohX9B0ZedQxb8uuQsLrbWw
-F6YSjNRieOpWauwK0kDDPAUwPk2Ut59KA9N9J0u2/kTO+hkzGm2kQtHdzMjI1xZS
-g081lLMSVX3l4kLr5JyTCcBMWwerx20RoFAXlCOotQqSD7J6wWAsOMwaplv/8gzj
-qh8c3LigkyfeY+N/IZ865Z764BNqdeuWXGKRlI5nU7aJ+BIJy29SWwNyhlCVCNSN
-h4YVH5Uk2KRvms6knZtt0rJ2BobGVgjF6wnaNsIbW0G+YSrjcOa4pvi2WsS9Iff/
-ql+hbHY5ZtbqTFXhADObE5hjyW/QASAJN1LnDE8+zbz1X5YnpyACleAu6AdBBR8V
-btaw5BngDwKTACdyxYvRVB9dSsNAl35VpnzBMwQUAR1JIGkLGZOdblgi90AMRgwj
-Y/M50n92Uaf0yKHxDHYiI0ZSKS3io0EHVmmY0gUJvGnHWmHNj4FgFU2A3ZDifcRQ
-8ow7bkrHxuaAKzyBvBGAFhAn1/DNP3nMcyrDflOR1m749fPH0FFNjkulW+YZFzvW
-gQncItzujrnEj1PhZ7szuIgVRs/taTX/dQ1G885x4cVrhkIGuUE=
------END CERTIFICATE-----
-
 # Issuer: CN=OISTE WISeKey Global Root GB CA O=WISeKey OU=OISTE Foundation Endorsed
 # Subject: CN=OISTE WISeKey Global Root GB CA O=WISeKey OU=OISTE Foundation Endorsed
 # Label: "OISTE WISeKey Global Root GB CA"
@@ -3849,47 +3682,6 @@ CkcO8DdZEv8tmZQoTipPNU0zWgIxAOp1AE47xDqUEpHJWEadIRNyp4iciuRMStuW
 1KyLa2tJElMzrdfkviT8tQp21KW8EA==
 -----END CERTIFICATE-----
 
-# Issuer: CN=LuxTrust Global Root 2 O=LuxTrust S.A.
-# Subject: CN=LuxTrust Global Root 2 O=LuxTrust S.A.
-# Label: "LuxTrust Global Root 2"
-# Serial: 59914338225734147123941058376788110305822489521
-# MD5 Fingerprint: b2:e1:09:00:61:af:f7:f1:91:6f:c4:ad:8d:5e:3b:7c
-# SHA1 Fingerprint: 1e:0e:56:19:0a:d1:8b:25:98:b2:04:44:ff:66:8a:04:17:99:5f:3f
-# SHA256 Fingerprint: 54:45:5f:71:29:c2:0b:14:47:c4:18:f9:97:16:8f:24:c5:8f:c5:02:3b:f5:da:5b:e2:eb:6e:1d:d8:90:2e:d5
------BEGIN CERTIFICATE-----
-MIIFwzCCA6ugAwIBAgIUCn6m30tEntpqJIWe5rgV0xZ/u7EwDQYJKoZIhvcNAQEL
-BQAwRjELMAkGA1UEBhMCTFUxFjAUBgNVBAoMDUx1eFRydXN0IFMuQS4xHzAdBgNV
-BAMMFkx1eFRydXN0IEdsb2JhbCBSb290IDIwHhcNMTUwMzA1MTMyMTU3WhcNMzUw
-MzA1MTMyMTU3WjBGMQswCQYDVQQGEwJMVTEWMBQGA1UECgwNTHV4VHJ1c3QgUy5B
-LjEfMB0GA1UEAwwWTHV4VHJ1c3QgR2xvYmFsIFJvb3QgMjCCAiIwDQYJKoZIhvcN
-AQEBBQADggIPADCCAgoCggIBANeFl78RmOnwYoNMPIf5U2o3C/IPPIfOb9wmKb3F
-ibrJgz337spbxm1Jc7TJRqMbNBM/wYlFV/TZsfs2ZUv7COJIcRHIbjuend+JZTem
-hfY7RBi2xjcwYkSSl2l9QjAk5A0MiWtj3sXh306pFGxT4GHO9hcvHTy95iJMHZP1
-EMShduxq3sVs35a0VkBCwGKSMKEtFZSg0iAGCW5qbeXrt77U8PEVfIvmTroTzEsn
-Xpk8F12PgX8zPU/TPxvsXD/wPEx1bvKm1Z3aLQdjAsZy6ZS8TEmVT4hSyNvoaYL4
-zDRbIvCGp4m9SAptZoFtyMhk+wHh9OHe2Z7d21vUKpkmFRseTJIpgp7VkoGSQXAZ
-96Tlk0u8d2cx3Rz9MXANF5kM+Qw5GSoXtTBxVdUPrljhPS80m8+f9niFwpN6cj5m
-j5wWEWCPnolvZ77gR1o7DJpni89Gxq44o/KnvObWhWszJHAiS8sIm7vI+AIpHb4g
-DEa/a4ebsypmQjVGbKq6rfmYe+lQVRQxv7HaLe2ArWgk+2mr2HETMOZns4dA/Yl+
-8kPREd8vZS9kzl8UubG/Mb2HeFpZZYiq/FkySIbWTLkpS5XTdvN3JW1CHDiDTf2j
-X5t/Lax5Gw5CMZdjpPuKadUiDTSQMC6otOBttpSsvItO13D8xTiOZCXhTTmQzsmH
-hFhxAgMBAAGjgagwgaUwDwYDVR0TAQH/BAUwAwEB/zBCBgNVHSAEOzA5MDcGByuB
-KwEBAQowLDAqBggrBgEFBQcCARYeaHR0cHM6Ly9yZXBvc2l0b3J5Lmx1eHRydXN0
-Lmx1MA4GA1UdDwEB/wQEAwIBBjAfBgNVHSMEGDAWgBT/GCh2+UgFLKGu8SsbK7JT
-+Et8szAdBgNVHQ4EFgQU/xgodvlIBSyhrvErGyuyU/hLfLMwDQYJKoZIhvcNAQEL
-BQADggIBAGoZFO1uecEsh9QNcH7X9njJCwROxLHOk3D+sFTAMs2ZMGQXvw/l4jP9
-BzZAcg4atmpZ1gDlaCDdLnINH2pkMSCEfUmmWjfrRcmF9dTHF5kH5ptV5AzoqbTO
-jFu1EVzPig4N1qx3gf4ynCSecs5U89BvolbW7MM3LGVYvlcAGvI1+ut7MV3CwRI9
-loGIlonBWVx65n9wNOeD4rHh4bhY79SV5GCc8JaXcozrhAIuZY+kt9J/Z93I055c
-qqmkoCUUBpvsT34tC38ddfEz2O3OuHVtPlu5mB0xDVbYQw8wkbIEa91WvpWAVWe+
-2M2D2RjuLg+GLZKecBPs3lHJQ3gCpU3I+V/EkVhGFndadKpAvAefMLmx9xIX3eP/
-JEAdemrRTxgKqpAd60Ae36EeRJIQmvKN4dFLRp7oRUKX6kWZ8+xm1QL68qZKJKre
-zrnK+T+Tb/mjuuqlPpmt/f97mfVl7vBZKGfXkJWkE4SphMHozs51k2MavDzq1WQf
-LSoSOcbDWjLtR5EWDrw4wVDej8oqkDQc7kGUnF4ZLvhFSZl0kbAEb+MEWrGrKqv+
-x9CWttrhSmQGbmBNvUJO/3jaJMobtNeWOWyu8Q6qp31IiyBMz2TWuJdGsE7RKlY6
-oJO9r4Ak4Ap+58rVyuiFVdw2KuGUaJPHZnJED4AhMmwlxyOAgwrr
------END CERTIFICATE-----
-
 # Issuer: CN=TUBITAK Kamu SM SSL Kok Sertifikasi - Surum 1 O=Turkiye Bilimsel ve Teknolojik Arastirma Kurumu - TUBITAK OU=Kamu Sertifikasyon Merkezi - Kamu SM
 # Subject: CN=TUBITAK Kamu SM SSL Kok Sertifikasi - Surum 1 O=Turkiye Bilimsel ve Teknolojik Arastirma Kurumu - TUBITAK OU=Kamu Sertifikasyon Merkezi - Kamu SM
 # Label: "TUBITAK Kamu SM SSL Kok Sertifikasi - Surum 1"
@@ -4656,3 +4448,173 @@ L5/ndtFhKvshuzHQqp9HpLIiyhY6UFfEW0NnxWViA0kB60PZ2Pierc+xYw5F9KBa
 LJstxabArahH9CdMOA0uG0k7UvToiIMrVCjU8jVStDKDYmlkDJGcn5fqdBb9HxEG
 mpv0
 -----END CERTIFICATE-----
+
+# Issuer: CN=Entrust Root Certification Authority - G4 O=Entrust, Inc. OU=See www.entrust.net/legal-terms/(c) 2015 Entrust, Inc. - for authorized use only
+# Subject: CN=Entrust Root Certification Authority - G4 O=Entrust, Inc. OU=See www.entrust.net/legal-terms/(c) 2015 Entrust, Inc. - for authorized use only
+# Label: "Entrust Root Certification Authority - G4"
+# Serial: 289383649854506086828220374796556676440
+# MD5 Fingerprint: 89:53:f1:83:23:b7:7c:8e:05:f1:8c:71:38:4e:1f:88
+# SHA1 Fingerprint: 14:88:4e:86:26:37:b0:26:af:59:62:5c:40:77:ec:35:29:ba:96:01
+# SHA256 Fingerprint: db:35:17:d1:f6:73:2a:2d:5a:b9:7c:53:3e:c7:07:79:ee:32:70:a6:2f:b4:ac:42:38:37:24:60:e6:f0:1e:88
+-----BEGIN CERTIFICATE-----
+MIIGSzCCBDOgAwIBAgIRANm1Q3+vqTkPAAAAAFVlrVgwDQYJKoZIhvcNAQELBQAw
+gb4xCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1FbnRydXN0LCBJbmMuMSgwJgYDVQQL
+Ex9TZWUgd3d3LmVudHJ1c3QubmV0L2xlZ2FsLXRlcm1zMTkwNwYDVQQLEzAoYykg
+MjAxNSBFbnRydXN0LCBJbmMuIC0gZm9yIGF1dGhvcml6ZWQgdXNlIG9ubHkxMjAw
+BgNVBAMTKUVudHJ1c3QgUm9vdCBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eSAtIEc0
+MB4XDTE1MDUyNzExMTExNloXDTM3MTIyNzExNDExNlowgb4xCzAJBgNVBAYTAlVT
+MRYwFAYDVQQKEw1FbnRydXN0LCBJbmMuMSgwJgYDVQQLEx9TZWUgd3d3LmVudHJ1
+c3QubmV0L2xlZ2FsLXRlcm1zMTkwNwYDVQQLEzAoYykgMjAxNSBFbnRydXN0LCBJ
+bmMuIC0gZm9yIGF1dGhvcml6ZWQgdXNlIG9ubHkxMjAwBgNVBAMTKUVudHJ1c3Qg
+Um9vdCBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eSAtIEc0MIICIjANBgkqhkiG9w0B
+AQEFAAOCAg8AMIICCgKCAgEAsewsQu7i0TD/pZJH4i3DumSXbcr3DbVZwbPLqGgZ
+2K+EbTBwXX7zLtJTmeH+H17ZSK9dE43b/2MzTdMAArzE+NEGCJR5WIoV3imz/f3E
+T+iq4qA7ec2/a0My3dl0ELn39GjUu9CH1apLiipvKgS1sqbHoHrmSKvS0VnM1n4j
+5pds8ELl3FFLFUHtSUrJ3hCX1nbB76W1NhSXNdh4IjVS70O92yfbYVaCNNzLiGAM
+C1rlLAHGVK/XqsEQe9IFWrhAnoanw5CGAlZSCXqc0ieCU0plUmr1POeo8pyvi73T
+DtTUXm6Hnmo9RR3RXRv06QqsYJn7ibT/mCzPfB3pAqoEmh643IhuJbNsZvc8kPNX
+wbMv9W3y+8qh+CmdRouzavbmZwe+LGcKKh9asj5XxNMhIWNlUpEbsZmOeX7m640A
+2Vqq6nPopIICR5b+W45UYaPrL0swsIsjdXJ8ITzI9vF01Bx7owVV7rtNOzK+mndm
+nqxpkCIHH2E6lr7lmk/MBTwoWdPBDFSoWWG9yHJM6Nyfh3+9nEg2XpWjDrk4JFX8
+dWbrAuMINClKxuMrLzOg2qOGpRKX/YAr2hRC45K9PvJdXmd0LhyIRyk0X+IyqJwl
+N4y6mACXi0mWHv0liqzc2thddG5msP9E36EYxr5ILzeUePiVSj9/E15dWf10hkNj
+c0kCAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYwHQYD
+VR0OBBYEFJ84xFYjwznooHFs6FRM5Og6sb9nMA0GCSqGSIb3DQEBCwUAA4ICAQAS
+5UKme4sPDORGpbZgQIeMJX6tuGguW8ZAdjwD+MlZ9POrYs4QjbRaZIxowLByQzTS
+Gwv2LFPSypBLhmb8qoMi9IsabyZIrHZ3CL/FmFz0Jomee8O5ZDIBf9PD3Vht7LGr
+hFV0d4QEJ1JrhkzO3bll/9bGXp+aEJlLdWr+aumXIOTkdnrG0CSqkM0gkLpHZPt/
+B7NTeLUKYvJzQ85BK4FqLoUWlFPUa19yIqtRLULVAJyZv967lDtX/Zr1hstWO1uI
+AeV8KEsD+UmDfLJ/fOPtjqF/YFOOVZ1QNBIPt5d7bIdKROf1beyAN/BYGW5KaHbw
+H5Lk6rWS02FREAutp9lfx1/cH6NcjKF+m7ee01ZvZl4HliDtC3T7Zk6LERXpgUl+
+b7DUUH8i119lAg2m9IUe2K4GS0qn0jFmwvjO5QimpAKWRGhXxNUzzxkvFMSUHHuk
+2fCfDrGA4tGeEWSpiBE6doLlYsKA2KSD7ZPvfC+QsDJMlhVoSFLUmQjAJOgc47Ol
+IQ6SwJAfzyBfyjs4x7dtOvPmRLgOMWuIjnDrnBdSqEGULoe256YSxXXfW8AKbnuk
+5F6G+TaU33fD6Q3AOfF5u0aOq0NZJ7cguyPpVkAh7DE9ZapD8j3fcEThuk0mEDuY
+n/PIjhs4ViFqUZPTkcpG2om3PVODLAgfi49T3f+sHw==
+-----END CERTIFICATE-----
+
+# Issuer: CN=Microsoft ECC Root Certificate Authority 2017 O=Microsoft Corporation
+# Subject: CN=Microsoft ECC Root Certificate Authority 2017 O=Microsoft Corporation
+# Label: "Microsoft ECC Root Certificate Authority 2017"
+# Serial: 136839042543790627607696632466672567020
+# MD5 Fingerprint: dd:a1:03:e6:4a:93:10:d1:bf:f0:19:42:cb:fe:ed:67
+# SHA1 Fingerprint: 99:9a:64:c3:7f:f4:7d:9f:ab:95:f1:47:69:89:14:60:ee:c4:c3:c5
+# SHA256 Fingerprint: 35:8d:f3:9d:76:4a:f9:e1:b7:66:e9:c9:72:df:35:2e:e1:5c:fa:c2:27:af:6a:d1:d7:0e:8e:4a:6e:dc:ba:02
+-----BEGIN CERTIFICATE-----
+MIICWTCCAd+gAwIBAgIQZvI9r4fei7FK6gxXMQHC7DAKBggqhkjOPQQDAzBlMQsw
+CQYDVQQGEwJVUzEeMBwGA1UEChMVTWljcm9zb2Z0IENvcnBvcmF0aW9uMTYwNAYD
+VQQDEy1NaWNyb3NvZnQgRUNDIFJvb3QgQ2VydGlmaWNhdGUgQXV0aG9yaXR5IDIw
+MTcwHhcNMTkxMjE4MjMwNjQ1WhcNNDIwNzE4MjMxNjA0WjBlMQswCQYDVQQGEwJV
+UzEeMBwGA1UEChMVTWljcm9zb2Z0IENvcnBvcmF0aW9uMTYwNAYDVQQDEy1NaWNy
+b3NvZnQgRUNDIFJvb3QgQ2VydGlmaWNhdGUgQXV0aG9yaXR5IDIwMTcwdjAQBgcq
+hkjOPQIBBgUrgQQAIgNiAATUvD0CQnVBEyPNgASGAlEvaqiBYgtlzPbKnR5vSmZR
+ogPZnZH6thaxjG7efM3beaYvzrvOcS/lpaso7GMEZpn4+vKTEAXhgShC48Zo9OYb
+hGBKia/teQ87zvH2RPUBeMCjVDBSMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMBAf8E
+BTADAQH/MB0GA1UdDgQWBBTIy5lycFIM+Oa+sgRXKSrPQhDtNTAQBgkrBgEEAYI3
+FQEEAwIBADAKBggqhkjOPQQDAwNoADBlAjBY8k3qDPlfXu5gKcs68tvWMoQZP3zV
+L8KxzJOuULsJMsbG7X7JNpQS5GiFBqIb0C8CMQCZ6Ra0DvpWSNSkMBaReNtUjGUB
+iudQZsIxtzm6uBoiB078a1QWIP8rtedMDE2mT3M=
+-----END CERTIFICATE-----
+
+# Issuer: CN=Microsoft RSA Root Certificate Authority 2017 O=Microsoft Corporation
+# Subject: CN=Microsoft RSA Root Certificate Authority 2017 O=Microsoft Corporation
+# Label: "Microsoft RSA Root Certificate Authority 2017"
+# Serial: 40975477897264996090493496164228220339
+# MD5 Fingerprint: 10:ff:00:ff:cf:c9:f8:c7:7a:c0:ee:35:8e:c9:0f:47
+# SHA1 Fingerprint: 73:a5:e6:4a:3b:ff:83:16:ff:0e:dc:cc:61:8a:90:6e:4e:ae:4d:74
+# SHA256 Fingerprint: c7:41:f7:0f:4b:2a:8d:88:bf:2e:71:c1:41:22:ef:53:ef:10:eb:a0:cf:a5:e6:4c:fa:20:f4:18:85:30:73:e0
+-----BEGIN CERTIFICATE-----
+MIIFqDCCA5CgAwIBAgIQHtOXCV/YtLNHcB6qvn9FszANBgkqhkiG9w0BAQwFADBl
+MQswCQYDVQQGEwJVUzEeMBwGA1UEChMVTWljcm9zb2Z0IENvcnBvcmF0aW9uMTYw
+NAYDVQQDEy1NaWNyb3NvZnQgUlNBIFJvb3QgQ2VydGlmaWNhdGUgQXV0aG9yaXR5
+IDIwMTcwHhcNMTkxMjE4MjI1MTIyWhcNNDIwNzE4MjMwMDIzWjBlMQswCQYDVQQG
+EwJVUzEeMBwGA1UEChMVTWljcm9zb2Z0IENvcnBvcmF0aW9uMTYwNAYDVQQDEy1N
+aWNyb3NvZnQgUlNBIFJvb3QgQ2VydGlmaWNhdGUgQXV0aG9yaXR5IDIwMTcwggIi
+MA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQDKW76UM4wplZEWCpW9R2LBifOZ
+Nt9GkMml7Xhqb0eRaPgnZ1AzHaGm++DlQ6OEAlcBXZxIQIJTELy/xztokLaCLeX0
+ZdDMbRnMlfl7rEqUrQ7eS0MdhweSE5CAg2Q1OQT85elss7YfUJQ4ZVBcF0a5toW1
+HLUX6NZFndiyJrDKxHBKrmCk3bPZ7Pw71VdyvD/IybLeS2v4I2wDwAW9lcfNcztm
+gGTjGqwu+UcF8ga2m3P1eDNbx6H7JyqhtJqRjJHTOoI+dkC0zVJhUXAoP8XFWvLJ
+jEm7FFtNyP9nTUwSlq31/niol4fX/V4ggNyhSyL71Imtus5Hl0dVe49FyGcohJUc
+aDDv70ngNXtk55iwlNpNhTs+VcQor1fznhPbRiefHqJeRIOkpcrVE7NLP8TjwuaG
+YaRSMLl6IE9vDzhTyzMMEyuP1pq9KsgtsRx9S1HKR9FIJ3Jdh+vVReZIZZ2vUpC6
+W6IYZVcSn2i51BVrlMRpIpj0M+Dt+VGOQVDJNE92kKz8OMHY4Xu54+OU4UZpyw4K
+UGsTuqwPN1q3ErWQgR5WrlcihtnJ0tHXUeOrO8ZV/R4O03QK0dqq6mm4lyiPSMQH
++FJDOvTKVTUssKZqwJz58oHhEmrARdlns87/I6KJClTUFLkqqNfs+avNJVgyeY+Q
+W5g5xAgGwax/Dj0ApQIDAQABo1QwUjAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/
+BAUwAwEB/zAdBgNVHQ4EFgQUCctZf4aycI8awznjwNnpv7tNsiMwEAYJKwYBBAGC
+NxUBBAMCAQAwDQYJKoZIhvcNAQEMBQADggIBAKyvPl3CEZaJjqPnktaXFbgToqZC
+LgLNFgVZJ8og6Lq46BrsTaiXVq5lQ7GPAJtSzVXNUzltYkyLDVt8LkS/gxCP81OC
+gMNPOsduET/m4xaRhPtthH80dK2Jp86519efhGSSvpWhrQlTM93uCupKUY5vVau6
+tZRGrox/2KJQJWVggEbbMwSubLWYdFQl3JPk+ONVFT24bcMKpBLBaYVu32TxU5nh
+SnUgnZUP5NbcA/FZGOhHibJXWpS2qdgXKxdJ5XbLwVaZOjex/2kskZGT4d9Mozd2
+TaGf+G0eHdP67Pv0RR0Tbc/3WeUiJ3IrhvNXuzDtJE3cfVa7o7P4NHmJweDyAmH3
+pvwPuxwXC65B2Xy9J6P9LjrRk5Sxcx0ki69bIImtt2dmefU6xqaWM/5TkshGsRGR
+xpl/j8nWZjEgQRCHLQzWwa80mMpkg/sTV9HB8Dx6jKXB/ZUhoHHBk2dxEuqPiApp
+GWSZI1b7rCoucL5mxAyE7+WL85MB+GqQk2dLsmijtWKP6T+MejteD+eMuMZ87zf9
+dOLITzNy4ZQ5bb0Sr74MTnB8G2+NszKTc0QWbej09+CVgI+WXTik9KveCjCHk9hN
+AHFiRSdLOkKEW39lt2c0Ui2cFmuqqNh7o0JMcccMyj6D5KbvtwEwXlGjefVwaaZB
+RA+GsCyRxj3qrg+E
+-----END CERTIFICATE-----
+
+# Issuer: CN=e-Szigno Root CA 2017 O=Microsec Ltd.
+# Subject: CN=e-Szigno Root CA 2017 O=Microsec Ltd.
+# Label: "e-Szigno Root CA 2017"
+# Serial: 411379200276854331539784714
+# MD5 Fingerprint: de:1f:f6:9e:84:ae:a7:b4:21:ce:1e:58:7d:d1:84:98
+# SHA1 Fingerprint: 89:d4:83:03:4f:9e:9a:48:80:5f:72:37:d4:a9:a6:ef:cb:7c:1f:d1
+# SHA256 Fingerprint: be:b0:0b:30:83:9b:9b:c3:2c:32:e4:44:79:05:95:06:41:f2:64:21:b1:5e:d0:89:19:8b:51:8a:e2:ea:1b:99
+-----BEGIN CERTIFICATE-----
+MIICQDCCAeWgAwIBAgIMAVRI7yH9l1kN9QQKMAoGCCqGSM49BAMCMHExCzAJBgNV
+BAYTAkhVMREwDwYDVQQHDAhCdWRhcGVzdDEWMBQGA1UECgwNTWljcm9zZWMgTHRk
+LjEXMBUGA1UEYQwOVkFUSFUtMjM1ODQ0OTcxHjAcBgNVBAMMFWUtU3ppZ25vIFJv
+b3QgQ0EgMjAxNzAeFw0xNzA4MjIxMjA3MDZaFw00MjA4MjIxMjA3MDZaMHExCzAJ
+BgNVBAYTAkhVMREwDwYDVQQHDAhCdWRhcGVzdDEWMBQGA1UECgwNTWljcm9zZWMg
+THRkLjEXMBUGA1UEYQwOVkFUSFUtMjM1ODQ0OTcxHjAcBgNVBAMMFWUtU3ppZ25v
+IFJvb3QgQ0EgMjAxNzBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABJbcPYrYsHtv
+xie+RJCxs1YVe45DJH0ahFnuY2iyxl6H0BVIHqiQrb1TotreOpCmYF9oMrWGQd+H
+Wyx7xf58etqjYzBhMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMB0G
+A1UdDgQWBBSHERUI0arBeAyxr87GyZDvvzAEwDAfBgNVHSMEGDAWgBSHERUI0arB
+eAyxr87GyZDvvzAEwDAKBggqhkjOPQQDAgNJADBGAiEAtVfd14pVCzbhhkT61Nlo
+jbjcI4qKDdQvfepz7L9NbKgCIQDLpbQS+ue16M9+k/zzNY9vTlp8tLxOsvxyqltZ
++efcMQ==
+-----END CERTIFICATE-----
+
+# Issuer: O=CERTSIGN SA OU=certSIGN ROOT CA G2
+# Subject: O=CERTSIGN SA OU=certSIGN ROOT CA G2
+# Label: "certSIGN Root CA G2"
+# Serial: 313609486401300475190
+# MD5 Fingerprint: 8c:f1:75:8a:c6:19:cf:94:b7:f7:65:20:87:c3:97:c7
+# SHA1 Fingerprint: 26:f9:93:b4:ed:3d:28:27:b0:b9:4b:a7:e9:15:1d:a3:8d:92:e5:32
+# SHA256 Fingerprint: 65:7c:fe:2f:a7:3f:aa:38:46:25:71:f3:32:a2:36:3a:46:fc:e7:02:09:51:71:07:02:cd:fb:b6:ee:da:33:05
+-----BEGIN CERTIFICATE-----
+MIIFRzCCAy+gAwIBAgIJEQA0tk7GNi02MA0GCSqGSIb3DQEBCwUAMEExCzAJBgNV
+BAYTAlJPMRQwEgYDVQQKEwtDRVJUU0lHTiBTQTEcMBoGA1UECxMTY2VydFNJR04g
+Uk9PVCBDQSBHMjAeFw0xNzAyMDYwOTI3MzVaFw00MjAyMDYwOTI3MzVaMEExCzAJ
+BgNVBAYTAlJPMRQwEgYDVQQKEwtDRVJUU0lHTiBTQTEcMBoGA1UECxMTY2VydFNJ
+R04gUk9PVCBDQSBHMjCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAMDF
+dRmRfUR0dIf+DjuW3NgBFszuY5HnC2/OOwppGnzC46+CjobXXo9X69MhWf05N0Iw
+vlDqtg+piNguLWkh59E3GE59kdUWX2tbAMI5Qw02hVK5U2UPHULlj88F0+7cDBrZ
+uIt4ImfkabBoxTzkbFpG583H+u/E7Eu9aqSs/cwoUe+StCmrqzWaTOTECMYmzPhp
+n+Sc8CnTXPnGFiWeI8MgwT0PPzhAsP6CRDiqWhqKa2NYOLQV07YRaXseVO6MGiKs
+cpc/I1mbySKEwQdPzH/iV8oScLumZfNpdWO9lfsbl83kqK/20U6o2YpxJM02PbyW
+xPFsqa7lzw1uKA2wDrXKUXt4FMMgL3/7FFXhEZn91QqhngLjYl/rNUssuHLoPj1P
+rCy7Lobio3aP5ZMqz6WryFyNSwb/EkaseMsUBzXgqd+L6a8VTxaJW732jcZZroiF
+DsGJ6x9nxUWO/203Nit4ZoORUSs9/1F3dmKh7Gc+PoGD4FapUB8fepmrY7+EF3fx
+DTvf95xhszWYijqy7DwaNz9+j5LP2RIUZNoQAhVB/0/E6xyjyfqZ90bp4RjZsbgy
+LcsUDFDYg2WD7rlcz8sFWkz6GZdr1l0T08JcVLwyc6B49fFtHsufpaafItzRUZ6C
+eWRgKRM+o/1Pcmqr4tTluCRVLERLiohEnMqE0yo7AgMBAAGjQjBAMA8GA1UdEwEB
+/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMB0GA1UdDgQWBBSCIS1mxteg4BXrzkwJ
+d8RgnlRuAzANBgkqhkiG9w0BAQsFAAOCAgEAYN4auOfyYILVAzOBywaK8SJJ6ejq
+kX/GM15oGQOGO0MBzwdw5AgeZYWR5hEit/UCI46uuR59H35s5r0l1ZUa8gWmr4UC
+b6741jH/JclKyMeKqdmfS0mbEVeZkkMR3rYzpMzXjWR91M08KCy0mpbqTfXERMQl
+qiCA2ClV9+BB/AYm/7k29UMUA2Z44RGx2iBfRgB4ACGlHgAoYXhvqAEBj500mv/0
+OJD7uNGzcgbJceaBxXntC6Z58hMLnPddDnskk7RI24Zf3lCGeOdA5jGokHZwYa+c
+NywRtYK3qq4kNFtyDGkNzVmf9nGvnAvRCjj5BiKDUyUM/FHE5r7iOZULJK2v0ZXk
+ltd0ZGtxTgI8qoXzIKNDOXZbbFD+mpwUHmUUihW9o4JFWklWatKcsWMy5WHgUyIO
+pwpJ6st+H6jiYoD2EEVSmAYY3qXNL3+q1Ok+CHLsIwMCPKaq2LxndD0UF/tUSxfj
+03k9bWtJySgOLnRQvwzZRjoQhsmnP+mg7H/rpXdYaXHmgwo38oZJar55CJD2AhZk
+PuXaTH4MNMn5X7azKFGnpyuqSfqNZSlO42sTp5SjLVFteAxEy9/eCG/Oo2Sr05WE
+1LlSVHJ7liXMvGnjSG4N0MedJ5qq+BOS3R7fY581qRY27Iy4g/Q9iY/NtBde17MX
+QRBdJ3NghVdJIgc=
+-----END CERTIFICATE-----

Contents/Libraries/Shared/certifi/core.py

@@ -4,12 +4,57 @@
 certifi.py
 ~~~~~~~~~~
 
-This module returns the installation location of cacert.pem.
+This module returns the installation location of cacert.pem or its contents.
 """
 import os
 
+try:
+    from importlib.resources import path as get_path, read_text
 
-def where():
-    f = os.path.dirname(__file__)
+    _CACERT_CTX = None
+    _CACERT_PATH = None
 
-    return os.path.join(f, 'cacert.pem')
+    def where():
+        # This is slightly terrible, but we want to delay extracting the file
+        # in cases where we're inside of a zipimport situation until someone
+        # actually calls where(), but we don't want to re-extract the file
+        # on every call of where(), so we'll do it once then store it in a
+        # global variable.
+        global _CACERT_CTX
+        global _CACERT_PATH
+        if _CACERT_PATH is None:
+            # This is slightly janky, the importlib.resources API wants you to
+            # manage the cleanup of this file, so it doesn't actually return a
+            # path, it returns a context manager that will give you the path
+            # when you enter it and will do any cleanup when you leave it. In
+            # the common case of not needing a temporary file, it will just
+            # return the file system location and the __exit__() is a no-op.
+            #
+            # We also have to hold onto the actual context manager, because
+            # it will do the cleanup whenever it gets garbage collected, so
+            # we will also store that at the global level as well.
+            _CACERT_CTX = get_path("certifi", "cacert.pem")
+            _CACERT_PATH = str(_CACERT_CTX.__enter__())
+
+        return _CACERT_PATH
+
+
+except ImportError:
+    # This fallback will work for Python versions prior to 3.7 that lack the
+    # importlib.resources module but relies on the existing `where` function
+    # so won't address issues with environments like PyOxidizer that don't set
+    # __file__ on modules.
+    def read_text(_module, _path, encoding="ascii"):
+        with open(where(), "r", encoding=encoding) as data:
+            return data.read()
+
+    # If we don't have importlib.resources, then we will just do the old logic
+    # of assuming we're on the filesystem and munge the path directly.
+    def where():
+        f = os.path.dirname(__file__)
+
+        return os.path.join(f, "cacert.pem")
+
+
+def contents():
+    return read_text("certifi", "cacert.pem", encoding="ascii")

Contents/Libraries/Shared/pbkdf2.py

@@ -0,0 +1,297 @@
+#!/usr/bin/python
+# -*- coding: ascii -*-
+###########################################################################
+# pbkdf2 - PKCS#5 v2.0 Password-Based Key Derivation
+#
+# Copyright (C) 2007-2011 Dwayne C. Litzenberger <dlitz@dlitz.net>
+#
+# Permission is hereby granted, free of charge, to any person obtaining
+# a copy of this software and associated documentation files (the
+# "Software"), to deal in the Software without restriction, including
+# without limitation the rights to use, copy, modify, merge, publish,
+# distribute, sublicense, and/or sell copies of the Software, and to
+# permit persons to whom the Software is furnished to do so, subject to
+# the following conditions:
+#
+# The above copyright notice and this permission notice shall be
+# included in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+# LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+# WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+#
+# Country of origin: Canada
+#
+###########################################################################
+# Sample PBKDF2 usage:
+#   from Crypto.Cipher import AES
+#   from pbkdf2 import PBKDF2
+#   import os
+#
+#   salt = os.urandom(8)    # 64-bit salt
+#   key = PBKDF2("This passphrase is a secret.", salt).read(32) # 256-bit key
+#   iv = os.urandom(16)     # 128-bit IV
+#   cipher = AES.new(key, AES.MODE_CBC, iv)
+#     ...
+#
+# Sample crypt() usage:
+#   from pbkdf2 import crypt
+#   pwhash = crypt("secret")
+#   alleged_pw = raw_input("Enter password: ")
+#   if pwhash == crypt(alleged_pw, pwhash):
+#       print "Password good"
+#   else:
+#       print "Invalid password"
+#
+###########################################################################
+
+__version__ = "1.3"
+__all__ = ['PBKDF2', 'crypt']
+
+from struct import pack
+from random import randint
+import string
+import sys
+
+try:
+    # Use PyCrypto (if available).
+    from Crypto.Hash import HMAC, SHA as SHA1
+except ImportError:
+    # PyCrypto not available.  Use the Python standard library.
+    import hmac as HMAC
+    try:
+        from hashlib import sha1 as SHA1
+    except ImportError:
+        # hashlib not available.  Use the old sha module.
+        import sha as SHA1
+
+#
+# Python 2.1 thru 3.2 compatibility
+#
+
+if sys.version_info[0] == 2:
+    _0xffffffffL = long(1) << 32
+    def isunicode(s):
+        return isinstance(s, unicode)
+    def isbytes(s):
+        return isinstance(s, str)
+    def isinteger(n):
+        return isinstance(n, (int, long))
+    def b(s):
+        return s
+    def binxor(a, b):
+        return "".join([chr(ord(x) ^ ord(y)) for (x, y) in zip(a, b)])
+    def b64encode(data, chars="+/"):
+        tt = string.maketrans("+/", chars)
+        return data.encode('base64').replace("\n", "").translate(tt)
+    from binascii import b2a_hex
+else:
+    _0xffffffffL = 0xffffffff
+    def isunicode(s):
+        return isinstance(s, str)
+    def isbytes(s):
+        return isinstance(s, bytes)
+    def isinteger(n):
+        return isinstance(n, int)
+    def callable(obj):
+        return hasattr(obj, '__call__')
+    def b(s):
+       return s.encode("latin-1")
+    def binxor(a, b):
+        return bytes([x ^ y for (x, y) in zip(a, b)])
+    from base64 import b64encode as _b64encode
+    def b64encode(data, chars="+/"):
+        if isunicode(chars):
+            return _b64encode(data, chars.encode('utf-8')).decode('utf-8')
+        else:
+            return _b64encode(data, chars)
+    from binascii import b2a_hex as _b2a_hex
+    def b2a_hex(s):
+        return _b2a_hex(s).decode('us-ascii')
+    xrange = range
+
+class PBKDF2(object):
+    """PBKDF2.py : PKCS#5 v2.0 Password-Based Key Derivation
+
+    This implementation takes a passphrase and a salt (and optionally an
+    iteration count, a digest module, and a MAC module) and provides a
+    file-like object from which an arbitrarily-sized key can be read.
+
+    If the passphrase and/or salt are unicode objects, they are encoded as
+    UTF-8 before they are processed.
+
+    The idea behind PBKDF2 is to derive a cryptographic key from a
+    passphrase and a salt.
+
+    PBKDF2 may also be used as a strong salted password hash.  The
+    'crypt' function is provided for that purpose.
+
+    Remember: Keys generated using PBKDF2 are only as strong as the
+    passphrases they are derived from.
+    """
+
+    def __init__(self, passphrase, salt, iterations=1000,
+                 digestmodule=SHA1, macmodule=HMAC):
+        self.__macmodule = macmodule
+        self.__digestmodule = digestmodule
+        self._setup(passphrase, salt, iterations, self._pseudorandom)
+
+    def _pseudorandom(self, key, msg):
+        """Pseudorandom function.  e.g. HMAC-SHA1"""
+        return self.__macmodule.new(key=key, msg=msg,
+            digestmod=self.__digestmodule).digest()
+
+    def read(self, bytes):
+        """Read the specified number of key bytes."""
+        if self.closed:
+            raise ValueError("file-like object is closed")
+
+        size = len(self.__buf)
+        blocks = [self.__buf]
+        i = self.__blockNum
+        while size < bytes:
+            i += 1
+            if i > _0xffffffffL or i < 1:
+                # We could return "" here, but
+                raise OverflowError("derived key too long")
+            block = self.__f(i)
+            blocks.append(block)
+            size += len(block)
+        buf = b("").join(blocks)
+        retval = buf[:bytes]
+        self.__buf = buf[bytes:]
+        self.__blockNum = i
+        return retval
+
+    def __f(self, i):
+        # i must fit within 32 bits
+        assert 1 <= i <= _0xffffffffL
+        U = self.__prf(self.__passphrase, self.__salt + pack("!L", i))
+        result = U
+        for j in xrange(2, 1+self.__iterations):
+            U = self.__prf(self.__passphrase, U)
+            result = binxor(result, U)
+        return result
+
+    def hexread(self, octets):
+        """Read the specified number of octets. Return them as hexadecimal.
+
+        Note that len(obj.hexread(n)) == 2*n.
+        """
+        return b2a_hex(self.read(octets))
+
+    def _setup(self, passphrase, salt, iterations, prf):
+        # Sanity checks:
+
+        # passphrase and salt must be str or unicode (in the latter
+        # case, we convert to UTF-8)
+        if isunicode(passphrase):
+            passphrase = passphrase.encode("UTF-8")
+        elif not isbytes(passphrase):
+            raise TypeError("passphrase must be str or unicode")
+        if isunicode(salt):
+            salt = salt.encode("UTF-8")
+        elif not isbytes(salt):
+            raise TypeError("salt must be str or unicode")
+
+        # iterations must be an integer >= 1
+        if not isinteger(iterations):
+            raise TypeError("iterations must be an integer")
+        if iterations < 1:
+            raise ValueError("iterations must be at least 1")
+
+        # prf must be callable
+        if not callable(prf):
+            raise TypeError("prf must be callable")
+
+        self.__passphrase = passphrase
+        self.__salt = salt
+        self.__iterations = iterations
+        self.__prf = prf
+        self.__blockNum = 0
+        self.__buf = b("")
+        self.closed = False
+
+    def close(self):
+        """Close the stream."""
+        if not self.closed:
+            del self.__passphrase
+            del self.__salt
+            del self.__iterations
+            del self.__prf
+            del self.__blockNum
+            del self.__buf
+            self.closed = True
+
+def crypt(word, salt=None, iterations=None):
+    """PBKDF2-based unix crypt(3) replacement.
+
+    The number of iterations specified in the salt overrides the 'iterations'
+    parameter.
+
+    The effective hash length is 192 bits.
+    """
+
+    # Generate a (pseudo-)random salt if the user hasn't provided one.
+    if salt is None:
+        salt = _makesalt()
+
+    # salt must be a string or the us-ascii subset of unicode
+    if isunicode(salt):
+        salt = salt.encode('us-ascii').decode('us-ascii')
+    elif isbytes(salt):
+        salt = salt.decode('us-ascii')
+    else:
+        raise TypeError("salt must be a string")
+
+    # word must be a string or unicode (in the latter case, we convert to UTF-8)
+    if isunicode(word):
+        word = word.encode("UTF-8")
+    elif not isbytes(word):
+        raise TypeError("word must be a string or unicode")
+
+    # Try to extract the real salt and iteration count from the salt
+    if salt.startswith("$p5k2$"):
+        (iterations, salt, dummy) = salt.split("$")[2:5]
+        if iterations == "":
+            iterations = 400
+        else:
+            converted = int(iterations, 16)
+            if iterations != "%x" % converted:  # lowercase hex, minimum digits
+                raise ValueError("Invalid salt")
+            iterations = converted
+            if not (iterations >= 1):
+                raise ValueError("Invalid salt")
+
+    # Make sure the salt matches the allowed character set
+    allowed = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789./"
+    for ch in salt:
+        if ch not in allowed:
+            raise ValueError("Illegal character %r in salt" % (ch,))
+
+    if iterations is None or iterations == 400:
+        iterations = 400
+        salt = "$p5k2$$" + salt
+    else:
+        salt = "$p5k2$%x$%s" % (iterations, salt)
+    rawhash = PBKDF2(word, salt, iterations).read(24)
+    return salt + "$" + b64encode(rawhash, "./")
+
+# Add crypt as a static method of the PBKDF2 class
+# This makes it easier to do "from PBKDF2 import PBKDF2" and still use
+# crypt.
+PBKDF2.crypt = staticmethod(crypt)
+
+def _makesalt():
+    """Return a 48-bit pseudorandom salt for crypt().
+
+    This function is not suitable for generating cryptographic secrets.
+    """
+    binarysalt = b("").join([pack("@H", randint(0, 0xffff)) for i in range(3)])
+    return b64encode(binarysalt, "./")
+
+# vim:set ts=4 sw=4 sts=4 expandtab:

Contents/Libraries/Shared/provider_test.sh

@@ -43,6 +43,8 @@ python -c "import logging; logging.basicConfig(level=logging.DEBUG); logging.get
 # subscenter:list
 python -c "import logging; logging.basicConfig(level=logging.DEBUG); logging.getLogger('rebulk').setLevel(logging.WARNING); import subliminal_patch, subliminal; subliminal.region.configure('dogpile.cache.memory'); from subliminal_patch.core import SZProviderPool; from babelfish import Language; from subliminal.core import scan_video; print SZProviderPool(providers=['subscenter'], )['subscenter'].list_subtitles(scan_video('FULL_PATH'), languages=[Language('heb')])"
 
+# subscene:list
+python -c "import logging; logging.basicConfig(level=logging.DEBUG); logging.getLogger('rebulk').setLevel(logging.WARNING); import subliminal_patch, subliminal; subliminal.region.configure('dogpile.cache.memory'); from subliminal_patch.core import SZProviderPool; from subzero.language import Language; from subzero.video import parse_video; SZProviderPool(providers=['subscene'], provider_configs={'subscene': {'username': 'USERNAME', 'password': 'PASSWORD'}})['subscene'].list_subtitles(parse_video('FILENAME', {}, {'type': 'episode'}, dry_run=True), languages=[Language('eng')])"
 
 # refining
 python -c "import logging; logging.basicConfig(level=logging.DEBUG); logging.getLogger('rebulk').setLevel(logging.WARNING); import os; os.environ['U1pfT01EQl9LRVk'] = '789CF30DAC2C8B0AF433F5C9AD34290A712DF30D7135F12D0FB3E502006FDE081E'; import subliminal_patch, subliminal; subliminal.region.configure('dogpile.cache.memory'); from subzero.video import parse_video, refine_video; video = parse_video('FILE_NAME', {'type': 'episode'}, dry_run=True); print refine_video(video)"

Contents/Libraries/Shared/pyaes/__init__.py

@@ -0,0 +1,53 @@
+# The MIT License (MIT)
+#
+# Copyright (c) 2014 Richard Moore
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+# THE SOFTWARE.
+
+# This is a pure-Python implementation of the AES algorithm and AES common
+# modes of operation.
+
+# See: https://en.wikipedia.org/wiki/Advanced_Encryption_Standard
+# See: https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation
+
+
+# Supported key sizes:
+#   128-bit
+#   192-bit
+#   256-bit
+
+
+# Supported modes of operation:
+#   ECB - Electronic Codebook
+#   CBC - Cipher-Block Chaining
+#   CFB - Cipher Feedback
+#   OFB - Output Feedback
+#   CTR - Counter
+
+# See the README.md for API details and general information.
+
+# Also useful, PyCrypto, a crypto library implemented in C with Python bindings:
+# https://www.dlitz.net/software/pycrypto/
+
+
+VERSION = [1, 3, 0]
+
+from .aes import AES, AESModeOfOperationCTR, AESModeOfOperationCBC, AESModeOfOperationCFB, AESModeOfOperationECB, AESModeOfOperationOFB, AESModesOfOperation, Counter
+from .blockfeeder import decrypt_stream, Decrypter, encrypt_stream, Encrypter
+from .blockfeeder import PADDING_NONE, PADDING_DEFAULT

Contents/Libraries/Shared/pyaes/aes.py

@@ -0,0 +1,589 @@
+# The MIT License (MIT)
+#
+# Copyright (c) 2014 Richard Moore
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+# THE SOFTWARE.
+
+# This is a pure-Python implementation of the AES algorithm and AES common
+# modes of operation.
+
+# See: https://en.wikipedia.org/wiki/Advanced_Encryption_Standard
+
+# Honestly, the best description of the modes of operations are the wonderful
+# diagrams on Wikipedia. They explain in moments what my words could never
+# achieve. Hence the inline documentation here is sparer than I'd prefer.
+# See: https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation
+
+# Also useful, PyCrypto, a crypto library implemented in C with Python bindings:
+# https://www.dlitz.net/software/pycrypto/
+
+
+# Supported key sizes:
+#   128-bit
+#   192-bit
+#   256-bit
+
+
+# Supported modes of operation:
+#   ECB - Electronic Codebook
+#   CBC - Cipher-Block Chaining
+#   CFB - Cipher Feedback
+#   OFB - Output Feedback
+#   CTR - Counter
+
+
+# See the README.md for API details and general information.
+
+
+import copy
+import struct
+
+__all__ = ["AES", "AESModeOfOperationCTR", "AESModeOfOperationCBC", "AESModeOfOperationCFB",
+           "AESModeOfOperationECB", "AESModeOfOperationOFB", "AESModesOfOperation", "Counter"]
+
+
+def _compact_word(word):
+    return (word[0] << 24) | (word[1] << 16) | (word[2] << 8) | word[3]
+
+def _string_to_bytes(text):
+    return list(ord(c) for c in text)
+
+def _bytes_to_string(binary):
+    return "".join(chr(b) for b in binary)
+
+def _concat_list(a, b):
+    return a + b
+
+
+# Python 3 compatibility
+try:
+    xrange
+except Exception:
+    xrange = range
+
+    # Python 3 supports bytes, which is already an array of integers
+    def _string_to_bytes(text):
+        if isinstance(text, bytes):
+            return text
+        return [ord(c) for c in text]
+
+    # In Python 3, we return bytes
+    def _bytes_to_string(binary):
+        return bytes(binary)
+
+    # Python 3 cannot concatenate a list onto a bytes, so we bytes-ify it first
+    def _concat_list(a, b):
+        return a + bytes(b)
+
+
+# Based *largely* on the Rijndael implementation
+# See: http://csrc.nist.gov/publications/fips/fips197/fips-197.pdf
+class AES(object):
+    '''Encapsulates the AES block cipher.
+
+    You generally should not need this. Use the AESModeOfOperation classes
+    below instead.'''
+
+    # Number of rounds by keysize
+    number_of_rounds = {16: 10, 24: 12, 32: 14}
+
+    # Round constant words
+    rcon = [ 0x01, 0x02, 0x04, 0x08, 0x10, 0x20, 0x40, 0x80, 0x1b, 0x36, 0x6c, 0xd8, 0xab, 0x4d, 0x9a, 0x2f, 0x5e, 0xbc, 0x63, 0xc6, 0x97, 0x35, 0x6a, 0xd4, 0xb3, 0x7d, 0xfa, 0xef, 0xc5, 0x91 ]
+
+    # S-box and Inverse S-box (S is for Substitution)
+    S = [ 0x63, 0x7c, 0x77, 0x7b, 0xf2, 0x6b, 0x6f, 0xc5, 0x30, 0x01, 0x67, 0x2b, 0xfe, 0xd7, 0xab, 0x76, 0xca, 0x82, 0xc9, 0x7d, 0xfa, 0x59, 0x47, 0xf0, 0xad, 0xd4, 0xa2, 0xaf, 0x9c, 0xa4, 0x72, 0xc0, 0xb7, 0xfd, 0x93, 0x26, 0x36, 0x3f, 0xf7, 0xcc, 0x34, 0xa5, 0xe5, 0xf1, 0x71, 0xd8, 0x31, 0x15, 0x04, 0xc7, 0x23, 0xc3, 0x18, 0x96, 0x05, 0x9a, 0x07, 0x12, 0x80, 0xe2, 0xeb, 0x27, 0xb2, 0x75, 0x09, 0x83, 0x2c, 0x1a, 0x1b, 0x6e, 0x5a, 0xa0, 0x52, 0x3b, 0xd6, 0xb3, 0x29, 0xe3, 0x2f, 0x84, 0x53, 0xd1, 0x00, 0xed, 0x20, 0xfc, 0xb1, 0x5b, 0x6a, 0xcb, 0xbe, 0x39, 0x4a, 0x4c, 0x58, 0xcf, 0xd0, 0xef, 0xaa, 0xfb, 0x43, 0x4d, 0x33, 0x85, 0x45, 0xf9, 0x02, 0x7f, 0x50, 0x3c, 0x9f, 0xa8, 0x51, 0xa3, 0x40, 0x8f, 0x92, 0x9d, 0x38, 0xf5, 0xbc, 0xb6, 0xda, 0x21, 0x10, 0xff, 0xf3, 0xd2, 0xcd, 0x0c, 0x13, 0xec, 0x5f, 0x97, 0x44, 0x17, 0xc4, 0xa7, 0x7e, 0x3d, 0x64, 0x5d, 0x19, 0x73, 0x60, 0x81, 0x4f, 0xdc, 0x22, 0x2a, 0x90, 0x88, 0x46, 0xee, 0xb8, 0x14, 0xde, 0x5e, 0x0b, 0xdb, 0xe0, 0x32, 0x3a, 0x0a, 0x49, 0x06, 0x24, 0x5c, 0xc2, 0xd3, 0xac, 0x62, 0x91, 0x95, 0xe4, 0x79, 0xe7, 0xc8, 0x37, 0x6d, 0x8d, 0xd5, 0x4e, 0xa9, 0x6c, 0x56, 0xf4, 0xea, 0x65, 0x7a, 0xae, 0x08, 0xba, 0x78, 0x25, 0x2e, 0x1c, 0xa6, 0xb4, 0xc6, 0xe8, 0xdd, 0x74, 0x1f, 0x4b, 0xbd, 0x8b, 0x8a, 0x70, 0x3e, 0xb5, 0x66, 0x48, 0x03, 0xf6, 0x0e, 0x61, 0x35, 0x57, 0xb9, 0x86, 0xc1, 0x1d, 0x9e, 0xe1, 0xf8, 0x98, 0x11, 0x69, 0xd9, 0x8e, 0x94, 0x9b, 0x1e, 0x87, 0xe9, 0xce, 0x55, 0x28, 0xdf, 0x8c, 0xa1, 0x89, 0x0d, 0xbf, 0xe6, 0x42, 0x68, 0x41, 0x99, 0x2d, 0x0f, 0xb0, 0x54, 0xbb, 0x16 ]
+    Si =[ 0x52, 0x09, 0x6a, 0xd5, 0x30, 0x36, 0xa5, 0x38, 0xbf, 0x40, 0xa3, 0x9e, 0x81, 0xf3, 0xd7, 0xfb, 0x7c, 0xe3, 0x39, 0x82, 0x9b, 0x2f, 0xff, 0x87, 0x34, 0x8e, 0x43, 0x44, 0xc4, 0xde, 0xe9, 0xcb, 0x54, 0x7b, 0x94, 0x32, 0xa6, 0xc2, 0x23, 0x3d, 0xee, 0x4c, 0x95, 0x0b, 0x42, 0xfa, 0xc3, 0x4e, 0x08, 0x2e, 0xa1, 0x66, 0x28, 0xd9, 0x24, 0xb2, 0x76, 0x5b, 0xa2, 0x49, 0x6d, 0x8b, 0xd1, 0x25, 0x72, 0xf8, 0xf6, 0x64, 0x86, 0x68, 0x98, 0x16, 0xd4, 0xa4, 0x5c, 0xcc, 0x5d, 0x65, 0xb6, 0x92, 0x6c, 0x70, 0x48, 0x50, 0xfd, 0xed, 0xb9, 0xda, 0x5e, 0x15, 0x46, 0x57, 0xa7, 0x8d, 0x9d, 0x84, 0x90, 0xd8, 0xab, 0x00, 0x8c, 0xbc, 0xd3, 0x0a, 0xf7, 0xe4, 0x58, 0x05, 0xb8, 0xb3, 0x45, 0x06, 0xd0, 0x2c, 0x1e, 0x8f, 0xca, 0x3f, 0x0f, 0x02, 0xc1, 0xaf, 0xbd, 0x03, 0x01, 0x13, 0x8a, 0x6b, 0x3a, 0x91, 0x11, 0x41, 0x4f, 0x67, 0xdc, 0xea, 0x97, 0xf2, 0xcf, 0xce, 0xf0, 0xb4, 0xe6, 0x73, 0x96, 0xac, 0x74, 0x22, 0xe7, 0xad, 0x35, 0x85, 0xe2, 0xf9, 0x37, 0xe8, 0x1c, 0x75, 0xdf, 0x6e, 0x47, 0xf1, 0x1a, 0x71, 0x1d, 0x29, 0xc5, 0x89, 0x6f, 0xb7, 0x62, 0x0e, 0xaa, 0x18, 0xbe, 0x1b, 0xfc, 0x56, 0x3e, 0x4b, 0xc6, 0xd2, 0x79, 0x20, 0x9a, 0xdb, 0xc0, 0xfe, 0x78, 0xcd, 0x5a, 0xf4, 0x1f, 0xdd, 0xa8, 0x33, 0x88, 0x07, 0xc7, 0x31, 0xb1, 0x12, 0x10, 0x59, 0x27, 0x80, 0xec, 0x5f, 0x60, 0x51, 0x7f, 0xa9, 0x19, 0xb5, 0x4a, 0x0d, 0x2d, 0xe5, 0x7a, 0x9f, 0x93, 0xc9, 0x9c, 0xef, 0xa0, 0xe0, 0x3b, 0x4d, 0xae, 0x2a, 0xf5, 0xb0, 0xc8, 0xeb, 0xbb, 0x3c, 0x83, 0x53, 0x99, 0x61, 0x17, 0x2b, 0x04, 0x7e, 0xba, 0x77, 0xd6, 0x26, 0xe1, 0x69, 0x14, 0x63, 0x55, 0x21, 0x0c, 0x7d ] 
+
+    # Transformations for encryption
+    T1 = [ 0xc66363a5, 0xf87c7c84, 0xee777799, 0xf67b7b8d, 0xfff2f20d, 0xd66b6bbd, 0xde6f6fb1, 0x91c5c554, 0x60303050, 0x02010103, 0xce6767a9, 0x562b2b7d, 0xe7fefe19, 0xb5d7d762, 0x4dababe6, 0xec76769a, 0x8fcaca45, 0x1f82829d, 0x89c9c940, 0xfa7d7d87, 0xeffafa15, 0xb25959eb, 0x8e4747c9, 0xfbf0f00b, 0x41adadec, 0xb3d4d467, 0x5fa2a2fd, 0x45afafea, 0x239c9cbf, 0x53a4a4f7, 0xe4727296, 0x9bc0c05b, 0x75b7b7c2, 0xe1fdfd1c, 0x3d9393ae, 0x4c26266a, 0x6c36365a, 0x7e3f3f41, 0xf5f7f702, 0x83cccc4f, 0x6834345c, 0x51a5a5f4, 0xd1e5e534, 0xf9f1f108, 0xe2717193, 0xabd8d873, 0x62313153, 0x2a15153f, 0x0804040c, 0x95c7c752, 0x46232365, 0x9dc3c35e, 0x30181828, 0x379696a1, 0x0a05050f, 0x2f9a9ab5, 0x0e070709, 0x24121236, 0x1b80809b, 0xdfe2e23d, 0xcdebeb26, 0x4e272769, 0x7fb2b2cd, 0xea75759f, 0x1209091b, 0x1d83839e, 0x582c2c74, 0x341a1a2e, 0x361b1b2d, 0xdc6e6eb2, 0xb45a5aee, 0x5ba0a0fb, 0xa45252f6, 0x763b3b4d, 0xb7d6d661, 0x7db3b3ce, 0x5229297b, 0xdde3e33e, 0x5e2f2f71, 0x13848497, 0xa65353f5, 0xb9d1d168, 0x00000000, 0xc1eded2c, 0x40202060, 0xe3fcfc1f, 0x79b1b1c8, 0xb65b5bed, 0xd46a6abe, 0x8dcbcb46, 0x67bebed9, 0x7239394b, 0x944a4ade, 0x984c4cd4, 0xb05858e8, 0x85cfcf4a, 0xbbd0d06b, 0xc5efef2a, 0x4faaaae5, 0xedfbfb16, 0x864343c5, 0x9a4d4dd7, 0x66333355, 0x11858594, 0x8a4545cf, 0xe9f9f910, 0x04020206, 0xfe7f7f81, 0xa05050f0, 0x783c3c44, 0x259f9fba, 0x4ba8a8e3, 0xa25151f3, 0x5da3a3fe, 0x804040c0, 0x058f8f8a, 0x3f9292ad, 0x219d9dbc, 0x70383848, 0xf1f5f504, 0x63bcbcdf, 0x77b6b6c1, 0xafdada75, 0x42212163, 0x20101030, 0xe5ffff1a, 0xfdf3f30e, 0xbfd2d26d, 0x81cdcd4c, 0x180c0c14, 0x26131335, 0xc3ecec2f, 0xbe5f5fe1, 0x359797a2, 0x884444cc, 0x2e171739, 0x93c4c457, 0x55a7a7f2, 0xfc7e7e82, 0x7a3d3d47, 0xc86464ac, 0xba5d5de7, 0x3219192b, 0xe6737395, 0xc06060a0, 0x19818198, 0x9e4f4fd1, 0xa3dcdc7f, 0x44222266, 0x542a2a7e, 0x3b9090ab, 0x0b888883, 0x8c4646ca, 0xc7eeee29, 0x6bb8b8d3, 0x2814143c, 0xa7dede79, 0xbc5e5ee2, 0x160b0b1d, 0xaddbdb76, 0xdbe0e03b, 0x64323256, 0x743a3a4e, 0x140a0a1e, 0x924949db, 0x0c06060a, 0x4824246c, 0xb85c5ce4, 0x9fc2c25d, 0xbdd3d36e, 0x43acacef, 0xc46262a6, 0x399191a8, 0x319595a4, 0xd3e4e437, 0xf279798b, 0xd5e7e732, 0x8bc8c843, 0x6e373759, 0xda6d6db7, 0x018d8d8c, 0xb1d5d564, 0x9c4e4ed2, 0x49a9a9e0, 0xd86c6cb4, 0xac5656fa, 0xf3f4f407, 0xcfeaea25, 0xca6565af, 0xf47a7a8e, 0x47aeaee9, 0x10080818, 0x6fbabad5, 0xf0787888, 0x4a25256f, 0x5c2e2e72, 0x381c1c24, 0x57a6a6f1, 0x73b4b4c7, 0x97c6c651, 0xcbe8e823, 0xa1dddd7c, 0xe874749c, 0x3e1f1f21, 0x964b4bdd, 0x61bdbddc, 0x0d8b8b86, 0x0f8a8a85, 0xe0707090, 0x7c3e3e42, 0x71b5b5c4, 0xcc6666aa, 0x904848d8, 0x06030305, 0xf7f6f601, 0x1c0e0e12, 0xc26161a3, 0x6a35355f, 0xae5757f9, 0x69b9b9d0, 0x17868691, 0x99c1c158, 0x3a1d1d27, 0x279e9eb9, 0xd9e1e138, 0xebf8f813, 0x2b9898b3, 0x22111133, 0xd26969bb, 0xa9d9d970, 0x078e8e89, 0x339494a7, 0x2d9b9bb6, 0x3c1e1e22, 0x15878792, 0xc9e9e920, 0x87cece49, 0xaa5555ff, 0x50282878, 0xa5dfdf7a, 0x038c8c8f, 0x59a1a1f8, 0x09898980, 0x1a0d0d17, 0x65bfbfda, 0xd7e6e631, 0x844242c6, 0xd06868b8, 0x824141c3, 0x299999b0, 0x5a2d2d77, 0x1e0f0f11, 0x7bb0b0cb, 0xa85454fc, 0x6dbbbbd6, 0x2c16163a ]
+    T2 = [ 0xa5c66363, 0x84f87c7c, 0x99ee7777, 0x8df67b7b, 0x0dfff2f2, 0xbdd66b6b, 0xb1de6f6f, 0x5491c5c5, 0x50603030, 0x03020101, 0xa9ce6767, 0x7d562b2b, 0x19e7fefe, 0x62b5d7d7, 0xe64dabab, 0x9aec7676, 0x458fcaca, 0x9d1f8282, 0x4089c9c9, 0x87fa7d7d, 0x15effafa, 0xebb25959, 0xc98e4747, 0x0bfbf0f0, 0xec41adad, 0x67b3d4d4, 0xfd5fa2a2, 0xea45afaf, 0xbf239c9c, 0xf753a4a4, 0x96e47272, 0x5b9bc0c0, 0xc275b7b7, 0x1ce1fdfd, 0xae3d9393, 0x6a4c2626, 0x5a6c3636, 0x417e3f3f, 0x02f5f7f7, 0x4f83cccc, 0x5c683434, 0xf451a5a5, 0x34d1e5e5, 0x08f9f1f1, 0x93e27171, 0x73abd8d8, 0x53623131, 0x3f2a1515, 0x0c080404, 0x5295c7c7, 0x65462323, 0x5e9dc3c3, 0x28301818, 0xa1379696, 0x0f0a0505, 0xb52f9a9a, 0x090e0707, 0x36241212, 0x9b1b8080, 0x3ddfe2e2, 0x26cdebeb, 0x694e2727, 0xcd7fb2b2, 0x9fea7575, 0x1b120909, 0x9e1d8383, 0x74582c2c, 0x2e341a1a, 0x2d361b1b, 0xb2dc6e6e, 0xeeb45a5a, 0xfb5ba0a0, 0xf6a45252, 0x4d763b3b, 0x61b7d6d6, 0xce7db3b3, 0x7b522929, 0x3edde3e3, 0x715e2f2f, 0x97138484, 0xf5a65353, 0x68b9d1d1, 0x00000000, 0x2cc1eded, 0x60402020, 0x1fe3fcfc, 0xc879b1b1, 0xedb65b5b, 0xbed46a6a, 0x468dcbcb, 0xd967bebe, 0x4b723939, 0xde944a4a, 0xd4984c4c, 0xe8b05858, 0x4a85cfcf, 0x6bbbd0d0, 0x2ac5efef, 0xe54faaaa, 0x16edfbfb, 0xc5864343, 0xd79a4d4d, 0x55663333, 0x94118585, 0xcf8a4545, 0x10e9f9f9, 0x06040202, 0x81fe7f7f, 0xf0a05050, 0x44783c3c, 0xba259f9f, 0xe34ba8a8, 0xf3a25151, 0xfe5da3a3, 0xc0804040, 0x8a058f8f, 0xad3f9292, 0xbc219d9d, 0x48703838, 0x04f1f5f5, 0xdf63bcbc, 0xc177b6b6, 0x75afdada, 0x63422121, 0x30201010, 0x1ae5ffff, 0x0efdf3f3, 0x6dbfd2d2, 0x4c81cdcd, 0x14180c0c, 0x35261313, 0x2fc3ecec, 0xe1be5f5f, 0xa2359797, 0xcc884444, 0x392e1717, 0x5793c4c4, 0xf255a7a7, 0x82fc7e7e, 0x477a3d3d, 0xacc86464, 0xe7ba5d5d, 0x2b321919, 0x95e67373, 0xa0c06060, 0x98198181, 0xd19e4f4f, 0x7fa3dcdc, 0x66442222, 0x7e542a2a, 0xab3b9090, 0x830b8888, 0xca8c4646, 0x29c7eeee, 0xd36bb8b8, 0x3c281414, 0x79a7dede, 0xe2bc5e5e, 0x1d160b0b, 0x76addbdb, 0x3bdbe0e0, 0x56643232, 0x4e743a3a, 0x1e140a0a, 0xdb924949, 0x0a0c0606, 0x6c482424, 0xe4b85c5c, 0x5d9fc2c2, 0x6ebdd3d3, 0xef43acac, 0xa6c46262, 0xa8399191, 0xa4319595, 0x37d3e4e4, 0x8bf27979, 0x32d5e7e7, 0x438bc8c8, 0x596e3737, 0xb7da6d6d, 0x8c018d8d, 0x64b1d5d5, 0xd29c4e4e, 0xe049a9a9, 0xb4d86c6c, 0xfaac5656, 0x07f3f4f4, 0x25cfeaea, 0xafca6565, 0x8ef47a7a, 0xe947aeae, 0x18100808, 0xd56fbaba, 0x88f07878, 0x6f4a2525, 0x725c2e2e, 0x24381c1c, 0xf157a6a6, 0xc773b4b4, 0x5197c6c6, 0x23cbe8e8, 0x7ca1dddd, 0x9ce87474, 0x213e1f1f, 0xdd964b4b, 0xdc61bdbd, 0x860d8b8b, 0x850f8a8a, 0x90e07070, 0x427c3e3e, 0xc471b5b5, 0xaacc6666, 0xd8904848, 0x05060303, 0x01f7f6f6, 0x121c0e0e, 0xa3c26161, 0x5f6a3535, 0xf9ae5757, 0xd069b9b9, 0x91178686, 0x5899c1c1, 0x273a1d1d, 0xb9279e9e, 0x38d9e1e1, 0x13ebf8f8, 0xb32b9898, 0x33221111, 0xbbd26969, 0x70a9d9d9, 0x89078e8e, 0xa7339494, 0xb62d9b9b, 0x223c1e1e, 0x92158787, 0x20c9e9e9, 0x4987cece, 0xffaa5555, 0x78502828, 0x7aa5dfdf, 0x8f038c8c, 0xf859a1a1, 0x80098989, 0x171a0d0d, 0xda65bfbf, 0x31d7e6e6, 0xc6844242, 0xb8d06868, 0xc3824141, 0xb0299999, 0x775a2d2d, 0x111e0f0f, 0xcb7bb0b0, 0xfca85454, 0xd66dbbbb, 0x3a2c1616 ]
+    T3 = [ 0x63a5c663, 0x7c84f87c, 0x7799ee77, 0x7b8df67b, 0xf20dfff2, 0x6bbdd66b, 0x6fb1de6f, 0xc55491c5, 0x30506030, 0x01030201, 0x67a9ce67, 0x2b7d562b, 0xfe19e7fe, 0xd762b5d7, 0xabe64dab, 0x769aec76, 0xca458fca, 0x829d1f82, 0xc94089c9, 0x7d87fa7d, 0xfa15effa, 0x59ebb259, 0x47c98e47, 0xf00bfbf0, 0xadec41ad, 0xd467b3d4, 0xa2fd5fa2, 0xafea45af, 0x9cbf239c, 0xa4f753a4, 0x7296e472, 0xc05b9bc0, 0xb7c275b7, 0xfd1ce1fd, 0x93ae3d93, 0x266a4c26, 0x365a6c36, 0x3f417e3f, 0xf702f5f7, 0xcc4f83cc, 0x345c6834, 0xa5f451a5, 0xe534d1e5, 0xf108f9f1, 0x7193e271, 0xd873abd8, 0x31536231, 0x153f2a15, 0x040c0804, 0xc75295c7, 0x23654623, 0xc35e9dc3, 0x18283018, 0x96a13796, 0x050f0a05, 0x9ab52f9a, 0x07090e07, 0x12362412, 0x809b1b80, 0xe23ddfe2, 0xeb26cdeb, 0x27694e27, 0xb2cd7fb2, 0x759fea75, 0x091b1209, 0x839e1d83, 0x2c74582c, 0x1a2e341a, 0x1b2d361b, 0x6eb2dc6e, 0x5aeeb45a, 0xa0fb5ba0, 0x52f6a452, 0x3b4d763b, 0xd661b7d6, 0xb3ce7db3, 0x297b5229, 0xe33edde3, 0x2f715e2f, 0x84971384, 0x53f5a653, 0xd168b9d1, 0x00000000, 0xed2cc1ed, 0x20604020, 0xfc1fe3fc, 0xb1c879b1, 0x5bedb65b, 0x6abed46a, 0xcb468dcb, 0xbed967be, 0x394b7239, 0x4ade944a, 0x4cd4984c, 0x58e8b058, 0xcf4a85cf, 0xd06bbbd0, 0xef2ac5ef, 0xaae54faa, 0xfb16edfb, 0x43c58643, 0x4dd79a4d, 0x33556633, 0x85941185, 0x45cf8a45, 0xf910e9f9, 0x02060402, 0x7f81fe7f, 0x50f0a050, 0x3c44783c, 0x9fba259f, 0xa8e34ba8, 0x51f3a251, 0xa3fe5da3, 0x40c08040, 0x8f8a058f, 0x92ad3f92, 0x9dbc219d, 0x38487038, 0xf504f1f5, 0xbcdf63bc, 0xb6c177b6, 0xda75afda, 0x21634221, 0x10302010, 0xff1ae5ff, 0xf30efdf3, 0xd26dbfd2, 0xcd4c81cd, 0x0c14180c, 0x13352613, 0xec2fc3ec, 0x5fe1be5f, 0x97a23597, 0x44cc8844, 0x17392e17, 0xc45793c4, 0xa7f255a7, 0x7e82fc7e, 0x3d477a3d, 0x64acc864, 0x5de7ba5d, 0x192b3219, 0x7395e673, 0x60a0c060, 0x81981981, 0x4fd19e4f, 0xdc7fa3dc, 0x22664422, 0x2a7e542a, 0x90ab3b90, 0x88830b88, 0x46ca8c46, 0xee29c7ee, 0xb8d36bb8, 0x143c2814, 0xde79a7de, 0x5ee2bc5e, 0x0b1d160b, 0xdb76addb, 0xe03bdbe0, 0x32566432, 0x3a4e743a, 0x0a1e140a, 0x49db9249, 0x060a0c06, 0x246c4824, 0x5ce4b85c, 0xc25d9fc2, 0xd36ebdd3, 0xacef43ac, 0x62a6c462, 0x91a83991, 0x95a43195, 0xe437d3e4, 0x798bf279, 0xe732d5e7, 0xc8438bc8, 0x37596e37, 0x6db7da6d, 0x8d8c018d, 0xd564b1d5, 0x4ed29c4e, 0xa9e049a9, 0x6cb4d86c, 0x56faac56, 0xf407f3f4, 0xea25cfea, 0x65afca65, 0x7a8ef47a, 0xaee947ae, 0x08181008, 0xbad56fba, 0x7888f078, 0x256f4a25, 0x2e725c2e, 0x1c24381c, 0xa6f157a6, 0xb4c773b4, 0xc65197c6, 0xe823cbe8, 0xdd7ca1dd, 0x749ce874, 0x1f213e1f, 0x4bdd964b, 0xbddc61bd, 0x8b860d8b, 0x8a850f8a, 0x7090e070, 0x3e427c3e, 0xb5c471b5, 0x66aacc66, 0x48d89048, 0x03050603, 0xf601f7f6, 0x0e121c0e, 0x61a3c261, 0x355f6a35, 0x57f9ae57, 0xb9d069b9, 0x86911786, 0xc15899c1, 0x1d273a1d, 0x9eb9279e, 0xe138d9e1, 0xf813ebf8, 0x98b32b98, 0x11332211, 0x69bbd269, 0xd970a9d9, 0x8e89078e, 0x94a73394, 0x9bb62d9b, 0x1e223c1e, 0x87921587, 0xe920c9e9, 0xce4987ce, 0x55ffaa55, 0x28785028, 0xdf7aa5df, 0x8c8f038c, 0xa1f859a1, 0x89800989, 0x0d171a0d, 0xbfda65bf, 0xe631d7e6, 0x42c68442, 0x68b8d068, 0x41c38241, 0x99b02999, 0x2d775a2d, 0x0f111e0f, 0xb0cb7bb0, 0x54fca854, 0xbbd66dbb, 0x163a2c16 ]
+    T4 = [ 0x6363a5c6, 0x7c7c84f8, 0x777799ee, 0x7b7b8df6, 0xf2f20dff, 0x6b6bbdd6, 0x6f6fb1de, 0xc5c55491, 0x30305060, 0x01010302, 0x6767a9ce, 0x2b2b7d56, 0xfefe19e7, 0xd7d762b5, 0xababe64d, 0x76769aec, 0xcaca458f, 0x82829d1f, 0xc9c94089, 0x7d7d87fa, 0xfafa15ef, 0x5959ebb2, 0x4747c98e, 0xf0f00bfb, 0xadadec41, 0xd4d467b3, 0xa2a2fd5f, 0xafafea45, 0x9c9cbf23, 0xa4a4f753, 0x727296e4, 0xc0c05b9b, 0xb7b7c275, 0xfdfd1ce1, 0x9393ae3d, 0x26266a4c, 0x36365a6c, 0x3f3f417e, 0xf7f702f5, 0xcccc4f83, 0x34345c68, 0xa5a5f451, 0xe5e534d1, 0xf1f108f9, 0x717193e2, 0xd8d873ab, 0x31315362, 0x15153f2a, 0x04040c08, 0xc7c75295, 0x23236546, 0xc3c35e9d, 0x18182830, 0x9696a137, 0x05050f0a, 0x9a9ab52f, 0x0707090e, 0x12123624, 0x80809b1b, 0xe2e23ddf, 0xebeb26cd, 0x2727694e, 0xb2b2cd7f, 0x75759fea, 0x09091b12, 0x83839e1d, 0x2c2c7458, 0x1a1a2e34, 0x1b1b2d36, 0x6e6eb2dc, 0x5a5aeeb4, 0xa0a0fb5b, 0x5252f6a4, 0x3b3b4d76, 0xd6d661b7, 0xb3b3ce7d, 0x29297b52, 0xe3e33edd, 0x2f2f715e, 0x84849713, 0x5353f5a6, 0xd1d168b9, 0x00000000, 0xeded2cc1, 0x20206040, 0xfcfc1fe3, 0xb1b1c879, 0x5b5bedb6, 0x6a6abed4, 0xcbcb468d, 0xbebed967, 0x39394b72, 0x4a4ade94, 0x4c4cd498, 0x5858e8b0, 0xcfcf4a85, 0xd0d06bbb, 0xefef2ac5, 0xaaaae54f, 0xfbfb16ed, 0x4343c586, 0x4d4dd79a, 0x33335566, 0x85859411, 0x4545cf8a, 0xf9f910e9, 0x02020604, 0x7f7f81fe, 0x5050f0a0, 0x3c3c4478, 0x9f9fba25, 0xa8a8e34b, 0x5151f3a2, 0xa3a3fe5d, 0x4040c080, 0x8f8f8a05, 0x9292ad3f, 0x9d9dbc21, 0x38384870, 0xf5f504f1, 0xbcbcdf63, 0xb6b6c177, 0xdada75af, 0x21216342, 0x10103020, 0xffff1ae5, 0xf3f30efd, 0xd2d26dbf, 0xcdcd4c81, 0x0c0c1418, 0x13133526, 0xecec2fc3, 0x5f5fe1be, 0x9797a235, 0x4444cc88, 0x1717392e, 0xc4c45793, 0xa7a7f255, 0x7e7e82fc, 0x3d3d477a, 0x6464acc8, 0x5d5de7ba, 0x19192b32, 0x737395e6, 0x6060a0c0, 0x81819819, 0x4f4fd19e, 0xdcdc7fa3, 0x22226644, 0x2a2a7e54, 0x9090ab3b, 0x8888830b, 0x4646ca8c, 0xeeee29c7, 0xb8b8d36b, 0x14143c28, 0xdede79a7, 0x5e5ee2bc, 0x0b0b1d16, 0xdbdb76ad, 0xe0e03bdb, 0x32325664, 0x3a3a4e74, 0x0a0a1e14, 0x4949db92, 0x06060a0c, 0x24246c48, 0x5c5ce4b8, 0xc2c25d9f, 0xd3d36ebd, 0xacacef43, 0x6262a6c4, 0x9191a839, 0x9595a431, 0xe4e437d3, 0x79798bf2, 0xe7e732d5, 0xc8c8438b, 0x3737596e, 0x6d6db7da, 0x8d8d8c01, 0xd5d564b1, 0x4e4ed29c, 0xa9a9e049, 0x6c6cb4d8, 0x5656faac, 0xf4f407f3, 0xeaea25cf, 0x6565afca, 0x7a7a8ef4, 0xaeaee947, 0x08081810, 0xbabad56f, 0x787888f0, 0x25256f4a, 0x2e2e725c, 0x1c1c2438, 0xa6a6f157, 0xb4b4c773, 0xc6c65197, 0xe8e823cb, 0xdddd7ca1, 0x74749ce8, 0x1f1f213e, 0x4b4bdd96, 0xbdbddc61, 0x8b8b860d, 0x8a8a850f, 0x707090e0, 0x3e3e427c, 0xb5b5c471, 0x6666aacc, 0x4848d890, 0x03030506, 0xf6f601f7, 0x0e0e121c, 0x6161a3c2, 0x35355f6a, 0x5757f9ae, 0xb9b9d069, 0x86869117, 0xc1c15899, 0x1d1d273a, 0x9e9eb927, 0xe1e138d9, 0xf8f813eb, 0x9898b32b, 0x11113322, 0x6969bbd2, 0xd9d970a9, 0x8e8e8907, 0x9494a733, 0x9b9bb62d, 0x1e1e223c, 0x87879215, 0xe9e920c9, 0xcece4987, 0x5555ffaa, 0x28287850, 0xdfdf7aa5, 0x8c8c8f03, 0xa1a1f859, 0x89898009, 0x0d0d171a, 0xbfbfda65, 0xe6e631d7, 0x4242c684, 0x6868b8d0, 0x4141c382, 0x9999b029, 0x2d2d775a, 0x0f0f111e, 0xb0b0cb7b, 0x5454fca8, 0xbbbbd66d, 0x16163a2c ]
+
+    # Transformations for decryption
+    T5 = [ 0x51f4a750, 0x7e416553, 0x1a17a4c3, 0x3a275e96, 0x3bab6bcb, 0x1f9d45f1, 0xacfa58ab, 0x4be30393, 0x2030fa55, 0xad766df6, 0x88cc7691, 0xf5024c25, 0x4fe5d7fc, 0xc52acbd7, 0x26354480, 0xb562a38f, 0xdeb15a49, 0x25ba1b67, 0x45ea0e98, 0x5dfec0e1, 0xc32f7502, 0x814cf012, 0x8d4697a3, 0x6bd3f9c6, 0x038f5fe7, 0x15929c95, 0xbf6d7aeb, 0x955259da, 0xd4be832d, 0x587421d3, 0x49e06929, 0x8ec9c844, 0x75c2896a, 0xf48e7978, 0x99583e6b, 0x27b971dd, 0xbee14fb6, 0xf088ad17, 0xc920ac66, 0x7dce3ab4, 0x63df4a18, 0xe51a3182, 0x97513360, 0x62537f45, 0xb16477e0, 0xbb6bae84, 0xfe81a01c, 0xf9082b94, 0x70486858, 0x8f45fd19, 0x94de6c87, 0x527bf8b7, 0xab73d323, 0x724b02e2, 0xe31f8f57, 0x6655ab2a, 0xb2eb2807, 0x2fb5c203, 0x86c57b9a, 0xd33708a5, 0x302887f2, 0x23bfa5b2, 0x02036aba, 0xed16825c, 0x8acf1c2b, 0xa779b492, 0xf307f2f0, 0x4e69e2a1, 0x65daf4cd, 0x0605bed5, 0xd134621f, 0xc4a6fe8a, 0x342e539d, 0xa2f355a0, 0x058ae132, 0xa4f6eb75, 0x0b83ec39, 0x4060efaa, 0x5e719f06, 0xbd6e1051, 0x3e218af9, 0x96dd063d, 0xdd3e05ae, 0x4de6bd46, 0x91548db5, 0x71c45d05, 0x0406d46f, 0x605015ff, 0x1998fb24, 0xd6bde997, 0x894043cc, 0x67d99e77, 0xb0e842bd, 0x07898b88, 0xe7195b38, 0x79c8eedb, 0xa17c0a47, 0x7c420fe9, 0xf8841ec9, 0x00000000, 0x09808683, 0x322bed48, 0x1e1170ac, 0x6c5a724e, 0xfd0efffb, 0x0f853856, 0x3daed51e, 0x362d3927, 0x0a0fd964, 0x685ca621, 0x9b5b54d1, 0x24362e3a, 0x0c0a67b1, 0x9357e70f, 0xb4ee96d2, 0x1b9b919e, 0x80c0c54f, 0x61dc20a2, 0x5a774b69, 0x1c121a16, 0xe293ba0a, 0xc0a02ae5, 0x3c22e043, 0x121b171d, 0x0e090d0b, 0xf28bc7ad, 0x2db6a8b9, 0x141ea9c8, 0x57f11985, 0xaf75074c, 0xee99ddbb, 0xa37f60fd, 0xf701269f, 0x5c72f5bc, 0x44663bc5, 0x5bfb7e34, 0x8b432976, 0xcb23c6dc, 0xb6edfc68, 0xb8e4f163, 0xd731dcca, 0x42638510, 0x13972240, 0x84c61120, 0x854a247d, 0xd2bb3df8, 0xaef93211, 0xc729a16d, 0x1d9e2f4b, 0xdcb230f3, 0x0d8652ec, 0x77c1e3d0, 0x2bb3166c, 0xa970b999, 0x119448fa, 0x47e96422, 0xa8fc8cc4, 0xa0f03f1a, 0x567d2cd8, 0x223390ef, 0x87494ec7, 0xd938d1c1, 0x8ccaa2fe, 0x98d40b36, 0xa6f581cf, 0xa57ade28, 0xdab78e26, 0x3fadbfa4, 0x2c3a9de4, 0x5078920d, 0x6a5fcc9b, 0x547e4662, 0xf68d13c2, 0x90d8b8e8, 0x2e39f75e, 0x82c3aff5, 0x9f5d80be, 0x69d0937c, 0x6fd52da9, 0xcf2512b3, 0xc8ac993b, 0x10187da7, 0xe89c636e, 0xdb3bbb7b, 0xcd267809, 0x6e5918f4, 0xec9ab701, 0x834f9aa8, 0xe6956e65, 0xaaffe67e, 0x21bccf08, 0xef15e8e6, 0xbae79bd9, 0x4a6f36ce, 0xea9f09d4, 0x29b07cd6, 0x31a4b2af, 0x2a3f2331, 0xc6a59430, 0x35a266c0, 0x744ebc37, 0xfc82caa6, 0xe090d0b0, 0x33a7d815, 0xf104984a, 0x41ecdaf7, 0x7fcd500e, 0x1791f62f, 0x764dd68d, 0x43efb04d, 0xccaa4d54, 0xe49604df, 0x9ed1b5e3, 0x4c6a881b, 0xc12c1fb8, 0x4665517f, 0x9d5eea04, 0x018c355d, 0xfa877473, 0xfb0b412e, 0xb3671d5a, 0x92dbd252, 0xe9105633, 0x6dd64713, 0x9ad7618c, 0x37a10c7a, 0x59f8148e, 0xeb133c89, 0xcea927ee, 0xb761c935, 0xe11ce5ed, 0x7a47b13c, 0x9cd2df59, 0x55f2733f, 0x1814ce79, 0x73c737bf, 0x53f7cdea, 0x5ffdaa5b, 0xdf3d6f14, 0x7844db86, 0xcaaff381, 0xb968c43e, 0x3824342c, 0xc2a3405f, 0x161dc372, 0xbce2250c, 0x283c498b, 0xff0d9541, 0x39a80171, 0x080cb3de, 0xd8b4e49c, 0x6456c190, 0x7bcb8461, 0xd532b670, 0x486c5c74, 0xd0b85742 ]
+    T6 = [ 0x5051f4a7, 0x537e4165, 0xc31a17a4, 0x963a275e, 0xcb3bab6b, 0xf11f9d45, 0xabacfa58, 0x934be303, 0x552030fa, 0xf6ad766d, 0x9188cc76, 0x25f5024c, 0xfc4fe5d7, 0xd7c52acb, 0x80263544, 0x8fb562a3, 0x49deb15a, 0x6725ba1b, 0x9845ea0e, 0xe15dfec0, 0x02c32f75, 0x12814cf0, 0xa38d4697, 0xc66bd3f9, 0xe7038f5f, 0x9515929c, 0xebbf6d7a, 0xda955259, 0x2dd4be83, 0xd3587421, 0x2949e069, 0x448ec9c8, 0x6a75c289, 0x78f48e79, 0x6b99583e, 0xdd27b971, 0xb6bee14f, 0x17f088ad, 0x66c920ac, 0xb47dce3a, 0x1863df4a, 0x82e51a31, 0x60975133, 0x4562537f, 0xe0b16477, 0x84bb6bae, 0x1cfe81a0, 0x94f9082b, 0x58704868, 0x198f45fd, 0x8794de6c, 0xb7527bf8, 0x23ab73d3, 0xe2724b02, 0x57e31f8f, 0x2a6655ab, 0x07b2eb28, 0x032fb5c2, 0x9a86c57b, 0xa5d33708, 0xf2302887, 0xb223bfa5, 0xba02036a, 0x5ced1682, 0x2b8acf1c, 0x92a779b4, 0xf0f307f2, 0xa14e69e2, 0xcd65daf4, 0xd50605be, 0x1fd13462, 0x8ac4a6fe, 0x9d342e53, 0xa0a2f355, 0x32058ae1, 0x75a4f6eb, 0x390b83ec, 0xaa4060ef, 0x065e719f, 0x51bd6e10, 0xf93e218a, 0x3d96dd06, 0xaedd3e05, 0x464de6bd, 0xb591548d, 0x0571c45d, 0x6f0406d4, 0xff605015, 0x241998fb, 0x97d6bde9, 0xcc894043, 0x7767d99e, 0xbdb0e842, 0x8807898b, 0x38e7195b, 0xdb79c8ee, 0x47a17c0a, 0xe97c420f, 0xc9f8841e, 0x00000000, 0x83098086, 0x48322bed, 0xac1e1170, 0x4e6c5a72, 0xfbfd0eff, 0x560f8538, 0x1e3daed5, 0x27362d39, 0x640a0fd9, 0x21685ca6, 0xd19b5b54, 0x3a24362e, 0xb10c0a67, 0x0f9357e7, 0xd2b4ee96, 0x9e1b9b91, 0x4f80c0c5, 0xa261dc20, 0x695a774b, 0x161c121a, 0x0ae293ba, 0xe5c0a02a, 0x433c22e0, 0x1d121b17, 0x0b0e090d, 0xadf28bc7, 0xb92db6a8, 0xc8141ea9, 0x8557f119, 0x4caf7507, 0xbbee99dd, 0xfda37f60, 0x9ff70126, 0xbc5c72f5, 0xc544663b, 0x345bfb7e, 0x768b4329, 0xdccb23c6, 0x68b6edfc, 0x63b8e4f1, 0xcad731dc, 0x10426385, 0x40139722, 0x2084c611, 0x7d854a24, 0xf8d2bb3d, 0x11aef932, 0x6dc729a1, 0x4b1d9e2f, 0xf3dcb230, 0xec0d8652, 0xd077c1e3, 0x6c2bb316, 0x99a970b9, 0xfa119448, 0x2247e964, 0xc4a8fc8c, 0x1aa0f03f, 0xd8567d2c, 0xef223390, 0xc787494e, 0xc1d938d1, 0xfe8ccaa2, 0x3698d40b, 0xcfa6f581, 0x28a57ade, 0x26dab78e, 0xa43fadbf, 0xe42c3a9d, 0x0d507892, 0x9b6a5fcc, 0x62547e46, 0xc2f68d13, 0xe890d8b8, 0x5e2e39f7, 0xf582c3af, 0xbe9f5d80, 0x7c69d093, 0xa96fd52d, 0xb3cf2512, 0x3bc8ac99, 0xa710187d, 0x6ee89c63, 0x7bdb3bbb, 0x09cd2678, 0xf46e5918, 0x01ec9ab7, 0xa8834f9a, 0x65e6956e, 0x7eaaffe6, 0x0821bccf, 0xe6ef15e8, 0xd9bae79b, 0xce4a6f36, 0xd4ea9f09, 0xd629b07c, 0xaf31a4b2, 0x312a3f23, 0x30c6a594, 0xc035a266, 0x37744ebc, 0xa6fc82ca, 0xb0e090d0, 0x1533a7d8, 0x4af10498, 0xf741ecda, 0x0e7fcd50, 0x2f1791f6, 0x8d764dd6, 0x4d43efb0, 0x54ccaa4d, 0xdfe49604, 0xe39ed1b5, 0x1b4c6a88, 0xb8c12c1f, 0x7f466551, 0x049d5eea, 0x5d018c35, 0x73fa8774, 0x2efb0b41, 0x5ab3671d, 0x5292dbd2, 0x33e91056, 0x136dd647, 0x8c9ad761, 0x7a37a10c, 0x8e59f814, 0x89eb133c, 0xeecea927, 0x35b761c9, 0xede11ce5, 0x3c7a47b1, 0x599cd2df, 0x3f55f273, 0x791814ce, 0xbf73c737, 0xea53f7cd, 0x5b5ffdaa, 0x14df3d6f, 0x867844db, 0x81caaff3, 0x3eb968c4, 0x2c382434, 0x5fc2a340, 0x72161dc3, 0x0cbce225, 0x8b283c49, 0x41ff0d95, 0x7139a801, 0xde080cb3, 0x9cd8b4e4, 0x906456c1, 0x617bcb84, 0x70d532b6, 0x74486c5c, 0x42d0b857 ]
+    T7 = [ 0xa75051f4, 0x65537e41, 0xa4c31a17, 0x5e963a27, 0x6bcb3bab, 0x45f11f9d, 0x58abacfa, 0x03934be3, 0xfa552030, 0x6df6ad76, 0x769188cc, 0x4c25f502, 0xd7fc4fe5, 0xcbd7c52a, 0x44802635, 0xa38fb562, 0x5a49deb1, 0x1b6725ba, 0x0e9845ea, 0xc0e15dfe, 0x7502c32f, 0xf012814c, 0x97a38d46, 0xf9c66bd3, 0x5fe7038f, 0x9c951592, 0x7aebbf6d, 0x59da9552, 0x832dd4be, 0x21d35874, 0x692949e0, 0xc8448ec9, 0x896a75c2, 0x7978f48e, 0x3e6b9958, 0x71dd27b9, 0x4fb6bee1, 0xad17f088, 0xac66c920, 0x3ab47dce, 0x4a1863df, 0x3182e51a, 0x33609751, 0x7f456253, 0x77e0b164, 0xae84bb6b, 0xa01cfe81, 0x2b94f908, 0x68587048, 0xfd198f45, 0x6c8794de, 0xf8b7527b, 0xd323ab73, 0x02e2724b, 0x8f57e31f, 0xab2a6655, 0x2807b2eb, 0xc2032fb5, 0x7b9a86c5, 0x08a5d337, 0x87f23028, 0xa5b223bf, 0x6aba0203, 0x825ced16, 0x1c2b8acf, 0xb492a779, 0xf2f0f307, 0xe2a14e69, 0xf4cd65da, 0xbed50605, 0x621fd134, 0xfe8ac4a6, 0x539d342e, 0x55a0a2f3, 0xe132058a, 0xeb75a4f6, 0xec390b83, 0xefaa4060, 0x9f065e71, 0x1051bd6e, 0x8af93e21, 0x063d96dd, 0x05aedd3e, 0xbd464de6, 0x8db59154, 0x5d0571c4, 0xd46f0406, 0x15ff6050, 0xfb241998, 0xe997d6bd, 0x43cc8940, 0x9e7767d9, 0x42bdb0e8, 0x8b880789, 0x5b38e719, 0xeedb79c8, 0x0a47a17c, 0x0fe97c42, 0x1ec9f884, 0x00000000, 0x86830980, 0xed48322b, 0x70ac1e11, 0x724e6c5a, 0xfffbfd0e, 0x38560f85, 0xd51e3dae, 0x3927362d, 0xd9640a0f, 0xa621685c, 0x54d19b5b, 0x2e3a2436, 0x67b10c0a, 0xe70f9357, 0x96d2b4ee, 0x919e1b9b, 0xc54f80c0, 0x20a261dc, 0x4b695a77, 0x1a161c12, 0xba0ae293, 0x2ae5c0a0, 0xe0433c22, 0x171d121b, 0x0d0b0e09, 0xc7adf28b, 0xa8b92db6, 0xa9c8141e, 0x198557f1, 0x074caf75, 0xddbbee99, 0x60fda37f, 0x269ff701, 0xf5bc5c72, 0x3bc54466, 0x7e345bfb, 0x29768b43, 0xc6dccb23, 0xfc68b6ed, 0xf163b8e4, 0xdccad731, 0x85104263, 0x22401397, 0x112084c6, 0x247d854a, 0x3df8d2bb, 0x3211aef9, 0xa16dc729, 0x2f4b1d9e, 0x30f3dcb2, 0x52ec0d86, 0xe3d077c1, 0x166c2bb3, 0xb999a970, 0x48fa1194, 0x642247e9, 0x8cc4a8fc, 0x3f1aa0f0, 0x2cd8567d, 0x90ef2233, 0x4ec78749, 0xd1c1d938, 0xa2fe8cca, 0x0b3698d4, 0x81cfa6f5, 0xde28a57a, 0x8e26dab7, 0xbfa43fad, 0x9de42c3a, 0x920d5078, 0xcc9b6a5f, 0x4662547e, 0x13c2f68d, 0xb8e890d8, 0xf75e2e39, 0xaff582c3, 0x80be9f5d, 0x937c69d0, 0x2da96fd5, 0x12b3cf25, 0x993bc8ac, 0x7da71018, 0x636ee89c, 0xbb7bdb3b, 0x7809cd26, 0x18f46e59, 0xb701ec9a, 0x9aa8834f, 0x6e65e695, 0xe67eaaff, 0xcf0821bc, 0xe8e6ef15, 0x9bd9bae7, 0x36ce4a6f, 0x09d4ea9f, 0x7cd629b0, 0xb2af31a4, 0x23312a3f, 0x9430c6a5, 0x66c035a2, 0xbc37744e, 0xcaa6fc82, 0xd0b0e090, 0xd81533a7, 0x984af104, 0xdaf741ec, 0x500e7fcd, 0xf62f1791, 0xd68d764d, 0xb04d43ef, 0x4d54ccaa, 0x04dfe496, 0xb5e39ed1, 0x881b4c6a, 0x1fb8c12c, 0x517f4665, 0xea049d5e, 0x355d018c, 0x7473fa87, 0x412efb0b, 0x1d5ab367, 0xd25292db, 0x5633e910, 0x47136dd6, 0x618c9ad7, 0x0c7a37a1, 0x148e59f8, 0x3c89eb13, 0x27eecea9, 0xc935b761, 0xe5ede11c, 0xb13c7a47, 0xdf599cd2, 0x733f55f2, 0xce791814, 0x37bf73c7, 0xcdea53f7, 0xaa5b5ffd, 0x6f14df3d, 0xdb867844, 0xf381caaf, 0xc43eb968, 0x342c3824, 0x405fc2a3, 0xc372161d, 0x250cbce2, 0x498b283c, 0x9541ff0d, 0x017139a8, 0xb3de080c, 0xe49cd8b4, 0xc1906456, 0x84617bcb, 0xb670d532, 0x5c74486c, 0x5742d0b8 ]
+    T8 = [ 0xf4a75051, 0x4165537e, 0x17a4c31a, 0x275e963a, 0xab6bcb3b, 0x9d45f11f, 0xfa58abac, 0xe303934b, 0x30fa5520, 0x766df6ad, 0xcc769188, 0x024c25f5, 0xe5d7fc4f, 0x2acbd7c5, 0x35448026, 0x62a38fb5, 0xb15a49de, 0xba1b6725, 0xea0e9845, 0xfec0e15d, 0x2f7502c3, 0x4cf01281, 0x4697a38d, 0xd3f9c66b, 0x8f5fe703, 0x929c9515, 0x6d7aebbf, 0x5259da95, 0xbe832dd4, 0x7421d358, 0xe0692949, 0xc9c8448e, 0xc2896a75, 0x8e7978f4, 0x583e6b99, 0xb971dd27, 0xe14fb6be, 0x88ad17f0, 0x20ac66c9, 0xce3ab47d, 0xdf4a1863, 0x1a3182e5, 0x51336097, 0x537f4562, 0x6477e0b1, 0x6bae84bb, 0x81a01cfe, 0x082b94f9, 0x48685870, 0x45fd198f, 0xde6c8794, 0x7bf8b752, 0x73d323ab, 0x4b02e272, 0x1f8f57e3, 0x55ab2a66, 0xeb2807b2, 0xb5c2032f, 0xc57b9a86, 0x3708a5d3, 0x2887f230, 0xbfa5b223, 0x036aba02, 0x16825ced, 0xcf1c2b8a, 0x79b492a7, 0x07f2f0f3, 0x69e2a14e, 0xdaf4cd65, 0x05bed506, 0x34621fd1, 0xa6fe8ac4, 0x2e539d34, 0xf355a0a2, 0x8ae13205, 0xf6eb75a4, 0x83ec390b, 0x60efaa40, 0x719f065e, 0x6e1051bd, 0x218af93e, 0xdd063d96, 0x3e05aedd, 0xe6bd464d, 0x548db591, 0xc45d0571, 0x06d46f04, 0x5015ff60, 0x98fb2419, 0xbde997d6, 0x4043cc89, 0xd99e7767, 0xe842bdb0, 0x898b8807, 0x195b38e7, 0xc8eedb79, 0x7c0a47a1, 0x420fe97c, 0x841ec9f8, 0x00000000, 0x80868309, 0x2bed4832, 0x1170ac1e, 0x5a724e6c, 0x0efffbfd, 0x8538560f, 0xaed51e3d, 0x2d392736, 0x0fd9640a, 0x5ca62168, 0x5b54d19b, 0x362e3a24, 0x0a67b10c, 0x57e70f93, 0xee96d2b4, 0x9b919e1b, 0xc0c54f80, 0xdc20a261, 0x774b695a, 0x121a161c, 0x93ba0ae2, 0xa02ae5c0, 0x22e0433c, 0x1b171d12, 0x090d0b0e, 0x8bc7adf2, 0xb6a8b92d, 0x1ea9c814, 0xf1198557, 0x75074caf, 0x99ddbbee, 0x7f60fda3, 0x01269ff7, 0x72f5bc5c, 0x663bc544, 0xfb7e345b, 0x4329768b, 0x23c6dccb, 0xedfc68b6, 0xe4f163b8, 0x31dccad7, 0x63851042, 0x97224013, 0xc6112084, 0x4a247d85, 0xbb3df8d2, 0xf93211ae, 0x29a16dc7, 0x9e2f4b1d, 0xb230f3dc, 0x8652ec0d, 0xc1e3d077, 0xb3166c2b, 0x70b999a9, 0x9448fa11, 0xe9642247, 0xfc8cc4a8, 0xf03f1aa0, 0x7d2cd856, 0x3390ef22, 0x494ec787, 0x38d1c1d9, 0xcaa2fe8c, 0xd40b3698, 0xf581cfa6, 0x7ade28a5, 0xb78e26da, 0xadbfa43f, 0x3a9de42c, 0x78920d50, 0x5fcc9b6a, 0x7e466254, 0x8d13c2f6, 0xd8b8e890, 0x39f75e2e, 0xc3aff582, 0x5d80be9f, 0xd0937c69, 0xd52da96f, 0x2512b3cf, 0xac993bc8, 0x187da710, 0x9c636ee8, 0x3bbb7bdb, 0x267809cd, 0x5918f46e, 0x9ab701ec, 0x4f9aa883, 0x956e65e6, 0xffe67eaa, 0xbccf0821, 0x15e8e6ef, 0xe79bd9ba, 0x6f36ce4a, 0x9f09d4ea, 0xb07cd629, 0xa4b2af31, 0x3f23312a, 0xa59430c6, 0xa266c035, 0x4ebc3774, 0x82caa6fc, 0x90d0b0e0, 0xa7d81533, 0x04984af1, 0xecdaf741, 0xcd500e7f, 0x91f62f17, 0x4dd68d76, 0xefb04d43, 0xaa4d54cc, 0x9604dfe4, 0xd1b5e39e, 0x6a881b4c, 0x2c1fb8c1, 0x65517f46, 0x5eea049d, 0x8c355d01, 0x877473fa, 0x0b412efb, 0x671d5ab3, 0xdbd25292, 0x105633e9, 0xd647136d, 0xd7618c9a, 0xa10c7a37, 0xf8148e59, 0x133c89eb, 0xa927eece, 0x61c935b7, 0x1ce5ede1, 0x47b13c7a, 0xd2df599c, 0xf2733f55, 0x14ce7918, 0xc737bf73, 0xf7cdea53, 0xfdaa5b5f, 0x3d6f14df, 0x44db8678, 0xaff381ca, 0x68c43eb9, 0x24342c38, 0xa3405fc2, 0x1dc37216, 0xe2250cbc, 0x3c498b28, 0x0d9541ff, 0xa8017139, 0x0cb3de08, 0xb4e49cd8, 0x56c19064, 0xcb84617b, 0x32b670d5, 0x6c5c7448, 0xb85742d0 ]
+
+    # Transformations for decryption key expansion
+    U1 = [ 0x00000000, 0x0e090d0b, 0x1c121a16, 0x121b171d, 0x3824342c, 0x362d3927, 0x24362e3a, 0x2a3f2331, 0x70486858, 0x7e416553, 0x6c5a724e, 0x62537f45, 0x486c5c74, 0x4665517f, 0x547e4662, 0x5a774b69, 0xe090d0b0, 0xee99ddbb, 0xfc82caa6, 0xf28bc7ad, 0xd8b4e49c, 0xd6bde997, 0xc4a6fe8a, 0xcaaff381, 0x90d8b8e8, 0x9ed1b5e3, 0x8ccaa2fe, 0x82c3aff5, 0xa8fc8cc4, 0xa6f581cf, 0xb4ee96d2, 0xbae79bd9, 0xdb3bbb7b, 0xd532b670, 0xc729a16d, 0xc920ac66, 0xe31f8f57, 0xed16825c, 0xff0d9541, 0xf104984a, 0xab73d323, 0xa57ade28, 0xb761c935, 0xb968c43e, 0x9357e70f, 0x9d5eea04, 0x8f45fd19, 0x814cf012, 0x3bab6bcb, 0x35a266c0, 0x27b971dd, 0x29b07cd6, 0x038f5fe7, 0x0d8652ec, 0x1f9d45f1, 0x119448fa, 0x4be30393, 0x45ea0e98, 0x57f11985, 0x59f8148e, 0x73c737bf, 0x7dce3ab4, 0x6fd52da9, 0x61dc20a2, 0xad766df6, 0xa37f60fd, 0xb16477e0, 0xbf6d7aeb, 0x955259da, 0x9b5b54d1, 0x894043cc, 0x87494ec7, 0xdd3e05ae, 0xd33708a5, 0xc12c1fb8, 0xcf2512b3, 0xe51a3182, 0xeb133c89, 0xf9082b94, 0xf701269f, 0x4de6bd46, 0x43efb04d, 0x51f4a750, 0x5ffdaa5b, 0x75c2896a, 0x7bcb8461, 0x69d0937c, 0x67d99e77, 0x3daed51e, 0x33a7d815, 0x21bccf08, 0x2fb5c203, 0x058ae132, 0x0b83ec39, 0x1998fb24, 0x1791f62f, 0x764dd68d, 0x7844db86, 0x6a5fcc9b, 0x6456c190, 0x4e69e2a1, 0x4060efaa, 0x527bf8b7, 0x5c72f5bc, 0x0605bed5, 0x080cb3de, 0x1a17a4c3, 0x141ea9c8, 0x3e218af9, 0x302887f2, 0x223390ef, 0x2c3a9de4, 0x96dd063d, 0x98d40b36, 0x8acf1c2b, 0x84c61120, 0xaef93211, 0xa0f03f1a, 0xb2eb2807, 0xbce2250c, 0xe6956e65, 0xe89c636e, 0xfa877473, 0xf48e7978, 0xdeb15a49, 0xd0b85742, 0xc2a3405f, 0xccaa4d54, 0x41ecdaf7, 0x4fe5d7fc, 0x5dfec0e1, 0x53f7cdea, 0x79c8eedb, 0x77c1e3d0, 0x65daf4cd, 0x6bd3f9c6, 0x31a4b2af, 0x3fadbfa4, 0x2db6a8b9, 0x23bfa5b2, 0x09808683, 0x07898b88, 0x15929c95, 0x1b9b919e, 0xa17c0a47, 0xaf75074c, 0xbd6e1051, 0xb3671d5a, 0x99583e6b, 0x97513360, 0x854a247d, 0x8b432976, 0xd134621f, 0xdf3d6f14, 0xcd267809, 0xc32f7502, 0xe9105633, 0xe7195b38, 0xf5024c25, 0xfb0b412e, 0x9ad7618c, 0x94de6c87, 0x86c57b9a, 0x88cc7691, 0xa2f355a0, 0xacfa58ab, 0xbee14fb6, 0xb0e842bd, 0xea9f09d4, 0xe49604df, 0xf68d13c2, 0xf8841ec9, 0xd2bb3df8, 0xdcb230f3, 0xcea927ee, 0xc0a02ae5, 0x7a47b13c, 0x744ebc37, 0x6655ab2a, 0x685ca621, 0x42638510, 0x4c6a881b, 0x5e719f06, 0x5078920d, 0x0a0fd964, 0x0406d46f, 0x161dc372, 0x1814ce79, 0x322bed48, 0x3c22e043, 0x2e39f75e, 0x2030fa55, 0xec9ab701, 0xe293ba0a, 0xf088ad17, 0xfe81a01c, 0xd4be832d, 0xdab78e26, 0xc8ac993b, 0xc6a59430, 0x9cd2df59, 0x92dbd252, 0x80c0c54f, 0x8ec9c844, 0xa4f6eb75, 0xaaffe67e, 0xb8e4f163, 0xb6edfc68, 0x0c0a67b1, 0x02036aba, 0x10187da7, 0x1e1170ac, 0x342e539d, 0x3a275e96, 0x283c498b, 0x26354480, 0x7c420fe9, 0x724b02e2, 0x605015ff, 0x6e5918f4, 0x44663bc5, 0x4a6f36ce, 0x587421d3, 0x567d2cd8, 0x37a10c7a, 0x39a80171, 0x2bb3166c, 0x25ba1b67, 0x0f853856, 0x018c355d, 0x13972240, 0x1d9e2f4b, 0x47e96422, 0x49e06929, 0x5bfb7e34, 0x55f2733f, 0x7fcd500e, 0x71c45d05, 0x63df4a18, 0x6dd64713, 0xd731dcca, 0xd938d1c1, 0xcb23c6dc, 0xc52acbd7, 0xef15e8e6, 0xe11ce5ed, 0xf307f2f0, 0xfd0efffb, 0xa779b492, 0xa970b999, 0xbb6bae84, 0xb562a38f, 0x9f5d80be, 0x91548db5, 0x834f9aa8, 0x8d4697a3 ]
+    U2 = [ 0x00000000, 0x0b0e090d, 0x161c121a, 0x1d121b17, 0x2c382434, 0x27362d39, 0x3a24362e, 0x312a3f23, 0x58704868, 0x537e4165, 0x4e6c5a72, 0x4562537f, 0x74486c5c, 0x7f466551, 0x62547e46, 0x695a774b, 0xb0e090d0, 0xbbee99dd, 0xa6fc82ca, 0xadf28bc7, 0x9cd8b4e4, 0x97d6bde9, 0x8ac4a6fe, 0x81caaff3, 0xe890d8b8, 0xe39ed1b5, 0xfe8ccaa2, 0xf582c3af, 0xc4a8fc8c, 0xcfa6f581, 0xd2b4ee96, 0xd9bae79b, 0x7bdb3bbb, 0x70d532b6, 0x6dc729a1, 0x66c920ac, 0x57e31f8f, 0x5ced1682, 0x41ff0d95, 0x4af10498, 0x23ab73d3, 0x28a57ade, 0x35b761c9, 0x3eb968c4, 0x0f9357e7, 0x049d5eea, 0x198f45fd, 0x12814cf0, 0xcb3bab6b, 0xc035a266, 0xdd27b971, 0xd629b07c, 0xe7038f5f, 0xec0d8652, 0xf11f9d45, 0xfa119448, 0x934be303, 0x9845ea0e, 0x8557f119, 0x8e59f814, 0xbf73c737, 0xb47dce3a, 0xa96fd52d, 0xa261dc20, 0xf6ad766d, 0xfda37f60, 0xe0b16477, 0xebbf6d7a, 0xda955259, 0xd19b5b54, 0xcc894043, 0xc787494e, 0xaedd3e05, 0xa5d33708, 0xb8c12c1f, 0xb3cf2512, 0x82e51a31, 0x89eb133c, 0x94f9082b, 0x9ff70126, 0x464de6bd, 0x4d43efb0, 0x5051f4a7, 0x5b5ffdaa, 0x6a75c289, 0x617bcb84, 0x7c69d093, 0x7767d99e, 0x1e3daed5, 0x1533a7d8, 0x0821bccf, 0x032fb5c2, 0x32058ae1, 0x390b83ec, 0x241998fb, 0x2f1791f6, 0x8d764dd6, 0x867844db, 0x9b6a5fcc, 0x906456c1, 0xa14e69e2, 0xaa4060ef, 0xb7527bf8, 0xbc5c72f5, 0xd50605be, 0xde080cb3, 0xc31a17a4, 0xc8141ea9, 0xf93e218a, 0xf2302887, 0xef223390, 0xe42c3a9d, 0x3d96dd06, 0x3698d40b, 0x2b8acf1c, 0x2084c611, 0x11aef932, 0x1aa0f03f, 0x07b2eb28, 0x0cbce225, 0x65e6956e, 0x6ee89c63, 0x73fa8774, 0x78f48e79, 0x49deb15a, 0x42d0b857, 0x5fc2a340, 0x54ccaa4d, 0xf741ecda, 0xfc4fe5d7, 0xe15dfec0, 0xea53f7cd, 0xdb79c8ee, 0xd077c1e3, 0xcd65daf4, 0xc66bd3f9, 0xaf31a4b2, 0xa43fadbf, 0xb92db6a8, 0xb223bfa5, 0x83098086, 0x8807898b, 0x9515929c, 0x9e1b9b91, 0x47a17c0a, 0x4caf7507, 0x51bd6e10, 0x5ab3671d, 0x6b99583e, 0x60975133, 0x7d854a24, 0x768b4329, 0x1fd13462, 0x14df3d6f, 0x09cd2678, 0x02c32f75, 0x33e91056, 0x38e7195b, 0x25f5024c, 0x2efb0b41, 0x8c9ad761, 0x8794de6c, 0x9a86c57b, 0x9188cc76, 0xa0a2f355, 0xabacfa58, 0xb6bee14f, 0xbdb0e842, 0xd4ea9f09, 0xdfe49604, 0xc2f68d13, 0xc9f8841e, 0xf8d2bb3d, 0xf3dcb230, 0xeecea927, 0xe5c0a02a, 0x3c7a47b1, 0x37744ebc, 0x2a6655ab, 0x21685ca6, 0x10426385, 0x1b4c6a88, 0x065e719f, 0x0d507892, 0x640a0fd9, 0x6f0406d4, 0x72161dc3, 0x791814ce, 0x48322bed, 0x433c22e0, 0x5e2e39f7, 0x552030fa, 0x01ec9ab7, 0x0ae293ba, 0x17f088ad, 0x1cfe81a0, 0x2dd4be83, 0x26dab78e, 0x3bc8ac99, 0x30c6a594, 0x599cd2df, 0x5292dbd2, 0x4f80c0c5, 0x448ec9c8, 0x75a4f6eb, 0x7eaaffe6, 0x63b8e4f1, 0x68b6edfc, 0xb10c0a67, 0xba02036a, 0xa710187d, 0xac1e1170, 0x9d342e53, 0x963a275e, 0x8b283c49, 0x80263544, 0xe97c420f, 0xe2724b02, 0xff605015, 0xf46e5918, 0xc544663b, 0xce4a6f36, 0xd3587421, 0xd8567d2c, 0x7a37a10c, 0x7139a801, 0x6c2bb316, 0x6725ba1b, 0x560f8538, 0x5d018c35, 0x40139722, 0x4b1d9e2f, 0x2247e964, 0x2949e069, 0x345bfb7e, 0x3f55f273, 0x0e7fcd50, 0x0571c45d, 0x1863df4a, 0x136dd647, 0xcad731dc, 0xc1d938d1, 0xdccb23c6, 0xd7c52acb, 0xe6ef15e8, 0xede11ce5, 0xf0f307f2, 0xfbfd0eff, 0x92a779b4, 0x99a970b9, 0x84bb6bae, 0x8fb562a3, 0xbe9f5d80, 0xb591548d, 0xa8834f9a, 0xa38d4697 ]
+    U3 = [ 0x00000000, 0x0d0b0e09, 0x1a161c12, 0x171d121b, 0x342c3824, 0x3927362d, 0x2e3a2436, 0x23312a3f, 0x68587048, 0x65537e41, 0x724e6c5a, 0x7f456253, 0x5c74486c, 0x517f4665, 0x4662547e, 0x4b695a77, 0xd0b0e090, 0xddbbee99, 0xcaa6fc82, 0xc7adf28b, 0xe49cd8b4, 0xe997d6bd, 0xfe8ac4a6, 0xf381caaf, 0xb8e890d8, 0xb5e39ed1, 0xa2fe8cca, 0xaff582c3, 0x8cc4a8fc, 0x81cfa6f5, 0x96d2b4ee, 0x9bd9bae7, 0xbb7bdb3b, 0xb670d532, 0xa16dc729, 0xac66c920, 0x8f57e31f, 0x825ced16, 0x9541ff0d, 0x984af104, 0xd323ab73, 0xde28a57a, 0xc935b761, 0xc43eb968, 0xe70f9357, 0xea049d5e, 0xfd198f45, 0xf012814c, 0x6bcb3bab, 0x66c035a2, 0x71dd27b9, 0x7cd629b0, 0x5fe7038f, 0x52ec0d86, 0x45f11f9d, 0x48fa1194, 0x03934be3, 0x0e9845ea, 0x198557f1, 0x148e59f8, 0x37bf73c7, 0x3ab47dce, 0x2da96fd5, 0x20a261dc, 0x6df6ad76, 0x60fda37f, 0x77e0b164, 0x7aebbf6d, 0x59da9552, 0x54d19b5b, 0x43cc8940, 0x4ec78749, 0x05aedd3e, 0x08a5d337, 0x1fb8c12c, 0x12b3cf25, 0x3182e51a, 0x3c89eb13, 0x2b94f908, 0x269ff701, 0xbd464de6, 0xb04d43ef, 0xa75051f4, 0xaa5b5ffd, 0x896a75c2, 0x84617bcb, 0x937c69d0, 0x9e7767d9, 0xd51e3dae, 0xd81533a7, 0xcf0821bc, 0xc2032fb5, 0xe132058a, 0xec390b83, 0xfb241998, 0xf62f1791, 0xd68d764d, 0xdb867844, 0xcc9b6a5f, 0xc1906456, 0xe2a14e69, 0xefaa4060, 0xf8b7527b, 0xf5bc5c72, 0xbed50605, 0xb3de080c, 0xa4c31a17, 0xa9c8141e, 0x8af93e21, 0x87f23028, 0x90ef2233, 0x9de42c3a, 0x063d96dd, 0x0b3698d4, 0x1c2b8acf, 0x112084c6, 0x3211aef9, 0x3f1aa0f0, 0x2807b2eb, 0x250cbce2, 0x6e65e695, 0x636ee89c, 0x7473fa87, 0x7978f48e, 0x5a49deb1, 0x5742d0b8, 0x405fc2a3, 0x4d54ccaa, 0xdaf741ec, 0xd7fc4fe5, 0xc0e15dfe, 0xcdea53f7, 0xeedb79c8, 0xe3d077c1, 0xf4cd65da, 0xf9c66bd3, 0xb2af31a4, 0xbfa43fad, 0xa8b92db6, 0xa5b223bf, 0x86830980, 0x8b880789, 0x9c951592, 0x919e1b9b, 0x0a47a17c, 0x074caf75, 0x1051bd6e, 0x1d5ab367, 0x3e6b9958, 0x33609751, 0x247d854a, 0x29768b43, 0x621fd134, 0x6f14df3d, 0x7809cd26, 0x7502c32f, 0x5633e910, 0x5b38e719, 0x4c25f502, 0x412efb0b, 0x618c9ad7, 0x6c8794de, 0x7b9a86c5, 0x769188cc, 0x55a0a2f3, 0x58abacfa, 0x4fb6bee1, 0x42bdb0e8, 0x09d4ea9f, 0x04dfe496, 0x13c2f68d, 0x1ec9f884, 0x3df8d2bb, 0x30f3dcb2, 0x27eecea9, 0x2ae5c0a0, 0xb13c7a47, 0xbc37744e, 0xab2a6655, 0xa621685c, 0x85104263, 0x881b4c6a, 0x9f065e71, 0x920d5078, 0xd9640a0f, 0xd46f0406, 0xc372161d, 0xce791814, 0xed48322b, 0xe0433c22, 0xf75e2e39, 0xfa552030, 0xb701ec9a, 0xba0ae293, 0xad17f088, 0xa01cfe81, 0x832dd4be, 0x8e26dab7, 0x993bc8ac, 0x9430c6a5, 0xdf599cd2, 0xd25292db, 0xc54f80c0, 0xc8448ec9, 0xeb75a4f6, 0xe67eaaff, 0xf163b8e4, 0xfc68b6ed, 0x67b10c0a, 0x6aba0203, 0x7da71018, 0x70ac1e11, 0x539d342e, 0x5e963a27, 0x498b283c, 0x44802635, 0x0fe97c42, 0x02e2724b, 0x15ff6050, 0x18f46e59, 0x3bc54466, 0x36ce4a6f, 0x21d35874, 0x2cd8567d, 0x0c7a37a1, 0x017139a8, 0x166c2bb3, 0x1b6725ba, 0x38560f85, 0x355d018c, 0x22401397, 0x2f4b1d9e, 0x642247e9, 0x692949e0, 0x7e345bfb, 0x733f55f2, 0x500e7fcd, 0x5d0571c4, 0x4a1863df, 0x47136dd6, 0xdccad731, 0xd1c1d938, 0xc6dccb23, 0xcbd7c52a, 0xe8e6ef15, 0xe5ede11c, 0xf2f0f307, 0xfffbfd0e, 0xb492a779, 0xb999a970, 0xae84bb6b, 0xa38fb562, 0x80be9f5d, 0x8db59154, 0x9aa8834f, 0x97a38d46 ]
+    U4 = [ 0x00000000, 0x090d0b0e, 0x121a161c, 0x1b171d12, 0x24342c38, 0x2d392736, 0x362e3a24, 0x3f23312a, 0x48685870, 0x4165537e, 0x5a724e6c, 0x537f4562, 0x6c5c7448, 0x65517f46, 0x7e466254, 0x774b695a, 0x90d0b0e0, 0x99ddbbee, 0x82caa6fc, 0x8bc7adf2, 0xb4e49cd8, 0xbde997d6, 0xa6fe8ac4, 0xaff381ca, 0xd8b8e890, 0xd1b5e39e, 0xcaa2fe8c, 0xc3aff582, 0xfc8cc4a8, 0xf581cfa6, 0xee96d2b4, 0xe79bd9ba, 0x3bbb7bdb, 0x32b670d5, 0x29a16dc7, 0x20ac66c9, 0x1f8f57e3, 0x16825ced, 0x0d9541ff, 0x04984af1, 0x73d323ab, 0x7ade28a5, 0x61c935b7, 0x68c43eb9, 0x57e70f93, 0x5eea049d, 0x45fd198f, 0x4cf01281, 0xab6bcb3b, 0xa266c035, 0xb971dd27, 0xb07cd629, 0x8f5fe703, 0x8652ec0d, 0x9d45f11f, 0x9448fa11, 0xe303934b, 0xea0e9845, 0xf1198557, 0xf8148e59, 0xc737bf73, 0xce3ab47d, 0xd52da96f, 0xdc20a261, 0x766df6ad, 0x7f60fda3, 0x6477e0b1, 0x6d7aebbf, 0x5259da95, 0x5b54d19b, 0x4043cc89, 0x494ec787, 0x3e05aedd, 0x3708a5d3, 0x2c1fb8c1, 0x2512b3cf, 0x1a3182e5, 0x133c89eb, 0x082b94f9, 0x01269ff7, 0xe6bd464d, 0xefb04d43, 0xf4a75051, 0xfdaa5b5f, 0xc2896a75, 0xcb84617b, 0xd0937c69, 0xd99e7767, 0xaed51e3d, 0xa7d81533, 0xbccf0821, 0xb5c2032f, 0x8ae13205, 0x83ec390b, 0x98fb2419, 0x91f62f17, 0x4dd68d76, 0x44db8678, 0x5fcc9b6a, 0x56c19064, 0x69e2a14e, 0x60efaa40, 0x7bf8b752, 0x72f5bc5c, 0x05bed506, 0x0cb3de08, 0x17a4c31a, 0x1ea9c814, 0x218af93e, 0x2887f230, 0x3390ef22, 0x3a9de42c, 0xdd063d96, 0xd40b3698, 0xcf1c2b8a, 0xc6112084, 0xf93211ae, 0xf03f1aa0, 0xeb2807b2, 0xe2250cbc, 0x956e65e6, 0x9c636ee8, 0x877473fa, 0x8e7978f4, 0xb15a49de, 0xb85742d0, 0xa3405fc2, 0xaa4d54cc, 0xecdaf741, 0xe5d7fc4f, 0xfec0e15d, 0xf7cdea53, 0xc8eedb79, 0xc1e3d077, 0xdaf4cd65, 0xd3f9c66b, 0xa4b2af31, 0xadbfa43f, 0xb6a8b92d, 0xbfa5b223, 0x80868309, 0x898b8807, 0x929c9515, 0x9b919e1b, 0x7c0a47a1, 0x75074caf, 0x6e1051bd, 0x671d5ab3, 0x583e6b99, 0x51336097, 0x4a247d85, 0x4329768b, 0x34621fd1, 0x3d6f14df, 0x267809cd, 0x2f7502c3, 0x105633e9, 0x195b38e7, 0x024c25f5, 0x0b412efb, 0xd7618c9a, 0xde6c8794, 0xc57b9a86, 0xcc769188, 0xf355a0a2, 0xfa58abac, 0xe14fb6be, 0xe842bdb0, 0x9f09d4ea, 0x9604dfe4, 0x8d13c2f6, 0x841ec9f8, 0xbb3df8d2, 0xb230f3dc, 0xa927eece, 0xa02ae5c0, 0x47b13c7a, 0x4ebc3774, 0x55ab2a66, 0x5ca62168, 0x63851042, 0x6a881b4c, 0x719f065e, 0x78920d50, 0x0fd9640a, 0x06d46f04, 0x1dc37216, 0x14ce7918, 0x2bed4832, 0x22e0433c, 0x39f75e2e, 0x30fa5520, 0x9ab701ec, 0x93ba0ae2, 0x88ad17f0, 0x81a01cfe, 0xbe832dd4, 0xb78e26da, 0xac993bc8, 0xa59430c6, 0xd2df599c, 0xdbd25292, 0xc0c54f80, 0xc9c8448e, 0xf6eb75a4, 0xffe67eaa, 0xe4f163b8, 0xedfc68b6, 0x0a67b10c, 0x036aba02, 0x187da710, 0x1170ac1e, 0x2e539d34, 0x275e963a, 0x3c498b28, 0x35448026, 0x420fe97c, 0x4b02e272, 0x5015ff60, 0x5918f46e, 0x663bc544, 0x6f36ce4a, 0x7421d358, 0x7d2cd856, 0xa10c7a37, 0xa8017139, 0xb3166c2b, 0xba1b6725, 0x8538560f, 0x8c355d01, 0x97224013, 0x9e2f4b1d, 0xe9642247, 0xe0692949, 0xfb7e345b, 0xf2733f55, 0xcd500e7f, 0xc45d0571, 0xdf4a1863, 0xd647136d, 0x31dccad7, 0x38d1c1d9, 0x23c6dccb, 0x2acbd7c5, 0x15e8e6ef, 0x1ce5ede1, 0x07f2f0f3, 0x0efffbfd, 0x79b492a7, 0x70b999a9, 0x6bae84bb, 0x62a38fb5, 0x5d80be9f, 0x548db591, 0x4f9aa883, 0x4697a38d ]
+
+    def __init__(self, key):
+
+        if len(key) not in (16, 24, 32):
+            raise ValueError('Invalid key size')
+
+        rounds = self.number_of_rounds[len(key)]
+
+        # Encryption round keys
+        self._Ke = [[0] * 4 for i in xrange(rounds + 1)]
+
+        # Decryption round keys
+        self._Kd = [[0] * 4 for i in xrange(rounds + 1)]
+
+        round_key_count = (rounds + 1) * 4
+        KC = len(key) // 4
+
+        # Convert the key into ints
+        tk = [ struct.unpack('>i', key[i:i + 4])[0] for i in xrange(0, len(key), 4) ]
+
+        # Copy values into round key arrays
+        for i in xrange(0, KC):
+            self._Ke[i // 4][i % 4] = tk[i]
+            self._Kd[rounds - (i // 4)][i % 4] = tk[i]
+
+        # Key expansion (fips-197 section 5.2)
+        rconpointer = 0
+        t = KC
+        while t < round_key_count:
+
+            tt = tk[KC - 1]
+            tk[0] ^= ((self.S[(tt >> 16) & 0xFF] << 24) ^
+                      (self.S[(tt >>  8) & 0xFF] << 16) ^
+                      (self.S[ tt        & 0xFF] <<  8) ^
+                       self.S[(tt >> 24) & 0xFF]        ^
+                      (self.rcon[rconpointer] << 24))
+            rconpointer += 1
+
+            if KC != 8:
+                for i in xrange(1, KC):
+                    tk[i] ^= tk[i - 1]
+
+            # Key expansion for 256-bit keys is "slightly different" (fips-197)
+            else:
+                for i in xrange(1, KC // 2):
+                    tk[i] ^= tk[i - 1]
+                tt = tk[KC // 2 - 1]
+
+                tk[KC // 2] ^= (self.S[ tt        & 0xFF]        ^
+                               (self.S[(tt >>  8) & 0xFF] <<  8) ^
+                               (self.S[(tt >> 16) & 0xFF] << 16) ^
+                               (self.S[(tt >> 24) & 0xFF] << 24))
+
+                for i in xrange(KC // 2 + 1, KC):
+                    tk[i] ^= tk[i - 1]
+
+            # Copy values into round key arrays
+            j = 0
+            while j < KC and t < round_key_count:
+                self._Ke[t // 4][t % 4] = tk[j]
+                self._Kd[rounds - (t // 4)][t % 4] = tk[j]
+                j += 1
+                t += 1
+
+        # Inverse-Cipher-ify the decryption round key (fips-197 section 5.3)
+        for r in xrange(1, rounds):
+            for j in xrange(0, 4):
+                tt = self._Kd[r][j]
+                self._Kd[r][j] = (self.U1[(tt >> 24) & 0xFF] ^
+                                  self.U2[(tt >> 16) & 0xFF] ^
+                                  self.U3[(tt >>  8) & 0xFF] ^
+                                  self.U4[ tt        & 0xFF])
+
+    def encrypt(self, plaintext):
+        'Encrypt a block of plain text using the AES block cipher.'
+
+        if len(plaintext) != 16:
+            raise ValueError('wrong block length')
+
+        rounds = len(self._Ke) - 1
+        (s1, s2, s3) = [1, 2, 3]
+        a = [0, 0, 0, 0]
+
+        # Convert plaintext to (ints ^ key)
+        t = [(_compact_word(plaintext[4 * i:4 * i + 4]) ^ self._Ke[0][i]) for i in xrange(0, 4)]
+
+        # Apply round transforms
+        for r in xrange(1, rounds):
+            for i in xrange(0, 4):
+                a[i] = (self.T1[(t[ i          ] >> 24) & 0xFF] ^
+                        self.T2[(t[(i + s1) % 4] >> 16) & 0xFF] ^
+                        self.T3[(t[(i + s2) % 4] >>  8) & 0xFF] ^
+                        self.T4[ t[(i + s3) % 4]        & 0xFF] ^
+                        self._Ke[r][i])
+            t = copy.copy(a)
+
+        # The last round is special
+        result = [ ]
+        for i in xrange(0, 4):
+            tt = self._Ke[rounds][i]
+            result.append((self.S[(t[ i           ] >> 24) & 0xFF] ^ (tt >> 24)) & 0xFF)
+            result.append((self.S[(t[(i + s1) % 4] >> 16) & 0xFF] ^ (tt >> 16)) & 0xFF)
+            result.append((self.S[(t[(i + s2) % 4] >>  8) & 0xFF] ^ (tt >>  8)) & 0xFF)
+            result.append((self.S[ t[(i + s3) % 4]        & 0xFF] ^  tt       ) & 0xFF)
+
+        return result
+
+    def decrypt(self, ciphertext):
+        'Decrypt a block of cipher text using the AES block cipher.'
+
+        if len(ciphertext) != 16:
+            raise ValueError('wrong block length')
+
+        rounds = len(self._Kd) - 1
+        (s1, s2, s3) = [3, 2, 1]
+        a = [0, 0, 0, 0]
+
+        # Convert ciphertext to (ints ^ key)
+        t = [(_compact_word(ciphertext[4 * i:4 * i + 4]) ^ self._Kd[0][i]) for i in xrange(0, 4)]
+
+        # Apply round transforms
+        for r in xrange(1, rounds):
+            for i in xrange(0, 4):
+                a[i] = (self.T5[(t[ i          ] >> 24) & 0xFF] ^
+                        self.T6[(t[(i + s1) % 4] >> 16) & 0xFF] ^
+                        self.T7[(t[(i + s2) % 4] >>  8) & 0xFF] ^
+                        self.T8[ t[(i + s3) % 4]        & 0xFF] ^
+                        self._Kd[r][i])
+            t = copy.copy(a)
+
+        # The last round is special
+        result = [ ]
+        for i in xrange(0, 4):
+            tt = self._Kd[rounds][i]
+            result.append((self.Si[(t[ i           ] >> 24) & 0xFF] ^ (tt >> 24)) & 0xFF)
+            result.append((self.Si[(t[(i + s1) % 4] >> 16) & 0xFF] ^ (tt >> 16)) & 0xFF)
+            result.append((self.Si[(t[(i + s2) % 4] >>  8) & 0xFF] ^ (tt >>  8)) & 0xFF)
+            result.append((self.Si[ t[(i + s3) % 4]        & 0xFF] ^  tt       ) & 0xFF)
+
+        return result
+
+
+class Counter(object):
+    '''A counter object for the Counter (CTR) mode of operation.
+
+       To create a custom counter, you can usually just override the
+       increment method.'''
+
+    def __init__(self, initial_value = 1):
+
+        # Convert the value into an array of bytes long
+        self._counter = [ ((initial_value >> i) % 256) for i in xrange(128 - 8, -1, -8) ]
+
+    value = property(lambda s: s._counter)
+
+    def increment(self):
+        '''Increment the counter (overflow rolls back to 0).'''
+
+        for i in xrange(len(self._counter) - 1, -1, -1):
+            self._counter[i] += 1
+
+            if self._counter[i] < 256: break
+
+            # Carry the one
+            self._counter[i] = 0
+
+        # Overflow
+        else:
+            self._counter = [ 0 ] * len(self._counter)
+
+
+class AESBlockModeOfOperation(object):
+    '''Super-class for AES modes of operation that require blocks.'''
+    def __init__(self, key):
+        self._aes = AES(key)
+
+    def decrypt(self, ciphertext):
+        raise Exception('not implemented')
+
+    def encrypt(self, plaintext):
+        raise Exception('not implemented')
+
+
+class AESStreamModeOfOperation(AESBlockModeOfOperation):
+    '''Super-class for AES modes of operation that are stream-ciphers.'''
+
+class AESSegmentModeOfOperation(AESStreamModeOfOperation):
+    '''Super-class for AES modes of operation that segment data.'''
+
+    segment_bytes = 16
+
+
+
+class AESModeOfOperationECB(AESBlockModeOfOperation):
+    '''AES Electronic Codebook Mode of Operation.
+
+       o Block-cipher, so data must be padded to 16 byte boundaries
+
+   Security Notes:
+       o This mode is not recommended
+       o Any two identical blocks produce identical encrypted values,
+         exposing data patterns. (See the image of Tux on wikipedia)
+
+   Also see:
+       o https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Electronic_codebook_.28ECB.29
+       o See NIST SP800-38A (http://csrc.nist.gov/publications/nistpubs/800-38a/sp800-38a.pdf); section 6.1'''
+
+
+    name = "Electronic Codebook (ECB)"
+
+    def encrypt(self, plaintext):
+        if len(plaintext) != 16:
+            raise ValueError('plaintext block must be 16 bytes')
+
+        plaintext = _string_to_bytes(plaintext)
+        return _bytes_to_string(self._aes.encrypt(plaintext))
+
+    def decrypt(self, ciphertext):
+        if len(ciphertext) != 16:
+            raise ValueError('ciphertext block must be 16 bytes')
+
+        ciphertext = _string_to_bytes(ciphertext)
+        return _bytes_to_string(self._aes.decrypt(ciphertext))
+
+
+
+class AESModeOfOperationCBC(AESBlockModeOfOperation):
+    '''AES Cipher-Block Chaining Mode of Operation.
+
+       o The Initialization Vector (IV)
+       o Block-cipher, so data must be padded to 16 byte boundaries
+       o An incorrect initialization vector will only cause the first
+         block to be corrupt; all other blocks will be intact
+       o A corrupt bit in the cipher text will cause a block to be
+         corrupted, and the next block to be inverted, but all other
+         blocks will be intact.
+
+   Security Notes:
+       o This method (and CTR) ARE recommended.
+
+   Also see:
+       o https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Cipher-block_chaining_.28CBC.29
+       o See NIST SP800-38A (http://csrc.nist.gov/publications/nistpubs/800-38a/sp800-38a.pdf); section 6.2'''
+
+
+    name = "Cipher-Block Chaining (CBC)"
+
+    def __init__(self, key, iv = None):
+        if iv is None:
+            self._last_cipherblock = [ 0 ] * 16
+        elif len(iv) != 16:
+            raise ValueError('initialization vector must be 16 bytes')
+        else:
+            self._last_cipherblock = _string_to_bytes(iv)
+
+        AESBlockModeOfOperation.__init__(self, key)
+
+    def encrypt(self, plaintext):
+        if len(plaintext) != 16:
+            raise ValueError('plaintext block must be 16 bytes')
+
+        plaintext = _string_to_bytes(plaintext)
+        precipherblock = [ (p ^ l) for (p, l) in zip(plaintext, self._last_cipherblock) ]
+        self._last_cipherblock = self._aes.encrypt(precipherblock)
+
+        return _bytes_to_string(self._last_cipherblock)
+
+    def decrypt(self, ciphertext):
+        if len(ciphertext) != 16:
+            raise ValueError('ciphertext block must be 16 bytes')
+
+        cipherblock = _string_to_bytes(ciphertext)
+        plaintext = [ (p ^ l) for (p, l) in zip(self._aes.decrypt(cipherblock), self._last_cipherblock) ]
+        self._last_cipherblock = cipherblock
+
+        return _bytes_to_string(plaintext)
+
+
+
+class AESModeOfOperationCFB(AESSegmentModeOfOperation):
+    '''AES Cipher Feedback Mode of Operation.
+
+       o A stream-cipher, so input does not need to be padded to blocks,
+         but does need to be padded to segment_size
+
+    Also see:
+       o https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Cipher_feedback_.28CFB.29
+       o See NIST SP800-38A (http://csrc.nist.gov/publications/nistpubs/800-38a/sp800-38a.pdf); section 6.3'''
+
+
+    name = "Cipher Feedback (CFB)"
+
+    def __init__(self, key, iv, segment_size = 1):
+        if segment_size == 0: segment_size = 1
+
+        if iv is None:
+            self._shift_register = [ 0 ] * 16
+        elif len(iv) != 16:
+            raise ValueError('initialization vector must be 16 bytes')
+        else:
+          self._shift_register = _string_to_bytes(iv)
+
+        self._segment_bytes = segment_size
+
+        AESBlockModeOfOperation.__init__(self, key)
+
+    segment_bytes = property(lambda s: s._segment_bytes)
+
+    def encrypt(self, plaintext):
+        if len(plaintext) % self._segment_bytes != 0:
+            raise ValueError('plaintext block must be a multiple of segment_size')
+
+        plaintext = _string_to_bytes(plaintext)
+
+        # Break block into segments
+        encrypted = [ ]
+        for i in xrange(0, len(plaintext), self._segment_bytes):
+            plaintext_segment = plaintext[i: i + self._segment_bytes]
+            xor_segment = self._aes.encrypt(self._shift_register)[:len(plaintext_segment)]
+            cipher_segment = [ (p ^ x) for (p, x) in zip(plaintext_segment, xor_segment) ]
+
+            # Shift the top bits out and the ciphertext in
+            self._shift_register = _concat_list(self._shift_register[len(cipher_segment):], cipher_segment)
+
+            encrypted.extend(cipher_segment)
+
+        return _bytes_to_string(encrypted)
+
+    def decrypt(self, ciphertext):
+        if len(ciphertext) % self._segment_bytes != 0:
+            raise ValueError('ciphertext block must be a multiple of segment_size')
+
+        ciphertext = _string_to_bytes(ciphertext)
+
+        # Break block into segments
+        decrypted = [ ]
+        for i in xrange(0, len(ciphertext), self._segment_bytes):
+            cipher_segment = ciphertext[i: i + self._segment_bytes]
+            xor_segment = self._aes.encrypt(self._shift_register)[:len(cipher_segment)]
+            plaintext_segment = [ (p ^ x) for (p, x) in zip(cipher_segment, xor_segment) ]
+
+            # Shift the top bits out and the ciphertext in
+            self._shift_register = _concat_list(self._shift_register[len(cipher_segment):], cipher_segment)
+
+            decrypted.extend(plaintext_segment)
+
+        return _bytes_to_string(decrypted)
+
+
+
+class AESModeOfOperationOFB(AESStreamModeOfOperation):
+    '''AES Output Feedback Mode of Operation.
+
+       o A stream-cipher, so input does not need to be padded to blocks,
+         allowing arbitrary length data.
+       o A bit twiddled in the cipher text, twiddles the same bit in the
+         same bit in the plain text, which can be useful for error
+         correction techniques.
+
+    Also see:
+       o https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Output_feedback_.28OFB.29
+       o See NIST SP800-38A (http://csrc.nist.gov/publications/nistpubs/800-38a/sp800-38a.pdf); section 6.4'''
+
+
+    name = "Output Feedback (OFB)"
+
+    def __init__(self, key, iv = None):
+        if iv is None:
+            self._last_precipherblock = [ 0 ] * 16
+        elif len(iv) != 16:
+            raise ValueError('initialization vector must be 16 bytes')
+        else:
+          self._last_precipherblock = _string_to_bytes(iv)
+
+        self._remaining_block = [ ]
+
+        AESBlockModeOfOperation.__init__(self, key)
+
+    def encrypt(self, plaintext):
+        encrypted = [ ]
+        for p in _string_to_bytes(plaintext):
+            if len(self._remaining_block) == 0:
+                self._remaining_block = self._aes.encrypt(self._last_precipherblock)
+                self._last_precipherblock = [ ]
+            precipherbyte = self._remaining_block.pop(0)
+            self._last_precipherblock.append(precipherbyte)
+            cipherbyte = p ^ precipherbyte
+            encrypted.append(cipherbyte)
+
+        return _bytes_to_string(encrypted)
+
+    def decrypt(self, ciphertext):
+        # AES-OFB is symetric
+        return self.encrypt(ciphertext)
+
+
+
+class AESModeOfOperationCTR(AESStreamModeOfOperation):
+    '''AES Counter Mode of Operation.
+
+       o A stream-cipher, so input does not need to be padded to blocks,
+         allowing arbitrary length data.
+       o The counter must be the same size as the key size (ie. len(key))
+       o Each block independant of the other, so a corrupt byte will not
+         damage future blocks.
+       o Each block has a uniue counter value associated with it, which
+         contributes to the encrypted value, so no data patterns are
+         leaked.
+       o Also known as: Counter Mode (CM), Integer Counter Mode (ICM) and
+         Segmented Integer Counter (SIC
+
+   Security Notes:
+       o This method (and CBC) ARE recommended.
+       o Each message block is associated with a counter value which must be
+         unique for ALL messages with the same key. Otherwise security may be
+         compromised.
+
+    Also see:
+
+       o https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Counter_.28CTR.29
+       o See NIST SP800-38A (http://csrc.nist.gov/publications/nistpubs/800-38a/sp800-38a.pdf); section 6.5
+         and Appendix B for managing the initial counter'''
+
+
+    name = "Counter (CTR)"
+
+    def __init__(self, key, counter = None):
+        AESBlockModeOfOperation.__init__(self, key)
+
+        if counter is None:
+            counter = Counter()
+
+        self._counter = counter
+        self._remaining_counter = [ ]
+
+    def encrypt(self, plaintext):
+        while len(self._remaining_counter) < len(plaintext):
+            self._remaining_counter += self._aes.encrypt(self._counter.value)
+            self._counter.increment()
+
+        plaintext = _string_to_bytes(plaintext)
+
+        encrypted = [ (p ^ c) for (p, c) in zip(plaintext, self._remaining_counter) ]
+        self._remaining_counter = self._remaining_counter[len(encrypted):]
+
+        return _bytes_to_string(encrypted)
+
+    def decrypt(self, crypttext):
+        # AES-CTR is symetric
+        return self.encrypt(crypttext)
+
+
+# Simple lookup table for each mode
+AESModesOfOperation = dict(
+    ctr = AESModeOfOperationCTR,
+    cbc = AESModeOfOperationCBC,
+    cfb = AESModeOfOperationCFB,
+    ecb = AESModeOfOperationECB,
+    ofb = AESModeOfOperationOFB,
+)

Contents/Libraries/Shared/pyaes/blockfeeder.py

@@ -0,0 +1,227 @@
+# The MIT License (MIT)
+#
+# Copyright (c) 2014 Richard Moore
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+# THE SOFTWARE.
+
+
+from .aes import AESBlockModeOfOperation, AESSegmentModeOfOperation, AESStreamModeOfOperation
+from .util import append_PKCS7_padding, strip_PKCS7_padding, to_bufferable
+
+
+# First we inject three functions to each of the modes of operations
+#
+#    _can_consume(size)
+#       - Given a size, determine how many bytes could be consumed in
+#         a single call to either the decrypt or encrypt method
+#
+#    _final_encrypt(data, padding = PADDING_DEFAULT)
+#       - call and return encrypt on this (last) chunk of data,
+#         padding as necessary; this will always be at least 16
+#         bytes unless the total incoming input was less than 16
+#         bytes
+#
+#    _final_decrypt(data, padding = PADDING_DEFAULT)
+#       - same as _final_encrypt except for decrypt, for
+#         stripping off padding
+#
+
+PADDING_NONE       = 'none'
+PADDING_DEFAULT    = 'default'
+
+# @TODO: Ciphertext stealing and explicit PKCS#7
+# PADDING_CIPHERTEXT_STEALING
+# PADDING_PKCS7
+
+# ECB and CBC are block-only ciphers
+
+def _block_can_consume(self, size):
+    if size >= 16: return 16
+    return 0
+
+# After padding, we may have more than one block
+def _block_final_encrypt(self, data, padding = PADDING_DEFAULT):
+    if padding == PADDING_DEFAULT:
+        data = append_PKCS7_padding(data)
+
+    elif padding == PADDING_NONE:
+        if len(data) != 16:
+            raise Exception('invalid data length for final block')
+    else:
+        raise Exception('invalid padding option')
+
+    if len(data) == 32:
+        return self.encrypt(data[:16]) + self.encrypt(data[16:])
+
+    return self.encrypt(data)
+
+
+def _block_final_decrypt(self, data, padding = PADDING_DEFAULT):
+    if padding == PADDING_DEFAULT:
+        return strip_PKCS7_padding(self.decrypt(data))
+
+    if padding == PADDING_NONE:
+        if len(data) != 16:
+            raise Exception('invalid data length for final block')
+        return self.decrypt(data)
+
+    raise Exception('invalid padding option')
+
+AESBlockModeOfOperation._can_consume = _block_can_consume
+AESBlockModeOfOperation._final_encrypt = _block_final_encrypt
+AESBlockModeOfOperation._final_decrypt = _block_final_decrypt
+
+
+
+# CFB is a segment cipher
+
+def _segment_can_consume(self, size):
+    return self.segment_bytes * int(size // self.segment_bytes)
+
+# CFB can handle a non-segment-sized block at the end using the remaining cipherblock
+def _segment_final_encrypt(self, data, padding = PADDING_DEFAULT):
+    if padding != PADDING_DEFAULT:
+        raise Exception('invalid padding option')
+
+    faux_padding = (chr(0) * (self.segment_bytes - (len(data) % self.segment_bytes)))
+    padded = data + to_bufferable(faux_padding)
+    return self.encrypt(padded)[:len(data)]
+
+# CFB can handle a non-segment-sized block at the end using the remaining cipherblock
+def _segment_final_decrypt(self, data, padding = PADDING_DEFAULT):
+    if padding != PADDING_DEFAULT:
+        raise Exception('invalid padding option')
+
+    faux_padding = (chr(0) * (self.segment_bytes - (len(data) % self.segment_bytes)))
+    padded = data + to_bufferable(faux_padding)
+    return self.decrypt(padded)[:len(data)]
+
+AESSegmentModeOfOperation._can_consume = _segment_can_consume
+AESSegmentModeOfOperation._final_encrypt = _segment_final_encrypt
+AESSegmentModeOfOperation._final_decrypt = _segment_final_decrypt
+
+
+
+# OFB and CTR are stream ciphers
+
+def _stream_can_consume(self, size):
+    return size
+
+def _stream_final_encrypt(self, data, padding = PADDING_DEFAULT):
+    if padding not in [PADDING_NONE, PADDING_DEFAULT]:
+        raise Exception('invalid padding option')
+
+    return self.encrypt(data)
+
+def _stream_final_decrypt(self, data, padding = PADDING_DEFAULT):
+    if padding not in [PADDING_NONE, PADDING_DEFAULT]:
+        raise Exception('invalid padding option')
+
+    return self.decrypt(data)
+
+AESStreamModeOfOperation._can_consume = _stream_can_consume
+AESStreamModeOfOperation._final_encrypt = _stream_final_encrypt
+AESStreamModeOfOperation._final_decrypt = _stream_final_decrypt
+
+
+
+class BlockFeeder(object):
+    '''The super-class for objects to handle chunking a stream of bytes
+       into the appropriate block size for the underlying mode of operation
+       and applying (or stripping) padding, as necessary.'''
+
+    def __init__(self, mode, feed, final, padding = PADDING_DEFAULT):
+        self._mode = mode
+        self._feed = feed
+        self._final = final
+        self._buffer = to_bufferable("")
+        self._padding = padding
+
+    def feed(self, data = None):
+        '''Provide bytes to encrypt (or decrypt), returning any bytes
+           possible from this or any previous calls to feed.
+
+           Call with None or an empty string to flush the mode of
+           operation and return any final bytes; no further calls to
+           feed may be made.'''
+
+        if self._buffer is None:
+            raise ValueError('already finished feeder')
+
+        # Finalize; process the spare bytes we were keeping
+        if data is None:
+            result = self._final(self._buffer, self._padding)
+            self._buffer = None
+            return result
+
+        self._buffer += to_bufferable(data)
+
+        # We keep 16 bytes around so we can determine padding
+        result = to_bufferable('')
+        while len(self._buffer) > 16:
+            can_consume = self._mode._can_consume(len(self._buffer) - 16)
+            if can_consume == 0: break
+            result += self._feed(self._buffer[:can_consume])
+            self._buffer = self._buffer[can_consume:]
+
+        return result
+
+
+class Encrypter(BlockFeeder):
+    'Accepts bytes of plaintext and returns encrypted ciphertext.'
+
+    def __init__(self, mode, padding = PADDING_DEFAULT):
+        BlockFeeder.__init__(self, mode, mode.encrypt, mode._final_encrypt, padding)
+
+
+class Decrypter(BlockFeeder):
+    'Accepts bytes of ciphertext and returns decrypted plaintext.'
+
+    def __init__(self, mode, padding = PADDING_DEFAULT):
+        BlockFeeder.__init__(self, mode, mode.decrypt, mode._final_decrypt, padding)
+
+
+# 8kb blocks
+BLOCK_SIZE = (1 << 13)
+
+def _feed_stream(feeder, in_stream, out_stream, block_size = BLOCK_SIZE):
+    'Uses feeder to read and convert from in_stream and write to out_stream.'
+
+    while True:
+        chunk = in_stream.read(block_size)
+        if not chunk:
+            break
+        converted = feeder.feed(chunk)
+        out_stream.write(converted)
+    converted = feeder.feed()
+    out_stream.write(converted)
+
+
+def encrypt_stream(mode, in_stream, out_stream, block_size = BLOCK_SIZE, padding = PADDING_DEFAULT):
+    'Encrypts a stream of bytes from in_stream to out_stream using mode.'
+
+    encrypter = Encrypter(mode, padding = padding)
+    _feed_stream(encrypter, in_stream, out_stream, block_size)
+
+
+def decrypt_stream(mode, in_stream, out_stream, block_size = BLOCK_SIZE, padding = PADDING_DEFAULT):
+    'Decrypts a stream of bytes from in_stream to out_stream using mode.'
+
+    decrypter = Decrypter(mode, padding = padding)
+    _feed_stream(decrypter, in_stream, out_stream, block_size)

Contents/Libraries/Shared/pyaes/util.py

@@ -0,0 +1,60 @@
+# The MIT License (MIT)
+#
+# Copyright (c) 2014 Richard Moore
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+# THE SOFTWARE.
+
+# Why to_bufferable?
+# Python 3 is very different from Python 2.x when it comes to strings of text
+# and strings of bytes; in Python 3, strings of bytes do not exist, instead to
+# represent arbitrary binary data, we must use the "bytes" object. This method
+# ensures the object behaves as we need it to.
+
+def to_bufferable(binary):
+    return binary
+
+def _get_byte(c):
+    return ord(c)
+
+try:
+    xrange
+except:
+
+    def to_bufferable(binary):
+        if isinstance(binary, bytes):
+            return binary
+        return bytes(ord(b) for b in binary)
+
+    def _get_byte(c):
+        return c
+
+def append_PKCS7_padding(data):
+    pad = 16 - (len(data) % 16)
+    return data + to_bufferable(chr(pad) * pad)
+
+def strip_PKCS7_padding(data):
+    if len(data) % 16 != 0:
+        raise ValueError("invalid length")
+
+    pad = _get_byte(data[-1])
+
+    if pad > 16:
+        raise ValueError("invalid padding byte")
+
+    return data[:-pad]

Contents/Libraries/Shared/pysubs2/exceptions.py

@@ -12,3 +12,6 @@ class UnknownFormatIdentifierError(Pysubs2Error):
 
 class FormatAutodetectionError(Pysubs2Error):
     """Subtitle format is ambiguous or unknown."""
+
+class ContentNotUsable(Pysubs2Error):
+    """Current content not usable for specified format"""

Contents/Libraries/Shared/pysubs2/ssastyle.py

@@ -41,6 +41,7 @@ class SSAStyle(object):
         self.italic = False #: Italic
         self.underline = False #: Underline (ASS only)
         self.strikeout = False #: Strikeout (ASS only)
+        self.drawing = False #: Drawing (ASS only, see http://docs.aegisub.org/3.1/ASS_Tags/#drawing-tags
         self.scalex = 100.0 #: Horizontal scaling (ASS only)
         self.scaley = 100.0 #: Vertical scaling (ASS only)
         self.spacing = 0.0 #: Letter spacing (ASS only)

Contents/Libraries/Shared/pysubs2/subrip.py

@@ -5,6 +5,7 @@ from .formatbase import FormatBase
 from .ssaevent import SSAEvent
 from .ssastyle import SSAStyle
 from .substation import parse_tags
+from .exceptions import ContentNotUsable
 from .time import ms_to_times, make_time, TIMESTAMP, timestamp_to_ms
 
 #: Largest timestamp allowed in SubRip, ie. 99:59:59,999.
@@ -81,6 +82,7 @@ class SubripFormat(FormatBase):
                 if sty.italic: fragment = "<i>%s</i>" % fragment
                 if sty.underline: fragment = "<u>%s</u>" % fragment
                 if sty.strikeout: fragment = "<s>%s</s>" % fragment
+                if sty.drawing: raise ContentNotUsable
                 body.append(fragment)
 
             return re.sub("\n+", "\n", "".join(body).strip())
@@ -90,7 +92,10 @@ class SubripFormat(FormatBase):
         for i, line in enumerate(visible_lines, 1):
             start = ms_to_timestamp(line.start)
             end = ms_to_timestamp(line.end)
-            text = prepare_text(line.text, subs.styles.get(line.style, SSAStyle.DEFAULT_STYLE))
+            try:
+                text = prepare_text(line.text, subs.styles.get(line.style, SSAStyle.DEFAULT_STYLE))
+            except ContentNotUsable:
+                continue
 
             print("%d" % i, file=fp) # Python 2.7 compat
             print(start, "-->", end, file=fp)

Contents/Libraries/Shared/pysubs2/substation.py

@@ -110,7 +110,7 @@ def parse_tags(text, style=SSAStyle.DEFAULT_STYLE, styles={}):
     
     def apply_overrides(all_overrides):
         s = style.copy()
-        for tag in re.findall(r"\\[ibus][10]|\\r[a-zA-Z_0-9 ]*", all_overrides):
+        for tag in re.findall(r"\\[ibusp][0-9]|\\r[a-zA-Z_0-9 ]*", all_overrides):
             if tag == r"\r":
                 s = style.copy() # reset to original line style
             elif tag.startswith(r"\r"):
@@ -122,6 +122,13 @@ def parse_tags(text, style=SSAStyle.DEFAULT_STYLE, styles={}):
                 elif "b" in tag: s.bold = "1" in tag
                 elif "u" in tag: s.underline = "1" in tag
                 elif "s" in tag: s.strikeout = "1" in tag
+                elif "p" in tag:
+                    try:
+                        scale = int(tag[2:])
+                    except (ValueError, IndexError):
+                        continue
+
+                    s.drawing = scale > 0
         return s
     
     overrides = SSAEvent.OVERRIDE_SEQUENCE.findall(text)

Contents/Libraries/Shared/subliminal/exceptions.py

@@ -27,3 +27,8 @@ class ServiceUnavailable(ProviderError):
 class DownloadLimitExceeded(ProviderError):
     """Exception raised by providers when download limit is exceeded."""
     pass
+
+
+class DownloadLimitPerDayExceeded(ProviderError):
+    """Exception raised by providers when download limit is exceeded."""
+    pass

Contents/Libraries/Shared/subliminal/extensions.py

@@ -95,7 +95,8 @@ provider_manager = RegistrableExtensionManager('subliminal.providers', [
     'shooter = subliminal.providers.shooter:ShooterProvider',
     'thesubdb = subliminal.providers.thesubdb:TheSubDBProvider',
     'tvsubtitles = subliminal.providers.tvsubtitles:TVsubtitlesProvider',
-    'screwzira = subliminal.providers.screwzira:ScrewZiraProvider'
+    'ktuvit = subliminal.providers.ktuvit:KtuvitProvider'
+    'wizdom = subliminal.providers.wizdom:WizdomProvider'
 ])
 
 #: Refiner manager

Contents/Libraries/Shared/subliminal_patch/core.py

@@ -108,10 +108,12 @@ class SZProviderPool(ProviderPool):
         try:
             logger.info('Terminating provider %s', name)
             self.initialized_providers[name].terminate()
-        except (requests.Timeout, socket.timeout):
+        except (requests.Timeout, socket.timeout) as e:
             logger.error('Provider %r timed out, improperly terminated', name)
-        except:
+            self.throttle_callback(name, e)
+        except Exception as e:
             logger.exception('Provider %r terminated unexpectedly', name)
+            self.throttle_callback(name, e)
 
         del self.initialized_providers[name]
 
@@ -183,8 +185,9 @@ class SZProviderPool(ProviderPool):
 
             return out
 
-        except (requests.Timeout, socket.timeout):
-            logger.error('Provider %r timed out', provider)
+        except (requests.Timeout, socket.timeout) as e:
+            logger.exception('Provider %r timed out', provider)
+            self.throttle_callback(provider, e)
 
         except Exception as e:
             logger.exception('Unexpected error in provider %r: %s', provider, traceback.format_exc())
@@ -263,10 +266,11 @@ class SZProviderPool(ProviderPool):
                     requests.exceptions.ProxyError,
                     requests.exceptions.SSLError,
                     requests.Timeout,
-                    socket.timeout):
-                logger.error('Provider %r connection error', subtitle.provider_name)
+                    socket.timeout) as e:
+                logger.exception('Provider %r connection error', subtitle.provider_name)
+                self.throttle_callback(subtitle.provider_name, e)
 
-            except ResponseNotReady:
+            except ResponseNotReady as e:
                 logger.error('Provider %r response error, reinitializing', subtitle.provider_name)
                 try:
                     self[subtitle.provider_name].terminate()
@@ -274,6 +278,7 @@ class SZProviderPool(ProviderPool):
                 except:
                     logger.error('Provider %r reinitialization error: %s', subtitle.provider_name,
                                  traceback.format_exc())
+                    self.throttle_callback(subtitle.provider_name, e)
 
             except rarfile.BadRarFile:
                 logger.error('Malformed RAR file from provider %r, skipping subtitle.', subtitle.provider_name)
@@ -354,15 +359,16 @@ class SZProviderPool(ProviderPool):
             orig_matches = matches.copy()
 
             logger.debug('%r: Found matches %r', s, matches)
+            score, score_without_hash = compute_score(matches, s, video, hearing_impaired=use_hearing_impaired)
             unsorted_subtitles.append(
-                (s, compute_score(matches, s, video, hearing_impaired=use_hearing_impaired), matches, orig_matches))
+                (s, score, score_without_hash, matches, orig_matches))
 
         # sort subtitles by score
-        scored_subtitles = sorted(unsorted_subtitles, key=operator.itemgetter(1), reverse=True)
+        scored_subtitles = sorted(unsorted_subtitles, key=operator.itemgetter(1, 2), reverse=True)
 
         # download best subtitles, falling back on the next on error
         downloaded_subtitles = []
-        for subtitle, score, matches, orig_matches in scored_subtitles:
+        for subtitle, score, score_without_hash, matches, orig_matches in scored_subtitles:
             # check score
             if score < min_score:
                 logger.info('%r: Score %d is below min_score (%d)', subtitle, score, min_score)
@@ -523,7 +529,7 @@ def scan_video(path, dont_use_actual_file=False, hints=None, providers=None, ski
     video.hints = hints
 
     # get possibly alternative title from the filename itself
-    alt_guess = guessit(filename, options=hints)
+    alt_guess = guessit(filename, options={k: v for k, v in hints.items() if k not in ('expected_title', 'title')})
     if "title" in alt_guess and alt_guess["title"] != guessed_result["title"]:
         if video_type == "episode":
             video.alternative_series.append(alt_guess["title"])
@@ -548,6 +554,9 @@ def scan_video(path, dont_use_actual_file=False, hints=None, providers=None, ski
             if "opensubtitles" in providers:
                 video.hashes['opensubtitles'] = osub_hash = osub_hash or hash_opensubtitles(hash_path)
 
+            if "opensubtitlescom" in providers:
+                video.hashes['opensubtitlescom'] = osub_hash = osub_hash or hash_opensubtitles(hash_path)
+
             if "shooter" in providers:
                 video.hashes['shooter'] = hash_shooter(hash_path)
 

Contents/Libraries/Shared/subliminal_patch/http.py

@@ -35,12 +35,8 @@ except ImportError:
 from subzero.lib.io import get_viable_encoding
 
 logger = logging.getLogger(__name__)
-pem_file = os.path.normpath(os.path.join(os.path.dirname(os.path.realpath(unicode(__file__, get_viable_encoding()))), "..", certifi.where()))
-try:
-    default_ssl_context = ssl.create_default_context(cafile=pem_file)
-except AttributeError:
-    # < Python 2.7.9
-    default_ssl_context = None
+pem_file = os.path.normpath(os.path.join(os.path.dirname(os.path.realpath(unicode(__file__, get_viable_encoding()))),
+                                         "..", certifi.where()))
 
 
 class TimeoutSession(requests.Session):
@@ -58,9 +54,9 @@ class TimeoutSession(requests.Session):
 
 
 class CertifiSession(TimeoutSession):
-    def __init__(self):
+    def __init__(self, verify=None):
         super(CertifiSession, self).__init__()
-        self.verify = pem_file
+        self.verify = verify or pem_file
 
 
 class NeedsCaptchaException(Exception):
@@ -241,12 +237,20 @@ class SubZeroRequestsTransport(xmlrpclib.SafeTransport):
     # change our user agent to reflect Requests
     user_agent = "Python XMLRPC with Requests (python-requests.org)"
     proxies = None
+    xm_ver = 1
+    session_var = "PHPSESSID"
 
     def __init__(self, use_https=True, verify=None, user_agent=None, timeout=10, *args, **kwargs):
         self.verify = pem_file if verify is None else verify
         self.use_https = use_https
         self.user_agent = user_agent if user_agent is not None else self.user_agent
         self.timeout = timeout
+        self.session = requests.Session()
+        self.session.headers['User-Agent'] = self.user_agent
+        # if 'requests' in self.session.headers['User-Agent']:
+        #     # Set a random User-Agent if no custom User-Agent has been set
+        #     self.session.headers = User_Agent(allow_brotli=False).headers
+
         proxy = os.environ.get('SZ_HTTP_PROXY')
         if proxy:
             self.proxies = {
@@ -260,18 +264,40 @@ class SubZeroRequestsTransport(xmlrpclib.SafeTransport):
         """
         Make an xmlrpc request.
         """
-        headers = {'User-Agent': self.user_agent}
         url = self._build_url(host, handler)
+        cache_key = "xm%s_%s" % (self.xm_ver, host)
+
+        old_sessvar = self.session.cookies.get(self.session_var, "")
+        if not old_sessvar:
+            data = region.get(cache_key)
+            if data is not NO_VALUE:
+                logger.debug("Trying to re-use headers/cookies for %s" % host)
+                self.session.cookies, self.session.headers = data
+                old_sessvar = self.session.cookies.get(self.session_var, "")
+
         try:
-            resp = requests.post(url, data=request_body, headers=headers,
-                                 stream=True, timeout=self.timeout, proxies=self.proxies,
-                                 verify=self.verify)
+            resp = self.session.post(url, data=request_body,
+                                     stream=True, timeout=self.timeout, proxies=self.proxies,
+                                     verify=self.verify)
+
+            if self.session_var in resp.cookies and resp.cookies[self.session_var] != old_sessvar:
+                logger.debug("Storing %s cookies" % host)
+                region.set(cache_key, [self.session.cookies, self.session.headers])
         except ValueError:
+            logger.debug("Wiping cookies/headers cache (VE) for %s" % host)
+            region.delete(cache_key)
             raise
         except Exception:
+            logger.debug("Wiping cookies/headers cache (EX) for %s" % host)
+            region.delete(cache_key)
             raise  # something went wrong
         else:
-            resp.raise_for_status()
+            try:
+                resp.raise_for_status()
+            except requests.exceptions.HTTPError:
+                logger.debug("Wiping cookies/headers cache (RE) for %s" % host)
+                region.delete(cache_key)
+                raise
 
             try:
                 if 'x-ratelimit-remaining' in resp.headers and int(resp.headers['x-ratelimit-remaining']) <= 2:

Contents/Libraries/Shared/subliminal_patch/providers/addic7ed.py

@@ -2,6 +2,8 @@
 import logging
 import re
 import datetime
+import types
+
 import subliminal
 import time
 
@@ -10,7 +12,8 @@ from random import randint
 from dogpile.cache.api import NO_VALUE
 from requests import Session
 from subliminal.cache import region
-from subliminal.exceptions import DownloadLimitExceeded, AuthenticationError, ConfigurationError
+from subliminal.exceptions import DownloadLimitExceeded, AuthenticationError, ConfigurationError, \
+    DownloadLimitPerDayExceeded
 from subliminal.providers.addic7ed import Addic7edProvider as _Addic7edProvider, \
     Addic7edSubtitle as _Addic7edSubtitle, ParserBeautifulSoup
 from subliminal.subtitle import fix_line_ending
@@ -64,6 +67,7 @@ class Addic7edProvider(_Addic7edProvider):
         'slk', 'slv', 'spa', 'sqi', 'srp', 'swe', 'tha', 'tur', 'ukr', 'vie', 'zho'
     ]} | {Language.fromietf(l) for l in ["sr-Latn", "sr-Cyrl"]}
 
+    vip = False
     USE_ADDICTED_RANDOM_AGENTS = False
     hearing_impaired_verifiable = True
     subtitle_class = Addic7edSubtitle
@@ -72,9 +76,10 @@ class Addic7edProvider(_Addic7edProvider):
     sanitize_characters = {'-', ':', '(', ')', '.', '/'}
     last_show_ids_fetch_key = "addic7ed_last_id_fetch"
 
-    def __init__(self, username=None, password=None, use_random_agents=False):
+    def __init__(self, username=None, password=None, use_random_agents=False, is_vip=False):
         super(Addic7edProvider, self).__init__(username=username, password=password)
         self.USE_ADDICTED_RANDOM_AGENTS = use_random_agents
+        self.vip = is_vip
 
         if not all((username, password)):
             raise ConfigurationError('Username and password must be specified')
@@ -397,6 +402,27 @@ class Addic7edProvider(_Addic7edProvider):
         return subtitles
 
     def download_subtitle(self, subtitle):
+        last_dls = region.get("addic7ed_dls")
+        now = datetime.datetime.now()
+        one_day = datetime.timedelta(hours=24)
+
+        def raise_limit():
+            logger.info("Addic7ed: Downloads per day exceeded (%s)", cap)
+            raise DownloadLimitPerDayExceeded
+
+        if not isinstance(last_dls, types.ListType):
+            last_dls = []
+        else:
+            # filter all non-expired DLs
+            last_dls = filter(lambda t: t + one_day > now, last_dls)
+            region.set("addic7ed_dls", last_dls)
+
+        cap = self.vip and 80 or 40
+        amount = len(last_dls)
+
+        if amount >= cap:
+            raise_limit()
+
         # download the subtitle
         r = self.session.get(self.server_url + subtitle.download_link, headers={'Referer': subtitle.page_link},
                              timeout=10)
@@ -408,7 +434,7 @@ class Addic7edProvider(_Addic7edProvider):
         if not r.content:
             # Provider wrongful return a status of 304 Not Modified with an empty content
             # raise_for_status won't raise exception for that status code
-            logger.error('Unable to download subtitle. No data returned from provider')
+            logger.error('Addic7ed: Unable to download subtitle. No data returned from provider')
             return
 
         # detect download limit exceeded
@@ -416,3 +442,10 @@ class Addic7edProvider(_Addic7edProvider):
             raise DownloadLimitExceeded
 
         subtitle.content = fix_line_ending(r.content)
+        last_dls.append(datetime.datetime.now())
+        region.set("addic7ed_dls", last_dls)
+        logger.info("Addic7ed: Used %s/%s downloads", amount + 1, cap)
+
+        if amount + 1 >= cap:
+            raise_limit()
+

Contents/Libraries/Shared/subliminal_patch/providers/bsplayer.py

@@ -19,9 +19,11 @@ from xml.etree import ElementTree
 
 logger = logging.getLogger(__name__)
 
+
 class BSPlayerSubtitle(Subtitle):
     """BSPlayer Subtitle."""
     provider_name = 'bsplayer'
+    hash_verifiable = True
 
     def __init__(self, language, filename, subtype, video, link):
         super(BSPlayerSubtitle, self).__init__(language)
@@ -41,27 +43,12 @@ class BSPlayerSubtitle(Subtitle):
 
     def get_matches(self, video):
         matches = set()
-
-        video_filename = video.name
-        video_filename = os.path.basename(video_filename)
-        video_filename, _ = os.path.splitext(video_filename)
-        video_filename = sanitize_release_group(video_filename)
-
-        subtitle_filename = self.filename
-        subtitle_filename = os.path.basename(subtitle_filename)
-        subtitle_filename, _ = os.path.splitext(subtitle_filename)
-        subtitle_filename = sanitize_release_group(subtitle_filename)
-
-
         matches |= guess_matches(video, guessit(self.filename))
-
-        matches.add(id(self))
         matches.add('hash')
 
         return matches
 
 
-
 class BSPlayerProvider(Provider):
     """BSPlayer Provider."""
     languages = {Language('por', 'BR')} | {Language(l) for l in [
@@ -69,6 +56,7 @@ class BSPlayerProvider(Provider):
         'ron', 'rus', 'spa', 'swe', 'tur', 'ukr', 'zho'
     ]}
     SEARCH_THROTTLE = 8
+    hash_verifiable = True
 
     # batantly based on kodi's bsplayer plugin
     # also took from BSPlayer-Subtitles-Downloader
@@ -108,18 +96,11 @@ class BSPlayerProvider(Provider):
                 res = self.session.post(self.search_url, data)
                 return ElementTree.fromstring(res.text)
 
-                ### with requests
-                # res = requests.post(
-                #     url=self.search_url,
-                #     data=data,
-                #     headers=headers
-                # ) 
-                # return ElementTree.fromstring(res.text)
-               
             except Exception as ex:
                 logger.info("ERROR: %s." % ex)
                 if func_name == 'logIn':
                     self.search_url = self.get_sub_domain()
+
                 sleep(1)
         logger.info('ERROR: Too many tries (%d)...' % tries)
         raise Exception('Too many tries...')
@@ -167,7 +148,6 @@ class BSPlayerProvider(Provider):
             # language_ids = 'spa'
             language_ids = ','.join(sorted(l.opensubtitles for l in language))
 
-
         if video.imdb_id is None:
             imdbId = '*'
         else:
@@ -193,13 +173,13 @@ class BSPlayerProvider(Provider):
         if items:
             logger.info("Subtitles Found.")
             for item in items:
-                subID=item.find('subID').text
-                subDownloadLink=item.find('subDownloadLink').text
-                subLang= Language.fromopensubtitles(item.find('subLang').text)
-                subName=item.find('subName').text
-                subFormat=item.find('subFormat').text
+                subID = item.find('subID').text
+                subDownloadLink = item.find('subDownloadLink').text
+                subLang = Language.fromopensubtitles(item.find('subLang').text)
+                subName = item.find('subName').text
+                subFormat = item.find('subFormat').text
                 subtitles.append(
-                    BSPlayerSubtitle(subLang,subName, subFormat, video, subDownloadLink)
+                    BSPlayerSubtitle(subLang, subName, subFormat, video, subDownloadLink)
                 )
         return subtitles
 
@@ -207,9 +187,9 @@ class BSPlayerProvider(Provider):
         return self.query(video, video.hashes['bsplayer'], languages)
 
     def get_sub_domain(self):
-    # s1-9, s101-109
+        # s1-9, s101-109
         SUB_DOMAINS = ['s1', 's2', 's3', 's4', 's5', 's6', 's7', 's8', 's9',
-                's101', 's102', 's103', 's104', 's105', 's106', 's107', 's108', 's109']
+                       's101', 's102', 's103', 's104', 's105', 's106', 's107', 's108', 's109']
         API_URL_TEMPLATE = "http://{sub_domain}.api.bsplayer-subtitles.com/v1.php"
         sub_domains_end = len(SUB_DOMAINS) - 1
         return API_URL_TEMPLATE.format(sub_domain=SUB_DOMAINS[random.randint(0, sub_domains_end)])
@@ -226,10 +206,8 @@ class BSPlayerProvider(Provider):
                 raise ValueError('Error 500 on server')
 
             with gzip.GzipFile(fileobj=io.BytesIO(res.content)) as gf:
-                subtitle.content = gf.read() 
+                subtitle.content = gf.read()
                 subtitle.normalize()
 
             return subtitle
         raise ValueError('Problems conecting to the server')
-
-

Contents/Libraries/Shared/subliminal_patch/providers/ktuvit.py

@@ -1,5 +1,7 @@
 # -*- coding: utf-8 -*-
+import re
 from requests import Session
+from requests.cookies import RequestsCookieJar
 import json
 import logging
 from subzero.language import Language
@@ -11,17 +13,25 @@ from subliminal.providers import Episode, Movie
 from subliminal_patch.utils import sanitize
 from subliminal_patch.subtitle import Subtitle, guess_matches
 from subliminal.subtitle import fix_line_ending
+from subliminal.exceptions import ConfigurationError, AuthenticationError
+
+from string import hexdigits
+from collections import defaultdict
+import pbkdf2
+from base64 import b64decode, b64encode
+from hashlib import sha256
+import pyaes
 
 __author__ = "Dor Nizar"
 
 logger = logging.getLogger(__name__)
 
 
-class ScrewZiraSubtitle(Subtitle):
-    provider_name = 'screwzira'
+class KtuvitSubtitle(Subtitle):
+    provider_name = 'ktuvit'
 
     def __init__(self, language, title_id, subtitle_id, series, season, episode, release, year):
-        super(ScrewZiraSubtitle, self).__init__(language, subtitle_id)
+        super(KtuvitSubtitle, self).__init__(language, subtitle_id)
         self.title_id = title_id
         self.subtitle_id = subtitle_id
         self.series = series
@@ -32,7 +42,7 @@ class ScrewZiraSubtitle(Subtitle):
 
     def get_matches(self, video):
         matches = set()
-        logger.debug("--ScrewZiraSubtitle--\n{}".format(self.__dict__))
+        logger.debug("[{}]\n{}".format(self.__class__.__name__, self.__dict__))
 
         # episode
         if isinstance(video, Episode):
@@ -59,7 +69,7 @@ class ScrewZiraSubtitle(Subtitle):
             # guess
             matches |= guess_matches(video, guessit(self.release, {'type': 'movie'}))
 
-        logger.debug("ScrewZira subtitle criteria match:\n{}".format(matches))
+        logger.debug("Ktuvit subtitle criteria match:\n{}".format(matches))
         return matches
 
     @property
@@ -67,30 +77,108 @@ class ScrewZiraSubtitle(Subtitle):
         return self.subtitle_id
 
 
-class ScrewZiraProvider(Provider):
-    subtitle_class = ScrewZiraSubtitle
+class KtuvitProvider(Provider):
+    subtitle_class = KtuvitSubtitle
     languages = {Language.fromalpha2(l) for l in ['he']}
-    URL_SERVER = 'https://www.screwzira.com/'
+    URL_SERVER = 'https://www.ktuvit.me/'
 
+    URI_LOGIN = 'Login.aspx'
+    URI_LOGIN_POST = 'Services/MembershipService.svc/Login'
     URI_SEARCH_TITLE = 'Services/ContentProvider.svc/GetSearchForecast'
     URI_SEARCH_SERIES_SUBTITLE = 'Services/GetModuleAjax.ashx'
     URI_SEARCH_MOVIE_SUBTITLE = "MovieInfo.aspx"
     URI_REQ_SUBTITLE_ID = "Services/ContentProvider.svc/RequestSubtitleDownload"
     URI_DOWNLOAD_SUBTITLE = "Services/DownloadFile.ashx"
 
+    def __init__(self, username=None, password=None):
+        if not all((username, password)):
+            raise ConfigurationError('Username and password must be specified')
+
+        self.session = None
+        self.username = username
+        self.password = password
+        self.encrypted_password = None
+        self.salt = None
+
+    def encrypt_password(self):
+        logger.debug("password: {}".format(self.password))
+        encrypted_password = KtuvitEncryptor(self.username, self.password, self.salt).encrypt()
+        if not encrypted_password:
+            logger.error("Could not encrypt password")
+            return False
+        self.encrypted_password = encrypted_password
+        return True
+
+    def get_encryption_salt(self):
+        p = re.compile(r"var encryptionSalt = '(.*)';")
+        r = self.session.get(self.URL_SERVER + self.URI_LOGIN)
+        r.raise_for_status()
+
+        logger.debug("Searching for encryptionSalt...")
+        script_list = [i for i in BeautifulSoup(r.content, 'html.parser').select('div#navbar script') if i.contents]
+        for item in script_list:
+            search_salt = p.search(item.contents[0])
+            if search_salt:
+                self.salt = search_salt.group(1)
+                return True
+        logger.error("Could not get encryptionSalt")
+        return False
+
+    def login(self):
+        if not self.get_encryption_salt():
+            return False
+        if not self.encrypt_password():
+            return False
+        data = {
+            "request": {
+                "Email": self.username,
+                "Password": self.encrypted_password
+            }
+        }
+        logger.debug("Trying to log in using: {}".format(json.dumps(data)))
+        r = self.session.post(self.URL_SERVER + self.URI_LOGIN_POST, json=data)
+        r_result = r.json()
+        login_results = ""
+        if 'd' in r_result:
+            try:
+                login_results = json.loads(r_result['d'])
+            except ValueError:
+                logger.error("Could not process JSON from login response")
+                return False
+
+        if 'IsSuccess' not in login_results:
+            logger.error("Login response is different than expected")
+            return False
+
+        if not login_results['IsSuccess']:
+            logger.error("Wrong username or password!")
+            raise AuthenticationError('Wrong username or password!')
+            return False
+
+        if not r.cookies or type(r.cookies) is not RequestsCookieJar:
+            logger.error("Could not get the cookie response of the login")
+            return False
+
+        if 'Login' not in r.cookies.keys():
+            logger.error("Could not found login cookie!")
+            return False
+
+        logger.info("Connected successfully to Ktuvit!")
+        return True
+
     def initialize(self):
-        logger.debug("ScrewZira initialize")
+        logger.debug("Ktuvit initialize")
         self.session = Session()
         self.session.headers[
             'User-Agent'] = 'Mozilla/5.0 (Windows NT 10.0; ' \
                             'Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36'
 
-    def terminate(self):
-        logger.debug("ScrewZira terminate")
-        self.session.close()
+        if not self.login():
+            return False
 
-    def __init__(self):
-        self.session = None
+    def terminate(self):
+        logger.debug("Ktuvit terminate")
+        self.session.close()
 
     def _search_series(self, title):
         logger.debug("Searching '{}'".format(title))
@@ -225,17 +313,105 @@ class ScrewZiraProvider(Provider):
         return [s for s in self.query(title, season, episode, year) if s.language in languages]
 
     def download_subtitle(self, subtitle):
-        # type: (ScrewZiraSubtitle) -> None
+        # type: (KtuvitSubtitle) -> None
 
-        logger.info('Downloading subtitle from ScrewZira: %r', subtitle)
-        downloadID = self._req_download_identifier(subtitle.title_id, subtitle.subtitle_id)
-        if not downloadID:
+        logger.info('Downloading subtitle from Ktuvit: %r', subtitle)
+        download_id = self._req_download_identifier(subtitle.title_id, subtitle.subtitle_id)
+        if not download_id:
             logger.debug('Unable to retrieve download identifier')
             return None
 
-        content = self._download_subtitles(downloadID)
+        content = self._download_subtitles(download_id)
         if not content:
             logger.debug('Unable to download subtitle')
             return None
 
         subtitle.content = fix_line_ending(content)
+
+
+class KtuvitEncryptor:
+    def __init__(self, username, password, salt):
+        if not all((username, password, salt)):
+            raise Exception("Encryptor did not get all required arguments")
+
+        self.encrypted_password = None
+        self.username = username
+        self.password = password
+        self.salt = salt
+
+    @staticmethod
+    def rshift(val, n):
+        return (val % 0x100000000) >> n
+
+    @staticmethod
+    def js_parseint(s, rad=10):
+        digits = ''
+        for c in str(s).strip():
+            if c not in hexdigits:
+                break
+            digits += c
+
+        return int(digits, rad) if digits else 0
+
+    @staticmethod
+    def to_signed32(n):
+        n = n & 0xffffffff
+        return n | (-(n & 0x80000000))
+
+    def stringify(self, words, length):
+        sigbytes = int(length / 2) + int((length % 2) > 0)
+        hex_chars = list()
+
+        for i in xrange(0, sigbytes):
+            bite = self.rshift(words[self.rshift(i, 2)], (24 - (i % 4) * 8)) & 0xff
+            hex_chars.append(format(self.rshift(bite, 4), 'x'))
+            hex_chars.append(format(bite & 0x0f, 'x'))
+        return ''.join(hex_chars)
+
+    def cryptojs_hexparse(self, s):
+        words = defaultdict(int)
+        for i in range(0, len(s), 2):
+            tmp1 = (self.js_parseint(s[i:i + 2], 16))
+            tmp2 = (24 - (i % 8) * 4)
+            tmp3 = self.to_signed32(tmp1 << tmp2)
+            words[self.rshift(i, 3)] |= tmp3
+        return self.stringify(words, len(s))
+
+    def cryptojs_pad_iv(self, iv):
+        return str.ljust(self.cryptojs_hexparse(iv), 32, '0').decode('hex')
+
+    @staticmethod
+    def pbkdf2_encrypt(key, salt):
+        a = pbkdf2.PBKDF2(salt, key, 3000)
+        return a.read(16)
+
+    @staticmethod
+    def pad(m):
+        return m + chr(16 - len(m) % 16) * (16 - len(m) % 16)
+
+    def aes_encrypt(self, msg, key, iv):
+        if len(iv) != 16:
+            logger.error("iv (Len: {}) - {} is not 16 length".format(len(iv), iv))
+            return False
+        msg = self.pad(msg)
+
+        aes = pyaes.AESModeOfOperationCBC(key, iv=iv[:16])
+        return b64encode(aes.encrypt(msg))
+
+    def encrypt(self):
+        if not self.salt:
+            logger.error("No salt was instantiated!")
+            return False
+        msg = self.password.encode('utf-8')
+        iv = self.cryptojs_pad_iv(self.username)
+        key = self.pbkdf2_encrypt(self.username, self.salt.encode('utf-8'))
+
+        cipher = self.aes_encrypt(msg, key, iv)
+        if not cipher:
+            return False
+
+        hash_sha256 = sha256(b64decode(cipher))
+        self.encrypted_password = b64encode(hash_sha256.digest())
+        logger.debug("Encrypted password: {}".format(self.encrypted_password))
+        logger.debug("Original password: {}".format(self.password))
+        return self.encrypted_password

Contents/Libraries/Shared/subliminal_patch/providers/opensubtitles.py

@@ -105,7 +105,7 @@ class OpenSubtitlesProvider(ProviderRetryMixin, _OpenSubtitlesProvider):
 
     def __init__(self, username=None, password=None, use_tag_search=False, only_foreign=False, also_foreign=False,
                  skip_wrong_fps=True, is_vip=False, use_ssl=True, timeout=15):
-        if any((username, password)) and not all((username, password)):
+        if not all((username, password)):
             raise ConfigurationError('Username and password must be specified')
 
         self.username = username or ''
@@ -154,6 +154,7 @@ class OpenSubtitlesProvider(ProviderRetryMixin, _OpenSubtitlesProvider):
         logger.debug('Logged in with token %r', self.token[:10]+"X"*(len(self.token)-10))
 
         region.set("os_token", self.token)
+        time.sleep(1)
 
     def use_token_or_login(self, func):
         if not self.token:
@@ -162,6 +163,7 @@ class OpenSubtitlesProvider(ProviderRetryMixin, _OpenSubtitlesProvider):
         try:
             return func()
         except Unauthorized:
+            logger.debug("Token not valid, logging in again")
             self.log_in()
             return func()
 
@@ -197,16 +199,11 @@ class OpenSubtitlesProvider(ProviderRetryMixin, _OpenSubtitlesProvider):
                     return
 
             logger.error("Login failed, please check your credentials")
+            raise
                 
     def terminate(self):
-        if self.token:
-            try:
-                checked(lambda: self.server.LogOut(self.token))
-            except:
-                logger.error("Logout failed: %s", traceback.format_exc())
-
         self.server = None
-        self.token = None
+        #self.token = None
 
     def list_subtitles(self, video, languages):
         """

Contents/Libraries/Shared/subliminal_patch/providers/opensubtitlescom.py

@@ -0,0 +1,579 @@
+# -*- coding: utf-8 -*-
+import logging
+import os
+import time
+import datetime
+import json
+import types
+
+from requests import Session, ConnectionError, Timeout, ReadTimeout, RequestException
+from simplejson import JSONDecodeError
+from subzero.language import Language
+
+from babelfish import language_converters
+from subliminal import Episode, Movie
+from subliminal.score import get_equivalent_release_groups
+from subliminal.utils import sanitize_release_group, sanitize
+from subliminal_patch.exceptions import TooManyRequests, APIThrottled
+from subliminal.exceptions import DownloadLimitExceeded, AuthenticationError, ConfigurationError, ServiceUnavailable, \
+    ProviderError
+from .mixins import ProviderRetryMixin
+from subliminal_patch.subtitle import Subtitle
+from subliminal.subtitle import fix_line_ending, SUBTITLE_EXTENSIONS
+from subliminal_patch.providers import Provider
+from subliminal_patch.subtitle import guess_matches
+from subliminal_patch.utils import fix_inconsistent_naming
+from subliminal.cache import region
+from dogpile.cache.api import NO_VALUE
+from guessit import guessit
+
+logger = logging.getLogger(__name__)
+
+SHOW_EXPIRATION_TIME = datetime.timedelta(weeks=1).total_seconds()
+TOKEN_EXPIRATION_TIME = datetime.timedelta(hours=12).total_seconds()
+
+retry_amount = 3
+
+
+def fix_tv_naming(title):
+    """Fix TV show titles with inconsistent naming using dictionary, but do not sanitize them.
+
+    :param str title: original title.
+    :return: new title.
+    :rtype: str
+
+    """
+    return fix_inconsistent_naming(title, {"Superman & Lois": "Superman and Lois",
+                                           }, True)
+
+
+def fix_movie_naming(title):
+    return fix_inconsistent_naming(title, {
+    }, True)
+
+
+custom_languages = {
+    'pt': 'pt-PT',
+    'zh': 'zh-CN',
+}
+
+
+def to_opensubtitlescom(lang):
+    if lang in custom_languages.keys():
+        return custom_languages[lang]
+    else:
+        return lang
+
+
+def from_opensubtitlescom(lang):
+    from_custom_languages = {v: k for k, v in custom_languages.items()}
+    if lang in from_custom_languages.keys():
+        return from_custom_languages[lang]
+    else:
+        return lang
+
+
+class OpenSubtitlesComSubtitle(Subtitle):
+    provider_name = 'opensubtitlescom'
+    hash_verifiable = True
+    hearing_impaired_verifiable = True
+
+    def __init__(self, language, forced, hearing_impaired, page_link, file_id, releases, uploader, title, year,
+                 hash_matched, file_hash=None, season=None, episode=None, imdb_match=False):
+        super(OpenSubtitlesComSubtitle, self).__init__(language, hearing_impaired, page_link)
+        language = Language.rebuild(language, hi=hearing_impaired, forced=forced)
+
+        self.title = title
+        self.year = year
+        self.season = season
+        self.episode = episode
+        self.releases = releases
+        self.release_info = releases
+        self.language = language
+        self.hearing_impaired = hearing_impaired
+        self.forced = forced
+        self.file_id = file_id
+        self.page_link = page_link
+        self.download_link = None
+        self.uploader = uploader
+        self.matches = None
+        self.hash = file_hash
+        self.encoding = 'utf-8'
+        self.hash_matched = hash_matched
+        self.imdb_match = imdb_match
+
+    @property
+    def id(self):
+        return self.file_id
+
+    def get_matches(self, video):
+        matches = set()
+        type_ = "movie" if isinstance(video, Movie) else "episode"
+
+        # handle movies and series separately
+        if type_ == "episode":
+            # series
+            matches.add('series')
+            # season
+            if video.season == self.season:
+                matches.add('season')
+            # episode
+            if video.episode == self.episode:
+                matches.add('episode')
+            # imdb
+            if self.imdb_match:
+                matches.add('series_imdb_id')
+        else:
+            # title
+            matches.add('title')
+            # imdb
+            if self.imdb_match:
+                matches.add('imdb_id')
+
+        # rest is same for both groups
+
+        # year
+        if video.year == self.year:
+            matches.add('year')
+
+        # release_group
+        if (video.release_group and self.releases and
+                any(r in sanitize_release_group(self.releases)
+                    for r in get_equivalent_release_groups(sanitize_release_group(video.release_group)))):
+            matches.add('release_group')
+
+        if self.hash_matched:
+            matches.add('hash')
+
+        # other properties
+        matches |= guess_matches(video, guessit(self.releases, {"type": type_}))
+
+        self.matches = matches
+
+        return matches
+
+
+class OpenSubtitlesComProvider(ProviderRetryMixin, Provider):
+    """OpenSubtitlesCom Provider"""
+    server_url = 'https://api.opensubtitles.com/api/v1/'
+
+    languages = {Language.fromopensubtitles(lang) for lang in language_converters['szopensubtitles'].codes}
+    languages.update(set(Language.rebuild(lang, forced=True) for lang in languages))
+    languages.update(set(Language.rebuild(l, hi=True) for l in languages))
+
+    video_types = (Episode, Movie)
+
+    def __init__(self, username=None, password=None, use_hash=True, include_ai_translated=False, api_key=None):
+        if not all((username, password)):
+            raise ConfigurationError('Username and password must be specified')
+
+        if not api_key:
+            raise ConfigurationError('Api_key must be specified')
+
+        if not all((username, password)):
+            raise ConfigurationError('Username and password must be specified')
+
+        self.session = Session()
+        self.session.headers = {'User-Agent': os.environ.get("SZ_USER_AGENT", "Sub-Zero/2"),
+                                'Api-Key': api_key,
+                                'Content-Type': 'application/json'}
+        self.token = None
+        self.username = username
+        self.password = password
+        self.video = None
+        self.use_hash = use_hash
+        self.include_ai_translated = include_ai_translated
+        self._started = None
+
+    def initialize(self):
+        self._started = time.time()
+
+        if region.get("oscom_token", expiration_time=TOKEN_EXPIRATION_TIME) is NO_VALUE:
+            logger.debug("No cached token, we'll try to login again.")
+            self.login()
+        else:
+            self.token = region.get("oscom_token", expiration_time=TOKEN_EXPIRATION_TIME)
+
+    def terminate(self):
+        self.session.close()
+
+    def ping(self):
+        return self._started and (time.time() - self._started) < TOKEN_EXPIRATION_TIME
+
+    def login(self, is_retry=False):
+        r = self.checked(
+            lambda: self.session.post(self.server_url + 'login',
+                                      json={"username": self.username, "password": self.password},
+                                      allow_redirects=False,
+                                      timeout=30),
+            is_retry=is_retry)
+
+        try:
+            self.token = r.json()['token']
+        except (ValueError, JSONDecodeError):
+            log_request_response(r)
+            raise ProviderError("Cannot get token from provider login response")
+        else:
+            log_request_response(r, non_standard=False)
+            region.set("oscom_token", self.token)
+
+    @staticmethod
+    def sanitize_external_ids(external_id):
+        if isinstance(external_id, types.StringTypes):
+            external_id = external_id.lower().lstrip('tt').lstrip('0')
+        sanitized_id = external_id[:-1].lstrip('0') + external_id[-1]
+        return int(sanitized_id)
+
+    @region.cache_on_arguments(expiration_time=SHOW_EXPIRATION_TIME)
+    def search_titles(self, title):
+        title_id = None
+
+        parameters = {'query': title.lower()}
+        logging.debug('Searching using this title: %s' % title)
+
+        results = self.retry(
+            lambda: self.checked(
+                lambda: self.session.get(self.server_url + 'features', params=parameters, timeout=30),
+                validate_json=True,
+                json_key_name='data'
+            ),
+            amount=retry_amount
+        )
+
+        # deserialize results
+        results_dict = results.json()['data']
+
+        # loop over results
+        for result in results_dict:
+            if 'title' in result['attributes']:
+                if isinstance(self.video, Episode):
+                    if fix_tv_naming(title).lower() == result['attributes']['title'].lower() and \
+                            (not self.video.year or self.video.year == int(result['attributes']['year'])):
+                        title_id = result['id']
+                        break
+                else:
+                    if fix_movie_naming(title).lower() == result['attributes']['title'].lower() and \
+                            (not self.video.year or self.video.year == int(result['attributes']['year'])):
+                        title_id = result['id']
+                        break
+            else:
+                continue
+
+        if title_id:
+            logging.debug('Found this title ID: %s' % title_id)
+            return self.sanitize_external_ids(title_id)
+
+        if not title_id:
+            logger.debug('No match found for %s' % title)
+
+    def query(self, languages, video):
+        self.video = video
+        if self.use_hash:
+            file_hash = self.video.hashes.get('opensubtitlescom')
+            logging.debug('Searching using this hash: %s' % hash)
+        else:
+            file_hash = None
+
+        if isinstance(self.video, Episode):
+            title = self.video.series
+        else:
+            title = self.video.title
+
+        imdb_id = None
+        if isinstance(self.video, Episode) and self.video.series_imdb_id:
+            imdb_id = self.sanitize_external_ids(self.video.series_imdb_id)
+        elif isinstance(self.video, Movie) and self.video.imdb_id:
+            imdb_id = self.sanitize_external_ids(self.video.imdb_id)
+
+        title_id = None
+        if not imdb_id:
+            title_id = self.search_titles(title)
+            if not title_id:
+                return []
+
+        # be sure to remove duplicates using list(set())
+        langs_list = sorted(list(set([to_opensubtitlescom(lang.basename).lower() for lang in languages])))
+
+        langs = ','.join(langs_list)
+        logging.debug('Searching for those languages: %s' % langs)
+
+        # query the server
+        if isinstance(self.video, Episode):
+            res = self.retry(
+                lambda: self.checked(
+                    lambda: self.session.get(self.server_url + 'subtitles',
+                                             params=(('ai_translated', 'exclude' if not self.include_ai_translated
+                                                     else 'include'),
+                                                     ('episode_number', self.video.episode),
+                                                     ('imdb_id', imdb_id if not title_id else None),
+                                                     ('languages', langs),
+                                                     ('moviehash', file_hash),
+                                                     ('parent_feature_id', title_id if title_id else None),
+                                                     ('season_number', self.video.season)),
+                                             timeout=30),
+                    validate_json=True,
+                    json_key_name='data'
+                ),
+                amount=retry_amount
+            )
+        else:
+            res = self.retry(
+                lambda: self.checked(
+                    lambda: self.session.get(self.server_url + 'subtitles',
+                                             params=(('ai_translated', 'exclude' if not self.include_ai_translated
+                                                     else 'include'),
+                                                     ('id', title_id if title_id else None),
+                                                     ('imdb_id', imdb_id if not title_id else None),
+                                                     ('languages', langs),
+                                                     ('moviehash', file_hash)),
+                                             timeout=30),
+                    validate_json=True,
+                    json_key_name='data'
+                ),
+                amount=retry_amount
+            )
+
+        subtitles = []
+
+        result = res.json()
+
+        # filter out forced subtitles or not depending on the required languages
+        if all([lang.forced for lang in languages]):  # only forced
+            result['data'] = [x for x in result['data'] if x['attributes']['foreign_parts_only']]
+        elif any([lang.forced for lang in languages]):  # also forced
+            pass
+        else:  # not forced
+            result['data'] = [x for x in result['data'] if not x['attributes']['foreign_parts_only']]
+
+        logging.debug("Query returned %s subtitles" % len(result['data']))
+
+        if len(result['data']):
+            for item in result['data']:
+                # ignore AI translated subtitles
+                if 'ai_translated' in item['attributes'] and item['attributes']['ai_translated']:
+                    logging.debug("Skipping AI translated subtitles")
+                    continue
+
+                # ignore machine translated subtitles
+                if 'machine_translated' in item['attributes'] and item['attributes']['machine_translated']:
+                    logging.debug("Skipping machine translated subtitles")
+                    continue
+
+                if 'season_number' in item['attributes']['feature_details']:
+                    season_number = item['attributes']['feature_details']['season_number']
+                else:
+                    season_number = None
+
+                if 'episode_number' in item['attributes']['feature_details']:
+                    episode_number = item['attributes']['feature_details']['episode_number']
+                else:
+                    episode_number = None
+
+                if 'moviehash_match' in item['attributes']:
+                    moviehash_match = item['attributes']['moviehash_match']
+                else:
+                    moviehash_match = False
+
+                try:
+                    year = int(item['attributes']['feature_details']['year'])
+                except TypeError:
+                    year = item['attributes']['feature_details']['year']
+
+                if len(item['attributes']['files']):
+                    subtitle = OpenSubtitlesComSubtitle(
+                        language=Language.fromietf(from_opensubtitlescom(item['attributes']['language'])),
+                        forced=item['attributes']['foreign_parts_only'],
+                        hearing_impaired=item['attributes']['hearing_impaired'],
+                        page_link=item['attributes']['url'],
+                        file_id=item['attributes']['files'][0]['file_id'],
+                        releases=item['attributes']['release'],
+                        uploader=item['attributes']['uploader']['name'],
+                        title=item['attributes']['feature_details']['movie_name'],
+                        year=year,
+                        season=season_number,
+                        episode=episode_number,
+                        hash_matched=moviehash_match,
+                        imdb_match=True if imdb_id else False
+                    )
+                    subtitle.get_matches(self.video)
+                    subtitles.append(subtitle)
+
+        return subtitles
+
+    def list_subtitles(self, video, languages):
+        return self.query(languages, video)
+
+    def download_subtitle(self, subtitle):
+        logger.info('Downloading subtitle %r', subtitle)
+
+        headers = {'Accept': 'application/json', 'Content-Type': 'application/json',
+                   'Authorization': 'Bearer ' + self.token}
+        res = self.retry(
+            lambda: self.checked(
+                lambda: self.session.post(self.server_url + 'download',
+                                          json={'file_id': subtitle.file_id, 'sub_format': 'srt'},
+                                          headers=headers,
+                                          timeout=30),
+                validate_json=True,
+                json_key_name='link'
+            ),
+            amount=retry_amount
+        )
+
+        download_data = res.json()
+        subtitle.download_link = download_data['link']
+
+        r = self.retry(
+            lambda: self.checked(
+                lambda: self.session.get(subtitle.download_link, timeout=30),
+                validate_content=True
+            ),
+            amount=retry_amount
+        )
+
+        if not r:
+            logger.debug('Could not download subtitle from %s' % subtitle.download_link)
+            subtitle.content = None
+            return
+        else:
+            subtitle_content = r.content
+            subtitle.content = fix_line_ending(subtitle_content)
+
+    @staticmethod
+    def reset_token():
+        logging.debug('Authentication failed: clearing cache and attempting to login.')
+        region.delete("oscom_token")
+        return
+
+    def checked(self, fn, raise_api_limit=False, validate_json=False, json_key_name=None, validate_content=False,
+                is_retry=False):
+        """Run :fn: and check the response status before returning it.
+
+        :param fn: the function to make an API call to OpenSubtitles.com.
+        :param raise_api_limit: if True we wait a little bit longer before running the call again.
+        :param validate_json: test if response is valid json.
+        :param json_key_name: test if returned json contain a specific key.
+        :param validate_content: test if response have a content (used with download).
+        :param is_retry: prevent additional retries with login endpoint.
+        :return: the response.
+
+        """
+        response = None
+        try:
+            try:
+                response = fn()
+            except APIThrottled:
+                if not raise_api_limit:
+                    logger.info("API request limit hit, waiting and trying again once.")
+                    time.sleep(15)
+                    return self.checked(fn, raise_api_limit=True)
+                raise
+            except (ConnectionError, Timeout, ReadTimeout):
+                raise ServiceUnavailable('Unknown Error, empty response: {}: {}'.format(response.status_code, response))
+            except Exception:
+                logging.exception('Unhandled exception raised.')
+                raise ProviderError('Unhandled exception raised. Check log.')
+            else:
+                status_code = response.status_code
+        except Exception:
+            status_code = None
+        else:
+            if status_code == 400:
+                try:
+                    json_response = response.json()
+                    message = json_response['message']
+                except JSONDecodeError:
+                    raise ProviderError('Invalid JSON returned by provider')
+                else:
+                    log_request_response(response)
+                    raise ConfigurationError(message)
+            elif status_code == 401:
+                log_request_response(response)
+                self.reset_token()
+                if is_retry:
+                    raise AuthenticationError('Login failed')
+                else:
+                    time.sleep(1)
+                    self.login(is_retry=True)
+                    self.checked(fn, raise_api_limit=raise_api_limit, validate_json=validate_json,
+                                 json_key_name=json_key_name, validate_content=validate_content, is_retry=True)
+            elif status_code == 403:
+                log_request_response(response)
+                raise ProviderError("Bazarr API key seems to be in problem")
+            elif status_code == 406:
+                try:
+                    json_response = response.json()
+                    download_count = json_response['requests']
+                    remaining_download = json_response['remaining']
+                    quota_reset_time = json_response['reset_time']
+                except JSONDecodeError:
+                    raise ProviderError('Invalid JSON returned by provider')
+                else:
+                    log_request_response(response)
+                    raise DownloadLimitExceeded("Daily download limit reached. {} subtitles have been "
+                                                "downloaded and {} remaining subtitles can be "
+                                                "downloaded. Quota will be reset in {}.".format(download_count, remaining_download, quota_reset_time))
+            elif status_code == 410:
+                log_request_response(response)
+                raise ProviderError("Download as expired")
+            elif status_code == 429:
+                log_request_response(response)
+                raise TooManyRequests()
+            elif status_code == 500:
+                logging.debug("Server side exception raised while downloading from opensubtitles.com website. They "
+                              "should mitigate this soon.")
+                return None
+            elif status_code == 502:
+                # this one should deal with Bad Gateway issue on their side.
+                raise APIThrottled()
+            elif 500 <= status_code <= 599:
+                raise ProviderError(response.reason)
+
+            if status_code != 200:
+                log_request_response(response)
+                raise ProviderError('Bad status code: %s' % response.status_code)
+
+            if validate_json:
+                try:
+                    json_test = response.json()
+                except JSONDecodeError:
+                    raise ProviderError('Invalid JSON returned by provider')
+                else:
+                    if json_key_name not in json_test:
+                        raise ProviderError('Invalid JSON returned by provider: no %s key in returned json.' % json_key_name)
+
+            if validate_content:
+                if not hasattr(response, 'content'):
+                    logging.error('Download link returned no content attribute.')
+                    return False
+                elif not response.content:
+                    logging.error('This download link returned empty content: %s' % response.url)
+                    return False
+
+        return response
+
+
+def log_request_response(response, non_standard=True):
+    redacted_request_headers = response.request.headers
+    if 'Authorization' in redacted_request_headers and isinstance(redacted_request_headers['Authorization'], str):
+        redacted_request_headers['Authorization'] = redacted_request_headers['Authorization'][:-8]+8*'x'
+
+    redacted_request_body = json.loads(response.request.body)
+    if 'password' in redacted_request_body:
+        redacted_request_body['password'] = 'redacted'
+
+    redacted_response_body = json.loads(response.text)
+    if 'token' in redacted_response_body and isinstance(redacted_response_body['token'], str):
+        redacted_response_body['token'] = redacted_response_body['token'][:-8] + 8 * 'x'
+
+    if non_standard:
+        logging.debug("opensubtitlescom returned a non standard response. Logging request/response for debugging "
+                      "purpose.")
+    else:
+        logging.debug("opensubtitlescom returned a standard response. Logging request/response for debugging purpose.")
+    logging.debug("Request URL: %s" % response.request.url)
+    logging.debug("Request Headers: %s" % redacted_request_headers)
+    logging.debug("Request Body: %s" % json.dumps(redacted_request_body))
+    logging.debug("Response Status Code: %s" % {response.status_code})
+    logging.debug("Response Headers: %s" % response.headers)
+    logging.debug("Response Body: %s" % json.dumps(redacted_response_body))

Contents/Libraries/Shared/subliminal_patch/providers/supersubtitles.py

@@ -143,7 +143,7 @@ class SuperSubtitlesProvider(Provider, ProviderSubtitleArchiveMixin):
     ]}
     video_types = (Episode, Movie)
     # https://www.feliratok.info/?search=&soriSorszam=&nyelv=&sorozatnev=The+Flash+%282014%29&sid=3212&complexsearch=true&knyelv=0&evad=4&epizod1=1&cimke=0&minoseg=0&rlsr=0&tab=all
-    server_url = 'https://www.feliratok.info/'
+    server_url = 'https://www.feliratok.eu/'
     subtitle_class = SuperSubtitlesSubtitle
     hearing_impaired_verifiable = False
     multi_result_throttle = 2  # seconds

Contents/Libraries/Shared/subliminal_patch/providers/wizdom.py

@@ -0,0 +1,235 @@
+# -*- coding: utf-8 -*-
+from requests import Session
+import logging
+from subzero.language import Language
+from guessit import guessit
+
+from subliminal_patch.providers import Provider
+from subliminal.providers import Episode, Movie
+from subliminal_patch.utils import sanitize
+from subliminal_patch.subtitle import Subtitle, guess_matches
+from subliminal.subtitle import fix_line_ending
+
+from io import BytesIO
+from zipfile import ZipFile
+
+__author__ = "Dor Nizar"
+
+logger = logging.getLogger(__name__)
+
+
+class WizdomSubtitle(Subtitle):
+    provider_name = 'wizdom'
+
+    def __init__(self, language, title_id, subtitle_id, series, season, episode, release, year, page_link):
+        super(WizdomSubtitle, self).__init__(language, subtitle_id)
+        self.title_id = title_id
+        self.subtitle_id = subtitle_id
+        self.series = series
+        self.season = season
+        self.episode = episode
+        self.release = release
+        self.year = year
+        self.page_link = page_link
+
+    def get_matches(self, video):
+        matches = set()
+        logger.debug("[{}]\n{}".format(self.__class__.__name__, self.__dict__))
+
+        # episode
+        if isinstance(video, Episode):
+            # series
+            if video.series and sanitize(self.series) == sanitize(video.series):
+                matches.add('series')
+            # season
+            if video.season and self.season == video.season:
+                matches.add('season')
+            # episode
+            if video.episode and self.episode == video.episode:
+                matches.add('episode')
+            # guess
+            matches |= guess_matches(video, guessit(self.release, {'type': 'episode'}))
+        # movie
+        elif isinstance(video, Movie):
+            # title
+            if video.title and (sanitize(self.series) in (
+                    sanitize(name) for name in [video.title] + video.alternative_titles)):
+                matches.add('title')
+            # year
+            if video.year and self.year == video.year:
+                matches.add('year')
+            # guess
+            matches |= guess_matches(video, guessit(self.release, {'type': 'movie'}))
+
+        logger.debug("Wizdom subtitle criteria match:\n{}".format(matches))
+        return matches
+
+    @property
+    def id(self):
+        return self.subtitle_id
+
+
+class WizdomProvider(Provider):
+    subtitle_class = WizdomSubtitle
+    languages = {Language.fromalpha2(lng) for lng in ['he']}
+    URL_JSON_SERVER = 'https://json.wizdom.xyz/'
+    URL_DOWNLOAD_SERVER = 'https://zip.wizdom.xyz/'
+    URL_WIZDOM_PAGELINK = 'https://wizdom.xyz/{}/{}'
+
+    URL_SEARCH = URL_JSON_SERVER + "search.php?search={}"
+    URL_INFO = URL_JSON_SERVER + "{}.json"
+    URL_DOWNLOAD_SUBTITLE = URL_DOWNLOAD_SERVER + "{}.zip"
+
+    def __init__(self):
+        self.session = None
+
+    def initialize(self):
+        logger.info("Wizdom initialize")
+        self.session = Session()
+        self.session.headers[
+            'User-Agent'] = 'Mozilla/5.0 (Windows NT 10.0; ' \
+                            'Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36'
+        return True
+
+    def terminate(self):
+        logger.info("Wizdom terminate")
+        self.session.close()
+
+    def _search_series(self, title, movie=False):
+        logger.debug("Searching '{}'".format(title))
+        r = self.session.get(self.URL_SEARCH.format(title))
+        if not r.ok:
+            logger.error("Bad response from server while searching: '{}']".format(title))
+            return []
+        try:
+            found = r.json()
+        except ValueError:
+            logger.error("Could not extract JSON from response")
+            return []
+
+        if movie:
+            series_list = [x for x in found if 'type' in x and x['type'] == 'movie']
+        else:
+            series_list = [x for x in found if 'type' in x and x['type'] == 'tv']
+
+        logger.debug("Found the following titles: {}".format([x['title_en'] for x in series_list if 'title_en' in x]))
+        return series_list
+
+    def _search_subtitles(self, title_id, season=None, episode=None):
+        r = self.session.get(self.URL_INFO.format(title_id))
+
+        if not r.ok:
+            logger.error("Bad response from server while searching subtitles [title_id={}]".format(title_id))
+            return []
+        if not r.content:
+            return []
+        try:
+            results = r.json()
+        except ValueError:
+            return []
+
+        if 'subs' not in results:
+            return []
+
+        if season and episode:
+            s_season = str(season)
+            s_episode = str(episode)
+            if s_season in results['subs'] and s_episode in results['subs'][s_season]:
+                return results['subs'][s_season][s_episode]
+        else:
+            return results['subs']
+        return []
+
+    def query(self, title, season=None, episode=None, year=None):
+        subtitles = []
+        if season and episode:
+            logger.debug("Searching for:\nTitle: {}\nSeason: {}\nEpisode: {}\nYear: {}".format(title, season,
+                                                                                               episode, year))
+            titles = self._search_series(title)
+        else:
+            logger.debug("Searching for:\nTitle: {}\nYear: {}\n".format(title, year))
+            titles = self._search_series(title, movie=True)
+
+        for title in titles:
+            title_name = title['title_en']
+            title_id = title['imdb']
+            if season and episode:
+                results = self._search_subtitles(title_id, season, episode)
+                page_link = self.URL_WIZDOM_PAGELINK.format("tv", title_id)
+            else:
+                results = self._search_subtitles(title_id)
+                page_link = self.URL_WIZDOM_PAGELINK.format("movie", title_id)
+
+            if not results:
+                logger.info("No subtitles found for: {}".format(title_name))
+                continue
+
+            for result in results:
+                subtitle_id, release = result['id'], result['version']
+                subtitles.append(self.subtitle_class(next(iter(self.languages)), title_id, subtitle_id,
+                                                     title_name, season, episode, release, year, page_link))
+
+        if subtitles:
+            logger.debug("Found Subtitle Candidates: {}".format([x.release for x in subtitles]))
+        return subtitles
+
+    def list_subtitles(self, video, languages):
+        season = episode = year = title = None
+
+        if isinstance(video, Episode):
+            logger.info("list_subtitles Series: {}, season: {}, episode: {}".format(video.series,
+                                                                                    video.season,
+                                                                                    video.episode))
+            title = video.series
+            season = video.season
+            if video.episode == 0:
+                episode = 1
+            else:
+                episode = video.episode
+        elif isinstance(video, Movie):
+            logger.info("list_subtitles Movie: {}, year: {}".format(video.title, video.year))
+            title = video.title
+            year = video.year
+
+        return [s for s in self.query(title, season, episode, year) if s.language in languages]
+
+    def _download_subtitles(self, subtitle_id):
+        logger.debug("Downloading subtitle id - {}".format(subtitle_id))
+        r = self.session.get(self.URL_DOWNLOAD_SUBTITLE.format(subtitle_id))
+
+        if not r.ok:
+            logger.error("Bad response from server while downloding zip [id={}]".format(subtitle_id))
+            return None
+        if not r.content:
+            logger.error("Unable to download zip [id={}]".format(subtitle_id))
+            return None
+
+        return r.content
+
+    def download_subtitle(self, subtitle):
+        # type: (WizdomSubtitle) -> None
+
+        logger.info('Downloading subtitle from WizdomSubs: %r', subtitle)
+
+        content = None
+        zip_content = self._download_subtitles(subtitle.subtitle_id)
+        if not zip_content:
+            return None
+        if not zip_content[:2] == "PK":
+            logger.warning("Response did not contain zip file")
+            return None
+
+        zfile = ZipFile(BytesIO(zip_content))
+        if not len(zfile.namelist()):
+            logger.warning("Response did not contain files inside the zip archive")
+            return None
+
+        sub_files = [x for x in zfile.namelist() if x.endswith(('.srt', '.idx', '.sub'))]
+        if sub_files:
+            content = zfile.open(sub_files[0]).read()
+
+        if not content:
+            logger.warning("File inside zip is empty")
+            return None
+        subtitle.content = fix_line_ending(content)
+        logger.info("Downloaded {} successfuly!".format(subtitle))

Contents/Libraries/Shared/subliminal_patch/refiners/drone.py

@@ -19,11 +19,11 @@ class DroneAPIClient(object):
     _fill_attrs = None
 
     def __init__(self, version=1, session=None, headers=None, timeout=10, base_url=None, api_key=None,
-                 ssl_no_verify=False):
+                 ssl_no_verify=False, pem_file=None):
         headers = dict(headers or {}, **{"X-Api-Key": api_key})
 
         #: Session for the requests
-        self.session = session or CertifiSession()
+        self.session = session or CertifiSession(verify=pem_file)
         if ssl_no_verify:
             self.session.verify = False
 

Contents/Libraries/Shared/subliminal_patch/score.py

@@ -60,6 +60,8 @@ def compute_score(matches, subtitle, video, hearing_impaired=None):
     episode_hash_valid_if = {"series", "season", "episode", "format"}
     movie_hash_valid_if = {"video_codec", "format"}
 
+    orig_matches = matches.copy()
+
     # on hash match, discard everything else
     if subtitle.hash_verifiable:
         if 'hash' in matches:
@@ -83,41 +85,47 @@ def compute_score(matches, subtitle, video, hearing_impaired=None):
         matches &= {'hash'}
 
     # handle equivalent matches
+    eq_matches = set()
     if is_episode:
         if 'title' in matches:
             logger.debug('Adding title match equivalent')
-            matches.add('episode')
+            eq_matches.add('episode')
         if 'series_imdb_id' in matches:
             logger.debug('Adding series_imdb_id match equivalent')
-            matches |= {'series', 'year'}
+            eq_matches |= {'series', 'year'}
         if 'imdb_id' in matches:
             logger.debug('Adding imdb_id match equivalents')
-            matches |= {'series', 'year', 'season', 'episode'}
+            eq_matches |= {'series', 'year', 'season', 'episode'}
         if 'tvdb_id' in matches:
             logger.debug('Adding tvdb_id match equivalents')
-            matches |= {'series', 'year', 'season', 'episode', 'title'}
+            eq_matches |= {'series', 'year', 'season', 'episode', 'title'}
         if 'series_tvdb_id' in matches:
             logger.debug('Adding series_tvdb_id match equivalents')
-            matches |= {'series', 'year'}
+            eq_matches |= {'series', 'year'}
 
         # specials
         if video.is_special and 'title' in matches and 'series' in matches \
                 and 'year' in matches:
             logger.debug('Adding special title match equivalent')
-            matches |= {'season', 'episode'}
+            eq_matches |= {'season', 'episode'}
 
     elif is_movie:
         if 'imdb_id' in matches:
             logger.debug('Adding imdb_id match equivalents')
-            matches |= {'title', 'year'}
+            eq_matches |= {'title', 'year'}
+
+    matches |= eq_matches
 
     # handle hearing impaired
     if hearing_impaired is not None and subtitle.hearing_impaired == hearing_impaired:
         logger.debug('Matched hearing_impaired')
         matches.add('hearing_impaired')
+        orig_matches.add('hearing_impaired')
 
     # compute the score
     score = sum((scores.get(match, 0) for match in matches))
     logger.info('%r: Computed score %r with final matches %r', subtitle, score, matches)
 
-    return score
+    score_without_hash = sum((scores.get(match, 0) for match in orig_matches | eq_matches if match != "hash"))
+
+    return score, score_without_hash

Contents/Libraries/Shared/subliminal_patch/subtitle.py

@@ -278,6 +278,12 @@ class Subtitle(Subtitle_):
 
     @classmethod
     def pysubs2_to_unicode(cls, sub, format="srt"):
+        """
+        this is a modified version of pysubs2.SubripFormat.to_file with special handling for drawing tags in ASS
+        :param sub:
+        :param format:
+        :return:
+        """
         def ms_to_timestamp(ms, mssep=","):
             """Convert ms to 'HH:MM:SS,mmm'"""
             # XXX throw on overflow/underflow?
@@ -289,9 +295,12 @@ class Subtitle(Subtitle_):
         def prepare_text(text, style):
             body = []
             for fragment, sty in parse_tags(text, style, sub.styles):
-                fragment = fragment.replace(ur"\h", u" ")
-                fragment = fragment.replace(ur"\n", u"\n")
-                fragment = fragment.replace(ur"\N", u"\n")
+                fragment = fragment.replace(r"\h", u" ")
+                fragment = fragment.replace(r"\n", u"\n")
+                fragment = fragment.replace(r"\N", u"\n")
+                if sty.drawing:
+                    raise pysubs2.ContentNotUsable
+
                 if format == "srt":
                     if sty.italic:
                         fragment = u"<i>%s</i>" % fragment
@@ -323,7 +332,10 @@ class Subtitle(Subtitle_):
         for i, line in enumerate(visible_lines, 1):
             start = ms_to_timestamp(line.start, mssep=mssep)
             end = ms_to_timestamp(line.end, mssep=mssep)
-            text = prepare_text(line.text, sub.styles.get(line.style, SSAStyle.DEFAULT_STYLE))
+            try:
+                text = prepare_text(line.text, sub.styles.get(line.style, SSAStyle.DEFAULT_STYLE))
+            except pysubs2.ContentNotUsable:
+                continue
 
             out.append(u"%d\n" % i)
             out.append(u"%s --> %s\n" % (start, end))

Contents/Libraries/Shared/submod_test.py

@@ -24,6 +24,7 @@ if debug:
 sub = Subtitle(Language.fromietf("eng"), mods=["common", "remove_HI", "OCR_fixes", "fix_uppercase", "shift_offset(ms=0,s=1)"])
 sub.content = open(fn).read()
 sub.normalize()
+sub.is_valid()
 content = sub.get_modified_content(debug=True)
 
 #submod = SubMod(debug=debug)

Contents/Libraries/Shared/subzero/language.py

@@ -1,9 +1,11 @@
 # coding=utf-8
+from __future__ import absolute_import
 import types
 import re
 
 from babelfish.exceptions import LanguageError
 from babelfish import Language as Language_, basestr, LANGUAGE_MATRIX
+from six.moves import zip
 
 repl_map = {
     "dk": "da",
@@ -30,11 +32,15 @@ repl_map = {
     "tib": "bo",
 }
 
+CUSTOM_LIST = ["chs", "sc", "zhs", "hans", "gb", u"简", u"双语",
+               "cht", "tc", "zht", "hant", "big5", u"繁", u"雙語",
+               "spl", "ea", "pob", "pb"]
 
 ALPHA2_LIST = list(set(filter(lambda x: x, map(lambda x: x.alpha2, LANGUAGE_MATRIX)))) + list(repl_map.values())
 ALPHA3b_LIST = list(set(filter(lambda x: x, map(lambda x: x.alpha3, LANGUAGE_MATRIX)))) + \
                list(set(filter(lambda x: len(x) == 3, list(repl_map.keys()))))
 FULL_LANGUAGE_LIST = ALPHA2_LIST + ALPHA3b_LIST
+FULL_LANGUAGE_LIST.extend(CUSTOM_LIST)
 
 
 def language_from_stream(l):
@@ -61,7 +67,8 @@ def wrap_forced(f):
         args = args[1:]
         s = args.pop(0)
         forced = None
-        if isinstance(s, types.StringTypes):
+        hi = None
+        if isinstance(s, (str,)):
             base, forced = s.split(":") if ":" in s else (s, False)
         else:
             base = s
@@ -69,6 +76,7 @@ def wrap_forced(f):
         instance = f(cls, base, *args, **kwargs)
         if isinstance(instance, Language):
             instance.forced = forced == "forced"
+            instance.hi = hi == "hi"
         return instance
 
     return inner
@@ -76,16 +84,21 @@ def wrap_forced(f):
 
 class Language(Language_):
     forced = False
+    hi = False
 
-    def __init__(self, language, country=None, script=None, unknown=None, forced=False):
+    def __init__(self, language, country=None, script=None, unknown=None, forced=False, hi=False):
         self.forced = forced
+        self.hi = hi
         super(Language, self).__init__(language, country=country, script=script, unknown=unknown)
 
     def __getstate__(self):
-        return self.alpha3, self.country, self.script, self.forced
+        return self.alpha3, self.country, self.script, self.hi, self.forced
 
-    def __setstate__(self, state):
-        self.alpha3, self.country, self.script, self.forced = state
+    def __setstate__(self, forced):
+        self.alpha3, self.country, self.script, self.hi, self.forced = forced
+
+    def __hash__(self):
+        return hash(str(self))
 
     def __eq__(self, other):
         if isinstance(other, basestr):
@@ -95,11 +108,16 @@ class Language(Language_):
         return (self.alpha3 == other.alpha3 and
                 self.country == other.country and
                 self.script == other.script and
-                bool(self.forced) == bool(other.forced))
+                bool(self.forced) == bool(other.forced) and
+                bool(self.hi) == bool(other.hi))
 
     def __str__(self):
         return super(Language, self).__str__() + (":forced" if self.forced else "")
 
+    def __repr__(self):
+        info = ";".join("{}={}".format(k, v) for k, v in vars(self).items() if v)
+        return "<{}: {}>".format(self.__class__.__name__, info)
+
     @property
     def basename(self):
         return super(Language, self).__str__()
@@ -108,14 +126,15 @@ class Language(Language_):
         ret = super(Language, self).__getattr__(name)
         if isinstance(ret, Language):
             ret.forced = self.forced
+            ret.hi = self.hi
         return ret
 
     @classmethod
     def rebuild(cls, instance, **replkw):
         state = instance.__getstate__()
-        attrs = ("country", "script", "forced")
+        attrs = ("country", "script", "hi", "forced")
         language = state[0]
-        kwa = dict(zip(attrs, state[1:]))
+        kwa = dict(list(zip(attrs, state[1:])))
         kwa.update(replkw)
         return cls(language, **kwa)
 

Contents/Libraries/Shared/subzero/modification/exc.py

@@ -0,0 +1,7 @@
+# coding=utf-8
+class EmptyEntryError(Exception):
+    pass
+
+
+class EmptyLineError(Exception):
+    pass

Contents/Libraries/Shared/subzero/modification/main.py

@@ -6,7 +6,8 @@ import pysubs2
 import logging
 import time
 
-from mods import EMPTY_TAG_PROCESSOR, EmptyEntryError
+from mods import EMPTY_TAG_PROCESSOR
+from exc import EmptyEntryError
 from registry import registry
 from subzero.language import Language
 
@@ -300,11 +301,11 @@ class SubtitleModifications(object):
                     mod = self.initialized_mods[identifier]
 
                     try:
-                        line = mod.modify(line.strip(), entry=entry.text, debug=self.debug, parent=self, index=index,
+                        line = mod.modify(line.strip(), entry=t, debug=self.debug, parent=self, index=index,
                                           **args)
                     except EmptyEntryError:
                         if self.debug:
-                            logger.debug(u"%d: %s: %r -> ''", index, identifier, entry.text)
+                            logger.debug(u"%d: %s: %r -> ''", index, identifier, t)
                         skip_entry = True
                         break
 
@@ -329,11 +330,11 @@ class SubtitleModifications(object):
                     mod = self.initialized_mods[identifier]
 
                     try:
-                        line = mod.modify(line.strip(), entry=entry.text, debug=self.debug, parent=self, index=index,
+                        line = mod.modify(line.strip(), entry=t, debug=self.debug, parent=self, index=index,
                                           procs=["last_process"], **args)
                     except EmptyEntryError:
                         if self.debug:
-                            logger.debug(u"%d: %s: %r -> ''", index, identifier, entry.text)
+                            logger.debug(u"%d: %s: %r -> ''", index, identifier, t)
                         skip_entry = True
                         break
 

Contents/Libraries/Shared/subzero/modification/mods/__init__.py

@@ -107,9 +107,3 @@ empty_line_post_processors = [
 ]
 
 
-class EmptyEntryError(Exception):
-    pass
-
-
-class EmptyLineError(Exception):
-    pass

Contents/Libraries/Shared/subzero/modification/mods/common.py

@@ -7,6 +7,7 @@ from subzero.modification.mods import SubtitleTextModification, empty_line_post_
 from subzero.modification.processors import FuncProcessor
 from subzero.modification.processors.re_processor import NReProcessor
 from subzero.modification import registry
+from tld import get_tld
 
 
 ENGLISH = Language("eng")
@@ -28,7 +29,7 @@ class CommonFixes(SubtitleTextModification):
         NReProcessor(re.compile(r'(?u)(\w|\b|\s|^)(-\s?-{1,2})'), ur"\1—", name="CM_multidash"),
 
         # line = _/-/\s
-        NReProcessor(re.compile(r'(?u)(^\W*[-_.:>~]+\W*$)'), "", name="<CM_non_word_only"),
+        NReProcessor(re.compile(r'(?u)(^\W*[-_.:<>~"\']+\W*$)'), "", name="CM_non_word_only"),
 
         # remove >>
         NReProcessor(re.compile(r'(?u)^\s?>>\s*'), "", name="CM_leading_crocodiles"),
@@ -113,7 +114,9 @@ class CommonFixes(SubtitleTextModification):
         NReProcessor(re.compile(r'(?u)(?:(?<=^)|(?<=\w)) +([!?.,](?![!?.,]| \.))'), r"\1", name="CM_punctuation_space"),
 
         # add space after punctuation
-        NReProcessor(re.compile(r'(?u)([!?.,:])([A-zÀ-ž]{2,})'), r"\1 \2", name="CM_punctuation_space2"),
+        NReProcessor(re.compile(r'(?u)(([^\s]*)([!?.,:])([A-zÀ-ž]{2,}))'),
+                     lambda match: u"%s%s %s" % (match.group(2), match.group(3), match.group(4)) if not get_tld(match.group(1), fail_silently=True, fix_protocol=True) else match.group(1),
+                     name="CM_punctuation_space2"),
 
         # fix lowercase I in english
         NReProcessor(re.compile(r'(?u)(\b)i(\b)'), r"\1I\2", name="CM_EN_lowercase_i",
@@ -178,6 +181,14 @@ class FixUppercase(SubtitleModification):
             entry.plaintext = self.capitalize(entry.plaintext)
 
 
+"""
+subsync
+
+subsync --cli --offline --overwrite --window-size=600 --max-point-dist=2.0 --min-points-no=20 --min-word-prob=0.3 --min-word-len=5 --min-correlation=0.9999 --min-words-sim=0.6 --out-time-offset=-0.08 sync --out SUBTITLE -s SUBTITLE --sub-lang=eng --sub-enc=utf-8 -r REF_FILE --ref-stream-by-type=audio --ref-stream-by-lang=eng
+
+"""
+
+
 registry.register(CommonFixes)
 registry.register(RemoveTags)
 registry.register(ReverseRTL)

Contents/Libraries/Shared/subzero/modification/mods/hearing_impaired.py

@@ -1,7 +1,8 @@
 # coding=utf-8
 import re
 
-from subzero.modification.mods import SubtitleTextModification, empty_line_post_processors, EmptyEntryError, TAG
+from subzero.modification.mods import SubtitleTextModification, empty_line_post_processors, TAG
+from subzero.modification.exc import EmptyEntryError
 from subzero.modification.processors.re_processor import NReProcessor
 from subzero.modification import registry
 
@@ -46,7 +47,7 @@ class HearingImpaired(SubtitleTextModification):
                      name="HI_before_colon_noncaps"),
 
         # brackets (only remove if at least 3 chars in brackets)
-        NReProcessor(re.compile(ur'(?sux)-?%(t)s[([][^([)\]]+?(?=[A-zÀ-ž"\'.]{3,})[^([)\]]+[)\]][\s:]*%(t)s' %
+        NReProcessor(re.compile(ur'(?sux)-?%(t)s["\']*[([][^([)\]]+?(?=[A-zÀ-ž"\'.]{3,})[^([)\]]+[)\]]["\']*[\s:]*%(t)s' %
                                 {"t": TAG}), "", name="HI_brackets"),
 
         #NReProcessor(re.compile(ur'(?sux)-?%(t)s[([]%(t)s(?=[A-zÀ-ž"\'.]{3,})[^([)\]]+%(t)s$' % {"t": TAG}),
@@ -90,8 +91,8 @@ class HearingImpaired(SubtitleTextModification):
                      "", name="HI_music_symbols_only"),
 
         # remove music entries
-        NReProcessor(re.compile(ur'(?ums)(^[-\s>~]*[♫♪]+\s*.+|.+\s*[♫♪]+\s*$)'),
-                     "", name="HI_music"),
+        NReProcessor(re.compile(ur'(?ums)(^[-\s>~]*[*#¶♫♪]+\s*.+|.+\s*[*#¶♫♪]+\s*$)'),
+                     "", name="HI_music", entry=True),
     ]
 
 

Contents/Libraries/Shared/subzero/modification/processors/__init__.py

@@ -10,7 +10,7 @@ class Processor(object):
     supported = None
     enabled = True
 
-    def __init__(self, name=None, parent=None, supported=None):
+    def __init__(self, name=None, parent=None, supported=None, **kwargs):
         self.name = name
         self.parent = parent
         self.supported = supported if supported else lambda parent: True
@@ -35,7 +35,7 @@ class Processor(object):
 class FuncProcessor(Processor):
     func = None
 
-    def __init__(self, func, name=None, parent=None, supported=None):
+    def __init__(self, func, name=None, parent=None, supported=None, **kwargs):
         super(FuncProcessor, self).__init__(name=name, supported=supported)
         self.func = func
 

Contents/Libraries/Shared/subzero/modification/processors/re_processor.py

@@ -2,6 +2,7 @@
 import re
 import logging
 
+from subzero.modification.exc import EmptyEntryError
 from subzero.modification.processors import Processor
 
 logger = logging.getLogger(__name__)
@@ -14,13 +15,22 @@ class ReProcessor(Processor):
     pattern = None
     replace_with = None
 
-    def __init__(self, pattern, replace_with, name=None, supported=None):
+    def __init__(self, pattern, replace_with, name=None, supported=None, entry=False, **kwargs):
         super(ReProcessor, self).__init__(name=name, supported=supported)
         self.pattern = pattern
         self.replace_with = replace_with
+        self.use_entry = entry
 
-    def process(self, content, debug=False, **kwargs):
-        return self.pattern.sub(self.replace_with, content)
+    def process(self, content, debug=False, entry=None, **kwargs):
+        if not self.use_entry:
+            return self.pattern.sub(self.replace_with, content)
+
+        ret = self.pattern.sub(self.replace_with, entry)
+        if not ret:
+            raise EmptyEntryError()
+        elif ret != entry:
+            return ret
+        return content
 
 
 class NReProcessor(ReProcessor):
@@ -36,7 +46,7 @@ class MultipleWordReProcessor(ReProcessor):
     }
     replaces found key in pattern with the corresponding value in data
     """
-    def __init__(self, snr_dict, name=None, parent=None, supported=None):
+    def __init__(self, snr_dict, name=None, parent=None, supported=None, **kwargs):
         super(ReProcessor, self).__init__(name=name, supported=supported)
         self.snr_dict = snr_dict
 

Contents/Libraries/Shared/subzero/modification/processors/string_processor.py

@@ -12,7 +12,7 @@ class StringProcessor(Processor):
     String replacement processor base
     """
 
-    def __init__(self, search, replace, name=None, parent=None, supported=None):
+    def __init__(self, search, replace, name=None, parent=None, supported=None, **kwargs):
         super(StringProcessor, self).__init__(name=name, supported=supported)
         self.search = search
         self.replace = replace
@@ -31,7 +31,7 @@ class MultipleLineProcessor(Processor):
         "data": {"old_value": "new_value"}
     }
     """
-    def __init__(self, snr_dict, name=None, parent=None, supported=None):
+    def __init__(self, snr_dict, name=None, parent=None, supported=None, **kwargs):
         super(MultipleLineProcessor, self).__init__(name=name, supported=supported)
         self.snr_dict = snr_dict
 

Contents/Libraries/Shared/test.srt

@@ -19,6 +19,8 @@ I can't keep running. L can't!
 <b>i don't know. Some kind of wrong "1 00" number---
 of signal, drawing the Tardis off.... course.</b>
 # I'm singing in the rain
+www.website.com
+www.nowebsite.badlol
 
 4
 00:00:16,099 --> 00:00:17,224

Contents/Libraries/Shared/tld/__init__.py

@@ -0,0 +1,14 @@
+# -*- coding: utf-8 -*-
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+from __future__ import unicode_literals
+from .utils import get_fld, get_tld, get_tld_names, is_tld, parse_tld, Result, update_tld_names
+__title__ = u'tld'
+__version__ = u'0.11.10'
+__author__ = u'Artur Barseghyan'
+__copyright__ = u'2013-2019 Artur Barseghyan'
+__license__ = u'MPL-1.1 OR GPL-2.0-only OR LGPL-2.1-or-later'
+__all__ = (u'get_fld', u'get_tld', u'get_tld_names', u'is_tld',
+           u'parse_tld', u'Result', u'update_tld_names')

Contents/Libraries/Shared/tld/base.py

@@ -0,0 +1,57 @@
+# -*- coding: utf-8 -*-
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+from __future__ import unicode_literals
+from six import with_metaclass as _py_backwards_six_withmetaclass
+from codecs import open as codecs_open
+try:
+    from urllib.request import urlopen
+except ImportError:
+    from six.moves.urllib.request import urlopen as urlopen
+from .exceptions import TldIOError, TldImproperlyConfigured
+from .helpers import project_dir
+from .registry import Registry
+__author__ = u'Artur Barseghyan'
+__copyright__ = u'2013-2019 Artur Barseghyan'
+__license__ = u'MPL-1.1 OR GPL-2.0-only OR LGPL-2.1-or-later'
+__all__ = (u'BaseTLDSourceParser',)
+
+
+class BaseTLDSourceParser(
+        _py_backwards_six_withmetaclass(Registry, *[object])):
+    u'Base TLD source parser.'
+    uid = None
+    source_url = None
+    local_path = None
+
+    @classmethod
+    def validate(cls):
+        u'Constructor.'
+        if (not cls.uid):
+            raise TldImproperlyConfigured(
+                u'The `uid` property of the TLD source parser shall be defined.')
+
+    @classmethod
+    def get_tld_names(cls, fail_silently=False, retry_count=0):
+        u'Get tld names.\n\n        :param fail_silently:\n        :param retry_count:\n        :return:\n        '
+        cls.validate()
+        raise NotImplementedError(
+            u'Your TLD source parser shall implement `get_tld_names` method.')
+
+    @classmethod
+    def update_tld_names(cls, fail_silently=False):
+        u'Update the local copy of the TLD file.\n\n        :param fail_silently:\n        :return:\n        '
+        try:
+            remote_file = urlopen(cls.source_url)
+            local_file = codecs_open(project_dir(
+                cls.local_path), u'wb', encoding='utf8')
+            local_file.write(remote_file.read().decode(u'utf8'))
+            local_file.close()
+            remote_file.close()
+        except Exception as err:
+            if fail_silently:
+                return False
+            raise TldIOError(err)
+        return True

Contents/Libraries/Shared/tld/conf.py

@@ -0,0 +1,45 @@
+# -*- coding: utf-8 -*-
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+from __future__ import unicode_literals
+from typing import Any
+from . import defaults
+__author__ = u'Artur Barseghyan'
+__copyright__ = u'2013-2019 Artur Barseghyan'
+__license__ = u'MPL-1.1 OR GPL-2.0-only OR LGPL-2.1-or-later'
+__all__ = (u'get_setting', u'reset_settings', u'set_setting', u'settings')
+
+
+class Settings(object):
+    u'Settings registry.'
+
+    def __init__(self):
+        self._settings = {
+
+        }
+        self._settings_get = self._settings.get
+
+    def set(self, name, value):
+        u'\n        Override default settings.\n\n        :param str name:\n        :param mixed value:\n        '
+        self._settings[name] = value
+
+    def get(self, name, default=None):
+        u'\n        Gets a variable from local settings.\n\n        :param str name:\n        :param mixed default: Default value.\n        :return mixed:\n        '
+        if (name in self._settings):
+            return self._settings_get(name, default)
+        elif hasattr(defaults, name):
+            return getattr(defaults, name, default)
+        return default
+
+    def reset(self):
+        u'Reset settings.'
+        for name in defaults.__all__:
+            self.set(name, getattr(defaults, name))
+
+
+settings = Settings()
+get_setting = settings.get
+set_setting = settings.set
+reset_settings = settings.reset

Contents/Libraries/Shared/tld/defaults.py

@@ -0,0 +1,13 @@
+# -*- coding: utf-8 -*-
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+from __future__ import unicode_literals
+from os.path import dirname
+__author__ = u'Artur Barseghyan'
+__copyright__ = u'2013-2019 Artur Barseghyan'
+__license__ = u'MPL-1.1 OR GPL-2.0-only OR LGPL-2.1-or-later'
+__all__ = (u'DEBUG', u'NAMES_LOCAL_PATH_PARENT')
+NAMES_LOCAL_PATH_PARENT = dirname(__file__)
+DEBUG = False

Contents/Libraries/Shared/tld/exceptions.py

@@ -0,0 +1,49 @@
+# -*- coding: utf-8 -*-
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+from __future__ import unicode_literals
+from .conf import get_setting
+__author__ = u'Artur Barseghyan'
+__copyright__ = u'2013-2019 Artur Barseghyan'
+__license__ = u'MPL-1.1 OR GPL-2.0-only OR LGPL-2.1-or-later'
+__all__ = (u'TldBadUrl', u'TldDomainNotFound',
+           u'TldImproperlyConfigured', u'TldIOError')
+
+
+class TldIOError(IOError):
+    u'TldIOError.\n\n    Supposed to be thrown when problems with reading/writing occur.\n    '
+
+    def __init__(self, msg=None):
+        tld_names_local_path = get_setting(u'NAMES_LOCAL_PATH')
+        if (msg is None):
+            msg = (u"Can't read from or write to the %s file!" %
+                   tld_names_local_path)
+        super(TldIOError, self).__init__(msg)
+
+
+class TldDomainNotFound(ValueError):
+    u"TldDomainNotFound.\n\n    Supposed to be thrown when domain name is not found (didn't match) the\n    local TLD policy.\n    "
+
+    def __init__(self, domain_name):
+        super(TldDomainNotFound, self).__init__(
+            (u"Domain %s didn't match any existing TLD name!" % domain_name))
+
+
+class TldBadUrl(ValueError):
+    u'TldBadUrl.\n\n    Supposed to be thrown when bad URL is given.\n    '
+
+    def __init__(self, url):
+        super(TldBadUrl, self).__init__((u'Is not a valid URL %s!' % url))
+
+
+class TldImproperlyConfigured(Exception):
+    u'TldImproperlyConfigured.\n\n    Supposed to be thrown when code is improperly configured. Typical use-case\n    is when user tries to use `get_tld` function with both `search_public` and\n    `search_private` set to False.\n    '
+
+    def __init__(self, msg=None):
+        if (msg is None):
+            msg = u'Improperly configured.'
+        else:
+            msg = (u'Improperly configured. %s' % msg)
+        super(TldImproperlyConfigured, self).__init__(msg)

Contents/Libraries/Shared/tld/helpers.py

@@ -0,0 +1,21 @@
+# -*- coding: utf-8 -*-
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+from __future__ import unicode_literals
+from os.path import abspath, join
+from .conf import get_setting
+__author__ = u'Artur Barseghyan'
+__copyright__ = u'2013-2019 Artur Barseghyan'
+__license__ = u'MPL-1.1 OR GPL-2.0-only OR LGPL-2.1-or-later'
+__all__ = (u'project_dir', u'PROJECT_DIR')
+
+
+def project_dir(base):
+    u'Project dir.'
+    tld_names_local_path_parent = get_setting(u'NAMES_LOCAL_PATH_PARENT')
+    return abspath(join(tld_names_local_path_parent, base).replace(u'\\', u'/'))
+
+
+PROJECT_DIR = project_dir

Contents/Libraries/Shared/tld/parsers/__init__.py

@@ -0,0 +1,5 @@
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+from __future__ import unicode_literals

Contents/Libraries/Shared/tld/parsers/mozilla.py

@@ -0,0 +1,5 @@
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+from __future__ import unicode_literals

Contents/Libraries/Shared/tld/registry.py

@@ -0,0 +1,41 @@
+# -*- coding: utf-8 -*-
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+from __future__ import unicode_literals
+from typing import Type, Dict
+__author__ = u'Artur Barseghyan'
+__copyright__ = u'2013-2019 Artur Barseghyan'
+__license__ = u'MPL-1.1 OR GPL-2.0-only OR LGPL-2.1-or-later'
+__all__ = (u'Registry',)
+
+
+class Registry(type):
+    REGISTRY = {
+
+    }
+
+    def __new__(cls, name, bases, attrs):
+        new_cls = type.__new__(cls, name, bases, attrs)
+        if getattr(new_cls, u'_uid', None):
+            cls.REGISTRY[new_cls._uid] = new_cls
+        return new_cls
+
+    @property
+    def _uid(cls):
+        return getattr(cls, 'uid', cls.__name__)
+
+    @classmethod
+    def reset(cls):
+        cls.REGISTRY = {
+
+        }
+
+    @classmethod
+    def get(cls, key, default=None):
+        return cls.REGISTRY.get(key, default)
+
+    @classmethod
+    def items(cls):
+        return cls.REGISTRY.items()

Contents/Libraries/Shared/tld/result.py

@@ -0,0 +1,57 @@
+# -*- coding: utf-8 -*-
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+from __future__ import unicode_literals
+from typing import Dict
+try:
+    from urllib.parse import SplitResult
+except ImportError:
+    from six.moves.urllib_parse import SplitResult
+__author__ = u'Artur Barseghyan'
+__copyright__ = u'2013-2019 Artur Barseghyan'
+__license__ = u'MPL-1.1 OR GPL-2.0-only OR LGPL-2.1-or-later'
+__all__ = (u'Result',)
+
+
+class Result(object):
+    u'Container.'
+    __slots__ = (u'subdomain', u'domain', u'tld', u'__fld', u'parsed_url')
+
+    def __init__(self, tld, domain, subdomain, parsed_url):
+        self.tld = tld
+        self.domain = (domain if (domain != u'') else tld)
+        self.subdomain = subdomain
+        self.parsed_url = parsed_url
+        if domain:
+            self.__fld = u''.join(
+                [u'{}'.format(self.domain), u'.', u'{}'.format(self.tld)])
+        else:
+            self.__fld = self.tld
+
+    @property
+    def extension(self):
+        u'Alias of ``tld``.\n\n        :return str:\n        '
+        return self.tld
+    suffix = extension
+
+    @property
+    def fld(self):
+        u'First level domain.\n\n        :return:\n        :rtype: str\n        '
+        return self.__fld
+
+    def __str__(self):
+        return self.tld
+    __repr__ = __str__
+
+    @property
+    def __dict__(self):
+        u'Mimic __dict__ functionality.\n\n        :return:\n        :rtype: dict\n        '
+        return {
+            u'tld': self.tld,
+            u'domain': self.domain,
+            u'subdomain': self.subdomain,
+            u'fld': self.fld,
+            u'parsed_url': self.parsed_url,
+        }

Contents/Libraries/Shared/tld/tests/__init__.py

@@ -0,0 +1,11 @@
+# -*- coding: utf-8 -*-
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+from __future__ import unicode_literals
+import unittest
+from .test_core import *
+from .test_commands import *
+if (__name__ == u'__main__'):
+    unittest.main()

Contents/Libraries/Shared/tld/tests/base.py

@@ -0,0 +1,65 @@
+# -*- coding: utf-8 -*-
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+from __future__ import unicode_literals
+from backports.functools_lru_cache import lru_cache
+import logging
+import socket
+__author__ = u'Artur Barseghyan'
+__copyright__ = u'2013-2019 Artur Barseghyan'
+__license__ = u'MPL-1.1 OR GPL-2.0-only OR LGPL-2.1-or-later'
+__all__ = (u'internet_available_only', u'log_info')
+LOG_INFO = True
+LOGGER = logging.getLogger(__name__)
+
+
+def log_info(func):
+    u'Log some useful info.'
+    if (not LOG_INFO):
+        return func
+
+    def inner(self, *args, **kwargs):
+        u'Inner.'
+        result = func(*([self] + list(args)), **kwargs)
+        LOGGER.debug(u'\n\n%s', func.__name__)
+        LOGGER.debug(u'============================')
+        if func.__doc__:
+            LOGGER.debug(u'""" %s """', func.__doc__.strip())
+        LOGGER.debug(u'----------------------------')
+        if (result is not None):
+            LOGGER.debug(result)
+        LOGGER.debug(u'\n++++++++++++++++++++++++++++')
+        return result
+    return inner
+
+
+@lru_cache(maxsize=32)
+def is_internet_available(host='8.8.8.8', port=53, timeout=3):
+    u'Check if internet is available.\n\n    Host: 8.8.8.8 (google-public-dns-a.google.com)\n    OpenPort: 53/tcp\n    Service: domain (DNS/TCP)\n    '
+    try:
+        socket.setdefaulttimeout(timeout)
+        socket.socket(socket.AF_INET, socket.SOCK_STREAM).connect((host, port))
+        return True
+    except socket.error as ex:
+        print(ex)
+        return False
+
+
+def internet_available_only(func):
+
+    def inner(self, *args, **kwargs):
+        u'Inner.'
+        if (not is_internet_available()):
+            LOGGER.debug(u'\n\n%s', func.__name__)
+            LOGGER.debug(u'============================')
+            if func.__doc__:
+                LOGGER.debug(u'""" %s """', func.__doc__.strip())
+            LOGGER.debug(u'----------------------------')
+            LOGGER.debug(u'Skipping because no Internet connection available.')
+            LOGGER.debug(u'\n++++++++++++++++++++++++++++')
+            return None
+        result = func(*([self] + list(args)), **kwargs)
+        return result
+    return inner

Contents/Libraries/Shared/tld/tests/test_commands.py

@@ -0,0 +1,43 @@
+# -*- coding: utf-8 -*-
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+from __future__ import unicode_literals
+import logging
+import unittest
+import subprocess
+from .base import log_info, internet_available_only
+__author__ = u'Artur Barseghyan'
+__copyright__ = u'2013-2019 Artur Barseghyan'
+__license__ = u'GPL 2.0/LGPL 2.1'
+__all__ = (u'TestCommands',)
+LOGGER = logging.getLogger(__name__)
+
+
+class TestCommands(unittest.TestCase):
+    u'Tld commands tests.'
+
+    def setUp(self):
+        u'Set up.'
+
+    @internet_available_only
+    @log_info
+    def test_1_update_tld_names_command(self):
+        u'Test updating the tld names (re-fetch mozilla source).'
+        res = subprocess.check_output([u'update-tld-names']).strip()
+        self.assertEqual(res, b'')
+        return res
+
+    @internet_available_only
+    @log_info
+    def test_1_update_tld_names_mozilla_command(self):
+        u'Test updating the tld names (re-fetch mozilla source).'
+        res = subprocess.check_output(
+            [u'update-tld-names', u'mozilla']).strip()
+        self.assertEqual(res, b'')
+        return res
+
+
+if (__name__ == u'__main__'):
+    unittest.main()

Contents/Libraries/Shared/tld/tests/test_core.py

@@ -0,0 +1,708 @@
+# -*- coding: utf-8 -*-
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import copy
+import logging
+from os.path import abspath, join
+import unittest
+from tempfile import gettempdir
+from typing import Type
+try:
+    from urllib.parse import urlsplit
+except ImportError:
+    from six.moves.urllib_parse import urlsplit
+from faker import Faker
+from .. import defaults
+from ..base import BaseTLDSourceParser
+from ..conf import get_setting, reset_settings, set_setting
+from ..exceptions import TldBadUrl, TldDomainNotFound, TldImproperlyConfigured, TldIOError
+from ..helpers import project_dir
+from ..registry import Registry
+from ..utils import get_fld, get_tld, get_tld_names, get_tld_names_container, is_tld, MozillaTLDSourceParser, BaseMozillaTLDSourceParser, parse_tld, reset_tld_names, update_tld_names, update_tld_names_cli
+from .base import internet_available_only, log_info
+__author__ = u'Artur Barseghyan'
+__copyright__ = u'2013-2019 Artur Barseghyan'
+__license__ = u'MPL-1.1 OR GPL-2.0-only OR LGPL-2.1-or-later'
+__all__ = (u'TestCore',)
+LOGGER = logging.getLogger(__name__)
+
+
+class TestCore(unittest.TestCase):
+    u'Core tld functionality tests.'
+
+    @classmethod
+    def setUpClass(cls):
+        cls.faker = Faker()
+        cls.temp_dir = gettempdir()
+
+    def setUp(self):
+        u'Set up.'
+        self.good_patterns = [{
+            u'url': u'http://www.google.co.uk',
+            u'fld': u'google.co.uk',
+            u'subdomain': u'www',
+            u'domain': u'google',
+            u'suffix': u'co.uk',
+            u'tld': u'co.uk',
+            u'kwargs': {
+                u'fail_silently': True,
+            },
+        }, {
+            u'url': u'http://www.v2.google.co.uk',
+            u'fld': u'google.co.uk',
+            u'subdomain': u'www.v2',
+            u'domain': u'google',
+            u'suffix': u'co.uk',
+            u'tld': u'co.uk',
+            u'kwargs': {
+                u'fail_silently': True,
+            },
+        }, {
+            u'url': u'http://хром.гугл.рф',
+            u'fld': u'гугл.рф',
+            u'subdomain': u'хром',
+            u'domain': u'гугл',
+            u'suffix': u'рф',
+            u'tld': u'рф',
+            u'kwargs': {
+                u'fail_silently': True,
+            },
+        }, {
+            u'url': u'http://www.google.co.uk:8001/lorem-ipsum/',
+            u'fld': u'google.co.uk',
+            u'subdomain': u'www',
+            u'domain': u'google',
+            u'suffix': u'co.uk',
+            u'tld': u'co.uk',
+            u'kwargs': {
+                u'fail_silently': True,
+            },
+        }, {
+            u'url': u'http://www.me.cloudfront.net',
+            u'fld': u'me.cloudfront.net',
+            u'subdomain': u'www',
+            u'domain': u'me',
+            u'suffix': u'cloudfront.net',
+            u'tld': u'cloudfront.net',
+            u'kwargs': {
+                u'fail_silently': True,
+            },
+        }, {
+            u'url': u'http://www.v2.forum.tech.google.co.uk:8001/lorem-ipsum/',
+            u'fld': u'google.co.uk',
+            u'subdomain': u'www.v2.forum.tech',
+            u'domain': u'google',
+            u'suffix': u'co.uk',
+            u'tld': u'co.uk',
+            u'kwargs': {
+                u'fail_silently': True,
+            },
+        }, {
+            u'url': u'https://pantheon.io/',
+            u'fld': u'pantheon.io',
+            u'subdomain': u'',
+            u'domain': u'pantheon',
+            u'suffix': u'io',
+            u'tld': u'io',
+            u'kwargs': {
+                u'fail_silently': True,
+            },
+        }, {
+            u'url': u'v2.www.google.com',
+            u'fld': u'google.com',
+            u'subdomain': u'v2.www',
+            u'domain': u'google',
+            u'suffix': u'com',
+            u'tld': u'com',
+            u'kwargs': {
+                u'fail_silently': True,
+                u'fix_protocol': True,
+            },
+        }, {
+            u'url': u'//v2.www.google.com',
+            u'fld': u'google.com',
+            u'subdomain': u'v2.www',
+            u'domain': u'google',
+            u'suffix': u'com',
+            u'tld': u'com',
+            u'kwargs': {
+                u'fail_silently': True,
+                u'fix_protocol': True,
+            },
+        }, {
+            u'url': u'http://foo@bar.com',
+            u'fld': u'bar.com',
+            u'subdomain': u'',
+            u'domain': u'bar',
+            u'suffix': u'com',
+            u'tld': u'com',
+            u'kwargs': {
+                u'fail_silently': True,
+            },
+        }, {
+            u'url': u'http://user:foo@bar.com',
+            u'fld': u'bar.com',
+            u'subdomain': u'',
+            u'domain': u'bar',
+            u'suffix': u'com',
+            u'tld': u'com',
+            u'kwargs': {
+                u'fail_silently': True,
+            },
+        }, {
+            u'url': u'https://faguoren.xn--fiqs8s',
+            u'fld': u'faguoren.xn--fiqs8s',
+            u'subdomain': u'',
+            u'domain': u'faguoren',
+            u'suffix': u'xn--fiqs8s',
+            u'tld': u'xn--fiqs8s',
+            u'kwargs': {
+                u'fail_silently': True,
+            },
+        }, {
+            u'url': u'blogs.lemonde.paris',
+            u'fld': u'lemonde.paris',
+            u'subdomain': u'blogs',
+            u'domain': u'lemonde',
+            u'suffix': u'paris',
+            u'tld': u'paris',
+            u'kwargs': {
+                u'fail_silently': True,
+                u'fix_protocol': True,
+            },
+        }, {
+            u'url': u'axel.brighton.ac.uk',
+            u'fld': u'brighton.ac.uk',
+            u'subdomain': u'axel',
+            u'domain': u'brighton',
+            u'suffix': u'ac.uk',
+            u'tld': u'ac.uk',
+            u'kwargs': {
+                u'fail_silently': True,
+                u'fix_protocol': True,
+            },
+        }, {
+            u'url': u'm.fr.blogspot.com.au',
+            u'fld': u'fr.blogspot.com.au',
+            u'subdomain': u'm',
+            u'domain': u'fr',
+            u'suffix': u'blogspot.com.au',
+            u'tld': u'blogspot.com.au',
+            u'kwargs': {
+                u'fail_silently': True,
+                u'fix_protocol': True,
+            },
+        }, {
+            u'url': u'help.www.福岡.jp',
+            u'fld': u'www.福岡.jp',
+            u'subdomain': u'help',
+            u'domain': u'www',
+            u'suffix': u'福岡.jp',
+            u'tld': u'福岡.jp',
+            u'kwargs': {
+                u'fail_silently': True,
+                u'fix_protocol': True,
+            },
+        }, {
+            u'url': u'syria.arabic.variant.سوريا',
+            u'fld': u'variant.سوريا',
+            u'subdomain': u'syria.arabic',
+            u'domain': u'variant',
+            u'suffix': u'سوريا',
+            u'tld': u'سوريا',
+            u'kwargs': {
+                u'fail_silently': True,
+                u'fix_protocol': True,
+            },
+        }, {
+            u'url': u'http://www.help.kawasaki.jp',
+            u'fld': u'www.help.kawasaki.jp',
+            u'subdomain': u'',
+            u'domain': u'www',
+            u'suffix': u'help.kawasaki.jp',
+            u'tld': u'help.kawasaki.jp',
+            u'kwargs': {
+                u'fail_silently': True,
+            },
+        }, {
+            u'url': u'http://www.city.kawasaki.jp',
+            u'fld': u'city.kawasaki.jp',
+            u'subdomain': u'www',
+            u'domain': u'city',
+            u'suffix': u'kawasaki.jp',
+            u'tld': u'kawasaki.jp',
+            u'kwargs': {
+                u'fail_silently': True,
+            },
+        }, {
+            u'url': u'http://fedoraproject.org',
+            u'fld': u'fedoraproject.org',
+            u'subdomain': u'',
+            u'domain': u'fedoraproject',
+            u'suffix': u'org',
+            u'tld': u'org',
+            u'kwargs': {
+                u'fail_silently': True,
+            },
+        }, {
+            u'url': u'http://www.cloud.fedoraproject.org',
+            u'fld': u'www.cloud.fedoraproject.org',
+            u'subdomain': u'',
+            u'domain': u'www',
+            u'suffix': u'cloud.fedoraproject.org',
+            u'tld': u'cloud.fedoraproject.org',
+            u'kwargs': {
+                u'fail_silently': True,
+            },
+        }, {
+            u'url': u'https://www.john.app.os.fedoraproject.org',
+            u'fld': u'john.app.os.fedoraproject.org',
+            u'subdomain': u'www',
+            u'domain': u'john',
+            u'suffix': u'app.os.fedoraproject.org',
+            u'tld': u'app.os.fedoraproject.org',
+            u'kwargs': {
+                u'fail_silently': True,
+            },
+        }, {
+            u'url': u'ftp://www.xn--mxail5aa.xn--11b4c3d',
+            u'fld': u'xn--mxail5aa.xn--11b4c3d',
+            u'subdomain': u'www',
+            u'domain': u'xn--mxail5aa',
+            u'suffix': u'xn--11b4c3d',
+            u'tld': u'xn--11b4c3d',
+            u'kwargs': {
+                u'fail_silently': True,
+            },
+        }, {
+            u'url': u'http://cloud.fedoraproject.org',
+            u'fld': u'cloud.fedoraproject.org',
+            u'subdomain': u'',
+            u'domain': u'cloud.fedoraproject.org',
+            u'suffix': u'cloud.fedoraproject.org',
+            u'tld': u'cloud.fedoraproject.org',
+            u'kwargs': {
+                u'fail_silently': True,
+            },
+        }, {
+            u'url': u'github.io',
+            u'fld': u'github.io',
+            u'subdomain': u'',
+            u'domain': u'github.io',
+            u'suffix': u'github.io',
+            u'tld': u'github.io',
+            u'kwargs': {
+                u'fail_silently': True,
+                u'fix_protocol': True,
+            },
+        }, {
+            u'url': urlsplit(u'http://lemonde.fr/article.html'),
+            u'fld': u'lemonde.fr',
+            u'subdomain': u'',
+            u'domain': u'lemonde',
+            u'suffix': u'fr',
+            u'tld': u'fr',
+            u'kwargs': {
+                u'fail_silently': True,
+            },
+        }]
+        self.bad_patterns = {
+            u'v2.www.google.com': {
+                u'exception': TldBadUrl,
+            },
+            u'/index.php?a=1&b=2': {
+                u'exception': TldBadUrl,
+            },
+            u'http://www.tld.doesnotexist': {
+                u'exception': TldDomainNotFound,
+            },
+            u'https://2001:0db8:0000:85a3:0000:0000:ac1f:8001': {
+                u'exception': TldDomainNotFound,
+            },
+            u'http://192.169.1.1': {
+                u'exception': TldDomainNotFound,
+            },
+            u'http://localhost:8080': {
+                u'exception': TldDomainNotFound,
+            },
+            u'https://localhost': {
+                u'exception': TldDomainNotFound,
+            },
+            u'https://localhost2': {
+                u'exception': TldImproperlyConfigured,
+                u'kwargs': {
+                    u'search_public': False,
+                    u'search_private': False,
+                },
+            },
+        }
+        self.invalid_tlds = {u'v2.www.google.com', u'tld.doesnotexist',
+                             u'2001:0db8:0000:85a3:0000:0000:ac1f', u'192.169.1.1', 'localhost', u'google.com'}
+        self.tld_names_local_path_custom = project_dir(
+            join(u'tests', u'res', u'effective_tld_names_custom.dat.txt'))
+        self.good_patterns_custom_parser = [{
+            u'url': u'http://www.foreverchild',
+            u'fld': u'www.foreverchild',
+            u'subdomain': u'',
+            u'domain': u'www',
+            u'suffix': u'foreverchild',
+            u'tld': u'foreverchild',
+            u'kwargs': {
+                u'fail_silently': True,
+            },
+        }, {
+            u'url': u'http://www.v2.foreverchild',
+            u'fld': u'v2.foreverchild',
+            u'subdomain': u'www',
+            u'domain': u'v2',
+            u'suffix': u'foreverchild',
+            u'tld': u'foreverchild',
+            u'kwargs': {
+                u'fail_silently': True,
+            },
+        }]
+        reset_settings()
+
+    def tearDown(self):
+        u'Tear down.'
+        reset_settings()
+        Registry.reset()
+
+    @property
+    def good_url(self):
+        return self.good_patterns[0][u'url']
+
+    @property
+    def bad_url(self):
+        return list(self.bad_patterns.keys())[0]
+
+    def get_custom_parser_class(self, uid='custom_mozilla', source_url=None, local_path='tests/res/effective_tld_names_custom.dat.txt'):
+        parser_class = type('CustomMozillaTLDSourceParser', (BaseMozillaTLDSourceParser,), {
+            'uid': uid,
+            'source_url': source_url,
+            'local_path': local_path,
+        })
+        return parser_class
+
+    @log_info
+    def test_0_tld_names_loaded(self):
+        u'Test if tld names are loaded.'
+        get_fld(u'http://www.google.co.uk')
+        from ..utils import tld_names
+        res = (len(tld_names) > 0)
+        self.assertTrue(res)
+        return res
+
+    @internet_available_only
+    @log_info
+    def test_1_update_tld_names(self):
+        u'Test updating the tld names (re-fetch mozilla source).'
+        res = update_tld_names(fail_silently=False)
+        self.assertTrue(res)
+        return res
+
+    @log_info
+    def test_2_fld_good_patterns_pass(self):
+        u'Test good URL patterns.'
+        res = []
+        for data in self.good_patterns:
+            _res = get_fld(data[u'url'], **data[u'kwargs'])
+            self.assertEqual(_res, data[u'fld'])
+            res.append(_res)
+        return res
+
+    @log_info
+    def test_3_fld_bad_patterns_pass(self):
+        u'Test bad URL patterns.'
+        res = []
+        for (url, params) in self.bad_patterns.items():
+            _res = get_fld(url, fail_silently=True)
+            self.assertEqual(_res, None)
+            res.append(_res)
+        return res
+
+    @log_info
+    def test_4_override_settings(self):
+        u'Testing settings override.'
+
+        def override_settings():
+            u'Override settings.'
+            return get_setting(u'DEBUG')
+        self.assertEqual(defaults.DEBUG, override_settings())
+        set_setting(u'DEBUG', True)
+        self.assertEqual(True, override_settings())
+        return override_settings()
+
+    @log_info
+    def test_5_tld_good_patterns_pass_parsed_object(self):
+        u'Test good URL patterns.'
+        res = []
+        for data in self.good_patterns:
+            kwargs = copy.copy(data[u'kwargs'])
+            kwargs.update({
+                u'as_object': True,
+            })
+            _res = get_tld(data[u'url'], **kwargs)
+            self.assertEqual(_res.tld, data[u'tld'])
+            self.assertEqual(_res.subdomain, data[u'subdomain'])
+            self.assertEqual(_res.domain, data[u'domain'])
+            self.assertEqual(_res.suffix, data[u'suffix'])
+            self.assertEqual(_res.fld, data[u'fld'])
+            self.assertEqual(unicode(_res).encode(u'utf8'),
+                             data[u'tld'].encode(u'utf8'))
+            self.assertEqual(_res.__dict__, {
+                u'tld': _res.tld,
+                u'domain': _res.domain,
+                u'subdomain': _res.subdomain,
+                u'fld': _res.fld,
+                u'parsed_url': _res.parsed_url,
+            })
+            res.append(_res)
+        return res
+
+    @log_info
+    def test_6_override_full_names_path(self):
+        default = project_dir(u'dummy.txt')
+        override_base = u'/tmp/test'
+        set_setting(u'NAMES_LOCAL_PATH_PARENT', override_base)
+        modified = project_dir(u'dummy.txt')
+        self.assertNotEqual(default, modified)
+        self.assertEqual(modified, abspath(u'/tmp/test/dummy.txt'))
+
+    @log_info
+    def test_7_public_private(self):
+        res = get_fld(u'http://silly.cc.ua',
+                      fail_silently=True, search_private=False)
+        self.assertEqual(res, None)
+        res = get_fld(u'http://silly.cc.ua',
+                      fail_silently=True, search_private=True)
+        self.assertEqual(res, u'silly.cc.ua')
+        res = get_fld(u'mercy.compute.amazonaws.com',
+                      fail_silently=True, search_private=False, fix_protocol=True)
+        self.assertEqual(res, None)
+        res = get_fld(u'http://whatever.com',
+                      fail_silently=True, search_public=False)
+        self.assertEqual(res, None)
+
+    @log_info
+    def test_8_fld_bad_patterns_exceptions(self):
+        u'Test exceptions.'
+        res = []
+        for (url, params) in self.bad_patterns.items():
+            kwargs = (params[u'kwargs'] if (u'kwargs' in params) else {
+
+            })
+            kwargs.update({
+                u'fail_silently': False,
+            })
+            with self.assertRaises(params[u'exception']):
+                _res = get_fld(url, **kwargs)
+                res.append(_res)
+        return res
+
+    @log_info
+    def test_9_tld_good_patterns_pass(self):
+        u'Test `get_tld` good URL patterns.'
+        res = []
+        for data in self.good_patterns:
+            _res = get_tld(data[u'url'], **data[u'kwargs'])
+            self.assertEqual(_res, data[u'tld'])
+            res.append(_res)
+        return res
+
+    @log_info
+    def test_10_tld_bad_patterns_pass(self):
+        u'Test `get_tld` bad URL patterns.'
+        res = []
+        for (url, params) in self.bad_patterns.items():
+            _res = get_tld(url, fail_silently=True)
+            self.assertEqual(_res, None)
+            res.append(_res)
+        return res
+
+    @log_info
+    def test_11_parse_tld_good_patterns(self):
+        u'Test `parse_tld` good URL patterns.'
+        res = []
+        for data in self.good_patterns:
+            _res = parse_tld(data[u'url'], **data[u'kwargs'])
+            self.assertEqual(
+                _res, (data[u'tld'], data[u'domain'], data[u'subdomain']))
+            res.append(_res)
+        return res
+
+    @log_info
+    def test_12_is_tld_good_patterns(self):
+        u'Test `is_tld` good URL patterns.'
+        for data in self.good_patterns:
+            self.assertTrue(is_tld(data[u'tld']))
+
+    @log_info
+    def test_13_is_tld_bad_patterns(self):
+        u'Test `is_tld` bad URL patterns.'
+        for _tld in self.invalid_tlds:
+            self.assertFalse(is_tld(_tld))
+
+    @log_info
+    def test_14_fail_update_tld_names(self):
+        u'Test fail `update_tld_names`.'
+        parser_class = self.get_custom_parser_class(
+            uid='custom_mozilla_2', source_url='i-do-not-exist')
+        with self.assertRaises(TldIOError):
+            update_tld_names(fail_silently=False, parser_uid=parser_class.uid)
+        self.assertFalse(update_tld_names(
+            fail_silently=True, parser_uid=parser_class.uid))
+
+    @log_info
+    def test_15_fail_get_fld_wrong_kwargs(self):
+        u'Test fail `get_fld` with wrong kwargs.'
+        with self.assertRaises(TldImproperlyConfigured):
+            get_fld(self.good_url, as_object=True)
+
+    @log_info
+    def test_16_fail_parse_tld(self):
+        u'Test fail `parse_tld`.\n\n        Assert raise TldIOError on wrong `NAMES_SOURCE_URL` for `parse_tld`.\n        '
+        parser_class = self.get_custom_parser_class(
+            source_url='i-do-not-exist')
+        parsed_tld = parse_tld(
+            self.bad_url, fail_silently=False, parser_class=parser_class)
+        self.assertEqual(parsed_tld, (None, None, None))
+
+    @log_info
+    def test_17_get_tld_names_and_reset_tld_names(self):
+        u'Test fail `get_tld_names` and repair using `reset_tld_names`.'
+        tmp_filename = join(gettempdir(), u''.join(
+            [u'{}'.format(self.faker.uuid4()), u'.dat.txt']))
+        parser_class = self.get_custom_parser_class(
+            source_url='i-do-not-exist', local_path=tmp_filename)
+        reset_tld_names()
+        if True:
+            with self.assertRaises(TldIOError):
+                get_tld_names(fail_silently=False, parser_class=parser_class)
+        tmp_filename = join(gettempdir(), u''.join(
+            [u'{}'.format(self.faker.uuid4()), u'.dat.txt']))
+        parser_class_2 = self.get_custom_parser_class(
+            source_url='i-do-not-exist-2', local_path=tmp_filename)
+        reset_tld_names()
+        if True:
+            self.assertIsNone(get_tld_names(
+                fail_silently=True, parser_class=parser_class_2))
+
+    @internet_available_only
+    @log_info
+    def test_18_update_tld_names_cli(self):
+        u'Test the return code of the CLI version of `update_tld_names`.'
+        reset_tld_names()
+        res = update_tld_names_cli()
+        self.assertEqual(res, 0)
+
+    @log_info
+    def test_19_parse_tld_custom_tld_names_good_patterns(self):
+        u'Test `parse_tld` good URL patterns for custom tld names.'
+        res = []
+        for data in self.good_patterns_custom_parser:
+            kwargs = copy.copy(data[u'kwargs'])
+            kwargs.update({
+                u'parser_class': self.get_custom_parser_class(),
+            })
+            _res = parse_tld(data[u'url'], **kwargs)
+            self.assertEqual(
+                _res, (data[u'tld'], data[u'domain'], data[u'subdomain']))
+            res.append(_res)
+        return res
+
+    @log_info
+    def test_20_tld_custom_tld_names_good_patterns_pass_parsed_object(self):
+        u'Test `get_tld` good URL patterns for custom tld names.'
+        res = []
+        for data in self.good_patterns_custom_parser:
+            kwargs = copy.copy(data[u'kwargs'])
+            kwargs.update({
+                u'as_object': True,
+                u'parser_class': self.get_custom_parser_class(),
+            })
+            _res = get_tld(data[u'url'], **kwargs)
+            self.assertEqual(_res.tld, data[u'tld'])
+            self.assertEqual(_res.subdomain, data[u'subdomain'])
+            self.assertEqual(_res.domain, data[u'domain'])
+            self.assertEqual(_res.suffix, data[u'suffix'])
+            self.assertEqual(_res.fld, data[u'fld'])
+            self.assertEqual(unicode(_res).encode(u'utf8'),
+                             data[u'tld'].encode(u'utf8'))
+            self.assertEqual(_res.__dict__, {
+                u'tld': _res.tld,
+                u'domain': _res.domain,
+                u'subdomain': _res.subdomain,
+                u'fld': _res.fld,
+                u'parsed_url': _res.parsed_url,
+            })
+            res.append(_res)
+        return res
+
+    @log_info
+    def test_21_reset_tld_names_for_custom_parser(self):
+        u'Test `reset_tld_names` for `tld_names_local_path`.'
+        res = []
+        parser_class = self.get_custom_parser_class()
+        for data in self.good_patterns_custom_parser:
+            kwargs = copy.copy(data[u'kwargs'])
+            kwargs.update({
+                u'as_object': True,
+                u'parser_class': self.get_custom_parser_class(),
+            })
+            _res = get_tld(data[u'url'], **kwargs)
+            self.assertEqual(_res.tld, data[u'tld'])
+            self.assertEqual(_res.subdomain, data[u'subdomain'])
+            self.assertEqual(_res.domain, data[u'domain'])
+            self.assertEqual(_res.suffix, data[u'suffix'])
+            self.assertEqual(_res.fld, data[u'fld'])
+            self.assertEqual(unicode(_res).encode(u'utf8'),
+                             data[u'tld'].encode(u'utf8'))
+            self.assertEqual(_res.__dict__, {
+                u'tld': _res.tld,
+                u'domain': _res.domain,
+                u'subdomain': _res.subdomain,
+                u'fld': _res.fld,
+                u'parsed_url': _res.parsed_url,
+            })
+            res.append(_res)
+        tld_names = get_tld_names_container()
+        self.assertIn(parser_class.local_path, tld_names)
+        reset_tld_names(parser_class.local_path)
+        self.assertNotIn(parser_class.local_path, tld_names)
+        return res
+
+    @log_info
+    def test_22_fail_define_custom_parser_class_without_uid(self):
+        u'Test fail define custom parser class without `uid`.'
+
+        class CustomParser(BaseTLDSourceParser):
+            pass
+
+        class AnotherCustomParser(BaseTLDSourceParser):
+            uid = u'another-custom-parser'
+        with self.assertRaises(TldImproperlyConfigured):
+            CustomParser.get_tld_names()
+        with self.assertRaises(NotImplementedError):
+            AnotherCustomParser.get_tld_names()
+
+    @log_info
+    def test_23_len_trie_nodes(self):
+        u'Test len of the trie nodes.'
+        get_tld(u'http://delusionalinsanity.com')
+        tld_names = get_tld_names_container()
+        self.assertGreater(
+            len(tld_names[MozillaTLDSourceParser.local_path]), 0)
+
+    @log_info
+    def test_24_get_tld_names_no_arguments(self):
+        u'Test len of the trie nodes.'
+        tld_names = get_tld_names()
+        self.assertGreater(len(tld_names), 0)
+
+
+if (__name__ == u'__main__'):
+    unittest.main()

Contents/Libraries/Shared/tld/trie.py

@@ -0,0 +1,54 @@
+# -*- coding: utf-8 -*-
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+from __future__ import unicode_literals
+__author__ = u'Artur Barseghyan'
+__copyright__ = u'2013-2019 Artur Barseghyan'
+__license__ = u'MPL-1.1 OR GPL-2.0-only OR LGPL-2.1-or-later'
+__all__ = (u'Trie', u'TrieNode')
+
+
+class TrieNode(object):
+    u'Class representing a single Trie node.'
+    __slots__ = (u'children', u'exception', u'leaf', u'private')
+
+    def __init__(self):
+        self.children = None
+        self.exception = None
+        self.leaf = False
+        self.private = False
+
+
+class Trie(object):
+    u'An adhoc Trie data structure to store tlds in reverse notation order.'
+
+    def __init__(self):
+        self.root = TrieNode()
+        self.__nodes = 0
+
+    def __len__(self):
+        return self.__nodes
+
+    def add(self, tld, private=False):
+        node = self.root
+        for part in reversed(tld.split(u'.')):
+            if part.startswith(u'!'):
+                node.exception = part[1:]
+                break
+            if (node.children is None):
+                node.children = {
+
+                }
+                child = TrieNode()
+            else:
+                child = node.children.get(part)
+                if (child is None):
+                    child = TrieNode()
+            node.children[part] = child
+            node = child
+        node.leaf = True
+        if private:
+            node.private = True
+        self.__nodes += 1

Contents/Libraries/Shared/tld/utils.py

@@ -0,0 +1,271 @@
+# -*- coding: utf-8 -*-
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+from __future__ import unicode_literals
+from __future__ import unicode_literals
+import argparse
+from codecs import open as codecs_open
+from backports.functools_lru_cache import lru_cache
+from os.path import isabs
+import sys
+from typing import Dict, Type, Union, Tuple, List
+try:
+    from urllib.parse import urlsplit, SplitResult
+except ImportError:
+    from six.moves.urllib_parse import urlsplit, SplitResult
+from .base import BaseTLDSourceParser
+from .exceptions import TldBadUrl, TldDomainNotFound, TldImproperlyConfigured, TldIOError
+from .helpers import project_dir
+from .trie import Trie
+from .registry import Registry
+from .result import Result
+__author__ = u'Artur Barseghyan'
+__copyright__ = u'2013-2019 Artur Barseghyan'
+__license__ = u'MPL-1.1 OR GPL-2.0-only OR LGPL-2.1-or-later'
+__all__ = (u'BaseMozillaTLDSourceParser', u'get_fld', u'get_tld', u'get_tld_names', u'get_tld_names_container', u'is_tld', u'MozillaTLDSourceParser', u'parse_tld',
+           u'pop_tld_names_container', u'process_url', u'reset_tld_names', u'Result', u'update_tld_names', u'update_tld_names_cli', u'update_tld_names_container')
+tld_names = {
+
+}
+
+
+def get_tld_names_container():
+    u'Get container of all tld names.\n\n    :return:\n    :rtype dict:\n    '
+    global tld_names
+    return tld_names
+
+
+def update_tld_names_container(tld_names_local_path, trie_obj):
+    u'Update TLD Names container item.\n\n    :param tld_names_local_path:\n    :param trie_obj:\n    :return:\n    '
+    global tld_names
+    tld_names.update({
+        tld_names_local_path: trie_obj,
+    })
+
+
+def pop_tld_names_container(tld_names_local_path):
+    u'Remove TLD names container item.\n\n    :param tld_names_local_path:\n    :return:\n    '
+    global tld_names
+    tld_names.pop(tld_names_local_path, None)
+
+
+@lru_cache(maxsize=128, typed=True)
+def update_tld_names(fail_silently=False, parser_uid=None):
+    u'Update TLD names.\n\n    :param fail_silently:\n    :param parser_uid:\n    :return:\n    '
+    results = []
+    results_append = results.append
+    if parser_uid:
+        parser_cls = Registry.get(parser_uid, None)
+        if (parser_cls and parser_cls.source_url):
+            results_append(parser_cls.update_tld_names(
+                fail_silently=fail_silently))
+    else:
+        for (parser_uid, parser_cls) in Registry.items():
+            if (parser_cls and parser_cls.source_url):
+                results_append(parser_cls.update_tld_names(
+                    fail_silently=fail_silently))
+    return all(results)
+
+
+def update_tld_names_cli():
+    u'CLI wrapper for update_tld_names.\n\n    Since update_tld_names returns True on success, we need to negate the\n    result to match CLI semantics.\n    '
+    parser = argparse.ArgumentParser(description='Update TLD names')
+    parser.add_argument(u'parser_uid', nargs='?', default=None,
+                        help='UID of the parser to update TLD names for.')
+    parser.add_argument(u'--fail-silently', dest='fail_silently',
+                        default=False, action='store_true', help='Fail silently')
+    args = parser.parse_args(sys.argv[1:])
+    parser_uid = args.parser_uid
+    fail_silently = args.fail_silently
+    return int((not update_tld_names(parser_uid=parser_uid, fail_silently=fail_silently)))
+
+
+def get_tld_names(fail_silently=False, retry_count=0, parser_class=None):
+    u'Build the ``tlds`` list if empty. Recursive.\n\n    :param fail_silently: If set to True, no exceptions are raised and None\n        is returned on failure.\n    :param retry_count: If greater than 1, we raise an exception in order\n        to avoid infinite loops.\n    :param parser_class:\n    :type fail_silently: bool\n    :type retry_count: int\n    :type parser_class: BaseTLDSourceParser\n    :return: List of TLD names\n    :rtype: obj:`tld.utils.Trie`\n    '
+    if (not parser_class):
+        parser_class = MozillaTLDSourceParser
+    return parser_class.get_tld_names(fail_silently=fail_silently, retry_count=retry_count)
+
+
+class BaseMozillaTLDSourceParser(BaseTLDSourceParser):
+
+    @classmethod
+    def get_tld_names(cls, fail_silently=False, retry_count=0):
+        u'Parse.\n\n        :param fail_silently:\n        :param retry_count:\n        :return:\n        '
+        if (retry_count > 1):
+            if fail_silently:
+                return None
+            else:
+                raise TldIOError
+        global tld_names
+        _tld_names = tld_names
+        if ((cls.local_path in _tld_names) and (_tld_names[cls.local_path] is not None)):
+            return _tld_names
+        local_file = None
+        try:
+            if isabs(cls.local_path):
+                local_path = cls.local_path
+            else:
+                local_path = project_dir(cls.local_path)
+            local_file = codecs_open(local_path, u'r', encoding='utf8')
+            trie = Trie()
+            trie_add = trie.add
+            private_section = False
+            for line in local_file:
+                if (u'===BEGIN PRIVATE DOMAINS===' in line):
+                    private_section = True
+                if (u'// xn--' in line):
+                    line = line.split()[1]
+                if (line[0] in (u'/', u'\n')):
+                    continue
+                trie_add(u''.join([u'{}'.format(line.strip())]),
+                         private=private_section)
+            update_tld_names_container(cls.local_path, trie)
+            local_file.close()
+        except IOError as err:
+            cls.update_tld_names(fail_silently=fail_silently)
+            retry_count += 1
+            return cls.get_tld_names(fail_silently=fail_silently, retry_count=retry_count)
+        except Exception as err:
+            if fail_silently:
+                return None
+            else:
+                raise err
+        finally:
+            try:
+                local_file.close()
+            except Exception:
+                pass
+        return _tld_names
+
+
+class MozillaTLDSourceParser(BaseMozillaTLDSourceParser):
+    u'Mozilla TLD source.'
+    uid = u'mozilla'
+    source_url = u'http://mxr.mozilla.org/mozilla/source/netwerk/dns/src/effective_tld_names.dat?raw=1'
+    local_path = u'res/effective_tld_names.dat.txt'
+
+
+def process_url(url, fail_silently=False, fix_protocol=False, search_public=True, search_private=True, parser_class=MozillaTLDSourceParser):
+    u'Process URL.\n\n    :param parser_class:\n    :param url:\n    :param fail_silently:\n    :param fix_protocol:\n    :param search_public:\n    :param search_private:\n    :return:\n    '
+    if (not (search_public or search_private)):
+        raise TldImproperlyConfigured(
+            u'Either `search_public` or `search_private` (or both) shall be set to True.')
+    _tld_names = get_tld_names(
+        fail_silently=fail_silently, parser_class=parser_class)
+    if (not isinstance(url, SplitResult)):
+        url = url.lower()
+        if (fix_protocol and (not url.startswith((u'//', u'http://', u'https://')))):
+            url = u''.join([u'https://', u'{}'.format(url)])
+        parsed_url = urlsplit(url)
+    else:
+        parsed_url = url
+    domain_name = parsed_url.hostname
+    if (not domain_name):
+        if fail_silently:
+            return (None, None, parsed_url)
+        else:
+            raise TldBadUrl(url=url)
+    domain_parts = domain_name.split(u'.')
+    tld_names_local_path = parser_class.local_path
+    node = _tld_names[tld_names_local_path].root
+    current_length = 0
+    tld_length = 0
+    match = None
+    len_domain_parts = len(domain_parts)
+    for i in reversed(range(len_domain_parts)):
+        part = domain_parts[i]
+        if (node.children is None):
+            break
+        if (part == node.exception):
+            break
+        child = node.children.get(part)
+        if (child is None):
+            child = node.children.get(u'*')
+        if (child is None):
+            break
+        current_length += 1
+        node = child
+        if node.leaf:
+            tld_length = current_length
+            match = node
+    if ((match is None) or (not match.leaf) or ((not search_public) and (not match.private)) or ((not search_private) and match.private)):
+        if fail_silently:
+            return (None, None, parsed_url)
+        else:
+            raise TldDomainNotFound(domain_name=domain_name)
+    if (len_domain_parts == tld_length):
+        non_zero_i = (- 1)
+    else:
+        non_zero_i = max(1, (len_domain_parts - tld_length))
+    return (domain_parts, non_zero_i, parsed_url)
+
+
+def get_fld(url, fail_silently=False, fix_protocol=False, search_public=True, search_private=True, parser_class=MozillaTLDSourceParser, **kwargs):
+    u"Extract the first level domain.\n\n    Extract the top level domain based on the mozilla's effective TLD names\n    dat file. Returns a string. May throw ``TldBadUrl`` or\n    ``TldDomainNotFound`` exceptions if there's bad URL provided or no TLD\n    match found respectively.\n\n    :param url: URL to get top level domain from.\n    :param fail_silently: If set to True, no exceptions are raised and None\n        is returned on failure.\n    :param fix_protocol: If set to True, missing or wrong protocol is\n        ignored (https is appended instead).\n    :param search_public: If set to True, search in public domains.\n    :param search_private: If set to True, search in private domains.\n    :param parser_class:\n    :type url: str\n    :type fail_silently: bool\n    :type fix_protocol: bool\n    :type search_public: bool\n    :type search_private: bool\n    :return: String with top level domain (if ``as_object`` argument\n        is set to False) or a ``tld.utils.Result`` object (if ``as_object``\n        argument is set to True); returns None on failure.\n    :rtype: str\n    "
+    if (u'as_object' in kwargs):
+        raise TldImproperlyConfigured(
+            u'`as_object` argument is deprecated for `get_fld`. Use `get_tld` instead.')
+    (domain_parts, non_zero_i, parsed_url) = process_url(url=url, fail_silently=fail_silently,
+                                                         fix_protocol=fix_protocol, search_public=search_public, search_private=search_private, parser_class=parser_class)
+    if (domain_parts is None):
+        return None
+    if (non_zero_i < 0):
+        return parsed_url.hostname
+    return u'.'.join(domain_parts[(non_zero_i - 1):])
+
+
+def get_tld(url, fail_silently=False, as_object=False, fix_protocol=False, search_public=True, search_private=True, parser_class=MozillaTLDSourceParser):
+    u"Extract the top level domain.\n\n    Extract the top level domain based on the mozilla's effective TLD names\n    dat file. Returns a string. May throw ``TldBadUrl`` or\n    ``TldDomainNotFound`` exceptions if there's bad URL provided or no TLD\n    match found respectively.\n\n    :param url: URL to get top level domain from.\n    :param fail_silently: If set to True, no exceptions are raised and None\n        is returned on failure.\n    :param as_object: If set to True, ``tld.utils.Result`` object is returned,\n        ``domain``, ``suffix`` and ``tld`` properties.\n    :param fix_protocol: If set to True, missing or wrong protocol is\n        ignored (https is appended instead).\n    :param search_public: If set to True, search in public domains.\n    :param search_private: If set to True, search in private domains.\n    :param parser_class:\n    :type url: str\n    :type fail_silently: bool\n    :type as_object: bool\n    :type fix_protocol: bool\n    :type search_public: bool\n    :type search_private: bool\n    :return: String with top level domain (if ``as_object`` argument\n        is set to False) or a ``tld.utils.Result`` object (if ``as_object``\n        argument is set to True); returns None on failure.\n    :rtype: str\n    "
+    (domain_parts, non_zero_i, parsed_url) = process_url(url=url, fail_silently=fail_silently,
+                                                         fix_protocol=fix_protocol, search_public=search_public, search_private=search_private, parser_class=parser_class)
+    if (domain_parts is None):
+        return None
+    if (not as_object):
+        if (non_zero_i < 0):
+            return parsed_url.hostname
+        return u'.'.join(domain_parts[non_zero_i:])
+    if (non_zero_i < 0):
+        subdomain = u''
+        domain = u''
+        _tld = parsed_url.hostname
+    else:
+        subdomain = u'.'.join(domain_parts[:(non_zero_i - 1)])
+        domain = u'.'.join(domain_parts[(non_zero_i - 1):non_zero_i])
+        _tld = u'.'.join(domain_parts[non_zero_i:])
+    return Result(subdomain=subdomain, domain=domain, tld=_tld, parsed_url=parsed_url)
+
+
+def parse_tld(url, fail_silently=False, fix_protocol=False, search_public=True, search_private=True, parser_class=MozillaTLDSourceParser):
+    u'Parse TLD into parts.\n\n    :param url:\n    :param fail_silently:\n    :param fix_protocol:\n    :param search_public:\n    :param search_private:\n    :param parser_class:\n    :return:\n    :rtype: tuple\n    '
+    try:
+        obj = get_tld(url, fail_silently=fail_silently, as_object=True, fix_protocol=fix_protocol,
+                      search_public=search_public, search_private=search_private, parser_class=parser_class)
+        _tld = obj.tld
+        domain = obj.domain
+        subdomain = obj.subdomain
+    except (TldBadUrl, TldDomainNotFound, TldImproperlyConfigured, TldIOError):
+        _tld = None
+        domain = None
+        subdomain = None
+    return (_tld, domain, subdomain)
+
+
+def is_tld(value, search_public=True, search_private=True, parser_class=MozillaTLDSourceParser):
+    u'Check if given URL is tld.\n\n    :param value: URL to get top level domain from.\n    :param search_public: If set to True, search in public domains.\n    :param search_private: If set to True, search in private domains.\n    :param parser_class:\n    :type value: str\n    :type search_public: bool\n    :type search_private: bool\n    :return:\n    :rtype: bool\n    '
+    _tld = get_tld(url=value, fail_silently=True, fix_protocol=True,
+                   search_public=search_public, search_private=search_private, parser_class=parser_class)
+    return (value == _tld)
+
+
+def reset_tld_names(tld_names_local_path=None):
+    u'Reset the ``tld_names`` to empty value.\n\n    If ``tld_names_local_path`` is given, removes specified\n    entry from ``tld_names`` instead.\n\n    :param tld_names_local_path:\n    :type tld_names_local_path: str\n    :return:\n    '
+    if tld_names_local_path:
+        pop_tld_names_container(tld_names_local_path)
+    else:
+        global tld_names
+        tld_names = {
+
+        }

Contents/advanced_settings.json.template

@@ -22,17 +22,20 @@ Don't expect support if you mess this up.
     // when the find better subtitles task finds a subtitle for an item that has an active extracted embedded subtitle
     // set, what should be the minimum score that subtitle has to have in order to be considered better?
     "find_better_as_extracted_tv_score": 352,
-    "find_better_as_extracted_movie_score": 82,
+    "find_better_as_extracted_movie_score": 112,
 
     // SZ can use mediainfo if present to detect titles/forced state of MP4 MOV_TEXT, because the PMS currently doesn't
     // set the title attribute
-    "dont_use_mediainfo_mp4": False,
+    "dont_use_mediainfo_mp4": false,
 
     // specific mediainfo binary path
     "mediainfo_bin": null,
 
     "debug_i18n": false,
 
+    // refresh item after plugin refresh has been called (default: 5)
+    "refresh_after_called": 5,
+
     // per-provider-config
     "providers": {
         // enabled_for: specifies which media type to enable the provider for. does not override global/throttled
@@ -85,9 +88,12 @@ Don't expect support if you mess this up.
         "sonarr": {
             // don't verify HTTPS certificates? Set to True for self-signed certificates
             "ssl_no_verify": false,
+            // custom path to certificate pem file
+            "pem_file": null,
         },
         "radarr": {
             "ssl_no_verify": false,
+            "pem_file": null,
         },
     }
 }

README.md

@@ -1,7 +1,7 @@
 # Sub-Zero for Plex
 [![](https://img.shields.io/github/release/pannal/Sub-Zero.bundle.svg?style=flat&label=stable)](https://github.com/pannal/Sub-Zero.bundle/releases/latest) 
 [![master](https://img.shields.io/badge/master-stable-green.svg?maxAge=2592000)]()
-[![Maintenance](https://img.shields.io/maintenance/yes/2020.svg)]()
+[![Maintenance](https://img.shields.io/maintenance/yes/2024.svg)]()
 [![Slack Status](https://szslack.fragstore.net/badge.svg)](https://szslack.fragstore.net)
 [![FOSSA Status](https://app.fossa.io/api/projects/git%2Bgithub.com%2Fpannal%2FSub-Zero.bundle.svg?type=shield)](https://app.fossa.io/projects/git%2Bgithub.com%2Fpannal%2FSub-Zero.bundle?ref=badge_shield)
 
@@ -12,6 +12,11 @@ Check out **[the Sub-Zero Wiki](https://github.com/pannal/Sub-Zero.bundle/wiki)*
 
 <br />
 
+# DEPRECATED, USE [BAZARR](https://www.bazarr.media/)
+
+## Legacy maintenance mode
+This addon will not be developed any further. It still works and arguably is still the best for managing subtitles when using Plex. As long as Plex Inc. supports agents, Sub-Zero will be maintained to work with the latest PMS version.
+
 ---
 
 **[Kitana is now required to have a UI](https://github.com/pannal/Kitana)**
@@ -24,7 +29,7 @@ Check out **[the Sub-Zero Wiki](https://github.com/pannal/Sub-Zero.bundle/wiki)*
 
 ## Helping development
 
-If you like this, buy me a beer: <br>[![Donate](https://www.paypalobjects.com/en_US/i/btn/btn_donate_LG.gif)](https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=G9VKR2B8PMNKG) <br>or become a Patreon starting at **1 $ / month** <br><a href="https://www.patreon.com/subzero_plex" target="_blank"><img src="https://i0.wp.com/tablecakes.com/wp-content/uploads/2018/08/become-a-patron-button.png" height="54" /></a> <br>or use the OpenSubtitles Sub-Zero affiliate link to become VIP <br>**10€/year, ad-free subs, 1000 subs/day, no-cache *VIP* server**<br><a href="http://v.ht/osvip" target="_blank"><img src="https://static.opensubtitles.org/gfx/logo.gif" height="50" /></a> 
+If you like this, buy me a beer: <br>[![Donate](https://www.paypalobjects.com/en_US/i/btn/btn_donate_LG.gif)](https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=G9VKR2B8PMNKG) <br>or become a Patreon starting at **1 $ / month** <br><a href="https://www.patreon.com/subzero_plex" target="_blank"><img src="https://images.squarespace-cdn.com/content/v1/56c7831bf8baf3ae17ce9259/1561826418792-IJOMFASTOR6CW80N5W0Z/ke17ZwdGBToddI8pDm48kEycOuEejcFJqqLot0yQ4VVZw-zPPgdn4jUwVcJE1ZvWEtT5uBSRWt4vQZAgTJucoTqqXjS3CfNDSuuf31e0tVFrfGYmOrPFZFUXr1UxW4wA0PgPOjs31URy2JeWL9DdYhur-lC0WofN0YB1wFg-ZW0/footer-patreon.png?format=500w" height="54" /></a>
 
 If you register with an anti-captcha service and you decide to use [Anti-Captcha.com](http://getcaptchasolution.com/kkvviom7nh), you can use [this affiliate link](http://getcaptchasolution.com/kkvviom7nh) to help development.
 
@@ -86,31 +91,43 @@ In addition to that Sub-Zero also fixes problems introduced by the subtitle crea
 Ever had broken music icons in a subtitle? Nordic characters like `Å` which turned into `Ã¥`? Not anymore.
 
 ## Installation
-Simply go to the Plex Plugins in your Plex Media Server, search for Sub-Zero and install it.
-For further help or manual installation, [please go to the wiki](https://github.com/pannal/Sub-Zero.bundle/wiki/Installation).
+[Please go to the wiki](https://github.com/pannal/Sub-Zero.bundle/wiki/Installation).
 
 ## Big thanks to the beta/i18n testing team (in no particular order)!
 the.vbm, mmgoodnow, Vertig0ne, thliu78, tattoomees, ostman, count_confucius, eherberg, tywilliams_88, Swanny, Jippo, Joost1991 / joost, Marik, Jon, AmbyDK, Clay, Abenlog, michael, smikwily, shoghicp, Zuikkis, Isilorn, Jacob K, Ninjouz, chopeta, fvb, Uthman, Claus Møller, Semi Doludizgin, Rafael, sugarman402, Morpheus1333, Yamil.llanos, Notorius28 
 
 ## Changelog
 
+2.6.5.3280
 
-2.6.5.3217
+temporarily enable OpenSubtitles.com instead of OpenSubtitles.org.
+You need to have an account there and an API consumer configured. Enter your API key in settings.
 
+This is barely tested but should work for basic usage.
+
+THIS PLUGIN IS DEPRECATED, PLEASE USE BAZARR!
+
+Changelog
+- cheaply backport opensubtitlescom from bazarr
+
+
+2.6.5.3277
+- core: fix enabled library/agents detection (Plex removed certain features)
+  fix Plex agent integration; Plex Inc removed certain attributes; SZ is now limited to thetvdb, thetvdbdvdorder, hama, themoviedb, imdb)
+
+
+
+2.6.5.3268
 subscene, addic7ed
 - either of those providers might impose a reCAPTCHA verification. In order to use those providers, please create an account at an AntiCaptcha service ([anti-captcha.com](http://getcaptchasolution.com/kkvviom7nh) or [deathbycaptcha.com](http://deathbycaptcha.com)), add funds, then supply your credentials/apikey in the configuration
 
 Changelog
-- core: also extract (missing) embedded subtitles when SearchAllRecentlyAddedMissing is running
-- core: core: clarify detecting streams (in logs)
-- core: UnRAR: set binary to executable, even if not checked out from git; might fix #693
-- core: bazarr-backport: morpheus65535/bazarr#703: use proper language code detection instead of a wild guess; should fix bad existing subtitle detection
-- core: bazarr-backport: morpheus65535/bazarr#660: fix BOM encoding stuff
-- core: bazarr-backport: morpheus65535/bazarr#656 further generalize formats; skip release group match if format match failed
-- core: fix stream detection when using mediainfo (#711)
-- config/core: make periodic SZ-internal subtitle maintenance interval configurable
-- providers: add BSPlayer Subtitles
-- providers: add ScrewZira (Hebrew)
+- clarify README
+- core: fix custom folder handling; #761
+- core: providers: screwzira: move to ktuvit and wizdom
+- core: add option to not download subtitles for certain audio languages existing; and/or no audio stream; fix #756
+- core: delay item refreshes after refresh call (default: 5 seconds; exposed in advanced settings)
+- menu: allow extraction of embedded subtitles for whole tv shows
 
 [older changes](CHANGELOG.md)