Update README.md

fix advanced_settings.json.template
Update README.md
2024-01-27 02:28:07 +01:00 · 2024-01-27 01:45:43 +01:00 · 2024-01-27 00:11:07 +01:00 · 2024-01-27 00:09:06 +01:00 · 2024-01-27 00:08:25 +01:00 · 2024-01-27 00:06:17 +01:00
121 changed files with 58183 additions and 1303 deletions
@@ -1,3 +1,201 @@
+2.6.5.3280
+
+temporarily enable OpenSubtitles.com instead of OpenSubtitles.org.
+You need to have an account there and an API consumer configured. Enter your API key in settings.
+
+This is barely tested but should work for basic usage.
+
+THIS PLUGIN IS DEPRECATED, PLEASE USE BAZARR!
+
+Changelog
+- cheaply backport opensubtitlescom from bazarr
+
+
+2.6.5.3268
+subscene, addic7ed
+- either of those providers might impose a reCAPTCHA verification. In order to use those providers, please create an account at an AntiCaptcha service ([anti-captcha.com](http://getcaptchasolution.com/kkvviom7nh) or [deathbycaptcha.com](http://deathbycaptcha.com)), add funds, then supply your credentials/apikey in the configuration
+
+Changelog
+- clarify README
+- core: fix custom folder handling; #761
+- core: providers: screwzira: move to ktuvit and wizdom
+- core: add option to not download subtitles for certain audio languages existing; and/or no audio stream; fix #756
+- core: delay item refreshes after refresh call (default: 5 seconds; exposed in advanced settings)
+- menu: allow extraction of embedded subtitles for whole tv shows
+
+
+2.6.5.3247
+
+subscene, addic7ed
+
+either of those providers might impose a reCAPTCHA verification. In order to use those providers, please create an account at an AntiCaptcha service (anti-captcha.com or deathbycaptcha.com), add funds, then supply your credentials/apikey in the configuration
+Changelog
+core: fix for tv.plex.agents.movie not populating its media types
+core: tasks: findBetterSubtitles: increase minimum score for better subtitles for movies with extracted embedded subs from 82 to 112
+
+
+2.6.5.3241
+
+subscene, addic7ed
+
+either of those providers might impose a reCAPTCHA verification. In order to use those providers, please create an account at an AntiCaptcha service (anti-captcha.com or deathbycaptcha.com), add funds, then supply your credentials/apikey in the configuration
+Changelog
+
+core: add support for new Plex Movie agent
+core: remove py3 compat breaking unnecessary change
+core: skip drawing tags for SRT
+core: advanced_settings: refiners: drone: add custom pem_file support; fixes #735
+providers: core: set DownloadLimitPerDayExceeded timeout to 4 hours (was one day);
+providers: addic7ed: limit downloads per day; add vip setting
+providers: addic7ed: properly compare last_dl, add last_reset tracking info to log #723
+providers: addic7ed: properly implement limits
+submod: HI: remove more music tags
+submod: common CM_punctuation_space2: detect AND don't try changing domain/url/host when fixing punctuation
+
+
+2.6.5.3223
+
+subscene, addic7ed
+
+either of those providers might impose a reCAPTCHA verification. In order to use those providers, please create an account at an AntiCaptcha service (anti-captcha.com or deathbycaptcha.com), add funds, then supply your credentials/apikey in the configuration
+Changelog
+
+core: scoring: reorder subtitles based on second non-hash-score if main hash score is the same; morpheus65535/bazarr#821
+providers: bsplayer: verify hash; clean up
+
+
+2.6.5.3217
+
+subscene, addic7ed
+- either of those providers might impose a reCAPTCHA verification. In order to use those providers, please create an account at an AntiCaptcha service ([anti-captcha.com](http://getcaptchasolution.com/kkvviom7nh) or [deathbycaptcha.com](http://deathbycaptcha.com)), add funds, then supply your credentials/apikey in the configuration
+
+Changelog
+- core: also extract (missing) embedded subtitles when SearchAllRecentlyAddedMissing is running
+- core: core: clarify detecting streams (in logs)
+- core: UnRAR: set binary to executable, even if not checked out from git; might fix #693
+- core: bazarr-backport: morpheus65535/bazarr#703: use proper language code detection instead of a wild guess; should fix bad existing subtitle detection
+- core: bazarr-backport: morpheus65535/bazarr#660: fix BOM encoding stuff
+- core: bazarr-backport: morpheus65535/bazarr#656 further generalize formats; skip release group match if format match failed
+- core: fix stream detection when using mediainfo (#711)
+- config/core: make periodic SZ-internal subtitle maintenance interval configurable
+- providers: add BSPlayer Subtitles
+- providers: add ScrewZira (Hebrew)
+
+2.6.5.3183
+
+
+subscene, addic7ed
+- either of those providers might impose a reCAPTCHA verification. In order to use those providers, please create an account at an AntiCaptcha service ([anti-captcha.com](http://getcaptchasolution.com/kkvviom7nh) or [deathbycaptcha.com](http://deathbycaptcha.com)), add funds, then supply your credentials/apikey in the configuration
+
+Changelog
+- core: don't fall back to default providers if none enabled
+- core: don't process any further if stream info is missing
+- core: support using mediainfo for retrieving MP4 MOV_TEXT subtitle stream titles (PMS bug)
+- core: fix embedded subtitle extraction in some cases (#681, #680)
+- core: scanning: add additional INFO logging for undetected languages
+- core: bazarr-backport: remove existing subtitle file, to support MergerFS
+- core: bazarr-backport: generic 10 minute throttling if uncaught exception occurs
+- providers: addic7ed: fix recaptcha solving; fix show ID retrieval (#681)
+- providers: addic7ed: add timeout on authentication error
+- providers: addic7ed: fix shows with dots in them (Mayans M.C.)
+- providers: addic7ed: fix detection of completed subtitle for non-english users (#686)
+- providers: addic7ed: add more timeouts in the login process
+- providers: argenteam: bazarr-backport: use new url; fixes
+
+
+2.6.5.3152
+
+subscene, addic7ed
+- either of those providers might impose a reCAPTCHA verification. In order to use those providers, please create an account at an AntiCaptcha service ([anti-captcha.com](http://getcaptchasolution.com/kkvviom7nh) or [deathbycaptcha.com](http://deathbycaptcha.com)), add funds, then supply your credentials/apikey in the configuration
+
+Changelog
+- core: fix core issue possibly impacting results on OpenSubtitles in certain conditions
+- core: fix default values of opensubtitles-skip-wrong-fps, use_https; fix #676
+- core: fix for determining whether to search under certain circumstances; fixes #666
+- core: #664 fix missing language processing of multiple videos refreshed at once
+- core: #661 fix match strictness when determining preexisting external subtitles
+- providers: titlovi: New implementation of Titlovi using API (thanks @viking1304)
+
+
+2.6.5.3124
+
+subscene, addic7ed and titlovi
+- either of those providers might impose a reCAPTCHA verification. In order to use those providers, please create an account at an AntiCaptcha service ([anti-captcha.com](http://getcaptchasolution.com/kkvviom7nh) or [deathbycaptcha.com](http://deathbycaptcha.com)), add funds, then supply your credentials/apikey in the configuration
+
+Changelog
+- core: http: fallback to default DNS when normal resolving fails; fixes #657
+- core: extract embedded/menu: fix detection of unknown streams; don't use unknown streams if a known language was previously found
+- core: language: use replacement map from bazarr
+- providers: titlovi: fix matching
+- providers: subscene: fix unknown language code error when "empty" result is returned
+- providers: subscene: add support for pt-BR (based on Diaoul/subliminal@b22cf08)
+- providers: subscene: explicitly set account filters for languages
+- providers: subscene: limit alternative searches to 3; set throttle to 8
+- providers: subscene: move login/cookies to initialization sequence
+- submod: generic: en: fix ";='s
+
+
+2.6.5.3109
+
+subscene, addic7ed and titlovi
+- either of those providers might impose a reCAPTCHA verification. In order to use those providers, please create an account at an AntiCaptcha service ([anti-captcha.com](http://getcaptchasolution.com/kkvviom7nh) or [deathbycaptcha.com](http://deathbycaptcha.com)), add funds, then supply your credentials/apikey in the configuration
+
+Changelog
+- providers: add Napisy24 (polish)
+- providers: subscene: reduce provider load by possibly half
+- providers: subscene: support logging in (username/password are now required)
+- providers: subscene: fallback to non year results if none found with year
+
+
+2.6.5.3099
+
+subscene, addic7ed and titlovi
+- either of those providers might impose a reCAPTCHA verification. In order to use those providers, please create an account at an AntiCaptcha service ([anti-captcha.com](http://getcaptchasolution.com/kkvviom7nh) or [deathbycaptcha.com](http://deathbycaptcha.com)), add funds, then supply your credentials/apikey in the configuration
+
+Changelog
+- core: allow system DNS again by putting "system" as the DNS
+- providers: subscene: fix again (subscene, contact us please, so we can end this)
+
+
+2.6.5.3092
+
+subscene, addic7ed and titlovi
+- either of those providers might impose a reCAPTCHA verification. In order to use those providers, please create an account at an AntiCaptcha service ([anti-captcha.com](http://getcaptchasolution.com/kkvviom7nh) or [deathbycaptcha.com](http://deathbycaptcha.com)), add funds, then supply your credentials/apikey in the configuration
+
+Changelog
+- providers: subscene: fix endpoint (hopefully for longer now)
+- providers: subscene: don't search for season packs (broken for now; relieves 50% of server load on provider)
+- providers: subscene: don't calculate video fn for now
+- providers: argenteam: backport fixes from bazarr
+- subtitle: try decoding with utf-16 by default as well (zho/farsi)
+- submod: HI: remove music tags by default
+- core: compat (bazarr): add env var SZ_KEEP_ENCODING to keep encoding of subtitles
+
+
+2.6.5.3074
+
+Changelog
+- core: cf: bypass cf 95% of the time without captchas
+- core: fix breaking line endings of certain languages (chinese, UTF-16); fixes #646
+- core: update pysubs2 to 0.2.3
+
+
+2.6.5.3062
+
+Changelog
+- core: cf: optimize
+- core: http: don't query DNS with IPs. thanks @fgump (fixes sonarr/radarr)
+
+
+2.6.5.3041
+
+Changelog
+- core: only reference guessed title if there actually is one
+- core: cf: optimize
+- core/config: add setting for one existing language to be enough, fixes #491
+- core/compat: dns: support nameservers via ENV[dns_resolvers]; don't fall back to default DNS when configured custom DNS failed
+- providers: titlovi: prevent repeated captcha solving for CF
+

 2.6.5.3017

@@ -21,20 +21,21 @@ import support
 import interface
 sys.modules["interface"] = interface

-from subzero.constants import OS_PLEX_USERAGENT, PERSONAL_MEDIA_IDENTIFIER
+from subzero.constants import OS_PLEX_USERAGENT
 from interface.menu import *
 from support.plex_media import media_to_videos, get_media_item_ids
+from support.extract import agent_extract_embedded
 from support.scanning import scan_videos
-from support.storage import save_subtitles, store_subtitle_info, get_subtitle_storage
+from support.storage import save_subtitles, store_subtitle_info
 from support.items import is_wanted
 from support.config import config
 from support.lib import get_intent
-from support.helpers import track_usage, get_title_for_video_metadata, get_identifier, cast_bool, \
-    audio_streams_match_languages
+from support.helpers import track_usage, get_title_for_video_metadata, get_identifier, cast_bool
 from support.history import get_history
 from support.data import dispatch_migrate
 from support.activities import activity
 from support.download import download_best_subtitles
+from support.localmedia import find_subtitles


 def Start():
@@ -96,57 +97,7 @@ def Start():

 def update_local_media(videos, ignore_parts_cleanup=None):
    for video in videos:
-        support.localmedia.find_subtitles(video["plex_part"], ignore_parts_cleanup=ignore_parts_cleanup)
-
-
-def agent_extract_embedded(video_part_map):
-    try:
-        subtitle_storage = get_subtitle_storage()
-
-        to_extract = []
-        item_count = 0
-
-        for scanned_video, part_info in video_part_map.iteritems():
-            plexapi_item = scanned_video.plexapi_metadata["item"]
-            stored_subs = subtitle_storage.load_or_new(plexapi_item)
-            valid_langs_in_media = audio_streams_match_languages(scanned_video, config.get_lang_list(ordered=True))
-
-            if not config.lang_list.difference(valid_langs_in_media):
-                Log.Debug("Skipping embedded subtitle extraction for %s, audio streams are in correct language(s)",
-                          plexapi_item.rating_key)
-                continue
-
-            for plexapi_part in get_all_parts(plexapi_item):
-                item_count = item_count + 1
-                used_one_unknown_stream = False
-                for requested_language in config.lang_list:
-                    embedded_subs = stored_subs.get_by_provider(plexapi_part.id, requested_language, "embedded")
-                    current = stored_subs.get_any(plexapi_part.id, requested_language) or \
-                        requested_language in scanned_video.external_subtitle_languages
-
-                    if not embedded_subs:
-                        stream_data = get_embedded_subtitle_streams(plexapi_part, requested_language=requested_language,
-                                                                    skip_unknown=used_one_unknown_stream)
-
-                        if stream_data:
-                            stream = stream_data[0]["stream"]
-                            if stream_data[0]["is_unknown"]:
-                                used_one_unknown_stream = True
-
-                            to_extract.append(({scanned_video: part_info}, plexapi_part, str(stream.index),
-                                               str(requested_language), not current))
-
-                            if not cast_bool(Prefs["subtitles.search_after_autoextract"]):
-                                scanned_video.subtitle_languages.update({requested_language})
-                    else:
-                        Log.Debug("Skipping embedded subtitle extraction for %s, already got %r from %s",
-                                  plexapi_item.rating_key, requested_language, embedded_subs[0].id)
-        if to_extract:
-            Log.Info("Triggering extraction of %d embedded subtitles of %d items", len(to_extract), item_count)
-            Thread.Create(multi_extract_embedded, stream_list=to_extract, refresh=True, with_mods=True,
-                          single_thread=not config.advanced.auto_extract_multithread)
-    except:
-        Log.Error("Something went wrong when auto-extracting subtitles, continuing: %s", traceback.format_exc())
+        find_subtitles(video["plex_part"], ignore_parts_cleanup=ignore_parts_cleanup)


 class SubZeroAgent(object):
@@ -311,7 +262,8 @@ class SubZeroAgent(object):


 class SubZeroSubtitlesAgentMovies(SubZeroAgent, Agent.Movies):
-    contributes_to = ['com.plexapp.agents.imdb', 'com.plexapp.agents.xbmcnfo', 'com.plexapp.agents.themoviedb', 'com.plexapp.agents.hama']
+    contributes_to = ['com.plexapp.agents.imdb', 'com.plexapp.agents.xbmcnfo', 'com.plexapp.agents.themoviedb',
+                      'com.plexapp.agents.hama',  'tv.plex.agents.movie']
    score_prefs_key = "subtitles.search.minimumMovieScore2"
    agent_type_verbose = "Movies"

@@ -7,14 +7,16 @@ from subzero.language import Language

 from sub_mod import SubtitleModificationsMenu
 from menu_helpers import debounce, SubFolderObjectContainer, default_thumb, add_incl_excl_options, get_item_task_data, \
-    set_refresh_menu_state, route, extract_embedded_sub
+    set_refresh_menu_state, route
+from support.extract import extract_embedded_sub

 from refresh_item import RefreshItem
 from subzero.constants import PREFIX
 from support.config import config, TEXT_SUBTITLE_EXTS
-from support.helpers import timestamp, df, get_language, display_language, get_language_from_stream, is_stream_forced
+from support.helpers import timestamp, df, get_language, display_language, get_language_from_stream
 from support.items import get_item_kind_from_rating_key, get_item, get_current_sub, get_item_title, save_stored_sub
-from support.plex_media import get_plex_metadata, get_part, get_embedded_subtitle_streams
+from support.plex_media import get_plex_metadata, get_part, get_embedded_subtitle_streams, is_stream_forced, \
+    update_stream_info
 from support.scanning import scan_videos
 from support.scheduler import scheduler
 from support.storage import get_subtitle_storage
@@ -118,6 +120,8 @@ def ItemDetailsMenu(rating_key, title=None, base_title=None, item_title=None, ra
            if not os.path.exists(part.file):
                continue

+            update_stream_info(part)
+
            part_id = str(part.id)
            part_index += 1

@@ -670,29 +674,28 @@ def ListEmbeddedSubsForItemMenu(**kwargs):
            stream = stream_data["stream"]
            is_forced = stream_data["is_forced"]

-            if language:
-                oc.add(DirectoryObject(
-                    key=Callback(TriggerExtractEmbeddedSubForItemMenu, randomize=timestamp(),
-                                 stream_index=str(stream.index), language=language, with_mods=True, **kwargs),
-                    title=_(u"Extract stream %(stream_index)s, %(language)s%(unknown_state)s%(forced_state)s"
-                            u"%(stream_title)s with default mods",
-                            stream_index=stream.index,
-                            language=display_language(language),
-                            unknown_state=_(" (unknown)") if is_unknown else "",
-                            forced_state=_(" (forced)") if is_forced else "",
-                            stream_title=" (\"%s\")" % stream.title if stream.title else ""),
-                ))
-                oc.add(DirectoryObject(
-                    key=Callback(TriggerExtractEmbeddedSubForItemMenu, randomize=timestamp(),
-                                 stream_index=str(stream.index), language=language, **kwargs),
-                    title=_(u"Extract stream %(stream_index)s, %(language)s%(unknown_state)s%(forced_state)s"
-                            u"%(stream_title)s",
-                            stream_index=stream.index,
-                            language=display_language(language),
-                            unknown_state=_(" (unknown)") if is_unknown else "",
-                            forced_state=_(" (forced)") if is_forced else "",
-                            stream_title=" (\"%s\")" % stream.title if stream.title else ""),
-                ))
+            oc.add(DirectoryObject(
+                key=Callback(TriggerExtractEmbeddedSubForItemMenu, randomize=timestamp(),
+                             stream_index=str(stream.index), language=language, with_mods=True, **kwargs),
+                title=_(u"Extract stream %(stream_index)s, %(language)s%(unknown_state)s%(forced_state)s"
+                        u"%(stream_title)s with default mods",
+                        stream_index=stream.index,
+                        language=display_language(language),
+                        unknown_state=_(" (unknown)") if is_unknown else "",
+                        forced_state=_(" (forced)") if is_forced else "",
+                        stream_title=" (\"%s\")" % stream.title if stream.title else ""),
+            ))
+            oc.add(DirectoryObject(
+                key=Callback(TriggerExtractEmbeddedSubForItemMenu, randomize=timestamp(),
+                             stream_index=str(stream.index), language=language, **kwargs),
+                title=_(u"Extract stream %(stream_index)s, %(language)s%(unknown_state)s%(forced_state)s"
+                        u"%(stream_title)s",
+                        stream_index=stream.index,
+                        language=display_language(language),
+                        unknown_state=_(" (unknown)") if is_unknown else "",
+                        forced_state=_(" (forced)") if is_forced else "",
+                        stream_title=" (\"%s\")" % stream.title if stream.title else ""),
+            ))
    return oc


@@ -12,19 +12,16 @@ from requests import HTTPError
 from item_details import ItemDetailsMenu
 from refresh_item import RefreshItem
 from menu_helpers import add_incl_excl_options, dig_tree, set_refresh_menu_state, \
-    default_thumb, debounce, ObjectContainer, SubFolderObjectContainer, route, \
-    extract_embedded_sub
+    default_thumb, debounce, ObjectContainer, SubFolderObjectContainer, route
 from main import fatality, InclExclMenu
 from advanced import DispatchRestart
 from subzero.constants import ART, PREFIX, DEPENDENCY_MODULE_NAMES
-from support.plex_media import get_all_parts, get_embedded_subtitle_streams
+from support.extract import season_extract_embedded
 from support.scheduler import scheduler
 from support.config import config
 from support.helpers import timestamp, df, display_language
 from support.ignore import get_decision_list
-from support.items import get_all_items, get_items_info, get_item_kind_from_rating_key, get_item, MI_KEY, \
-    get_item_title, get_item_thumb
-from support.storage import get_subtitle_storage
+from support.items import get_all_items, get_items_info, get_item_kind_from_rating_key, get_item, get_item_title
 from support.i18n import _

 # init GUI
@@ -111,28 +108,51 @@ def MetadataMenu(rating_key, title=None, base_title=None, display_items=False, p
            add_incl_excl_options(oc, current_kind, title=sub_title, rating_key=rating_key, callback_menu=InclExclMenu)

        # mass-extract embedded
-        if current_kind == "season" and config.plex_transcoder:
-            for lang in config.lang_list:
-                oc.add(DirectoryObject(
-                    key=Callback(SeasonExtractEmbedded, rating_key=rating_key, language=lang,
-                                 base_title=show.section.title, display_items=display_items, item_title=item_title,
-                                 title=title,
-                                 previous_item_type=previous_item_type, with_mods=True,
-                                 previous_rating_key=previous_rating_key, randomize=timestamp()),
-                    title=_(u"Extract missing %(language)s embedded subtitles", language=display_language(lang)),
-                    summary=_("Extracts the not yet extracted embedded subtitles of all episodes for the current "
-                              "season with all configured default modifications")
-                ))
-                oc.add(DirectoryObject(
-                    key=Callback(SeasonExtractEmbedded, rating_key=rating_key, language=lang,
-                                 base_title=show.section.title, display_items=display_items, item_title=item_title,
-                                 title=title, force=True,
-                                 previous_item_type=previous_item_type, with_mods=True,
-                                 previous_rating_key=previous_rating_key, randomize=timestamp()),
-                    title=_(u"Extract and activate %(language)s embedded subtitles", language=display_language(lang)),
-                    summary=_("Extracts embedded subtitles of all episodes for the current season "
-                              "with all configured default modifications")
-                ))
+        if config.plex_transcoder:
+            if current_kind == "season":
+                for lang in config.lang_list:
+                    oc.add(DirectoryObject(
+                        key=Callback(SeasonExtractEmbedded, rating_key=rating_key, language=lang,
+                                     base_title=show.section.title, display_items=display_items, item_title=item_title,
+                                     title=title,
+                                     previous_item_type=previous_item_type, with_mods=True,
+                                     previous_rating_key=previous_rating_key, randomize=timestamp()),
+                        title=_(u"Extract missing %(language)s embedded subtitles", language=display_language(lang)),
+                        summary=_("Extracts the not yet extracted embedded subtitles of all episodes for the current "
+                                  "season with all configured default modifications")
+                    ))
+                    oc.add(DirectoryObject(
+                        key=Callback(SeasonExtractEmbedded, rating_key=rating_key, language=lang,
+                                     base_title=show.section.title, display_items=display_items, item_title=item_title,
+                                     title=title, force=True,
+                                     previous_item_type=previous_item_type, with_mods=True,
+                                     previous_rating_key=previous_rating_key, randomize=timestamp()),
+                        title=_(u"Extract and activate %(language)s embedded subtitles", language=display_language(lang)),
+                        summary=_("Extracts embedded subtitles of all episodes for the current season "
+                                  "with all configured default modifications")
+                    ))
+            elif current_kind == "series":
+                for lang in config.lang_list:
+                    oc.add(DirectoryObject(
+                        key=Callback(SeasonExtractEmbedded, rating_key=rating_key, language=lang,
+                                     base_title=title, display_items=display_items, item_title=item_title,
+                                     title=title, mode="series",
+                                     previous_item_type=previous_item_type, with_mods=True,
+                                     previous_rating_key=previous_rating_key, randomize=timestamp()),
+                        title=_(u"Extract missing %(language)s embedded subtitles", language=display_language(lang)),
+                        summary=_("Extracts the not yet extracted embedded subtitles of all episodes for the current "
+                                  "series with all configured default modifications")
+                    ))
+                    oc.add(DirectoryObject(
+                        key=Callback(SeasonExtractEmbedded, rating_key=rating_key, language=lang,
+                                     base_title=title, display_items=display_items, item_title=item_title,
+                                     title=title, force=True, mode="series",
+                                     previous_item_type=previous_item_type, with_mods=True,
+                                     previous_rating_key=previous_rating_key, randomize=timestamp()),
+                        title=_(u"Extract and activate %(language)s embedded subtitles", language=display_language(lang)),
+                        summary=_("Extracts embedded subtitles of all episodes for the current series "
+                                  "with all configured default modifications")
+                    ))

        # add refresh
        oc.add(DirectoryObject(
@@ -163,9 +183,10 @@ def SeasonExtractEmbedded(**kwargs):
    item_title = kwargs.pop("item_title")
    title = kwargs.pop("title")
    force = kwargs.pop("force", False)
+    mode = kwargs.pop("mode", "season")

    Thread.Create(season_extract_embedded, **{"rating_key": rating_key, "requested_language": requested_language,
-                                              "with_mods": with_mods, "force": force})
+                                              "with_mods": with_mods, "force": force, "mode": mode})

    kwargs["header"] = _("Success")
    kwargs["message"] = _(u"Extracting of embedded subtitles for %s triggered", title)
@@ -174,53 +195,6 @@ def SeasonExtractEmbedded(**kwargs):
    return MetadataMenu(randomize=timestamp(), title=item_title, **kwargs)


-def multi_extract_embedded(stream_list, refresh=False, with_mods=False, single_thread=True, extract_mode="a",
-                           history_storage=None):
-    def execute():
-        for video_part_map, plexapi_part, stream_index, language, set_current in stream_list:
-            plexapi_item = video_part_map.keys()[0].plexapi_metadata["item"]
-
-            extract_embedded_sub(rating_key=plexapi_item.rating_key, part_id=plexapi_part.id,
-                                 plex_item=plexapi_item, part=plexapi_part, scanned_videos=video_part_map,
-                                 stream_index=stream_index, set_current=set_current,
-                                 language=language, with_mods=with_mods, refresh=refresh, extract_mode=extract_mode,
-                                 history_storage=history_storage)
-
-    if single_thread:
-        with Thread.Lock(key="extract_embedded"):
-            execute()
-    else:
-        execute()
-
-
-def season_extract_embedded(rating_key, requested_language, with_mods=False, force=False):
-    # get stored subtitle info for item id
-    subtitle_storage = get_subtitle_storage()
-
-    try:
-        for data in get_all_items(key="children", value=rating_key, base="library/metadata"):
-            item = get_item(data[MI_KEY])
-            if item:
-                stored_subs = subtitle_storage.load_or_new(item)
-                for part in get_all_parts(item):
-                    embedded_subs = stored_subs.get_by_provider(part.id, requested_language, "embedded")
-                    current = stored_subs.get_any(part.id, requested_language)
-                    if not embedded_subs or force:
-                        stream_data = get_embedded_subtitle_streams(part, requested_language=requested_language)
-                        if stream_data:
-                            stream = stream_data[0]["stream"]
-
-                            set_current = not current or force
-                            refresh = not current
-
-                            extract_embedded_sub(rating_key=item.rating_key, part_id=part.id,
-                                                 stream_index=str(stream.index), set_current=set_current,
-                                                 refresh=refresh, language=requested_language, with_mods=with_mods,
-                                                 extract_mode="m")
-    finally:
-        subtitle_storage.destroy()
-
-
@route(PREFIX + '/ignore_list')
 def IgnoreListMenu():
    ref_list = get_decision_list()
@@ -368,7 +342,7 @@ def ValidatePrefs():
            "subtitle_destination_folder", "include", "include_exclude_paths", "include_exclude_sz_files",
            "new_style_cache", "dbm_supported", "lang_list", "providers", "normal_subs", "forced_only", "forced_also",
            "plex_transcoder", "refiner_settings", "unrar", "adv_cfg_path", "use_custom_dns",
-            "has_anticaptcha", "anticaptcha_cls"]:
+            "has_anticaptcha", "anticaptcha_cls", "mediainfo_bin"]:

        value = getattr(config, attr)
        if isinstance(value, dict):
@@ -1,29 +1,16 @@
 # coding=utf-8
-import traceback
 import types
 import datetime
-import subprocess
-import os
-import operator

-from func import enable_channel_wrapper, route_wrapper, register_route_function
-from subzero.lib.io import get_viable_encoding
-from subzero.language import Language
+from func import enable_channel_wrapper, route_wrapper
 from support.i18n import is_localized_string, _
-from support.items import get_kind, get_item_thumb, get_item, get_item_kind_from_item, refresh_item
-from support.helpers import get_video_display_title, pad_title, display_language, quote_args, is_stream_forced, \
-    get_title_for_video_metadata, mswindows
-from support.history import get_history
+from support.items import get_item_thumb
+from support.helpers import get_video_display_title, pad_title
 from support.ignore import get_decision_list
 from support.lib import get_intent
 from support.config import config
 from subzero.constants import ICON_SUB, ICON
-from support.plex_media import get_part, get_plex_metadata
 from support.scheduler import scheduler
-from support.scanning import scan_videos
-from support.storage import save_subtitles
-
-from subliminal_patch.subtitle import ModifiedSubtitle

 default_thumb = R(ICON_SUB)
 main_icon = ICON if not config.is_development else "icon-dev.jpg"
@@ -156,89 +143,6 @@ def debounce(func):
    return func


-def extract_embedded_sub(**kwargs):
-    rating_key = kwargs["rating_key"]
-    part_id = kwargs.pop("part_id")
-    stream_index = kwargs.pop("stream_index")
-    with_mods = kwargs.pop("with_mods", False)
-    language = Language.fromietf(kwargs.pop("language"))
-    refresh = kwargs.pop("refresh", True)
-    set_current = kwargs.pop("set_current", True)
-
-    plex_item = kwargs.pop("plex_item", get_item(rating_key))
-    item_type = get_item_kind_from_item(plex_item)
-    part = kwargs.pop("part", get_part(plex_item, part_id))
-    scanned_videos = kwargs.pop("scanned_videos", None)
-    extract_mode = kwargs.pop("extract_mode", "a")
-
-    any_successful = False
-
-    if part:
-        if not scanned_videos:
-            metadata = get_plex_metadata(rating_key, part_id, item_type, plex_item=plex_item)
-            scanned_videos = scan_videos([metadata], ignore_all=True, skip_hashing=True)
-
-        for stream in part.streams:
-            # subtitle stream
-            if str(stream.index) == stream_index:
-                is_forced = is_stream_forced(stream)
-                bn = os.path.basename(part.file)
-
-                set_refresh_menu_state(_(u"Extracting subtitle %(stream_index)s of %(filename)s",
-                                         stream_index=stream_index,
-                                         filename=bn))
-                Log.Info(u"Extracting stream %s (%s) of %s", stream_index, str(language), bn)
-
-                out_codec = stream.codec if stream.codec != "mov_text" else "srt"
-
-                args = [
-                    config.plex_transcoder, "-i", part.file, "-map", "0:%s" % stream_index, "-f", out_codec, "-"
-                ]
-
-                cmdline = quote_args(args)
-                Log.Debug(u"Calling: %s", cmdline)
-                if mswindows:
-                    Log.Debug("MSWindows: Fixing encoding")
-                    cmdline = cmdline.encode("mbcs")
-
-                output = None
-                try:
-                    output = subprocess.check_output(cmdline, stderr=subprocess.PIPE, shell=True)
-                except:
-                    Log.Error("Extraction failed: %s", traceback.format_exc())
-
-                if output:
-                    subtitle = ModifiedSubtitle(language, mods=config.default_mods if with_mods else None)
-                    subtitle.content = output
-                    subtitle.provider_name = "embedded"
-                    subtitle.id = "stream_%s" % stream_index
-                    subtitle.score = 0
-                    subtitle.set_encoding("utf-8")
-
-                    # fixme: speedup video; only video.name is needed
-                    video = scanned_videos.keys()[0]
-                    save_successful = save_subtitles(scanned_videos, {video: [subtitle]}, mode="m",
-                                                     set_current=set_current)
-                    set_refresh_menu_state(None)
-
-                    if save_successful and refresh:
-                        refresh_item(rating_key)
-
-                    # add item to history
-                    item_title = get_title_for_video_metadata(video.plexapi_metadata,
-                                                              add_section_title=False, add_episode_title=True)
-
-                    history = get_history()
-                    history.add(item_title, video.id, section_title=video.plexapi_metadata["section"],
-                                thumb=video.plexapi_metadata["super_thumb"],
-                                subtitle=subtitle, mode=extract_mode)
-                    history.destroy()
-
-                    any_successful = True
-
-    return any_successful
-
-
 class SZObjectContainer(ObjectContainer):
    def __init__(self, *args, **kwargs):
        skip_pin_lock = kwargs.pop("skip_pin_lock", False)
@@ -19,6 +19,10 @@ sys.modules["support.i18n"] = i18n

 helpers._ = i18n._

+import history
+
+sys.modules["support.history"] = history
+
 import plex_media
 sys.modules["support.plex_media"] = plex_media

@@ -49,6 +53,10 @@ import missing_subtitles

 sys.modules["support.missing_subtitles"] = missing_subtitles

+import extract
+
+sys.modules["support.extract"] = extract
+
 import tasks

 sys.modules["support.tasks"] = tasks
@@ -57,10 +65,6 @@ import ignore

 sys.modules["support.ignore"] = ignore

-import history
-
-sys.modules["support.history"] = history
-
 import data

 sys.modules["support.data"] = data
@@ -8,12 +8,17 @@ import sys
 import rarfile
 import jstyleson
 import datetime
+import stat
+import traceback
+import socket
+import requests

 import subliminal
 import subliminal_patch
 import subzero.constants
 import lib
-from subliminal.exceptions import ServiceUnavailable, DownloadLimitExceeded, AuthenticationError
+from subliminal.exceptions import ServiceUnavailable, DownloadLimitExceeded, AuthenticationError, \
+    DownloadLimitPerDayExceeded
 from subliminal_patch.core import is_windows_special_path
 from whichdb import whichdb

@@ -22,6 +27,7 @@ from subzero.language import Language
 from subliminal.cli import MutexLock
 from subzero.lib.io import FileIO, get_viable_encoding
 from subzero.lib.dict import Dicked
+from subzero.lib.which import find_executable
 from subzero.util import get_root_path
 from subzero.constants import PLUGIN_NAME, PLUGIN_IDENTIFIER, MOVIE, SHOW, MEDIA_TYPE_TO_STRING
 from subzero.prefs import get_user_prefs, update_user_prefs
@@ -58,23 +64,36 @@ def int_or_default(s, default):
        return default


-VALID_THROTTLE_EXCEPTIONS = (TooManyRequests, DownloadLimitExceeded, ServiceUnavailable, APIThrottled)
+VALID_THROTTLE_EXCEPTIONS = (TooManyRequests, DownloadLimitExceeded, DownloadLimitPerDayExceeded,
+                             ServiceUnavailable, APIThrottled, requests.Timeout, requests.ReadTimeout, socket.timeout)
+
+def_timeout = (datetime.timedelta(minutes=20), "20 minutes")

 PROVIDER_THROTTLE_MAP = {
    "default": {
        TooManyRequests: (datetime.timedelta(hours=1), "1 hour"),
        DownloadLimitExceeded: (datetime.timedelta(hours=3), "3 hours"),
+        DownloadLimitPerDayExceeded: (datetime.timedelta(hours=4), "4 hours"),
        ServiceUnavailable: (datetime.timedelta(minutes=20), "20 minutes"),
        APIThrottled: (datetime.timedelta(minutes=10), "10 minutes"),
+        AuthenticationError: (datetime.timedelta(hours=2), "2 hours"),
+        requests.Timeout: def_timeout,
+        socket.timeout: def_timeout,
+        requests.ReadTimeout: def_timeout,
    },
    "opensubtitles": {
        TooManyRequests: (datetime.timedelta(hours=3), "3 hours"),
        DownloadLimitExceeded: (datetime.timedelta(hours=6), "6 hours"),
        APIThrottled: (datetime.timedelta(seconds=15), "15 seconds"),
    },
+    "opensubtitlescom": {
+        TooManyRequests: (datetime.timedelta(minutes=1), "1 minute"),
+        DownloadLimitExceeded: (datetime.timedelta(hours=24), "24 hours"),
+    },
    "addic7ed": {
        DownloadLimitExceeded: (datetime.timedelta(hours=3), "3 hours"),
        TooManyRequests: (datetime.timedelta(minutes=5), "5 minutes"),
+        AuthenticationError: (datetime.timedelta(hours=24), "24 hours"),
    }
 }

@@ -145,6 +164,8 @@ class Config(object):
    exact_filenames = False
    only_one = False
    any_language_is_enough = False
+    ignore_for_audio = False
+    ignore_subs_for_empty_audio = False
    embedded_auto_extract = False
    ietf_as_alpha3 = False
    unrar = None
@@ -153,6 +174,7 @@ class Config(object):
    anticaptcha_token = None
    anticaptcha_cls = None
    has_anticaptcha = False
+    mediainfo_bin = None

    store_recently_played_amount = 40

@@ -236,9 +258,12 @@ class Config(object):
        self.plex_transcoder = self.get_plex_transcoder()
        self.only_one = cast_bool(Prefs['subtitles.only_one'])
        self.any_language_is_enough = Prefs['subtitles.any_language_is_enough']
+        self.ignore_for_audio = self.ignore_subs_for_audio()
        self.embedded_auto_extract = cast_bool(Prefs["subtitles.embedded.autoextract"])
        self.ietf_as_alpha3 = cast_bool(Prefs["subtitles.language.ietf_normalize"])
        self.use_custom_dns = self.parse_custom_dns()
+        if not self.advanced.dont_use_mediainfo_mp4:
+            self.mediainfo_bin = self.advanced.mediainfo_bin or find_executable("mediainfo")
        self.initialized = True

    def migrate_prefs(self):
@@ -323,6 +348,19 @@ class Config(object):
        for exe in try_executables:
            rarfile.UNRAR_TOOL = exe
            rarfile.ORIG_UNRAR_TOOL = exe
+            if os.path.isfile(exe) and not os.access(exe, os.X_OK):
+                st = os.stat(exe)
+                try:
+                    Log.Debug("setting generic executable permissions for %s", exe)
+                    # fixme: too broad?
+                    os.chmod(exe, st.st_mode | stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH)
+                except:
+                    Log.Debug("failed setting generic executable permissions for %s: %s", exe, traceback.format_exc())
+                    try:
+                        Log.Debug("setting executable permissions for %s", exe)
+                        os.chmod(exe, st.st_mode | stat.S_IEXEC)
+                    except:
+                        Log.Debug("failed setting executable permissions for %s: %s", exe, traceback.format_exc())
            try:
                rarfile.custom_check([rarfile.UNRAR_TOOL], True)
            except:
@@ -637,17 +675,34 @@ class Config(object):
        enabled_for_primary_agents = {"movie": [], "show": []}
        enabled_sections = {}

+        legacy_agents = {
+            "com.plexapp.agents.thetvdb": [SHOW],
+            "com.plexapp.agents.thetvdbdvdorder": [SHOW],
+            "com.plexapp.agents.hama": [SHOW, MOVIE],
+            "com.plexapp.agents.themoviedb": [SHOW, MOVIE],
+            "com.plexapp.agents.imdb": [SHOW, MOVIE],
+        }
+
        # find which agents we're enabled for
        for agent in Plex.agents():
-            if not agent.primary:
+            #if not agent.primary:
+            #    continue
+            if agent.identifier not in legacy_agents:
                continue

-            for t in list(agent.media_types):
-                if t.media_type in (MOVIE, SHOW):
-                    related_agents = Plex.primary_agent(agent.identifier, t.media_type)
+            #media_types = [t.media_type for t in list(agent.media_types)]
+            media_types = legacy_agents[agent.identifier] + []
+
+            # the new movie agent doesn't populate its media types, workaround
+            if not media_types and agent.identifier == "tv.plex.agents.movie":
+                media_types = [MOVIE]
+
+            for media_type in media_types:
+                if media_type in (MOVIE, SHOW):
+                    related_agents = Plex.primary_agent(agent.identifier, media_type)
                    for a in related_agents:
                        if a.identifier == PLUGIN_IDENTIFIER and a.enabled:
-                            enabled_for_primary_agents[MEDIA_TYPE_TO_STRING[t.media_type]].append(agent.identifier)
+                            enabled_for_primary_agents[MEDIA_TYPE_TO_STRING[media_type]].append(agent.identifier)

        # find the libraries that use them
        for library in self.sections:
@@ -657,6 +712,22 @@ class Config(object):
        Log.Debug(u"I'm enabled for: %s" % [lib.title for key, lib in enabled_sections.iteritems()])
        return enabled_sections

+    def lang_str_to_list(self, s, l):
+        if len(s) and s != "None":
+            for lang in s.split(u","):
+                lang = lang.strip()
+                if lang == "NULL":
+                    l.append(lang)
+                    continue
+                try:
+                    real_lang = Language.fromietf(lang)
+                except:
+                    try:
+                        real_lang = Language.fromname(lang)
+                    except:
+                        continue
+                l.append(real_lang)
+
    # Prepare a list of languages we want subs for
    def get_lang_list(self, provider=None, ordered=False):
        # advanced settings
@@ -697,17 +768,9 @@ class Config(object):
            except:
                pass

-        if len(lang_custom) and lang_custom != "None":
-            for lang in lang_custom.split(u","):
-                lang = lang.strip()
-                try:
-                    real_lang = Language.fromietf(lang)
-                except:
-                    try:
-                        real_lang = Language.fromname(lang)
-                    except:
-                        continue
-                l.append(real_lang)
+        self.lang_str_to_list(lang_custom, l)
+        if "NULL" in l:
+            l.remove("NULL")

        if self.forced_also:
            if Prefs["subtitles.when_forced"] == "Always":
@@ -738,12 +801,22 @@ class Config(object):

    lang_list = property(get_lang_list)

+    def ignore_subs_for_audio(self):
+        c = Prefs['subtitles.ignore_for_audio'].strip()
+        l = []
+
+        self.lang_str_to_list(c, l)
+        if "NULL" in l:
+            l.remove("NULL")
+            self.ignore_subs_for_empty_audio = True
+        return l
+
    def get_subtitle_destination_folder(self):
        if not Prefs["subtitles.save.filesystem"]:
            return

-        fld_custom = Prefs["subtitles.save.subFolder.Custom"].strip() if cast_bool(
-            Prefs["subtitles.save.subFolder.Custom"]) else None
+        fld_custom = Prefs["subtitles.save.subFolder.Custom"].strip() if \
+            Prefs["subtitles.save.subFolder.Custom"] else None
        return fld_custom or (
            Prefs["subtitles.save.subFolder"] if Prefs["subtitles.save.subFolder"] != "current folder" else None)

@@ -758,9 +831,10 @@ class Config(object):

    @property
    def providers_by_prefs(self):
-        return {'opensubtitles': cast_bool(Prefs['provider.opensubtitles.enabled']),
+        return {'opensubtitlescom': cast_bool(Prefs['provider.opensubtitles.enabled']),
                     # 'thesubdb': Prefs['provider.thesubdb.enabled'],
                     'podnapisi': cast_bool(Prefs['provider.podnapisi.enabled']),
+                     'napisy24': cast_bool(Prefs['provider.napisy24.enabled']),
                     'titlovi': cast_bool(Prefs['provider.titlovi.enabled']),
                     'addic7ed': cast_bool(Prefs['provider.addic7ed.enabled']) and self.has_anticaptcha,
                     'tvsubtitles': cast_bool(Prefs['provider.tvsubtitles.enabled']),
@@ -773,7 +847,10 @@ class Config(object):
                     'argenteam': cast_bool(Prefs['provider.argenteam.enabled']),
                     'subscenter': False,
                     'assrt': cast_bool(Prefs['provider.assrt.enabled']),
-                     }
+                     'bsplayer': cast_bool(Prefs['provider.bsplayer.enabled']),
+                     'ktuvit': cast_bool(Prefs['provider.ktuvit.enabled']),
+                     'wizdom': cast_bool(Prefs['provider.wizdom.enabled']),
+                }

    @property
    def providers_enabled(self):
@@ -801,6 +878,10 @@ class Config(object):
            providers["argenteam"] = False
            providers["assrt"] = False
            providers["subscene"] = False
+            providers["napisy24"] = False
+            providers["bsplayer"] = False
+            providers["ktuvit"] = False
+            providers["wizdom"] = False
            providers_forced_off = dict(providers)

        if not self.unrar and providers["legendastv"]:
@@ -841,36 +922,51 @@ class Config(object):
    providers = property(get_providers)

    def get_provider_settings(self):
-        os_use_https = self.advanced.providers.opensubtitles.use_https \
-            if self.advanced.providers.opensubtitles.use_https is not None else True
-
-        os_skip_wrong_fps = self.advanced.providers.opensubtitles.skip_wrong_fps \
-            if self.advanced.providers.opensubtitles.skip_wrong_fps is not None else True
+        os_use_https = self.advanced.providers.opensubtitles.get("use_https", True)
+        os_skip_wrong_fps = self.advanced.providers.opensubtitles.get("skip_wrong_fps", True)

        provider_settings = {'addic7ed': {'username': Prefs['provider.addic7ed.username'],
                                          'password': Prefs['provider.addic7ed.password'],
+                                          'is_vip': cast_bool(Prefs['provider.addic7ed.is_vip']),
                                          },
-                             'opensubtitles': {'username': Prefs['provider.opensubtitles.username'],
+                             'opensubtitlescom': {'username': Prefs['provider.opensubtitles.username'],
                                               'password': Prefs['provider.opensubtitles.password'],
-                                               'use_tag_search': self.exact_filenames,
-                                               'only_foreign': self.forced_only,
-                                               'also_foreign': self.forced_also,
-                                               'is_vip': cast_bool(Prefs['provider.opensubtitles.is_vip']),
-                                               'use_ssl': os_use_https,
-                                               'timeout': self.advanced.providers.opensubtitles.timeout or 15,
-                                               'skip_wrong_fps': os_skip_wrong_fps,
+                                               #'use_tag_search': self.exact_filenames,
+                                               #'only_foreign': self.forced_only,
+                                               #'also_foreign': self.forced_also,
+                                               #'is_vip': cast_bool(Prefs['provider.opensubtitles.is_vip']),
+                                               #'use_ssl': os_use_https,
+                                               #'timeout': self.advanced.providers.opensubtitles.timeout or 15,
+                                               #'skip_wrong_fps': os_skip_wrong_fps,
+                                               'use_hash': cast_bool(Prefs['provider.opensubtitles.use_hash']),
+                                               'include_ai_translated': True,
+                                               'api_key': Prefs['provider.opensubtitles.api_key'],
                                               },
                             'podnapisi': {
                                 'only_foreign': self.forced_only,
                                 'also_foreign': self.forced_also,
                             },
+                             'titlovi': {
+                                 'username': Prefs['provider.titlovi.username'],
+                                 'password': Prefs['provider.titlovi.password'],
+                             },
+                             'napisy24': {
+                                 'username': Prefs['provider.napisy24.username'],
+                                 'password': Prefs['provider.napisy24.password'],
+                             },
                             'subscene': {
                                 'only_foreign': self.forced_only,
+                                 'username': Prefs['provider.subscene.username'],
+                                 'password': Prefs['provider.subscene.password'],
                             },
                             'legendastv': {'username': Prefs['provider.legendastv.username'],
                                            'password': Prefs['provider.legendastv.password'],
                                            },
-                             'assrt': {'token': Prefs['provider.assrt.token'], }
+                             'assrt': {'token': Prefs['provider.assrt.token'], },
+                             'ktuvit': {
+                                 'username': Prefs['provider.ktuvit.username'],
+                                 'password': Prefs['provider.ktuvit.password'],
+                             },
                             }

        return provider_settings
@@ -894,10 +990,10 @@ class Config(object):
        throttle_data = PROVIDER_THROTTLE_MAP.get(name, PROVIDER_THROTTLE_MAP["default"]).get(cls, None) or \
            PROVIDER_THROTTLE_MAP["default"].get(cls, None)

-        if not throttle_data:
-            return
-
-        throttle_delta, throttle_description = throttle_data
+        if throttle_data:
+            throttle_delta, throttle_description = throttle_data
+        else:
+            throttle_delta, throttle_description = datetime.timedelta(minutes=10), "10 minutes"

        if "provider_throttle" not in Dict:
            Dict["provider_throttle"] = {}
@@ -1084,11 +1180,12 @@ class Config(object):
    def parse_custom_dns(self):
        custom_dns = Prefs['use_custom_dns2'].strip()
        os.environ["dns_resolvers"] = ""
-        if custom_dns:
+
+        if custom_dns and custom_dns != "system":
            ips = filter(lambda x: x, [d.strip() for d in custom_dns.split(",")])
            if ips:
                os.environ["dns_resolvers"] = json.dumps(ips)
-                return os.environ["dns_resolvers"]
+            return os.environ["dns_resolvers"]

    def init_subliminal_patches(self):
        # configure custom subtitle destination folders for scanning pre-existing subs
@@ -16,6 +16,7 @@ from support.storage import get_pack_data, store_pack_data
 def get_missing_languages(video, part):
    languages_list = config.get_lang_list(ordered=True)
    languages = set(languages_list)
+    ignore_langs = set(config.ignore_for_audio)
    valid_langs_in_media = set()

    if Prefs["subtitles.when"] != "Always":
@@ -29,18 +30,27 @@ def get_missing_languages(video, part):
                  video)
        return set()

+    if config.ignore_subs_for_empty_audio and not video.audio_languages:
+        Log.Debug("Skipping subtitle search for %s, ignoring as no audio stream exists", video)
+        return set()
+
+    inter = ignore_langs.intersection(video.audio_languages)
+    if ignore_langs and inter:
+        Log.Debug("Skipping subtitle search for %s, ignoring due to existing audio streams: %s", video, inter)
+        return set()
+
    # should we treat IETF as alpha3? (ditch the country part)
    alpha3_map = {}
    if config.ietf_as_alpha3:
        for language in languages:
-            if language.country:
+            if language and language.country:
                alpha3_map[language.alpha3] = language.country
                language.country = None

    have_languages = video.subtitle_languages.copy()
    if config.ietf_as_alpha3:
        for language in have_languages:
-            if language.country:
+            if language and language.country:
                alpha3_map[language.alpha3] = language.country
                language.country = None

@@ -53,14 +63,14 @@ def get_missing_languages(video, part):
                filter(lambda l: not l.forced, video.subtitle_languages)
            if langs:
                Log.Debug("We have at least one subtitle for any configured language.")
-                return False
+                return set()

        elif "External subtitle" in config.any_language_is_enough:
-            langs = video.subtitle_languages if not not_in_forced else \
+            langs = video.external_subtitle_languages if not not_in_forced else \
                filter(lambda l: not l.forced, video.external_subtitle_languages)
            if langs:
                Log.Debug("We have at least one external subtitle for any configured language.")
-                return False
+                return set()

    # all languages are found if we either really have subs for all languages or we only want to have exactly one language
    # and we've only found one (the case for a selected language, Prefs['subtitles.only_one'] (one found sub matches any language))
@@ -70,7 +80,7 @@ def get_missing_languages(video, part):
            Log.Debug('Only one language was requested, and we\'ve got a subtitle for %s', video)
        else:
            Log.Debug('All languages %r exist for %s', languages, video)
-        return False
+        return set()

    # re-add country codes to the missing languages, in case we've removed them above
    if config.ietf_as_alpha3:
@@ -106,21 +116,22 @@ def language_hook(provider):
 def download_best_subtitles(video_part_map, min_score=0, throttle_time=None, providers=None):
    hearing_impaired = Prefs['subtitles.search.hearingImpaired']
    languages = set([Language.rebuild(l) for l in config.lang_list])
-    missing_languages = []
    if not languages:
        return

    use_videos = []
+    missing_languages = set()
    for video, part in video_part_map.iteritems():
        if not video.ignore_all:
-            missing_languages = get_missing_languages(video, part)
+            p_missing_languages = get_missing_languages(video, part)
        else:
-            missing_languages = languages
+            p_missing_languages = languages

-        if missing_languages:
-            Log.Info(u"%s has missing languages: %s", os.path.basename(video.name), missing_languages)
+        if p_missing_languages:
+            Log.Info(u"%s has missing languages: %s", os.path.basename(video.name), p_missing_languages)
            refine_video(video, refiner_settings=config.refiner_settings)
            use_videos.append(video)
+            missing_languages.update(p_missing_languages)

    # prepare blacklist
    blacklist = get_blacklist_from_part_map(video_part_map, languages)
@@ -0,0 +1,214 @@
+# coding=utf-8
+import os
+import subprocess
+import traceback
+
+from support.helpers import quote_args, mswindows, get_title_for_video_metadata, cast_bool, \
+    audio_streams_match_languages
+from support.i18n import _
+from support.items import get_item_kind_from_item, refresh_item, get_all_items, get_item, MI_KEY
+from support.storage import get_subtitle_storage, save_subtitles
+from support.config import config
+from support.history import get_history
+from support.plex_media import get_all_parts, get_embedded_subtitle_streams, get_part, get_plex_metadata, \
+    update_stream_info, is_stream_forced
+from support.scanning import scan_videos
+from subzero.language import Language
+from subliminal_patch.subtitle import ModifiedSubtitle
+
+
+def agent_extract_embedded(video_part_map, set_as_existing=False):
+    try:
+        subtitle_storage = get_subtitle_storage()
+
+        to_extract = []
+        item_count = 0
+
+        threads = []
+
+        for scanned_video, part_info in video_part_map.iteritems():
+            plexapi_item = scanned_video.plexapi_metadata["item"]
+            stored_subs = subtitle_storage.load_or_new(plexapi_item)
+            valid_langs_in_media = \
+                audio_streams_match_languages(scanned_video, config.get_lang_list(ordered=True))
+
+            if not config.lang_list.difference(valid_langs_in_media):
+                Log.Debug("Skipping embedded subtitle extraction for %s, audio streams are in correct language(s)",
+                          plexapi_item.rating_key)
+                continue
+
+            for plexapi_part in get_all_parts(plexapi_item):
+                item_count = item_count + 1
+                used_one_unknown_stream = False
+                used_one_known_stream = False
+                for requested_language in config.lang_list:
+                    skip_unknown = used_one_unknown_stream or used_one_known_stream
+                    embedded_subs = stored_subs.get_by_provider(plexapi_part.id, requested_language, "embedded")
+                    current = stored_subs.get_any(plexapi_part.id, requested_language) or \
+                        requested_language in scanned_video.external_subtitle_languages
+
+                    if not embedded_subs:
+                        stream_data = get_embedded_subtitle_streams(plexapi_part, requested_language=requested_language,
+                                                                    skip_unknown=skip_unknown)
+
+                        if stream_data and stream_data[0]["language"]:
+                            stream = stream_data[0]["stream"]
+                            if stream_data[0]["is_unknown"]:
+                                used_one_unknown_stream = True
+                            else:
+                                used_one_known_stream = True
+
+                            to_extract.append(({scanned_video: part_info}, plexapi_part, str(stream.index),
+                                               str(requested_language), not current))
+
+                            if not cast_bool(Prefs["subtitles.search_after_autoextract"]) or set_as_existing:
+                                scanned_video.subtitle_languages.update({requested_language})
+                    else:
+                        Log.Debug("Skipping embedded subtitle extraction for %s, already got %r from %s",
+                                  plexapi_item.rating_key, requested_language, embedded_subs[0].id)
+        if to_extract:
+            Log.Info("Triggering extraction of %d embedded subtitles of %d items", len(to_extract), item_count)
+            threads.append(Thread.Create(multi_extract_embedded, stream_list=to_extract, refresh=True, with_mods=True,
+                                         single_thread=not config.advanced.auto_extract_multithread))
+            return threads
+    except:
+        Log.Error("Something went wrong when auto-extracting subtitles, continuing: %s", traceback.format_exc())
+
+
+def multi_extract_embedded(stream_list, refresh=False, with_mods=False, single_thread=True, extract_mode="a",
+                           history_storage=None):
+    def execute():
+        for video_part_map, plexapi_part, stream_index, language, set_current in stream_list:
+            plexapi_item = video_part_map.keys()[0].plexapi_metadata["item"]
+
+            extract_embedded_sub(rating_key=plexapi_item.rating_key, part_id=plexapi_part.id,
+                                 plex_item=plexapi_item, part=plexapi_part, scanned_videos=video_part_map,
+                                 stream_index=stream_index, set_current=set_current,
+                                 language=language, with_mods=with_mods, refresh=refresh, extract_mode=extract_mode,
+                                 history_storage=history_storage)
+
+    if single_thread:
+        with Thread.Lock(key="extract_embedded"):
+            execute()
+    else:
+        execute()
+
+
+def season_extract_embedded(rating_key, requested_language, with_mods=False, force=False, mode="season"):
+    # get stored subtitle info for item id
+    subtitle_storage = get_subtitle_storage()
+
+    try:
+        rating_keys = [rating_key]
+        if mode == "series":
+            seasons = get_all_items(key="children", value=rating_key, base="library/metadata")
+            rating_keys = [season[MI_KEY] for season in seasons]
+
+        for rating_key in rating_keys:
+            for data in get_all_items(key="children", value=rating_key, base="library/metadata"):
+                item = get_item(data[MI_KEY])
+                if item:
+                    stored_subs = subtitle_storage.load_or_new(item)
+                    for part in get_all_parts(item):
+                        embedded_subs = stored_subs.get_by_provider(part.id, requested_language, "embedded")
+                        current = stored_subs.get_any(part.id, requested_language)
+                        if not embedded_subs or force:
+                            stream_data = get_embedded_subtitle_streams(part, requested_language=requested_language)
+                            if stream_data:
+                                stream = stream_data[0]["stream"]
+
+                                set_current = not current or force
+                                refresh = not current
+
+                                extract_embedded_sub(rating_key=item.rating_key, part_id=part.id,
+                                                     stream_index=str(stream.index), set_current=set_current,
+                                                     refresh=refresh, language=requested_language, with_mods=with_mods,
+                                                     extract_mode="m")
+    finally:
+        subtitle_storage.destroy()
+
+
+def extract_embedded_sub(**kwargs):
+    rating_key = kwargs["rating_key"]
+    part_id = kwargs.pop("part_id")
+    stream_index = kwargs.pop("stream_index")
+    with_mods = kwargs.pop("with_mods", False)
+    language = Language.fromietf(kwargs.pop("language"))
+    refresh = kwargs.pop("refresh", True)
+    set_current = kwargs.pop("set_current", True)
+
+    plex_item = kwargs.pop("plex_item", get_item(rating_key))
+    item_type = get_item_kind_from_item(plex_item)
+    part = kwargs.pop("part", get_part(plex_item, part_id))
+    scanned_videos = kwargs.pop("scanned_videos", None)
+    extract_mode = kwargs.pop("extract_mode", "a")
+
+    any_successful = False
+
+    from interface.menu_helpers import set_refresh_menu_state
+
+    if part:
+        if not scanned_videos:
+            metadata = get_plex_metadata(rating_key, part_id, item_type, plex_item=plex_item)
+            scanned_videos = scan_videos([metadata], ignore_all=True, skip_hashing=True)
+
+        update_stream_info(part)
+        for stream in part.streams:
+            # subtitle stream
+            if str(stream.index) == stream_index:
+                is_forced = is_stream_forced(stream)
+                bn = os.path.basename(part.file)
+
+                set_refresh_menu_state(_(u"Extracting subtitle %(stream_index)s of %(filename)s",
+                                         stream_index=stream_index,
+                                         filename=bn))
+                Log.Info(u"Extracting stream %s (%s) of %s", stream_index, str(language), bn)
+
+                out_codec = stream.codec if stream.codec != "mov_text" else "srt"
+
+                args = [
+                    config.plex_transcoder, "-i", part.file, "-map", "0:%s" % stream_index, "-f", out_codec, "-"
+                ]
+
+                cmdline = quote_args(args)
+                Log.Debug(u"Calling: %s", cmdline)
+                if mswindows:
+                    Log.Debug("MSWindows: Fixing encoding")
+                    cmdline = cmdline.encode("mbcs")
+
+                output = None
+                try:
+                    output = subprocess.check_output(cmdline, stderr=subprocess.PIPE, shell=True)
+                except:
+                    Log.Error("Extraction failed: %s", traceback.format_exc())
+
+                if output:
+                    subtitle = ModifiedSubtitle(language, mods=config.default_mods if with_mods else None)
+                    subtitle.content = output
+                    subtitle.provider_name = "embedded"
+                    subtitle.id = "stream_%s" % stream_index
+                    subtitle.score = 0
+                    subtitle.set_encoding("utf-8")
+
+                    # fixme: speedup video; only video.name is needed
+                    video = scanned_videos.keys()[0]
+                    save_successful = save_subtitles(scanned_videos, {video: [subtitle]}, mode="m",
+                                                     set_current=set_current)
+                    set_refresh_menu_state(None)
+
+                    if save_successful and refresh:
+                        refresh_item(rating_key)
+
+                    # add item to history
+                    item_title = get_title_for_video_metadata(video.plexapi_metadata,
+                                                              add_section_title=False, add_episode_title=True)
+
+                    history = get_history()
+                    history.add(item_title, video.id, section_title=video.plexapi_metadata["section"],
+                                thumb=video.plexapi_metadata["super_thumb"],
+                                subtitle=subtitle, mode=extract_mode)
+                    history.destroy()
+
+                    any_successful = True
+
+    return any_successful
@@ -394,7 +394,7 @@ def get_language_from_stream(lang_code):
            return Language.fromietf(lang)
        elif lang:
            try:
-                return language_from_stream(lang)
+                return language_from_stream(lang_code)
            except LanguageError:
                pass

@@ -437,17 +437,10 @@ def get_language(lang_short):


 def display_language(l):
+    if not l:
+        return "Unknown"
    return _(str(l.basename).lower()) + ((u" (%s)" % _("forced")) if l.forced else "")


-def is_stream_forced(stream):
-    stream_title = getattr(stream, "title", "") or ""
-    forced = getattr(stream, "forced", False)
-    if not forced and stream_title and "forced" in stream_title.strip().lower():
-        forced = True
-
-    return forced
-
-
 class PartUnknownException(Exception):
    pass
@@ -349,11 +349,13 @@ def refresh_item(rating_key, force=False, timeout=8000, refresh_kind=None, paren
        refresh = [item.rating_key for item in list(Plex["library/metadata"].children(int(rating_key)))]

    multiple = len(refresh) > 1
+    wait = config.advanced.get("refresh_after_called", 5)
+    Thread.Sleep(wait)
    for key in refresh:
        Log.Info("%s item %s", "Refreshing" if not force else "Forced-refreshing", key)
        Plex["library/metadata"].refresh(key)
        if multiple:
-            Thread.Sleep(10.0)
+            Thread.Sleep(wait)


 def get_current_sub(rating_key, part_id, language, plex_item=None):
@@ -7,6 +7,7 @@ import helpers
 import subtitlehelpers

 from config import config as sz_config
+from subzero.language import ENDSWITH_LANGUAGECODE_RE


 SECONDARY_TAGS = ['forced', 'normal', 'default', 'embedded', 'embedded-forced', 'custom', 'hi', 'cc', 'sdh']
@@ -125,7 +126,7 @@ def find_subtitles(part, ignore_parts_cleanup=None):
                            root = split_tag[0]

                        # get associated media file name without language
-                        sub_fn = subtitlehelpers.ENDSWITH_LANGUAGECODE_RE.sub("", root)
+                        sub_fn = ENDSWITH_LANGUAGECODE_RE.sub("", root)

                        # subtitle basename and basename without possible language tag  not found in collected
                        # media files? kill.
@@ -7,7 +7,8 @@ import os
 from babelfish import LanguageReverseError

 from support.config import config, TEXT_SUBTITLE_EXTS
-from support.helpers import get_plex_item_display_title, cast_bool, get_language_from_stream, is_stream_forced
+from support.helpers import get_plex_item_display_title, cast_bool, get_language_from_stream
+from support.plex_media import is_stream_forced, update_stream_info
 from support.items import get_item
 from support.lib import Plex
 from support.storage import get_subtitle_storage
@@ -35,7 +36,7 @@ def item_discover_missing_subs(rating_key, kind="show", added_at=None, section_t
    for media in item.media:
        existing_subs = {"internal": [], "external": [], "own_external": [], "count": 0}
        for part in media.parts:
-
+            update_stream_info(part)
            # did we already download an external subtitle before?
            if subtitle_target_dir and stored_subs:
                for language in languages_set:
@@ -1,6 +1,7 @@
 # coding=utf-8

 import os
+import subprocess

 import helpers
 from items import get_item
@@ -26,6 +27,9 @@ tvdb_guid_identifier = "com.plexapp.agents.thetvdb://"


 def get_plexapi_stream_info(plex_item, part_id=None):
+    if not plex_item:
+        return
+
    d = {"stream": {}}
    data = d["stream"]

@@ -100,6 +104,9 @@ def media_to_videos(media, kind="series"):
                plex_episode = get_item(ep.id)
                stream_info = get_plexapi_stream_info(plex_episode)

+                if not stream_info:
+                    continue
+
                for item in media.seasons[season].episodes[episode].items:
                    for part in item.parts:
                        videos.append(
@@ -121,22 +128,24 @@ def media_to_videos(media, kind="series"):
                        )
    else:
        stream_info = get_plexapi_stream_info(plex_item)
-        imdb_id = None
-        if imdb_guid_identifier in media.guid:
-            imdb_id = media.guid[len(imdb_guid_identifier):].split("?")[0]
-        for item in media.items:
-            for part in item.parts:
-                videos.append(
-                    get_metadata_dict(plex_item, part, dict(stream_info, **{"plex_part": part, "type": "movie",
-                                                                             "title": media.title, "id": media.id,
-                                                                             "super_thumb": plex_item.thumb,
-                                                                             "series_id": None, "year": year,
-                                                                             "season_id": None, "imdb_id": imdb_id,
-                                                                             "original_title": original_title,
-                                                                             "series_tvdb_id": None, "tvdb_id": None,
-                                                                             "section": plex_item.section.title})
-                                      )
-                )
+
+        if stream_info:
+            imdb_id = None
+            if imdb_guid_identifier in media.guid:
+                imdb_id = media.guid[len(imdb_guid_identifier):].split("?")[0]
+            for item in media.items:
+                for part in item.parts:
+                    videos.append(
+                        get_metadata_dict(plex_item, part, dict(stream_info, **{"plex_part": part, "type": "movie",
+                                                                                 "title": media.title, "id": media.id,
+                                                                                 "super_thumb": plex_item.thumb,
+                                                                                 "series_id": None, "year": year,
+                                                                                 "season_id": None, "imdb_id": imdb_id,
+                                                                                 "original_title": original_title,
+                                                                                 "series_tvdb_id": None, "tvdb_id": None,
+                                                                                 "section": plex_item.section.title})
+                                          )
+                    )
    return videos


@@ -174,42 +183,99 @@ def get_all_parts(plex_item):
    return parts


+def update_stream_info(part):
+    try:
+        return update_stream_info_(part)
+    except:
+        Log.Exception("Getting Mediainfo failed for: %s", part.file)
+
+
+def update_stream_info_(part):
+    if config.mediainfo_bin and part.container == "mp4":
+        cmdline = '%s --Inform="Text;-%%ID%%_%%Title%%" %s' % (config.mediainfo_bin, helpers.quote(part.file))
+        result = subprocess.check_output(cmdline, stderr=subprocess.PIPE, shell=True)
+        if result:
+            try:
+                stream_titles = {}
+                for pair in result[1:].split("-"):
+                    sid, title = pair.split("_")
+                    stream_titles[int(sid.strip())] = title.strip()
+            except:
+                pass
+            else:
+                filled = []
+                for stream in part.streams:
+                    if stream.index is None:
+                        Log.Debug("Found stream with no index: %r", stream)
+
+                    index = stream.index+1 if stream.index is not None else 1
+                    if index in stream_titles:
+                        stream.title = stream_titles[index]
+                        filled.append(index-1)
+                if filled:
+                    Log.Debug("Filled missing MP4 stream title info for streams: %s", filled)
+
+
+def is_stream_forced(stream):
+    stream_title = getattr(stream, "title", "") or ""
+    forced = getattr(stream, "forced", False)
+    if not forced and stream_title and "forced" in stream_title.strip().lower():
+        forced = True
+
+    return forced
+
+
 def get_embedded_subtitle_streams(part, requested_language=None, skip_duplicate_unknown=True, skip_unknown=False):
    streams = []
    streams_unknown = []
+    all_streams = []
    has_unknown = False
    found_requested_language = False
+    update_stream_info(part)
    for stream in part.streams:
        # subtitle stream
        if stream.stream_type == 3 and not stream.stream_key and stream.codec in TEXT_SUBTITLE_EXTS:
-            is_forced = helpers.is_stream_forced(stream)
+            is_forced = is_stream_forced(stream)
            language = helpers.get_language_from_stream(stream.language_code)
            if language:
                language = Language.rebuild(language, forced=is_forced)

            is_unknown = False
            found_requested_language = requested_language and requested_language == language
+            stream_data = None

-            if not language and config.treat_und_as_first:
+            if not language:
                # only consider first unknown subtitle stream
-                if has_unknown and skip_duplicate_unknown:
-                    continue
+                if config.treat_und_as_first:
+                    if has_unknown and skip_duplicate_unknown:
+                        Log.Debug("skipping duplicate unknown")
+                        continue

-                language = Language.rebuild(list(config.lang_list)[0], forced=is_forced)
+                    language = Language.rebuild(list(config.lang_list)[0], forced=is_forced)
+                else:
+                    language = None
                is_unknown = True
                has_unknown = True
-                streams_unknown.append({"stream": stream, "is_unknown": is_unknown, "language": language,
-                                        "is_forced": is_forced})
+                stream_data = {"stream": stream, "is_unknown": is_unknown, "language": language,
+                               "is_forced": is_forced}
+                streams_unknown.append(stream_data)

            if not requested_language or found_requested_language:
-                streams.append({"stream": stream, "is_unknown": is_unknown, "language": language,
-                                "is_forced": is_forced})
+                stream_data = {"stream": stream, "is_unknown": is_unknown, "language": language,
+                               "is_forced": is_forced}
+                streams.append(stream_data)

                if found_requested_language:
                    break

-    if streams_unknown and not found_requested_language and not skip_unknown:
-        streams = streams_unknown
+            if stream_data:
+                all_streams.append(stream_data)
+
+    if requested_language:
+        if streams_unknown and not found_requested_language and not skip_unknown:
+            streams = streams_unknown
+    else:
+        streams = all_streams

    return streams

@@ -245,6 +311,9 @@ def get_plex_metadata(rating_key, part_id, item_type, plex_item=None):

    stream_info = get_plexapi_stream_info(plex_item, part_id)

+    if not stream_info:
+        return
+
    # get normalized metadata
    # fixme: duplicated logic of media_to_videos
    if item_type == "episode":
@@ -366,3 +435,4 @@ class PMSMediaProxy(object):

            m = m.children[0]
        return parts
+
@@ -4,7 +4,7 @@ import helpers
 from babelfish.exceptions import LanguageError

 from support.lib import Plex, get_intent
-from support.plex_media import get_stream_fps
+from support.plex_media import get_stream_fps, is_stream_forced, update_stream_info
 from support.storage import get_subtitle_storage
 from support.config import config, TEXT_SUBTITLE_EXTS
 from support.subtitlehelpers import get_subtitles_from_metadata
@@ -29,7 +29,7 @@ def prepare_video(pms_video_info, ignore_all=False, hints=None, rating_key=None,
    if ignore_all:
        Log.Debug("Force refresh intended.")

-    Log.Debug("Detecting streams: %s, external_subtitles=%s, embedded_subtitles=%s" % (
+    Log.Debug("Detecting streams: %s, account_for_external_subtitles=%s, account_for_embedded_subtitles=%s" % (
        plex_part.file, external_subtitles, embedded_subtitles))

    known_embedded = []
@@ -46,23 +46,25 @@ def prepare_video(pms_video_info, ignore_all=False, hints=None, rating_key=None,
    # fixme: skip the whole scanning process if known_embedded == wanted languages?
    audio_languages = []
    if plexpy_part:
+        update_stream_info(plexpy_part)
        for stream in plexpy_part.streams:
            if stream.stream_type == 2:
                lang = None
                try:
                    lang = language_from_stream(stream.language_code)
                except LanguageError:
-                    Log.Debug("Couldn't detect embedded audio stream language: %s", stream.language_code)
+                    Log.Info("Couldn't detect embedded audio stream language: %s", stream.language_code)

                # treat unknown language as lang1?
                if not lang and config.treat_und_as_first:
                    lang = Language.rebuild(list(config.lang_list)[0])
+                    Log.Info("Assuming language %s for audio stream: %s", lang, getattr(stream, "index", None))

                audio_languages.append(lang)

            # subtitle stream
            elif stream.stream_type == 3 and embedded_subtitles:
-                is_forced = helpers.is_stream_forced(stream)
+                is_forced = is_stream_forced(stream)

                if ((config.forced_only or config.forced_also) and is_forced) or not is_forced:
                    # embedded subtitle
@@ -73,11 +75,13 @@ def prepare_video(pms_video_info, ignore_all=False, hints=None, rating_key=None,
                            try:
                                lang = language_from_stream(stream.language_code)
                            except LanguageError:
-                                Log.Debug("Couldn't detect embedded subtitle stream language: %s", stream.language_code)
+                                Log.Info("Couldn't detect embedded subtitle stream language: %s", stream.language_code)

                            # treat unknown language as lang1?
                            if not lang and config.treat_und_as_first:
                                lang = Language.rebuild(list(config.lang_list)[0])
+                                Log.Info("Assuming language %s for subtitle stream: %s", lang,
+                                         getattr(stream, "index", None))

                            if lang:
                                if is_forced:
@@ -127,7 +131,8 @@ def prepare_video(pms_video_info, ignore_all=False, hints=None, rating_key=None,
            set_existing_languages(video, pms_video_info, external_subtitles=external_subtitles,
                                   embedded_subtitles=embedded_subtitles, known_embedded=known_embedded,
                                   stored_subs=stored_subs, languages=config.lang_list,
-                                   only_one=config.only_one, known_metadata_subs=known_metadata_subs)
+                                   only_one=config.only_one, known_metadata_subs=known_metadata_subs,
+                                   match_strictness=config.ext_match_strictness)

        # add video fps info
        video.fps = plex_part.fps
@@ -5,6 +5,7 @@ import helpers

 from config import config, SUBTITLE_EXTS, TEXT_SUBTITLE_EXTS
 from bs4 import UnicodeDammit
+from subzero.language import match_ietf_language


 class SubtitleHelper(object):
@@ -85,19 +86,6 @@ class VobSubSubtitleHelper(SubtitleHelper):
 #####################################################################################################################


-IETF_MATCH = ".+\.([^-.]+)(?:-[A-Za-z]+)?$"
-ENDSWITH_LANGUAGECODE_RE = re.compile("\.([^-.]{2,3})(?:-[A-Za-z]{2,})?$")
-
-
-def match_ietf_language(s):
-    language_match = re.match(".+\.([^\.]+)$" if not helpers.cast_bool(Prefs["subtitles.language.ietf_display"])
-                              else IETF_MATCH, s)
-    if language_match and len(language_match.groups()) == 1:
-        language = language_match.groups()[0]
-        return language
-    return s
-
-
 class DefaultSubtitleHelper(SubtitleHelper):
    @classmethod
    def is_helper_for(cls, filename):
@@ -133,7 +121,7 @@ class DefaultSubtitleHelper(SubtitleHelper):
        # Attempt to extract the language from the filename (e.g. Avatar (2009).eng)
        # IETF support thanks to
        # https://github.com/hpsbranco/LocalMedia.bundle/commit/4fad9aefedece78a1fa96401304351347f644369
-        lang_part = match_ietf_language(file)
+        lang_part = match_ietf_language(file, ietf=helpers.cast_bool(Prefs["subtitles.language.ietf_display"]))
        if lang_part != file:
            language = Locale.Language.Match(lang_part)
        elif config.only_one:
@@ -19,6 +19,7 @@ from support.config import config
 from support.items import get_recent_items, get_item, is_wanted, get_item_title
 from support.helpers import track_usage, get_title_for_video_metadata, cast_bool, PartUnknownException
 from support.plex_media import get_plex_metadata
+from support.extract import agent_extract_embedded
 from support.scanning import scan_videos
 from support.i18n import _
 from download import download_best_subtitles, pre_download_hook, post_download_hook, language_hook
@@ -127,8 +128,8 @@ class SubtitleListingMixin(object):
        config.init_subliminal_patches()

        provider_settings = config.provider_settings
-        if not skip_wrong_fps:
-            provider_settings["opensubtitles"]["skip_wrong_fps"] = False
+        #if not skip_wrong_fps:
+        #    provider_settings["opensubtitlescom"]["skip_wrong_fps"] = False

        if item_type == "episode":
            min_score = 240
@@ -170,12 +171,15 @@ class SubtitleListingMixin(object):
                    else:
                        s.wrong_season_ep = True

+            orig_matches = matches.copy()
+            score, score_without_hash = compute_score(matches, s, video, hearing_impaired=use_hearing_impaired)
+
            unsorted_subtitles.append(
-                (s, compute_score(matches, s, video, hearing_impaired=use_hearing_impaired), matches))
-        scored_subtitles = sorted(unsorted_subtitles, key=operator.itemgetter(1), reverse=True)
+                (s, score, score_without_hash, matches, orig_matches))
+        scored_subtitles = sorted(unsorted_subtitles, key=operator.itemgetter(1, 2), reverse=True)

        subtitles = []
-        for subtitle, score, matches in scored_subtitles:
+        for subtitle, score, score_without_hash, matches, orig_matches in scored_subtitles:
            # check score
            if score < min_score and not subtitle.wrong_series:
                Log.Info(u'%s: Score %d is below min_score (%d)', self.name, score, min_score)
@@ -449,6 +453,17 @@ class SearchAllRecentlyAddedMissing(Task):
                    Log.Debug(u"%s: Looking for missing subtitles: %s", self.name, get_item_title(plex_item))
                    scanned_parts = scan_videos([metadata], providers=providers)

+                    # auto extract embedded
+                    if config.embedded_auto_extract:
+                        if config.plex_transcoder:
+                            ts = agent_extract_embedded(scanned_parts, set_as_existing=True)
+                            if ts:
+                                Log.Debug("Waiting for %i extraction threads to finish" % len(ts))
+                                for t in ts:
+                                    t.join()
+                        else:
+                            Log.Warn("Plex Transcoder not found, can't auto extract")
+
                    downloaded_subtitles = download_best_subtitles(scanned_parts, min_score=min_score,
                                                                   providers=providers)
                    hit_providers = downloaded_subtitles is not None
@@ -655,7 +670,7 @@ class FindBetterSubtitles(DownloadSubtitleMixin, SubtitleListingMixin, Task):
        min_score_series = int(Prefs["subtitles.search.minimumTVScore2"].strip())
        min_score_movies = int(Prefs["subtitles.search.minimumMovieScore2"].strip())
        min_score_extracted_series = config.advanced.find_better_as_extracted_tv_score or 352
-        min_score_extracted_movies = config.advanced.find_better_as_extracted_movie_score or 82
+        min_score_extracted_movies = config.advanced.find_better_as_extracted_movie_score or 112
        overwrite_manually_modified = cast_bool(
            Prefs["scheduler.tasks.FindBetterSubtitles.overwrite_manually_modified"])
        overwrite_manually_selected = cast_bool(
@@ -192,6 +192,12 @@
    ],
    "default": "Always"
  },
+  {
+    "id": "subtitles.ignore_for_audio",
+    "label": "Don't download subtitles for Audio languages (use ISO-639-1 codes; comma-separated; NULL=no audio)",
+    "type": "text",
+    "default": "None"
+  },
  {
    "id": "subtitles.when_forced",
    "label": "Download foreign/forced subtitles",
@@ -288,7 +294,7 @@
  },
  {
    "id": "anticaptcha.service",
-    "label": "AntiCaptcha-Service (needs paid account; enables Addic7ed, titlovi)",
+    "label": "AntiCaptcha-Service (needs paid account; enables Addic7ed)",
    "type": "enum",
    "values": [
      "none",
@@ -305,7 +311,7 @@
  },
  {
    "id": "provider.opensubtitles.enabled",
-    "label": "Provider: Enable OpenSubtitles",
+    "label": "Provider: Enable OpenSubtitles.com",
    "type": "bool",
    "default": "true"
  },
@@ -324,10 +330,16 @@
    "secure": "true"
  },
  {
-    "id": "provider.opensubtitles.is_vip",
-    "label": "OpenSubtitles VIP? (ad-free subs, 1000 subs/day, no-cache VIP server: http://v.ht/osvip)",
+    "id": "provider.opensubtitles.use_hash",
+    "label": "OpenSubtitles hash?",
    "type": "bool",
-    "default": "false"
+    "default": "true"
+  },
+  {
+    "id": "provider.opensubtitles.api_key",
+    "label": "OpenSubtitles APIKey",
+    "type": "text",
+    "default": ""
  },
  {
    "id": "provider.podnapisi.enabled",
@@ -335,6 +347,26 @@
    "type": "bool",
    "default": "true"
  },
+  {
+    "id": "provider.napisy24.enabled",
+    "label": "Provider: Enable Napisy24 (pl)",
+    "type": "bool",
+    "default": "false"
+  },
+  {
+    "id": "provider.napisy24.username",
+    "label": "Napisy24 Username",
+    "type": "text",
+    "default": ""
+  },
+  {
+    "id": "provider.napisy24.password",
+    "label": "Napisy24 Password",
+    "type": "text",
+    "option": "hidden",
+    "default": "",
+    "secure": "true"
+  },
  {
    "id": "provider.addic7ed.enabled",
    "label": "Provider: Enable Addic7ed (needs AntiCaptcha)",
@@ -355,6 +387,12 @@
    "default": "",
    "secure": "true"
  },
+  {
+    "id": "provider.addic7ed.is_vip",
+    "label": "Addic7ed VIP? (80 vs 40 downloads per day)",
+    "type": "bool",
+    "default": "false"
+  },
  {
    "id": "provider.addic7ed.boost_by2",
    "label": "Addic7ed: boost score (if requirements met)",
@@ -389,10 +427,24 @@
  },
  {
    "id": "provider.titlovi.enabled",
-    "label": "Provider: Enable Titlovi.com (might need AntiCaptcha)",
+    "label": "Provider: Enable Titlovi.com (User and Password required)",
    "type": "bool",
    "default": "true"
  },
+  {
+    "id": "provider.titlovi.username",
+    "label": "Titlovi Username",
+    "type": "text",
+    "default": ""
+  },
+  {
+    "id": "provider.titlovi.password",
+    "label": "Titlovi Password",
+    "type": "text",
+    "option": "hidden",
+    "default": "",
+    "secure": "true"
+  },
  {
    "id": "provider.legendastv.enabled",
    "label": "Provider: Enable Legendas TV (mostly pt-BR; UNRAR NEEDED)",
@@ -431,6 +483,20 @@
    "type": "bool",
    "default": "true"
  },
+  {
+    "id": "provider.subscene.username",
+    "label": "SubScene Username",
+    "type": "text",
+    "default": ""
+  },
+  {
+    "id": "provider.subscene.password",
+    "label": "SubScene Password",
+    "type": "text",
+    "option": "hidden",
+    "default": "",
+    "secure": "true"
+  },
  {
    "id": "provider.supersubtitles.enabled",
    "label": "Provider: Enable feliratok.info (Hungarian)",
@@ -461,6 +527,38 @@
    "type": "text",
    "default": ""
  },
+  {
+    "id": "provider.bsplayer.enabled",
+    "label": "Provider: Enable BSPlayer Subtitles",
+    "type": "bool",
+    "default": "true"
+  },
+  {
+    "id": "provider.wizdom.enabled",
+    "label": "Provider: Enable WizdomSubs (Hebrew)",
+    "type": "bool",
+    "default": "true"
+  },
+ {
+    "id": "provider.ktuvit.enabled",
+    "label": "Provider: Enable Ktuvit (Hebrew)",
+    "type": "bool",
+    "default": "true"
+  },
+  {
+    "id": "provider.ktuvit.username",
+    "label": "Ktuvit Username",
+    "type": "text",
+    "default": ""
+  },
+  {
+    "id": "provider.ktuvit.password",
+    "label": "Ktuvit Password",
+    "type": "text",
+    "option": "hidden",
+    "secure": "true",
+    "default": ""
+  },
  {
    "id": "providers.multithreading",
    "label": "Search enabled providers simultaneously (multithreading)",
@@ -746,6 +844,29 @@
    "type": "bool",
    "default": "false"
  },
+  {
+    "id": "scheduler.tasks.SubtitleStorageMaintenance.frequency",
+    "label": "Scheduler: Periodically run subtitle storage maintenance (SZ internal)",
+    "type": "enum",
+    "values": [
+      "never",
+      "every 6 hours",
+      "every 12 hours",
+      "every 24 hours",
+      "every 1 days",
+      "every 2 days",
+      "every 3 days",
+      "every 4 days",
+      "every 1 weeks",
+      "every 2 weeks",
+      "every 3 weeks",
+      "every 4 weeks",
+      "every 5 weeks",
+      "every 6 weeks",
+      "every 12 weeks"
+    ],
+    "default": "every 1 weeks"
+  },
  {
    "id": "history_size",
    "label": "History: amount of items to store historical data for",
@@ -861,7 +982,7 @@
  },
  {
    "id": "use_custom_dns2",
-    "label": "Use custom DNS (IPs, comma-separated, leave empty for system DNS. Default: Google/CF)",
+    "label": "Use custom DNS (IPs, comma-separated, set to 'system' for system DNS. Default: Google/CF)",
    "type": "text",
    "default": "1.1.1.1, 8.8.8.8"
  },
@@ -13,7 +13,7 @@
        <key>CFBundleSignature</key>
        <string>????</string>
        <key>CFBundleVersion</key>
-        <string>2.6.5.3062</string>
+        <string>2.6.5.3280</string>
        <key>PlexFrameworkVersion</key>
        <string>2</string>
        <key>PlexPluginClass</key>
@@ -32,7 +32,7 @@

 &lt;h1&gt;Sub-Zero for Plex&lt;/h1&gt;&lt;i&gt;Subtitles done right&lt;/i&gt;

-Version 2.6.5.3062
+Version 2.6.5.3280

 Originally based on @bramwalet's awesome &lt;a href=&quot;https://github.com/bramwalet/Subliminal.bundle&quot;&gt;Subliminal.bundle&lt;/a&gt;

@@ -0,0 +1 @@
+__path__ = __import__('pkgutil').extend_path(__path__, __name__)
@@ -0,0 +1,196 @@
+from __future__ import absolute_import
+
+import functools
+from collections import namedtuple
+from threading import RLock
+
+_CacheInfo = namedtuple("CacheInfo", ["hits", "misses", "maxsize", "currsize"])
+
+
+@functools.wraps(functools.update_wrapper)
+def update_wrapper(
+    wrapper,
+    wrapped,
+    assigned=functools.WRAPPER_ASSIGNMENTS,
+    updated=functools.WRAPPER_UPDATES,
+):
+    """
+    Patch two bugs in functools.update_wrapper.
+    """
+    # workaround for http://bugs.python.org/issue3445
+    assigned = tuple(attr for attr in assigned if hasattr(wrapped, attr))
+    wrapper = functools.update_wrapper(wrapper, wrapped, assigned, updated)
+    # workaround for https://bugs.python.org/issue17482
+    wrapper.__wrapped__ = wrapped
+    return wrapper
+
+
+class _HashedSeq(list):
+    __slots__ = 'hashvalue'
+
+    def __init__(self, tup, hash=hash):
+        self[:] = tup
+        self.hashvalue = hash(tup)
+
+    def __hash__(self):
+        return self.hashvalue
+
+
+def _make_key(
+    args,
+    kwds,
+    typed,
+    kwd_mark=(object(),),
+    fasttypes=set([int, str, frozenset, type(None)]),
+    sorted=sorted,
+    tuple=tuple,
+    type=type,
+    len=len,
+):
+    'Make a cache key from optionally typed positional and keyword arguments'
+    key = args
+    if kwds:
+        sorted_items = sorted(kwds.items())
+        key += kwd_mark
+        for item in sorted_items:
+            key += item
+    if typed:
+        key += tuple(type(v) for v in args)
+        if kwds:
+            key += tuple(type(v) for k, v in sorted_items)
+    elif len(key) == 1 and type(key[0]) in fasttypes:
+        return key[0]
+    return _HashedSeq(key)
+
+
+def lru_cache(maxsize=100, typed=False):
+    """Least-recently-used cache decorator.
+
+    If *maxsize* is set to None, the LRU features are disabled and the cache
+    can grow without bound.
+
+    If *typed* is True, arguments of different types will be cached separately.
+    For example, f(3.0) and f(3) will be treated as distinct calls with
+    distinct results.
+
+    Arguments to the cached function must be hashable.
+
+    View the cache statistics named tuple (hits, misses, maxsize, currsize) with
+    f.cache_info().  Clear the cache and statistics with f.cache_clear().
+    Access the underlying function with f.__wrapped__.
+
+    See:  http://en.wikipedia.org/wiki/Cache_algorithms#Least_Recently_Used
+
+    """
+
+    # Users should only access the lru_cache through its public API:
+    #       cache_info, cache_clear, and f.__wrapped__
+    # The internals of the lru_cache are encapsulated for thread safety and
+    # to allow the implementation to change (including a possible C version).
+
+    def decorating_function(user_function):
+
+        cache = dict()
+        stats = [0, 0]  # make statistics updateable non-locally
+        HITS, MISSES = 0, 1  # names for the stats fields
+        make_key = _make_key
+        cache_get = cache.get  # bound method to lookup key or return None
+        _len = len  # localize the global len() function
+        lock = RLock()  # because linkedlist updates aren't threadsafe
+        root = []  # root of the circular doubly linked list
+        root[:] = [root, root, None, None]  # initialize by pointing to self
+        nonlocal_root = [root]  # make updateable non-locally
+        PREV, NEXT, KEY, RESULT = 0, 1, 2, 3  # names for the link fields
+
+        if maxsize == 0:
+
+            def wrapper(*args, **kwds):
+                # no caching, just do a statistics update after a successful call
+                result = user_function(*args, **kwds)
+                stats[MISSES] += 1
+                return result
+
+        elif maxsize is None:
+
+            def wrapper(*args, **kwds):
+                # simple caching without ordering or size limit
+                key = make_key(args, kwds, typed)
+                result = cache_get(
+                    key, root
+                )  # root used here as a unique not-found sentinel
+                if result is not root:
+                    stats[HITS] += 1
+                    return result
+                result = user_function(*args, **kwds)
+                cache[key] = result
+                stats[MISSES] += 1
+                return result
+
+        else:
+
+            def wrapper(*args, **kwds):
+                # size limited caching that tracks accesses by recency
+                key = make_key(args, kwds, typed) if kwds or typed else args
+                with lock:
+                    link = cache_get(key)
+                    if link is not None:
+                        # record recent use of the key by moving it
+                        # to the front of the list
+                        root, = nonlocal_root
+                        link_prev, link_next, key, result = link
+                        link_prev[NEXT] = link_next
+                        link_next[PREV] = link_prev
+                        last = root[PREV]
+                        last[NEXT] = root[PREV] = link
+                        link[PREV] = last
+                        link[NEXT] = root
+                        stats[HITS] += 1
+                        return result
+                result = user_function(*args, **kwds)
+                with lock:
+                    root, = nonlocal_root
+                    if key in cache:
+                        # getting here means that this same key was added to the
+                        # cache while the lock was released.  since the link
+                        # update is already done, we need only return the
+                        # computed result and update the count of misses.
+                        pass
+                    elif _len(cache) >= maxsize:
+                        # use the old root to store the new key and result
+                        oldroot = root
+                        oldroot[KEY] = key
+                        oldroot[RESULT] = result
+                        # empty the oldest link and make it the new root
+                        root = nonlocal_root[0] = oldroot[NEXT]
+                        oldkey = root[KEY]
+                        root[KEY] = root[RESULT] = None
+                        # now update the cache dictionary for the new links
+                        del cache[oldkey]
+                        cache[key] = oldroot
+                    else:
+                        # put result in a new link at the front of the list
+                        last = root[PREV]
+                        link = [last, root, key, result]
+                        last[NEXT] = root[PREV] = cache[key] = link
+                    stats[MISSES] += 1
+                return result
+
+        def cache_info():
+            """Report cache statistics"""
+            with lock:
+                return _CacheInfo(stats[HITS], stats[MISSES], maxsize, len(cache))
+
+        def cache_clear():
+            """Clear the cache and cache statistics"""
+            with lock:
+                cache.clear()
+                root = nonlocal_root[0]
+                root[:] = [root, root, None, None]
+                stats[:] = [0, 0]
+
+        wrapper.__wrapped__ = user_function
+        wrapper.cache_info = cache_info
+        wrapper.cache_clear = cache_clear
+        return update_wrapper(wrapper, user_function)
+
+    return decorating_function
@@ -1,3 +1,3 @@
-from .core import where
+from .core import contents, where

-__version__ = "2019.03.09"
+__version__ = "2020.06.20"
@@ -1,2 +1,12 @@
-from certifi import where
-print(where())
+import argparse
+
+from certifi import contents, where
+
+parser = argparse.ArgumentParser()
+parser.add_argument("-c", "--contents", action="store_true")
+args = parser.parse_args()
+
+if args.contents:
+    print(contents())
+else:
+    print(where())
@@ -58,38 +58,6 @@ AfvDbbnvRG15RjF+Cv6pgsH/76tuIMRQyV+dTZsXjAzlAcmgQWpzU/qlULRuJQ/7
 TBj0/VLZjmmx6BEP3ojY+x1J96relc8geMJgEtslQIxq/H5COEBkEveegeGTLg==
 -----END CERTIFICATE-----

-# Issuer: CN=VeriSign Class 3 Public Primary Certification Authority - G3 O=VeriSign, Inc. OU=VeriSign Trust Network/(c) 1999 VeriSign, Inc. - For authorized use only
-# Subject: CN=VeriSign Class 3 Public Primary Certification Authority - G3 O=VeriSign, Inc. OU=VeriSign Trust Network/(c) 1999 VeriSign, Inc. - For authorized use only
-# Label: "Verisign Class 3 Public Primary Certification Authority - G3"
-# Serial: 206684696279472310254277870180966723415
-# MD5 Fingerprint: cd:68:b6:a7:c7:c4:ce:75:e0:1d:4f:57:44:61:92:09
-# SHA1 Fingerprint: 13:2d:0d:45:53:4b:69:97:cd:b2:d5:c3:39:e2:55:76:60:9b:5c:c6
-# SHA256 Fingerprint: eb:04:cf:5e:b1:f3:9a:fa:76:2f:2b:b1:20:f2:96:cb:a5:20:c1:b9:7d:b1:58:95:65:b8:1c:b9:a1:7b:72:44
-----BEGIN CERTIFICATE-----
-MIIEGjCCAwICEQCbfgZJoz5iudXukEhxKe9XMA0GCSqGSIb3DQEBBQUAMIHKMQsw
-CQYDVQQGEwJVUzEXMBUGA1UEChMOVmVyaVNpZ24sIEluYy4xHzAdBgNVBAsTFlZl
-cmlTaWduIFRydXN0IE5ldHdvcmsxOjA4BgNVBAsTMShjKSAxOTk5IFZlcmlTaWdu
-LCBJbmMuIC0gRm9yIGF1dGhvcml6ZWQgdXNlIG9ubHkxRTBDBgNVBAMTPFZlcmlT
-aWduIENsYXNzIDMgUHVibGljIFByaW1hcnkgQ2VydGlmaWNhdGlvbiBBdXRob3Jp
-dHkgLSBHMzAeFw05OTEwMDEwMDAwMDBaFw0zNjA3MTYyMzU5NTlaMIHKMQswCQYD
-VQQGEwJVUzEXMBUGA1UEChMOVmVyaVNpZ24sIEluYy4xHzAdBgNVBAsTFlZlcmlT
-aWduIFRydXN0IE5ldHdvcmsxOjA4BgNVBAsTMShjKSAxOTk5IFZlcmlTaWduLCBJ
-bmMuIC0gRm9yIGF1dGhvcml6ZWQgdXNlIG9ubHkxRTBDBgNVBAMTPFZlcmlTaWdu
-IENsYXNzIDMgUHVibGljIFByaW1hcnkgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkg
-LSBHMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMu6nFL8eB8aHm8b
-N3O9+MlrlBIwT/A2R/XQkQr1F8ilYcEWQE37imGQ5XYgwREGfassbqb1EUGO+i2t
-KmFZpGcmTNDovFJbcCAEWNF6yaRpvIMXZK0Fi7zQWM6NjPXr8EJJC52XJ2cybuGu
-kxUccLwgTS8Y3pKI6GyFVxEa6X7jJhFUokWWVYPKMIno3Nij7SqAP395ZVc+FSBm
-CC+Vk7+qRy+oRpfwEuL+wgorUeZ25rdGt+INpsyow0xZVYnm6FNcHOqd8GIWC6fJ
-Xwzw3sJ2zq/3avL6QaaiMxTJ5Xpj055iN9WFZZ4O5lMkdBteHRJTW8cs54NJOxWu
-imi5V5cCAwEAATANBgkqhkiG9w0BAQUFAAOCAQEAERSWwauSCPc/L8my/uRan2Te
-2yFPhpk0djZX3dAVL8WtfxUfN2JzPtTnX84XA9s1+ivbrmAJXx5fj267Cz3qWhMe
-DGBvtcC1IyIuBwvLqXTLR7sdwdela8wv0kL9Sd2nic9TutoAWii/gt/4uhMdUIaC
-/Y4wjylGsB49Ndo4YhYYSq3mtlFs3q9i6wHQHiT+eo8SGhJouPtmmRQURVyu565p
-F4ErWjfJXir0xuKhXFSbplQAz/DxwceYMBo7Nhbbo27q/a2ywtrvAkcTisDxszGt
-TxzhT5yvDwyd93gN2PQ1VoDat20Xj50egWTh/sVFuq1ruQp6Tk9LhO5L8X3dEQ==
-----END CERTIFICATE-----
-
 # Issuer: CN=Entrust.net Certification Authority (2048) O=Entrust.net OU=www.entrust.net/CPS_2048 incorp. by ref. (limits liab.)/(c) 1999 Entrust.net Limited
 # Subject: CN=Entrust.net Certification Authority (2048) O=Entrust.net OU=www.entrust.net/CPS_2048 incorp. by ref. (limits liab.)/(c) 1999 Entrust.net Limited
 # Label: "Entrust.net Premium 2048 Secure Server CA"
@@ -152,39 +120,6 @@ ksLi4xaNmjICq44Y3ekQEe5+NauQrz4wlHrQMz2nZQ/1/I6eYs9HRCwBXbsdtTLS
 R9I4LtD+gdwyah617jzV/OeBHRnDJELqYzmp
 -----END CERTIFICATE-----

-# Issuer: CN=AddTrust External CA Root O=AddTrust AB OU=AddTrust External TTP Network
-# Subject: CN=AddTrust External CA Root O=AddTrust AB OU=AddTrust External TTP Network
-# Label: "AddTrust External Root"
-# Serial: 1
-# MD5 Fingerprint: 1d:35:54:04:85:78:b0:3f:42:42:4d:bf:20:73:0a:3f
-# SHA1 Fingerprint: 02:fa:f3:e2:91:43:54:68:60:78:57:69:4d:f5:e4:5b:68:85:18:68
-# SHA256 Fingerprint: 68:7f:a4:51:38:22:78:ff:f0:c8:b1:1f:8d:43:d5:76:67:1c:6e:b2:bc:ea:b4:13:fb:83:d9:65:d0:6d:2f:f2
-----BEGIN CERTIFICATE-----
-MIIENjCCAx6gAwIBAgIBATANBgkqhkiG9w0BAQUFADBvMQswCQYDVQQGEwJTRTEU
-MBIGA1UEChMLQWRkVHJ1c3QgQUIxJjAkBgNVBAsTHUFkZFRydXN0IEV4dGVybmFs
-IFRUUCBOZXR3b3JrMSIwIAYDVQQDExlBZGRUcnVzdCBFeHRlcm5hbCBDQSBSb290
-MB4XDTAwMDUzMDEwNDgzOFoXDTIwMDUzMDEwNDgzOFowbzELMAkGA1UEBhMCU0Ux
-FDASBgNVBAoTC0FkZFRydXN0IEFCMSYwJAYDVQQLEx1BZGRUcnVzdCBFeHRlcm5h
-bCBUVFAgTmV0d29yazEiMCAGA1UEAxMZQWRkVHJ1c3QgRXh0ZXJuYWwgQ0EgUm9v
-dDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALf3GjPm8gAELTngTlvt
-H7xsD821+iO2zt6bETOXpClMfZOfvUq8k+0DGuOPz+VtUFrWlymUWoCwSXrbLpX9
-uMq/NzgtHj6RQa1wVsfwTz/oMp50ysiQVOnGXw94nZpAPA6sYapeFI+eh6FqUNzX
-mk6vBbOmcZSccbNQYArHE504B4YCqOmoaSYYkKtMsE8jqzpPhNjfzp/haW+710LX
-a0Tkx63ubUFfclpxCDezeWWkWaCUN/cALw3CknLa0Dhy2xSoRcRdKn23tNbE7qzN
-E0S3ySvdQwAl+mG5aWpYIxG3pzOPVnVZ9c0p10a3CitlttNCbxWyuHv77+ldU9U0
-WicCAwEAAaOB3DCB2TAdBgNVHQ4EFgQUrb2YejS0Jvf6xCZU7wO94CTLVBowCwYD
-VR0PBAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wgZkGA1UdIwSBkTCBjoAUrb2YejS0
-Jvf6xCZU7wO94CTLVBqhc6RxMG8xCzAJBgNVBAYTAlNFMRQwEgYDVQQKEwtBZGRU
-cnVzdCBBQjEmMCQGA1UECxMdQWRkVHJ1c3QgRXh0ZXJuYWwgVFRQIE5ldHdvcmsx
-IjAgBgNVBAMTGUFkZFRydXN0IEV4dGVybmFsIENBIFJvb3SCAQEwDQYJKoZIhvcN
-AQEFBQADggEBALCb4IUlwtYj4g+WBpKdQZic2YR5gdkeWxQHIzZlj7DYd7usQWxH
-YINRsPkyPef89iYTx4AWpb9a/IfPeHmJIZriTAcKhjW88t5RxNKWt9x+Tu5w/Rw5
-6wwCURQtjr0W4MHfRnXnJK3s9EK0hZNwEGe6nQY1ShjTK3rMUUKhemPR5ruhxSvC
-Nr4TDea9Y355e6cJDUCrat2PisP29owaQgVR1EX1n6diIWgVIEM8med8vSTYqZEX
-c4g/VhsxOBi0cQ+azcgOno4uG+GMmIPLHzHxREzGBHNJdmAPx/i9F4BrLunMTA5a
-mnkPIAou1Z5jJh5VkpTYghdae9C8x49OhgQ=
-----END CERTIFICATE-----
-
 # Issuer: CN=Entrust Root Certification Authority O=Entrust, Inc. OU=www.entrust.net/CPS is incorporated by reference/(c) 2006 Entrust, Inc.
 # Subject: CN=Entrust Root Certification Authority O=Entrust, Inc. OU=www.entrust.net/CPS is incorporated by reference/(c) 2006 Entrust, Inc.
 # Label: "Entrust Root Certification Authority"
@@ -771,36 +706,6 @@ vEsXCS+0yx5DaMkHJ8HSXPfqIbloEpw8nL+e/IBcm2PN7EeqJSdnoDfzAIJ9VNep
 +OkuE6N36B9K
 -----END CERTIFICATE-----

-# Issuer: CN=Class 2 Primary CA O=Certplus
-# Subject: CN=Class 2 Primary CA O=Certplus
-# Label: "Certplus Class 2 Primary CA"
-# Serial: 177770208045934040241468760488327595043
-# MD5 Fingerprint: 88:2c:8c:52:b8:a2:3c:f3:f7:bb:03:ea:ae:ac:42:0b
-# SHA1 Fingerprint: 74:20:74:41:72:9c:dd:92:ec:79:31:d8:23:10:8d:c2:81:92:e2:bb
-# SHA256 Fingerprint: 0f:99:3c:8a:ef:97:ba:af:56:87:14:0e:d5:9a:d1:82:1b:b4:af:ac:f0:aa:9a:58:b5:d5:7a:33:8a:3a:fb:cb
-----BEGIN CERTIFICATE-----
-MIIDkjCCAnqgAwIBAgIRAIW9S/PY2uNp9pTXX8OlRCMwDQYJKoZIhvcNAQEFBQAw
-PTELMAkGA1UEBhMCRlIxETAPBgNVBAoTCENlcnRwbHVzMRswGQYDVQQDExJDbGFz
-cyAyIFByaW1hcnkgQ0EwHhcNOTkwNzA3MTcwNTAwWhcNMTkwNzA2MjM1OTU5WjA9
-MQswCQYDVQQGEwJGUjERMA8GA1UEChMIQ2VydHBsdXMxGzAZBgNVBAMTEkNsYXNz
-IDIgUHJpbWFyeSBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBANxQ
-ltAS+DXSCHh6tlJw/W/uz7kRy1134ezpfgSN1sxvc0NXYKwzCkTsA18cgCSR5aiR
-VhKC9+Ar9NuuYS6JEI1rbLqzAr3VNsVINyPi8Fo3UjMXEuLRYE2+L0ER4/YXJQyL
-kcAbmXuZVg2v7tK8R1fjeUl7NIknJITesezpWE7+Tt9avkGtrAjFGA7v0lPubNCd
-EgETjdyAYveVqUSISnFOYFWe2yMZeVYHDD9jC1yw4r5+FfyUM1hBOHTE4Y+L3yas
-H7WLO7dDWWuwJKZtkIvEcupdM5i3y95ee++U8Rs+yskhwcWYAqqi9lt3m/V+llU0
-HGdpwPFC40es/CgcZlUCAwEAAaOBjDCBiTAPBgNVHRMECDAGAQH/AgEKMAsGA1Ud
-DwQEAwIBBjAdBgNVHQ4EFgQU43Mt38sOKAze3bOkynm4jrvoMIkwEQYJYIZIAYb4
-QgEBBAQDAgEGMDcGA1UdHwQwMC4wLKAqoCiGJmh0dHA6Ly93d3cuY2VydHBsdXMu
-Y29tL0NSTC9jbGFzczIuY3JsMA0GCSqGSIb3DQEBBQUAA4IBAQCnVM+IRBnL39R/
-AN9WM2K191EBkOvDP9GIROkkXe/nFL0gt5o8AP5tn9uQ3Nf0YtaLcF3n5QRIqWh8
-yfFC82x/xXp8HVGIutIKPidd3i1RTtMTZGnkLuPT55sJmabglZvOGtd/vjzOUrMR
-FcEPF80Du5wlFbqidon8BvEY0JNLDnyCt6X09l/+7UCmnYR0ObncHoUW2ikbhiMA
-ybuJfm6AiB4vFLQDJKgybwOaRywwvlbGp0ICcBvqQNi6BQNwB6SW//1IMwrh3KWB
-kJtN3X3n57LNXMhqlfil9o3EXXgIvnsG1knPGTZQIy4I5p4FTUcY1Rbpsda2ENW7
-l7+ijrRU
-----END CERTIFICATE-----
-
 # Issuer: CN=DST Root CA X3 O=Digital Signature Trust Co.
 # Subject: CN=DST Root CA X3 O=Digital Signature Trust Co.
 # Label: "DST Root CA X3"
@@ -1219,36 +1124,6 @@ t0QmwCbAr1UwnjvVNioZBPRcHv/PLLf/0P2HQBHVESO7SMAhqaQoLf0V+LBOK/Qw
 WyH8EZE0vkHve52Xdf+XlcCWWC/qu0bXu+TZLg==
 -----END CERTIFICATE-----

-# Issuer: CN=Deutsche Telekom Root CA 2 O=Deutsche Telekom AG OU=T-TeleSec Trust Center
-# Subject: CN=Deutsche Telekom Root CA 2 O=Deutsche Telekom AG OU=T-TeleSec Trust Center
-# Label: "Deutsche Telekom Root CA 2"
-# Serial: 38
-# MD5 Fingerprint: 74:01:4a:91:b1:08:c4:58:ce:47:cd:f0:dd:11:53:08
-# SHA1 Fingerprint: 85:a4:08:c0:9c:19:3e:5d:51:58:7d:cd:d6:13:30:fd:8c:de:37:bf
-# SHA256 Fingerprint: b6:19:1a:50:d0:c3:97:7f:7d:a9:9b:cd:aa:c8:6a:22:7d:ae:b9:67:9e:c7:0b:a3:b0:c9:d9:22:71:c1:70:d3
-----BEGIN CERTIFICATE-----
-MIIDnzCCAoegAwIBAgIBJjANBgkqhkiG9w0BAQUFADBxMQswCQYDVQQGEwJERTEc
-MBoGA1UEChMTRGV1dHNjaGUgVGVsZWtvbSBBRzEfMB0GA1UECxMWVC1UZWxlU2Vj
-IFRydXN0IENlbnRlcjEjMCEGA1UEAxMaRGV1dHNjaGUgVGVsZWtvbSBSb290IENB
-IDIwHhcNOTkwNzA5MTIxMTAwWhcNMTkwNzA5MjM1OTAwWjBxMQswCQYDVQQGEwJE
-RTEcMBoGA1UEChMTRGV1dHNjaGUgVGVsZWtvbSBBRzEfMB0GA1UECxMWVC1UZWxl
-U2VjIFRydXN0IENlbnRlcjEjMCEGA1UEAxMaRGV1dHNjaGUgVGVsZWtvbSBSb290
-IENBIDIwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCrC6M14IspFLEU
-ha88EOQ5bzVdSq7d6mGNlUn0b2SjGmBmpKlAIoTZ1KXleJMOaAGtuU1cOs7TuKhC
-QN/Po7qCWWqSG6wcmtoIKyUn+WkjR/Hg6yx6m/UTAtB+NHzCnjwAWav12gz1Mjwr
-rFDa1sPeg5TKqAyZMg4ISFZbavva4VhYAUlfckE8FQYBjl2tqriTtM2e66foai1S
-NNs671x1Udrb8zH57nGYMsRUFUQM+ZtV7a3fGAigo4aKSe5TBY8ZTNXeWHmb0moc
-QqvF1afPaA+W5OFhmHZhyJF81j4A4pFQh+GdCuatl9Idxjp9y7zaAzTVjlsB9WoH
-txa2bkp/AgMBAAGjQjBAMB0GA1UdDgQWBBQxw3kbuvVT1xfgiXotF2wKsyudMzAP
-BgNVHRMECDAGAQH/AgEFMA4GA1UdDwEB/wQEAwIBBjANBgkqhkiG9w0BAQUFAAOC
-AQEAlGRZrTlk5ynrE/5aw4sTV8gEJPB0d8Bg42f76Ymmg7+Wgnxu1MM9756Abrsp
-tJh6sTtU6zkXR34ajgv8HzFZMQSyzhfzLMdiNlXiItiJVbSYSKpk+tYcNthEeFpa
-IzpXl/V6ME+un2pMSyuOoAPjPuCp1NJ70rOo4nI8rZ7/gFnkm0W09juwzTkZmDLl
-6iFhkOQxIY40sfcvNUqFENrnijchvllj4PKFiDFT1FQUhXB59C4Gdyd1Lx+4ivn+
-xbrYNuSD7Odlt79jWvNGr4GUN9RBjNYj1h7P9WgbRGOiWrqnNVmh5XAFmw4jV5mU
-Cm26OWMohpLzGITY+9HPBVZkVw==
-----END CERTIFICATE-----
-
 # Issuer: CN=Cybertrust Global Root O=Cybertrust, Inc
 # Subject: CN=Cybertrust Global Root O=Cybertrust, Inc
 # Label: "Cybertrust Global Root"
@@ -1559,47 +1434,6 @@ uLjbvrW5KfnaNwUASZQDhETnv0Mxz3WLJdH0pmT1kvarBes96aULNmLazAZfNou2
 XjG4Kvte9nHfRCaexOYNkbQudZWAUWpLMKawYqGT8ZvYzsRjdT9ZR7E=
 -----END CERTIFICATE-----

-# Issuer: CN=Staat der Nederlanden Root CA - G2 O=Staat der Nederlanden
-# Subject: CN=Staat der Nederlanden Root CA - G2 O=Staat der Nederlanden
-# Label: "Staat der Nederlanden Root CA - G2"
-# Serial: 10000012
-# MD5 Fingerprint: 7c:a5:0f:f8:5b:9a:7d:6d:30:ae:54:5a:e3:42:a2:8a
-# SHA1 Fingerprint: 59:af:82:79:91:86:c7:b4:75:07:cb:cf:03:57:46:eb:04:dd:b7:16
-# SHA256 Fingerprint: 66:8c:83:94:7d:a6:3b:72:4b:ec:e1:74:3c:31:a0:e6:ae:d0:db:8e:c5:b3:1b:e3:77:bb:78:4f:91:b6:71:6f
-----BEGIN CERTIFICATE-----
-MIIFyjCCA7KgAwIBAgIEAJiWjDANBgkqhkiG9w0BAQsFADBaMQswCQYDVQQGEwJO
-TDEeMBwGA1UECgwVU3RhYXQgZGVyIE5lZGVybGFuZGVuMSswKQYDVQQDDCJTdGFh
-dCBkZXIgTmVkZXJsYW5kZW4gUm9vdCBDQSAtIEcyMB4XDTA4MDMyNjExMTgxN1oX
-DTIwMDMyNTExMDMxMFowWjELMAkGA1UEBhMCTkwxHjAcBgNVBAoMFVN0YWF0IGRl
-ciBOZWRlcmxhbmRlbjErMCkGA1UEAwwiU3RhYXQgZGVyIE5lZGVybGFuZGVuIFJv
-b3QgQ0EgLSBHMjCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAMVZ5291
-qj5LnLW4rJ4L5PnZyqtdj7U5EILXr1HgO+EASGrP2uEGQxGZqhQlEq0i6ABtQ8Sp
-uOUfiUtnvWFI7/3S4GCI5bkYYCjDdyutsDeqN95kWSpGV+RLufg3fNU254DBtvPU
-Z5uW6M7XxgpT0GtJlvOjCwV3SPcl5XCsMBQgJeN/dVrlSPhOewMHBPqCYYdu8DvE
-pMfQ9XQ+pV0aCPKbJdL2rAQmPlU6Yiile7Iwr/g3wtG61jj99O9JMDeZJiFIhQGp
-5Rbn3JBV3w/oOM2ZNyFPXfUib2rFEhZgF1XyZWampzCROME4HYYEhLoaJXhena/M
-UGDWE4dS7WMfbWV9whUYdMrhfmQpjHLYFhN9C0lK8SgbIHRrxT3dsKpICT0ugpTN
-GmXZK4iambwYfp/ufWZ8Pr2UuIHOzZgweMFvZ9C+X+Bo7d7iscksWXiSqt8rYGPy
-5V6548r6f1CGPqI0GAwJaCgRHOThuVw+R7oyPxjMW4T182t0xHJ04eOLoEq9jWYv
-6q012iDTiIJh8BIitrzQ1aTsr1SIJSQ8p22xcik/Plemf1WvbibG/ufMQFxRRIEK
-eN5KzlW/HdXZt1bv8Hb/C3m1r737qWmRRpdogBQ2HbN/uymYNqUg+oJgYjOk7Na6
-B6duxc8UpufWkjTYgfX8HV2qXB72o007uPc5AgMBAAGjgZcwgZQwDwYDVR0TAQH/
-BAUwAwEB/zBSBgNVHSAESzBJMEcGBFUdIAAwPzA9BggrBgEFBQcCARYxaHR0cDov
-L3d3dy5wa2lvdmVyaGVpZC5ubC9wb2xpY2llcy9yb290LXBvbGljeS1HMjAOBgNV
-HQ8BAf8EBAMCAQYwHQYDVR0OBBYEFJFoMocVHYnitfGsNig0jQt8YojrMA0GCSqG
-SIb3DQEBCwUAA4ICAQCoQUpnKpKBglBu4dfYszk78wIVCVBR7y29JHuIhjv5tLyS
-CZa59sCrI2AGeYwRTlHSeYAz+51IvuxBQ4EffkdAHOV6CMqqi3WtFMTC6GY8ggen
-5ieCWxjmD27ZUD6KQhgpxrRW/FYQoAUXvQwjf/ST7ZwaUb7dRUG/kSS0H4zpX897
-IZmflZ85OkYcbPnNe5yQzSipx6lVu6xiNGI1E0sUOlWDuYaNkqbG9AclVMwWVxJK
-gnjIFNkXgiYtXSAfea7+1HAWFpWD2DU5/1JddRwWxRNVz0fMdWVSSt7wsKfkCpYL
-+63C4iWEst3kvX5ZbJvw8NjnyvLplzh+ib7M+zkXYT9y2zqR2GUBGR2tUKRXCnxL
-vJxxcypFURmFzI79R6d0lR2o0a9OF7FpJsKqeFdbxU2n5Z4FF5TKsl+gSRiNNOkm
-bEgeqmiSBeGCc1qb3AdbCG19ndeNIdn8FCCqwkXfP+cAslHkwvgFuXkajDTznlvk
-N1trSt8sV4pAWja63XVECDdCcAz+3F4hoKOKwJCcaNpQ5kUQR3i2TtJlycM33+FC
-Y7BXN0Ute4qcvwXqZVUz9zkQxSgqIXobisQk+T8VyJoVIPVVYpbtbZNQvOSqeK3Z
-ywplh6ZmwcSBo3c6WB4L7oOLnR7SUqTMHW+wmG2UMbX4cQrcufx9MmDm66+KAQ==
-----END CERTIFICATE-----
-
 # Issuer: CN=Hongkong Post Root CA 1 O=Hongkong Post
 # Subject: CN=Hongkong Post Root CA 1 O=Hongkong Post
 # Label: "Hongkong Post Root CA 1"
@@ -2200,6 +2034,45 @@ t/2jioSgrGK+KwmHNPBqAbubKVY8/gA3zyNs8U6qtnRGEmyR7jTV7JqR50S+kDFy
 SjnRBUkLp7Y3gaVdjKozXoEofKd9J+sAro03
 -----END CERTIFICATE-----

+# Issuer: CN=EC-ACC O=Agencia Catalana de Certificacio (NIF Q-0801176-I) OU=Serveis Publics de Certificacio/Vegeu https://www.catcert.net/verarrel (c)03/Jerarquia Entitats de Certificacio Catalanes
+# Subject: CN=EC-ACC O=Agencia Catalana de Certificacio (NIF Q-0801176-I) OU=Serveis Publics de Certificacio/Vegeu https://www.catcert.net/verarrel (c)03/Jerarquia Entitats de Certificacio Catalanes
+# Label: "EC-ACC"
+# Serial: -23701579247955709139626555126524820479
+# MD5 Fingerprint: eb:f5:9d:29:0d:61:f9:42:1f:7c:c2:ba:6d:e3:15:09
+# SHA1 Fingerprint: 28:90:3a:63:5b:52:80:fa:e6:77:4c:0b:6d:a7:d6:ba:a6:4a:f2:e8
+# SHA256 Fingerprint: 88:49:7f:01:60:2f:31:54:24:6a:e2:8c:4d:5a:ef:10:f1:d8:7e:bb:76:62:6f:4a:e0:b7:f9:5b:a7:96:87:99
+-----BEGIN CERTIFICATE-----
+MIIFVjCCBD6gAwIBAgIQ7is969Qh3hSoYqwE893EATANBgkqhkiG9w0BAQUFADCB
+8zELMAkGA1UEBhMCRVMxOzA5BgNVBAoTMkFnZW5jaWEgQ2F0YWxhbmEgZGUgQ2Vy
+dGlmaWNhY2lvIChOSUYgUS0wODAxMTc2LUkpMSgwJgYDVQQLEx9TZXJ2ZWlzIFB1
+YmxpY3MgZGUgQ2VydGlmaWNhY2lvMTUwMwYDVQQLEyxWZWdldSBodHRwczovL3d3
+dy5jYXRjZXJ0Lm5ldC92ZXJhcnJlbCAoYykwMzE1MDMGA1UECxMsSmVyYXJxdWlh
+IEVudGl0YXRzIGRlIENlcnRpZmljYWNpbyBDYXRhbGFuZXMxDzANBgNVBAMTBkVD
+LUFDQzAeFw0wMzAxMDcyMzAwMDBaFw0zMTAxMDcyMjU5NTlaMIHzMQswCQYDVQQG
+EwJFUzE7MDkGA1UEChMyQWdlbmNpYSBDYXRhbGFuYSBkZSBDZXJ0aWZpY2FjaW8g
+KE5JRiBRLTA4MDExNzYtSSkxKDAmBgNVBAsTH1NlcnZlaXMgUHVibGljcyBkZSBD
+ZXJ0aWZpY2FjaW8xNTAzBgNVBAsTLFZlZ2V1IGh0dHBzOi8vd3d3LmNhdGNlcnQu
+bmV0L3ZlcmFycmVsIChjKTAzMTUwMwYDVQQLEyxKZXJhcnF1aWEgRW50aXRhdHMg
+ZGUgQ2VydGlmaWNhY2lvIENhdGFsYW5lczEPMA0GA1UEAxMGRUMtQUNDMIIBIjAN
+BgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAsyLHT+KXQpWIR4NA9h0X84NzJB5R
+85iKw5K4/0CQBXCHYMkAqbWUZRkiFRfCQ2xmRJoNBD45b6VLeqpjt4pEndljkYRm
+4CgPukLjbo73FCeTae6RDqNfDrHrZqJyTxIThmV6PttPB/SnCWDaOkKZx7J/sxaV
+HMf5NLWUhdWZXqBIoH7nF2W4onW4HvPlQn2v7fOKSGRdghST2MDk/7NQcvJ29rNd
+QlB50JQ+awwAvthrDk4q7D7SzIKiGGUzE3eeml0aE9jD2z3Il3rucO2n5nzbcc8t
+lGLfbdb1OL4/pYUKGbio2Al1QnDE6u/LDsg0qBIimAy4E5S2S+zw0JDnJwIDAQAB
+o4HjMIHgMB0GA1UdEQQWMBSBEmVjX2FjY0BjYXRjZXJ0Lm5ldDAPBgNVHRMBAf8E
+BTADAQH/MA4GA1UdDwEB/wQEAwIBBjAdBgNVHQ4EFgQUoMOLRKo3pUW/l4Ba0fF4
+opvpXY0wfwYDVR0gBHgwdjB0BgsrBgEEAfV4AQMBCjBlMCwGCCsGAQUFBwIBFiBo
+dHRwczovL3d3dy5jYXRjZXJ0Lm5ldC92ZXJhcnJlbDA1BggrBgEFBQcCAjApGidW
+ZWdldSBodHRwczovL3d3dy5jYXRjZXJ0Lm5ldC92ZXJhcnJlbCAwDQYJKoZIhvcN
+AQEFBQADggEBAKBIW4IB9k1IuDlVNZyAelOZ1Vr/sXE7zDkJlF7W2u++AVtd0x7Y
+/X1PzaBB4DSTv8vihpw3kpBWHNzrKQXlxJ7HNd+KDM3FIUPpqojlNcAZQmNaAl6k
+SBg6hW/cnbw/nZzBh7h6YQjpdwt/cKt63dmXLGQehb+8dJahw3oS7AwaboMMPOhy
+Rp/7SNVel+axofjk70YllJyJ22k4vuxcDlbHZVHlUIiIv0LVKz3l+bqeLrPK9HOS
+Agu+TGbrIP65y7WZf+a2E/rKS03Z7lNGBjvGTq2TWoF+bCpLagVFjPIhpDGQh2xl
+nJ2lYJU6Un/10asIbvPuW/mIPX64b24D5EI=
+-----END CERTIFICATE-----
+
 # Issuer: CN=Hellenic Academic and Research Institutions RootCA 2011 O=Hellenic Academic and Research Institutions Cert. Authority
 # Subject: CN=Hellenic Academic and Research Institutions RootCA 2011 O=Hellenic Academic and Research Institutions Cert. Authority
 # Label: "Hellenic Academic and Research Institutions RootCA 2011"
@@ -3453,46 +3326,6 @@ AAoACxGV2lZFA4gKn2fQ1XmxqI1AbQ3CekD6819kR5LLU7m7Wc5P/dAVUwHY3+vZ
 5nbv0CO7O6l5s9UCKc2Jo5YPSjXnTkLAdc0Hz+Ys63su
 -----END CERTIFICATE-----

-# Issuer: CN=Certinomis - Root CA O=Certinomis OU=0002 433998903
-# Subject: CN=Certinomis - Root CA O=Certinomis OU=0002 433998903
-# Label: "Certinomis - Root CA"
-# Serial: 1
-# MD5 Fingerprint: 14:0a:fd:8d:a8:28:b5:38:69:db:56:7e:61:22:03:3f
-# SHA1 Fingerprint: 9d:70:bb:01:a5:a4:a0:18:11:2e:f7:1c:01:b9:32:c5:34:e7:88:a8
-# SHA256 Fingerprint: 2a:99:f5:bc:11:74:b7:3c:bb:1d:62:08:84:e0:1c:34:e5:1c:cb:39:78:da:12:5f:0e:33:26:88:83:bf:41:58
-----BEGIN CERTIFICATE-----
-MIIFkjCCA3qgAwIBAgIBATANBgkqhkiG9w0BAQsFADBaMQswCQYDVQQGEwJGUjET
-MBEGA1UEChMKQ2VydGlub21pczEXMBUGA1UECxMOMDAwMiA0MzM5OTg5MDMxHTAb
-BgNVBAMTFENlcnRpbm9taXMgLSBSb290IENBMB4XDTEzMTAyMTA5MTcxOFoXDTMz
-MTAyMTA5MTcxOFowWjELMAkGA1UEBhMCRlIxEzARBgNVBAoTCkNlcnRpbm9taXMx
-FzAVBgNVBAsTDjAwMDIgNDMzOTk4OTAzMR0wGwYDVQQDExRDZXJ0aW5vbWlzIC0g
-Um9vdCBDQTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBANTMCQosP5L2
-fxSeC5yaah1AMGT9qt8OHgZbn1CF6s2Nq0Nn3rD6foCWnoR4kkjW4znuzuRZWJfl
-LieY6pOod5tK8O90gC3rMB+12ceAnGInkYjwSond3IjmFPnVAy//ldu9n+ws+hQV
-WZUKxkd8aRi5pwP5ynapz8dvtF4F/u7BUrJ1Mofs7SlmO/NKFoL21prbcpjp3vDF
-TKWrteoB4owuZH9kb/2jJZOLyKIOSY008B/sWEUuNKqEUL3nskoTuLAPrjhdsKkb
-5nPJWqHZZkCqqU2mNAKthH6yI8H7KsZn9DS2sJVqM09xRLWtwHkziOC/7aOgFLSc
-CbAK42C++PhmiM1b8XcF4LVzbsF9Ri6OSyemzTUK/eVNfaoqoynHWmgE6OXWk6Ri
-wsXm9E/G+Z8ajYJJGYrKWUM66A0ywfRMEwNvbqY/kXPLynNvEiCL7sCCeN5LLsJJ
-wx3tFvYk9CcbXFcx3FXuqB5vbKziRcxXV4p1VxngtViZSTYxPDMBbRZKzbgqg4SG
-m/lg0h9tkQPTYKbVPZrdd5A9NaSfD171UkRpucC63M9933zZxKyGIjK8e2uR73r4
-F2iw4lNVYC2vPsKD2NkJK/DAZNuHi5HMkesE/Xa0lZrmFAYb1TQdvtj/dBxThZng
-WVJKYe2InmtJiUZ+IFrZ50rlau7SZRFDAgMBAAGjYzBhMA4GA1UdDwEB/wQEAwIB
-BjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBTvkUz1pcMw6C8I6tNxIqSSaHh0
-2TAfBgNVHSMEGDAWgBTvkUz1pcMw6C8I6tNxIqSSaHh02TANBgkqhkiG9w0BAQsF
-AAOCAgEAfj1U2iJdGlg+O1QnurrMyOMaauo++RLrVl89UM7g6kgmJs95Vn6RHJk/
-0KGRHCwPT5iVWVO90CLYiF2cN/z7ZMF4jIuaYAnq1fohX9B0ZedQxb8uuQsLrbWw
-F6YSjNRieOpWauwK0kDDPAUwPk2Ut59KA9N9J0u2/kTO+hkzGm2kQtHdzMjI1xZS
-g081lLMSVX3l4kLr5JyTCcBMWwerx20RoFAXlCOotQqSD7J6wWAsOMwaplv/8gzj
-qh8c3LigkyfeY+N/IZ865Z764BNqdeuWXGKRlI5nU7aJ+BIJy29SWwNyhlCVCNSN
-h4YVH5Uk2KRvms6knZtt0rJ2BobGVgjF6wnaNsIbW0G+YSrjcOa4pvi2WsS9Iff/
-ql+hbHY5ZtbqTFXhADObE5hjyW/QASAJN1LnDE8+zbz1X5YnpyACleAu6AdBBR8V
-btaw5BngDwKTACdyxYvRVB9dSsNAl35VpnzBMwQUAR1JIGkLGZOdblgi90AMRgwj
-Y/M50n92Uaf0yKHxDHYiI0ZSKS3io0EHVmmY0gUJvGnHWmHNj4FgFU2A3ZDifcRQ
-8ow7bkrHxuaAKzyBvBGAFhAn1/DNP3nMcyrDflOR1m749fPH0FFNjkulW+YZFzvW
-gQncItzujrnEj1PhZ7szuIgVRs/taTX/dQ1G885x4cVrhkIGuUE=
-----END CERTIFICATE-----
-
 # Issuer: CN=OISTE WISeKey Global Root GB CA O=WISeKey OU=OISTE Foundation Endorsed
 # Subject: CN=OISTE WISeKey Global Root GB CA O=WISeKey OU=OISTE Foundation Endorsed
 # Label: "OISTE WISeKey Global Root GB CA"
@@ -3849,47 +3682,6 @@ CkcO8DdZEv8tmZQoTipPNU0zWgIxAOp1AE47xDqUEpHJWEadIRNyp4iciuRMStuW
 1KyLa2tJElMzrdfkviT8tQp21KW8EA==
 -----END CERTIFICATE-----

-# Issuer: CN=LuxTrust Global Root 2 O=LuxTrust S.A.
-# Subject: CN=LuxTrust Global Root 2 O=LuxTrust S.A.
-# Label: "LuxTrust Global Root 2"
-# Serial: 59914338225734147123941058376788110305822489521
-# MD5 Fingerprint: b2:e1:09:00:61:af:f7:f1:91:6f:c4:ad:8d:5e:3b:7c
-# SHA1 Fingerprint: 1e:0e:56:19:0a:d1:8b:25:98:b2:04:44:ff:66:8a:04:17:99:5f:3f
-# SHA256 Fingerprint: 54:45:5f:71:29:c2:0b:14:47:c4:18:f9:97:16:8f:24:c5:8f:c5:02:3b:f5:da:5b:e2:eb:6e:1d:d8:90:2e:d5
-----BEGIN CERTIFICATE-----
-MIIFwzCCA6ugAwIBAgIUCn6m30tEntpqJIWe5rgV0xZ/u7EwDQYJKoZIhvcNAQEL
-BQAwRjELMAkGA1UEBhMCTFUxFjAUBgNVBAoMDUx1eFRydXN0IFMuQS4xHzAdBgNV
-BAMMFkx1eFRydXN0IEdsb2JhbCBSb290IDIwHhcNMTUwMzA1MTMyMTU3WhcNMzUw
-MzA1MTMyMTU3WjBGMQswCQYDVQQGEwJMVTEWMBQGA1UECgwNTHV4VHJ1c3QgUy5B
-LjEfMB0GA1UEAwwWTHV4VHJ1c3QgR2xvYmFsIFJvb3QgMjCCAiIwDQYJKoZIhvcN
-AQEBBQADggIPADCCAgoCggIBANeFl78RmOnwYoNMPIf5U2o3C/IPPIfOb9wmKb3F
-ibrJgz337spbxm1Jc7TJRqMbNBM/wYlFV/TZsfs2ZUv7COJIcRHIbjuend+JZTem
-hfY7RBi2xjcwYkSSl2l9QjAk5A0MiWtj3sXh306pFGxT4GHO9hcvHTy95iJMHZP1
-EMShduxq3sVs35a0VkBCwGKSMKEtFZSg0iAGCW5qbeXrt77U8PEVfIvmTroTzEsn
-Xpk8F12PgX8zPU/TPxvsXD/wPEx1bvKm1Z3aLQdjAsZy6ZS8TEmVT4hSyNvoaYL4
-zDRbIvCGp4m9SAptZoFtyMhk+wHh9OHe2Z7d21vUKpkmFRseTJIpgp7VkoGSQXAZ
-96Tlk0u8d2cx3Rz9MXANF5kM+Qw5GSoXtTBxVdUPrljhPS80m8+f9niFwpN6cj5m
-j5wWEWCPnolvZ77gR1o7DJpni89Gxq44o/KnvObWhWszJHAiS8sIm7vI+AIpHb4g
-DEa/a4ebsypmQjVGbKq6rfmYe+lQVRQxv7HaLe2ArWgk+2mr2HETMOZns4dA/Yl+
-8kPREd8vZS9kzl8UubG/Mb2HeFpZZYiq/FkySIbWTLkpS5XTdvN3JW1CHDiDTf2j
-X5t/Lax5Gw5CMZdjpPuKadUiDTSQMC6otOBttpSsvItO13D8xTiOZCXhTTmQzsmH
-hFhxAgMBAAGjgagwgaUwDwYDVR0TAQH/BAUwAwEB/zBCBgNVHSAEOzA5MDcGByuB
-KwEBAQowLDAqBggrBgEFBQcCARYeaHR0cHM6Ly9yZXBvc2l0b3J5Lmx1eHRydXN0
-Lmx1MA4GA1UdDwEB/wQEAwIBBjAfBgNVHSMEGDAWgBT/GCh2+UgFLKGu8SsbK7JT
-+Et8szAdBgNVHQ4EFgQU/xgodvlIBSyhrvErGyuyU/hLfLMwDQYJKoZIhvcNAQEL
-BQADggIBAGoZFO1uecEsh9QNcH7X9njJCwROxLHOk3D+sFTAMs2ZMGQXvw/l4jP9
-BzZAcg4atmpZ1gDlaCDdLnINH2pkMSCEfUmmWjfrRcmF9dTHF5kH5ptV5AzoqbTO
-jFu1EVzPig4N1qx3gf4ynCSecs5U89BvolbW7MM3LGVYvlcAGvI1+ut7MV3CwRI9
-loGIlonBWVx65n9wNOeD4rHh4bhY79SV5GCc8JaXcozrhAIuZY+kt9J/Z93I055c
-qqmkoCUUBpvsT34tC38ddfEz2O3OuHVtPlu5mB0xDVbYQw8wkbIEa91WvpWAVWe+
-2M2D2RjuLg+GLZKecBPs3lHJQ3gCpU3I+V/EkVhGFndadKpAvAefMLmx9xIX3eP/
-JEAdemrRTxgKqpAd60Ae36EeRJIQmvKN4dFLRp7oRUKX6kWZ8+xm1QL68qZKJKre
-zrnK+T+Tb/mjuuqlPpmt/f97mfVl7vBZKGfXkJWkE4SphMHozs51k2MavDzq1WQf
-LSoSOcbDWjLtR5EWDrw4wVDej8oqkDQc7kGUnF4ZLvhFSZl0kbAEb+MEWrGrKqv+
-x9CWttrhSmQGbmBNvUJO/3jaJMobtNeWOWyu8Q6qp31IiyBMz2TWuJdGsE7RKlY6
-oJO9r4Ak4Ap+58rVyuiFVdw2KuGUaJPHZnJED4AhMmwlxyOAgwrr
-----END CERTIFICATE-----
-
 # Issuer: CN=TUBITAK Kamu SM SSL Kok Sertifikasi - Surum 1 O=Turkiye Bilimsel ve Teknolojik Arastirma Kurumu - TUBITAK OU=Kamu Sertifikasyon Merkezi - Kamu SM
 # Subject: CN=TUBITAK Kamu SM SSL Kok Sertifikasi - Surum 1 O=Turkiye Bilimsel ve Teknolojik Arastirma Kurumu - TUBITAK OU=Kamu Sertifikasyon Merkezi - Kamu SM
 # Label: "TUBITAK Kamu SM SSL Kok Sertifikasi - Surum 1"
@@ -4656,3 +4448,173 @@ L5/ndtFhKvshuzHQqp9HpLIiyhY6UFfEW0NnxWViA0kB60PZ2Pierc+xYw5F9KBa
 LJstxabArahH9CdMOA0uG0k7UvToiIMrVCjU8jVStDKDYmlkDJGcn5fqdBb9HxEG
 mpv0
 -----END CERTIFICATE-----
+
+# Issuer: CN=Entrust Root Certification Authority - G4 O=Entrust, Inc. OU=See www.entrust.net/legal-terms/(c) 2015 Entrust, Inc. - for authorized use only
+# Subject: CN=Entrust Root Certification Authority - G4 O=Entrust, Inc. OU=See www.entrust.net/legal-terms/(c) 2015 Entrust, Inc. - for authorized use only
+# Label: "Entrust Root Certification Authority - G4"
+# Serial: 289383649854506086828220374796556676440
+# MD5 Fingerprint: 89:53:f1:83:23:b7:7c:8e:05:f1:8c:71:38:4e:1f:88
+# SHA1 Fingerprint: 14:88:4e:86:26:37:b0:26:af:59:62:5c:40:77:ec:35:29:ba:96:01
+# SHA256 Fingerprint: db:35:17:d1:f6:73:2a:2d:5a:b9:7c:53:3e:c7:07:79:ee:32:70:a6:2f:b4:ac:42:38:37:24:60:e6:f0:1e:88
+-----BEGIN CERTIFICATE-----
+MIIGSzCCBDOgAwIBAgIRANm1Q3+vqTkPAAAAAFVlrVgwDQYJKoZIhvcNAQELBQAw
+gb4xCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1FbnRydXN0LCBJbmMuMSgwJgYDVQQL
+Ex9TZWUgd3d3LmVudHJ1c3QubmV0L2xlZ2FsLXRlcm1zMTkwNwYDVQQLEzAoYykg
+MjAxNSBFbnRydXN0LCBJbmMuIC0gZm9yIGF1dGhvcml6ZWQgdXNlIG9ubHkxMjAw
+BgNVBAMTKUVudHJ1c3QgUm9vdCBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eSAtIEc0
+MB4XDTE1MDUyNzExMTExNloXDTM3MTIyNzExNDExNlowgb4xCzAJBgNVBAYTAlVT
+MRYwFAYDVQQKEw1FbnRydXN0LCBJbmMuMSgwJgYDVQQLEx9TZWUgd3d3LmVudHJ1
+c3QubmV0L2xlZ2FsLXRlcm1zMTkwNwYDVQQLEzAoYykgMjAxNSBFbnRydXN0LCBJ
+bmMuIC0gZm9yIGF1dGhvcml6ZWQgdXNlIG9ubHkxMjAwBgNVBAMTKUVudHJ1c3Qg
+Um9vdCBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eSAtIEc0MIICIjANBgkqhkiG9w0B
+AQEFAAOCAg8AMIICCgKCAgEAsewsQu7i0TD/pZJH4i3DumSXbcr3DbVZwbPLqGgZ
+2K+EbTBwXX7zLtJTmeH+H17ZSK9dE43b/2MzTdMAArzE+NEGCJR5WIoV3imz/f3E
+T+iq4qA7ec2/a0My3dl0ELn39GjUu9CH1apLiipvKgS1sqbHoHrmSKvS0VnM1n4j
+5pds8ELl3FFLFUHtSUrJ3hCX1nbB76W1NhSXNdh4IjVS70O92yfbYVaCNNzLiGAM
+C1rlLAHGVK/XqsEQe9IFWrhAnoanw5CGAlZSCXqc0ieCU0plUmr1POeo8pyvi73T
+DtTUXm6Hnmo9RR3RXRv06QqsYJn7ibT/mCzPfB3pAqoEmh643IhuJbNsZvc8kPNX
+wbMv9W3y+8qh+CmdRouzavbmZwe+LGcKKh9asj5XxNMhIWNlUpEbsZmOeX7m640A
+2Vqq6nPopIICR5b+W45UYaPrL0swsIsjdXJ8ITzI9vF01Bx7owVV7rtNOzK+mndm
+nqxpkCIHH2E6lr7lmk/MBTwoWdPBDFSoWWG9yHJM6Nyfh3+9nEg2XpWjDrk4JFX8
+dWbrAuMINClKxuMrLzOg2qOGpRKX/YAr2hRC45K9PvJdXmd0LhyIRyk0X+IyqJwl
+N4y6mACXi0mWHv0liqzc2thddG5msP9E36EYxr5ILzeUePiVSj9/E15dWf10hkNj
+c0kCAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYwHQYD
+VR0OBBYEFJ84xFYjwznooHFs6FRM5Og6sb9nMA0GCSqGSIb3DQEBCwUAA4ICAQAS
+5UKme4sPDORGpbZgQIeMJX6tuGguW8ZAdjwD+MlZ9POrYs4QjbRaZIxowLByQzTS
+Gwv2LFPSypBLhmb8qoMi9IsabyZIrHZ3CL/FmFz0Jomee8O5ZDIBf9PD3Vht7LGr
+hFV0d4QEJ1JrhkzO3bll/9bGXp+aEJlLdWr+aumXIOTkdnrG0CSqkM0gkLpHZPt/
+B7NTeLUKYvJzQ85BK4FqLoUWlFPUa19yIqtRLULVAJyZv967lDtX/Zr1hstWO1uI
+AeV8KEsD+UmDfLJ/fOPtjqF/YFOOVZ1QNBIPt5d7bIdKROf1beyAN/BYGW5KaHbw
+H5Lk6rWS02FREAutp9lfx1/cH6NcjKF+m7ee01ZvZl4HliDtC3T7Zk6LERXpgUl+
+b7DUUH8i119lAg2m9IUe2K4GS0qn0jFmwvjO5QimpAKWRGhXxNUzzxkvFMSUHHuk
+2fCfDrGA4tGeEWSpiBE6doLlYsKA2KSD7ZPvfC+QsDJMlhVoSFLUmQjAJOgc47Ol
+IQ6SwJAfzyBfyjs4x7dtOvPmRLgOMWuIjnDrnBdSqEGULoe256YSxXXfW8AKbnuk
+5F6G+TaU33fD6Q3AOfF5u0aOq0NZJ7cguyPpVkAh7DE9ZapD8j3fcEThuk0mEDuY
+n/PIjhs4ViFqUZPTkcpG2om3PVODLAgfi49T3f+sHw==
+-----END CERTIFICATE-----
+
+# Issuer: CN=Microsoft ECC Root Certificate Authority 2017 O=Microsoft Corporation
+# Subject: CN=Microsoft ECC Root Certificate Authority 2017 O=Microsoft Corporation
+# Label: "Microsoft ECC Root Certificate Authority 2017"
+# Serial: 136839042543790627607696632466672567020
+# MD5 Fingerprint: dd:a1:03:e6:4a:93:10:d1:bf:f0:19:42:cb:fe:ed:67
+# SHA1 Fingerprint: 99:9a:64:c3:7f:f4:7d:9f:ab:95:f1:47:69:89:14:60:ee:c4:c3:c5
+# SHA256 Fingerprint: 35:8d:f3:9d:76:4a:f9:e1:b7:66:e9:c9:72:df:35:2e:e1:5c:fa:c2:27:af:6a:d1:d7:0e:8e:4a:6e:dc:ba:02
+-----BEGIN CERTIFICATE-----
+MIICWTCCAd+gAwIBAgIQZvI9r4fei7FK6gxXMQHC7DAKBggqhkjOPQQDAzBlMQsw
+CQYDVQQGEwJVUzEeMBwGA1UEChMVTWljcm9zb2Z0IENvcnBvcmF0aW9uMTYwNAYD
+VQQDEy1NaWNyb3NvZnQgRUNDIFJvb3QgQ2VydGlmaWNhdGUgQXV0aG9yaXR5IDIw
+MTcwHhcNMTkxMjE4MjMwNjQ1WhcNNDIwNzE4MjMxNjA0WjBlMQswCQYDVQQGEwJV
+UzEeMBwGA1UEChMVTWljcm9zb2Z0IENvcnBvcmF0aW9uMTYwNAYDVQQDEy1NaWNy
+b3NvZnQgRUNDIFJvb3QgQ2VydGlmaWNhdGUgQXV0aG9yaXR5IDIwMTcwdjAQBgcq
+hkjOPQIBBgUrgQQAIgNiAATUvD0CQnVBEyPNgASGAlEvaqiBYgtlzPbKnR5vSmZR
+ogPZnZH6thaxjG7efM3beaYvzrvOcS/lpaso7GMEZpn4+vKTEAXhgShC48Zo9OYb
+hGBKia/teQ87zvH2RPUBeMCjVDBSMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMBAf8E
+BTADAQH/MB0GA1UdDgQWBBTIy5lycFIM+Oa+sgRXKSrPQhDtNTAQBgkrBgEEAYI3
+FQEEAwIBADAKBggqhkjOPQQDAwNoADBlAjBY8k3qDPlfXu5gKcs68tvWMoQZP3zV
+L8KxzJOuULsJMsbG7X7JNpQS5GiFBqIb0C8CMQCZ6Ra0DvpWSNSkMBaReNtUjGUB
+iudQZsIxtzm6uBoiB078a1QWIP8rtedMDE2mT3M=
+-----END CERTIFICATE-----
+
+# Issuer: CN=Microsoft RSA Root Certificate Authority 2017 O=Microsoft Corporation
+# Subject: CN=Microsoft RSA Root Certificate Authority 2017 O=Microsoft Corporation
+# Label: "Microsoft RSA Root Certificate Authority 2017"
+# Serial: 40975477897264996090493496164228220339
+# MD5 Fingerprint: 10:ff:00:ff:cf:c9:f8:c7:7a:c0:ee:35:8e:c9:0f:47
+# SHA1 Fingerprint: 73:a5:e6:4a:3b:ff:83:16:ff:0e:dc:cc:61:8a:90:6e:4e:ae:4d:74
+# SHA256 Fingerprint: c7:41:f7:0f:4b:2a:8d:88:bf:2e:71:c1:41:22:ef:53:ef:10:eb:a0:cf:a5:e6:4c:fa:20:f4:18:85:30:73:e0
+-----BEGIN CERTIFICATE-----
+MIIFqDCCA5CgAwIBAgIQHtOXCV/YtLNHcB6qvn9FszANBgkqhkiG9w0BAQwFADBl
+MQswCQYDVQQGEwJVUzEeMBwGA1UEChMVTWljcm9zb2Z0IENvcnBvcmF0aW9uMTYw
+NAYDVQQDEy1NaWNyb3NvZnQgUlNBIFJvb3QgQ2VydGlmaWNhdGUgQXV0aG9yaXR5
+IDIwMTcwHhcNMTkxMjE4MjI1MTIyWhcNNDIwNzE4MjMwMDIzWjBlMQswCQYDVQQG
+EwJVUzEeMBwGA1UEChMVTWljcm9zb2Z0IENvcnBvcmF0aW9uMTYwNAYDVQQDEy1N
+aWNyb3NvZnQgUlNBIFJvb3QgQ2VydGlmaWNhdGUgQXV0aG9yaXR5IDIwMTcwggIi
+MA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQDKW76UM4wplZEWCpW9R2LBifOZ
+Nt9GkMml7Xhqb0eRaPgnZ1AzHaGm++DlQ6OEAlcBXZxIQIJTELy/xztokLaCLeX0
+ZdDMbRnMlfl7rEqUrQ7eS0MdhweSE5CAg2Q1OQT85elss7YfUJQ4ZVBcF0a5toW1
+HLUX6NZFndiyJrDKxHBKrmCk3bPZ7Pw71VdyvD/IybLeS2v4I2wDwAW9lcfNcztm
+gGTjGqwu+UcF8ga2m3P1eDNbx6H7JyqhtJqRjJHTOoI+dkC0zVJhUXAoP8XFWvLJ
+jEm7FFtNyP9nTUwSlq31/niol4fX/V4ggNyhSyL71Imtus5Hl0dVe49FyGcohJUc
+aDDv70ngNXtk55iwlNpNhTs+VcQor1fznhPbRiefHqJeRIOkpcrVE7NLP8TjwuaG
+YaRSMLl6IE9vDzhTyzMMEyuP1pq9KsgtsRx9S1HKR9FIJ3Jdh+vVReZIZZ2vUpC6
+W6IYZVcSn2i51BVrlMRpIpj0M+Dt+VGOQVDJNE92kKz8OMHY4Xu54+OU4UZpyw4K
+UGsTuqwPN1q3ErWQgR5WrlcihtnJ0tHXUeOrO8ZV/R4O03QK0dqq6mm4lyiPSMQH
+FJDOvTKVTUssKZqwJz58oHhEmrARdlns87/I6KJClTUFLkqqNfs+avNJVgyeY+Q
+W5g5xAgGwax/Dj0ApQIDAQABo1QwUjAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/
+BAUwAwEB/zAdBgNVHQ4EFgQUCctZf4aycI8awznjwNnpv7tNsiMwEAYJKwYBBAGC
+NxUBBAMCAQAwDQYJKoZIhvcNAQEMBQADggIBAKyvPl3CEZaJjqPnktaXFbgToqZC
+LgLNFgVZJ8og6Lq46BrsTaiXVq5lQ7GPAJtSzVXNUzltYkyLDVt8LkS/gxCP81OC
+gMNPOsduET/m4xaRhPtthH80dK2Jp86519efhGSSvpWhrQlTM93uCupKUY5vVau6
+tZRGrox/2KJQJWVggEbbMwSubLWYdFQl3JPk+ONVFT24bcMKpBLBaYVu32TxU5nh
+SnUgnZUP5NbcA/FZGOhHibJXWpS2qdgXKxdJ5XbLwVaZOjex/2kskZGT4d9Mozd2
+TaGf+G0eHdP67Pv0RR0Tbc/3WeUiJ3IrhvNXuzDtJE3cfVa7o7P4NHmJweDyAmH3
+pvwPuxwXC65B2Xy9J6P9LjrRk5Sxcx0ki69bIImtt2dmefU6xqaWM/5TkshGsRGR
+xpl/j8nWZjEgQRCHLQzWwa80mMpkg/sTV9HB8Dx6jKXB/ZUhoHHBk2dxEuqPiApp
+GWSZI1b7rCoucL5mxAyE7+WL85MB+GqQk2dLsmijtWKP6T+MejteD+eMuMZ87zf9
+dOLITzNy4ZQ5bb0Sr74MTnB8G2+NszKTc0QWbej09+CVgI+WXTik9KveCjCHk9hN
+AHFiRSdLOkKEW39lt2c0Ui2cFmuqqNh7o0JMcccMyj6D5KbvtwEwXlGjefVwaaZB
+RA+GsCyRxj3qrg+E
+-----END CERTIFICATE-----
+
+# Issuer: CN=e-Szigno Root CA 2017 O=Microsec Ltd.
+# Subject: CN=e-Szigno Root CA 2017 O=Microsec Ltd.
+# Label: "e-Szigno Root CA 2017"
+# Serial: 411379200276854331539784714
+# MD5 Fingerprint: de:1f:f6:9e:84:ae:a7:b4:21:ce:1e:58:7d:d1:84:98
+# SHA1 Fingerprint: 89:d4:83:03:4f:9e:9a:48:80:5f:72:37:d4:a9:a6:ef:cb:7c:1f:d1
+# SHA256 Fingerprint: be:b0:0b:30:83:9b:9b:c3:2c:32:e4:44:79:05:95:06:41:f2:64:21:b1:5e:d0:89:19:8b:51:8a:e2:ea:1b:99
+-----BEGIN CERTIFICATE-----
+MIICQDCCAeWgAwIBAgIMAVRI7yH9l1kN9QQKMAoGCCqGSM49BAMCMHExCzAJBgNV
+BAYTAkhVMREwDwYDVQQHDAhCdWRhcGVzdDEWMBQGA1UECgwNTWljcm9zZWMgTHRk
+LjEXMBUGA1UEYQwOVkFUSFUtMjM1ODQ0OTcxHjAcBgNVBAMMFWUtU3ppZ25vIFJv
+b3QgQ0EgMjAxNzAeFw0xNzA4MjIxMjA3MDZaFw00MjA4MjIxMjA3MDZaMHExCzAJ
+BgNVBAYTAkhVMREwDwYDVQQHDAhCdWRhcGVzdDEWMBQGA1UECgwNTWljcm9zZWMg
+THRkLjEXMBUGA1UEYQwOVkFUSFUtMjM1ODQ0OTcxHjAcBgNVBAMMFWUtU3ppZ25v
+IFJvb3QgQ0EgMjAxNzBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABJbcPYrYsHtv
+xie+RJCxs1YVe45DJH0ahFnuY2iyxl6H0BVIHqiQrb1TotreOpCmYF9oMrWGQd+H
+Wyx7xf58etqjYzBhMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMB0G
+A1UdDgQWBBSHERUI0arBeAyxr87GyZDvvzAEwDAfBgNVHSMEGDAWgBSHERUI0arB
+eAyxr87GyZDvvzAEwDAKBggqhkjOPQQDAgNJADBGAiEAtVfd14pVCzbhhkT61Nlo
+jbjcI4qKDdQvfepz7L9NbKgCIQDLpbQS+ue16M9+k/zzNY9vTlp8tLxOsvxyqltZ
+efcMQ==
+-----END CERTIFICATE-----
+
+# Issuer: O=CERTSIGN SA OU=certSIGN ROOT CA G2
+# Subject: O=CERTSIGN SA OU=certSIGN ROOT CA G2
+# Label: "certSIGN Root CA G2"
+# Serial: 313609486401300475190
+# MD5 Fingerprint: 8c:f1:75:8a:c6:19:cf:94:b7:f7:65:20:87:c3:97:c7
+# SHA1 Fingerprint: 26:f9:93:b4:ed:3d:28:27:b0:b9:4b:a7:e9:15:1d:a3:8d:92:e5:32
+# SHA256 Fingerprint: 65:7c:fe:2f:a7:3f:aa:38:46:25:71:f3:32:a2:36:3a:46:fc:e7:02:09:51:71:07:02:cd:fb:b6:ee:da:33:05
+-----BEGIN CERTIFICATE-----
+MIIFRzCCAy+gAwIBAgIJEQA0tk7GNi02MA0GCSqGSIb3DQEBCwUAMEExCzAJBgNV
+BAYTAlJPMRQwEgYDVQQKEwtDRVJUU0lHTiBTQTEcMBoGA1UECxMTY2VydFNJR04g
+Uk9PVCBDQSBHMjAeFw0xNzAyMDYwOTI3MzVaFw00MjAyMDYwOTI3MzVaMEExCzAJ
+BgNVBAYTAlJPMRQwEgYDVQQKEwtDRVJUU0lHTiBTQTEcMBoGA1UECxMTY2VydFNJ
+R04gUk9PVCBDQSBHMjCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAMDF
+dRmRfUR0dIf+DjuW3NgBFszuY5HnC2/OOwppGnzC46+CjobXXo9X69MhWf05N0Iw
+vlDqtg+piNguLWkh59E3GE59kdUWX2tbAMI5Qw02hVK5U2UPHULlj88F0+7cDBrZ
+uIt4ImfkabBoxTzkbFpG583H+u/E7Eu9aqSs/cwoUe+StCmrqzWaTOTECMYmzPhp
+n+Sc8CnTXPnGFiWeI8MgwT0PPzhAsP6CRDiqWhqKa2NYOLQV07YRaXseVO6MGiKs
+cpc/I1mbySKEwQdPzH/iV8oScLumZfNpdWO9lfsbl83kqK/20U6o2YpxJM02PbyW
+xPFsqa7lzw1uKA2wDrXKUXt4FMMgL3/7FFXhEZn91QqhngLjYl/rNUssuHLoPj1P
+rCy7Lobio3aP5ZMqz6WryFyNSwb/EkaseMsUBzXgqd+L6a8VTxaJW732jcZZroiF
+DsGJ6x9nxUWO/203Nit4ZoORUSs9/1F3dmKh7Gc+PoGD4FapUB8fepmrY7+EF3fx
+DTvf95xhszWYijqy7DwaNz9+j5LP2RIUZNoQAhVB/0/E6xyjyfqZ90bp4RjZsbgy
+LcsUDFDYg2WD7rlcz8sFWkz6GZdr1l0T08JcVLwyc6B49fFtHsufpaafItzRUZ6C
+eWRgKRM+o/1Pcmqr4tTluCRVLERLiohEnMqE0yo7AgMBAAGjQjBAMA8GA1UdEwEB
+/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMB0GA1UdDgQWBBSCIS1mxteg4BXrzkwJ
+d8RgnlRuAzANBgkqhkiG9w0BAQsFAAOCAgEAYN4auOfyYILVAzOBywaK8SJJ6ejq
+kX/GM15oGQOGO0MBzwdw5AgeZYWR5hEit/UCI46uuR59H35s5r0l1ZUa8gWmr4UC
+b6741jH/JclKyMeKqdmfS0mbEVeZkkMR3rYzpMzXjWR91M08KCy0mpbqTfXERMQl
+qiCA2ClV9+BB/AYm/7k29UMUA2Z44RGx2iBfRgB4ACGlHgAoYXhvqAEBj500mv/0
+OJD7uNGzcgbJceaBxXntC6Z58hMLnPddDnskk7RI24Zf3lCGeOdA5jGokHZwYa+c
+NywRtYK3qq4kNFtyDGkNzVmf9nGvnAvRCjj5BiKDUyUM/FHE5r7iOZULJK2v0ZXk
+ltd0ZGtxTgI8qoXzIKNDOXZbbFD+mpwUHmUUihW9o4JFWklWatKcsWMy5WHgUyIO
+pwpJ6st+H6jiYoD2EEVSmAYY3qXNL3+q1Ok+CHLsIwMCPKaq2LxndD0UF/tUSxfj
+03k9bWtJySgOLnRQvwzZRjoQhsmnP+mg7H/rpXdYaXHmgwo38oZJar55CJD2AhZk
+PuXaTH4MNMn5X7azKFGnpyuqSfqNZSlO42sTp5SjLVFteAxEy9/eCG/Oo2Sr05WE
+1LlSVHJ7liXMvGnjSG4N0MedJ5qq+BOS3R7fY581qRY27Iy4g/Q9iY/NtBde17MX
+QRBdJ3NghVdJIgc=
+-----END CERTIFICATE-----
@@ -4,12 +4,57 @@
 certifi.py
 ~~~~~~~~~~

-This module returns the installation location of cacert.pem.
+This module returns the installation location of cacert.pem or its contents.
 """
 import os

+try:
+    from importlib.resources import path as get_path, read_text

-def where():
-    f = os.path.dirname(__file__)
+    _CACERT_CTX = None
+    _CACERT_PATH = None

-    return os.path.join(f, 'cacert.pem')
+    def where():
+        # This is slightly terrible, but we want to delay extracting the file
+        # in cases where we're inside of a zipimport situation until someone
+        # actually calls where(), but we don't want to re-extract the file
+        # on every call of where(), so we'll do it once then store it in a
+        # global variable.
+        global _CACERT_CTX
+        global _CACERT_PATH
+        if _CACERT_PATH is None:
+            # This is slightly janky, the importlib.resources API wants you to
+            # manage the cleanup of this file, so it doesn't actually return a
+            # path, it returns a context manager that will give you the path
+            # when you enter it and will do any cleanup when you leave it. In
+            # the common case of not needing a temporary file, it will just
+            # return the file system location and the __exit__() is a no-op.
+            #
+            # We also have to hold onto the actual context manager, because
+            # it will do the cleanup whenever it gets garbage collected, so
+            # we will also store that at the global level as well.
+            _CACERT_CTX = get_path("certifi", "cacert.pem")
+            _CACERT_PATH = str(_CACERT_CTX.__enter__())
+
+        return _CACERT_PATH
+
+
+except ImportError:
+    # This fallback will work for Python versions prior to 3.7 that lack the
+    # importlib.resources module but relies on the existing `where` function
+    # so won't address issues with environments like PyOxidizer that don't set
+    # __file__ on modules.
+    def read_text(_module, _path, encoding="ascii"):
+        with open(where(), "r", encoding=encoding) as data:
+            return data.read()
+
+    # If we don't have importlib.resources, then we will just do the old logic
+    # of assuming we're on the filesystem and munge the path directly.
+    def where():
+        f = os.path.dirname(__file__)
+
+        return os.path.join(f, "cacert.pem")
+
+
+def contents():
+    return read_text("certifi", "cacert.pem", encoding="ascii")
@@ -1,11 +1,7 @@
-import re
-import ssl
-try:
-    import brotli
-except ImportError:
-    brotli = None
-
 import logging
+import re
+import sys
+import ssl

 from copy import deepcopy
 from time import sleep
@@ -14,7 +10,6 @@ from collections import OrderedDict
 from requests.sessions import Session
 from requests.adapters import HTTPAdapter
 from requests.packages.urllib3.util.ssl_ import create_urllib3_context
-from subliminal_patch.pitcher import pitchers

 from .interpreters import JavaScriptInterpreter
 from .user_agent import User_Agent
@@ -24,6 +19,11 @@ try:
 except ImportError:
    pass

+try:
+    import brotli
+except ImportError:
+    pass
+
 try:
    from urlparse import urlparse
    from urlparse import urlunparse
@@ -31,11 +31,9 @@ except ImportError:
    from urllib.parse import urlparse
    from urllib.parse import urlunparse

-logger = logging.getLogger(__name__)
-
 ##########################################################################################################################################################

-__version__ = '1.1.1'
+__version__ = '1.1.9'

 BUG_REPORT = 'Cloudflare may have changed their technique, or there may be a bug in the script.'

@@ -46,38 +44,41 @@ class CipherSuiteAdapter(HTTPAdapter):

    def __init__(self, cipherSuite=None, **kwargs):
        self.cipherSuite = cipherSuite
+
+        if hasattr(ssl, 'PROTOCOL_TLS'):
+            self.ssl_context = create_urllib3_context(
+                ssl_version=getattr(ssl, 'PROTOCOL_TLSv1_3', ssl.PROTOCOL_TLSv1_2),
+                ciphers=self.cipherSuite
+            )
+        else:
+            self.ssl_context = create_urllib3_context(ssl_version=ssl.PROTOCOL_TLSv1)
+
        super(CipherSuiteAdapter, self).__init__(**kwargs)

    ##########################################################################################################################################################

    def init_poolmanager(self, *args, **kwargs):
-        kwargs['ssl_context'] = create_urllib3_context(ciphers=self.cipherSuite)
+        kwargs['ssl_context'] = self.ssl_context
        return super(CipherSuiteAdapter, self).init_poolmanager(*args, **kwargs)

    ##########################################################################################################################################################

    def proxy_manager_for(self, *args, **kwargs):
-        kwargs['ssl_context'] = create_urllib3_context(ciphers=self.cipherSuite)
+        kwargs['ssl_context'] = self.ssl_context
        return super(CipherSuiteAdapter, self).proxy_manager_for(*args, **kwargs)

 ##########################################################################################################################################################


-class NeedsCaptchaException(Exception):
-    pass
-
-
 class CloudScraper(Session):
-    was_cf_request = False
-
    def __init__(self, *args, **kwargs):
        self.debug = kwargs.pop('debug', False)
        self.delay = kwargs.pop('delay', None)
        self.interpreter = kwargs.pop('interpreter', 'js2py')
-        self.allow_brotli = kwargs.pop('allow_brotli', True) and bool(brotli)
+        self.allow_brotli = kwargs.pop('allow_brotli', True if 'brotli' in sys.modules.keys() else False)
        self.cipherSuite = None

-        super(CloudScraper, self).__init__()
+        super(CloudScraper, self).__init__(*args, **kwargs)

        if 'requests' in self.headers['User-Agent']:
            # Set a random User-Agent if no custom User-Agent has been set
@@ -100,24 +101,27 @@ class CloudScraper(Session):
        if self.cipherSuite:
            return self.cipherSuite

-        ciphers = [
-            'GREASE_3A', 'GREASE_6A', 'AES128-GCM-SHA256', 'AES256-GCM-SHA256', 'AES256-GCM-SHA384', 'CHACHA20-POLY1305-SHA256',
-            'ECDHE-ECDSA-AES128-GCM-SHA256', 'ECDHE-RSA-AES128-GCM-SHA256', 'ECDHE-ECDSA-AES256-GCM-SHA384',
-            'ECDHE-RSA-AES256-GCM-SHA384', 'ECDHE-ECDSA-CHACHA20-POLY1305-SHA256', 'ECDHE-RSA-CHACHA20-POLY1305-SHA256',
-            'ECDHE-RSA-AES128-CBC-SHA', 'ECDHE-RSA-AES256-CBC-SHA', 'RSA-AES128-GCM-SHA256', 'RSA-AES256-GCM-SHA384',
-            'ECDHE-RSA-AES128-GCM-SHA256', 'RSA-AES256-SHA', '3DES-EDE-CBC'
-        ]
-
        self.cipherSuite = ''

-        ctx = ssl.SSLContext(ssl.PROTOCOL_SSLv23)
+        if hasattr(ssl, 'PROTOCOL_TLS'):
+            ciphers = [
+                'ECDHE-ECDSA-AES128-GCM-SHA256', 'ECDHE-RSA-AES128-GCM-SHA256', 'ECDHE-ECDSA-AES256-GCM-SHA384',
+                'ECDHE-RSA-AES256-GCM-SHA384', 'ECDHE-ECDSA-CHACHA20-POLY1305-SHA256', 'ECDHE-RSA-CHACHA20-POLY1305-SHA256',
+                'ECDHE-RSA-AES128-CBC-SHA', 'ECDHE-RSA-AES256-CBC-SHA', 'RSA-AES128-GCM-SHA256', 'RSA-AES256-GCM-SHA384',
+                'ECDHE-RSA-AES128-GCM-SHA256', 'RSA-AES256-SHA', '3DES-EDE-CBC'
+            ]

-        for cipher in ciphers:
-            try:
-                ctx.set_ciphers(cipher)
-                self.cipherSuite = '{}:{}'.format(self.cipherSuite, cipher).rstrip(':')
-            except ssl.SSLError:
-                pass
+            if hasattr(ssl, 'PROTOCOL_TLSv1_3'):
+                ciphers.insert(0, ['GREASE_3A', 'GREASE_6A', 'AES128-GCM-SHA256', 'AES256-GCM-SHA256', 'AES256-GCM-SHA384', 'CHACHA20-POLY1305-SHA256'])
+
+            ctx = ssl.SSLContext(getattr(ssl, 'PROTOCOL_TLSv1_3', ssl.PROTOCOL_TLSv1_2))
+
+            for cipher in ciphers:
+                try:
+                    ctx.set_ciphers(cipher)
+                    self.cipherSuite = '{}:{}'.format(self.cipherSuite, cipher).rstrip(':')
+                except ssl.SSLError:
+                    pass

        return self.cipherSuite

@@ -139,54 +143,15 @@ class CloudScraper(Session):
            self.debugRequest(resp)

        # Check if Cloudflare anti-bot is on
-        try:
-            if self.isChallengeRequest(resp):
-                self.was_cf_request = True
-                if resp.request.method != 'GET':
-                    # Work around if the initial request is not a GET,
-                    # Supersede with a GET then re-request the original METHOD.
-                    self.request('GET', resp.url)
-                    resp = ourSuper.request(method, url, *args, **kwargs)
-                else:
-                    # Solve Challenge
-                    resp = self.sendChallengeResponse(resp, **kwargs)
-        except NeedsCaptchaException:
-            self.was_cf_request = True
-            parsed_url = urlparse(url)
-            domain = parsed_url.netloc
-            # solve the captcha
-            site_key = re.search(r'data-sitekey="(.+?)"', resp.content).group(1)
-            challenge_s = re.search(r'type="hidden" name="s" value="(.+?)"', resp.content).group(1)
-            challenge_ray = re.search(r'data-ray="(.+?)"', resp.content).group(1)
-            if not all([site_key, challenge_s, challenge_ray]):
-                raise Exception("cf: Captcha site-key not found!")
-
-            pitcher = pitchers.get_pitcher()("cf: %s" % domain, resp.request.url, site_key,
-                                             user_agent=self.headers["User-Agent"],
-                                             cookies=self.cookies.get_dict(),
-                                             is_invisible=True)
-
-            parsed_url = urlparse(resp.url)
-            logger.info("cf: %s: Solving captcha", domain)
-            result = pitcher.throw()
-            if not result:
-                raise Exception("cf: Couldn't solve captcha!")
-
-            submit_url = '{}://{}/cdn-cgi/l/chk_captcha'.format(parsed_url.scheme, domain)
-            method = resp.request.method
-
-            cloudflare_kwargs = {
-                'allow_redirects': False,
-                'headers': {'Referer': resp.url},
-                'params': OrderedDict(
-                    [
-                        ('s', challenge_s),
-                        ('g-recaptcha-response', result)
-                    ]
-                )
-            }
-
-            return self.request(method, submit_url, **cloudflare_kwargs)
+        if self.isChallengeRequest(resp):
+            if resp.request.method != 'GET':
+                # Work around if the initial request is not a GET,
+                # Supersede with a GET then re-request the original METHOD.
+                self.request('GET', resp.url)
+                resp = ourSuper.request(method, url, *args, **kwargs)
+            else:
+                # Solve Challenge
+                resp = self.sendChallengeResponse(resp, **kwargs)

        return resp

@@ -196,7 +161,7 @@ class CloudScraper(Session):
    def isChallengeRequest(resp):
        if resp.headers.get('Server', '').startswith('cloudflare'):
            if b'why_captcha' in resp.content or b'/cdn-cgi/l/chk_captcha' in resp.content:
-                raise NeedsCaptchaException
+                raise ValueError('Captcha')

            return (
                resp.status_code in [429, 503]
@@ -52,10 +52,10 @@ class JavaScriptInterpreter(ABC):
        js += '\na.value;'

        jsEnv = '''
-            function italics (str) {{ return "<i>" + this + "</i>"; }};
+            String.prototype.italics=function(str) {{return "<i>" + this + "</i>";}};
            var document = {{
                createElement: function () {{
-                    return {{ firstChild: {{ href: "http://{domain}/" }} }}
+                    return {{ firstChild: {{ href: "https://{domain}/" }} }}
                }},
                getElementById: function () {{
                    return {{"innerHTML": "{innerHTML}"}};
@@ -5,6 +5,7 @@ import re
 from .translators.friendly_nodes import REGEXP_CONVERTER
 from .utils.injector import fix_js_args
 from types import FunctionType, ModuleType, GeneratorType, BuiltinFunctionType, MethodType, BuiltinMethodType
+from math import floor, log10
 import traceback
 try:
    import numpy
@@ -603,15 +604,7 @@ class PyJs(object):
        elif typ == 'Boolean':
            return Js('true') if self.value else Js('false')
        elif typ == 'Number':  #or self.Class=='Number':
-            if self.is_nan():
-                return Js('NaN')
-            elif self.is_infinity():
-                sign = '-' if self.value < 0 else ''
-                return Js(sign + 'Infinity')
-            elif isinstance(self.value,
-                            long) or self.value.is_integer():  # dont print .0
-                return Js(unicode(int(self.value)))
-            return Js(unicode(self.value))  # accurate enough
+            return Js(unicode(js_dtoa(self.value)))
        elif typ == 'String':
            return self
        else:  #object
@@ -1046,7 +1039,7 @@ def PyJsComma(a, b):
    return b


-from .internals.simplex import JsException as PyJsException
+from .internals.simplex import JsException as PyJsException, js_dtoa
 import pyjsparser
 pyjsparser.parser.ENABLE_JS2PY_ERRORS = lambda msg: MakeError('SyntaxError', msg)

@@ -116,10 +116,12 @@ def eval_js(js):


 def eval_js6(js):
+    """Just like eval_js but with experimental support for js6 via babel."""
    return eval_js(js6_to_js5(js))


 def translate_js6(js):
+    """Just like translate_js but with experimental support for js6 via babel."""
    return translate_js(js6_to_js5(js))


@@ -3,15 +3,19 @@ import re

 import datetime

-from desc import *
-from simplex import *
-from conversions import *
-import six
-from pyjsparser import PyJsParser
-from itertools import izip
+from .desc import *
+from .simplex import *
+from .conversions import *
+
+from pyjsparser import PyJsParser
+
+import six
+if six.PY2:
+    from itertools import izip
+else:
+    izip = zip
+

-from conversions import *
-from simplex import *


 def Type(obj):
@@ -1,8 +1,8 @@
-from code import Code
-from simplex import MakeError
-from opcodes import *
-from operations import *
-from trans_utils import *
+from .code import Code
+from .simplex import MakeError
+from .opcodes import *
+from .operations import *
+from .trans_utils import *

 SPECIAL_IDENTIFIERS = {'true', 'false', 'this'}

@@ -465,10 +465,11 @@ class ByteCodeGenerator:
        self.emit('LOAD_OBJECT', tuple(data))

    def Program(self, body, **kwargs):
+        old_tape_len = len(self.exe.tape)
        self.emit('LOAD_UNDEFINED')
        self.emit(body)
        # add function tape !
-        self.exe.tape = self.function_declaration_tape + self.exe.tape
+        self.exe.tape = self.exe.tape[:old_tape_len] + self.function_declaration_tape + self.exe.tape[old_tape_len:]

    def Pyimport(self, imp, **kwargs):
        raise NotImplementedError(
@@ -735,17 +736,17 @@ def main():
    #
    # }
    a.emit(d)
-    print a.declared_vars
-    print a.exe.tape
-    print len(a.exe.tape)
+    print(a.declared_vars)
+    print(a.exe.tape)
+    print(len(a.exe.tape))

    a.exe.compile()

    def log(this, args):
-        print args[0]
+        print(args[0])
        return 999

-    print a.exe.run(a.exe.space.GlobalObj)
+    print(a.exe.run(a.exe.space.GlobalObj))


 if __name__ == '__main__':
@@ -1,16 +1,17 @@
-from opcodes import *
-from space import *
-from base import *
+from .opcodes import *
+from .space import *
+from .base import *


 class Code:
    '''Can generate, store and run sequence of ops representing js code'''

-    def __init__(self, is_strict=False):
+    def __init__(self, is_strict=False, debug_mode=False):
        self.tape = []
        self.compiled = False
        self.label_locs = None
        self.is_strict = is_strict
+        self.debug_mode = debug_mode

        self.contexts = []
        self.current_ctx = None
@@ -22,6 +23,10 @@ class Code:
        self.GLOBAL_THIS = None
        self.space = None

+        # dbg
+        self.ctx_depth = 0
+
+
    def get_new_label(self):
        self._label_count += 1
        return self._label_count
@@ -74,21 +79,35 @@ class Code:
            # 0=normal, 1=return, 2=jump_outside, 3=errors
            # execute_fragment_under_context returns:
            # (return_value, typ, return_value/jump_loc/py_error)
-            # ctx.stack must be len 1 and its always empty after the call.
+            # IMPARTANT: It is guaranteed that the length of the ctx.stack is unchanged.
        '''
        old_curr_ctx = self.current_ctx
+        self.ctx_depth += 1
+        old_stack_len = len(ctx.stack)
+        old_ret_len = len(self.return_locs)
+        old_ctx_len = len(self.contexts)
        try:
            self.current_ctx = ctx
            return self._execute_fragment_under_context(
                ctx, start_label, end_label)
        except JsException as err:
-            # undo the things that were put on the stack (if any)
-            # don't worry, I know the recovery is possible through try statement and for this reason try statement
-            # has its own context and stack so it will not delete the contents of the outer stack
-            del ctx.stack[:]
+            if self.debug_mode:
+                self._on_fragment_exit("js errors")
+            # undo the things that were put on the stack (if any) to ensure a proper error recovery
+            del ctx.stack[old_stack_len:]
+            del self.return_locs[old_ret_len:]
+            del self.contexts[old_ctx_len :]
            return undefined, 3, err
        finally:
+            self.ctx_depth -= 1
            self.current_ctx = old_curr_ctx
+            assert old_stack_len == len(ctx.stack)
+
+    def _get_dbg_indent(self):
+        return self.ctx_depth * '  '
+
+    def _on_fragment_exit(self, mode):
+        print(self._get_dbg_indent() + 'ctx exit (%s)' % mode)

    def _execute_fragment_under_context(self, ctx, start_label, end_label):
        start, end = self.label_locs[start_label], self.label_locs[end_label]
@@ -97,16 +116,20 @@ class Code:
        entry_level = len(self.contexts)
        # for e in self.tape[start:end]:
        #     print e
-
+        if self.debug_mode:
+            print(self._get_dbg_indent() + 'ctx entry (from:%d, to:%d)' % (start, end))
        while loc < len(self.tape):
-            #print loc, self.tape[loc]
            if len(self.contexts) == entry_level and loc >= end:
+                if self.debug_mode:
+                    self._on_fragment_exit('normal')
                assert loc == end
-                assert len(ctx.stack) == (
-                    1 + initial_len), 'Stack change must be equal to +1!'
+                delta_stack = len(ctx.stack) - initial_len
+                assert delta_stack == +1, 'Stack change must be equal to +1! got %d' % delta_stack
                return ctx.stack.pop(), 0, None  # means normal return

            # execute instruction
+            if self.debug_mode:
+                print(self._get_dbg_indent() + str(loc), self.tape[loc])
            status = self.tape[loc].eval(ctx)

            # check status for special actions
@@ -116,9 +139,10 @@ class Code:
                    if len(self.contexts) == entry_level:
                        # check if jumped outside of the fragment and break if so
                        if not start <= loc < end:
-                            assert len(ctx.stack) == (
-                                1 + initial_len
-                            ), 'Stack change must be equal to +1!'
+                            if self.debug_mode:
+                                self._on_fragment_exit('jump outside loc:%d label:%d' % (loc, status))
+                            delta_stack = len(ctx.stack) - initial_len
+                            assert delta_stack == +1, 'Stack change must be equal to +1! got %d' % delta_stack
                            return ctx.stack.pop(), 2, status  # jump outside
                    continue

@@ -137,7 +161,10 @@ class Code:
                    # return: (None, None)
                    else:
                        if len(self.contexts) == entry_level:
-                            assert len(ctx.stack) == 1 + initial_len
+                            if self.debug_mode:
+                                self._on_fragment_exit('return')
+                            delta_stack = len(ctx.stack) - initial_len
+                            assert delta_stack == +1, 'Stack change must be equal to +1! got %d' % delta_stack
                            return undefined, 1, ctx.stack.pop(
                            )  # return signal
                        return_value = ctx.stack.pop()
@@ -149,6 +176,8 @@ class Code:
                        continue
            # next instruction
            loc += 1
+        if self.debug_mode:
+            self._on_fragment_exit('internal error - unexpected end of tape, will crash')
        assert False, 'Remember to add NOP at the end!'

    def run(self, ctx, starting_loc=0):
@@ -156,7 +185,8 @@ class Code:
        self.current_ctx = ctx
        while loc < len(self.tape):
            # execute instruction
-            #print loc, self.tape[loc]
+            if self.debug_mode:
+                print(loc, self.tape[loc])
            status = self.tape[loc].eval(ctx)

            # check status for special actions
@@ -42,6 +42,7 @@ def executable_code(code_str, space, global_context=True):
    space.byte_generator.emit('LABEL', skip)
    space.byte_generator.emit('NOP')
    space.byte_generator.restore_state()
+
    space.byte_generator.exe.compile(
        start_loc=old_tape_len
    )  # dont read the code from the beginning, dont be stupid!
@@ -71,5 +72,5 @@ def _eval(this, args):


 def log(this, args):
-    print ' '.join(map(to_string, args))
+    print(' '.join(map(to_string, args)))
    return undefined
@@ -1,6 +1,6 @@
 from __future__ import unicode_literals
 # Type Conversions. to_type. All must return PyJs subclass instance
-from simplex import *
+from .simplex import *


 def to_primitive(self, hint=None):
@@ -73,14 +73,7 @@ def to_string(self):
    elif typ == 'Boolean':
        return 'true' if self else 'false'
    elif typ == 'Number':  # or self.Class=='Number':
-        if is_nan(self):
-            return 'NaN'
-        elif is_infinity(self):
-            sign = '-' if self < 0 else ''
-            return sign + 'Infinity'
-        elif int(self) == self:  # integer value!
-            return unicode(int(self))
-        return unicode(self)  # todo make it print exactly like node.js
+        return js_dtoa(self)
    else:  # object
        return to_string(to_primitive(self, 'String'))

@@ -1,29 +1,22 @@
 from __future__ import unicode_literals

-from base import Scope
-from func_utils import *
-from conversions import *
+from .base import Scope
+from .func_utils import *
+from .conversions import *
 import six
-from prototypes.jsboolean import BooleanPrototype
-from prototypes.jserror import ErrorPrototype
-from prototypes.jsfunction import FunctionPrototype
-from prototypes.jsnumber import NumberPrototype
-from prototypes.jsobject import ObjectPrototype
-from prototypes.jsregexp import RegExpPrototype
-from prototypes.jsstring import StringPrototype
-from prototypes.jsarray import ArrayPrototype
-import prototypes.jsjson as jsjson
-import prototypes.jsutils as jsutils
+from .prototypes.jsboolean import BooleanPrototype
+from .prototypes.jserror import ErrorPrototype
+from .prototypes.jsfunction import FunctionPrototype
+from .prototypes.jsnumber import NumberPrototype
+from .prototypes.jsobject import ObjectPrototype
+from .prototypes.jsregexp import RegExpPrototype
+from .prototypes.jsstring import StringPrototype
+from .prototypes.jsarray import ArrayPrototype
+from .prototypes import jsjson
+from .prototypes import jsutils
+
+from .constructors import jsnumber, jsstring, jsarray, jsboolean, jsregexp, jsmath, jsobject, jsfunction, jsconsole

-from constructors import jsnumber
-from constructors import jsstring
-from constructors import jsarray
-from constructors import jsboolean
-from constructors import jsregexp
-from constructors import jsmath
-from constructors import jsobject
-from constructors import jsfunction
-from constructors import jsconsole


 def fill_proto(proto, proto_class, space):
@@ -155,7 +148,10 @@ def fill_space(space, byte_generator):

        j = easy_func(creator, space)
        j.name = unicode(typ)
-        j.prototype = space.ERROR_TYPES[typ]
+
+        set_protected(j, 'prototype', space.ERROR_TYPES[typ])
+
+        set_non_enumerable(space.ERROR_TYPES[typ], 'constructor', j)

        def new_create(args, space):
            message = get_arg(args, 0)
@@ -178,6 +174,7 @@ def fill_space(space, byte_generator):
        setattr(space, err_type_name + u'Prototype', extra_err)
        error_constructors[err_type_name] = construct_constructor(
            err_type_name)
+
    assert space.TypeErrorPrototype is not None

    # RegExp
@@ -1,5 +1,5 @@
-from simplex import *
-from conversions import *
+from .simplex import *
+from .conversions import *

 import six
 if six.PY3:
@@ -1,5 +1,5 @@
-from operations import *
-from base import get_member, get_member_dot, PyJsFunction, Scope
+from .operations import *
+from .base import get_member, get_member_dot, PyJsFunction, Scope


 class OP_CODE(object):
@@ -1,6 +1,6 @@
 from __future__ import unicode_literals
-from simplex import *
-from conversions import *
+from .simplex import *
+from .conversions import *

 # ------------------------------------------------------------------------------
 # Unary operations
@@ -4,7 +4,7 @@ from __future__ import unicode_literals
 import re
 from ..conversions import *
 from ..func_utils import *
-from jsregexp import RegExpExec
+from .jsregexp import RegExpExec

 DIGS = set(u'0123456789')
 WHITE = u"\u0009\u000A\u000B\u000C\u000D\u0020\u00A0\u1680\u180E\u2000\u2001\u2002\u2003\u2004\u2005\u2006\u2007\u2008\u2009\u200A\u2028\u2029\u202F\u205F\u3000\uFEFF"
@@ -1,11 +1,9 @@
 import pyjsparser
-from space import Space
-import fill_space
-from byte_trans import ByteCodeGenerator
-from code import Code
-from simplex import MakeError
-import sys
-sys.setrecursionlimit(100000)
+from .space import Space
+from . import fill_space
+from .byte_trans import ByteCodeGenerator
+from .code import Code
+from .simplex import *


 pyjsparser.parser.ENABLE_JS2PY_ERRORS = lambda msg: MakeError(u'SyntaxError', unicode(msg))
@@ -16,8 +14,8 @@ def get_js_bytecode(js):
    a.emit(d)
    return  a.exe.tape
    
-def eval_js_vm(js):
-    a = ByteCodeGenerator(Code())
+def eval_js_vm(js, debug=False):
+    a = ByteCodeGenerator(Code(debug_mode=debug))
    s = Space()
    a.exe.space = s
    s.exe = a.exe
@@ -26,7 +24,10 @@ def eval_js_vm(js):

    a.emit(d)
    fill_space.fill_space(s, a)
-    # print a.exe.tape
+    if debug:
+        from pprint import pprint
+        pprint(a.exe.tape)
+        print()
    a.exe.compile()

    return a.exe.run(a.exe.space.GlobalObj)
@@ -1,6 +1,10 @@
 from __future__ import unicode_literals
 import six
-
+if six.PY3:
+    basestring = str
+    long = int
+    xrange = range
+    unicode = str

 #Undefined
 class PyJsUndefined(object):
@@ -75,7 +79,7 @@ def is_callable(self):


 def is_infinity(self):
-    return self == float('inf') or self == -float('inf')
+    return self == Infinity or self == -Infinity


 def is_nan(self):
@@ -114,7 +118,7 @@ class JsException(Exception):
                return self.mes.to_string().value
        else:
            if self.throw is not None:
-                from conversions import to_string
+                from .conversions import to_string
                return to_string(self.throw)
            else:
                return self.typ + ': ' + self.message
@@ -131,3 +135,26 @@ def value_from_js_exception(js_exception, space):
        return js_exception.throw
    else:
        return space.NewError(js_exception.typ, js_exception.message)
+
+
+def js_dtoa(number):
+    if is_nan(number):
+        return u'NaN'
+    elif is_infinity(number):
+        if number > 0:
+            return u'Infinity'
+        return u'-Infinity'
+    elif number == 0.:
+        return u'0'
+    elif abs(number) < 1e-6 or abs(number) >= 1e21:
+        frac, exponent = unicode(repr(float(number))).split('e')
+        # Remove leading zeros from the exponent.
+        exponent = int(exponent)
+        return frac + ('e' if exponent < 0 else 'e+') + unicode(exponent)
+    elif abs(number) < 1e-4:  # python starts to return exp notation while we still want the prec
+        frac, exponent = unicode(repr(float(number))).split('e-')
+        base = u'0.' + u'0' * (int(exponent) - 1) + frac.lstrip('-').replace('.', '')
+        return base if number > 0. else u'-' + base
+    elif isinstance(number, long) or number.is_integer():  # dont print .0
+        return unicode(int(number))
+    return unicode(repr(number))  # python representation should be equivalent.
@@ -1,5 +1,5 @@
-from base import *
-from simplex import *
+from .base import *
+from .simplex import *


 class Space(object):
@@ -1,3 +1,10 @@
+import six
+if six.PY3:
+    basestring = str
+    long = int
+    xrange = range
+    unicode = str
+
 def to_key(literal_or_identifier):
    ''' returns string representation of this object'''
    if literal_or_identifier['type'] == 'Identifier':
@@ -6,8 +6,6 @@ if six.PY3:
    xrange = range
    unicode = str

-# todo fix apply and bind
-

 class FunctionPrototype:
    def toString():
@@ -41,6 +39,7 @@ class FunctionPrototype:
        return this.call(obj, args)

    def bind(thisArg):
+        arguments_ = arguments
        target = this
        if not target.is_callable():
            raise this.MakeError(
@@ -48,5 +47,5 @@ class FunctionPrototype:
        if len(arguments) <= 1:
            args = ()
        else:
-            args = tuple([arguments[e] for e in xrange(1, len(arguments))])
+            args = tuple([arguments_[e] for e in xrange(1, len(arguments_))])
        return this.PyJsBoundFunction(target, thisArg, args)
@@ -345,7 +345,7 @@ def BlockStatement(type, body):
        body)  # never returns empty string! In the worst case returns pass\n


-def ExpressionStatement(type, expression, **ommit):
+def ExpressionStatement(type, expression):
    return trans(expression) + '\n'  # end expression space with new line


@@ -0,0 +1,297 @@
+#!/usr/bin/python
+# -*- coding: ascii -*-
+###########################################################################
+# pbkdf2 - PKCS#5 v2.0 Password-Based Key Derivation
+#
+# Copyright (C) 2007-2011 Dwayne C. Litzenberger <dlitz@dlitz.net>
+#
+# Permission is hereby granted, free of charge, to any person obtaining
+# a copy of this software and associated documentation files (the
+# "Software"), to deal in the Software without restriction, including
+# without limitation the rights to use, copy, modify, merge, publish,
+# distribute, sublicense, and/or sell copies of the Software, and to
+# permit persons to whom the Software is furnished to do so, subject to
+# the following conditions:
+#
+# The above copyright notice and this permission notice shall be
+# included in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+# LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+# WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+#
+# Country of origin: Canada
+#
+###########################################################################
+# Sample PBKDF2 usage:
+#   from Crypto.Cipher import AES
+#   from pbkdf2 import PBKDF2
+#   import os
+#
+#   salt = os.urandom(8)    # 64-bit salt
+#   key = PBKDF2("This passphrase is a secret.", salt).read(32) # 256-bit key
+#   iv = os.urandom(16)     # 128-bit IV
+#   cipher = AES.new(key, AES.MODE_CBC, iv)
+#     ...
+#
+# Sample crypt() usage:
+#   from pbkdf2 import crypt
+#   pwhash = crypt("secret")
+#   alleged_pw = raw_input("Enter password: ")
+#   if pwhash == crypt(alleged_pw, pwhash):
+#       print "Password good"
+#   else:
+#       print "Invalid password"
+#
+###########################################################################
+
+__version__ = "1.3"
+__all__ = ['PBKDF2', 'crypt']
+
+from struct import pack
+from random import randint
+import string
+import sys
+
+try:
+    # Use PyCrypto (if available).
+    from Crypto.Hash import HMAC, SHA as SHA1
+except ImportError:
+    # PyCrypto not available.  Use the Python standard library.
+    import hmac as HMAC
+    try:
+        from hashlib import sha1 as SHA1
+    except ImportError:
+        # hashlib not available.  Use the old sha module.
+        import sha as SHA1
+
+#
+# Python 2.1 thru 3.2 compatibility
+#
+
+if sys.version_info[0] == 2:
+    _0xffffffffL = long(1) << 32
+    def isunicode(s):
+        return isinstance(s, unicode)
+    def isbytes(s):
+        return isinstance(s, str)
+    def isinteger(n):
+        return isinstance(n, (int, long))
+    def b(s):
+        return s
+    def binxor(a, b):
+        return "".join([chr(ord(x) ^ ord(y)) for (x, y) in zip(a, b)])
+    def b64encode(data, chars="+/"):
+        tt = string.maketrans("+/", chars)
+        return data.encode('base64').replace("\n", "").translate(tt)
+    from binascii import b2a_hex
+else:
+    _0xffffffffL = 0xffffffff
+    def isunicode(s):
+        return isinstance(s, str)
+    def isbytes(s):
+        return isinstance(s, bytes)
+    def isinteger(n):
+        return isinstance(n, int)
+    def callable(obj):
+        return hasattr(obj, '__call__')
+    def b(s):
+       return s.encode("latin-1")
+    def binxor(a, b):
+        return bytes([x ^ y for (x, y) in zip(a, b)])
+    from base64 import b64encode as _b64encode
+    def b64encode(data, chars="+/"):
+        if isunicode(chars):
+            return _b64encode(data, chars.encode('utf-8')).decode('utf-8')
+        else:
+            return _b64encode(data, chars)
+    from binascii import b2a_hex as _b2a_hex
+    def b2a_hex(s):
+        return _b2a_hex(s).decode('us-ascii')
+    xrange = range
+
+class PBKDF2(object):
+    """PBKDF2.py : PKCS#5 v2.0 Password-Based Key Derivation
+
+    This implementation takes a passphrase and a salt (and optionally an
+    iteration count, a digest module, and a MAC module) and provides a
+    file-like object from which an arbitrarily-sized key can be read.
+
+    If the passphrase and/or salt are unicode objects, they are encoded as
+    UTF-8 before they are processed.
+
+    The idea behind PBKDF2 is to derive a cryptographic key from a
+    passphrase and a salt.
+
+    PBKDF2 may also be used as a strong salted password hash.  The
+    'crypt' function is provided for that purpose.
+
+    Remember: Keys generated using PBKDF2 are only as strong as the
+    passphrases they are derived from.
+    """
+
+    def __init__(self, passphrase, salt, iterations=1000,
+                 digestmodule=SHA1, macmodule=HMAC):
+        self.__macmodule = macmodule
+        self.__digestmodule = digestmodule
+        self._setup(passphrase, salt, iterations, self._pseudorandom)
+
+    def _pseudorandom(self, key, msg):
+        """Pseudorandom function.  e.g. HMAC-SHA1"""
+        return self.__macmodule.new(key=key, msg=msg,
+            digestmod=self.__digestmodule).digest()
+
+    def read(self, bytes):
+        """Read the specified number of key bytes."""
+        if self.closed:
+            raise ValueError("file-like object is closed")
+
+        size = len(self.__buf)
+        blocks = [self.__buf]
+        i = self.__blockNum
+        while size < bytes:
+            i += 1
+            if i > _0xffffffffL or i < 1:
+                # We could return "" here, but
+                raise OverflowError("derived key too long")
+            block = self.__f(i)
+            blocks.append(block)
+            size += len(block)
+        buf = b("").join(blocks)
+        retval = buf[:bytes]
+        self.__buf = buf[bytes:]
+        self.__blockNum = i
+        return retval
+
+    def __f(self, i):
+        # i must fit within 32 bits
+        assert 1 <= i <= _0xffffffffL
+        U = self.__prf(self.__passphrase, self.__salt + pack("!L", i))
+        result = U
+        for j in xrange(2, 1+self.__iterations):
+            U = self.__prf(self.__passphrase, U)
+            result = binxor(result, U)
+        return result
+
+    def hexread(self, octets):
+        """Read the specified number of octets. Return them as hexadecimal.
+
+        Note that len(obj.hexread(n)) == 2*n.
+        """
+        return b2a_hex(self.read(octets))
+
+    def _setup(self, passphrase, salt, iterations, prf):
+        # Sanity checks:
+
+        # passphrase and salt must be str or unicode (in the latter
+        # case, we convert to UTF-8)
+        if isunicode(passphrase):
+            passphrase = passphrase.encode("UTF-8")
+        elif not isbytes(passphrase):
+            raise TypeError("passphrase must be str or unicode")
+        if isunicode(salt):
+            salt = salt.encode("UTF-8")
+        elif not isbytes(salt):
+            raise TypeError("salt must be str or unicode")
+
+        # iterations must be an integer >= 1
+        if not isinteger(iterations):
+            raise TypeError("iterations must be an integer")
+        if iterations < 1:
+            raise ValueError("iterations must be at least 1")
+
+        # prf must be callable
+        if not callable(prf):
+            raise TypeError("prf must be callable")
+
+        self.__passphrase = passphrase
+        self.__salt = salt
+        self.__iterations = iterations
+        self.__prf = prf
+        self.__blockNum = 0
+        self.__buf = b("")
+        self.closed = False
+
+    def close(self):
+        """Close the stream."""
+        if not self.closed:
+            del self.__passphrase
+            del self.__salt
+            del self.__iterations
+            del self.__prf
+            del self.__blockNum
+            del self.__buf
+            self.closed = True
+
+def crypt(word, salt=None, iterations=None):
+    """PBKDF2-based unix crypt(3) replacement.
+
+    The number of iterations specified in the salt overrides the 'iterations'
+    parameter.
+
+    The effective hash length is 192 bits.
+    """
+
+    # Generate a (pseudo-)random salt if the user hasn't provided one.
+    if salt is None:
+        salt = _makesalt()
+
+    # salt must be a string or the us-ascii subset of unicode
+    if isunicode(salt):
+        salt = salt.encode('us-ascii').decode('us-ascii')
+    elif isbytes(salt):
+        salt = salt.decode('us-ascii')
+    else:
+        raise TypeError("salt must be a string")
+
+    # word must be a string or unicode (in the latter case, we convert to UTF-8)
+    if isunicode(word):
+        word = word.encode("UTF-8")
+    elif not isbytes(word):
+        raise TypeError("word must be a string or unicode")
+
+    # Try to extract the real salt and iteration count from the salt
+    if salt.startswith("$p5k2$"):
+        (iterations, salt, dummy) = salt.split("$")[2:5]
+        if iterations == "":
+            iterations = 400
+        else:
+            converted = int(iterations, 16)
+            if iterations != "%x" % converted:  # lowercase hex, minimum digits
+                raise ValueError("Invalid salt")
+            iterations = converted
+            if not (iterations >= 1):
+                raise ValueError("Invalid salt")
+
+    # Make sure the salt matches the allowed character set
+    allowed = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789./"
+    for ch in salt:
+        if ch not in allowed:
+            raise ValueError("Illegal character %r in salt" % (ch,))
+
+    if iterations is None or iterations == 400:
+        iterations = 400
+        salt = "$p5k2$$" + salt
+    else:
+        salt = "$p5k2$%x$%s" % (iterations, salt)
+    rawhash = PBKDF2(word, salt, iterations).read(24)
+    return salt + "$" + b64encode(rawhash, "./")
+
+# Add crypt as a static method of the PBKDF2 class
+# This makes it easier to do "from PBKDF2 import PBKDF2" and still use
+# crypt.
+PBKDF2.crypt = staticmethod(crypt)
+
+def _makesalt():
+    """Return a 48-bit pseudorandom salt for crypt().
+
+    This function is not suitable for generating cryptographic secrets.
+    """
+    binarysalt = b("").join([pack("@H", randint(0, 0xffff)) for i in range(3)])
+    return b64encode(binarysalt, "./")
+
+# vim:set ts=4 sw=4 sts=4 expandtab:
@@ -43,6 +43,8 @@ python -c "import logging; logging.basicConfig(level=logging.DEBUG); logging.get
 # subscenter:list
 python -c "import logging; logging.basicConfig(level=logging.DEBUG); logging.getLogger('rebulk').setLevel(logging.WARNING); import subliminal_patch, subliminal; subliminal.region.configure('dogpile.cache.memory'); from subliminal_patch.core import SZProviderPool; from babelfish import Language; from subliminal.core import scan_video; print SZProviderPool(providers=['subscenter'], )['subscenter'].list_subtitles(scan_video('FULL_PATH'), languages=[Language('heb')])"

+# subscene:list
+python -c "import logging; logging.basicConfig(level=logging.DEBUG); logging.getLogger('rebulk').setLevel(logging.WARNING); import subliminal_patch, subliminal; subliminal.region.configure('dogpile.cache.memory'); from subliminal_patch.core import SZProviderPool; from subzero.language import Language; from subzero.video import parse_video; SZProviderPool(providers=['subscene'], provider_configs={'subscene': {'username': 'USERNAME', 'password': 'PASSWORD'}})['subscene'].list_subtitles(parse_video('FILENAME', {}, {'type': 'episode'}, dry_run=True), languages=[Language('eng')])"

 # refining
 python -c "import logging; logging.basicConfig(level=logging.DEBUG); logging.getLogger('rebulk').setLevel(logging.WARNING); import os; os.environ['U1pfT01EQl9LRVk'] = '789CF30DAC2C8B0AF433F5C9AD34290A712DF30D7135F12D0FB3E502006FDE081E'; import subliminal_patch, subliminal; subliminal.region.configure('dogpile.cache.memory'); from subzero.video import parse_video, refine_video; video = parse_video('FILE_NAME', {'type': 'episode'}, dry_run=True); print refine_video(video)"
@@ -0,0 +1,53 @@
+# The MIT License (MIT)
+#
+# Copyright (c) 2014 Richard Moore
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+# THE SOFTWARE.
+
+# This is a pure-Python implementation of the AES algorithm and AES common
+# modes of operation.
+
+# See: https://en.wikipedia.org/wiki/Advanced_Encryption_Standard
+# See: https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation
+
+
+# Supported key sizes:
+#   128-bit
+#   192-bit
+#   256-bit
+
+
+# Supported modes of operation:
+#   ECB - Electronic Codebook
+#   CBC - Cipher-Block Chaining
+#   CFB - Cipher Feedback
+#   OFB - Output Feedback
+#   CTR - Counter
+
+# See the README.md for API details and general information.
+
+# Also useful, PyCrypto, a crypto library implemented in C with Python bindings:
+# https://www.dlitz.net/software/pycrypto/
+
+
+VERSION = [1, 3, 0]
+
+from .aes import AES, AESModeOfOperationCTR, AESModeOfOperationCBC, AESModeOfOperationCFB, AESModeOfOperationECB, AESModeOfOperationOFB, AESModesOfOperation, Counter
+from .blockfeeder import decrypt_stream, Decrypter, encrypt_stream, Encrypter
+from .blockfeeder import PADDING_NONE, PADDING_DEFAULT
@@ -0,0 +1,589 @@
+# The MIT License (MIT)
+#
+# Copyright (c) 2014 Richard Moore
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+# THE SOFTWARE.
+
+# This is a pure-Python implementation of the AES algorithm and AES common
+# modes of operation.
+
+# See: https://en.wikipedia.org/wiki/Advanced_Encryption_Standard
+
+# Honestly, the best description of the modes of operations are the wonderful
+# diagrams on Wikipedia. They explain in moments what my words could never
+# achieve. Hence the inline documentation here is sparer than I'd prefer.
+# See: https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation
+
+# Also useful, PyCrypto, a crypto library implemented in C with Python bindings:
+# https://www.dlitz.net/software/pycrypto/
+
+
+# Supported key sizes:
+#   128-bit
+#   192-bit
+#   256-bit
+
+
+# Supported modes of operation:
+#   ECB - Electronic Codebook
+#   CBC - Cipher-Block Chaining
+#   CFB - Cipher Feedback
+#   OFB - Output Feedback
+#   CTR - Counter
+
+
+# See the README.md for API details and general information.
+
+
+import copy
+import struct
+
+__all__ = ["AES", "AESModeOfOperationCTR", "AESModeOfOperationCBC", "AESModeOfOperationCFB",
+           "AESModeOfOperationECB", "AESModeOfOperationOFB", "AESModesOfOperation", "Counter"]
+
+
+def _compact_word(word):
+    return (word[0] << 24) | (word[1] << 16) | (word[2] << 8) | word[3]
+
+def _string_to_bytes(text):
+    return list(ord(c) for c in text)
+
+def _bytes_to_string(binary):
+    return "".join(chr(b) for b in binary)
+
+def _concat_list(a, b):
+    return a + b
+
+
+# Python 3 compatibility
+try:
+    xrange
+except Exception:
+    xrange = range
+
+    # Python 3 supports bytes, which is already an array of integers
+    def _string_to_bytes(text):
+        if isinstance(text, bytes):
+            return text
+        return [ord(c) for c in text]
+
+    # In Python 3, we return bytes
+    def _bytes_to_string(binary):
+        return bytes(binary)
+
+    # Python 3 cannot concatenate a list onto a bytes, so we bytes-ify it first
+    def _concat_list(a, b):
+        return a + bytes(b)
+
+
+# Based *largely* on the Rijndael implementation
+# See: http://csrc.nist.gov/publications/fips/fips197/fips-197.pdf
+class AES(object):
+    '''Encapsulates the AES block cipher.
+
+    You generally should not need this. Use the AESModeOfOperation classes
+    below instead.'''
+
+    # Number of rounds by keysize
+    number_of_rounds = {16: 10, 24: 12, 32: 14}
+
+    # Round constant words
+    rcon = [ 0x01, 0x02, 0x04, 0x08, 0x10, 0x20, 0x40, 0x80, 0x1b, 0x36, 0x6c, 0xd8, 0xab, 0x4d, 0x9a, 0x2f, 0x5e, 0xbc, 0x63, 0xc6, 0x97, 0x35, 0x6a, 0xd4, 0xb3, 0x7d, 0xfa, 0xef, 0xc5, 0x91 ]
+
+    # S-box and Inverse S-box (S is for Substitution)
+    S = [ 0x63, 0x7c, 0x77, 0x7b, 0xf2, 0x6b, 0x6f, 0xc5, 0x30, 0x01, 0x67, 0x2b, 0xfe, 0xd7, 0xab, 0x76, 0xca, 0x82, 0xc9, 0x7d, 0xfa, 0x59, 0x47, 0xf0, 0xad, 0xd4, 0xa2, 0xaf, 0x9c, 0xa4, 0x72, 0xc0, 0xb7, 0xfd, 0x93, 0x26, 0x36, 0x3f, 0xf7, 0xcc, 0x34, 0xa5, 0xe5, 0xf1, 0x71, 0xd8, 0x31, 0x15, 0x04, 0xc7, 0x23, 0xc3, 0x18, 0x96, 0x05, 0x9a, 0x07, 0x12, 0x80, 0xe2, 0xeb, 0x27, 0xb2, 0x75, 0x09, 0x83, 0x2c, 0x1a, 0x1b, 0x6e, 0x5a, 0xa0, 0x52, 0x3b, 0xd6, 0xb3, 0x29, 0xe3, 0x2f, 0x84, 0x53, 0xd1, 0x00, 0xed, 0x20, 0xfc, 0xb1, 0x5b, 0x6a, 0xcb, 0xbe, 0x39, 0x4a, 0x4c, 0x58, 0xcf, 0xd0, 0xef, 0xaa, 0xfb, 0x43, 0x4d, 0x33, 0x85, 0x45, 0xf9, 0x02, 0x7f, 0x50, 0x3c, 0x9f, 0xa8, 0x51, 0xa3, 0x40, 0x8f, 0x92, 0x9d, 0x38, 0xf5, 0xbc, 0xb6, 0xda, 0x21, 0x10, 0xff, 0xf3, 0xd2, 0xcd, 0x0c, 0x13, 0xec, 0x5f, 0x97, 0x44, 0x17, 0xc4, 0xa7, 0x7e, 0x3d, 0x64, 0x5d, 0x19, 0x73, 0x60, 0x81, 0x4f, 0xdc, 0x22, 0x2a, 0x90, 0x88, 0x46, 0xee, 0xb8, 0x14, 0xde, 0x5e, 0x0b, 0xdb, 0xe0, 0x32, 0x3a, 0x0a, 0x49, 0x06, 0x24, 0x5c, 0xc2, 0xd3, 0xac, 0x62, 0x91, 0x95, 0xe4, 0x79, 0xe7, 0xc8, 0x37, 0x6d, 0x8d, 0xd5, 0x4e, 0xa9, 0x6c, 0x56, 0xf4, 0xea, 0x65, 0x7a, 0xae, 0x08, 0xba, 0x78, 0x25, 0x2e, 0x1c, 0xa6, 0xb4, 0xc6, 0xe8, 0xdd, 0x74, 0x1f, 0x4b, 0xbd, 0x8b, 0x8a, 0x70, 0x3e, 0xb5, 0x66, 0x48, 0x03, 0xf6, 0x0e, 0x61, 0x35, 0x57, 0xb9, 0x86, 0xc1, 0x1d, 0x9e, 0xe1, 0xf8, 0x98, 0x11, 0x69, 0xd9, 0x8e, 0x94, 0x9b, 0x1e, 0x87, 0xe9, 0xce, 0x55, 0x28, 0xdf, 0x8c, 0xa1, 0x89, 0x0d, 0xbf, 0xe6, 0x42, 0x68, 0x41, 0x99, 0x2d, 0x0f, 0xb0, 0x54, 0xbb, 0x16 ]
+    Si =[ 0x52, 0x09, 0x6a, 0xd5, 0x30, 0x36, 0xa5, 0x38, 0xbf, 0x40, 0xa3, 0x9e, 0x81, 0xf3, 0xd7, 0xfb, 0x7c, 0xe3, 0x39, 0x82, 0x9b, 0x2f, 0xff, 0x87, 0x34, 0x8e, 0x43, 0x44, 0xc4, 0xde, 0xe9, 0xcb, 0x54, 0x7b, 0x94, 0x32, 0xa6, 0xc2, 0x23, 0x3d, 0xee, 0x4c, 0x95, 0x0b, 0x42, 0xfa, 0xc3, 0x4e, 0x08, 0x2e, 0xa1, 0x66, 0x28, 0xd9, 0x24, 0xb2, 0x76, 0x5b, 0xa2, 0x49, 0x6d, 0x8b, 0xd1, 0x25, 0x72, 0xf8, 0xf6, 0x64, 0x86, 0x68, 0x98, 0x16, 0xd4, 0xa4, 0x5c, 0xcc, 0x5d, 0x65, 0xb6, 0x92, 0x6c, 0x70, 0x48, 0x50, 0xfd, 0xed, 0xb9, 0xda, 0x5e, 0x15, 0x46, 0x57, 0xa7, 0x8d, 0x9d, 0x84, 0x90, 0xd8, 0xab, 0x00, 0x8c, 0xbc, 0xd3, 0x0a, 0xf7, 0xe4, 0x58, 0x05, 0xb8, 0xb3, 0x45, 0x06, 0xd0, 0x2c, 0x1e, 0x8f, 0xca, 0x3f, 0x0f, 0x02, 0xc1, 0xaf, 0xbd, 0x03, 0x01, 0x13, 0x8a, 0x6b, 0x3a, 0x91, 0x11, 0x41, 0x4f, 0x67, 0xdc, 0xea, 0x97, 0xf2, 0xcf, 0xce, 0xf0, 0xb4, 0xe6, 0x73, 0x96, 0xac, 0x74, 0x22, 0xe7, 0xad, 0x35, 0x85, 0xe2, 0xf9, 0x37, 0xe8, 0x1c, 0x75, 0xdf, 0x6e, 0x47, 0xf1, 0x1a, 0x71, 0x1d, 0x29, 0xc5, 0x89, 0x6f, 0xb7, 0x62, 0x0e, 0xaa, 0x18, 0xbe, 0x1b, 0xfc, 0x56, 0x3e, 0x4b, 0xc6, 0xd2, 0x79, 0x20, 0x9a, 0xdb, 0xc0, 0xfe, 0x78, 0xcd, 0x5a, 0xf4, 0x1f, 0xdd, 0xa8, 0x33, 0x88, 0x07, 0xc7, 0x31, 0xb1, 0x12, 0x10, 0x59, 0x27, 0x80, 0xec, 0x5f, 0x60, 0x51, 0x7f, 0xa9, 0x19, 0xb5, 0x4a, 0x0d, 0x2d, 0xe5, 0x7a, 0x9f, 0x93, 0xc9, 0x9c, 0xef, 0xa0, 0xe0, 0x3b, 0x4d, 0xae, 0x2a, 0xf5, 0xb0, 0xc8, 0xeb, 0xbb, 0x3c, 0x83, 0x53, 0x99, 0x61, 0x17, 0x2b, 0x04, 0x7e, 0xba, 0x77, 0xd6, 0x26, 0xe1, 0x69, 0x14, 0x63, 0x55, 0x21, 0x0c, 0x7d ] 
+
+    # Transformations for encryption
+    T1 = [ 0xc66363a5, 0xf87c7c84, 0xee777799, 0xf67b7b8d, 0xfff2f20d, 0xd66b6bbd, 0xde6f6fb1, 0x91c5c554, 0x60303050, 0x02010103, 0xce6767a9, 0x562b2b7d, 0xe7fefe19, 0xb5d7d762, 0x4dababe6, 0xec76769a, 0x8fcaca45, 0x1f82829d, 0x89c9c940, 0xfa7d7d87, 0xeffafa15, 0xb25959eb, 0x8e4747c9, 0xfbf0f00b, 0x41adadec, 0xb3d4d467, 0x5fa2a2fd, 0x45afafea, 0x239c9cbf, 0x53a4a4f7, 0xe4727296, 0x9bc0c05b, 0x75b7b7c2, 0xe1fdfd1c, 0x3d9393ae, 0x4c26266a, 0x6c36365a, 0x7e3f3f41, 0xf5f7f702, 0x83cccc4f, 0x6834345c, 0x51a5a5f4, 0xd1e5e534, 0xf9f1f108, 0xe2717193, 0xabd8d873, 0x62313153, 0x2a15153f, 0x0804040c, 0x95c7c752, 0x46232365, 0x9dc3c35e, 0x30181828, 0x379696a1, 0x0a05050f, 0x2f9a9ab5, 0x0e070709, 0x24121236, 0x1b80809b, 0xdfe2e23d, 0xcdebeb26, 0x4e272769, 0x7fb2b2cd, 0xea75759f, 0x1209091b, 0x1d83839e, 0x582c2c74, 0x341a1a2e, 0x361b1b2d, 0xdc6e6eb2, 0xb45a5aee, 0x5ba0a0fb, 0xa45252f6, 0x763b3b4d, 0xb7d6d661, 0x7db3b3ce, 0x5229297b, 0xdde3e33e, 0x5e2f2f71, 0x13848497, 0xa65353f5, 0xb9d1d168, 0x00000000, 0xc1eded2c, 0x40202060, 0xe3fcfc1f, 0x79b1b1c8, 0xb65b5bed, 0xd46a6abe, 0x8dcbcb46, 0x67bebed9, 0x7239394b, 0x944a4ade, 0x984c4cd4, 0xb05858e8, 0x85cfcf4a, 0xbbd0d06b, 0xc5efef2a, 0x4faaaae5, 0xedfbfb16, 0x864343c5, 0x9a4d4dd7, 0x66333355, 0x11858594, 0x8a4545cf, 0xe9f9f910, 0x04020206, 0xfe7f7f81, 0xa05050f0, 0x783c3c44, 0x259f9fba, 0x4ba8a8e3, 0xa25151f3, 0x5da3a3fe, 0x804040c0, 0x058f8f8a, 0x3f9292ad, 0x219d9dbc, 0x70383848, 0xf1f5f504, 0x63bcbcdf, 0x77b6b6c1, 0xafdada75, 0x42212163, 0x20101030, 0xe5ffff1a, 0xfdf3f30e, 0xbfd2d26d, 0x81cdcd4c, 0x180c0c14, 0x26131335, 0xc3ecec2f, 0xbe5f5fe1, 0x359797a2, 0x884444cc, 0x2e171739, 0x93c4c457, 0x55a7a7f2, 0xfc7e7e82, 0x7a3d3d47, 0xc86464ac, 0xba5d5de7, 0x3219192b, 0xe6737395, 0xc06060a0, 0x19818198, 0x9e4f4fd1, 0xa3dcdc7f, 0x44222266, 0x542a2a7e, 0x3b9090ab, 0x0b888883, 0x8c4646ca, 0xc7eeee29, 0x6bb8b8d3, 0x2814143c, 0xa7dede79, 0xbc5e5ee2, 0x160b0b1d, 0xaddbdb76, 0xdbe0e03b, 0x64323256, 0x743a3a4e, 0x140a0a1e, 0x924949db, 0x0c06060a, 0x4824246c, 0xb85c5ce4, 0x9fc2c25d, 0xbdd3d36e, 0x43acacef, 0xc46262a6, 0x399191a8, 0x319595a4, 0xd3e4e437, 0xf279798b, 0xd5e7e732, 0x8bc8c843, 0x6e373759, 0xda6d6db7, 0x018d8d8c, 0xb1d5d564, 0x9c4e4ed2, 0x49a9a9e0, 0xd86c6cb4, 0xac5656fa, 0xf3f4f407, 0xcfeaea25, 0xca6565af, 0xf47a7a8e, 0x47aeaee9, 0x10080818, 0x6fbabad5, 0xf0787888, 0x4a25256f, 0x5c2e2e72, 0x381c1c24, 0x57a6a6f1, 0x73b4b4c7, 0x97c6c651, 0xcbe8e823, 0xa1dddd7c, 0xe874749c, 0x3e1f1f21, 0x964b4bdd, 0x61bdbddc, 0x0d8b8b86, 0x0f8a8a85, 0xe0707090, 0x7c3e3e42, 0x71b5b5c4, 0xcc6666aa, 0x904848d8, 0x06030305, 0xf7f6f601, 0x1c0e0e12, 0xc26161a3, 0x6a35355f, 0xae5757f9, 0x69b9b9d0, 0x17868691, 0x99c1c158, 0x3a1d1d27, 0x279e9eb9, 0xd9e1e138, 0xebf8f813, 0x2b9898b3, 0x22111133, 0xd26969bb, 0xa9d9d970, 0x078e8e89, 0x339494a7, 0x2d9b9bb6, 0x3c1e1e22, 0x15878792, 0xc9e9e920, 0x87cece49, 0xaa5555ff, 0x50282878, 0xa5dfdf7a, 0x038c8c8f, 0x59a1a1f8, 0x09898980, 0x1a0d0d17, 0x65bfbfda, 0xd7e6e631, 0x844242c6, 0xd06868b8, 0x824141c3, 0x299999b0, 0x5a2d2d77, 0x1e0f0f11, 0x7bb0b0cb, 0xa85454fc, 0x6dbbbbd6, 0x2c16163a ]
+    T2 = [ 0xa5c66363, 0x84f87c7c, 0x99ee7777, 0x8df67b7b, 0x0dfff2f2, 0xbdd66b6b, 0xb1de6f6f, 0x5491c5c5, 0x50603030, 0x03020101, 0xa9ce6767, 0x7d562b2b, 0x19e7fefe, 0x62b5d7d7, 0xe64dabab, 0x9aec7676, 0x458fcaca, 0x9d1f8282, 0x4089c9c9, 0x87fa7d7d, 0x15effafa, 0xebb25959, 0xc98e4747, 0x0bfbf0f0, 0xec41adad, 0x67b3d4d4, 0xfd5fa2a2, 0xea45afaf, 0xbf239c9c, 0xf753a4a4, 0x96e47272, 0x5b9bc0c0, 0xc275b7b7, 0x1ce1fdfd, 0xae3d9393, 0x6a4c2626, 0x5a6c3636, 0x417e3f3f, 0x02f5f7f7, 0x4f83cccc, 0x5c683434, 0xf451a5a5, 0x34d1e5e5, 0x08f9f1f1, 0x93e27171, 0x73abd8d8, 0x53623131, 0x3f2a1515, 0x0c080404, 0x5295c7c7, 0x65462323, 0x5e9dc3c3, 0x28301818, 0xa1379696, 0x0f0a0505, 0xb52f9a9a, 0x090e0707, 0x36241212, 0x9b1b8080, 0x3ddfe2e2, 0x26cdebeb, 0x694e2727, 0xcd7fb2b2, 0x9fea7575, 0x1b120909, 0x9e1d8383, 0x74582c2c, 0x2e341a1a, 0x2d361b1b, 0xb2dc6e6e, 0xeeb45a5a, 0xfb5ba0a0, 0xf6a45252, 0x4d763b3b, 0x61b7d6d6, 0xce7db3b3, 0x7b522929, 0x3edde3e3, 0x715e2f2f, 0x97138484, 0xf5a65353, 0x68b9d1d1, 0x00000000, 0x2cc1eded, 0x60402020, 0x1fe3fcfc, 0xc879b1b1, 0xedb65b5b, 0xbed46a6a, 0x468dcbcb, 0xd967bebe, 0x4b723939, 0xde944a4a, 0xd4984c4c, 0xe8b05858, 0x4a85cfcf, 0x6bbbd0d0, 0x2ac5efef, 0xe54faaaa, 0x16edfbfb, 0xc5864343, 0xd79a4d4d, 0x55663333, 0x94118585, 0xcf8a4545, 0x10e9f9f9, 0x06040202, 0x81fe7f7f, 0xf0a05050, 0x44783c3c, 0xba259f9f, 0xe34ba8a8, 0xf3a25151, 0xfe5da3a3, 0xc0804040, 0x8a058f8f, 0xad3f9292, 0xbc219d9d, 0x48703838, 0x04f1f5f5, 0xdf63bcbc, 0xc177b6b6, 0x75afdada, 0x63422121, 0x30201010, 0x1ae5ffff, 0x0efdf3f3, 0x6dbfd2d2, 0x4c81cdcd, 0x14180c0c, 0x35261313, 0x2fc3ecec, 0xe1be5f5f, 0xa2359797, 0xcc884444, 0x392e1717, 0x5793c4c4, 0xf255a7a7, 0x82fc7e7e, 0x477a3d3d, 0xacc86464, 0xe7ba5d5d, 0x2b321919, 0x95e67373, 0xa0c06060, 0x98198181, 0xd19e4f4f, 0x7fa3dcdc, 0x66442222, 0x7e542a2a, 0xab3b9090, 0x830b8888, 0xca8c4646, 0x29c7eeee, 0xd36bb8b8, 0x3c281414, 0x79a7dede, 0xe2bc5e5e, 0x1d160b0b, 0x76addbdb, 0x3bdbe0e0, 0x56643232, 0x4e743a3a, 0x1e140a0a, 0xdb924949, 0x0a0c0606, 0x6c482424, 0xe4b85c5c, 0x5d9fc2c2, 0x6ebdd3d3, 0xef43acac, 0xa6c46262, 0xa8399191, 0xa4319595, 0x37d3e4e4, 0x8bf27979, 0x32d5e7e7, 0x438bc8c8, 0x596e3737, 0xb7da6d6d, 0x8c018d8d, 0x64b1d5d5, 0xd29c4e4e, 0xe049a9a9, 0xb4d86c6c, 0xfaac5656, 0x07f3f4f4, 0x25cfeaea, 0xafca6565, 0x8ef47a7a, 0xe947aeae, 0x18100808, 0xd56fbaba, 0x88f07878, 0x6f4a2525, 0x725c2e2e, 0x24381c1c, 0xf157a6a6, 0xc773b4b4, 0x5197c6c6, 0x23cbe8e8, 0x7ca1dddd, 0x9ce87474, 0x213e1f1f, 0xdd964b4b, 0xdc61bdbd, 0x860d8b8b, 0x850f8a8a, 0x90e07070, 0x427c3e3e, 0xc471b5b5, 0xaacc6666, 0xd8904848, 0x05060303, 0x01f7f6f6, 0x121c0e0e, 0xa3c26161, 0x5f6a3535, 0xf9ae5757, 0xd069b9b9, 0x91178686, 0x5899c1c1, 0x273a1d1d, 0xb9279e9e, 0x38d9e1e1, 0x13ebf8f8, 0xb32b9898, 0x33221111, 0xbbd26969, 0x70a9d9d9, 0x89078e8e, 0xa7339494, 0xb62d9b9b, 0x223c1e1e, 0x92158787, 0x20c9e9e9, 0x4987cece, 0xffaa5555, 0x78502828, 0x7aa5dfdf, 0x8f038c8c, 0xf859a1a1, 0x80098989, 0x171a0d0d, 0xda65bfbf, 0x31d7e6e6, 0xc6844242, 0xb8d06868, 0xc3824141, 0xb0299999, 0x775a2d2d, 0x111e0f0f, 0xcb7bb0b0, 0xfca85454, 0xd66dbbbb, 0x3a2c1616 ]
+    T3 = [ 0x63a5c663, 0x7c84f87c, 0x7799ee77, 0x7b8df67b, 0xf20dfff2, 0x6bbdd66b, 0x6fb1de6f, 0xc55491c5, 0x30506030, 0x01030201, 0x67a9ce67, 0x2b7d562b, 0xfe19e7fe, 0xd762b5d7, 0xabe64dab, 0x769aec76, 0xca458fca, 0x829d1f82, 0xc94089c9, 0x7d87fa7d, 0xfa15effa, 0x59ebb259, 0x47c98e47, 0xf00bfbf0, 0xadec41ad, 0xd467b3d4, 0xa2fd5fa2, 0xafea45af, 0x9cbf239c, 0xa4f753a4, 0x7296e472, 0xc05b9bc0, 0xb7c275b7, 0xfd1ce1fd, 0x93ae3d93, 0x266a4c26, 0x365a6c36, 0x3f417e3f, 0xf702f5f7, 0xcc4f83cc, 0x345c6834, 0xa5f451a5, 0xe534d1e5, 0xf108f9f1, 0x7193e271, 0xd873abd8, 0x31536231, 0x153f2a15, 0x040c0804, 0xc75295c7, 0x23654623, 0xc35e9dc3, 0x18283018, 0x96a13796, 0x050f0a05, 0x9ab52f9a, 0x07090e07, 0x12362412, 0x809b1b80, 0xe23ddfe2, 0xeb26cdeb, 0x27694e27, 0xb2cd7fb2, 0x759fea75, 0x091b1209, 0x839e1d83, 0x2c74582c, 0x1a2e341a, 0x1b2d361b, 0x6eb2dc6e, 0x5aeeb45a, 0xa0fb5ba0, 0x52f6a452, 0x3b4d763b, 0xd661b7d6, 0xb3ce7db3, 0x297b5229, 0xe33edde3, 0x2f715e2f, 0x84971384, 0x53f5a653, 0xd168b9d1, 0x00000000, 0xed2cc1ed, 0x20604020, 0xfc1fe3fc, 0xb1c879b1, 0x5bedb65b, 0x6abed46a, 0xcb468dcb, 0xbed967be, 0x394b7239, 0x4ade944a, 0x4cd4984c, 0x58e8b058, 0xcf4a85cf, 0xd06bbbd0, 0xef2ac5ef, 0xaae54faa, 0xfb16edfb, 0x43c58643, 0x4dd79a4d, 0x33556633, 0x85941185, 0x45cf8a45, 0xf910e9f9, 0x02060402, 0x7f81fe7f, 0x50f0a050, 0x3c44783c, 0x9fba259f, 0xa8e34ba8, 0x51f3a251, 0xa3fe5da3, 0x40c08040, 0x8f8a058f, 0x92ad3f92, 0x9dbc219d, 0x38487038, 0xf504f1f5, 0xbcdf63bc, 0xb6c177b6, 0xda75afda, 0x21634221, 0x10302010, 0xff1ae5ff, 0xf30efdf3, 0xd26dbfd2, 0xcd4c81cd, 0x0c14180c, 0x13352613, 0xec2fc3ec, 0x5fe1be5f, 0x97a23597, 0x44cc8844, 0x17392e17, 0xc45793c4, 0xa7f255a7, 0x7e82fc7e, 0x3d477a3d, 0x64acc864, 0x5de7ba5d, 0x192b3219, 0x7395e673, 0x60a0c060, 0x81981981, 0x4fd19e4f, 0xdc7fa3dc, 0x22664422, 0x2a7e542a, 0x90ab3b90, 0x88830b88, 0x46ca8c46, 0xee29c7ee, 0xb8d36bb8, 0x143c2814, 0xde79a7de, 0x5ee2bc5e, 0x0b1d160b, 0xdb76addb, 0xe03bdbe0, 0x32566432, 0x3a4e743a, 0x0a1e140a, 0x49db9249, 0x060a0c06, 0x246c4824, 0x5ce4b85c, 0xc25d9fc2, 0xd36ebdd3, 0xacef43ac, 0x62a6c462, 0x91a83991, 0x95a43195, 0xe437d3e4, 0x798bf279, 0xe732d5e7, 0xc8438bc8, 0x37596e37, 0x6db7da6d, 0x8d8c018d, 0xd564b1d5, 0x4ed29c4e, 0xa9e049a9, 0x6cb4d86c, 0x56faac56, 0xf407f3f4, 0xea25cfea, 0x65afca65, 0x7a8ef47a, 0xaee947ae, 0x08181008, 0xbad56fba, 0x7888f078, 0x256f4a25, 0x2e725c2e, 0x1c24381c, 0xa6f157a6, 0xb4c773b4, 0xc65197c6, 0xe823cbe8, 0xdd7ca1dd, 0x749ce874, 0x1f213e1f, 0x4bdd964b, 0xbddc61bd, 0x8b860d8b, 0x8a850f8a, 0x7090e070, 0x3e427c3e, 0xb5c471b5, 0x66aacc66, 0x48d89048, 0x03050603, 0xf601f7f6, 0x0e121c0e, 0x61a3c261, 0x355f6a35, 0x57f9ae57, 0xb9d069b9, 0x86911786, 0xc15899c1, 0x1d273a1d, 0x9eb9279e, 0xe138d9e1, 0xf813ebf8, 0x98b32b98, 0x11332211, 0x69bbd269, 0xd970a9d9, 0x8e89078e, 0x94a73394, 0x9bb62d9b, 0x1e223c1e, 0x87921587, 0xe920c9e9, 0xce4987ce, 0x55ffaa55, 0x28785028, 0xdf7aa5df, 0x8c8f038c, 0xa1f859a1, 0x89800989, 0x0d171a0d, 0xbfda65bf, 0xe631d7e6, 0x42c68442, 0x68b8d068, 0x41c38241, 0x99b02999, 0x2d775a2d, 0x0f111e0f, 0xb0cb7bb0, 0x54fca854, 0xbbd66dbb, 0x163a2c16 ]
+    T4 = [ 0x6363a5c6, 0x7c7c84f8, 0x777799ee, 0x7b7b8df6, 0xf2f20dff, 0x6b6bbdd6, 0x6f6fb1de, 0xc5c55491, 0x30305060, 0x01010302, 0x6767a9ce, 0x2b2b7d56, 0xfefe19e7, 0xd7d762b5, 0xababe64d, 0x76769aec, 0xcaca458f, 0x82829d1f, 0xc9c94089, 0x7d7d87fa, 0xfafa15ef, 0x5959ebb2, 0x4747c98e, 0xf0f00bfb, 0xadadec41, 0xd4d467b3, 0xa2a2fd5f, 0xafafea45, 0x9c9cbf23, 0xa4a4f753, 0x727296e4, 0xc0c05b9b, 0xb7b7c275, 0xfdfd1ce1, 0x9393ae3d, 0x26266a4c, 0x36365a6c, 0x3f3f417e, 0xf7f702f5, 0xcccc4f83, 0x34345c68, 0xa5a5f451, 0xe5e534d1, 0xf1f108f9, 0x717193e2, 0xd8d873ab, 0x31315362, 0x15153f2a, 0x04040c08, 0xc7c75295, 0x23236546, 0xc3c35e9d, 0x18182830, 0x9696a137, 0x05050f0a, 0x9a9ab52f, 0x0707090e, 0x12123624, 0x80809b1b, 0xe2e23ddf, 0xebeb26cd, 0x2727694e, 0xb2b2cd7f, 0x75759fea, 0x09091b12, 0x83839e1d, 0x2c2c7458, 0x1a1a2e34, 0x1b1b2d36, 0x6e6eb2dc, 0x5a5aeeb4, 0xa0a0fb5b, 0x5252f6a4, 0x3b3b4d76, 0xd6d661b7, 0xb3b3ce7d, 0x29297b52, 0xe3e33edd, 0x2f2f715e, 0x84849713, 0x5353f5a6, 0xd1d168b9, 0x00000000, 0xeded2cc1, 0x20206040, 0xfcfc1fe3, 0xb1b1c879, 0x5b5bedb6, 0x6a6abed4, 0xcbcb468d, 0xbebed967, 0x39394b72, 0x4a4ade94, 0x4c4cd498, 0x5858e8b0, 0xcfcf4a85, 0xd0d06bbb, 0xefef2ac5, 0xaaaae54f, 0xfbfb16ed, 0x4343c586, 0x4d4dd79a, 0x33335566, 0x85859411, 0x4545cf8a, 0xf9f910e9, 0x02020604, 0x7f7f81fe, 0x5050f0a0, 0x3c3c4478, 0x9f9fba25, 0xa8a8e34b, 0x5151f3a2, 0xa3a3fe5d, 0x4040c080, 0x8f8f8a05, 0x9292ad3f, 0x9d9dbc21, 0x38384870, 0xf5f504f1, 0xbcbcdf63, 0xb6b6c177, 0xdada75af, 0x21216342, 0x10103020, 0xffff1ae5, 0xf3f30efd, 0xd2d26dbf, 0xcdcd4c81, 0x0c0c1418, 0x13133526, 0xecec2fc3, 0x5f5fe1be, 0x9797a235, 0x4444cc88, 0x1717392e, 0xc4c45793, 0xa7a7f255, 0x7e7e82fc, 0x3d3d477a, 0x6464acc8, 0x5d5de7ba, 0x19192b32, 0x737395e6, 0x6060a0c0, 0x81819819, 0x4f4fd19e, 0xdcdc7fa3, 0x22226644, 0x2a2a7e54, 0x9090ab3b, 0x8888830b, 0x4646ca8c, 0xeeee29c7, 0xb8b8d36b, 0x14143c28, 0xdede79a7, 0x5e5ee2bc, 0x0b0b1d16, 0xdbdb76ad, 0xe0e03bdb, 0x32325664, 0x3a3a4e74, 0x0a0a1e14, 0x4949db92, 0x06060a0c, 0x24246c48, 0x5c5ce4b8, 0xc2c25d9f, 0xd3d36ebd, 0xacacef43, 0x6262a6c4, 0x9191a839, 0x9595a431, 0xe4e437d3, 0x79798bf2, 0xe7e732d5, 0xc8c8438b, 0x3737596e, 0x6d6db7da, 0x8d8d8c01, 0xd5d564b1, 0x4e4ed29c, 0xa9a9e049, 0x6c6cb4d8, 0x5656faac, 0xf4f407f3, 0xeaea25cf, 0x6565afca, 0x7a7a8ef4, 0xaeaee947, 0x08081810, 0xbabad56f, 0x787888f0, 0x25256f4a, 0x2e2e725c, 0x1c1c2438, 0xa6a6f157, 0xb4b4c773, 0xc6c65197, 0xe8e823cb, 0xdddd7ca1, 0x74749ce8, 0x1f1f213e, 0x4b4bdd96, 0xbdbddc61, 0x8b8b860d, 0x8a8a850f, 0x707090e0, 0x3e3e427c, 0xb5b5c471, 0x6666aacc, 0x4848d890, 0x03030506, 0xf6f601f7, 0x0e0e121c, 0x6161a3c2, 0x35355f6a, 0x5757f9ae, 0xb9b9d069, 0x86869117, 0xc1c15899, 0x1d1d273a, 0x9e9eb927, 0xe1e138d9, 0xf8f813eb, 0x9898b32b, 0x11113322, 0x6969bbd2, 0xd9d970a9, 0x8e8e8907, 0x9494a733, 0x9b9bb62d, 0x1e1e223c, 0x87879215, 0xe9e920c9, 0xcece4987, 0x5555ffaa, 0x28287850, 0xdfdf7aa5, 0x8c8c8f03, 0xa1a1f859, 0x89898009, 0x0d0d171a, 0xbfbfda65, 0xe6e631d7, 0x4242c684, 0x6868b8d0, 0x4141c382, 0x9999b029, 0x2d2d775a, 0x0f0f111e, 0xb0b0cb7b, 0x5454fca8, 0xbbbbd66d, 0x16163a2c ]
+
+    # Transformations for decryption
+    T5 = [ 0x51f4a750, 0x7e416553, 0x1a17a4c3, 0x3a275e96, 0x3bab6bcb, 0x1f9d45f1, 0xacfa58ab, 0x4be30393, 0x2030fa55, 0xad766df6, 0x88cc7691, 0xf5024c25, 0x4fe5d7fc, 0xc52acbd7, 0x26354480, 0xb562a38f, 0xdeb15a49, 0x25ba1b67, 0x45ea0e98, 0x5dfec0e1, 0xc32f7502, 0x814cf012, 0x8d4697a3, 0x6bd3f9c6, 0x038f5fe7, 0x15929c95, 0xbf6d7aeb, 0x955259da, 0xd4be832d, 0x587421d3, 0x49e06929, 0x8ec9c844, 0x75c2896a, 0xf48e7978, 0x99583e6b, 0x27b971dd, 0xbee14fb6, 0xf088ad17, 0xc920ac66, 0x7dce3ab4, 0x63df4a18, 0xe51a3182, 0x97513360, 0x62537f45, 0xb16477e0, 0xbb6bae84, 0xfe81a01c, 0xf9082b94, 0x70486858, 0x8f45fd19, 0x94de6c87, 0x527bf8b7, 0xab73d323, 0x724b02e2, 0xe31f8f57, 0x6655ab2a, 0xb2eb2807, 0x2fb5c203, 0x86c57b9a, 0xd33708a5, 0x302887f2, 0x23bfa5b2, 0x02036aba, 0xed16825c, 0x8acf1c2b, 0xa779b492, 0xf307f2f0, 0x4e69e2a1, 0x65daf4cd, 0x0605bed5, 0xd134621f, 0xc4a6fe8a, 0x342e539d, 0xa2f355a0, 0x058ae132, 0xa4f6eb75, 0x0b83ec39, 0x4060efaa, 0x5e719f06, 0xbd6e1051, 0x3e218af9, 0x96dd063d, 0xdd3e05ae, 0x4de6bd46, 0x91548db5, 0x71c45d05, 0x0406d46f, 0x605015ff, 0x1998fb24, 0xd6bde997, 0x894043cc, 0x67d99e77, 0xb0e842bd, 0x07898b88, 0xe7195b38, 0x79c8eedb, 0xa17c0a47, 0x7c420fe9, 0xf8841ec9, 0x00000000, 0x09808683, 0x322bed48, 0x1e1170ac, 0x6c5a724e, 0xfd0efffb, 0x0f853856, 0x3daed51e, 0x362d3927, 0x0a0fd964, 0x685ca621, 0x9b5b54d1, 0x24362e3a, 0x0c0a67b1, 0x9357e70f, 0xb4ee96d2, 0x1b9b919e, 0x80c0c54f, 0x61dc20a2, 0x5a774b69, 0x1c121a16, 0xe293ba0a, 0xc0a02ae5, 0x3c22e043, 0x121b171d, 0x0e090d0b, 0xf28bc7ad, 0x2db6a8b9, 0x141ea9c8, 0x57f11985, 0xaf75074c, 0xee99ddbb, 0xa37f60fd, 0xf701269f, 0x5c72f5bc, 0x44663bc5, 0x5bfb7e34, 0x8b432976, 0xcb23c6dc, 0xb6edfc68, 0xb8e4f163, 0xd731dcca, 0x42638510, 0x13972240, 0x84c61120, 0x854a247d, 0xd2bb3df8, 0xaef93211, 0xc729a16d, 0x1d9e2f4b, 0xdcb230f3, 0x0d8652ec, 0x77c1e3d0, 0x2bb3166c, 0xa970b999, 0x119448fa, 0x47e96422, 0xa8fc8cc4, 0xa0f03f1a, 0x567d2cd8, 0x223390ef, 0x87494ec7, 0xd938d1c1, 0x8ccaa2fe, 0x98d40b36, 0xa6f581cf, 0xa57ade28, 0xdab78e26, 0x3fadbfa4, 0x2c3a9de4, 0x5078920d, 0x6a5fcc9b, 0x547e4662, 0xf68d13c2, 0x90d8b8e8, 0x2e39f75e, 0x82c3aff5, 0x9f5d80be, 0x69d0937c, 0x6fd52da9, 0xcf2512b3, 0xc8ac993b, 0x10187da7, 0xe89c636e, 0xdb3bbb7b, 0xcd267809, 0x6e5918f4, 0xec9ab701, 0x834f9aa8, 0xe6956e65, 0xaaffe67e, 0x21bccf08, 0xef15e8e6, 0xbae79bd9, 0x4a6f36ce, 0xea9f09d4, 0x29b07cd6, 0x31a4b2af, 0x2a3f2331, 0xc6a59430, 0x35a266c0, 0x744ebc37, 0xfc82caa6, 0xe090d0b0, 0x33a7d815, 0xf104984a, 0x41ecdaf7, 0x7fcd500e, 0x1791f62f, 0x764dd68d, 0x43efb04d, 0xccaa4d54, 0xe49604df, 0x9ed1b5e3, 0x4c6a881b, 0xc12c1fb8, 0x4665517f, 0x9d5eea04, 0x018c355d, 0xfa877473, 0xfb0b412e, 0xb3671d5a, 0x92dbd252, 0xe9105633, 0x6dd64713, 0x9ad7618c, 0x37a10c7a, 0x59f8148e, 0xeb133c89, 0xcea927ee, 0xb761c935, 0xe11ce5ed, 0x7a47b13c, 0x9cd2df59, 0x55f2733f, 0x1814ce79, 0x73c737bf, 0x53f7cdea, 0x5ffdaa5b, 0xdf3d6f14, 0x7844db86, 0xcaaff381, 0xb968c43e, 0x3824342c, 0xc2a3405f, 0x161dc372, 0xbce2250c, 0x283c498b, 0xff0d9541, 0x39a80171, 0x080cb3de, 0xd8b4e49c, 0x6456c190, 0x7bcb8461, 0xd532b670, 0x486c5c74, 0xd0b85742 ]
+    T6 = [ 0x5051f4a7, 0x537e4165, 0xc31a17a4, 0x963a275e, 0xcb3bab6b, 0xf11f9d45, 0xabacfa58, 0x934be303, 0x552030fa, 0xf6ad766d, 0x9188cc76, 0x25f5024c, 0xfc4fe5d7, 0xd7c52acb, 0x80263544, 0x8fb562a3, 0x49deb15a, 0x6725ba1b, 0x9845ea0e, 0xe15dfec0, 0x02c32f75, 0x12814cf0, 0xa38d4697, 0xc66bd3f9, 0xe7038f5f, 0x9515929c, 0xebbf6d7a, 0xda955259, 0x2dd4be83, 0xd3587421, 0x2949e069, 0x448ec9c8, 0x6a75c289, 0x78f48e79, 0x6b99583e, 0xdd27b971, 0xb6bee14f, 0x17f088ad, 0x66c920ac, 0xb47dce3a, 0x1863df4a, 0x82e51a31, 0x60975133, 0x4562537f, 0xe0b16477, 0x84bb6bae, 0x1cfe81a0, 0x94f9082b, 0x58704868, 0x198f45fd, 0x8794de6c, 0xb7527bf8, 0x23ab73d3, 0xe2724b02, 0x57e31f8f, 0x2a6655ab, 0x07b2eb28, 0x032fb5c2, 0x9a86c57b, 0xa5d33708, 0xf2302887, 0xb223bfa5, 0xba02036a, 0x5ced1682, 0x2b8acf1c, 0x92a779b4, 0xf0f307f2, 0xa14e69e2, 0xcd65daf4, 0xd50605be, 0x1fd13462, 0x8ac4a6fe, 0x9d342e53, 0xa0a2f355, 0x32058ae1, 0x75a4f6eb, 0x390b83ec, 0xaa4060ef, 0x065e719f, 0x51bd6e10, 0xf93e218a, 0x3d96dd06, 0xaedd3e05, 0x464de6bd, 0xb591548d, 0x0571c45d, 0x6f0406d4, 0xff605015, 0x241998fb, 0x97d6bde9, 0xcc894043, 0x7767d99e, 0xbdb0e842, 0x8807898b, 0x38e7195b, 0xdb79c8ee, 0x47a17c0a, 0xe97c420f, 0xc9f8841e, 0x00000000, 0x83098086, 0x48322bed, 0xac1e1170, 0x4e6c5a72, 0xfbfd0eff, 0x560f8538, 0x1e3daed5, 0x27362d39, 0x640a0fd9, 0x21685ca6, 0xd19b5b54, 0x3a24362e, 0xb10c0a67, 0x0f9357e7, 0xd2b4ee96, 0x9e1b9b91, 0x4f80c0c5, 0xa261dc20, 0x695a774b, 0x161c121a, 0x0ae293ba, 0xe5c0a02a, 0x433c22e0, 0x1d121b17, 0x0b0e090d, 0xadf28bc7, 0xb92db6a8, 0xc8141ea9, 0x8557f119, 0x4caf7507, 0xbbee99dd, 0xfda37f60, 0x9ff70126, 0xbc5c72f5, 0xc544663b, 0x345bfb7e, 0x768b4329, 0xdccb23c6, 0x68b6edfc, 0x63b8e4f1, 0xcad731dc, 0x10426385, 0x40139722, 0x2084c611, 0x7d854a24, 0xf8d2bb3d, 0x11aef932, 0x6dc729a1, 0x4b1d9e2f, 0xf3dcb230, 0xec0d8652, 0xd077c1e3, 0x6c2bb316, 0x99a970b9, 0xfa119448, 0x2247e964, 0xc4a8fc8c, 0x1aa0f03f, 0xd8567d2c, 0xef223390, 0xc787494e, 0xc1d938d1, 0xfe8ccaa2, 0x3698d40b, 0xcfa6f581, 0x28a57ade, 0x26dab78e, 0xa43fadbf, 0xe42c3a9d, 0x0d507892, 0x9b6a5fcc, 0x62547e46, 0xc2f68d13, 0xe890d8b8, 0x5e2e39f7, 0xf582c3af, 0xbe9f5d80, 0x7c69d093, 0xa96fd52d, 0xb3cf2512, 0x3bc8ac99, 0xa710187d, 0x6ee89c63, 0x7bdb3bbb, 0x09cd2678, 0xf46e5918, 0x01ec9ab7, 0xa8834f9a, 0x65e6956e, 0x7eaaffe6, 0x0821bccf, 0xe6ef15e8, 0xd9bae79b, 0xce4a6f36, 0xd4ea9f09, 0xd629b07c, 0xaf31a4b2, 0x312a3f23, 0x30c6a594, 0xc035a266, 0x37744ebc, 0xa6fc82ca, 0xb0e090d0, 0x1533a7d8, 0x4af10498, 0xf741ecda, 0x0e7fcd50, 0x2f1791f6, 0x8d764dd6, 0x4d43efb0, 0x54ccaa4d, 0xdfe49604, 0xe39ed1b5, 0x1b4c6a88, 0xb8c12c1f, 0x7f466551, 0x049d5eea, 0x5d018c35, 0x73fa8774, 0x2efb0b41, 0x5ab3671d, 0x5292dbd2, 0x33e91056, 0x136dd647, 0x8c9ad761, 0x7a37a10c, 0x8e59f814, 0x89eb133c, 0xeecea927, 0x35b761c9, 0xede11ce5, 0x3c7a47b1, 0x599cd2df, 0x3f55f273, 0x791814ce, 0xbf73c737, 0xea53f7cd, 0x5b5ffdaa, 0x14df3d6f, 0x867844db, 0x81caaff3, 0x3eb968c4, 0x2c382434, 0x5fc2a340, 0x72161dc3, 0x0cbce225, 0x8b283c49, 0x41ff0d95, 0x7139a801, 0xde080cb3, 0x9cd8b4e4, 0x906456c1, 0x617bcb84, 0x70d532b6, 0x74486c5c, 0x42d0b857 ]
+    T7 = [ 0xa75051f4, 0x65537e41, 0xa4c31a17, 0x5e963a27, 0x6bcb3bab, 0x45f11f9d, 0x58abacfa, 0x03934be3, 0xfa552030, 0x6df6ad76, 0x769188cc, 0x4c25f502, 0xd7fc4fe5, 0xcbd7c52a, 0x44802635, 0xa38fb562, 0x5a49deb1, 0x1b6725ba, 0x0e9845ea, 0xc0e15dfe, 0x7502c32f, 0xf012814c, 0x97a38d46, 0xf9c66bd3, 0x5fe7038f, 0x9c951592, 0x7aebbf6d, 0x59da9552, 0x832dd4be, 0x21d35874, 0x692949e0, 0xc8448ec9, 0x896a75c2, 0x7978f48e, 0x3e6b9958, 0x71dd27b9, 0x4fb6bee1, 0xad17f088, 0xac66c920, 0x3ab47dce, 0x4a1863df, 0x3182e51a, 0x33609751, 0x7f456253, 0x77e0b164, 0xae84bb6b, 0xa01cfe81, 0x2b94f908, 0x68587048, 0xfd198f45, 0x6c8794de, 0xf8b7527b, 0xd323ab73, 0x02e2724b, 0x8f57e31f, 0xab2a6655, 0x2807b2eb, 0xc2032fb5, 0x7b9a86c5, 0x08a5d337, 0x87f23028, 0xa5b223bf, 0x6aba0203, 0x825ced16, 0x1c2b8acf, 0xb492a779, 0xf2f0f307, 0xe2a14e69, 0xf4cd65da, 0xbed50605, 0x621fd134, 0xfe8ac4a6, 0x539d342e, 0x55a0a2f3, 0xe132058a, 0xeb75a4f6, 0xec390b83, 0xefaa4060, 0x9f065e71, 0x1051bd6e, 0x8af93e21, 0x063d96dd, 0x05aedd3e, 0xbd464de6, 0x8db59154, 0x5d0571c4, 0xd46f0406, 0x15ff6050, 0xfb241998, 0xe997d6bd, 0x43cc8940, 0x9e7767d9, 0x42bdb0e8, 0x8b880789, 0x5b38e719, 0xeedb79c8, 0x0a47a17c, 0x0fe97c42, 0x1ec9f884, 0x00000000, 0x86830980, 0xed48322b, 0x70ac1e11, 0x724e6c5a, 0xfffbfd0e, 0x38560f85, 0xd51e3dae, 0x3927362d, 0xd9640a0f, 0xa621685c, 0x54d19b5b, 0x2e3a2436, 0x67b10c0a, 0xe70f9357, 0x96d2b4ee, 0x919e1b9b, 0xc54f80c0, 0x20a261dc, 0x4b695a77, 0x1a161c12, 0xba0ae293, 0x2ae5c0a0, 0xe0433c22, 0x171d121b, 0x0d0b0e09, 0xc7adf28b, 0xa8b92db6, 0xa9c8141e, 0x198557f1, 0x074caf75, 0xddbbee99, 0x60fda37f, 0x269ff701, 0xf5bc5c72, 0x3bc54466, 0x7e345bfb, 0x29768b43, 0xc6dccb23, 0xfc68b6ed, 0xf163b8e4, 0xdccad731, 0x85104263, 0x22401397, 0x112084c6, 0x247d854a, 0x3df8d2bb, 0x3211aef9, 0xa16dc729, 0x2f4b1d9e, 0x30f3dcb2, 0x52ec0d86, 0xe3d077c1, 0x166c2bb3, 0xb999a970, 0x48fa1194, 0x642247e9, 0x8cc4a8fc, 0x3f1aa0f0, 0x2cd8567d, 0x90ef2233, 0x4ec78749, 0xd1c1d938, 0xa2fe8cca, 0x0b3698d4, 0x81cfa6f5, 0xde28a57a, 0x8e26dab7, 0xbfa43fad, 0x9de42c3a, 0x920d5078, 0xcc9b6a5f, 0x4662547e, 0x13c2f68d, 0xb8e890d8, 0xf75e2e39, 0xaff582c3, 0x80be9f5d, 0x937c69d0, 0x2da96fd5, 0x12b3cf25, 0x993bc8ac, 0x7da71018, 0x636ee89c, 0xbb7bdb3b, 0x7809cd26, 0x18f46e59, 0xb701ec9a, 0x9aa8834f, 0x6e65e695, 0xe67eaaff, 0xcf0821bc, 0xe8e6ef15, 0x9bd9bae7, 0x36ce4a6f, 0x09d4ea9f, 0x7cd629b0, 0xb2af31a4, 0x23312a3f, 0x9430c6a5, 0x66c035a2, 0xbc37744e, 0xcaa6fc82, 0xd0b0e090, 0xd81533a7, 0x984af104, 0xdaf741ec, 0x500e7fcd, 0xf62f1791, 0xd68d764d, 0xb04d43ef, 0x4d54ccaa, 0x04dfe496, 0xb5e39ed1, 0x881b4c6a, 0x1fb8c12c, 0x517f4665, 0xea049d5e, 0x355d018c, 0x7473fa87, 0x412efb0b, 0x1d5ab367, 0xd25292db, 0x5633e910, 0x47136dd6, 0x618c9ad7, 0x0c7a37a1, 0x148e59f8, 0x3c89eb13, 0x27eecea9, 0xc935b761, 0xe5ede11c, 0xb13c7a47, 0xdf599cd2, 0x733f55f2, 0xce791814, 0x37bf73c7, 0xcdea53f7, 0xaa5b5ffd, 0x6f14df3d, 0xdb867844, 0xf381caaf, 0xc43eb968, 0x342c3824, 0x405fc2a3, 0xc372161d, 0x250cbce2, 0x498b283c, 0x9541ff0d, 0x017139a8, 0xb3de080c, 0xe49cd8b4, 0xc1906456, 0x84617bcb, 0xb670d532, 0x5c74486c, 0x5742d0b8 ]
+    T8 = [ 0xf4a75051, 0x4165537e, 0x17a4c31a, 0x275e963a, 0xab6bcb3b, 0x9d45f11f, 0xfa58abac, 0xe303934b, 0x30fa5520, 0x766df6ad, 0xcc769188, 0x024c25f5, 0xe5d7fc4f, 0x2acbd7c5, 0x35448026, 0x62a38fb5, 0xb15a49de, 0xba1b6725, 0xea0e9845, 0xfec0e15d, 0x2f7502c3, 0x4cf01281, 0x4697a38d, 0xd3f9c66b, 0x8f5fe703, 0x929c9515, 0x6d7aebbf, 0x5259da95, 0xbe832dd4, 0x7421d358, 0xe0692949, 0xc9c8448e, 0xc2896a75, 0x8e7978f4, 0x583e6b99, 0xb971dd27, 0xe14fb6be, 0x88ad17f0, 0x20ac66c9, 0xce3ab47d, 0xdf4a1863, 0x1a3182e5, 0x51336097, 0x537f4562, 0x6477e0b1, 0x6bae84bb, 0x81a01cfe, 0x082b94f9, 0x48685870, 0x45fd198f, 0xde6c8794, 0x7bf8b752, 0x73d323ab, 0x4b02e272, 0x1f8f57e3, 0x55ab2a66, 0xeb2807b2, 0xb5c2032f, 0xc57b9a86, 0x3708a5d3, 0x2887f230, 0xbfa5b223, 0x036aba02, 0x16825ced, 0xcf1c2b8a, 0x79b492a7, 0x07f2f0f3, 0x69e2a14e, 0xdaf4cd65, 0x05bed506, 0x34621fd1, 0xa6fe8ac4, 0x2e539d34, 0xf355a0a2, 0x8ae13205, 0xf6eb75a4, 0x83ec390b, 0x60efaa40, 0x719f065e, 0x6e1051bd, 0x218af93e, 0xdd063d96, 0x3e05aedd, 0xe6bd464d, 0x548db591, 0xc45d0571, 0x06d46f04, 0x5015ff60, 0x98fb2419, 0xbde997d6, 0x4043cc89, 0xd99e7767, 0xe842bdb0, 0x898b8807, 0x195b38e7, 0xc8eedb79, 0x7c0a47a1, 0x420fe97c, 0x841ec9f8, 0x00000000, 0x80868309, 0x2bed4832, 0x1170ac1e, 0x5a724e6c, 0x0efffbfd, 0x8538560f, 0xaed51e3d, 0x2d392736, 0x0fd9640a, 0x5ca62168, 0x5b54d19b, 0x362e3a24, 0x0a67b10c, 0x57e70f93, 0xee96d2b4, 0x9b919e1b, 0xc0c54f80, 0xdc20a261, 0x774b695a, 0x121a161c, 0x93ba0ae2, 0xa02ae5c0, 0x22e0433c, 0x1b171d12, 0x090d0b0e, 0x8bc7adf2, 0xb6a8b92d, 0x1ea9c814, 0xf1198557, 0x75074caf, 0x99ddbbee, 0x7f60fda3, 0x01269ff7, 0x72f5bc5c, 0x663bc544, 0xfb7e345b, 0x4329768b, 0x23c6dccb, 0xedfc68b6, 0xe4f163b8, 0x31dccad7, 0x63851042, 0x97224013, 0xc6112084, 0x4a247d85, 0xbb3df8d2, 0xf93211ae, 0x29a16dc7, 0x9e2f4b1d, 0xb230f3dc, 0x8652ec0d, 0xc1e3d077, 0xb3166c2b, 0x70b999a9, 0x9448fa11, 0xe9642247, 0xfc8cc4a8, 0xf03f1aa0, 0x7d2cd856, 0x3390ef22, 0x494ec787, 0x38d1c1d9, 0xcaa2fe8c, 0xd40b3698, 0xf581cfa6, 0x7ade28a5, 0xb78e26da, 0xadbfa43f, 0x3a9de42c, 0x78920d50, 0x5fcc9b6a, 0x7e466254, 0x8d13c2f6, 0xd8b8e890, 0x39f75e2e, 0xc3aff582, 0x5d80be9f, 0xd0937c69, 0xd52da96f, 0x2512b3cf, 0xac993bc8, 0x187da710, 0x9c636ee8, 0x3bbb7bdb, 0x267809cd, 0x5918f46e, 0x9ab701ec, 0x4f9aa883, 0x956e65e6, 0xffe67eaa, 0xbccf0821, 0x15e8e6ef, 0xe79bd9ba, 0x6f36ce4a, 0x9f09d4ea, 0xb07cd629, 0xa4b2af31, 0x3f23312a, 0xa59430c6, 0xa266c035, 0x4ebc3774, 0x82caa6fc, 0x90d0b0e0, 0xa7d81533, 0x04984af1, 0xecdaf741, 0xcd500e7f, 0x91f62f17, 0x4dd68d76, 0xefb04d43, 0xaa4d54cc, 0x9604dfe4, 0xd1b5e39e, 0x6a881b4c, 0x2c1fb8c1, 0x65517f46, 0x5eea049d, 0x8c355d01, 0x877473fa, 0x0b412efb, 0x671d5ab3, 0xdbd25292, 0x105633e9, 0xd647136d, 0xd7618c9a, 0xa10c7a37, 0xf8148e59, 0x133c89eb, 0xa927eece, 0x61c935b7, 0x1ce5ede1, 0x47b13c7a, 0xd2df599c, 0xf2733f55, 0x14ce7918, 0xc737bf73, 0xf7cdea53, 0xfdaa5b5f, 0x3d6f14df, 0x44db8678, 0xaff381ca, 0x68c43eb9, 0x24342c38, 0xa3405fc2, 0x1dc37216, 0xe2250cbc, 0x3c498b28, 0x0d9541ff, 0xa8017139, 0x0cb3de08, 0xb4e49cd8, 0x56c19064, 0xcb84617b, 0x32b670d5, 0x6c5c7448, 0xb85742d0 ]
+
+    # Transformations for decryption key expansion
+    U1 = [ 0x00000000, 0x0e090d0b, 0x1c121a16, 0x121b171d, 0x3824342c, 0x362d3927, 0x24362e3a, 0x2a3f2331, 0x70486858, 0x7e416553, 0x6c5a724e, 0x62537f45, 0x486c5c74, 0x4665517f, 0x547e4662, 0x5a774b69, 0xe090d0b0, 0xee99ddbb, 0xfc82caa6, 0xf28bc7ad, 0xd8b4e49c, 0xd6bde997, 0xc4a6fe8a, 0xcaaff381, 0x90d8b8e8, 0x9ed1b5e3, 0x8ccaa2fe, 0x82c3aff5, 0xa8fc8cc4, 0xa6f581cf, 0xb4ee96d2, 0xbae79bd9, 0xdb3bbb7b, 0xd532b670, 0xc729a16d, 0xc920ac66, 0xe31f8f57, 0xed16825c, 0xff0d9541, 0xf104984a, 0xab73d323, 0xa57ade28, 0xb761c935, 0xb968c43e, 0x9357e70f, 0x9d5eea04, 0x8f45fd19, 0x814cf012, 0x3bab6bcb, 0x35a266c0, 0x27b971dd, 0x29b07cd6, 0x038f5fe7, 0x0d8652ec, 0x1f9d45f1, 0x119448fa, 0x4be30393, 0x45ea0e98, 0x57f11985, 0x59f8148e, 0x73c737bf, 0x7dce3ab4, 0x6fd52da9, 0x61dc20a2, 0xad766df6, 0xa37f60fd, 0xb16477e0, 0xbf6d7aeb, 0x955259da, 0x9b5b54d1, 0x894043cc, 0x87494ec7, 0xdd3e05ae, 0xd33708a5, 0xc12c1fb8, 0xcf2512b3, 0xe51a3182, 0xeb133c89, 0xf9082b94, 0xf701269f, 0x4de6bd46, 0x43efb04d, 0x51f4a750, 0x5ffdaa5b, 0x75c2896a, 0x7bcb8461, 0x69d0937c, 0x67d99e77, 0x3daed51e, 0x33a7d815, 0x21bccf08, 0x2fb5c203, 0x058ae132, 0x0b83ec39, 0x1998fb24, 0x1791f62f, 0x764dd68d, 0x7844db86, 0x6a5fcc9b, 0x6456c190, 0x4e69e2a1, 0x4060efaa, 0x527bf8b7, 0x5c72f5bc, 0x0605bed5, 0x080cb3de, 0x1a17a4c3, 0x141ea9c8, 0x3e218af9, 0x302887f2, 0x223390ef, 0x2c3a9de4, 0x96dd063d, 0x98d40b36, 0x8acf1c2b, 0x84c61120, 0xaef93211, 0xa0f03f1a, 0xb2eb2807, 0xbce2250c, 0xe6956e65, 0xe89c636e, 0xfa877473, 0xf48e7978, 0xdeb15a49, 0xd0b85742, 0xc2a3405f, 0xccaa4d54, 0x41ecdaf7, 0x4fe5d7fc, 0x5dfec0e1, 0x53f7cdea, 0x79c8eedb, 0x77c1e3d0, 0x65daf4cd, 0x6bd3f9c6, 0x31a4b2af, 0x3fadbfa4, 0x2db6a8b9, 0x23bfa5b2, 0x09808683, 0x07898b88, 0x15929c95, 0x1b9b919e, 0xa17c0a47, 0xaf75074c, 0xbd6e1051, 0xb3671d5a, 0x99583e6b, 0x97513360, 0x854a247d, 0x8b432976, 0xd134621f, 0xdf3d6f14, 0xcd267809, 0xc32f7502, 0xe9105633, 0xe7195b38, 0xf5024c25, 0xfb0b412e, 0x9ad7618c, 0x94de6c87, 0x86c57b9a, 0x88cc7691, 0xa2f355a0, 0xacfa58ab, 0xbee14fb6, 0xb0e842bd, 0xea9f09d4, 0xe49604df, 0xf68d13c2, 0xf8841ec9, 0xd2bb3df8, 0xdcb230f3, 0xcea927ee, 0xc0a02ae5, 0x7a47b13c, 0x744ebc37, 0x6655ab2a, 0x685ca621, 0x42638510, 0x4c6a881b, 0x5e719f06, 0x5078920d, 0x0a0fd964, 0x0406d46f, 0x161dc372, 0x1814ce79, 0x322bed48, 0x3c22e043, 0x2e39f75e, 0x2030fa55, 0xec9ab701, 0xe293ba0a, 0xf088ad17, 0xfe81a01c, 0xd4be832d, 0xdab78e26, 0xc8ac993b, 0xc6a59430, 0x9cd2df59, 0x92dbd252, 0x80c0c54f, 0x8ec9c844, 0xa4f6eb75, 0xaaffe67e, 0xb8e4f163, 0xb6edfc68, 0x0c0a67b1, 0x02036aba, 0x10187da7, 0x1e1170ac, 0x342e539d, 0x3a275e96, 0x283c498b, 0x26354480, 0x7c420fe9, 0x724b02e2, 0x605015ff, 0x6e5918f4, 0x44663bc5, 0x4a6f36ce, 0x587421d3, 0x567d2cd8, 0x37a10c7a, 0x39a80171, 0x2bb3166c, 0x25ba1b67, 0x0f853856, 0x018c355d, 0x13972240, 0x1d9e2f4b, 0x47e96422, 0x49e06929, 0x5bfb7e34, 0x55f2733f, 0x7fcd500e, 0x71c45d05, 0x63df4a18, 0x6dd64713, 0xd731dcca, 0xd938d1c1, 0xcb23c6dc, 0xc52acbd7, 0xef15e8e6, 0xe11ce5ed, 0xf307f2f0, 0xfd0efffb, 0xa779b492, 0xa970b999, 0xbb6bae84, 0xb562a38f, 0x9f5d80be, 0x91548db5, 0x834f9aa8, 0x8d4697a3 ]
+    U2 = [ 0x00000000, 0x0b0e090d, 0x161c121a, 0x1d121b17, 0x2c382434, 0x27362d39, 0x3a24362e, 0x312a3f23, 0x58704868, 0x537e4165, 0x4e6c5a72, 0x4562537f, 0x74486c5c, 0x7f466551, 0x62547e46, 0x695a774b, 0xb0e090d0, 0xbbee99dd, 0xa6fc82ca, 0xadf28bc7, 0x9cd8b4e4, 0x97d6bde9, 0x8ac4a6fe, 0x81caaff3, 0xe890d8b8, 0xe39ed1b5, 0xfe8ccaa2, 0xf582c3af, 0xc4a8fc8c, 0xcfa6f581, 0xd2b4ee96, 0xd9bae79b, 0x7bdb3bbb, 0x70d532b6, 0x6dc729a1, 0x66c920ac, 0x57e31f8f, 0x5ced1682, 0x41ff0d95, 0x4af10498, 0x23ab73d3, 0x28a57ade, 0x35b761c9, 0x3eb968c4, 0x0f9357e7, 0x049d5eea, 0x198f45fd, 0x12814cf0, 0xcb3bab6b, 0xc035a266, 0xdd27b971, 0xd629b07c, 0xe7038f5f, 0xec0d8652, 0xf11f9d45, 0xfa119448, 0x934be303, 0x9845ea0e, 0x8557f119, 0x8e59f814, 0xbf73c737, 0xb47dce3a, 0xa96fd52d, 0xa261dc20, 0xf6ad766d, 0xfda37f60, 0xe0b16477, 0xebbf6d7a, 0xda955259, 0xd19b5b54, 0xcc894043, 0xc787494e, 0xaedd3e05, 0xa5d33708, 0xb8c12c1f, 0xb3cf2512, 0x82e51a31, 0x89eb133c, 0x94f9082b, 0x9ff70126, 0x464de6bd, 0x4d43efb0, 0x5051f4a7, 0x5b5ffdaa, 0x6a75c289, 0x617bcb84, 0x7c69d093, 0x7767d99e, 0x1e3daed5, 0x1533a7d8, 0x0821bccf, 0x032fb5c2, 0x32058ae1, 0x390b83ec, 0x241998fb, 0x2f1791f6, 0x8d764dd6, 0x867844db, 0x9b6a5fcc, 0x906456c1, 0xa14e69e2, 0xaa4060ef, 0xb7527bf8, 0xbc5c72f5, 0xd50605be, 0xde080cb3, 0xc31a17a4, 0xc8141ea9, 0xf93e218a, 0xf2302887, 0xef223390, 0xe42c3a9d, 0x3d96dd06, 0x3698d40b, 0x2b8acf1c, 0x2084c611, 0x11aef932, 0x1aa0f03f, 0x07b2eb28, 0x0cbce225, 0x65e6956e, 0x6ee89c63, 0x73fa8774, 0x78f48e79, 0x49deb15a, 0x42d0b857, 0x5fc2a340, 0x54ccaa4d, 0xf741ecda, 0xfc4fe5d7, 0xe15dfec0, 0xea53f7cd, 0xdb79c8ee, 0xd077c1e3, 0xcd65daf4, 0xc66bd3f9, 0xaf31a4b2, 0xa43fadbf, 0xb92db6a8, 0xb223bfa5, 0x83098086, 0x8807898b, 0x9515929c, 0x9e1b9b91, 0x47a17c0a, 0x4caf7507, 0x51bd6e10, 0x5ab3671d, 0x6b99583e, 0x60975133, 0x7d854a24, 0x768b4329, 0x1fd13462, 0x14df3d6f, 0x09cd2678, 0x02c32f75, 0x33e91056, 0x38e7195b, 0x25f5024c, 0x2efb0b41, 0x8c9ad761, 0x8794de6c, 0x9a86c57b, 0x9188cc76, 0xa0a2f355, 0xabacfa58, 0xb6bee14f, 0xbdb0e842, 0xd4ea9f09, 0xdfe49604, 0xc2f68d13, 0xc9f8841e, 0xf8d2bb3d, 0xf3dcb230, 0xeecea927, 0xe5c0a02a, 0x3c7a47b1, 0x37744ebc, 0x2a6655ab, 0x21685ca6, 0x10426385, 0x1b4c6a88, 0x065e719f, 0x0d507892, 0x640a0fd9, 0x6f0406d4, 0x72161dc3, 0x791814ce, 0x48322bed, 0x433c22e0, 0x5e2e39f7, 0x552030fa, 0x01ec9ab7, 0x0ae293ba, 0x17f088ad, 0x1cfe81a0, 0x2dd4be83, 0x26dab78e, 0x3bc8ac99, 0x30c6a594, 0x599cd2df, 0x5292dbd2, 0x4f80c0c5, 0x448ec9c8, 0x75a4f6eb, 0x7eaaffe6, 0x63b8e4f1, 0x68b6edfc, 0xb10c0a67, 0xba02036a, 0xa710187d, 0xac1e1170, 0x9d342e53, 0x963a275e, 0x8b283c49, 0x80263544, 0xe97c420f, 0xe2724b02, 0xff605015, 0xf46e5918, 0xc544663b, 0xce4a6f36, 0xd3587421, 0xd8567d2c, 0x7a37a10c, 0x7139a801, 0x6c2bb316, 0x6725ba1b, 0x560f8538, 0x5d018c35, 0x40139722, 0x4b1d9e2f, 0x2247e964, 0x2949e069, 0x345bfb7e, 0x3f55f273, 0x0e7fcd50, 0x0571c45d, 0x1863df4a, 0x136dd647, 0xcad731dc, 0xc1d938d1, 0xdccb23c6, 0xd7c52acb, 0xe6ef15e8, 0xede11ce5, 0xf0f307f2, 0xfbfd0eff, 0x92a779b4, 0x99a970b9, 0x84bb6bae, 0x8fb562a3, 0xbe9f5d80, 0xb591548d, 0xa8834f9a, 0xa38d4697 ]
+    U3 = [ 0x00000000, 0x0d0b0e09, 0x1a161c12, 0x171d121b, 0x342c3824, 0x3927362d, 0x2e3a2436, 0x23312a3f, 0x68587048, 0x65537e41, 0x724e6c5a, 0x7f456253, 0x5c74486c, 0x517f4665, 0x4662547e, 0x4b695a77, 0xd0b0e090, 0xddbbee99, 0xcaa6fc82, 0xc7adf28b, 0xe49cd8b4, 0xe997d6bd, 0xfe8ac4a6, 0xf381caaf, 0xb8e890d8, 0xb5e39ed1, 0xa2fe8cca, 0xaff582c3, 0x8cc4a8fc, 0x81cfa6f5, 0x96d2b4ee, 0x9bd9bae7, 0xbb7bdb3b, 0xb670d532, 0xa16dc729, 0xac66c920, 0x8f57e31f, 0x825ced16, 0x9541ff0d, 0x984af104, 0xd323ab73, 0xde28a57a, 0xc935b761, 0xc43eb968, 0xe70f9357, 0xea049d5e, 0xfd198f45, 0xf012814c, 0x6bcb3bab, 0x66c035a2, 0x71dd27b9, 0x7cd629b0, 0x5fe7038f, 0x52ec0d86, 0x45f11f9d, 0x48fa1194, 0x03934be3, 0x0e9845ea, 0x198557f1, 0x148e59f8, 0x37bf73c7, 0x3ab47dce, 0x2da96fd5, 0x20a261dc, 0x6df6ad76, 0x60fda37f, 0x77e0b164, 0x7aebbf6d, 0x59da9552, 0x54d19b5b, 0x43cc8940, 0x4ec78749, 0x05aedd3e, 0x08a5d337, 0x1fb8c12c, 0x12b3cf25, 0x3182e51a, 0x3c89eb13, 0x2b94f908, 0x269ff701, 0xbd464de6, 0xb04d43ef, 0xa75051f4, 0xaa5b5ffd, 0x896a75c2, 0x84617bcb, 0x937c69d0, 0x9e7767d9, 0xd51e3dae, 0xd81533a7, 0xcf0821bc, 0xc2032fb5, 0xe132058a, 0xec390b83, 0xfb241998, 0xf62f1791, 0xd68d764d, 0xdb867844, 0xcc9b6a5f, 0xc1906456, 0xe2a14e69, 0xefaa4060, 0xf8b7527b, 0xf5bc5c72, 0xbed50605, 0xb3de080c, 0xa4c31a17, 0xa9c8141e, 0x8af93e21, 0x87f23028, 0x90ef2233, 0x9de42c3a, 0x063d96dd, 0x0b3698d4, 0x1c2b8acf, 0x112084c6, 0x3211aef9, 0x3f1aa0f0, 0x2807b2eb, 0x250cbce2, 0x6e65e695, 0x636ee89c, 0x7473fa87, 0x7978f48e, 0x5a49deb1, 0x5742d0b8, 0x405fc2a3, 0x4d54ccaa, 0xdaf741ec, 0xd7fc4fe5, 0xc0e15dfe, 0xcdea53f7, 0xeedb79c8, 0xe3d077c1, 0xf4cd65da, 0xf9c66bd3, 0xb2af31a4, 0xbfa43fad, 0xa8b92db6, 0xa5b223bf, 0x86830980, 0x8b880789, 0x9c951592, 0x919e1b9b, 0x0a47a17c, 0x074caf75, 0x1051bd6e, 0x1d5ab367, 0x3e6b9958, 0x33609751, 0x247d854a, 0x29768b43, 0x621fd134, 0x6f14df3d, 0x7809cd26, 0x7502c32f, 0x5633e910, 0x5b38e719, 0x4c25f502, 0x412efb0b, 0x618c9ad7, 0x6c8794de, 0x7b9a86c5, 0x769188cc, 0x55a0a2f3, 0x58abacfa, 0x4fb6bee1, 0x42bdb0e8, 0x09d4ea9f, 0x04dfe496, 0x13c2f68d, 0x1ec9f884, 0x3df8d2bb, 0x30f3dcb2, 0x27eecea9, 0x2ae5c0a0, 0xb13c7a47, 0xbc37744e, 0xab2a6655, 0xa621685c, 0x85104263, 0x881b4c6a, 0x9f065e71, 0x920d5078, 0xd9640a0f, 0xd46f0406, 0xc372161d, 0xce791814, 0xed48322b, 0xe0433c22, 0xf75e2e39, 0xfa552030, 0xb701ec9a, 0xba0ae293, 0xad17f088, 0xa01cfe81, 0x832dd4be, 0x8e26dab7, 0x993bc8ac, 0x9430c6a5, 0xdf599cd2, 0xd25292db, 0xc54f80c0, 0xc8448ec9, 0xeb75a4f6, 0xe67eaaff, 0xf163b8e4, 0xfc68b6ed, 0x67b10c0a, 0x6aba0203, 0x7da71018, 0x70ac1e11, 0x539d342e, 0x5e963a27, 0x498b283c, 0x44802635, 0x0fe97c42, 0x02e2724b, 0x15ff6050, 0x18f46e59, 0x3bc54466, 0x36ce4a6f, 0x21d35874, 0x2cd8567d, 0x0c7a37a1, 0x017139a8, 0x166c2bb3, 0x1b6725ba, 0x38560f85, 0x355d018c, 0x22401397, 0x2f4b1d9e, 0x642247e9, 0x692949e0, 0x7e345bfb, 0x733f55f2, 0x500e7fcd, 0x5d0571c4, 0x4a1863df, 0x47136dd6, 0xdccad731, 0xd1c1d938, 0xc6dccb23, 0xcbd7c52a, 0xe8e6ef15, 0xe5ede11c, 0xf2f0f307, 0xfffbfd0e, 0xb492a779, 0xb999a970, 0xae84bb6b, 0xa38fb562, 0x80be9f5d, 0x8db59154, 0x9aa8834f, 0x97a38d46 ]
+    U4 = [ 0x00000000, 0x090d0b0e, 0x121a161c, 0x1b171d12, 0x24342c38, 0x2d392736, 0x362e3a24, 0x3f23312a, 0x48685870, 0x4165537e, 0x5a724e6c, 0x537f4562, 0x6c5c7448, 0x65517f46, 0x7e466254, 0x774b695a, 0x90d0b0e0, 0x99ddbbee, 0x82caa6fc, 0x8bc7adf2, 0xb4e49cd8, 0xbde997d6, 0xa6fe8ac4, 0xaff381ca, 0xd8b8e890, 0xd1b5e39e, 0xcaa2fe8c, 0xc3aff582, 0xfc8cc4a8, 0xf581cfa6, 0xee96d2b4, 0xe79bd9ba, 0x3bbb7bdb, 0x32b670d5, 0x29a16dc7, 0x20ac66c9, 0x1f8f57e3, 0x16825ced, 0x0d9541ff, 0x04984af1, 0x73d323ab, 0x7ade28a5, 0x61c935b7, 0x68c43eb9, 0x57e70f93, 0x5eea049d, 0x45fd198f, 0x4cf01281, 0xab6bcb3b, 0xa266c035, 0xb971dd27, 0xb07cd629, 0x8f5fe703, 0x8652ec0d, 0x9d45f11f, 0x9448fa11, 0xe303934b, 0xea0e9845, 0xf1198557, 0xf8148e59, 0xc737bf73, 0xce3ab47d, 0xd52da96f, 0xdc20a261, 0x766df6ad, 0x7f60fda3, 0x6477e0b1, 0x6d7aebbf, 0x5259da95, 0x5b54d19b, 0x4043cc89, 0x494ec787, 0x3e05aedd, 0x3708a5d3, 0x2c1fb8c1, 0x2512b3cf, 0x1a3182e5, 0x133c89eb, 0x082b94f9, 0x01269ff7, 0xe6bd464d, 0xefb04d43, 0xf4a75051, 0xfdaa5b5f, 0xc2896a75, 0xcb84617b, 0xd0937c69, 0xd99e7767, 0xaed51e3d, 0xa7d81533, 0xbccf0821, 0xb5c2032f, 0x8ae13205, 0x83ec390b, 0x98fb2419, 0x91f62f17, 0x4dd68d76, 0x44db8678, 0x5fcc9b6a, 0x56c19064, 0x69e2a14e, 0x60efaa40, 0x7bf8b752, 0x72f5bc5c, 0x05bed506, 0x0cb3de08, 0x17a4c31a, 0x1ea9c814, 0x218af93e, 0x2887f230, 0x3390ef22, 0x3a9de42c, 0xdd063d96, 0xd40b3698, 0xcf1c2b8a, 0xc6112084, 0xf93211ae, 0xf03f1aa0, 0xeb2807b2, 0xe2250cbc, 0x956e65e6, 0x9c636ee8, 0x877473fa, 0x8e7978f4, 0xb15a49de, 0xb85742d0, 0xa3405fc2, 0xaa4d54cc, 0xecdaf741, 0xe5d7fc4f, 0xfec0e15d, 0xf7cdea53, 0xc8eedb79, 0xc1e3d077, 0xdaf4cd65, 0xd3f9c66b, 0xa4b2af31, 0xadbfa43f, 0xb6a8b92d, 0xbfa5b223, 0x80868309, 0x898b8807, 0x929c9515, 0x9b919e1b, 0x7c0a47a1, 0x75074caf, 0x6e1051bd, 0x671d5ab3, 0x583e6b99, 0x51336097, 0x4a247d85, 0x4329768b, 0x34621fd1, 0x3d6f14df, 0x267809cd, 0x2f7502c3, 0x105633e9, 0x195b38e7, 0x024c25f5, 0x0b412efb, 0xd7618c9a, 0xde6c8794, 0xc57b9a86, 0xcc769188, 0xf355a0a2, 0xfa58abac, 0xe14fb6be, 0xe842bdb0, 0x9f09d4ea, 0x9604dfe4, 0x8d13c2f6, 0x841ec9f8, 0xbb3df8d2, 0xb230f3dc, 0xa927eece, 0xa02ae5c0, 0x47b13c7a, 0x4ebc3774, 0x55ab2a66, 0x5ca62168, 0x63851042, 0x6a881b4c, 0x719f065e, 0x78920d50, 0x0fd9640a, 0x06d46f04, 0x1dc37216, 0x14ce7918, 0x2bed4832, 0x22e0433c, 0x39f75e2e, 0x30fa5520, 0x9ab701ec, 0x93ba0ae2, 0x88ad17f0, 0x81a01cfe, 0xbe832dd4, 0xb78e26da, 0xac993bc8, 0xa59430c6, 0xd2df599c, 0xdbd25292, 0xc0c54f80, 0xc9c8448e, 0xf6eb75a4, 0xffe67eaa, 0xe4f163b8, 0xedfc68b6, 0x0a67b10c, 0x036aba02, 0x187da710, 0x1170ac1e, 0x2e539d34, 0x275e963a, 0x3c498b28, 0x35448026, 0x420fe97c, 0x4b02e272, 0x5015ff60, 0x5918f46e, 0x663bc544, 0x6f36ce4a, 0x7421d358, 0x7d2cd856, 0xa10c7a37, 0xa8017139, 0xb3166c2b, 0xba1b6725, 0x8538560f, 0x8c355d01, 0x97224013, 0x9e2f4b1d, 0xe9642247, 0xe0692949, 0xfb7e345b, 0xf2733f55, 0xcd500e7f, 0xc45d0571, 0xdf4a1863, 0xd647136d, 0x31dccad7, 0x38d1c1d9, 0x23c6dccb, 0x2acbd7c5, 0x15e8e6ef, 0x1ce5ede1, 0x07f2f0f3, 0x0efffbfd, 0x79b492a7, 0x70b999a9, 0x6bae84bb, 0x62a38fb5, 0x5d80be9f, 0x548db591, 0x4f9aa883, 0x4697a38d ]
+
+    def __init__(self, key):
+
+        if len(key) not in (16, 24, 32):
+            raise ValueError('Invalid key size')
+
+        rounds = self.number_of_rounds[len(key)]
+
+        # Encryption round keys
+        self._Ke = [[0] * 4 for i in xrange(rounds + 1)]
+
+        # Decryption round keys
+        self._Kd = [[0] * 4 for i in xrange(rounds + 1)]
+
+        round_key_count = (rounds + 1) * 4
+        KC = len(key) // 4
+
+        # Convert the key into ints
+        tk = [ struct.unpack('>i', key[i:i + 4])[0] for i in xrange(0, len(key), 4) ]
+
+        # Copy values into round key arrays
+        for i in xrange(0, KC):
+            self._Ke[i // 4][i % 4] = tk[i]
+            self._Kd[rounds - (i // 4)][i % 4] = tk[i]
+
+        # Key expansion (fips-197 section 5.2)
+        rconpointer = 0
+        t = KC
+        while t < round_key_count:
+
+            tt = tk[KC - 1]
+            tk[0] ^= ((self.S[(tt >> 16) & 0xFF] << 24) ^
+                      (self.S[(tt >>  8) & 0xFF] << 16) ^
+                      (self.S[ tt        & 0xFF] <<  8) ^
+                       self.S[(tt >> 24) & 0xFF]        ^
+                      (self.rcon[rconpointer] << 24))
+            rconpointer += 1
+
+            if KC != 8:
+                for i in xrange(1, KC):
+                    tk[i] ^= tk[i - 1]
+
+            # Key expansion for 256-bit keys is "slightly different" (fips-197)
+            else:
+                for i in xrange(1, KC // 2):
+                    tk[i] ^= tk[i - 1]
+                tt = tk[KC // 2 - 1]
+
+                tk[KC // 2] ^= (self.S[ tt        & 0xFF]        ^
+                               (self.S[(tt >>  8) & 0xFF] <<  8) ^
+                               (self.S[(tt >> 16) & 0xFF] << 16) ^
+                               (self.S[(tt >> 24) & 0xFF] << 24))
+
+                for i in xrange(KC // 2 + 1, KC):
+                    tk[i] ^= tk[i - 1]
+
+            # Copy values into round key arrays
+            j = 0
+            while j < KC and t < round_key_count:
+                self._Ke[t // 4][t % 4] = tk[j]
+                self._Kd[rounds - (t // 4)][t % 4] = tk[j]
+                j += 1
+                t += 1
+
+        # Inverse-Cipher-ify the decryption round key (fips-197 section 5.3)
+        for r in xrange(1, rounds):
+            for j in xrange(0, 4):
+                tt = self._Kd[r][j]
+                self._Kd[r][j] = (self.U1[(tt >> 24) & 0xFF] ^
+                                  self.U2[(tt >> 16) & 0xFF] ^
+                                  self.U3[(tt >>  8) & 0xFF] ^
+                                  self.U4[ tt        & 0xFF])
+
+    def encrypt(self, plaintext):
+        'Encrypt a block of plain text using the AES block cipher.'
+
+        if len(plaintext) != 16:
+            raise ValueError('wrong block length')
+
+        rounds = len(self._Ke) - 1
+        (s1, s2, s3) = [1, 2, 3]
+        a = [0, 0, 0, 0]
+
+        # Convert plaintext to (ints ^ key)
+        t = [(_compact_word(plaintext[4 * i:4 * i + 4]) ^ self._Ke[0][i]) for i in xrange(0, 4)]
+
+        # Apply round transforms
+        for r in xrange(1, rounds):
+            for i in xrange(0, 4):
+                a[i] = (self.T1[(t[ i          ] >> 24) & 0xFF] ^
+                        self.T2[(t[(i + s1) % 4] >> 16) & 0xFF] ^
+                        self.T3[(t[(i + s2) % 4] >>  8) & 0xFF] ^
+                        self.T4[ t[(i + s3) % 4]        & 0xFF] ^
+                        self._Ke[r][i])
+            t = copy.copy(a)
+
+        # The last round is special
+        result = [ ]
+        for i in xrange(0, 4):
+            tt = self._Ke[rounds][i]
+            result.append((self.S[(t[ i           ] >> 24) & 0xFF] ^ (tt >> 24)) & 0xFF)
+            result.append((self.S[(t[(i + s1) % 4] >> 16) & 0xFF] ^ (tt >> 16)) & 0xFF)
+            result.append((self.S[(t[(i + s2) % 4] >>  8) & 0xFF] ^ (tt >>  8)) & 0xFF)
+            result.append((self.S[ t[(i + s3) % 4]        & 0xFF] ^  tt       ) & 0xFF)
+
+        return result
+
+    def decrypt(self, ciphertext):
+        'Decrypt a block of cipher text using the AES block cipher.'
+
+        if len(ciphertext) != 16:
+            raise ValueError('wrong block length')
+
+        rounds = len(self._Kd) - 1
+        (s1, s2, s3) = [3, 2, 1]
+        a = [0, 0, 0, 0]
+
+        # Convert ciphertext to (ints ^ key)
+        t = [(_compact_word(ciphertext[4 * i:4 * i + 4]) ^ self._Kd[0][i]) for i in xrange(0, 4)]
+
+        # Apply round transforms
+        for r in xrange(1, rounds):
+            for i in xrange(0, 4):
+                a[i] = (self.T5[(t[ i          ] >> 24) & 0xFF] ^
+                        self.T6[(t[(i + s1) % 4] >> 16) & 0xFF] ^
+                        self.T7[(t[(i + s2) % 4] >>  8) & 0xFF] ^
+                        self.T8[ t[(i + s3) % 4]        & 0xFF] ^
+                        self._Kd[r][i])
+            t = copy.copy(a)
+
+        # The last round is special
+        result = [ ]
+        for i in xrange(0, 4):
+            tt = self._Kd[rounds][i]
+            result.append((self.Si[(t[ i           ] >> 24) & 0xFF] ^ (tt >> 24)) & 0xFF)
+            result.append((self.Si[(t[(i + s1) % 4] >> 16) & 0xFF] ^ (tt >> 16)) & 0xFF)
+            result.append((self.Si[(t[(i + s2) % 4] >>  8) & 0xFF] ^ (tt >>  8)) & 0xFF)
+            result.append((self.Si[ t[(i + s3) % 4]        & 0xFF] ^  tt       ) & 0xFF)
+
+        return result
+
+
+class Counter(object):
+    '''A counter object for the Counter (CTR) mode of operation.
+
+       To create a custom counter, you can usually just override the
+       increment method.'''
+
+    def __init__(self, initial_value = 1):
+
+        # Convert the value into an array of bytes long
+        self._counter = [ ((initial_value >> i) % 256) for i in xrange(128 - 8, -1, -8) ]
+
+    value = property(lambda s: s._counter)
+
+    def increment(self):
+        '''Increment the counter (overflow rolls back to 0).'''
+
+        for i in xrange(len(self._counter) - 1, -1, -1):
+            self._counter[i] += 1
+
+            if self._counter[i] < 256: break
+
+            # Carry the one
+            self._counter[i] = 0
+
+        # Overflow
+        else:
+            self._counter = [ 0 ] * len(self._counter)
+
+
+class AESBlockModeOfOperation(object):
+    '''Super-class for AES modes of operation that require blocks.'''
+    def __init__(self, key):
+        self._aes = AES(key)
+
+    def decrypt(self, ciphertext):
+        raise Exception('not implemented')
+
+    def encrypt(self, plaintext):
+        raise Exception('not implemented')
+
+
+class AESStreamModeOfOperation(AESBlockModeOfOperation):
+    '''Super-class for AES modes of operation that are stream-ciphers.'''
+
+class AESSegmentModeOfOperation(AESStreamModeOfOperation):
+    '''Super-class for AES modes of operation that segment data.'''
+
+    segment_bytes = 16
+
+
+
+class AESModeOfOperationECB(AESBlockModeOfOperation):
+    '''AES Electronic Codebook Mode of Operation.
+
+       o Block-cipher, so data must be padded to 16 byte boundaries
+
+   Security Notes:
+       o This mode is not recommended
+       o Any two identical blocks produce identical encrypted values,
+         exposing data patterns. (See the image of Tux on wikipedia)
+
+   Also see:
+       o https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Electronic_codebook_.28ECB.29
+       o See NIST SP800-38A (http://csrc.nist.gov/publications/nistpubs/800-38a/sp800-38a.pdf); section 6.1'''
+
+
+    name = "Electronic Codebook (ECB)"
+
+    def encrypt(self, plaintext):
+        if len(plaintext) != 16:
+            raise ValueError('plaintext block must be 16 bytes')
+
+        plaintext = _string_to_bytes(plaintext)
+        return _bytes_to_string(self._aes.encrypt(plaintext))
+
+    def decrypt(self, ciphertext):
+        if len(ciphertext) != 16:
+            raise ValueError('ciphertext block must be 16 bytes')
+
+        ciphertext = _string_to_bytes(ciphertext)
+        return _bytes_to_string(self._aes.decrypt(ciphertext))
+
+
+
+class AESModeOfOperationCBC(AESBlockModeOfOperation):
+    '''AES Cipher-Block Chaining Mode of Operation.
+
+       o The Initialization Vector (IV)
+       o Block-cipher, so data must be padded to 16 byte boundaries
+       o An incorrect initialization vector will only cause the first
+         block to be corrupt; all other blocks will be intact
+       o A corrupt bit in the cipher text will cause a block to be
+         corrupted, and the next block to be inverted, but all other
+         blocks will be intact.
+
+   Security Notes:
+       o This method (and CTR) ARE recommended.
+
+   Also see:
+       o https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Cipher-block_chaining_.28CBC.29
+       o See NIST SP800-38A (http://csrc.nist.gov/publications/nistpubs/800-38a/sp800-38a.pdf); section 6.2'''
+
+
+    name = "Cipher-Block Chaining (CBC)"
+
+    def __init__(self, key, iv = None):
+        if iv is None:
+            self._last_cipherblock = [ 0 ] * 16
+        elif len(iv) != 16:
+            raise ValueError('initialization vector must be 16 bytes')
+        else:
+            self._last_cipherblock = _string_to_bytes(iv)
+
+        AESBlockModeOfOperation.__init__(self, key)
+
+    def encrypt(self, plaintext):
+        if len(plaintext) != 16:
+            raise ValueError('plaintext block must be 16 bytes')
+
+        plaintext = _string_to_bytes(plaintext)
+        precipherblock = [ (p ^ l) for (p, l) in zip(plaintext, self._last_cipherblock) ]
+        self._last_cipherblock = self._aes.encrypt(precipherblock)
+
+        return _bytes_to_string(self._last_cipherblock)
+
+    def decrypt(self, ciphertext):
+        if len(ciphertext) != 16:
+            raise ValueError('ciphertext block must be 16 bytes')
+
+        cipherblock = _string_to_bytes(ciphertext)
+        plaintext = [ (p ^ l) for (p, l) in zip(self._aes.decrypt(cipherblock), self._last_cipherblock) ]
+        self._last_cipherblock = cipherblock
+
+        return _bytes_to_string(plaintext)
+
+
+
+class AESModeOfOperationCFB(AESSegmentModeOfOperation):
+    '''AES Cipher Feedback Mode of Operation.
+
+       o A stream-cipher, so input does not need to be padded to blocks,
+         but does need to be padded to segment_size
+
+    Also see:
+       o https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Cipher_feedback_.28CFB.29
+       o See NIST SP800-38A (http://csrc.nist.gov/publications/nistpubs/800-38a/sp800-38a.pdf); section 6.3'''
+
+
+    name = "Cipher Feedback (CFB)"
+
+    def __init__(self, key, iv, segment_size = 1):
+        if segment_size == 0: segment_size = 1
+
+        if iv is None:
+            self._shift_register = [ 0 ] * 16
+        elif len(iv) != 16:
+            raise ValueError('initialization vector must be 16 bytes')
+        else:
+          self._shift_register = _string_to_bytes(iv)
+
+        self._segment_bytes = segment_size
+
+        AESBlockModeOfOperation.__init__(self, key)
+
+    segment_bytes = property(lambda s: s._segment_bytes)
+
+    def encrypt(self, plaintext):
+        if len(plaintext) % self._segment_bytes != 0:
+            raise ValueError('plaintext block must be a multiple of segment_size')
+
+        plaintext = _string_to_bytes(plaintext)
+
+        # Break block into segments
+        encrypted = [ ]
+        for i in xrange(0, len(plaintext), self._segment_bytes):
+            plaintext_segment = plaintext[i: i + self._segment_bytes]
+            xor_segment = self._aes.encrypt(self._shift_register)[:len(plaintext_segment)]
+            cipher_segment = [ (p ^ x) for (p, x) in zip(plaintext_segment, xor_segment) ]
+
+            # Shift the top bits out and the ciphertext in
+            self._shift_register = _concat_list(self._shift_register[len(cipher_segment):], cipher_segment)
+
+            encrypted.extend(cipher_segment)
+
+        return _bytes_to_string(encrypted)
+
+    def decrypt(self, ciphertext):
+        if len(ciphertext) % self._segment_bytes != 0:
+            raise ValueError('ciphertext block must be a multiple of segment_size')
+
+        ciphertext = _string_to_bytes(ciphertext)
+
+        # Break block into segments
+        decrypted = [ ]
+        for i in xrange(0, len(ciphertext), self._segment_bytes):
+            cipher_segment = ciphertext[i: i + self._segment_bytes]
+            xor_segment = self._aes.encrypt(self._shift_register)[:len(cipher_segment)]
+            plaintext_segment = [ (p ^ x) for (p, x) in zip(cipher_segment, xor_segment) ]
+
+            # Shift the top bits out and the ciphertext in
+            self._shift_register = _concat_list(self._shift_register[len(cipher_segment):], cipher_segment)
+
+            decrypted.extend(plaintext_segment)
+
+        return _bytes_to_string(decrypted)
+
+
+
+class AESModeOfOperationOFB(AESStreamModeOfOperation):
+    '''AES Output Feedback Mode of Operation.
+
+       o A stream-cipher, so input does not need to be padded to blocks,
+         allowing arbitrary length data.
+       o A bit twiddled in the cipher text, twiddles the same bit in the
+         same bit in the plain text, which can be useful for error
+         correction techniques.
+
+    Also see:
+       o https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Output_feedback_.28OFB.29
+       o See NIST SP800-38A (http://csrc.nist.gov/publications/nistpubs/800-38a/sp800-38a.pdf); section 6.4'''
+
+
+    name = "Output Feedback (OFB)"
+
+    def __init__(self, key, iv = None):
+        if iv is None:
+            self._last_precipherblock = [ 0 ] * 16
+        elif len(iv) != 16:
+            raise ValueError('initialization vector must be 16 bytes')
+        else:
+          self._last_precipherblock = _string_to_bytes(iv)
+
+        self._remaining_block = [ ]
+
+        AESBlockModeOfOperation.__init__(self, key)
+
+    def encrypt(self, plaintext):
+        encrypted = [ ]
+        for p in _string_to_bytes(plaintext):
+            if len(self._remaining_block) == 0:
+                self._remaining_block = self._aes.encrypt(self._last_precipherblock)
+                self._last_precipherblock = [ ]
+            precipherbyte = self._remaining_block.pop(0)
+            self._last_precipherblock.append(precipherbyte)
+            cipherbyte = p ^ precipherbyte
+            encrypted.append(cipherbyte)
+
+        return _bytes_to_string(encrypted)
+
+    def decrypt(self, ciphertext):
+        # AES-OFB is symetric
+        return self.encrypt(ciphertext)
+
+
+
+class AESModeOfOperationCTR(AESStreamModeOfOperation):
+    '''AES Counter Mode of Operation.
+
+       o A stream-cipher, so input does not need to be padded to blocks,
+         allowing arbitrary length data.
+       o The counter must be the same size as the key size (ie. len(key))
+       o Each block independant of the other, so a corrupt byte will not
+         damage future blocks.
+       o Each block has a uniue counter value associated with it, which
+         contributes to the encrypted value, so no data patterns are
+         leaked.
+       o Also known as: Counter Mode (CM), Integer Counter Mode (ICM) and
+         Segmented Integer Counter (SIC
+
+   Security Notes:
+       o This method (and CBC) ARE recommended.
+       o Each message block is associated with a counter value which must be
+         unique for ALL messages with the same key. Otherwise security may be
+         compromised.
+
+    Also see:
+
+       o https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Counter_.28CTR.29
+       o See NIST SP800-38A (http://csrc.nist.gov/publications/nistpubs/800-38a/sp800-38a.pdf); section 6.5
+         and Appendix B for managing the initial counter'''
+
+
+    name = "Counter (CTR)"
+
+    def __init__(self, key, counter = None):
+        AESBlockModeOfOperation.__init__(self, key)
+
+        if counter is None:
+            counter = Counter()
+
+        self._counter = counter
+        self._remaining_counter = [ ]
+
+    def encrypt(self, plaintext):
+        while len(self._remaining_counter) < len(plaintext):
+            self._remaining_counter += self._aes.encrypt(self._counter.value)
+            self._counter.increment()
+
+        plaintext = _string_to_bytes(plaintext)
+
+        encrypted = [ (p ^ c) for (p, c) in zip(plaintext, self._remaining_counter) ]
+        self._remaining_counter = self._remaining_counter[len(encrypted):]
+
+        return _bytes_to_string(encrypted)
+
+    def decrypt(self, crypttext):
+        # AES-CTR is symetric
+        return self.encrypt(crypttext)
+
+
+# Simple lookup table for each mode
+AESModesOfOperation = dict(
+    ctr = AESModeOfOperationCTR,
+    cbc = AESModeOfOperationCBC,
+    cfb = AESModeOfOperationCFB,
+    ecb = AESModeOfOperationECB,
+    ofb = AESModeOfOperationOFB,
+)
@@ -0,0 +1,227 @@
+# The MIT License (MIT)
+#
+# Copyright (c) 2014 Richard Moore
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+# THE SOFTWARE.
+
+
+from .aes import AESBlockModeOfOperation, AESSegmentModeOfOperation, AESStreamModeOfOperation
+from .util import append_PKCS7_padding, strip_PKCS7_padding, to_bufferable
+
+
+# First we inject three functions to each of the modes of operations
+#
+#    _can_consume(size)
+#       - Given a size, determine how many bytes could be consumed in
+#         a single call to either the decrypt or encrypt method
+#
+#    _final_encrypt(data, padding = PADDING_DEFAULT)
+#       - call and return encrypt on this (last) chunk of data,
+#         padding as necessary; this will always be at least 16
+#         bytes unless the total incoming input was less than 16
+#         bytes
+#
+#    _final_decrypt(data, padding = PADDING_DEFAULT)
+#       - same as _final_encrypt except for decrypt, for
+#         stripping off padding
+#
+
+PADDING_NONE       = 'none'
+PADDING_DEFAULT    = 'default'
+
+# @TODO: Ciphertext stealing and explicit PKCS#7
+# PADDING_CIPHERTEXT_STEALING
+# PADDING_PKCS7
+
+# ECB and CBC are block-only ciphers
+
+def _block_can_consume(self, size):
+    if size >= 16: return 16
+    return 0
+
+# After padding, we may have more than one block
+def _block_final_encrypt(self, data, padding = PADDING_DEFAULT):
+    if padding == PADDING_DEFAULT:
+        data = append_PKCS7_padding(data)
+
+    elif padding == PADDING_NONE:
+        if len(data) != 16:
+            raise Exception('invalid data length for final block')
+    else:
+        raise Exception('invalid padding option')
+
+    if len(data) == 32:
+        return self.encrypt(data[:16]) + self.encrypt(data[16:])
+
+    return self.encrypt(data)
+
+
+def _block_final_decrypt(self, data, padding = PADDING_DEFAULT):
+    if padding == PADDING_DEFAULT:
+        return strip_PKCS7_padding(self.decrypt(data))
+
+    if padding == PADDING_NONE:
+        if len(data) != 16:
+            raise Exception('invalid data length for final block')
+        return self.decrypt(data)
+
+    raise Exception('invalid padding option')
+
+AESBlockModeOfOperation._can_consume = _block_can_consume
+AESBlockModeOfOperation._final_encrypt = _block_final_encrypt
+AESBlockModeOfOperation._final_decrypt = _block_final_decrypt
+
+
+
+# CFB is a segment cipher
+
+def _segment_can_consume(self, size):
+    return self.segment_bytes * int(size // self.segment_bytes)
+
+# CFB can handle a non-segment-sized block at the end using the remaining cipherblock
+def _segment_final_encrypt(self, data, padding = PADDING_DEFAULT):
+    if padding != PADDING_DEFAULT:
+        raise Exception('invalid padding option')
+
+    faux_padding = (chr(0) * (self.segment_bytes - (len(data) % self.segment_bytes)))
+    padded = data + to_bufferable(faux_padding)
+    return self.encrypt(padded)[:len(data)]
+
+# CFB can handle a non-segment-sized block at the end using the remaining cipherblock
+def _segment_final_decrypt(self, data, padding = PADDING_DEFAULT):
+    if padding != PADDING_DEFAULT:
+        raise Exception('invalid padding option')
+
+    faux_padding = (chr(0) * (self.segment_bytes - (len(data) % self.segment_bytes)))
+    padded = data + to_bufferable(faux_padding)
+    return self.decrypt(padded)[:len(data)]
+
+AESSegmentModeOfOperation._can_consume = _segment_can_consume
+AESSegmentModeOfOperation._final_encrypt = _segment_final_encrypt
+AESSegmentModeOfOperation._final_decrypt = _segment_final_decrypt
+
+
+
+# OFB and CTR are stream ciphers
+
+def _stream_can_consume(self, size):
+    return size
+
+def _stream_final_encrypt(self, data, padding = PADDING_DEFAULT):
+    if padding not in [PADDING_NONE, PADDING_DEFAULT]:
+        raise Exception('invalid padding option')
+
+    return self.encrypt(data)
+
+def _stream_final_decrypt(self, data, padding = PADDING_DEFAULT):
+    if padding not in [PADDING_NONE, PADDING_DEFAULT]:
+        raise Exception('invalid padding option')
+
+    return self.decrypt(data)
+
+AESStreamModeOfOperation._can_consume = _stream_can_consume
+AESStreamModeOfOperation._final_encrypt = _stream_final_encrypt
+AESStreamModeOfOperation._final_decrypt = _stream_final_decrypt
+
+
+
+class BlockFeeder(object):
+    '''The super-class for objects to handle chunking a stream of bytes
+       into the appropriate block size for the underlying mode of operation
+       and applying (or stripping) padding, as necessary.'''
+
+    def __init__(self, mode, feed, final, padding = PADDING_DEFAULT):
+        self._mode = mode
+        self._feed = feed
+        self._final = final
+        self._buffer = to_bufferable("")
+        self._padding = padding
+
+    def feed(self, data = None):
+        '''Provide bytes to encrypt (or decrypt), returning any bytes
+           possible from this or any previous calls to feed.
+
+           Call with None or an empty string to flush the mode of
+           operation and return any final bytes; no further calls to
+           feed may be made.'''
+
+        if self._buffer is None:
+            raise ValueError('already finished feeder')
+
+        # Finalize; process the spare bytes we were keeping
+        if data is None:
+            result = self._final(self._buffer, self._padding)
+            self._buffer = None
+            return result
+
+        self._buffer += to_bufferable(data)
+
+        # We keep 16 bytes around so we can determine padding
+        result = to_bufferable('')
+        while len(self._buffer) > 16:
+            can_consume = self._mode._can_consume(len(self._buffer) - 16)
+            if can_consume == 0: break
+            result += self._feed(self._buffer[:can_consume])
+            self._buffer = self._buffer[can_consume:]
+
+        return result
+
+
+class Encrypter(BlockFeeder):
+    'Accepts bytes of plaintext and returns encrypted ciphertext.'
+
+    def __init__(self, mode, padding = PADDING_DEFAULT):
+        BlockFeeder.__init__(self, mode, mode.encrypt, mode._final_encrypt, padding)
+
+
+class Decrypter(BlockFeeder):
+    'Accepts bytes of ciphertext and returns decrypted plaintext.'
+
+    def __init__(self, mode, padding = PADDING_DEFAULT):
+        BlockFeeder.__init__(self, mode, mode.decrypt, mode._final_decrypt, padding)
+
+
+# 8kb blocks
+BLOCK_SIZE = (1 << 13)
+
+def _feed_stream(feeder, in_stream, out_stream, block_size = BLOCK_SIZE):
+    'Uses feeder to read and convert from in_stream and write to out_stream.'
+
+    while True:
+        chunk = in_stream.read(block_size)
+        if not chunk:
+            break
+        converted = feeder.feed(chunk)
+        out_stream.write(converted)
+    converted = feeder.feed()
+    out_stream.write(converted)
+
+
+def encrypt_stream(mode, in_stream, out_stream, block_size = BLOCK_SIZE, padding = PADDING_DEFAULT):
+    'Encrypts a stream of bytes from in_stream to out_stream using mode.'
+
+    encrypter = Encrypter(mode, padding = padding)
+    _feed_stream(encrypter, in_stream, out_stream, block_size)
+
+
+def decrypt_stream(mode, in_stream, out_stream, block_size = BLOCK_SIZE, padding = PADDING_DEFAULT):
+    'Decrypts a stream of bytes from in_stream to out_stream using mode.'
+
+    decrypter = Decrypter(mode, padding = padding)
+    _feed_stream(decrypter, in_stream, out_stream, block_size)
@@ -0,0 +1,60 @@
+# The MIT License (MIT)
+#
+# Copyright (c) 2014 Richard Moore
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+# THE SOFTWARE.
+
+# Why to_bufferable?
+# Python 3 is very different from Python 2.x when it comes to strings of text
+# and strings of bytes; in Python 3, strings of bytes do not exist, instead to
+# represent arbitrary binary data, we must use the "bytes" object. This method
+# ensures the object behaves as we need it to.
+
+def to_bufferable(binary):
+    return binary
+
+def _get_byte(c):
+    return ord(c)
+
+try:
+    xrange
+except:
+
+    def to_bufferable(binary):
+        if isinstance(binary, bytes):
+            return binary
+        return bytes(ord(b) for b in binary)
+
+    def _get_byte(c):
+        return c
+
+def append_PKCS7_padding(data):
+    pad = 16 - (len(data) % 16)
+    return data + to_bufferable(chr(pad) * pad)
+
+def strip_PKCS7_padding(data):
+    if len(data) % 16 != 0:
+        raise ValueError("invalid length")
+
+    pad = _get_byte(data[-1])
+
+    if pad > 16:
+        raise ValueError("invalid padding byte")
+
+    return data[:-pad]
@@ -163,3 +163,13 @@ class Pysubs2CLI(object):
        elif args.transform_framerate is not None:
            in_fps, out_fps = args.transform_framerate
            subs.transform_framerate(in_fps, out_fps)
+
+
+def __main__():
+    cli = Pysubs2CLI()
+    rv = cli(sys.argv[1:])
+    sys.exit(rv)
+
+
+if __name__ == "__main__":
+    __main__()
@@ -17,12 +17,14 @@ class Color(_Color):
        return _Color.__new__(cls, r, g, b, a)

 #: Version of the pysubs2 library.
-VERSION = "0.2.1"
+VERSION = "0.2.3"


 PY3 = sys.version_info.major == 3

 if PY3:
    text_type = str
+    binary_string_type = bytes
 else:
    text_type = unicode
+    binary_string_type = str
@@ -12,3 +12,6 @@ class UnknownFormatIdentifierError(Pysubs2Error):

 class FormatAutodetectionError(Pysubs2Error):
    """Subtitle format is ambiguous or unknown."""
+
+class ContentNotUsable(Pysubs2Error):
+    """Current content not usable for specified format"""
@@ -3,7 +3,7 @@ from .microdvd import MicroDVDFormat
 from .subrip import SubripFormat
 from .jsonformat import JSONFormat
 from .substation import SubstationFormat
-from .txt_generic import TXTGenericFormat, MPL2Format
+from .mpl2 import MPL2Format
 from .exceptions import *

 #: Dict mapping file extensions to format identifiers.
@@ -13,7 +13,6 @@ FILE_EXTENSION_TO_FORMAT_IDENTIFIER = {
    ".ssa": "ssa",
    ".sub": "microdvd",
    ".json": "json",
-    ".txt": "txt_generic",
 }

 #: Dict mapping format identifiers to implementations (FormatBase subclasses).
@@ -23,7 +22,6 @@ FORMAT_IDENTIFIER_TO_FORMAT_CLASS = {
    "ssa": SubstationFormat,
    "microdvd": MicroDVDFormat,
    "json": JSONFormat,
-    "txt_generic": TXTGenericFormat,
    "mpl2": MPL2Format,
 }

@@ -2,44 +2,48 @@

 from __future__ import print_function, division, unicode_literals
 import re
-from numbers import Number

-from pysubs2.time import times_to_ms
+from .time import times_to_ms
 from .formatbase import FormatBase
 from .ssaevent import SSAEvent
-from .ssastyle import SSAStyle


 # thanks to http://otsaloma.io/gaupol/doc/api/aeidon.files.mpl2_source.html
-MPL2_FORMAT = re.compile(r"^(?um)\[(-?\d+)\]\[(-?\d+)\](.*?)$")
-
-
-class TXTGenericFormat(FormatBase):
-    @classmethod
-    def guess_format(cls, text):
-        if MPL2_FORMAT.match(text):
-            return "mpl2"
+MPL2_FORMAT = re.compile(r"^(?um)\[(-?\d+)\]\[(-?\d+)\](.*)")


 class MPL2Format(FormatBase):
    @classmethod
    def guess_format(cls, text):
-        return TXTGenericFormat.guess_format(text)
+        if MPL2_FORMAT.search(text):
+            return "mpl2"

    @classmethod
    def from_file(cls, subs, fp, format_, **kwargs):
        def prepare_text(lines):
            out = []
            for s in lines.split("|"):
+                s = s.strip()
+
                if s.startswith("/"):
-                    out.append(r"{\i1}%s{\i0}" % s[1:])
-                    continue
+                    # line beginning with '/' is in italics
+                    s = r"{\i1}%s{\i0}" % s[1:].strip()
+
                out.append(s)
-            return "\n".join(out)
+            return "\\N".join(out)

        subs.events = [SSAEvent(start=times_to_ms(s=float(start) / 10), end=times_to_ms(s=float(end) / 10),
                       text=prepare_text(text)) for start, end, text in MPL2_FORMAT.findall(fp.getvalue())]

    @classmethod
    def to_file(cls, subs, fp, format_, **kwargs):
-        raise NotImplemented
+
+        # TODO handle italics
+        for line in subs:
+            if line.is_comment:
+                continue
+
+            print("[{start}][{end}] {text}".format(start=int(line.start // 100),
+                                                   end=int(line.end // 100),
+                                                   text=line.plaintext.replace("\n", "|")),
+                  file=fp)
@@ -41,6 +41,7 @@ class SSAStyle(object):
        self.italic = False #: Italic
        self.underline = False #: Underline (ASS only)
        self.strikeout = False #: Strikeout (ASS only)
+        self.drawing = False #: Drawing (ASS only, see http://docs.aegisub.org/3.1/ASS_Tags/#drawing-tags
        self.scalex = 100.0 #: Horizontal scaling (ASS only)
        self.scaley = 100.0 #: Vertical scaling (ASS only)
        self.spacing = 0.0 #: Letter spacing (ASS only)
@@ -78,7 +79,7 @@ class SSAStyle(object):
        s += "%rpx " % self.fontsize
        if self.bold: s += "bold "
        if self.italic: s += "italic "
-        s += "'%s'>" % self.fontname
+        s += "{!r}>".format(self.fontname)
        if not PY3: s = s.encode("utf-8")
        return s

@@ -5,6 +5,7 @@ from .formatbase import FormatBase
 from .ssaevent import SSAEvent
 from .ssastyle import SSAStyle
 from .substation import parse_tags
+from .exceptions import ContentNotUsable
 from .time import ms_to_times, make_time, TIMESTAMP, timestamp_to_ms

 #: Largest timestamp allowed in SubRip, ie. 99:59:59,999.
@@ -46,8 +47,16 @@ class SubripFormat(FormatBase):
                    following_lines[-1].append(line)

        def prepare_text(lines):
+            # Handle the "happy" empty subtitle case, which is timestamp line followed by blank line(s)
+            # followed by number line and timestamp line of the next subtitle. Fixes issue #11.
+            if (len(lines) >= 2
+                    and all(re.match("\s*$", line) for line in lines[:-1])
+                    and re.match("\s*\d+\s*$", lines[-1])):
+                return ""
+
+            # Handle the general case.
            s = "".join(lines).strip()
-            s = re.sub(r"\n* *\d+ *$", "", s) # strip number of next subtitle
+            s = re.sub(r"\n+ *\d+ *$", "", s) # strip number of next subtitle
            s = re.sub(r"< *i *>", r"{\i1}", s)
            s = re.sub(r"< */ *i *>", r"{\i0}", s)
            s = re.sub(r"< *s *>", r"{\s1}", s)
@@ -73,6 +82,7 @@ class SubripFormat(FormatBase):
                if sty.italic: fragment = "<i>%s</i>" % fragment
                if sty.underline: fragment = "<u>%s</u>" % fragment
                if sty.strikeout: fragment = "<s>%s</s>" % fragment
+                if sty.drawing: raise ContentNotUsable
                body.append(fragment)

            return re.sub("\n+", "\n", "".join(body).strip())
@@ -82,7 +92,10 @@ class SubripFormat(FormatBase):
        for i, line in enumerate(visible_lines, 1):
            start = ms_to_timestamp(line.start)
            end = ms_to_timestamp(line.end)
-            text = prepare_text(line.text, subs.styles.get(line.style, SSAStyle.DEFAULT_STYLE))
+            try:
+                text = prepare_text(line.text, subs.styles.get(line.style, SSAStyle.DEFAULT_STYLE))
+            except ContentNotUsable:
+                continue

            print("%d" % i, file=fp) # Python 2.7 compat
            print(start, "-->", end, file=fp)
@@ -4,7 +4,7 @@ from numbers import Number
 from .formatbase import FormatBase
 from .ssaevent import SSAEvent
 from .ssastyle import SSAStyle
-from .common import text_type, Color
+from .common import text_type, Color, PY3, binary_string_type
 from .time import make_time, ms_to_times, timestamp_to_ms, TIMESTAMP

 SSA_ALIGNMENT = (1, 2, 3, 9, 10, 11, 5, 6, 7)
@@ -110,7 +110,7 @@ def parse_tags(text, style=SSAStyle.DEFAULT_STYLE, styles={}):
    
    def apply_overrides(all_overrides):
        s = style.copy()
-        for tag in re.findall(r"\\[ibus][10]|\\r[a-zA-Z_0-9 ]*", all_overrides):
+        for tag in re.findall(r"\\[ibusp][0-9]|\\r[a-zA-Z_0-9 ]*", all_overrides):
            if tag == r"\r":
                s = style.copy() # reset to original line style
            elif tag.startswith(r"\r"):
@@ -122,6 +122,13 @@ def parse_tags(text, style=SSAStyle.DEFAULT_STYLE, styles={}):
                elif "b" in tag: s.bold = "1" in tag
                elif "u" in tag: s.underline = "1" in tag
                elif "s" in tag: s.strikeout = "1" in tag
+                elif "p" in tag:
+                    try:
+                        scale = int(tag[2:])
+                    except (ValueError, IndexError):
+                        continue
+
+                    s.drawing = scale > 0
        return s
    
    overrides = SSAEvent.OVERRIDE_SEQUENCE.findall(text)
@@ -150,14 +157,7 @@ class SubstationFormat(FormatBase):
                if format_ == "ass":
                    return ass_rgba_to_color(v)
                else:
-                    try:
-                        return ssa_rgb_to_color(v)
-                    except ValueError:
-                        try:
-                            return ass_rgba_to_color(v)
-                        except:
-                            return Color(255, 255, 255, 0)
-
+                    return ssa_rgb_to_color(v)
            elif f in {"bold", "underline", "italic", "strikeout"}:
                return v == "-1"
            elif f in {"borderstyle", "encoding", "marginl", "marginr", "marginv", "layer", "alphalevel"}:
@@ -229,7 +229,7 @@ class SubstationFormat(FormatBase):
            for k, v in subs.aegisub_project.items():
                print(k, v, sep=": ", file=fp)

-        def field_to_string(f, v):
+        def field_to_string(f, v, line):
            if f in {"start", "end"}:
                return ms_to_timestamp(v)
            elif f == "marked":
@@ -240,23 +240,31 @@ class SubstationFormat(FormatBase):
                return "-1" if v else "0"
            elif isinstance(v, (text_type, Number)):
                return text_type(v)
+            elif not PY3 and isinstance(v, binary_string_type):
+                # A convenience feature, see issue #12 - accept non-unicode strings
+                # when they are ASCII; this is useful in Python 2, especially for non-text
+                # fields like style names, where requiring Unicode type seems too stringent
+                if all(ord(c) < 128 for c in v):
+                    return text_type(v)
+                else:
+                    raise TypeError("Encountered binary string with non-ASCII codepoint in SubStation field {!r} for line {!r} - please use unicode string instead of str".format(f, line))
            elif isinstance(v, Color):
                if format_ == "ass":
                    return color_to_ass_rgba(v)
                else:
                    return color_to_ssa_rgb(v)
            else:
-                raise TypeError("Unexpected type when writing a SubStation field")
+                raise TypeError("Unexpected type when writing a SubStation field {!r} for line {!r}".format(f, line))

        print("\n[V4+ Styles]" if format_ == "ass" else "\n[V4 Styles]", file=fp)
        print(STYLE_FORMAT_LINE[format_], file=fp)
        for name, sty in subs.styles.items():
-            fields = [field_to_string(f, getattr(sty, f)) for f in STYLE_FIELDS[format_]]
+            fields = [field_to_string(f, getattr(sty, f), sty) for f in STYLE_FIELDS[format_]]
            print("Style: %s" % name, *fields, sep=",", file=fp)

        print("\n[Events]", file=fp)
        print(EVENT_FORMAT_LINE[format_], file=fp)
        for ev in subs.events:
-            fields = [field_to_string(f, getattr(ev, f)) for f in EVENT_FIELDS[format_]]
+            fields = [field_to_string(f, getattr(ev, f), ev) for f in EVENT_FIELDS[format_]]
            print(ev.type, end=": ", file=fp)
            print(*fields, sep=",", file=fp)
@@ -27,3 +27,8 @@ class ServiceUnavailable(ProviderError):
 class DownloadLimitExceeded(ProviderError):
    """Exception raised by providers when download limit is exceeded."""
    pass
+
+
+class DownloadLimitPerDayExceeded(ProviderError):
+    """Exception raised by providers when download limit is exceeded."""
+    pass
@@ -94,7 +94,9 @@ provider_manager = RegistrableExtensionManager('subliminal.providers', [
    'podnapisi = subliminal.providers.podnapisi:PodnapisiProvider',
    'shooter = subliminal.providers.shooter:ShooterProvider',
    'thesubdb = subliminal.providers.thesubdb:TheSubDBProvider',
-    'tvsubtitles = subliminal.providers.tvsubtitles:TVsubtitlesProvider'
+    'tvsubtitles = subliminal.providers.tvsubtitles:TVsubtitlesProvider',
+    'ktuvit = subliminal.providers.ktuvit:KtuvitProvider'
+    'wizdom = subliminal.providers.wizdom:WizdomProvider'
 ])

 #: Refiner manager
@@ -258,4 +258,4 @@ def fix_line_ending(content):
    :rtype: bytes

    """
-    return content.replace(b'\r\n', b'\n').replace(b'\r', b'\n')
+    return content.replace(b'\r\n', b'\n')
@@ -12,6 +12,13 @@ from_subscene = {
        'Malay': 'msa', 'Pashto': 'pus', 'Punjabi': 'pan', 'Swahili': 'swa'
 }

+from_subscene_with_country = {
+    'Brazillian Portuguese': ('por', 'BR')
+}
+
+to_subscene_with_country = {val: key for key, val in from_subscene_with_country.items()}
+
+
 to_subscene = {v: k for k, v in from_subscene.items()}

 exact_languages_alpha3 = [
@@ -34,12 +41,12 @@ language_ids = {
        'mkd': 48, 'mal': 64, 'mni': 65, 'mon': 72, 'pus': 67, 'pol': 31,
        'por': 32, 'pan': 66, 'rus': 34, 'srp': 35, 'sin': 58, 'slk': 36,
        'slv': 37, 'som': 70, 'tgl': 53, 'tam': 59, 'tel': 63, 'tha': 40,
-        'tur': 41, 'ukr': 56, 'urd': 42, 'yor': 71
+        'tur': 41, 'ukr': 56, 'urd': 42, 'yor': 71, 'pt-BR': 4
 }

 # TODO: specify codes for unspecified_languages
 unspecified_languages = [
-        'Big 5 code', 'Brazillian Portuguese', 'Bulgarian/ English',
+        'Big 5 code', 'Bulgarian/ English',
        'Chinese BG code', 'Dutch/ English', 'English/ German',
        'Hungarian/ English', 'Rohingya'
 ]
@@ -50,6 +57,8 @@ alpha3_of_code = {l.name: l.alpha3 for l in supported_languages}

 supported_languages.update({Language(l) for l in to_subscene})

+supported_languages.update({Language(lang, cr) for lang, cr in to_subscene_with_country})
+

 class SubsceneConverter(LanguageReverseConverter):
    codes = {l.name for l in supported_languages}
@@ -61,9 +70,15 @@ class SubsceneConverter(LanguageReverseConverter):
        if alpha3 in to_subscene:
            return to_subscene[alpha3]

+        if (alpha3, country) in to_subscene_with_country:
+            return to_subscene_with_country[(alpha3, country)]
+
        raise ConfigurationError('Unsupported language for subscene: %s, %s, %s' % (alpha3, country, script))

    def reverse(self, code):
+        if code in from_subscene_with_country:
+            return from_subscene_with_country[code]
+
        if code in from_subscene:
            return (from_subscene[code],)

@@ -27,16 +27,6 @@ class TitloviConverter(LanguageReverseConverter):
                           }
        self.codes = set(self.from_titlovi.keys())

-        # temporary fix, should be removed as soon as API is used
-        self.lang_from_countrycode = {'ba': ('bos',),
-                                 'en': ('eng',),
-                                 'hr': ('hrv',),
-                                 'mk': ('mkd',),
-                                 'rs': ('srp',),
-                                 'rsc': ('srp', None, 'Cyrl'),
-                                 'si': ('slv',)
-                                 }
-
    def convert(self, alpha3, country=None, script=None):
        if (alpha3, country, script) in self.to_titlovi:
            return self.to_titlovi[(alpha3, country, script)]
@@ -49,9 +39,5 @@ class TitloviConverter(LanguageReverseConverter):
        if titlovi in self.from_titlovi:
            return self.from_titlovi[titlovi]

-        # temporary fix, should be removed as soon as API is used
-        if titlovi in self.lang_from_countrycode:
-            return self.lang_from_countrycode[titlovi]
-
        raise ConfigurationError('Unsupported language number for titlovi: %s' % titlovi)

@@ -30,7 +30,7 @@ from subliminal.core import guessit, ProviderPool, io, is_windows_special_path,
    ThreadPoolExecutor, check_video
 from subliminal_patch.exceptions import TooManyRequests, APIThrottled

-from subzero.language import Language
+from subzero.language import Language, ENDSWITH_LANGUAGECODE_RE, FULL_LANGUAGE_LIST
 from scandir import scandir, scandir_generic as _scandir_generic

 logger = logging.getLogger(__name__)
@@ -62,7 +62,7 @@ class SZProviderPool(ProviderPool):
    def __init__(self, providers=None, provider_configs=None, blacklist=None, throttle_callback=None,
                 pre_download_hook=None, post_download_hook=None, language_hook=None):
        #: Name of providers to use
-        self.providers = providers or provider_registry.names()
+        self.providers = providers

        #: Provider configuration
        self.provider_configs = provider_configs or {}
@@ -108,10 +108,12 @@ class SZProviderPool(ProviderPool):
        try:
            logger.info('Terminating provider %s', name)
            self.initialized_providers[name].terminate()
-        except (requests.Timeout, socket.timeout):
+        except (requests.Timeout, socket.timeout) as e:
            logger.error('Provider %r timed out, improperly terminated', name)
-        except:
+            self.throttle_callback(name, e)
+        except Exception as e:
            logger.exception('Provider %r terminated unexpectedly', name)
+            self.throttle_callback(name, e)

        del self.initialized_providers[name]

@@ -183,15 +185,13 @@ class SZProviderPool(ProviderPool):

            return out

-        except (requests.Timeout, socket.timeout):
-            logger.error('Provider %r timed out', provider)
-
-        except (TooManyRequests, DownloadLimitExceeded, ServiceUnavailable, APIThrottled), e:
+        except (requests.Timeout, socket.timeout) as e:
+            logger.exception('Provider %r timed out', provider)
            self.throttle_callback(provider, e)
-            return

-        except:
+        except Exception as e:
            logger.exception('Unexpected error in provider %r: %s', provider, traceback.format_exc())
+            self.throttle_callback(provider, e)

    def list_subtitles(self, video, languages):
        """List subtitles.
@@ -266,10 +266,11 @@ class SZProviderPool(ProviderPool):
                    requests.exceptions.ProxyError,
                    requests.exceptions.SSLError,
                    requests.Timeout,
-                    socket.timeout):
-                logger.error('Provider %r connection error', subtitle.provider_name)
+                    socket.timeout) as e:
+                logger.exception('Provider %r connection error', subtitle.provider_name)
+                self.throttle_callback(subtitle.provider_name, e)

-            except ResponseNotReady:
+            except ResponseNotReady as e:
                logger.error('Provider %r response error, reinitializing', subtitle.provider_name)
                try:
                    self[subtitle.provider_name].terminate()
@@ -277,20 +278,17 @@ class SZProviderPool(ProviderPool):
                except:
                    logger.error('Provider %r reinitialization error: %s', subtitle.provider_name,
                                 traceback.format_exc())
+                    self.throttle_callback(subtitle.provider_name, e)

            except rarfile.BadRarFile:
                logger.error('Malformed RAR file from provider %r, skipping subtitle.', subtitle.provider_name)
                logger.debug("RAR Traceback: %s", traceback.format_exc())
                return False

-            except (TooManyRequests, DownloadLimitExceeded, ServiceUnavailable, APIThrottled), e:
-                self.throttle_callback(subtitle.provider_name, e)
-                self.discarded_providers.add(subtitle.provider_name)
-                return False
-
-            except:
+            except Exception as e:
                logger.exception('Unexpected error in provider %r, Traceback: %s', subtitle.provider_name,
                                 traceback.format_exc())
+                self.throttle_callback(subtitle.provider_name, e)
                self.discarded_providers.add(subtitle.provider_name)
                return False

@@ -309,7 +307,8 @@ class SZProviderPool(ProviderPool):
            logger.error('Invalid subtitle')
            return False

-        subtitle.normalize()
+        if not os.environ.get("SZ_KEEP_ENCODING", False):
+            subtitle.normalize()

        return True

@@ -360,15 +359,16 @@ class SZProviderPool(ProviderPool):
            orig_matches = matches.copy()

            logger.debug('%r: Found matches %r', s, matches)
+            score, score_without_hash = compute_score(matches, s, video, hearing_impaired=use_hearing_impaired)
            unsorted_subtitles.append(
-                (s, compute_score(matches, s, video, hearing_impaired=use_hearing_impaired), matches, orig_matches))
+                (s, score, score_without_hash, matches, orig_matches))

        # sort subtitles by score
-        scored_subtitles = sorted(unsorted_subtitles, key=operator.itemgetter(1), reverse=True)
+        scored_subtitles = sorted(unsorted_subtitles, key=operator.itemgetter(1, 2), reverse=True)

        # download best subtitles, falling back on the next on error
        downloaded_subtitles = []
-        for subtitle, score, matches, orig_matches in scored_subtitles:
+        for subtitle, score, score_without_hash, matches, orig_matches in scored_subtitles:
            # check score
            if score < min_score:
                logger.info('%r: Score %d is below min_score (%d)', subtitle, score, min_score)
@@ -472,7 +472,7 @@ if is_windows_special_path:
    SZAsyncProviderPool = SZProviderPool


-def scan_video(path, dont_use_actual_file=False, hints=None, providers=None, skip_hashing=False):
+def scan_video(path, dont_use_actual_file=False, hints=None, providers=None, skip_hashing=False, hash_from=None):
    """Scan a video from a `path`.

    patch:
@@ -529,7 +529,7 @@ def scan_video(path, dont_use_actual_file=False, hints=None, providers=None, ski
    video.hints = hints

    # get possibly alternative title from the filename itself
-    alt_guess = guessit(filename, options=hints)
+    alt_guess = guessit(filename, options={k: v for k, v in hints.items() if k not in ('expected_title', 'title')})
    if "title" in alt_guess and alt_guess["title"] != guessed_result["title"]:
        if video_type == "episode":
            video.alternative_series.append(alt_guess["title"])
@@ -537,28 +537,41 @@ def scan_video(path, dont_use_actual_file=False, hints=None, providers=None, ski
            video.alternative_titles.append(alt_guess["title"])
        logger.debug("Adding alternative title: %s", alt_guess["title"])

-    if dont_use_actual_file:
+    if dont_use_actual_file and not hash_from:
        return video

    # size and hashes
    if not skip_hashing:
-        video.size = os.path.getsize(path)
+        hash_path = hash_from or path
+        video.size = os.path.getsize(hash_path)
        if video.size > 10485760:
            logger.debug('Size is %d', video.size)
+            osub_hash = None
+
+            if "bsplayer" in providers:
+                video.hashes['bsplayer'] = osub_hash = hash_opensubtitles(hash_path)
+
            if "opensubtitles" in providers:
-                video.hashes['opensubtitles'] = hash_opensubtitles(path)
+                video.hashes['opensubtitles'] = osub_hash = osub_hash or hash_opensubtitles(hash_path)
+
+            if "opensubtitlescom" in providers:
+                video.hashes['opensubtitlescom'] = osub_hash = osub_hash or hash_opensubtitles(hash_path)

            if "shooter" in providers:
-                video.hashes['shooter'] = hash_shooter(path)
+                video.hashes['shooter'] = hash_shooter(hash_path)

            if "thesubdb" in providers:
-                video.hashes['thesubdb'] = hash_thesubdb(path)
+                video.hashes['thesubdb'] = hash_thesubdb(hash_path)

            if "napiprojekt" in providers:
                try:
-                    video.hashes['napiprojekt'] = hash_napiprojekt(path)
+                    video.hashes['napiprojekt'] = hash_napiprojekt(hash_path)
                except MemoryError:
-                    logger.warning(u"Couldn't compute napiprojekt hash for %s", path)
+                    logger.warning(u"Couldn't compute napiprojekt hash for %s", hash_path)
+
+            if "napisy24" in providers:
+                # Napisy24 uses the same hash as opensubtitles
+                video.hashes['napisy24'] = osub_hash or hash_opensubtitles(hash_path)

            logger.debug('Computed hashes %r', video.hashes)
        else:
@@ -567,14 +580,16 @@ def scan_video(path, dont_use_actual_file=False, hints=None, providers=None, ski
    return video


-def _search_external_subtitles(path, languages=None, only_one=False, scandir_generic=False):
+def _search_external_subtitles(path, languages=None, only_one=False, scandir_generic=False, match_strictness="strict"):
    dirpath, filename = os.path.split(path)
    dirpath = dirpath or '.'
-    fileroot, fileext = os.path.splitext(filename)
+    fn_no_ext, fileext = os.path.splitext(filename)
+    fn_no_ext_lower = fn_no_ext.lower()
    subtitles = {}
    _scandir = _scandir_generic if scandir_generic else scandir
+
    for entry in _scandir(dirpath):
-        if not entry.name and not scandir_generic:
+        if (not entry.name or entry.name in ('\x0c', '$', ',', '\x7f')) and not scandir_generic:
            logger.debug('Could not determine the name of the file, retrying with scandir_generic')
            return _search_external_subtitles(path, languages, only_one, True)
        if not entry.is_file(follow_symlinks=False):
@@ -583,9 +598,11 @@ def _search_external_subtitles(path, languages=None, only_one=False, scandir_gen
        p = entry.name

        # keep only valid subtitle filenames
-        if not p.lower().startswith(fileroot.lower()) or not p.lower().endswith(SUBTITLE_EXTENSIONS):
+        if not p.lower().endswith(SUBTITLE_EXTENSIONS):
            continue

+        # not p.lower().startswith(fileroot.lower()) or not
+
        p_root, p_ext = os.path.splitext(p)
        if not INCLUDE_EXOTIC_SUBS and p_ext not in (".srt", ".ass", ".ssa", ".vtt"):
            continue
@@ -603,22 +620,34 @@ def _search_external_subtitles(path, languages=None, only_one=False, scandir_gen
        if adv_tag:
            forced = "forced" in adv_tag

+        # remove possible language code for matching
+        p_root_bare = ENDSWITH_LANGUAGECODE_RE.sub(
+            lambda m: "" if str(m.group(1)).lower() in FULL_LANGUAGE_LIST else m.group(0), p_root)
+
+        p_root_lower = p_root_bare.lower()
+
+        filename_matches = p_root_lower == fn_no_ext_lower
+        filename_contains = p_root_lower in fn_no_ext_lower
+
+        if not filename_matches:
+            if match_strictness == "strict" or (match_strictness == "loose" and not filename_contains):
+                continue
+
+        language = None
+
        # extract the potential language code
-        language_code = p_root[len(fileroot):].replace('_', '-')[1:]
-
-        # default language is undefined
-        language = Language('und')
-
-        # attempt to parse
-        if language_code:
+        try:
+            language_code = p_root.rsplit(".", 1)[1].replace('_', '-')
            try:
                language = Language.fromietf(language_code)
                language.forced = forced
-            except ValueError:
+            except (ValueError, LanguageReverseError):
                logger.error('Cannot parse language code %r', language_code)
-                language = None
+                language_code = None
+        except IndexError:
+            language_code = None

-        elif not language_code and only_one:
+        if not language and not language_code and only_one:
            language = Language.rebuild(list(languages)[0], forced=forced)

        subtitles[p] = language
@@ -628,7 +657,7 @@ def _search_external_subtitles(path, languages=None, only_one=False, scandir_gen
    return subtitles


-def search_external_subtitles(path, languages=None, only_one=False):
+def search_external_subtitles(path, languages=None, only_one=False, match_strictness="strict"):
    """
    wrap original search_external_subtitles function to search multiple paths for one given video
    # todo: cleanup and merge with _search_external_subtitles
@@ -649,10 +678,11 @@ def search_external_subtitles(path, languages=None, only_one=False):
        if os.path.isdir(os.path.dirname(abspath)):
            try:
                subtitles.update(_search_external_subtitles(abspath, languages=languages,
-                                                            only_one=only_one))
+                                                            only_one=only_one, match_strictness=match_strictness))
            except OSError:
                subtitles.update(_search_external_subtitles(abspath, languages=languages,
-                                                            only_one=only_one, scandir_generic=True))
+                                                            only_one=only_one, match_strictness=match_strictness,
+                                                            scandir_generic=True))
    logger.debug("external subs: found %s", subtitles)
    return subtitles

@@ -845,6 +875,9 @@ def save_subtitles(file_path, subtitles, single=False, directory=None, chmod=Non
            logger.debug(u"Saving %r to %r", subtitle, subtitle_path)
            content = subtitle.get_modified_content(format=format, debug=debug_mods)
            if content:
+                if os.path.exists(subtitle_path):
+                    os.remove(subtitle_path)
+
                with open(subtitle_path, 'w') as f:
                    f.write(content)
                subtitle.storage_path = subtitle_path
@@ -9,3 +9,8 @@ class TooManyRequests(ProviderError):

 class APIThrottled(ProviderError):
    pass
+
+
+class ParseResponseError(ProviderError):
+    """Exception raised by providers when they are not able to parse the response."""
+    pass
@@ -11,6 +11,7 @@ import requests
 import xmlrpclib
 import dns.resolver
 import ipaddress
+import re

 from requests import exceptions
 from urllib3.util import connection
@@ -18,8 +19,14 @@ from retry.api import retry_call
 from exceptions import APIThrottled
 from dogpile.cache.api import NO_VALUE
 from subliminal.cache import region
+from subliminal_patch.pitcher import pitchers
 from cloudscraper import CloudScraper

+try:
+    import brotli
+except:
+    pass
+
 try:
    from urlparse import urlparse
 except ImportError:
@@ -28,12 +35,8 @@ except ImportError:
 from subzero.lib.io import get_viable_encoding

 logger = logging.getLogger(__name__)
-pem_file = os.path.normpath(os.path.join(os.path.dirname(os.path.realpath(unicode(__file__, get_viable_encoding()))), "..", certifi.where()))
-try:
-    default_ssl_context = ssl.create_default_context(cafile=pem_file)
-except AttributeError:
-    # < Python 2.7.9
-    default_ssl_context = None
+pem_file = os.path.normpath(os.path.join(os.path.dirname(os.path.realpath(unicode(__file__, get_viable_encoding()))),
+                                         "..", certifi.where()))


 class TimeoutSession(requests.Session):
@@ -51,18 +54,86 @@ class TimeoutSession(requests.Session):


 class CertifiSession(TimeoutSession):
-    def __init__(self):
+    def __init__(self, verify=None):
        super(CertifiSession, self).__init__()
-        self.verify = pem_file
+        self.verify = verify or pem_file
+
+
+class NeedsCaptchaException(Exception):
+    pass


 class CFSession(CloudScraper):
-    _hdrs = None
-
    def __init__(self, *args, **kwargs):
        super(CFSession, self).__init__(*args, **kwargs)
        self.debug = os.environ.get("CF_DEBUG", False)
-        self._was_cf = False
+
+    def _request(self, method, url, *args, **kwargs):
+        ourSuper = super(CloudScraper, self)
+        resp = ourSuper.request(method, url, *args, **kwargs)
+
+        if resp.headers.get('Content-Encoding') == 'br':
+            if self.allow_brotli and resp._content:
+                resp._content = brotli.decompress(resp.content)
+            else:
+                logging.warning('Brotli content detected, But option is disabled, we will not continue.')
+                return resp
+
+        # Debug request
+        if self.debug:
+            self.debugRequest(resp)
+
+        # Check if Cloudflare anti-bot is on
+        try:
+            if self.isChallengeRequest(resp):
+                if resp.request.method != 'GET':
+                    # Work around if the initial request is not a GET,
+                    # Supersede with a GET then re-request the original METHOD.
+                    CloudScraper.request(self, 'GET', resp.url)
+                    resp = ourSuper.request(method, url, *args, **kwargs)
+                else:
+                    # Solve Challenge
+                    resp = self.sendChallengeResponse(resp, **kwargs)
+
+        except ValueError, e:
+            if e.message == "Captcha":
+                parsed_url = urlparse(url)
+                domain = parsed_url.netloc
+                # solve the captcha
+                site_key = re.search(r'data-sitekey="(.+?)"', resp.content).group(1)
+                challenge_s = re.search(r'type="hidden" name="s" value="(.+?)"', resp.content).group(1)
+                challenge_ray = re.search(r'data-ray="(.+?)"', resp.content).group(1)
+                if not all([site_key, challenge_s, challenge_ray]):
+                    raise Exception("cf: Captcha site-key not found!")
+
+                pitcher = pitchers.get_pitcher()("cf: %s" % domain, resp.request.url, site_key,
+                                                 user_agent=self.headers["User-Agent"],
+                                                 cookies=self.cookies.get_dict(),
+                                                 is_invisible=True)
+
+                parsed_url = urlparse(resp.url)
+                logger.info("cf: %s: Solving captcha", domain)
+                result = pitcher.throw()
+                if not result:
+                    raise Exception("cf: Couldn't solve captcha!")
+
+                submit_url = '{}://{}/cdn-cgi/l/chk_captcha'.format(parsed_url.scheme, domain)
+                method = resp.request.method
+
+                cloudflare_kwargs = {
+                    'allow_redirects': False,
+                    'headers': {'Referer': resp.url},
+                    'params': OrderedDict(
+                        [
+                            ('s', challenge_s),
+                            ('g-recaptcha-response', result)
+                        ]
+                    )
+                }
+
+                return CloudScraper.request(self, method, submit_url, **cloudflare_kwargs)
+
+        return resp

    def request(self, method, url, *args, **kwargs):
        parsed_url = urlparse(url)
@@ -80,20 +151,19 @@ class CFSession(CloudScraper):

                self.headers = hdrs

-        ret = super(CFSession, self).request(method, url, *args, **kwargs)
+        ret = self._request(method, url, *args, **kwargs)

-        if self.was_cf_request:
-            self.was_cf_request = False
-            logger.debug("We've hit CF, seeing if we need to store previous data")
-            try:
-                cf_data = self.get_cf_live_tokens(domain)
-            except:
-                logger.debug("Couldn't get CF live tokens for re-use. Cookies: %r", self.cookies)
-                pass
-            else:
-                if cf_data != region.get(cache_key) and cf_data[0]["cf_clearance"]:
+        try:
+            cf_data = self.get_cf_live_tokens(domain)
+        except:
+            pass
+        else:
+            if cf_data and "cf_clearance" in cf_data[0] and cf_data[0]["cf_clearance"]:
+                if cf_data != region.get(cache_key):
                    logger.debug("Storing cf data for %s: %s", domain, cf_data)
                    region.set(cache_key, cf_data)
+                elif cf_data[0]["cf_clearance"]:
+                    logger.debug("CF Live tokens not updated")

        return ret

@@ -167,12 +237,20 @@ class SubZeroRequestsTransport(xmlrpclib.SafeTransport):
    # change our user agent to reflect Requests
    user_agent = "Python XMLRPC with Requests (python-requests.org)"
    proxies = None
+    xm_ver = 1
+    session_var = "PHPSESSID"

    def __init__(self, use_https=True, verify=None, user_agent=None, timeout=10, *args, **kwargs):
        self.verify = pem_file if verify is None else verify
        self.use_https = use_https
        self.user_agent = user_agent if user_agent is not None else self.user_agent
        self.timeout = timeout
+        self.session = requests.Session()
+        self.session.headers['User-Agent'] = self.user_agent
+        # if 'requests' in self.session.headers['User-Agent']:
+        #     # Set a random User-Agent if no custom User-Agent has been set
+        #     self.session.headers = User_Agent(allow_brotli=False).headers
+
        proxy = os.environ.get('SZ_HTTP_PROXY')
        if proxy:
            self.proxies = {
@@ -186,18 +264,40 @@ class SubZeroRequestsTransport(xmlrpclib.SafeTransport):
        """
        Make an xmlrpc request.
        """
-        headers = {'User-Agent': self.user_agent}
        url = self._build_url(host, handler)
+        cache_key = "xm%s_%s" % (self.xm_ver, host)
+
+        old_sessvar = self.session.cookies.get(self.session_var, "")
+        if not old_sessvar:
+            data = region.get(cache_key)
+            if data is not NO_VALUE:
+                logger.debug("Trying to re-use headers/cookies for %s" % host)
+                self.session.cookies, self.session.headers = data
+                old_sessvar = self.session.cookies.get(self.session_var, "")
+
        try:
-            resp = requests.post(url, data=request_body, headers=headers,
-                                 stream=True, timeout=self.timeout, proxies=self.proxies,
-                                 verify=self.verify)
+            resp = self.session.post(url, data=request_body,
+                                     stream=True, timeout=self.timeout, proxies=self.proxies,
+                                     verify=self.verify)
+
+            if self.session_var in resp.cookies and resp.cookies[self.session_var] != old_sessvar:
+                logger.debug("Storing %s cookies" % host)
+                region.set(cache_key, [self.session.cookies, self.session.headers])
        except ValueError:
+            logger.debug("Wiping cookies/headers cache (VE) for %s" % host)
+            region.delete(cache_key)
            raise
        except Exception:
+            logger.debug("Wiping cookies/headers cache (EX) for %s" % host)
+            region.delete(cache_key)
            raise  # something went wrong
        else:
-            resp.raise_for_status()
+            try:
+                resp.raise_for_status()
+            except requests.exceptions.HTTPError:
+                logger.debug("Wiping cookies/headers cache (RE) for %s" % host)
+                region.delete(cache_key)
+                raise

            try:
                if 'x-ratelimit-remaining' in resp.headers and int(resp.headers['x-ratelimit-remaining']) <= 2:
@@ -280,7 +380,6 @@ def patch_create_connection():
                        return _orig_create_connection((ip, port), *args, **kwargs)
                    except dns.exception.DNSException:
                        logger.warning("DNS: Couldn't resolve %s with DNS: %s", host, custom_resolver.nameservers)
-                        raise

        logger.debug("DNS: Falling back to default DNS or IP on %s", host)
        return _orig_create_connection((host, port), *args, **kwargs)
@@ -2,15 +2,20 @@
 import logging
 import re
 import datetime
+import types
+
 import subliminal
 import time

 from random import randint
+
+from dogpile.cache.api import NO_VALUE
 from requests import Session
 from subliminal.cache import region
-from subliminal.exceptions import DownloadLimitExceeded, AuthenticationError
+from subliminal.exceptions import DownloadLimitExceeded, AuthenticationError, ConfigurationError, \
+    DownloadLimitPerDayExceeded
 from subliminal.providers.addic7ed import Addic7edProvider as _Addic7edProvider, \
-    Addic7edSubtitle as _Addic7edSubtitle, ParserBeautifulSoup, show_cells_re
+    Addic7edSubtitle as _Addic7edSubtitle, ParserBeautifulSoup
 from subliminal.subtitle import fix_line_ending
 from subliminal_patch.utils import sanitize
 from subliminal_patch.exceptions import TooManyRequests
@@ -19,6 +24,8 @@ from subzero.language import Language

 logger = logging.getLogger(__name__)

+show_cells_re = re.compile(b'<td class="(?:version|vr)">.*?</td>', re.DOTALL)
+
 #: Series header parsing regex
 series_year_re = re.compile(r'^(?P<series>[ \w\'.:(),*&!?-]+?)(?: \((?P<year>\d{4})\))?$')

@@ -60,16 +67,22 @@ class Addic7edProvider(_Addic7edProvider):
        'slk', 'slv', 'spa', 'sqi', 'srp', 'swe', 'tha', 'tur', 'ukr', 'vie', 'zho'
    ]} | {Language.fromietf(l) for l in ["sr-Latn", "sr-Cyrl"]}

+    vip = False
    USE_ADDICTED_RANDOM_AGENTS = False
    hearing_impaired_verifiable = True
    subtitle_class = Addic7edSubtitle
    server_url = 'https://www.addic7ed.com/'

    sanitize_characters = {'-', ':', '(', ')', '.', '/'}
+    last_show_ids_fetch_key = "addic7ed_last_id_fetch"

-    def __init__(self, username=None, password=None, use_random_agents=False):
+    def __init__(self, username=None, password=None, use_random_agents=False, is_vip=False):
        super(Addic7edProvider, self).__init__(username=username, password=password)
        self.USE_ADDICTED_RANDOM_AGENTS = use_random_agents
+        self.vip = is_vip
+
+        if not all((username, password)):
+            raise ConfigurationError('Username and password must be specified')

    def initialize(self):
        self.session = Session()
@@ -101,13 +114,18 @@ class Addic7edProvider(_Addic7edProvider):
                    'remember': 'true'}

            tries = 0
-            while tries < 3:
+            while tries <= 3:
+                tries += 1
                r = self.session.get(self.server_url + 'login.php', timeout=10, headers={"Referer": self.server_url})
-                if "grecaptcha" in r.content:
+                if "g-recaptcha" in r.content or "grecaptcha" in r.content:
                    logger.info('Addic7ed: Solving captcha. This might take a couple of minutes, but should only '
                                'happen once every so often')

-                    site_key = re.search(r'grecaptcha.execute\(\'(.+?)\',', r.content).group(1)
+                    for g, s in (("g-recaptcha-response", r'g-recaptcha.+?data-sitekey=\"(.+?)\"'),
+                                 ("recaptcha_response", r'grecaptcha.execute\(\'(.+?)\',')):
+                        site_key = re.search(s, r.content).group(1)
+                        if site_key:
+                            break
                    if not site_key:
                        logger.error("Addic7ed: Captcha site-key not found!")
                        return
@@ -119,23 +137,31 @@ class Addic7edProvider(_Addic7edProvider):

                    result = pitcher.throw()
                    if not result:
-                        raise Exception("Addic7ed: Couldn't solve captcha!")
+                        if tries >= 3:
+                            raise Exception("Addic7ed: Couldn't solve captcha!")
+                        logger.info("Addic7ed: Couldn't solve captcha! Retrying")
+                        time.sleep(4)
+                        continue

-                    data["recaptcha_response"] = result
+                    data[g] = result

+                time.sleep(1)
                r = self.session.post(self.server_url + 'dologin.php', data, allow_redirects=False, timeout=10,
                                      headers={"Referer": self.server_url + "login.php"})

                if "relax, slow down" in r.content:
                    raise TooManyRequests(self.username)

-                if r.status_code != 302:
-                    if "User <b></b> doesn't exist" in r.content and tries <= 2:
-                        logger.info("Addic7ed: Error, trying again. (%s/%s)", tries+1, 3)
-                        tries += 1
-                        continue
-
+                if "Wrong password" in r.content or "doesn't exist" in r.content:
                    raise AuthenticationError(self.username)
+
+                if r.status_code != 302:
+                    if tries >= 3:
+                        logger.error("Addic7ed: Something went wrong when logging in")
+                        raise AuthenticationError(self.username)
+                    logger.info("Addic7ed: Something went wrong when logging in; retrying")
+                    time.sleep(4)
+                    continue
                break

            store_verification("addic7ed", self.session)
@@ -143,10 +169,12 @@ class Addic7edProvider(_Addic7edProvider):
            logger.debug('Addic7ed: Logged in')
            self.logged_in = True

+            time.sleep(2)
+
    def terminate(self):
        self.session.close()

-    def get_show_id(self, series, year=None, country_code=None):
+    def get_show_id(self, series, year=None, country_code=None, ignore_cache=False):
        """Get the best matching show id for `series`, `year` and `country_code`.

        First search in the result of :meth:`_get_show_ids` and fallback on a search with :meth:`_search_show_id`.
@@ -158,32 +186,45 @@ class Addic7edProvider(_Addic7edProvider):
        :type country_code: str
        :return: the show id, if found.
        :rtype: int
-
        """
-        series_sanitized = sanitize(series).lower()
-        show_ids = self._get_show_ids()
        show_id = None
+        ids_to_look_for = {sanitize(series).lower(), sanitize(series.replace(".", "")).lower()}
+        show_ids = self._get_show_ids()
+        if ignore_cache or not show_ids:
+            show_ids = self._get_show_ids.refresh(self)

-        # attempt with country
-        if not show_id and country_code:
-            logger.debug('Getting show id with country')
-            show_id = show_ids.get('%s %s' % (series_sanitized, country_code.lower()))
+        logger.debug("Trying show ids: %s", ids_to_look_for)
+        for series_sanitized in ids_to_look_for:
+            # attempt with country
+            if not show_id and country_code:
+                logger.debug('Getting show id with country')
+                show_id = show_ids.get('%s %s' % (series_sanitized, country_code.lower()))

-        # attempt with year
-        if not show_id and year:
-            logger.debug('Getting show id with year')
-            show_id = show_ids.get('%s %d' % (series_sanitized, year))
+            # attempt with year
+            if not show_id and year:
+                logger.debug('Getting show id with year')
+                show_id = show_ids.get('%s %d' % (series_sanitized, year))

-        # attempt clean
-        if not show_id:
-            logger.debug('Getting show id')
-            show_id = show_ids.get(series_sanitized)
+            # attempt clean
+            if not show_id:
+                logger.debug('Getting show id')
+                show_id = show_ids.get(series_sanitized)

-        # search as last resort
-        # broken right now
-        # if not show_id:
-        #     logger.warning('Series %s not found in show ids', series)
-        #     show_id = self._search_show_id(series)
+                if not show_id:
+                    now = datetime.datetime.now()
+                    last_fetch = region.get(self.last_show_ids_fetch_key)
+
+                    # re-fetch show ids once per day if any show ID not found
+                    if not ignore_cache and last_fetch != NO_VALUE and last_fetch + datetime.timedelta(days=1) < now:
+                        logger.info("Show id not found; re-fetching show ids")
+                        return self.get_show_id(series, year=year, country_code=country_code, ignore_cache=True)
+                    logger.debug("Not refreshing show ids, as the last fetch has been too recent")
+
+            # search as last resort
+            # broken right now
+            # if not show_id:
+            #     logger.warning('Series %s not found in show ids', series)
+            #     show_id = self._search_show_id(series)

        return show_id

@@ -197,6 +238,8 @@ class Addic7edProvider(_Addic7edProvider):
        """
        # get the show page
        logger.info('Getting show ids')
+        region.set(self.last_show_ids_fetch_key, datetime.datetime.now())
+
        r = self.session.get(self.server_url + 'shows.php', timeout=10)
        r.raise_for_status()

@@ -205,14 +248,15 @@ class Addic7edProvider(_Addic7edProvider):
        # Assuming the site's markup is bad, and stripping it down to only contain what's needed.
        show_cells = re.findall(show_cells_re, r.content)
        if show_cells:
-            soup = ParserBeautifulSoup(b''.join(show_cells), ['lxml', 'html.parser'])
+            soup = ParserBeautifulSoup(b''.join(show_cells).decode('utf-8', 'ignore'), ['lxml', 'html.parser'])
        else:
            # If RegEx fails, fall back to original r.content and use 'html.parser'
            soup = ParserBeautifulSoup(r.content, ['html.parser'])

        # populate the show ids
        show_ids = {}
-        for show in soup.select('td > h3 > a[href^="/show/"]'):
+        shows = soup.select('td > h3 > a[href^="/show/"]')
+        for show in shows:
            show_clean = sanitize(show.text, default_characters=self.sanitize_characters)
            try:
                show_id = int(show['href'][6:])
@@ -230,6 +274,9 @@ class Addic7edProvider(_Addic7edProvider):

        logger.debug('Found %d show ids', len(show_ids))

+        if not show_ids:
+            raise Exception("Addic7ed: No show IDs found!")
+
        return show_ids

    @region.cache_on_arguments(expiration_time=SHOW_EXPIRATION_TIME)
@@ -329,7 +376,7 @@ class Addic7edProvider(_Addic7edProvider):

            # ignore incomplete subtitles
            status = cells[5].text
-            if status != 'Completed':
+            if "%" in status:
                logger.debug('Ignoring subtitle with status %s', status)
                continue

@@ -355,6 +402,27 @@ class Addic7edProvider(_Addic7edProvider):
        return subtitles

    def download_subtitle(self, subtitle):
+        last_dls = region.get("addic7ed_dls")
+        now = datetime.datetime.now()
+        one_day = datetime.timedelta(hours=24)
+
+        def raise_limit():
+            logger.info("Addic7ed: Downloads per day exceeded (%s)", cap)
+            raise DownloadLimitPerDayExceeded
+
+        if not isinstance(last_dls, types.ListType):
+            last_dls = []
+        else:
+            # filter all non-expired DLs
+            last_dls = filter(lambda t: t + one_day > now, last_dls)
+            region.set("addic7ed_dls", last_dls)
+
+        cap = self.vip and 80 or 40
+        amount = len(last_dls)
+
+        if amount >= cap:
+            raise_limit()
+
        # download the subtitle
        r = self.session.get(self.server_url + subtitle.download_link, headers={'Referer': subtitle.page_link},
                             timeout=10)
@@ -366,7 +434,7 @@ class Addic7edProvider(_Addic7edProvider):
        if not r.content:
            # Provider wrongful return a status of 304 Not Modified with an empty content
            # raise_for_status won't raise exception for that status code
-            logger.error('Unable to download subtitle. No data returned from provider')
+            logger.error('Addic7ed: Unable to download subtitle. No data returned from provider')
            return

        # detect download limit exceeded
@@ -374,3 +442,10 @@ class Addic7edProvider(_Addic7edProvider):
            raise DownloadLimitExceeded

        subtitle.content = fix_line_ending(r.content)
+        last_dls.append(datetime.datetime.now())
+        region.set("addic7ed_dls", last_dls)
+        logger.info("Addic7ed: Used %s/%s downloads", amount + 1, cap)
+
+        if amount + 1 >= cap:
+            raise_limit()
+
@@ -23,9 +23,10 @@ class ArgenteamSubtitle(Subtitle):
    hearing_impaired_verifiable = False
    _release_info = None

-    def __init__(self, language, download_link, movie_kind, title, season, episode, year, release, version, source,
+    def __init__(self, language, page_link, download_link, movie_kind, title, season, episode, year, release, version, source,
                 video_codec, tvdb_id, imdb_id, asked_for_episode=None, asked_for_release_group=None, *args, **kwargs):
-        super(ArgenteamSubtitle, self).__init__(language, download_link, *args, **kwargs)
+        super(ArgenteamSubtitle, self).__init__(language, page_link=page_link, *args, **kwargs)
+        self.page_link = page_link
        self.download_link = download_link
        self.movie_kind = movie_kind
        self.title = title
@@ -135,7 +136,8 @@ class ArgenteamProvider(Provider, ProviderSubtitleArchiveMixin):
    provider_name = 'argenteam'
    languages = {Language.fromalpha2(l) for l in ['es']}
    video_types = (Episode, Movie)
-    API_URL = "http://argenteam.net/api/v1/"
+    BASE_URL = "https://argenteam.net/"
+    API_URL = BASE_URL + "api/v1/"
    subtitle_class = ArgenteamSubtitle
    hearing_impaired_verifiable = False
    language_list = list(languages)
@@ -240,12 +242,15 @@ class ArgenteamProvider(Provider, ProviderSubtitleArchiveMixin):

            for r in content['releases']:
                for s in r['subtitles']:
-                    sub = ArgenteamSubtitle(language, s['uri'], "episode" if is_episode else "movie", returned_title,
+                    movie_kind = "episode" if is_episode else "movie"
+                    page_link = self.BASE_URL + movie_kind + "/" + str(aid)
+                    # use https and new domain
+                    download_link = s['uri'].replace('http://www.argenteam.net/', self.BASE_URL)
+                    sub = ArgenteamSubtitle(language, page_link, download_link, movie_kind, returned_title,
                                            season, episode, year, r.get('team'), r.get('tags'),
                                            r.get('source'), r.get('codec'), content.get("tvdb"), imdb_id,
                                            asked_for_release_group=video.release_group,
-                                            asked_for_episode=episode
-                                            )
+                                            asked_for_episode=episode)
                    subtitles.append(sub)

            if has_multiple_ids:
@@ -0,0 +1,213 @@
+# -*- coding: utf-8 -*-
+from __future__ import absolute_import
+import logging
+import io
+import os
+
+from requests import Session
+from guessit import guessit
+from subliminal_patch.providers import Provider
+from subliminal_patch.subtitle import Subtitle
+from subliminal.utils import sanitize_release_group
+from subliminal.subtitle import guess_matches
+from subzero.language import Language
+
+import gzip
+import random
+from time import sleep
+from xml.etree import ElementTree
+
+logger = logging.getLogger(__name__)
+
+
+class BSPlayerSubtitle(Subtitle):
+    """BSPlayer Subtitle."""
+    provider_name = 'bsplayer'
+    hash_verifiable = True
+
+    def __init__(self, language, filename, subtype, video, link):
+        super(BSPlayerSubtitle, self).__init__(language)
+        self.language = language
+        self.filename = filename
+        self.page_link = link
+        self.subtype = subtype
+        self.video = video
+
+    @property
+    def id(self):
+        return self.page_link
+
+    @property
+    def release_info(self):
+        return self.filename
+
+    def get_matches(self, video):
+        matches = set()
+        matches |= guess_matches(video, guessit(self.filename))
+        matches.add('hash')
+
+        return matches
+
+
+class BSPlayerProvider(Provider):
+    """BSPlayer Provider."""
+    languages = {Language('por', 'BR')} | {Language(l) for l in [
+        'ara', 'bul', 'ces', 'dan', 'deu', 'ell', 'eng', 'fin', 'fra', 'hun', 'ita', 'jpn', 'kor', 'nld', 'pol', 'por',
+        'ron', 'rus', 'spa', 'swe', 'tur', 'ukr', 'zho'
+    ]}
+    SEARCH_THROTTLE = 8
+    hash_verifiable = True
+
+    # batantly based on kodi's bsplayer plugin
+    # also took from BSPlayer-Subtitles-Downloader
+    def __init__(self):
+        self.initialize()
+
+    def initialize(self):
+        self.session = Session()
+        self.search_url = self.get_sub_domain()
+        self.token = None
+        self.login()
+
+    def terminate(self):
+        self.session.close()
+        self.logout()
+
+    def api_request(self, func_name='logIn', params='', tries=5):
+        headers = {
+            'User-Agent': 'BSPlayer/2.x (1022.12360)',
+            'Content-Type': 'text/xml; charset=utf-8',
+            'Connection': 'close',
+            'SOAPAction': '"http://api.bsplayer-subtitles.com/v1.php#{func_name}"'.format(func_name=func_name)
+        }
+        data = (
+            '<?xml version="1.0" encoding="UTF-8"?>\n'
+            '<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" '
+            'xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" '
+            'xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" '
+            'xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:ns1="{search_url}">'
+            '<SOAP-ENV:Body SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">'
+            '<ns1:{func_name}>{params}</ns1:{func_name}></SOAP-ENV:Body></SOAP-ENV:Envelope>'
+        ).format(search_url=self.search_url, func_name=func_name, params=params)
+        logger.info('Sending request: %s.' % func_name)
+        for i in iter(range(tries)):
+            try:
+                self.session.headers.update(headers.items())
+                res = self.session.post(self.search_url, data)
+                return ElementTree.fromstring(res.text)
+
+            except Exception as ex:
+                logger.info("ERROR: %s." % ex)
+                if func_name == 'logIn':
+                    self.search_url = self.get_sub_domain()
+
+                sleep(1)
+        logger.info('ERROR: Too many tries (%d)...' % tries)
+        raise Exception('Too many tries...')
+
+    def login(self):
+        # If already logged in
+        if self.token:
+            return True
+
+        root = self.api_request(
+            func_name='logIn',
+            params=('<username></username>'
+                    '<password></password>'
+                    '<AppID>BSPlayer v2.67</AppID>')
+        )
+        res = root.find('.//return')
+        if res.find('status').text == 'OK':
+            self.token = res.find('data').text
+            logger.info("Logged In Successfully.")
+            return True
+        return False
+
+    def logout(self):
+        # If already logged out / not logged in
+        if not self.token:
+            return True
+
+        root = self.api_request(
+            func_name='logOut',
+            params='<handle>{token}</handle>'.format(token=self.token)
+        )
+        res = root.find('.//return')
+        self.token = None
+        if res.find('status').text == 'OK':
+            logger.info("Logged Out Successfully.")
+            return True
+        return False
+
+    def query(self, video, video_hash, language):
+        if not self.login():
+            return []
+
+        if isinstance(language, (tuple, list, set)):
+            # language_ids = ",".join(language)
+            # language_ids = 'spa'
+            language_ids = ','.join(sorted(l.opensubtitles for l in language))
+
+        if video.imdb_id is None:
+            imdbId = '*'
+        else:
+            imdbId = video.imdb_id
+        sleep(self.SEARCH_THROTTLE)
+        root = self.api_request(
+            func_name='searchSubtitles',
+            params=(
+                '<handle>{token}</handle>'
+                '<movieHash>{movie_hash}</movieHash>'
+                '<movieSize>{movie_size}</movieSize>'
+                '<languageId>{language_ids}</languageId>'
+                '<imdbId>{imdbId}</imdbId>'
+            ).format(token=self.token, movie_hash=video_hash,
+                     movie_size=video.size, language_ids=language_ids, imdbId=imdbId)
+        )
+        res = root.find('.//return/result')
+        if res.find('status').text != 'OK':
+            return []
+
+        items = root.findall('.//return/data/item')
+        subtitles = []
+        if items:
+            logger.info("Subtitles Found.")
+            for item in items:
+                subID = item.find('subID').text
+                subDownloadLink = item.find('subDownloadLink').text
+                subLang = Language.fromopensubtitles(item.find('subLang').text)
+                subName = item.find('subName').text
+                subFormat = item.find('subFormat').text
+                subtitles.append(
+                    BSPlayerSubtitle(subLang, subName, subFormat, video, subDownloadLink)
+                )
+        return subtitles
+
+    def list_subtitles(self, video, languages):
+        return self.query(video, video.hashes['bsplayer'], languages)
+
+    def get_sub_domain(self):
+        # s1-9, s101-109
+        SUB_DOMAINS = ['s1', 's2', 's3', 's4', 's5', 's6', 's7', 's8', 's9',
+                       's101', 's102', 's103', 's104', 's105', 's106', 's107', 's108', 's109']
+        API_URL_TEMPLATE = "http://{sub_domain}.api.bsplayer-subtitles.com/v1.php"
+        sub_domains_end = len(SUB_DOMAINS) - 1
+        return API_URL_TEMPLATE.format(sub_domain=SUB_DOMAINS[random.randint(0, sub_domains_end)])
+
+    def download_subtitle(self, subtitle):
+        session = Session()
+        _addheaders = {
+            'User-Agent': 'Mozilla/4.0 (compatible; Synapse)'
+        }
+        session.headers.update(_addheaders)
+        res = session.get(subtitle.page_link)
+        if res:
+            if res.text == '500':
+                raise ValueError('Error 500 on server')
+
+            with gzip.GzipFile(fileobj=io.BytesIO(res.content)) as gf:
+                subtitle.content = gf.read()
+                subtitle.normalize()
+
+            return subtitle
+        raise ValueError('Problems conecting to the server')
@@ -0,0 +1,417 @@
+# -*- coding: utf-8 -*-
+import re
+from requests import Session
+from requests.cookies import RequestsCookieJar
+import json
+import logging
+from subzero.language import Language
+from bs4 import BeautifulSoup
+from guessit import guessit
+
+from subliminal_patch.providers import Provider
+from subliminal.providers import Episode, Movie
+from subliminal_patch.utils import sanitize
+from subliminal_patch.subtitle import Subtitle, guess_matches
+from subliminal.subtitle import fix_line_ending
+from subliminal.exceptions import ConfigurationError, AuthenticationError
+
+from string import hexdigits
+from collections import defaultdict
+import pbkdf2
+from base64 import b64decode, b64encode
+from hashlib import sha256
+import pyaes
+
+__author__ = "Dor Nizar"
+
+logger = logging.getLogger(__name__)
+
+
+class KtuvitSubtitle(Subtitle):
+    provider_name = 'ktuvit'
+
+    def __init__(self, language, title_id, subtitle_id, series, season, episode, release, year):
+        super(KtuvitSubtitle, self).__init__(language, subtitle_id)
+        self.title_id = title_id
+        self.subtitle_id = subtitle_id
+        self.series = series
+        self.season = season
+        self.episode = episode
+        self.release = release
+        self.year = year
+
+    def get_matches(self, video):
+        matches = set()
+        logger.debug("[{}]\n{}".format(self.__class__.__name__, self.__dict__))
+
+        # episode
+        if isinstance(video, Episode):
+            # series
+            if video.series and sanitize(self.series) == sanitize(video.series):
+                matches.add('series')
+            # season
+            if video.season and self.season == video.season:
+                matches.add('season')
+            # episode
+            if video.episode and self.episode == video.episode:
+                matches.add('episode')
+            # guess
+            matches |= guess_matches(video, guessit(self.release, {'type': 'episode'}))
+        # movie
+        elif isinstance(video, Movie):
+            # title
+            if video.title and (sanitize(self.series) in (
+                    sanitize(name) for name in [video.title] + video.alternative_titles)):
+                matches.add('title')
+            # year
+            if video.year and self.year == video.year:
+                matches.add('year')
+            # guess
+            matches |= guess_matches(video, guessit(self.release, {'type': 'movie'}))
+
+        logger.debug("Ktuvit subtitle criteria match:\n{}".format(matches))
+        return matches
+
+    @property
+    def id(self):
+        return self.subtitle_id
+
+
+class KtuvitProvider(Provider):
+    subtitle_class = KtuvitSubtitle
+    languages = {Language.fromalpha2(l) for l in ['he']}
+    URL_SERVER = 'https://www.ktuvit.me/'
+
+    URI_LOGIN = 'Login.aspx'
+    URI_LOGIN_POST = 'Services/MembershipService.svc/Login'
+    URI_SEARCH_TITLE = 'Services/ContentProvider.svc/GetSearchForecast'
+    URI_SEARCH_SERIES_SUBTITLE = 'Services/GetModuleAjax.ashx'
+    URI_SEARCH_MOVIE_SUBTITLE = "MovieInfo.aspx"
+    URI_REQ_SUBTITLE_ID = "Services/ContentProvider.svc/RequestSubtitleDownload"
+    URI_DOWNLOAD_SUBTITLE = "Services/DownloadFile.ashx"
+
+    def __init__(self, username=None, password=None):
+        if not all((username, password)):
+            raise ConfigurationError('Username and password must be specified')
+
+        self.session = None
+        self.username = username
+        self.password = password
+        self.encrypted_password = None
+        self.salt = None
+
+    def encrypt_password(self):
+        logger.debug("password: {}".format(self.password))
+        encrypted_password = KtuvitEncryptor(self.username, self.password, self.salt).encrypt()
+        if not encrypted_password:
+            logger.error("Could not encrypt password")
+            return False
+        self.encrypted_password = encrypted_password
+        return True
+
+    def get_encryption_salt(self):
+        p = re.compile(r"var encryptionSalt = '(.*)';")
+        r = self.session.get(self.URL_SERVER + self.URI_LOGIN)
+        r.raise_for_status()
+
+        logger.debug("Searching for encryptionSalt...")
+        script_list = [i for i in BeautifulSoup(r.content, 'html.parser').select('div#navbar script') if i.contents]
+        for item in script_list:
+            search_salt = p.search(item.contents[0])
+            if search_salt:
+                self.salt = search_salt.group(1)
+                return True
+        logger.error("Could not get encryptionSalt")
+        return False
+
+    def login(self):
+        if not self.get_encryption_salt():
+            return False
+        if not self.encrypt_password():
+            return False
+        data = {
+            "request": {
+                "Email": self.username,
+                "Password": self.encrypted_password
+            }
+        }
+        logger.debug("Trying to log in using: {}".format(json.dumps(data)))
+        r = self.session.post(self.URL_SERVER + self.URI_LOGIN_POST, json=data)
+        r_result = r.json()
+        login_results = ""
+        if 'd' in r_result:
+            try:
+                login_results = json.loads(r_result['d'])
+            except ValueError:
+                logger.error("Could not process JSON from login response")
+                return False
+
+        if 'IsSuccess' not in login_results:
+            logger.error("Login response is different than expected")
+            return False
+
+        if not login_results['IsSuccess']:
+            logger.error("Wrong username or password!")
+            raise AuthenticationError('Wrong username or password!')
+            return False
+
+        if not r.cookies or type(r.cookies) is not RequestsCookieJar:
+            logger.error("Could not get the cookie response of the login")
+            return False
+
+        if 'Login' not in r.cookies.keys():
+            logger.error("Could not found login cookie!")
+            return False
+
+        logger.info("Connected successfully to Ktuvit!")
+        return True
+
+    def initialize(self):
+        logger.debug("Ktuvit initialize")
+        self.session = Session()
+        self.session.headers[
+            'User-Agent'] = 'Mozilla/5.0 (Windows NT 10.0; ' \
+                            'Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36'
+
+        if not self.login():
+            return False
+
+    def terminate(self):
+        logger.debug("Ktuvit terminate")
+        self.session.close()
+
+    def _search_series(self, title):
+        logger.debug("Searching '{}'".format(title))
+        title_request = {
+            "request": {
+                "SearchString": title,
+                "SearchType": "Film"
+            }
+        }
+        r = self.session.post(self.URL_SERVER + self.URI_SEARCH_TITLE, json=title_request, allow_redirects=False,
+                              timeout=10)
+        r.raise_for_status()
+        series_found = r.json()
+        if 'd' in series_found:
+            try:
+                series_found = json.loads(series_found['d'])
+            except ValueError:
+                series_found = None
+        if 'Items' in series_found:
+            return series_found['Items']
+        return []
+
+    def _search_subtitles(self, title_id, season=None, episode=None):
+        if season and episode:
+            params = {
+                'moduleName': 'SubtitlesList',
+                'SeriesID': title_id,
+                'Season': season,
+                'Episode': episode
+            }
+            r = self.session.get(url=self.URL_SERVER + self.URI_SEARCH_SERIES_SUBTITLE, params=params)
+        else:
+            params = {
+                'ID': title_id,
+            }
+            r = self.session.get(url=self.URL_SERVER + self.URI_SEARCH_MOVIE_SUBTITLE, params=params)
+
+        r.raise_for_status()
+        results = r.content
+        if not results:
+            return []
+        subtitles = BeautifulSoup(results, 'html.parser').select('a.fa')
+        logger.debug("[BS4] Elements found:\n{}".format(subtitles))
+        subtitle_list = []
+        for i in subtitles:
+            subtitle_id = i.attrs['data-subtitle-id']
+            release = i.findParent().findParent().text.strip().split('\n')[0]
+            subtitle_list.append((subtitle_id, release))
+
+        return subtitle_list  # [(Subtitle ID, name), (....)]
+
+    def _req_download_identifier(self, title_id, subtitle_id):
+        logger.debug("Request subtitle identifier for: title id: {}, subtitle id: {}".format(title_id, subtitle_id))
+        data = {
+            'request': {
+                'FilmID': title_id,
+                'SubtitleID': subtitle_id,
+                'FontSize': 0,
+                'FontColor': "",
+                'PredefinedLayout': -1
+            }
+        }
+
+        r = self.session.post(self.URL_SERVER + self.URI_REQ_SUBTITLE_ID, json=data, allow_redirects=False,
+                              timeout=10)
+        r.raise_for_status()
+        try:
+            r = json.loads(r.json()['d'])
+        except ValueError:
+            r = {}
+
+        if 'DownloadIdentifier' not in r:
+            logger.error("Download Identifier not found")
+            return None
+        return r['DownloadIdentifier']
+
+    def _download_subtitles(self, download_id):
+        logger.debug("Downloading subtitles by download identifier - {}".format(download_id))
+        data = {'DownloadIdentifier': download_id}
+        r = self.session.get(self.URL_SERVER + self.URI_DOWNLOAD_SUBTITLE, params=data,
+                             timeout=10)
+        r.raise_for_status()
+        if not r.content:
+            logger.debug("Download subtitle failed")
+            return None
+
+        logger.debug("Download subtitle success")
+        return r.content
+
+    def query(self, title, season=None, episode=None, year=None):
+        subtitles = []
+        titles = self._search_series(title)
+        if season and episode:
+            logger.debug("Searching for:\nTitle: {}\nSeason: {}\nEpisode: {}\nYear: {}".format(title, season,
+                                                                                               episode, year))
+        else:
+            logger.debug("Searching for:\nTitle: {}\nYear: {}\n".format(title, year))
+        for title in titles:
+            logger.debug("Title Candidate: {}".format(title))
+            title_id = title['ID']
+            if season and episode:
+                result = self._search_subtitles(title_id, season, episode)
+            else:
+                result = self._search_subtitles(title_id)
+
+            if not result:
+                continue
+
+            for subtitle_id, release in result:
+                subtitles.append(self.subtitle_class(next(iter(self.languages)), title_id, subtitle_id,
+                                                     title['EngName'], season, episode, release, year))
+
+        if subtitles:
+            logger.debug("Found Subtitle Candidates: {}".format(subtitles))
+        return subtitles
+
+    def list_subtitles(self, video, languages):
+        season = episode = year = title = None
+
+        if isinstance(video, Episode):
+            logger.info("list_subtitles Series: {}, season: {}, episode: {}".format(video.series,
+                                                                                    video.season,
+                                                                                    video.episode))
+            title = video.series
+            season = video.season
+            episode = video.episode
+        elif isinstance(video, Movie):
+            logger.info("list_subtitles Movie: {}, year: {}".format(video.title, video.year))
+            title = video.title
+            year = video.year
+
+        return [s for s in self.query(title, season, episode, year) if s.language in languages]
+
+    def download_subtitle(self, subtitle):
+        # type: (KtuvitSubtitle) -> None
+
+        logger.info('Downloading subtitle from Ktuvit: %r', subtitle)
+        download_id = self._req_download_identifier(subtitle.title_id, subtitle.subtitle_id)
+        if not download_id:
+            logger.debug('Unable to retrieve download identifier')
+            return None
+
+        content = self._download_subtitles(download_id)
+        if not content:
+            logger.debug('Unable to download subtitle')
+            return None
+
+        subtitle.content = fix_line_ending(content)
+
+
+class KtuvitEncryptor:
+    def __init__(self, username, password, salt):
+        if not all((username, password, salt)):
+            raise Exception("Encryptor did not get all required arguments")
+
+        self.encrypted_password = None
+        self.username = username
+        self.password = password
+        self.salt = salt
+
+    @staticmethod
+    def rshift(val, n):
+        return (val % 0x100000000) >> n
+
+    @staticmethod
+    def js_parseint(s, rad=10):
+        digits = ''
+        for c in str(s).strip():
+            if c not in hexdigits:
+                break
+            digits += c
+
+        return int(digits, rad) if digits else 0
+
+    @staticmethod
+    def to_signed32(n):
+        n = n & 0xffffffff
+        return n | (-(n & 0x80000000))
+
+    def stringify(self, words, length):
+        sigbytes = int(length / 2) + int((length % 2) > 0)
+        hex_chars = list()
+
+        for i in xrange(0, sigbytes):
+            bite = self.rshift(words[self.rshift(i, 2)], (24 - (i % 4) * 8)) & 0xff
+            hex_chars.append(format(self.rshift(bite, 4), 'x'))
+            hex_chars.append(format(bite & 0x0f, 'x'))
+        return ''.join(hex_chars)
+
+    def cryptojs_hexparse(self, s):
+        words = defaultdict(int)
+        for i in range(0, len(s), 2):
+            tmp1 = (self.js_parseint(s[i:i + 2], 16))
+            tmp2 = (24 - (i % 8) * 4)
+            tmp3 = self.to_signed32(tmp1 << tmp2)
+            words[self.rshift(i, 3)] |= tmp3
+        return self.stringify(words, len(s))
+
+    def cryptojs_pad_iv(self, iv):
+        return str.ljust(self.cryptojs_hexparse(iv), 32, '0').decode('hex')
+
+    @staticmethod
+    def pbkdf2_encrypt(key, salt):
+        a = pbkdf2.PBKDF2(salt, key, 3000)
+        return a.read(16)
+
+    @staticmethod
+    def pad(m):
+        return m + chr(16 - len(m) % 16) * (16 - len(m) % 16)
+
+    def aes_encrypt(self, msg, key, iv):
+        if len(iv) != 16:
+            logger.error("iv (Len: {}) - {} is not 16 length".format(len(iv), iv))
+            return False
+        msg = self.pad(msg)
+
+        aes = pyaes.AESModeOfOperationCBC(key, iv=iv[:16])
+        return b64encode(aes.encrypt(msg))
+
+    def encrypt(self):
+        if not self.salt:
+            logger.error("No salt was instantiated!")
+            return False
+        msg = self.password.encode('utf-8')
+        iv = self.cryptojs_pad_iv(self.username)
+        key = self.pbkdf2_encrypt(self.username, self.salt.encode('utf-8'))
+
+        cipher = self.aes_encrypt(msg, key, iv)
+        if not cipher:
+            return False
+
+        hash_sha256 = sha256(b64decode(cipher))
+        self.encrypted_password = b64encode(hash_sha256.digest())
+        logger.debug("Encrypted password: {}".format(self.encrypted_password))
+        logger.debug("Original password: {}".format(self.password))
+        return self.encrypted_password
@@ -0,0 +1,124 @@
+import logging
+import os
+from io import BytesIO
+from zipfile import ZipFile
+
+from requests import Session
+
+from subliminal_patch.subtitle import Subtitle
+from subliminal_patch.providers import Provider
+from subliminal import __short_version__
+from subliminal.exceptions import AuthenticationError, ConfigurationError
+from subliminal.subtitle import fix_line_ending
+from subzero.language import Language
+
+logger = logging.getLogger(__name__)
+
+
+class Napisy24Subtitle(Subtitle):
+    '''Napisy24 Subtitle.'''
+    provider_name = 'napisy24'
+
+    def __init__(self, language, hash, imdb_id, napis_id):
+        super(Napisy24Subtitle, self).__init__(language)
+        self.hash = hash
+        self.imdb_id = imdb_id
+        self.napis_id = napis_id
+
+    @property
+    def id(self):
+        return self.hash
+
+    def get_matches(self, video):
+        matches = set()
+
+        # hash
+        if 'napisy24' in video.hashes and video.hashes['napisy24'] == self.hash:
+            matches.add('hash')
+
+        # imdb_id
+        if video.imdb_id and self.imdb_id == video.imdb_id:
+            matches.add('imdb_id')
+
+        return matches
+
+
+class Napisy24Provider(Provider):
+    '''Napisy24 Provider.'''
+    languages = {Language(l) for l in ['pol']}
+    required_hash = 'napisy24'
+    api_url = 'http://napisy24.pl/run/CheckSubAgent.php'
+
+    def __init__(self, username=None, password=None):
+        if all((username, password)):
+            self.username = username
+            self.password = password
+        else:
+            self.username = 'subliminal'
+            self.password = 'lanimilbus'
+
+        self.session = None
+
+    def initialize(self):
+        self.session = Session()
+        self.session.headers['User-Agent'] = 'Subliminal/%s' % __short_version__
+        self.session.headers['Content-Type'] = 'application/x-www-form-urlencoded'
+
+    def terminate(self):
+        self.session.close()
+
+    def query(self, language, size, name, hash):
+        params = {
+            'postAction': 'CheckSub',
+            'ua': self.username,
+            'ap': self.password,
+            'fs': size,
+            'fh': hash,
+            'fn': os.path.basename(name),
+            'n24pref': 1
+        }
+
+        response = self.session.post(self.api_url, data=params, timeout=10)
+        response.raise_for_status()
+
+        response_content = response.content.split(b'||', 1)
+        n24_data = response_content[0].decode()
+
+        if n24_data[:2] != 'OK':
+            if n24_data[:11] == 'login error':
+                raise AuthenticationError('Login failed')
+            logger.error('Unknown response: %s', response.content)
+            return None
+
+        n24_status = n24_data[:4]
+        if n24_status == 'OK-0':
+            logger.info('No subtitles found')
+            return None
+
+        subtitle_info = dict(p.split(':', 1) for p in n24_data.split('|')[1:])
+        logger.debug('Subtitle info: %s', subtitle_info)
+
+        if n24_status == 'OK-1':
+            logger.info('No subtitles found but got video info')
+            return None
+        elif n24_status == 'OK-2':
+            logger.info('Found subtitles')
+        elif n24_status == 'OK-3':
+            logger.info('Found subtitles but not from Napisy24 database')
+            return None
+        
+        subtitle_content = response_content[1]
+
+        subtitle = Napisy24Subtitle(language, hash, 'tt%s' % subtitle_info['imdb'].zfill(7), subtitle_info['napisId'])
+        with ZipFile(BytesIO(subtitle_content)) as zf:
+            subtitle.content = fix_line_ending(zf.open(zf.namelist()[0]).read())
+
+        return subtitle
+
+    def list_subtitles(self, video, languages):
+        subtitles = [self.query(l, video.size, video.name, video.hashes['napisy24']) for l in languages]
+        return [s for s in subtitles if s is not None]
+
+    def download_subtitle(self, subtitle):
+        # there is no download step, content is already filled from listing subtitles
+        pass
@@ -105,7 +105,7 @@ class OpenSubtitlesProvider(ProviderRetryMixin, _OpenSubtitlesProvider):

    def __init__(self, username=None, password=None, use_tag_search=False, only_foreign=False, also_foreign=False,
                 skip_wrong_fps=True, is_vip=False, use_ssl=True, timeout=15):
-        if any((username, password)) and not all((username, password)):
+        if not all((username, password)):
            raise ConfigurationError('Username and password must be specified')

        self.username = username or ''
@@ -154,6 +154,7 @@ class OpenSubtitlesProvider(ProviderRetryMixin, _OpenSubtitlesProvider):
        logger.debug('Logged in with token %r', self.token[:10]+"X"*(len(self.token)-10))

        region.set("os_token", self.token)
+        time.sleep(1)

    def use_token_or_login(self, func):
        if not self.token:
@@ -162,6 +163,7 @@ class OpenSubtitlesProvider(ProviderRetryMixin, _OpenSubtitlesProvider):
        try:
            return func()
        except Unauthorized:
+            logger.debug("Token not valid, logging in again")
            self.log_in()
            return func()

@@ -197,16 +199,11 @@ class OpenSubtitlesProvider(ProviderRetryMixin, _OpenSubtitlesProvider):
                    return

            logger.error("Login failed, please check your credentials")
+            raise
                
    def terminate(self):
-        if self.token:
-            try:
-                checked(lambda: self.server.LogOut(self.token))
-            except:
-                logger.error("Logout failed: %s", traceback.format_exc())
-
        self.server = None
-        self.token = None
+        #self.token = None

    def list_subtitles(self, video, languages):
        """
@@ -0,0 +1,579 @@
+# -*- coding: utf-8 -*-
+import logging
+import os
+import time
+import datetime
+import json
+import types
+
+from requests import Session, ConnectionError, Timeout, ReadTimeout, RequestException
+from simplejson import JSONDecodeError
+from subzero.language import Language
+
+from babelfish import language_converters
+from subliminal import Episode, Movie
+from subliminal.score import get_equivalent_release_groups
+from subliminal.utils import sanitize_release_group, sanitize
+from subliminal_patch.exceptions import TooManyRequests, APIThrottled
+from subliminal.exceptions import DownloadLimitExceeded, AuthenticationError, ConfigurationError, ServiceUnavailable, \
+    ProviderError
+from .mixins import ProviderRetryMixin
+from subliminal_patch.subtitle import Subtitle
+from subliminal.subtitle import fix_line_ending, SUBTITLE_EXTENSIONS
+from subliminal_patch.providers import Provider
+from subliminal_patch.subtitle import guess_matches
+from subliminal_patch.utils import fix_inconsistent_naming
+from subliminal.cache import region
+from dogpile.cache.api import NO_VALUE
+from guessit import guessit
+
+logger = logging.getLogger(__name__)
+
+SHOW_EXPIRATION_TIME = datetime.timedelta(weeks=1).total_seconds()
+TOKEN_EXPIRATION_TIME = datetime.timedelta(hours=12).total_seconds()
+
+retry_amount = 3
+
+
+def fix_tv_naming(title):
+    """Fix TV show titles with inconsistent naming using dictionary, but do not sanitize them.
+
+    :param str title: original title.
+    :return: new title.
+    :rtype: str
+
+    """
+    return fix_inconsistent_naming(title, {"Superman & Lois": "Superman and Lois",
+                                           }, True)
+
+
+def fix_movie_naming(title):
+    return fix_inconsistent_naming(title, {
+    }, True)
+
+
+custom_languages = {
+    'pt': 'pt-PT',
+    'zh': 'zh-CN',
+}
+
+
+def to_opensubtitlescom(lang):
+    if lang in custom_languages.keys():
+        return custom_languages[lang]
+    else:
+        return lang
+
+
+def from_opensubtitlescom(lang):
+    from_custom_languages = {v: k for k, v in custom_languages.items()}
+    if lang in from_custom_languages.keys():
+        return from_custom_languages[lang]
+    else:
+        return lang
+
+
+class OpenSubtitlesComSubtitle(Subtitle):
+    provider_name = 'opensubtitlescom'
+    hash_verifiable = True
+    hearing_impaired_verifiable = True
+
+    def __init__(self, language, forced, hearing_impaired, page_link, file_id, releases, uploader, title, year,
+                 hash_matched, file_hash=None, season=None, episode=None, imdb_match=False):
+        super(OpenSubtitlesComSubtitle, self).__init__(language, hearing_impaired, page_link)
+        language = Language.rebuild(language, hi=hearing_impaired, forced=forced)
+
+        self.title = title
+        self.year = year
+        self.season = season
+        self.episode = episode
+        self.releases = releases
+        self.release_info = releases
+        self.language = language
+        self.hearing_impaired = hearing_impaired
+        self.forced = forced
+        self.file_id = file_id
+        self.page_link = page_link
+        self.download_link = None
+        self.uploader = uploader
+        self.matches = None
+        self.hash = file_hash
+        self.encoding = 'utf-8'
+        self.hash_matched = hash_matched
+        self.imdb_match = imdb_match
+
+    @property
+    def id(self):
+        return self.file_id
+
+    def get_matches(self, video):
+        matches = set()
+        type_ = "movie" if isinstance(video, Movie) else "episode"
+
+        # handle movies and series separately
+        if type_ == "episode":
+            # series
+            matches.add('series')
+            # season
+            if video.season == self.season:
+                matches.add('season')
+            # episode
+            if video.episode == self.episode:
+                matches.add('episode')
+            # imdb
+            if self.imdb_match:
+                matches.add('series_imdb_id')
+        else:
+            # title
+            matches.add('title')
+            # imdb
+            if self.imdb_match:
+                matches.add('imdb_id')
+
+        # rest is same for both groups
+
+        # year
+        if video.year == self.year:
+            matches.add('year')
+
+        # release_group
+        if (video.release_group and self.releases and
+                any(r in sanitize_release_group(self.releases)
+                    for r in get_equivalent_release_groups(sanitize_release_group(video.release_group)))):
+            matches.add('release_group')
+
+        if self.hash_matched:
+            matches.add('hash')
+
+        # other properties
+        matches |= guess_matches(video, guessit(self.releases, {"type": type_}))
+
+        self.matches = matches
+
+        return matches
+
+
+class OpenSubtitlesComProvider(ProviderRetryMixin, Provider):
+    """OpenSubtitlesCom Provider"""
+    server_url = 'https://api.opensubtitles.com/api/v1/'
+
+    languages = {Language.fromopensubtitles(lang) for lang in language_converters['szopensubtitles'].codes}
+    languages.update(set(Language.rebuild(lang, forced=True) for lang in languages))
+    languages.update(set(Language.rebuild(l, hi=True) for l in languages))
+
+    video_types = (Episode, Movie)
+
+    def __init__(self, username=None, password=None, use_hash=True, include_ai_translated=False, api_key=None):
+        if not all((username, password)):
+            raise ConfigurationError('Username and password must be specified')
+
+        if not api_key:
+            raise ConfigurationError('Api_key must be specified')
+
+        if not all((username, password)):
+            raise ConfigurationError('Username and password must be specified')
+
+        self.session = Session()
+        self.session.headers = {'User-Agent': os.environ.get("SZ_USER_AGENT", "Sub-Zero/2"),
+                                'Api-Key': api_key,
+                                'Content-Type': 'application/json'}
+        self.token = None
+        self.username = username
+        self.password = password
+        self.video = None
+        self.use_hash = use_hash
+        self.include_ai_translated = include_ai_translated
+        self._started = None
+
+    def initialize(self):
+        self._started = time.time()
+
+        if region.get("oscom_token", expiration_time=TOKEN_EXPIRATION_TIME) is NO_VALUE:
+            logger.debug("No cached token, we'll try to login again.")
+            self.login()
+        else:
+            self.token = region.get("oscom_token", expiration_time=TOKEN_EXPIRATION_TIME)
+
+    def terminate(self):
+        self.session.close()
+
+    def ping(self):
+        return self._started and (time.time() - self._started) < TOKEN_EXPIRATION_TIME
+
+    def login(self, is_retry=False):
+        r = self.checked(
+            lambda: self.session.post(self.server_url + 'login',
+                                      json={"username": self.username, "password": self.password},
+                                      allow_redirects=False,
+                                      timeout=30),
+            is_retry=is_retry)
+
+        try:
+            self.token = r.json()['token']
+        except (ValueError, JSONDecodeError):
+            log_request_response(r)
+            raise ProviderError("Cannot get token from provider login response")
+        else:
+            log_request_response(r, non_standard=False)
+            region.set("oscom_token", self.token)
+
+    @staticmethod
+    def sanitize_external_ids(external_id):
+        if isinstance(external_id, types.StringTypes):
+            external_id = external_id.lower().lstrip('tt').lstrip('0')
+        sanitized_id = external_id[:-1].lstrip('0') + external_id[-1]
+        return int(sanitized_id)
+
+    @region.cache_on_arguments(expiration_time=SHOW_EXPIRATION_TIME)
+    def search_titles(self, title):
+        title_id = None
+
+        parameters = {'query': title.lower()}
+        logging.debug('Searching using this title: %s' % title)
+
+        results = self.retry(
+            lambda: self.checked(
+                lambda: self.session.get(self.server_url + 'features', params=parameters, timeout=30),
+                validate_json=True,
+                json_key_name='data'
+            ),
+            amount=retry_amount
+        )
+
+        # deserialize results
+        results_dict = results.json()['data']
+
+        # loop over results
+        for result in results_dict:
+            if 'title' in result['attributes']:
+                if isinstance(self.video, Episode):
+                    if fix_tv_naming(title).lower() == result['attributes']['title'].lower() and \
+                            (not self.video.year or self.video.year == int(result['attributes']['year'])):
+                        title_id = result['id']
+                        break
+                else:
+                    if fix_movie_naming(title).lower() == result['attributes']['title'].lower() and \
+                            (not self.video.year or self.video.year == int(result['attributes']['year'])):
+                        title_id = result['id']
+                        break
+            else:
+                continue
+
+        if title_id:
+            logging.debug('Found this title ID: %s' % title_id)
+            return self.sanitize_external_ids(title_id)
+
+        if not title_id:
+            logger.debug('No match found for %s' % title)
+
+    def query(self, languages, video):
+        self.video = video
+        if self.use_hash:
+            file_hash = self.video.hashes.get('opensubtitlescom')
+            logging.debug('Searching using this hash: %s' % hash)
+        else:
+            file_hash = None
+
+        if isinstance(self.video, Episode):
+            title = self.video.series
+        else:
+            title = self.video.title
+
+        imdb_id = None
+        if isinstance(self.video, Episode) and self.video.series_imdb_id:
+            imdb_id = self.sanitize_external_ids(self.video.series_imdb_id)
+        elif isinstance(self.video, Movie) and self.video.imdb_id:
+            imdb_id = self.sanitize_external_ids(self.video.imdb_id)
+
+        title_id = None
+        if not imdb_id:
+            title_id = self.search_titles(title)
+            if not title_id:
+                return []
+
+        # be sure to remove duplicates using list(set())
+        langs_list = sorted(list(set([to_opensubtitlescom(lang.basename).lower() for lang in languages])))
+
+        langs = ','.join(langs_list)
+        logging.debug('Searching for those languages: %s' % langs)
+
+        # query the server
+        if isinstance(self.video, Episode):
+            res = self.retry(
+                lambda: self.checked(
+                    lambda: self.session.get(self.server_url + 'subtitles',
+                                             params=(('ai_translated', 'exclude' if not self.include_ai_translated
+                                                     else 'include'),
+                                                     ('episode_number', self.video.episode),
+                                                     ('imdb_id', imdb_id if not title_id else None),
+                                                     ('languages', langs),
+                                                     ('moviehash', file_hash),
+                                                     ('parent_feature_id', title_id if title_id else None),
+                                                     ('season_number', self.video.season)),
+                                             timeout=30),
+                    validate_json=True,
+                    json_key_name='data'
+                ),
+                amount=retry_amount
+            )
+        else:
+            res = self.retry(
+                lambda: self.checked(
+                    lambda: self.session.get(self.server_url + 'subtitles',
+                                             params=(('ai_translated', 'exclude' if not self.include_ai_translated
+                                                     else 'include'),
+                                                     ('id', title_id if title_id else None),
+                                                     ('imdb_id', imdb_id if not title_id else None),
+                                                     ('languages', langs),
+                                                     ('moviehash', file_hash)),
+                                             timeout=30),
+                    validate_json=True,
+                    json_key_name='data'
+                ),
+                amount=retry_amount
+            )
+
+        subtitles = []
+
+        result = res.json()
+
+        # filter out forced subtitles or not depending on the required languages
+        if all([lang.forced for lang in languages]):  # only forced
+            result['data'] = [x for x in result['data'] if x['attributes']['foreign_parts_only']]
+        elif any([lang.forced for lang in languages]):  # also forced
+            pass
+        else:  # not forced
+            result['data'] = [x for x in result['data'] if not x['attributes']['foreign_parts_only']]
+
+        logging.debug("Query returned %s subtitles" % len(result['data']))
+
+        if len(result['data']):
+            for item in result['data']:
+                # ignore AI translated subtitles
+                if 'ai_translated' in item['attributes'] and item['attributes']['ai_translated']:
+                    logging.debug("Skipping AI translated subtitles")
+                    continue
+
+                # ignore machine translated subtitles
+                if 'machine_translated' in item['attributes'] and item['attributes']['machine_translated']:
+                    logging.debug("Skipping machine translated subtitles")
+                    continue
+
+                if 'season_number' in item['attributes']['feature_details']:
+                    season_number = item['attributes']['feature_details']['season_number']
+                else:
+                    season_number = None
+
+                if 'episode_number' in item['attributes']['feature_details']:
+                    episode_number = item['attributes']['feature_details']['episode_number']
+                else:
+                    episode_number = None
+
+                if 'moviehash_match' in item['attributes']:
+                    moviehash_match = item['attributes']['moviehash_match']
+                else:
+                    moviehash_match = False
+
+                try:
+                    year = int(item['attributes']['feature_details']['year'])
+                except TypeError:
+                    year = item['attributes']['feature_details']['year']
+
+                if len(item['attributes']['files']):
+                    subtitle = OpenSubtitlesComSubtitle(
+                        language=Language.fromietf(from_opensubtitlescom(item['attributes']['language'])),
+                        forced=item['attributes']['foreign_parts_only'],
+                        hearing_impaired=item['attributes']['hearing_impaired'],
+                        page_link=item['attributes']['url'],
+                        file_id=item['attributes']['files'][0]['file_id'],
+                        releases=item['attributes']['release'],
+                        uploader=item['attributes']['uploader']['name'],
+                        title=item['attributes']['feature_details']['movie_name'],
+                        year=year,
+                        season=season_number,
+                        episode=episode_number,
+                        hash_matched=moviehash_match,
+                        imdb_match=True if imdb_id else False
+                    )
+                    subtitle.get_matches(self.video)
+                    subtitles.append(subtitle)
+
+        return subtitles
+
+    def list_subtitles(self, video, languages):
+        return self.query(languages, video)
+
+    def download_subtitle(self, subtitle):
+        logger.info('Downloading subtitle %r', subtitle)
+
+        headers = {'Accept': 'application/json', 'Content-Type': 'application/json',
+                   'Authorization': 'Bearer ' + self.token}
+        res = self.retry(
+            lambda: self.checked(
+                lambda: self.session.post(self.server_url + 'download',
+                                          json={'file_id': subtitle.file_id, 'sub_format': 'srt'},
+                                          headers=headers,
+                                          timeout=30),
+                validate_json=True,
+                json_key_name='link'
+            ),
+            amount=retry_amount
+        )
+
+        download_data = res.json()
+        subtitle.download_link = download_data['link']
+
+        r = self.retry(
+            lambda: self.checked(
+                lambda: self.session.get(subtitle.download_link, timeout=30),
+                validate_content=True
+            ),
+            amount=retry_amount
+        )
+
+        if not r:
+            logger.debug('Could not download subtitle from %s' % subtitle.download_link)
+            subtitle.content = None
+            return
+        else:
+            subtitle_content = r.content
+            subtitle.content = fix_line_ending(subtitle_content)
+
+    @staticmethod
+    def reset_token():
+        logging.debug('Authentication failed: clearing cache and attempting to login.')
+        region.delete("oscom_token")
+        return
+
+    def checked(self, fn, raise_api_limit=False, validate_json=False, json_key_name=None, validate_content=False,
+                is_retry=False):
+        """Run :fn: and check the response status before returning it.
+
+        :param fn: the function to make an API call to OpenSubtitles.com.
+        :param raise_api_limit: if True we wait a little bit longer before running the call again.
+        :param validate_json: test if response is valid json.
+        :param json_key_name: test if returned json contain a specific key.
+        :param validate_content: test if response have a content (used with download).
+        :param is_retry: prevent additional retries with login endpoint.
+        :return: the response.
+
+        """
+        response = None
+        try:
+            try:
+                response = fn()
+            except APIThrottled:
+                if not raise_api_limit:
+                    logger.info("API request limit hit, waiting and trying again once.")
+                    time.sleep(15)
+                    return self.checked(fn, raise_api_limit=True)
+                raise
+            except (ConnectionError, Timeout, ReadTimeout):
+                raise ServiceUnavailable('Unknown Error, empty response: {}: {}'.format(response.status_code, response))
+            except Exception:
+                logging.exception('Unhandled exception raised.')
+                raise ProviderError('Unhandled exception raised. Check log.')
+            else:
+                status_code = response.status_code
+        except Exception:
+            status_code = None
+        else:
+            if status_code == 400:
+                try:
+                    json_response = response.json()
+                    message = json_response['message']
+                except JSONDecodeError:
+                    raise ProviderError('Invalid JSON returned by provider')
+                else:
+                    log_request_response(response)
+                    raise ConfigurationError(message)
+            elif status_code == 401:
+                log_request_response(response)
+                self.reset_token()
+                if is_retry:
+                    raise AuthenticationError('Login failed')
+                else:
+                    time.sleep(1)
+                    self.login(is_retry=True)
+                    self.checked(fn, raise_api_limit=raise_api_limit, validate_json=validate_json,
+                                 json_key_name=json_key_name, validate_content=validate_content, is_retry=True)
+            elif status_code == 403:
+                log_request_response(response)
+                raise ProviderError("Bazarr API key seems to be in problem")
+            elif status_code == 406:
+                try:
+                    json_response = response.json()
+                    download_count = json_response['requests']
+                    remaining_download = json_response['remaining']
+                    quota_reset_time = json_response['reset_time']
+                except JSONDecodeError:
+                    raise ProviderError('Invalid JSON returned by provider')
+                else:
+                    log_request_response(response)
+                    raise DownloadLimitExceeded("Daily download limit reached. {} subtitles have been "
+                                                "downloaded and {} remaining subtitles can be "
+                                                "downloaded. Quota will be reset in {}.".format(download_count, remaining_download, quota_reset_time))
+            elif status_code == 410:
+                log_request_response(response)
+                raise ProviderError("Download as expired")
+            elif status_code == 429:
+                log_request_response(response)
+                raise TooManyRequests()
+            elif status_code == 500:
+                logging.debug("Server side exception raised while downloading from opensubtitles.com website. They "
+                              "should mitigate this soon.")
+                return None
+            elif status_code == 502:
+                # this one should deal with Bad Gateway issue on their side.
+                raise APIThrottled()
+            elif 500 <= status_code <= 599:
+                raise ProviderError(response.reason)
+
+            if status_code != 200:
+                log_request_response(response)
+                raise ProviderError('Bad status code: %s' % response.status_code)
+
+            if validate_json:
+                try:
+                    json_test = response.json()
+                except JSONDecodeError:
+                    raise ProviderError('Invalid JSON returned by provider')
+                else:
+                    if json_key_name not in json_test:
+                        raise ProviderError('Invalid JSON returned by provider: no %s key in returned json.' % json_key_name)
+
+            if validate_content:
+                if not hasattr(response, 'content'):
+                    logging.error('Download link returned no content attribute.')
+                    return False
+                elif not response.content:
+                    logging.error('This download link returned empty content: %s' % response.url)
+                    return False
+
+        return response
+
+
+def log_request_response(response, non_standard=True):
+    redacted_request_headers = response.request.headers
+    if 'Authorization' in redacted_request_headers and isinstance(redacted_request_headers['Authorization'], str):
+        redacted_request_headers['Authorization'] = redacted_request_headers['Authorization'][:-8]+8*'x'
+
+    redacted_request_body = json.loads(response.request.body)
+    if 'password' in redacted_request_body:
+        redacted_request_body['password'] = 'redacted'
+
+    redacted_response_body = json.loads(response.text)
+    if 'token' in redacted_response_body and isinstance(redacted_response_body['token'], str):
+        redacted_response_body['token'] = redacted_response_body['token'][:-8] + 8 * 'x'
+
+    if non_standard:
+        logging.debug("opensubtitlescom returned a non standard response. Logging request/response for debugging "
+                      "purpose.")
+    else:
+        logging.debug("opensubtitlescom returned a standard response. Logging request/response for debugging purpose.")
+    logging.debug("Request URL: %s" % response.request.url)
+    logging.debug("Request Headers: %s" % redacted_request_headers)
+    logging.debug("Request Body: %s" % json.dumps(redacted_request_body))
+    logging.debug("Response Status Code: %s" % {response.status_code})
+    logging.debug("Response Headers: %s" % response.headers)
+    logging.debug("Response Body: %s" % json.dumps(redacted_response_body))
@@ -4,24 +4,34 @@ import io
 import logging
 import os
 import time
+import traceback
+
+import requests
+
 import inflect
+import re
+import json
+import HTMLParser
+import urlparse

 from zipfile import ZipFile
 from babelfish import language_converters
 from guessit import guessit
+from dogpile.cache.api import NO_VALUE
 from subliminal import Episode, ProviderError
+from subliminal.exceptions import ConfigurationError, ServiceUnavailable
 from subliminal.utils import sanitize_release_group
+from subliminal.cache import region
 from subliminal_patch.http import RetryingCFSession
 from subliminal_patch.providers import Provider
 from subliminal_patch.providers.mixins import ProviderSubtitleArchiveMixin
 from subliminal_patch.subtitle import Subtitle, guess_matches
 from subliminal_patch.converters.subscene import language_ids, supported_languages
-from subscene_api.subscene import search, Subtitle as APISubtitle
+from subscene_api.subscene import search, Subtitle as APISubtitle, SITE_DOMAIN
 from subzero.language import Language

 p = inflect.engine()

-
 language_converters.register('subscene = subliminal_patch.converters.subscene:SubsceneConverter')
 logger = logging.getLogger(__name__)

@@ -112,28 +122,106 @@ class SubsceneProvider(Provider, ProviderSubtitleArchiveMixin):
    skip_wrong_fps = False
    hearing_impaired_verifiable = True
    only_foreign = False
+    username = None
+    password = None

-    search_throttle = 2  # seconds
+    search_throttle = 8  # seconds
+
+    def __init__(self, only_foreign=False, username=None, password=None):
+        if not all((username, password)):
+            raise ConfigurationError('Username and password must be specified')

-    def __init__(self, only_foreign=False):
        self.only_foreign = only_foreign
+        self.username = username
+        self.password = password

    def initialize(self):
        logger.info("Creating session")
        self.session = RetryingCFSession()

+        prev_cookies = region.get("subscene_cookies2")
+        if prev_cookies != NO_VALUE:
+            logger.debug("Re-using old subscene cookies: %r", prev_cookies)
+            self.session.cookies.update(prev_cookies)
+
+        else:
+            logger.debug("Logging in")
+            self.login()
+
+    def login(self):
+        r = self.session.get("https://subscene.com/account/login")
+        if "Server Error" in r.content:
+            logger.error("Login unavailable; Maintenance?")
+            raise ServiceUnavailable("Login unavailable; Maintenance?")
+
+        match = re.search(r"<script id='modelJson' type='application/json'>\s*(.+)\s*</script>", r.content)
+
+        if match:
+            h = HTMLParser.HTMLParser()
+            data = json.loads(h.unescape(match.group(1)))
+            login_url = urlparse.urljoin(data["siteUrl"], data["loginUrl"])
+            time.sleep(1.0)
+
+            r = self.session.post(login_url,
+                                  {
+                                      "username": self.username,
+                                      "password": self.password,
+                                      data["antiForgery"]["name"]: data["antiForgery"]["value"]
+                                  })
+            pep_content = re.search(r"<form method=\"post\" action=\"https://subscene\.com/\">"
+                                    r".+name=\"id_token\".+?value=\"(?P<id_token>.+?)\".*?"
+                                    r"access_token\".+?value=\"(?P<access_token>.+?)\".+?"
+                                    r"token_type.+?value=\"(?P<token_type>.+?)\".+?"
+                                    r"expires_in.+?value=\"(?P<expires_in>.+?)\".+?"
+                                    r"scope.+?value=\"(?P<scope>.+?)\".+?"
+                                    r"state.+?value=\"(?P<state>.+?)\".+?"
+                                    r"session_state.+?value=\"(?P<session_state>.+?)\"",
+                                    r.content, re.MULTILINE | re.DOTALL)
+
+            if pep_content:
+                r = self.session.post(SITE_DOMAIN, pep_content.groupdict())
+                try:
+                    r.raise_for_status()
+                except Exception:
+                    raise ProviderError("Something went wrong when trying to log in: %s", traceback.format_exc())
+                else:
+                    cj = self.session.cookies.copy()
+                    store_cks = ("scene", "idsrv", "idsrv.xsrf", "idsvr.clients", "idsvr.session", "idsvr.username")
+                    for cn in self.session.cookies.iterkeys():
+                        if cn not in store_cks:
+                            del cj[cn]
+
+                    logger.debug("Storing cookies: %r", cj)
+                    region.set("subscene_cookies2", cj)
+                    return
+        raise ProviderError("Something went wrong when trying to log in #1")
+
    def terminate(self):
        logger.info("Closing session")
        self.session.close()

    def _create_filters(self, languages):
        self.filters = dict(HearingImpaired="2")
+        acc_filters = self.filters.copy()
        if self.only_foreign:
            self.filters["ForeignOnly"] = "True"
+            acc_filters["ForeignOnly"] = self.filters["ForeignOnly"].lower()
            logger.info("Only searching for foreign/forced subtitles")

-        self.filters["LanguageFilter"] = ",".join((str(language_ids[l.alpha3]) for l in languages
-                                                   if l.alpha3 in language_ids))
+        selected_ids = []
+        for l in languages:
+            lid = language_ids.get(l.basename, language_ids.get(l.alpha3, None))
+            if lid:
+                selected_ids.append(str(lid))
+
+        acc_filters["SelectedIds"] = selected_ids
+        self.filters["LanguageFilter"] = ",".join(acc_filters["SelectedIds"])
+
+        last_filters = region.get("subscene_filters")
+        if last_filters != acc_filters:
+            region.set("subscene_filters", acc_filters)
+            logger.debug("Setting account filters to %r", acc_filters)
+            self.session.post("https://u.subscene.com/filter", acc_filters, allow_redirects=False)

        logger.debug("Filter created: '%s'" % self.filters)

@@ -176,7 +264,11 @@ class SubsceneProvider(Provider, ProviderSubtitleArchiveMixin):
    def parse_results(self, video, film):
        subtitles = []
        for s in film.subtitles:
-            subtitle = SubsceneSubtitle.from_api(s)
+            try:
+                subtitle = SubsceneSubtitle.from_api(s)
+            except NotImplementedError, e:
+                logger.info(e)
+                continue
            subtitle.asked_for_release_group = video.release_group
            if isinstance(video, Episode):
                subtitle.asked_for_episode = video.episode
@@ -189,10 +281,16 @@ class SubsceneProvider(Provider, ProviderSubtitleArchiveMixin):

        return subtitles

+    def do_search(self, *args, **kwargs):
+        try:
+            return search(*args, **kwargs)
+        except requests.HTTPError:
+            region.delete("subscene_cookies2")
+
    def query(self, video):
-        vfn = get_video_filename(video)
+        # vfn = get_video_filename(video)
        subtitles = []
-        #logger.debug(u"Searching for: %s", vfn)
+        # logger.debug(u"Searching for: %s", vfn)
        # film = search(vfn, session=self.session)
        #
        # if film and film.subtitles:
@@ -201,16 +299,17 @@ class SubsceneProvider(Provider, ProviderSubtitleArchiveMixin):
        # else:
        #     logger.debug('No release results found')

-        #time.sleep(self.search_throttle)
+        # time.sleep(self.search_throttle)

        # re-search for episodes without explicit release name
        if isinstance(video, Episode):
-            #term = u"%s S%02iE%02i" % (video.series, video.season, video.episode)
-            more_than_one = len([video.series] + video.alternative_series) > 1
-            for series in [video.series] + video.alternative_series:
+            titles = list(set([video.series] + video.alternative_series))[:2]
+            # term = u"%s S%02iE%02i" % (video.series, video.season, video.episode)
+            more_than_one = len(titles) > 1
+            for series in titles:
                term = u"%s - %s Season" % (series, p.number_to_words("%sth" % video.season).capitalize())
                logger.debug('Searching for alternative results: %s', term)
-                film = search(term, session=self.session, release=False)
+                film = self.do_search(term, session=self.session, release=False, throttle=self.search_throttle)
                if film and film.subtitles:
                    logger.debug('Alternative results found: %s', len(film.subtitles))
                    subtitles += self.parse_results(video, film)
@@ -218,25 +317,27 @@ class SubsceneProvider(Provider, ProviderSubtitleArchiveMixin):
                    logger.debug('No alternative results found')

                # packs
-                if video.season_fully_aired:
-                    term = u"%s S%02i" % (series, video.season)
-                    logger.debug('Searching for packs: %s', term)
-                    time.sleep(self.search_throttle)
-                    film = search(term, session=self.session)
-                    if film and film.subtitles:
-                        logger.debug('Pack results found: %s', len(film.subtitles))
-                        subtitles += self.parse_results(video, film)
-                    else:
-                        logger.debug('No pack results found')
-                else:
-                    logger.debug("Not searching for packs, because the season hasn't fully aired")
+                # if video.season_fully_aired:
+                #     term = u"%s S%02i" % (series, video.season)
+                #     logger.debug('Searching for packs: %s', term)
+                #     time.sleep(self.search_throttle)
+                #     film = search(term, session=self.session, throttle=self.search_throttle)
+                #     if film and film.subtitles:
+                #         logger.debug('Pack results found: %s', len(film.subtitles))
+                #         subtitles += self.parse_results(video, film)
+                #     else:
+                #         logger.debug('No pack results found')
+                # else:
+                #     logger.debug("Not searching for packs, because the season hasn't fully aired")
                if more_than_one:
                    time.sleep(self.search_throttle)
        else:
-            more_than_one = len([video.title] + video.alternative_titles) > 1
-            for title in [video.title] + video.alternative_titles:
-                logger.debug('Searching for movie results: %s', title)
-                film = search(title, year=video.year, session=self.session, limit_to=None, release=False)
+            titles = list(set([video.title] + video.alternative_titles))[:2]
+            more_than_one = len(titles) > 1
+            for title in titles:
+                logger.debug('Searching for movie results: %r', title)
+                film = self.do_search(title, year=video.year, session=self.session, limit_to=None, release=False,
+                                      throttle=self.search_throttle)
                if film and film.subtitles:
                    subtitles += self.parse_results(video, film)
                if more_than_one:
@@ -143,7 +143,7 @@ class SuperSubtitlesProvider(Provider, ProviderSubtitleArchiveMixin):
    ]}
    video_types = (Episode, Movie)
    # https://www.feliratok.info/?search=&soriSorszam=&nyelv=&sorozatnev=The+Flash+%282014%29&sid=3212&complexsearch=true&knyelv=0&evad=4&epizod1=1&cimke=0&minoseg=0&rlsr=0&tab=all
-    server_url = 'https://www.feliratok.info/'
+    server_url = 'https://www.feliratok.eu/'
    subtitle_class = SuperSubtitlesSubtitle
    hearing_impaired_verifiable = False
    multi_result_throttle = 2  # seconds
@@ -2,42 +2,35 @@

 import io
 import logging
-import math
 import re
-import time
+from datetime import datetime
+import dateutil.parser

 import rarfile

-from bs4 import BeautifulSoup
 from zipfile import ZipFile, is_zipfile
 from rarfile import RarFile, is_rarfile
 from babelfish import language_converters, Script
-from requests import RequestException
+from requests import RequestException, codes as request_codes
 from guessit import guessit
 from subliminal_patch.http import RetryingCFSession
 from subliminal_patch.providers import Provider
 from subliminal_patch.providers.mixins import ProviderSubtitleArchiveMixin
 from subliminal_patch.subtitle import Subtitle
 from subliminal_patch.utils import sanitize, fix_inconsistent_naming as _fix_inconsistent_naming
-from subliminal.exceptions import ProviderError
+from subliminal.exceptions import ProviderError, AuthenticationError, ConfigurationError
 from subliminal.score import get_equivalent_release_groups
 from subliminal.utils import sanitize_release_group
 from subliminal.subtitle import guess_matches
 from subliminal.video import Episode, Movie
 from subliminal.subtitle import fix_line_ending
-from subliminal_patch.pitcher import pitchers, load_verification, store_verification
-from subzero.language import Language

-from random import randint
-from .utils import FIRST_THOUSAND_OR_SO_USER_AGENTS as AGENT_LIST
+from subzero.language import Language
+from dogpile.cache.api import NO_VALUE
+from subliminal.cache import region

 # parsing regex definitions
 title_re = re.compile(r'(?P<title>(?:.+(?= [Aa][Kk][Aa] ))|.+)(?:(?:.+)(?P<altitle>(?<= [Aa][Kk][Aa] ).+))?')
-lang_re = re.compile(r'(?<=flags/)(?P<lang>.{2})(?:.)(?P<script>c?)(?:.+)')
-season_re = re.compile(r'Sezona (?P<season>\d+)')
-episode_re = re.compile(r'Epizoda (?P<episode>\d+)')
-year_re = re.compile(r'(?P<year>\d+)')
-fps_re = re.compile(r'fps: (?P<fps>.+)')


 def fix_inconsistent_naming(title):
@@ -51,6 +44,7 @@ def fix_inconsistent_naming(title):
    return _fix_inconsistent_naming(title, {"DC's Legends of Tomorrow": "Legends of Tomorrow",
                                            "Marvel's Jessica Jones": "Jessica Jones"})

+
 logger = logging.getLogger(__name__)

 # Configure :mod:`rarfile` to use the same path separator as :mod:`zipfile`
@@ -62,9 +56,9 @@ language_converters.register('titlovi = subliminal_patch.converters.titlovi:Titl
 class TitloviSubtitle(Subtitle):
    provider_name = 'titlovi'

-    def __init__(self, language, page_link, download_link, sid, releases, title, alt_title=None, season=None,
-                 episode=None, year=None, fps=None, asked_for_release_group=None, asked_for_episode=None):
-        super(TitloviSubtitle, self).__init__(language, page_link=page_link)
+    def __init__(self, language, download_link, sid, releases, title, alt_title=None, season=None,
+                 episode=None, year=None, rating=None, download_count=None, asked_for_release_group=None, asked_for_episode=None):
+        super(TitloviSubtitle, self).__init__(language)
        self.sid = sid
        self.releases = self.release_info = releases
        self.title = title
@@ -73,11 +67,21 @@ class TitloviSubtitle(Subtitle):
        self.episode = episode
        self.year = year
        self.download_link = download_link
-        self.fps = fps
+        self.rating = rating
+        self.download_count = download_count
        self.matches = None
        self.asked_for_release_group = asked_for_release_group
        self.asked_for_episode = asked_for_episode

+    def __repr__(self):
+        if self.season and self.episode:
+            return '<%s "%s (%r)" s%.2de%.2d [%s:%s] ID:%r R:%.2f D:%r>' % (
+                self.__class__.__name__, self.title, self.year, self.season, self.episode, self.language, self._guessed_encoding, self.sid,
+                self.rating, self.download_count)
+        else:
+            return '<%s "%s (%r)" [%s:%s] ID:%r R:%.2f D:%r>' % (
+                self.__class__.__name__, self.title, self.year,  self.language, self._guessed_encoding, self.sid, self.rating, self.download_count)
+
    @property
    def id(self):
        return self.sid
@@ -134,20 +138,62 @@ class TitloviSubtitle(Subtitle):
 class TitloviProvider(Provider, ProviderSubtitleArchiveMixin):
    subtitle_class = TitloviSubtitle
    languages = {Language.fromtitlovi(l) for l in language_converters['titlovi'].codes} | {Language.fromietf('sr-Latn')}
-    server_url = 'https://titlovi.com'
-    search_url = server_url + '/titlovi/?'
-    download_url = server_url + '/download/?type=1&mediaid='
+    api_url = 'https://kodi.titlovi.com/api/subtitles'
+    api_gettoken_url = api_url + '/gettoken'
+    api_search_url = api_url + '/search'
+
+    def __init__(self, username=None, password=None):
+        if not all((username, password)):
+            raise ConfigurationError('Username and password must be specified')
+
+        self.username = username
+        self.password = password
+
+        self.session = None
+
+        self.user_id = None
+        self.login_token = None
+        self.token_exp = None

    def initialize(self):
        self.session = RetryingCFSession()
        #load_verification("titlovi", self.session)

+        token = region.get("titlovi_token")
+        if token is not NO_VALUE:
+            self.user_id, self.login_token, self.token_exp = token
+            if datetime.now() > self.token_exp:
+                logger.debug('Token expired')
+                self.log_in()
+            else:
+                logger.debug('Use cached token')
+        else:
+            logger.debug('Token not found in cache')
+            self.log_in()
+
+    def log_in(self):
+        login_params = dict(username=self.username, password=self.password, json=True)
+        try:
+            response = self.session.post(self.api_gettoken_url, params=login_params)
+            if response.status_code == request_codes.ok:
+                resp_json = response.json()
+                self.login_token = resp_json.get('Token')
+                self.user_id = resp_json.get('UserId')
+                self.token_exp = dateutil.parser.parse(resp_json.get('ExpirationDate'))
+
+                region.set("titlovi_token", [self.user_id, self.login_token, self.token_exp])
+                logger.debug('New token obtained')
+
+            elif response.status_code == request_codes.unauthorized:
+                raise AuthenticationError('Login failed')
+
+        except RequestException as e:
+            logger.error(e)
    def terminate(self):
        self.session.close()

-    def query(self, languages, title, season=None, episode=None, year=None, video=None):
-        items_per_page = 10
-        current_page = 1
+    def query(self, languages, title, season=None, episode=None, year=None, imdb_id=None, video=None):
+        search_params = dict()

        used_languages = languages
        lang_strings = [str(lang) for lang in used_languages]
@@ -162,135 +208,73 @@ class TitloviProvider(Provider, ProviderSubtitleArchiveMixin):
        langs = '|'.join(map(str, [l.titlovi for l in used_languages]))

        # set query params
-        params = {'prijevod': title, 'jezik': langs}
+        search_params['query'] = title
+        search_params['lang'] = langs
        is_episode = False
        if season and episode:
            is_episode = True
-            params['s'] = season
-            params['e'] = episode
-        if year:
-            params['g'] = year
+            search_params['season'] = season
+            search_params['episode'] = episode
+        #if year:
+        #    search_params['year'] = year
+        if imdb_id:
+            search_params['imdbID'] = imdb_id

        # loop through paginated results
-        logger.info('Searching subtitles %r', params)
+        logger.info('Searching subtitles %r', search_params)
        subtitles = []
+        query_results = []

-        while True:
-            # query the server
-            try:
-                r = self.session.get(self.search_url, params=params, timeout=10)
-                r.raise_for_status()
-            except RequestException as e:
-                logger.exception('RequestException %s', e)
-                break
+        try:
+            search_params['token'] = self.login_token
+            search_params['userid'] = self.user_id
+            search_params['json'] = True
+
+            response = self.session.get(self.api_search_url, params=search_params)
+            resp_json = response.json()
+            if resp_json['SubtitleResults']:
+                query_results.extend(resp_json['SubtitleResults'])
+
+
+        except Exception as e:
+            logger.error(e)
+
+        for sub in query_results:
+
+            # title and alternate title
+            match = title_re.search(sub.get('Title'))
+            if match:
+                _title = match.group('title')
+                alt_title = match.group('altitle')
            else:
-                try:
-                    soup = BeautifulSoup(r.content, 'lxml')
+                continue

-                    # number of results
-                    result_count = int(soup.select_one('.results_count b').string)
-                except:
-                    result_count = None
+            # handle movies and series separately
+            if is_episode:
+                subtitle = self.subtitle_class(Language.fromtitlovi(sub.get('Lang')), sub.get('Link'), sub.get('Id'), sub.get('Release'), _title,
+                                               alt_title=alt_title, season=sub.get('Season'), episode=sub.get('Episode'),
+                                               year=sub.get('Year'), rating=sub.get('Rating'),
+                                               download_count=sub.get('DownloadCount'),
+                                               asked_for_release_group=video.release_group,
+                                               asked_for_episode=episode)
+            else:
+                subtitle = self.subtitle_class(Language.fromtitlovi(sub.get('Lang')), sub.get('Link'), sub.get('Id'), sub.get('Release'), _title,
+                                               alt_title=alt_title, year=sub.get('Year'), rating=sub.get('Rating'),
+                                               download_count=sub.get('DownloadCount'),
+                                               asked_for_release_group=video.release_group)
+            logger.debug('Found subtitle %r', subtitle)

-                # exit if no results
-                if not result_count:
-                    if not subtitles:
-                        logger.debug('No subtitles found')
-                    else:
-                        logger.debug("No more subtitles found")
-                    break
+            # prime our matches so we can use the values later
+            subtitle.get_matches(video)

-                # number of pages with results
-                pages = int(math.ceil(result_count / float(items_per_page)))
-
-                # get current page
-                if 'pg' in params:
-                    current_page = int(params['pg'])
-
-                try:
-                    sublist = soup.select('section.titlovi > ul.titlovi > li.subtitleContainer.canEdit')
-                    for sub in sublist:
-                        # subtitle id
-                        sid = sub.find(attrs={'data-id': True}).attrs['data-id']
-                        # get download link
-                        download_link = self.download_url + sid
-                        # title and alternate title
-                        match = title_re.search(sub.a.string)
-                        if match:
-                            _title = match.group('title')
-                            alt_title = match.group('altitle')
-                        else:
-                            continue
-
-                        # page link
-                        page_link = self.server_url + sub.a.attrs['href']
-                        # subtitle language
-                        _lang = sub.select_one('.lang')
-                        match = lang_re.search(_lang.attrs.get('src', _lang.attrs.get('cfsrc', '')))
-                        if match:
-                            try:
-                                # decode language
-                                lang = Language.fromtitlovi(match.group('lang')+match.group('script'))
-                            except ValueError:
-                                continue
-
-                        # relase year or series start year
-                        match = year_re.search(sub.find(attrs={'data-id': True}).parent.i.string)
-                        if match:
-                            r_year = int(match.group('year'))
-                        # fps
-                        match = fps_re.search(sub.select_one('.fps').string)
-                        if match:
-                            fps = match.group('fps')
-                        # releases
-                        releases = str(sub.select_one('.fps').parent.contents[0].string)
-
-                        # handle movies and series separately
-                        if is_episode:
-                            # season and episode info
-                            sxe = sub.select_one('.s0xe0y').string
-                            r_season = None
-                            r_episode = None
-                            if sxe:
-                                match = season_re.search(sxe)
-                                if match:
-                                    r_season = int(match.group('season'))
-                                match = episode_re.search(sxe)
-                                if match:
-                                    r_episode = int(match.group('episode'))
-
-                            subtitle = self.subtitle_class(lang, page_link, download_link, sid, releases, _title,
-                                                           alt_title=alt_title, season=r_season, episode=r_episode,
-                                                           year=r_year, fps=fps,
-                                                           asked_for_release_group=video.release_group,
-                                                           asked_for_episode=episode)
-                        else:
-                            subtitle = self.subtitle_class(lang, page_link, download_link, sid, releases, _title,
-                                                           alt_title=alt_title, year=r_year, fps=fps,
-                                                           asked_for_release_group=video.release_group)
-                        logger.debug('Found subtitle %r', subtitle)
-
-                        # prime our matches so we can use the values later
-                        subtitle.get_matches(video)
-
-                        # add found subtitles
-                        subtitles.append(subtitle)
-
-                finally:
-                    soup.decompose()
-
-                # stop on last page
-                if current_page >= pages:
-                    break
-
-                # increment current page
-                params['pg'] = current_page + 1
-                logger.debug('Getting page %d', params['pg'])
+            # add found subtitles
+            subtitles.append(subtitle)

        return subtitles

    def list_subtitles(self, video, languages):
        season = episode = None
+
        if isinstance(video, Episode):
            title = video.series
            season = video.season
@@ -300,6 +284,7 @@ class TitloviProvider(Provider, ProviderSubtitleArchiveMixin):

        return [s for s in
                self.query(languages, fix_inconsistent_naming(title), season=season, episode=episode, year=video.year,
+                           imdb_id=video.imdb_id,
                           video=video)]

    def download_subtitle(self, subtitle):
@@ -337,10 +322,12 @@ class TitloviProvider(Provider, ProviderSubtitleArchiveMixin):
        sub_to_extract = None

        for sub_name in subs_in_archive:
-            if not ('.cyr' in sub_name or '.cir' in sub_name):
+            _sub_name = sub_name.lower()
+
+            if not ('.cyr' in _sub_name or '.cir' in _sub_name or 'cyr)' in _sub_name):
                sr_lat_subs.append(sub_name)

-            if ('.cyr' in sub_name or '.cir' in sub_name) and not '.lat' in sub_name:
+            if ('.cyr' in sub_name or '.cir' in _sub_name) and not '.lat' in _sub_name.lower():
                sr_cyr_subs.append(sub_name)

        if subtitle.language == 'sr':
@@ -0,0 +1,235 @@
+# -*- coding: utf-8 -*-
+from requests import Session
+import logging
+from subzero.language import Language
+from guessit import guessit
+
+from subliminal_patch.providers import Provider
+from subliminal.providers import Episode, Movie
+from subliminal_patch.utils import sanitize
+from subliminal_patch.subtitle import Subtitle, guess_matches
+from subliminal.subtitle import fix_line_ending
+
+from io import BytesIO
+from zipfile import ZipFile
+
+__author__ = "Dor Nizar"
+
+logger = logging.getLogger(__name__)
+
+
+class WizdomSubtitle(Subtitle):
+    provider_name = 'wizdom'
+
+    def __init__(self, language, title_id, subtitle_id, series, season, episode, release, year, page_link):
+        super(WizdomSubtitle, self).__init__(language, subtitle_id)
+        self.title_id = title_id
+        self.subtitle_id = subtitle_id
+        self.series = series
+        self.season = season
+        self.episode = episode
+        self.release = release
+        self.year = year
+        self.page_link = page_link
+
+    def get_matches(self, video):
+        matches = set()
+        logger.debug("[{}]\n{}".format(self.__class__.__name__, self.__dict__))
+
+        # episode
+        if isinstance(video, Episode):
+            # series
+            if video.series and sanitize(self.series) == sanitize(video.series):
+                matches.add('series')
+            # season
+            if video.season and self.season == video.season:
+                matches.add('season')
+            # episode
+            if video.episode and self.episode == video.episode:
+                matches.add('episode')
+            # guess
+            matches |= guess_matches(video, guessit(self.release, {'type': 'episode'}))
+        # movie
+        elif isinstance(video, Movie):
+            # title
+            if video.title and (sanitize(self.series) in (
+                    sanitize(name) for name in [video.title] + video.alternative_titles)):
+                matches.add('title')
+            # year
+            if video.year and self.year == video.year:
+                matches.add('year')
+            # guess
+            matches |= guess_matches(video, guessit(self.release, {'type': 'movie'}))
+
+        logger.debug("Wizdom subtitle criteria match:\n{}".format(matches))
+        return matches
+
+    @property
+    def id(self):
+        return self.subtitle_id
+
+
+class WizdomProvider(Provider):
+    subtitle_class = WizdomSubtitle
+    languages = {Language.fromalpha2(lng) for lng in ['he']}
+    URL_JSON_SERVER = 'https://json.wizdom.xyz/'
+    URL_DOWNLOAD_SERVER = 'https://zip.wizdom.xyz/'
+    URL_WIZDOM_PAGELINK = 'https://wizdom.xyz/{}/{}'
+
+    URL_SEARCH = URL_JSON_SERVER + "search.php?search={}"
+    URL_INFO = URL_JSON_SERVER + "{}.json"
+    URL_DOWNLOAD_SUBTITLE = URL_DOWNLOAD_SERVER + "{}.zip"
+
+    def __init__(self):
+        self.session = None
+
+    def initialize(self):
+        logger.info("Wizdom initialize")
+        self.session = Session()
+        self.session.headers[
+            'User-Agent'] = 'Mozilla/5.0 (Windows NT 10.0; ' \
+                            'Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36'
+        return True
+
+    def terminate(self):
+        logger.info("Wizdom terminate")
+        self.session.close()
+
+    def _search_series(self, title, movie=False):
+        logger.debug("Searching '{}'".format(title))
+        r = self.session.get(self.URL_SEARCH.format(title))
+        if not r.ok:
+            logger.error("Bad response from server while searching: '{}']".format(title))
+            return []
+        try:
+            found = r.json()
+        except ValueError:
+            logger.error("Could not extract JSON from response")
+            return []
+
+        if movie:
+            series_list = [x for x in found if 'type' in x and x['type'] == 'movie']
+        else:
+            series_list = [x for x in found if 'type' in x and x['type'] == 'tv']
+
+        logger.debug("Found the following titles: {}".format([x['title_en'] for x in series_list if 'title_en' in x]))
+        return series_list
+
+    def _search_subtitles(self, title_id, season=None, episode=None):
+        r = self.session.get(self.URL_INFO.format(title_id))
+
+        if not r.ok:
+            logger.error("Bad response from server while searching subtitles [title_id={}]".format(title_id))
+            return []
+        if not r.content:
+            return []
+        try:
+            results = r.json()
+        except ValueError:
+            return []
+
+        if 'subs' not in results:
+            return []
+
+        if season and episode:
+            s_season = str(season)
+            s_episode = str(episode)
+            if s_season in results['subs'] and s_episode in results['subs'][s_season]:
+                return results['subs'][s_season][s_episode]
+        else:
+            return results['subs']
+        return []
+
+    def query(self, title, season=None, episode=None, year=None):
+        subtitles = []
+        if season and episode:
+            logger.debug("Searching for:\nTitle: {}\nSeason: {}\nEpisode: {}\nYear: {}".format(title, season,
+                                                                                               episode, year))
+            titles = self._search_series(title)
+        else:
+            logger.debug("Searching for:\nTitle: {}\nYear: {}\n".format(title, year))
+            titles = self._search_series(title, movie=True)
+
+        for title in titles:
+            title_name = title['title_en']
+            title_id = title['imdb']
+            if season and episode:
+                results = self._search_subtitles(title_id, season, episode)
+                page_link = self.URL_WIZDOM_PAGELINK.format("tv", title_id)
+            else:
+                results = self._search_subtitles(title_id)
+                page_link = self.URL_WIZDOM_PAGELINK.format("movie", title_id)
+
+            if not results:
+                logger.info("No subtitles found for: {}".format(title_name))
+                continue
+
+            for result in results:
+                subtitle_id, release = result['id'], result['version']
+                subtitles.append(self.subtitle_class(next(iter(self.languages)), title_id, subtitle_id,
+                                                     title_name, season, episode, release, year, page_link))
+
+        if subtitles:
+            logger.debug("Found Subtitle Candidates: {}".format([x.release for x in subtitles]))
+        return subtitles
+
+    def list_subtitles(self, video, languages):
+        season = episode = year = title = None
+
+        if isinstance(video, Episode):
+            logger.info("list_subtitles Series: {}, season: {}, episode: {}".format(video.series,
+                                                                                    video.season,
+                                                                                    video.episode))
+            title = video.series
+            season = video.season
+            if video.episode == 0:
+                episode = 1
+            else:
+                episode = video.episode
+        elif isinstance(video, Movie):
+            logger.info("list_subtitles Movie: {}, year: {}".format(video.title, video.year))
+            title = video.title
+            year = video.year
+
+        return [s for s in self.query(title, season, episode, year) if s.language in languages]
+
+    def _download_subtitles(self, subtitle_id):
+        logger.debug("Downloading subtitle id - {}".format(subtitle_id))
+        r = self.session.get(self.URL_DOWNLOAD_SUBTITLE.format(subtitle_id))
+
+        if not r.ok:
+            logger.error("Bad response from server while downloding zip [id={}]".format(subtitle_id))
+            return None
+        if not r.content:
+            logger.error("Unable to download zip [id={}]".format(subtitle_id))
+            return None
+
+        return r.content
+
+    def download_subtitle(self, subtitle):
+        # type: (WizdomSubtitle) -> None
+
+        logger.info('Downloading subtitle from WizdomSubs: %r', subtitle)
+
+        content = None
+        zip_content = self._download_subtitles(subtitle.subtitle_id)
+        if not zip_content:
+            return None
+        if not zip_content[:2] == "PK":
+            logger.warning("Response did not contain zip file")
+            return None
+
+        zfile = ZipFile(BytesIO(zip_content))
+        if not len(zfile.namelist()):
+            logger.warning("Response did not contain files inside the zip archive")
+            return None
+
+        sub_files = [x for x in zfile.namelist() if x.endswith(('.srt', '.idx', '.sub'))]
+        if sub_files:
+            content = zfile.open(sub_files[0]).read()
+
+        if not content:
+            logger.warning("File inside zip is empty")
+            return None
+        subtitle.content = fix_line_ending(content)
+        logger.info("Downloaded {} successfuly!".format(subtitle))
@@ -19,11 +19,11 @@ class DroneAPIClient(object):
    _fill_attrs = None

    def __init__(self, version=1, session=None, headers=None, timeout=10, base_url=None, api_key=None,
-                 ssl_no_verify=False):
+                 ssl_no_verify=False, pem_file=None):
        headers = dict(headers or {}, **{"X-Api-Key": api_key})

        #: Session for the requests
-        self.session = session or CertifiSession()
+        self.session = session or CertifiSession(verify=pem_file)
        if ssl_no_verify:
            self.session.verify = False

@@ -87,7 +87,10 @@ def refine(video, **kwargs):
        # parse series year
        series_year = None
        if result['firstAired']:
-            series_year = datetime.datetime.strptime(result['firstAired'], '%Y-%m-%d').year
+            try:
+                series_year = datetime.datetime.strptime(result['firstAired'], '%Y-%m-%d').year
+            except ValueError:
+                continue

        # discard mismatches on year
        if video.year and series_year and video.year != series_year:
@@ -60,6 +60,8 @@ def compute_score(matches, subtitle, video, hearing_impaired=None):
    episode_hash_valid_if = {"series", "season", "episode", "format"}
    movie_hash_valid_if = {"video_codec", "format"}

+    orig_matches = matches.copy()
+
    # on hash match, discard everything else
    if subtitle.hash_verifiable:
        if 'hash' in matches:
@@ -83,41 +85,47 @@ def compute_score(matches, subtitle, video, hearing_impaired=None):
        matches &= {'hash'}

    # handle equivalent matches
+    eq_matches = set()
    if is_episode:
        if 'title' in matches:
            logger.debug('Adding title match equivalent')
-            matches.add('episode')
+            eq_matches.add('episode')
        if 'series_imdb_id' in matches:
            logger.debug('Adding series_imdb_id match equivalent')
-            matches |= {'series', 'year'}
+            eq_matches |= {'series', 'year'}
        if 'imdb_id' in matches:
            logger.debug('Adding imdb_id match equivalents')
-            matches |= {'series', 'year', 'season', 'episode'}
+            eq_matches |= {'series', 'year', 'season', 'episode'}
        if 'tvdb_id' in matches:
            logger.debug('Adding tvdb_id match equivalents')
-            matches |= {'series', 'year', 'season', 'episode', 'title'}
+            eq_matches |= {'series', 'year', 'season', 'episode', 'title'}
        if 'series_tvdb_id' in matches:
            logger.debug('Adding series_tvdb_id match equivalents')
-            matches |= {'series', 'year'}
+            eq_matches |= {'series', 'year'}

        # specials
        if video.is_special and 'title' in matches and 'series' in matches \
                and 'year' in matches:
            logger.debug('Adding special title match equivalent')
-            matches |= {'season', 'episode'}
+            eq_matches |= {'season', 'episode'}

    elif is_movie:
        if 'imdb_id' in matches:
            logger.debug('Adding imdb_id match equivalents')
-            matches |= {'title', 'year'}
+            eq_matches |= {'title', 'year'}
+
+    matches |= eq_matches

    # handle hearing impaired
    if hearing_impaired is not None and subtitle.hearing_impaired == hearing_impaired:
        logger.debug('Matched hearing_impaired')
        matches.add('hearing_impaired')
+        orig_matches.add('hearing_impaired')

    # compute the score
    score = sum((scores.get(match, 0) for match in matches))
    logger.info('%r: Computed score %r with final matches %r', subtitle, score, matches)

-    return score
+    score_without_hash = sum((scores.get(match, 0) for match in orig_matches | eq_matches if match != "hash"))
+
+    return score, score_without_hash
@@ -19,6 +19,15 @@ from subliminal import Subtitle as Subtitle_
 from subliminal.subtitle import Episode, Movie, sanitize_release_group, get_equivalent_release_groups
 from subliminal_patch.utils import sanitize
 from ftfy import fix_text
+from codecs import BOM_UTF8, BOM_UTF16_BE, BOM_UTF16_LE, BOM_UTF32_BE, BOM_UTF32_LE
+
+BOMS = (
+    (BOM_UTF8, "UTF-8"),
+    (BOM_UTF32_BE, "UTF-32-BE"),
+    (BOM_UTF32_LE, "UTF-32-LE"),
+    (BOM_UTF16_BE, "UTF-16-BE"),
+    (BOM_UTF16_LE, "UTF-16-LE"),
+)

 logger = logging.getLogger(__name__)

@@ -105,6 +114,9 @@ class Subtitle(Subtitle_):
        # normalize line endings
        self.content = self.content.replace("\r\n", "\n").replace('\r', '\n')

+    def _check_bom(self, data):
+        return [encoding for bom, encoding in BOMS if data.startswith(bom)]
+
    def guess_encoding(self):
        """Guess encoding using the language, falling back on chardet.

@@ -119,11 +131,17 @@ class Subtitle(Subtitle_):

        encodings = ['utf-8']

+        # check UTF BOMs
+        bom_encodings = self._check_bom(self.content)
+        if bom_encodings:
+            encodings = list(set(enc.lower() for enc in bom_encodings + encodings))
+
        # add language-specific encodings
        # http://scratchpad.wikia.com/wiki/Character_Encoding_Recommendation_for_Languages

        if self.language.alpha3 == 'zho':
-            encodings.extend(['cp936', 'gb2312', 'cp950', 'gb18030', 'big5', 'big5hkscs'])
+            encodings.extend(['cp936', 'gb2312', 'gbk', 'gb18030', 'hz', 'iso2022_jp_2', 'cp950', 'gb18030', 'big5',
+                              'big5hkscs', 'utf-16'])
        elif self.language.alpha3 == 'jpn':
            encodings.extend(['shift-jis', 'cp932', 'euc_jp', 'iso2022_jp', 'iso2022_jp_1', 'iso2022_jp_2',
                              'iso2022_jp_2004', 'iso2022_jp_3', 'iso2022_jp_ext', ])
@@ -132,7 +150,7 @@ class Subtitle(Subtitle_):

        # arabian/farsi
        elif self.language.alpha3 in ('ara', 'fas', 'per'):
-            encodings.append('windows-1256')
+            encodings.extend(['windows-1256', 'utf-16'])
        elif self.language.alpha3 == 'heb':
            encodings.extend(['windows-1255', 'iso-8859-8'])
        elif self.language.alpha3 == 'tur':
@@ -250,8 +268,7 @@ class Subtitle(Subtitle_):
                subs = pysubs2.SSAFile.from_string(text, fps=self.plex_media_fps)

            unicontent = self.pysubs2_to_unicode(subs)
-            self.content = unicontent.encode("utf-8")
-            self._guessed_encoding = "utf-8"
+            self.content = unicontent.encode(self._guessed_encoding)
        except:
            logger.exception("Couldn't convert subtitle %s to .srt format: %s", self, traceback.format_exc())
            return False
@@ -261,6 +278,12 @@ class Subtitle(Subtitle_):

    @classmethod
    def pysubs2_to_unicode(cls, sub, format="srt"):
+        """
+        this is a modified version of pysubs2.SubripFormat.to_file with special handling for drawing tags in ASS
+        :param sub:
+        :param format:
+        :return:
+        """
        def ms_to_timestamp(ms, mssep=","):
            """Convert ms to 'HH:MM:SS,mmm'"""
            # XXX throw on overflow/underflow?
@@ -272,9 +295,12 @@ class Subtitle(Subtitle_):
        def prepare_text(text, style):
            body = []
            for fragment, sty in parse_tags(text, style, sub.styles):
-                fragment = fragment.replace(ur"\h", u" ")
-                fragment = fragment.replace(ur"\n", u"\n")
-                fragment = fragment.replace(ur"\N", u"\n")
+                fragment = fragment.replace(r"\h", u" ")
+                fragment = fragment.replace(r"\n", u"\n")
+                fragment = fragment.replace(r"\N", u"\n")
+                if sty.drawing:
+                    raise pysubs2.ContentNotUsable
+
                if format == "srt":
                    if sty.italic:
                        fragment = u"<i>%s</i>" % fragment
@@ -306,7 +332,10 @@ class Subtitle(Subtitle_):
        for i, line in enumerate(visible_lines, 1):
            start = ms_to_timestamp(line.start, mssep=mssep)
            end = ms_to_timestamp(line.end, mssep=mssep)
-            text = prepare_text(line.text, sub.styles.get(line.style, SSAStyle.DEFAULT_STYLE))
+            try:
+                text = prepare_text(line.text, sub.styles.get(line.style, SSAStyle.DEFAULT_STYLE))
+            except pysubs2.ContentNotUsable:
+                continue

            out.append(u"%d\n" % i)
            out.append(u"%s --> %s\n" % (start, end))
@@ -319,7 +348,8 @@ class Subtitle(Subtitle_):
        :return: string 
        """
        if not self.mods:
-            return fix_text(self.content.decode("utf-8"), **ftfy_defaults).encode(encoding="utf-8")
+            return fix_text(self.content.decode(encoding=self._guessed_encoding), **ftfy_defaults).encode(
+                encoding=self._guessed_encoding)

        submods = SubtitleModifications(debug=debug)
        if submods.load(content=self.text, language=self.language):
@@ -328,7 +358,7 @@ class Subtitle(Subtitle_):
            self.mods = submods.mods_used

            content = fix_text(self.pysubs2_to_unicode(submods.f, format=format), **ftfy_defaults)\
-                .encode(encoding="utf-8")
+                .encode(encoding=self._guessed_encoding)
            submods.f = None
            del submods
            return content
@@ -339,6 +369,15 @@ class ModifiedSubtitle(Subtitle):
    id = None


+MERGED_FORMATS = {
+    "TV": ("HDTV", "SDTV", "AHDTV", "UHDTV"),
+    "Air": ("SATRip", "DVB", "PPV"),
+    "Disk": ("DVD", "HD-DVD", "BluRay")
+}
+
+MERGED_FORMATS_REV = dict((v.lower(), k.lower()) for k in MERGED_FORMATS for v in MERGED_FORMATS[k])
+
+
 def guess_matches(video, guess, partial=False):
    """Get matches between a `video` and a `guess`.

@@ -421,21 +460,25 @@ def guess_matches(video, guess, partial=False):
            formats = [formats]

        if video.format:
-            video_format = video.format
-            if video_format in ("HDTV", "SDTV", "TV"):
-                video_format = "TV"
-                logger.debug("Treating HDTV/SDTV the same")
+            video_format = video.format.lower()
+            _video_gen_format = MERGED_FORMATS_REV.get(video_format)
+            if _video_gen_format:
+                logger.debug("Treating %s as %s the same", video_format, _video_gen_format)

            for frmt in formats:
-                if frmt in ("HDTV", "SDTV"):
-                    frmt = "TV"
+                _guess_gen_frmt = MERGED_FORMATS_REV.get(frmt.lower())

-                if frmt.lower() == video_format.lower():
+                if _guess_gen_frmt == _video_gen_format:
                    matches.add('format')
                    break
+        if "release_group" in matches and "format" not in matches:
+            logger.info("Release group matched but format didn't. Remnoving release group match.")
+            matches.remove("release_group")
+
    # video_codec
    if video.video_codec and 'video_codec' in guess and guess['video_codec'] == video.video_codec:
        matches.add('video_codec')
+
    # audio_codec
    if video.audio_codec and 'audio_codec' in guess and guess['audio_codec'] == video.audio_codec:
        matches.add('audio_codec')
@@ -21,9 +21,10 @@ if debug:
    logging.basicConfig(level=logging.DEBUG)

 #sub = Subtitle(Language.fromietf("eng:forced"), mods=["common", "remove_HI", "OCR_fixes", "fix_uppercase", "shift_offset(ms=-500)", "shift_offset(ms=500)", "shift_offset(s=2,ms=800)"])
-sub = Subtitle(Language.fromietf("eng:forced"), mods=["common", "remove_HI", "OCR_fixes", "fix_uppercase", "shift_offset(ms=0,s=1)"])
+sub = Subtitle(Language.fromietf("eng"), mods=["common", "remove_HI", "OCR_fixes", "fix_uppercase", "shift_offset(ms=0,s=1)"])
 sub.content = open(fn).read()
 sub.normalize()
+sub.is_valid()
 content = sub.get_modified_content(debug=True)

 #submod = SubMod(debug=debug)
@@ -28,6 +28,9 @@ import re

 import enum
 import sys
+import requests
+import time
+import logging

 is_PY2 = sys.version_info[0] < 3
 if is_PY2:
@@ -37,8 +40,13 @@ else:
    from contextlib import suppress
    from urllib2.request import Request, urlopen

+from dogpile.cache.api import NO_VALUE
+from subliminal.cache import region
 from bs4 import BeautifulSoup, NavigableString

+
+logger = logging.getLogger(__name__)
+
 # constants
 HEADERS = {
 }
@@ -48,14 +56,23 @@ DEFAULT_USER_AGENT = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWeb"\
                     "Kit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.47 Safari/537.36"


+ENDPOINT_RE = re.compile(ur'(?uis)<form.+?action="/subtitles/(.+)">.*?<input type="text"')
+
+
+class NewEndpoint(Exception):
+    pass
+
+
 # utils
-def soup_for(url, session=None, user_agent=DEFAULT_USER_AGENT):
+def soup_for(url, data=None, session=None, user_agent=DEFAULT_USER_AGENT):
    url = re.sub("\s", "+", url)
    if not session:
        r = Request(url, data=None, headers=dict(HEADERS, **{"User-Agent": user_agent}))
        html = urlopen(r).read().decode("utf-8")
    else:
-        html = session.get(url).text
+        ret = session.post(url, data=data)
+        ret.raise_for_status()
+        html = ret.text
    return BeautifulSoup(html, "html.parser")


@@ -108,7 +125,7 @@ class Subtitle(object):
        subtitles = []

        for row in rows:
-            if row.td.a is not None:
+            if row.td.a is not None and row.td.get("class", ["lazy"])[0] != "empty":
                subtitles.append(cls.from_row(row))

        return subtitles
@@ -238,22 +255,52 @@ def get_first_film(soup, section, year=None, session=None):
                url = SITE_DOMAIN + t.div.a.get("href")
                break
        if not url:
-            return
+            # fallback to non-year results
+            logger.info("Falling back to non-year results as year wasn't found (%s)", year)
+            url = SITE_DOMAIN + tag.findNext("ul").find("li").div.a.get("href")

    return Film.from_url(url, session=session)


-def search(term, release=True, session=None, year=None, limit_to=SearchTypes.Exact):
-    soup = soup_for("%s/subtitles/%s?q=%s" % (SITE_DOMAIN, "release" if release else "title", term), session=session)
+def find_endpoint(session, content=None):
+    endpoint = region.get("subscene_endpoint2")
+    if endpoint is NO_VALUE:
+        if not content:
+            content = session.get(SITE_DOMAIN).text

-    if "Subtitle search by" in str(soup):
-        rows = soup.find("table").tbody.find_all("tr")
-        subtitles = Subtitle.from_rows(rows)
-        return Film(term, subtitles=subtitles)
+        m = ENDPOINT_RE.search(content)
+        if m:
+            endpoint = m.group(1).strip()
+            logger.debug("Switching main endpoint to %s", endpoint)
+            region.set("subscene_endpoint2", endpoint)
+    return endpoint

-    for junk, search_type in SearchTypes.__members__.items():
-        if section_exists(soup, search_type):
-            return get_first_film(soup, search_type, year=year, session=session)

-        if limit_to == search_type:
-            return
+def search(term, release=True, session=None, year=None, limit_to=SearchTypes.Exact, throttle=0):
+    # note to subscene: if you actually start to randomize the endpoint, we'll have to query your server even more
+
+    if release:
+        endpoint = "release"
+    else:
+        endpoint = find_endpoint(session)
+        time.sleep(throttle)
+
+    if not endpoint:
+        logger.error("Couldn't find endpoint, exiting")
+        return
+
+    soup = soup_for("%s/subtitles/%s" % (SITE_DOMAIN, endpoint), data={"query": term},
+                    session=session)
+
+    if soup:
+        if "Subtitle search by" in str(soup):
+            rows = soup.find("table").tbody.find_all("tr")
+            subtitles = Subtitle.from_rows(rows)
+            return Film(term, subtitles=subtitles)
+
+        for junk, search_type in SearchTypes.__members__.items():
+            if section_exists(soup, search_type):
+                return get_first_film(soup, search_type, year=year, session=session)
+
+            if limit_to == search_type:
+                return
@@ -1,15 +1,47 @@
 # coding=utf-8
+from __future__ import absolute_import
 import types
+import re

 from babelfish.exceptions import LanguageError
-from babelfish import Language as Language_, basestr
+from babelfish import Language as Language_, basestr, LANGUAGE_MATRIX
+from six.moves import zip

 repl_map = {
    "dk": "da",
    "nld": "nl",
    "english": "en",
+    "alb": "sq",
+    "arm": "hy",
+    "baq": "eu",
+    "bur": "my",
+    "chi": "zh",
+    "cze": "cs",
+    "dut": "nl",
+    "fre": "fr",
+    "geo": "ka",
+    "ger": "de",
+    "gre": "el",
+    "ice": "is",
+    "mac": "mk",
+    "mao": "mi",
+    "may": "ms",
+    "per": "fa",
+    "rum": "ro",
+    "slo": "sk",
+    "tib": "bo",
 }

+CUSTOM_LIST = ["chs", "sc", "zhs", "hans", "gb", u"简", u"双语",
+               "cht", "tc", "zht", "hant", "big5", u"繁", u"雙語",
+               "spl", "ea", "pob", "pb"]
+
+ALPHA2_LIST = list(set(filter(lambda x: x, map(lambda x: x.alpha2, LANGUAGE_MATRIX)))) + list(repl_map.values())
+ALPHA3b_LIST = list(set(filter(lambda x: x, map(lambda x: x.alpha3, LANGUAGE_MATRIX)))) + \
+               list(set(filter(lambda x: len(x) == 3, list(repl_map.keys()))))
+FULL_LANGUAGE_LIST = ALPHA2_LIST + ALPHA3b_LIST
+FULL_LANGUAGE_LIST.extend(CUSTOM_LIST)
+

 def language_from_stream(l):
    if not l:
@@ -35,7 +67,8 @@ def wrap_forced(f):
        args = args[1:]
        s = args.pop(0)
        forced = None
-        if isinstance(s, types.StringTypes):
+        hi = None
+        if isinstance(s, (str,)):
            base, forced = s.split(":") if ":" in s else (s, False)
        else:
            base = s
@@ -43,6 +76,7 @@ def wrap_forced(f):
        instance = f(cls, base, *args, **kwargs)
        if isinstance(instance, Language):
            instance.forced = forced == "forced"
+            instance.hi = hi == "hi"
        return instance

    return inner
@@ -50,16 +84,21 @@ def wrap_forced(f):

 class Language(Language_):
    forced = False
+    hi = False

-    def __init__(self, language, country=None, script=None, unknown=None, forced=False):
+    def __init__(self, language, country=None, script=None, unknown=None, forced=False, hi=False):
        self.forced = forced
+        self.hi = hi
        super(Language, self).__init__(language, country=country, script=script, unknown=unknown)

    def __getstate__(self):
-        return self.alpha3, self.country, self.script, self.forced
+        return self.alpha3, self.country, self.script, self.hi, self.forced

-    def __setstate__(self, state):
-        self.alpha3, self.country, self.script, self.forced = state
+    def __setstate__(self, forced):
+        self.alpha3, self.country, self.script, self.hi, self.forced = forced
+
+    def __hash__(self):
+        return hash(str(self))

    def __eq__(self, other):
        if isinstance(other, basestr):
@@ -69,11 +108,16 @@ class Language(Language_):
        return (self.alpha3 == other.alpha3 and
                self.country == other.country and
                self.script == other.script and
-                bool(self.forced) == bool(other.forced))
+                bool(self.forced) == bool(other.forced) and
+                bool(self.hi) == bool(other.hi))

    def __str__(self):
        return super(Language, self).__str__() + (":forced" if self.forced else "")

+    def __repr__(self):
+        info = ";".join("{}={}".format(k, v) for k, v in vars(self).items() if v)
+        return "<{}: {}>".format(self.__class__.__name__, info)
+
    @property
    def basename(self):
        return super(Language, self).__str__()
@@ -82,14 +126,15 @@ class Language(Language_):
        ret = super(Language, self).__getattr__(name)
        if isinstance(ret, Language):
            ret.forced = self.forced
+            ret.hi = self.hi
        return ret

    @classmethod
    def rebuild(cls, instance, **replkw):
        state = instance.__getstate__()
-        attrs = ("country", "script", "forced")
+        attrs = ("country", "script", "hi", "forced")
        language = state[0]
-        kwa = dict(zip(attrs, state[1:]))
+        kwa = dict(list(zip(attrs, state[1:])))
        kwa.update(replkw)
        return cls(language, **kwa)

@@ -115,3 +160,16 @@ class Language(Language_):
            return Language(*Language_.fromietf(s).__getstate__())

        return Language(*Language_.fromalpha3b(s).__getstate__())
+
+
+IETF_MATCH = ".+\.([^-.]+)(?:-[A-Za-z]+)?$"
+ENDSWITH_LANGUAGECODE_RE = re.compile("\.([^-.]{2,3})(?:-[A-Za-z]{2,})?$")
+
+
+def match_ietf_language(s, ietf=False):
+    language_match = re.match(".+\.([^\.]+)$" if not ietf
+                              else IETF_MATCH, s)
+    if language_match and len(language_match.groups()) == 1:
+        language = language_match.groups()[0]
+        return language
+    return s
@@ -107,6 +107,12 @@ class Dicked(object):
        for key, value in entries.iteritems():
            self.__dict__[key] = (Dicked(**value) if isinstance(value, dict) else value)

+    def has(self, key):
+        return self._entries is not None and key in self._entries
+
+    def get(self, key, default=None):
+        return self._entries.get(key, default) if self._entries else default
+
    def __repr__(self):
        return str(self)

@@ -36,6 +36,7 @@ SZ_FIX_DATA = {
            u" l ": u" I ",
            u"'sjust": u"'s just",
            u"'tjust": u"'t just",
+            u"\";": u"'s",
        },
        "WholeWords": {
            u"I'11": u"I'll",
@@ -0,0 +1,7 @@
+# coding=utf-8
+class EmptyEntryError(Exception):
+    pass
+
+
+class EmptyLineError(Exception):
+    pass
@@ -6,7 +6,8 @@ import pysubs2
 import logging
 import time

-from mods import EMPTY_TAG_PROCESSOR, EmptyEntryError
+from mods import EMPTY_TAG_PROCESSOR
+from exc import EmptyEntryError
 from registry import registry
 from subzero.language import Language

@@ -293,15 +294,18 @@ class SubtitleModifications(object):
                    end_tag = line[-5:]
                    line = line[:-5]

+                last_procs_mods = []
+
+                # fixme: this double loop is ugly
                for order, identifier, args in mods:
                    mod = self.initialized_mods[identifier]

                    try:
-                        line = mod.modify(line.strip(), entry=entry.text, debug=self.debug, parent=self, index=index,
+                        line = mod.modify(line.strip(), entry=t, debug=self.debug, parent=self, index=index,
                                          **args)
                    except EmptyEntryError:
                        if self.debug:
-                            logger.debug(u"%d: %s: %r -> ''", index, identifier, entry.text)
+                            logger.debug(u"%d: %s: %r -> ''", index, identifier, t)
                        skip_entry = True
                        break

@@ -312,6 +316,33 @@ class SubtitleModifications(object):
                        break

                    applied_mods.append(identifier)
+                    if mod.last_processors:
+                        last_procs_mods.append([identifier, args])
+
+                if skip_entry:
+                    lines = []
+                    break
+
+                if skip_line:
+                    continue
+
+                for identifier, args in last_procs_mods:
+                    mod = self.initialized_mods[identifier]
+
+                    try:
+                        line = mod.modify(line.strip(), entry=t, debug=self.debug, parent=self, index=index,
+                                          procs=["last_process"], **args)
+                    except EmptyEntryError:
+                        if self.debug:
+                            logger.debug(u"%d: %s: %r -> ''", index, identifier, t)
+                        skip_entry = True
+                        break
+
+                    if not line:
+                        if self.debug:
+                            logger.debug(u"%d: %s: %r -> ''", index, identifier, old_line)
+                        skip_line = True
+                        break

                if skip_entry:
                    lines = []
@@ -21,6 +21,7 @@ class SubtitleModification(object):
    pre_processors = []
    processors = []
    post_processors = []
+    last_processors = []
    languages = []

    def __init__(self, parent):
@@ -67,15 +68,16 @@ class SubtitleModification(object):
    def post_process(self, content, debug=False, parent=None, **kwargs):
        return self._process(content, self.post_processors, debug=debug, parent=parent, **kwargs)

-    def modify(self, content, debug=False, parent=None, **kwargs):
+    def modify(self, content, debug=False, parent=None, procs=None, **kwargs):
        if not content:
            return

        new_content = content
-        for method in ("pre_process", "process", "post_process"):
+        for method in procs or ("pre_process", "process", "post_process"):
            if not new_content:
                return
-            new_content = getattr(self, method)(new_content, debug=debug, parent=parent, **kwargs)
+            new_content = self._process(new_content, getattr(self, "%sors" % method),
+                                        debug=debug, parent=parent, **kwargs)

        return new_content

@@ -105,5 +107,3 @@ empty_line_post_processors = [
 ]


-class EmptyEntryError(Exception):
-    pass
@@ -7,6 +7,7 @@ from subzero.modification.mods import SubtitleTextModification, empty_line_post_
 from subzero.modification.processors import FuncProcessor
 from subzero.modification.processors.re_processor import NReProcessor
 from subzero.modification import registry
+from tld import get_tld


 ENGLISH = Language("eng")
@@ -28,7 +29,7 @@ class CommonFixes(SubtitleTextModification):
        NReProcessor(re.compile(r'(?u)(\w|\b|\s|^)(-\s?-{1,2})'), ur"\1—", name="CM_multidash"),

        # line = _/-/\s
-        NReProcessor(re.compile(r'(?u)(^\W*[-_.:>~]+\W*$)'), "", name="CM_non_word_only"),
+        NReProcessor(re.compile(r'(?u)(^\W*[-_.:<>~"\']+\W*$)'), "", name="CM_non_word_only"),

        # remove >>
        NReProcessor(re.compile(r'(?u)^\s?>>\s*'), "", name="CM_leading_crocodiles"),
@@ -37,7 +38,7 @@ class CommonFixes(SubtitleTextModification):
        NReProcessor(re.compile(r'(?u)(^\W*:\s*(?=\w+))'), "", name="CM_empty_colon_start"),

        # fix music symbols
-        NReProcessor(re.compile(ur'(?u)(^[-\s>~]*[*#¶]+\s*)|(\s*[*#¶]+\s*$)'),
+        NReProcessor(re.compile(ur'(?u)(^[-\s>~]*[*#¶]+\s+)|(\s*[*#¶]+\s*$)'),
                     lambda x: u"♪ " if x.group(1) else u" ♪",
                     name="CM_music_symbols"),

@@ -113,7 +114,9 @@ class CommonFixes(SubtitleTextModification):
        NReProcessor(re.compile(r'(?u)(?:(?<=^)|(?<=\w)) +([!?.,](?![!?.,]| \.))'), r"\1", name="CM_punctuation_space"),

        # add space after punctuation
-        NReProcessor(re.compile(r'(?u)([!?.,:])([A-zÀ-ž]{2,})'), r"\1 \2", name="CM_punctuation_space2"),
+        NReProcessor(re.compile(r'(?u)(([^\s]*)([!?.,:])([A-zÀ-ž]{2,}))'),
+                     lambda match: u"%s%s %s" % (match.group(2), match.group(3), match.group(4)) if not get_tld(match.group(1), fail_silently=True, fix_protocol=True) else match.group(1),
+                     name="CM_punctuation_space2"),

        # fix lowercase I in english
        NReProcessor(re.compile(r'(?u)(\b)i(\b)'), r"\1I\2", name="CM_EN_lowercase_i",
@@ -178,6 +181,14 @@ class FixUppercase(SubtitleModification):
            entry.plaintext = self.capitalize(entry.plaintext)


+"""
+subsync
+
+subsync --cli --offline --overwrite --window-size=600 --max-point-dist=2.0 --min-points-no=20 --min-word-prob=0.3 --min-word-len=5 --min-correlation=0.9999 --min-words-sim=0.6 --out-time-offset=-0.08 sync --out SUBTITLE -s SUBTITLE --sub-lang=eng --sub-enc=utf-8 -r REF_FILE --ref-stream-by-type=audio --ref-stream-by-lang=eng
+
+"""
+
+
 registry.register(CommonFixes)
 registry.register(RemoveTags)
 registry.register(ReverseRTL)
@@ -1,7 +1,8 @@
 # coding=utf-8
 import re

-from subzero.modification.mods import SubtitleTextModification, empty_line_post_processors, EmptyEntryError, TAG
+from subzero.modification.mods import SubtitleTextModification, empty_line_post_processors, TAG
+from subzero.modification.exc import EmptyEntryError
 from subzero.modification.processors.re_processor import NReProcessor
 from subzero.modification import registry

@@ -46,14 +47,14 @@ class HearingImpaired(SubtitleTextModification):
                     name="HI_before_colon_noncaps"),

        # brackets (only remove if at least 3 chars in brackets)
-        NReProcessor(re.compile(ur'(?sux)-?%(t)s[([][^([)\]]+?(?=[A-zÀ-ž"\'.]{3,})[^([)\]]+[)\]][\s:]*%(t)s' %
+        NReProcessor(re.compile(ur'(?sux)-?%(t)s["\']*[([][^([)\]]+?(?=[A-zÀ-ž"\'.]{3,})[^([)\]]+[)\]]["\']*[\s:]*%(t)s' %
                                {"t": TAG}), "", name="HI_brackets"),

-        NReProcessor(re.compile(ur'(?sux)-?%(t)s[([]%(t)s(?=[A-zÀ-ž"\'.]{3,})[^([)\]]+%(t)s$' % {"t": TAG}),
-                     "", name="HI_bracket_open_start"),
+        #NReProcessor(re.compile(ur'(?sux)-?%(t)s[([]%(t)s(?=[A-zÀ-ž"\'.]{3,})[^([)\]]+%(t)s$' % {"t": TAG}),
+        #             "", name="HI_bracket_open_start"),

-        NReProcessor(re.compile(ur'(?sux)-?%(t)s(?=[A-zÀ-ž"\'.]{3,})[^([)\]]+[)\]][\s:]*%(t)s' % {"t": TAG}), "",
-                     name="HI_bracket_open_end"),
+        #NReProcessor(re.compile(ur'(?sux)-?%(t)s(?=[A-zÀ-ž"\'.]{3,})[^([)\]]+[)\]][\s:]*%(t)s' % {"t": TAG}), "",
+        #             name="HI_bracket_open_end"),

        # text before colon (and possible dash in front), max 11 chars after the first whitespace (if any)
        # NReProcessor(re.compile(r'(?u)(^[A-z\-\'"_]+[\w\s]{0,11}:[^0-9{2}][\s]*)'), "", name="HI_before_colon"),
@@ -73,7 +74,7 @@ class HearingImpaired(SubtitleTextModification):
                     supported=lambda p: not p.only_uppercase),

        # remove MAN:
-        NReProcessor(re.compile(ur'(?suxi)(.*MAN:\s*)'), "", name="HI_remove_man"),
+        NReProcessor(re.compile(ur'(?suxi)(\b(?:WO)MAN:\s*)'), "", name="HI_remove_man"),

        # dash in front
        # NReProcessor(re.compile(r'(?u)^\s*-\s*'), "", name="HI_starting_dash"),
@@ -81,13 +82,18 @@ class HearingImpaired(SubtitleTextModification):
        # all caps at start before new sentence
        NReProcessor(re.compile(ur'(?u)^(?=[A-ZÀ-Ž]{4,})[A-ZÀ-Ž-_\s]+\s([A-ZÀ-Ž][a-zà-ž].+)'), r"\1",
                     name="HI_starting_upper_then_sentence", supported=lambda p: not p.only_uppercase),
-
-        # remove music symbols
-        NReProcessor(re.compile(ur'(?u)(^%(t)s[*#¶♫♪\s]*%(t)s[*#¶♫♪\s]+%(t)s[*#¶♫♪\s]*%(t)s$)' % {"t": TAG}),
-                     "", name="HI_music_symbols_only"),
    ]

    post_processors = empty_line_post_processors
+    last_processors = [
+        # remove music symbols
+        NReProcessor(re.compile(ur'(?u)(^%(t)s[*#¶♫♪\s]*%(t)s[*#¶♫♪\s]+%(t)s[*#¶♫♪\s]*%(t)s$)' % {"t": TAG}),
+                     "", name="HI_music_symbols_only"),
+
+        # remove music entries
+        NReProcessor(re.compile(ur'(?ums)(^[-\s>~]*[*#¶♫♪]+\s*.+|.+\s*[*#¶♫♪]+\s*$)'),
+                     "", name="HI_music", entry=True),
+    ]


 registry.register(HearingImpaired)
@@ -10,7 +10,7 @@ class Processor(object):
    supported = None
    enabled = True

-    def __init__(self, name=None, parent=None, supported=None):
+    def __init__(self, name=None, parent=None, supported=None, **kwargs):
        self.name = name
        self.parent = parent
        self.supported = supported if supported else lambda parent: True
@@ -35,7 +35,7 @@ class Processor(object):
 class FuncProcessor(Processor):
    func = None

-    def __init__(self, func, name=None, parent=None, supported=None):
+    def __init__(self, func, name=None, parent=None, supported=None, **kwargs):
        super(FuncProcessor, self).__init__(name=name, supported=supported)
        self.func = func

@@ -2,6 +2,7 @@
 import re
 import logging

+from subzero.modification.exc import EmptyEntryError
 from subzero.modification.processors import Processor

 logger = logging.getLogger(__name__)
@@ -14,13 +15,22 @@ class ReProcessor(Processor):
    pattern = None
    replace_with = None

-    def __init__(self, pattern, replace_with, name=None, supported=None):
+    def __init__(self, pattern, replace_with, name=None, supported=None, entry=False, **kwargs):
        super(ReProcessor, self).__init__(name=name, supported=supported)
        self.pattern = pattern
        self.replace_with = replace_with
+        self.use_entry = entry

-    def process(self, content, debug=False, **kwargs):
-        return self.pattern.sub(self.replace_with, content)
+    def process(self, content, debug=False, entry=None, **kwargs):
+        if not self.use_entry:
+            return self.pattern.sub(self.replace_with, content)
+
+        ret = self.pattern.sub(self.replace_with, entry)
+        if not ret:
+            raise EmptyEntryError()
+        elif ret != entry:
+            return ret
+        return content


 class NReProcessor(ReProcessor):
@@ -36,7 +46,7 @@ class MultipleWordReProcessor(ReProcessor):
    }
    replaces found key in pattern with the corresponding value in data
    """
-    def __init__(self, snr_dict, name=None, parent=None, supported=None):
+    def __init__(self, snr_dict, name=None, parent=None, supported=None, **kwargs):
        super(ReProcessor, self).__init__(name=name, supported=supported)
        self.snr_dict = snr_dict

@@ -12,7 +12,7 @@ class StringProcessor(Processor):
    String replacement processor base
    """

-    def __init__(self, search, replace, name=None, parent=None, supported=None):
+    def __init__(self, search, replace, name=None, parent=None, supported=None, **kwargs):
        super(StringProcessor, self).__init__(name=name, supported=supported)
        self.search = search
        self.replace = replace
@@ -31,7 +31,7 @@ class MultipleLineProcessor(Processor):
        "data": {"old_value": "new_value"}
    }
    """
-    def __init__(self, snr_dict, name=None, parent=None, supported=None):
+    def __init__(self, snr_dict, name=None, parent=None, supported=None, **kwargs):
        super(MultipleLineProcessor, self).__init__(name=name, supported=supported)
        self.snr_dict = snr_dict

@@ -17,7 +17,8 @@ def has_external_subtitle(part_id, stored_subs, language):


 def set_existing_languages(video, video_info, external_subtitles=False, embedded_subtitles=False, known_embedded=None,
-                           stored_subs=None, languages=None, only_one=False, known_metadata_subs=None):
+                           stored_subs=None, languages=None, only_one=False, known_metadata_subs=None,
+                           match_strictness="strict"):
    logger.debug(u"Determining existing subtitles for %s", video.name)

    external_langs_found = set()
@@ -27,7 +28,8 @@ def set_existing_languages(video, video_info, external_subtitles=False, embedded
        external_langs_found = known_metadata_subs

    external_langs_found.update(set(search_external_subtitles(video.name, languages=languages,
-                                                              only_one=only_one).values()))
+                                                              only_one=only_one,
+                                                              match_strictness=match_strictness).values()))

    # found external subtitles should be considered?
    if external_subtitles:
@@ -52,10 +54,10 @@ def set_existing_languages(video, video_info, external_subtitles=False, embedded
            video.subtitle_languages.add(language)


-def parse_video(fn, hints, skip_hashing=False, dry_run=False, providers=None):
+def parse_video(fn, hints, skip_hashing=False, dry_run=False, providers=None, hash_from=None):
    logger.debug("Parsing video: %s, hints: %s", os.path.basename(fn), hints)
    return scan_video(fn, hints=hints, dont_use_actual_file=dry_run, providers=providers,
-                      skip_hashing=skip_hashing)
+                      skip_hashing=skip_hashing, hash_from=hash_from)


 def refine_video(video, no_refining=False, refiner_settings=None):
@@ -19,6 +19,8 @@ I can't keep running. L can't!
 <b>i don't know. Some kind of wrong "1 00" number---
 of signal, drawing the Tardis off.... course.</b>
 # I'm singing in the rain
+www.website.com
+www.nowebsite.badlol

 4
 00:00:16,099 --> 00:00:17,224
@@ -0,0 +1,14 @@
+# -*- coding: utf-8 -*-
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+from __future__ import unicode_literals
+from .utils import get_fld, get_tld, get_tld_names, is_tld, parse_tld, Result, update_tld_names
+__title__ = u'tld'
+__version__ = u'0.11.10'
+__author__ = u'Artur Barseghyan'
+__copyright__ = u'2013-2019 Artur Barseghyan'
+__license__ = u'MPL-1.1 OR GPL-2.0-only OR LGPL-2.1-or-later'
+__all__ = (u'get_fld', u'get_tld', u'get_tld_names', u'is_tld',
+           u'parse_tld', u'Result', u'update_tld_names')
@@ -0,0 +1,57 @@
+# -*- coding: utf-8 -*-
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+from __future__ import unicode_literals
+from six import with_metaclass as _py_backwards_six_withmetaclass
+from codecs import open as codecs_open
+try:
+    from urllib.request import urlopen
+except ImportError:
+    from six.moves.urllib.request import urlopen as urlopen
+from .exceptions import TldIOError, TldImproperlyConfigured
+from .helpers import project_dir
+from .registry import Registry
+__author__ = u'Artur Barseghyan'
+__copyright__ = u'2013-2019 Artur Barseghyan'
+__license__ = u'MPL-1.1 OR GPL-2.0-only OR LGPL-2.1-or-later'
+__all__ = (u'BaseTLDSourceParser',)
+
+
+class BaseTLDSourceParser(
+        _py_backwards_six_withmetaclass(Registry, *[object])):
+    u'Base TLD source parser.'
+    uid = None
+    source_url = None
+    local_path = None
+
+    @classmethod
+    def validate(cls):
+        u'Constructor.'
+        if (not cls.uid):
+            raise TldImproperlyConfigured(
+                u'The `uid` property of the TLD source parser shall be defined.')
+
+    @classmethod
+    def get_tld_names(cls, fail_silently=False, retry_count=0):
+        u'Get tld names.\n\n        :param fail_silently:\n        :param retry_count:\n        :return:\n        '
+        cls.validate()
+        raise NotImplementedError(
+            u'Your TLD source parser shall implement `get_tld_names` method.')
+
+    @classmethod
+    def update_tld_names(cls, fail_silently=False):
+        u'Update the local copy of the TLD file.\n\n        :param fail_silently:\n        :return:\n        '
+        try:
+            remote_file = urlopen(cls.source_url)
+            local_file = codecs_open(project_dir(
+                cls.local_path), u'wb', encoding='utf8')
+            local_file.write(remote_file.read().decode(u'utf8'))
+            local_file.close()
+            remote_file.close()
+        except Exception as err:
+            if fail_silently:
+                return False
+            raise TldIOError(err)
+        return True
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
pannal	3a09d9ffee	Update README.md	2024-01-27 02:28:07 +01:00
pannal	62439ef49f	fix advanced_settings.json.template	2024-01-27 01:45:43 +01:00
pannal	4458795293	Update README.md	2024-01-27 00:11:07 +01:00
pannal	e648e945c9	Update README.md	2024-01-27 00:09:06 +01:00
pannal	3d95c6e420	Update README.md	2024-01-27 00:08:25 +01:00
pannal	20c3cd2340	2.6.5.3280	2024-01-27 00:06:17 +01:00
pannal	483e0cf96e	extremely cheap and barely tested replacement of OpenSubtitles.org with OpenSubtitles.com (needs your own API key)	2024-01-27 00:00:00 +01:00
pannal	4ced7d8c8f	remove old TLD file due to licensing issues	2023-07-09 15:08:04 +02:00
pannal	f888cadb8f	release 2.6.5.3277	2023-07-09 04:49:32 +02:00
pannal	ccf264cffb	core: fix enabled library/agents detection (Plex removed certain features)	2023-07-09 04:42:24 +02:00
panni	19a66dcf1c	core: fix adding guessit-supplied titles as alternative titles (were overridden by plex before)	2023-07-09 04:42:24 +02:00
panni	67b20b357d	back to dev	2023-07-09 04:42:24 +02:00
pannal	d33bc1b148	Merge pull request #788 from sugarman402/patch-1 Change supersubtitles server url	2022-11-28 04:23:11 +01:00
Tamás Németh	8de63e92e5	Change supersupbtitles server url Supersubtitles' url changed from feliratok.info to feliratok.eu	2022-06-05 18:22:01 +02:00
pannal	2788b0e0b2	Update README.md	2021-05-07 02:36:16 +02:00
panni	dabef2240d	2.6.5.3268	2021-05-07 02:29:32 +02:00
panni	432255d088	update dev version to 2.6.5.3268	2021-05-07 02:28:55 +02:00
panni	69096d89b2	2.6.5.3268	2021-05-07 02:28:24 +02:00
panni	71fc91fe35	Merge branch 'master' into develop-2.6	2021-05-07 02:15:53 +02:00
panni	0a4014c4bf	core: delay subsequent refreshes for the same time core: advanced: add "refresh_after_called" (default: 5 seconds)	2021-05-07 02:07:43 +02:00
pannal	1b83203a64	Update README.md	2021-05-07 02:06:12 +02:00
panni	fc0ea00214	core: delay initial item refresh after refresh call (default: 5 seconds)	2021-05-07 02:04:48 +02:00
panni	60ff16c174	menu: allow extraction of embedded subtitles for whole tv shows	2021-03-22 18:46:37 +01:00
panni	a352be89f5	clarify subtitles.ignore_for_audio	2021-03-09 04:03:25 +01:00
pannal	06e8c0e1ff	Update README.md	2021-03-09 03:57:11 +01:00
panni	a0a29618bf	update readme	2021-03-09 03:56:38 +01:00
panni	2e5f0551a1	2.6.5.3254	2021-03-09 03:50:28 +01:00
panni	6bd0f26296	fix #756 - add option to not download subtitles for certain audio languages existing; and/or no audio stream	2021-03-09 03:47:41 +01:00
panni	58aa0da32f	fix #761	2021-03-08 03:48:07 +01:00
pannal	3f47c1bcb8	Update README.md	2021-01-24 04:54:20 +01:00
pannal	2625ca752f	Merge pull request #754 from glensc/patch-1 Install: need to enable for the libraries	2021-01-24 04:52:56 +01:00
pannal	56d16616ff	Merge pull request #751 from PlexIL/master Hebrew subtitle support for Ktuvit & WizdomSubs	2021-01-24 04:49:44 +01:00
Elan Ruusamäe	734dbf663e	Install: need to enable for the libraries	2020-12-10 11:39:01 +02:00
PlexIL	7dacd80660	Hebrew subtitle support for Ktuvit & WizdomSubs	2020-12-05 19:59:54 +02:00
panni	c77489a5be	the heat	2020-08-12 17:02:54 +02:00
panni	25f204b330	whoops, missed dev flag	2020-08-12 17:01:06 +02:00
panni	89dded387d	core: properly handle ReadTimeout	2020-08-12 16:09:53 +02:00
panni	a9b677f0ce	core: catch more exceptions	2020-08-08 03:46:57 +02:00
panni	6b918be799	release 2.6.5.3247	2020-07-26 03:12:35 +02:00
panni	f259682391	core: findBetterSubtitles: increase minimum score for better subtitles for movies with extracted embedded subs from 82 to 112	2020-07-26 03:08:46 +02:00
panni	8a059c988e	core: findBetterSubtitles: increase minimum score for better subtitles for movies with extracted embedded subs from 82 to 112	2020-07-26 03:07:04 +02:00
panni	8512940ccf	core: fix for tv.plex.agents.movie not populating its media types	2020-07-26 02:54:28 +02:00
panni	de2b11f69a	back to dev	2020-07-25 05:56:40 +02:00
panni	df1fba83a8	release 2.6.5.3241	2020-07-25 05:53:12 +02:00
panni	d98ae74c8a	2.6.5.3241	2020-07-25 05:51:09 +02:00
panni	6484646122	advanced_settings: refiners: drone: add custom pem_file support; fixes #735	2020-07-25 05:22:57 +02:00
panni	52bac14a2e	core: remove old remnants; core: update certifi to 2020.6.20	2020-07-25 05:08:51 +02:00
panni	0be589bc5f	add support for new Plex Movie agent	2020-07-25 05:01:12 +02:00
panni	e083e133eb	bump dev	2020-04-14 05:06:45 +02:00
panni	c787e671c3	providers: addic7ed: enforce limits once they're hit, to avoid unnecessary search queries #723	2020-04-14 05:06:02 +02:00
panni	31ff93c3f1	fix logging; set DownloadLimitPerDayExceeded timeout to 4 hours (was one day); #723	2020-04-13 05:57:41 +02:00
panni	289f174e2b	providers: addic7ed: properly implement limits #723	2020-04-13 05:54:04 +02:00
panni	ea03f3fc4d	providers: addic7ed: properly compare last_dl, add last_reset tracking info to log #723	2020-04-12 22:47:21 +02:00
panni	740fc93c13	bump dev	2020-04-12 06:54:36 +02:00
panni	e94bd3fcb9	providers: addic7ed: limit downloads per day; add vip setting	2020-04-12 06:52:45 +02:00
panni	dba469750b	submod: HI: remove more music tags core: update pysubs2 providers: opensubtitles: actually use sessions (they're broken) for checking for token state	2020-04-11 05:06:12 +02:00
panni	c7fe6076cb	bump version	2020-03-08 16:32:40 +01:00
panni	356f578014	remove py3 compat breaking unnecessary change	2020-03-08 16:29:49 +01:00
panni	b151ed4c55	core: mods: CM_punctuation_space2: detect AND don't try changing domain/url/host when fixing punctuation add python-tld; functools_lru_cache	2020-03-08 05:37:32 +01:00
panni	9455e3b52b	skip drawing tags for SRT	2020-03-08 05:18:20 +01:00
panni	60e2656541	back to dev	2020-02-16 06:06:23 +01:00
panni	fcb1a8a6a7	release 2.6.5.3223	2020-02-16 06:05:04 +01:00
panni	9e5829151d	release 2.6.5.3223	2020-02-16 06:03:21 +01:00
panni	1f0a713f9b	core: scoring: reorder subtitles based on second non-hash-score if main hash score is the same; morpheus65535/bazarr#821	2020-02-16 05:44:59 +01:00
panni	ff49dd4512	providers: bsplayer: verify hash; clean up	2020-02-16 05:08:50 +01:00
panni	3cf83b5bf7	back to dev	2020-02-15 03:17:36 +01:00
panni	c51ce55d32	update maintenance state in readme	2020-02-15 03:17:00 +01:00
panni	ee9268957d	release 2.6.5.3217	2020-02-15 03:16:32 +01:00
panni	5d1858d5da	2.6.5.3217	2020-02-15 03:15:17 +01:00
panni	d59abea1f5	changelog	2020-02-15 03:02:05 +01:00
panni	e6b79334d8	core: scheduler: add option to specify subtitle storage maintenance	2020-02-15 03:00:06 +01:00
panni	88d2a44f08	core: reuse OS hash core: refiners: tvdb: don't fail on bad firstAired date	2020-02-15 02:27:11 +01:00
panni	78e47d3cd5	providers: screwzira: move config; providers: add BSPlayer	2020-02-15 02:10:32 +01:00
panni	6b6af347da	providers: screwzira: disable when foreign only is enabled	2020-02-15 02:08:16 +01:00
panni	dccee96cf1	Merge branch 'master' into develop-2.6	2020-02-15 02:06:59 +01:00
pannal	92b24de7cd	Merge pull request #698 from dornizar/master ScrewZira Hebrew subtitle support	2020-02-15 02:04:53 +01:00
panni	1225a4887c	bump dev	2020-01-19 05:39:41 +01:00
panni	2a82857570	#711 plex security blerp	2020-01-19 05:38:59 +01:00
panni	831bec3630	bump dev	2020-01-19 05:35:10 +01:00
panni	55cbf2478a	morpheus65535/bazarr#656 further generalize formats; skip release group match if format match failed	2020-01-19 05:02:34 +01:00
panni	20a850b9e9	#711 use correct Log function	2020-01-18 23:26:43 +01:00
panni	11c649a7af	#711 don't fail on inexistant stream IDs when detecting extra stream info (mediainfo)	2020-01-18 23:21:01 +01:00
panni	c1e5a2077b	Merge branch 'bzr_#703' into develop-2.6	2019-12-25 01:39:13 +01:00
panni	dc96f626dd	bazarr #703 : py3 compat backport	2019-12-25 01:32:32 +01:00
panni	46f48023f4	bazarr #660 : lowercase all BOM encodings	2019-12-24 00:44:53 +01:00
panni	e3d83f6dc2	bazarr #660 : don't double-check encodings when BOM encoding detected	2019-12-24 00:42:50 +01:00
panni	fc484e569f	bazarr #660 : try detecting BOMs before doing any other encoding guessing	2019-12-24 00:38:50 +01:00
panni	a92f3e2480	bazarr #703 : use proper language code detection instead of a wild guess; should fix bad existing subtitle detection	2019-12-24 00:10:26 +01:00
dor	064c447528	changed imports to subliminal_patch	2019-11-09 21:32:02 +02:00
panni	64c1bcd9e6	bump dev	2019-11-09 04:46:56 +01:00
panni	0cca4a2ebe	core: UnRAR: set binary to executable, even if not checked out from git; might fix #693	2019-11-09 04:28:14 +01:00
panni	c4c26a76f1	core: clarify Detecting Streams	2019-11-09 03:57:38 +01:00
panni	1a638431d7	core: when extracting embedded subtitles from task, execute threads synchronously	2019-11-09 03:19:01 +01:00
panni	5d5fa21630	refactor all extraction into support.extract; extract also on SearchAllRecentlyAddedMissing	2019-11-09 02:59:36 +01:00
dor	e78ace4664	ScrewZira Hebrew subtitle support www.screwzira.com	2019-11-07 18:54:52 +02:00
panni	37596c412c	Merge branch 'master' into develop-2.6	2019-10-26 04:57:34 +02:00
panni	d200021243	Merge remote-tracking branch 'origin/master'	2019-10-26 04:44:55 +02:00
panni	1a999e202f	release 2.6.5.3183	2019-10-26 04:44:24 +02:00
panni	fd748b29e9	fix changelog	2019-10-26 04:41:26 +02:00
panni	775d1e3cf1	back to dev	2019-10-26 04:39:15 +02:00
panni	c6a1df9a79	release 2.6.5.3152	2019-10-26 04:38:34 +02:00
panni	77861a4c6d	release 2.6.5.3152	2019-10-26 04:38:03 +02:00
panni	bf1e1c3139	bump dev	2019-10-26 04:17:25 +02:00
panni	a564a1d808	providers: addic7ed: wait a short while between retries and after successfully logging in	2019-10-26 04:14:38 +02:00
panni	22ac935f9b	core: scanning: add additional INFO logging for undetected languages	2019-10-23 14:12:37 +02:00
panni	02e2bcb417	bump dev	2019-10-21 17:21:01 +02:00
panni	3445259cde	providers: addic7ed: fix detection of completed subtitle (#686 )	2019-10-21 17:15:02 +02:00
panni	c20c32c17d	providers: addic7ed: refresh show IDs if stored ones were empty	2019-10-21 17:10:34 +02:00
panni	3fec766890	core: fix #688	2019-10-21 16:03:40 +02:00
panni	f208a24213	providers: addic7ed: fix bungled show_ids reference; #686	2019-10-20 16:53:13 +02:00
panni	9e9dfb3f4d	providers: addic7ed: fix Mayans M.C.; add logging; fix AuthenticationError	2019-10-20 05:50:33 +02:00
panni	83eecf09ed	bump dev	2019-10-20 05:20:13 +02:00
panni	1aebe8d0dd	providers: addic7ed: fix getting show ids (failing on foreign characters) providers: addic7ed: don't run anything if no credentials given providers: addic7ed: actually try three times to log in providers: addic7ed: store last show ids fetch; when show id not found, re-try once per day	2019-10-20 05:19:05 +02:00
panni	bb64e482df	providers: addic7ed: fix getting show list (failing on foreign characters) providers: addic7ed: don't run anything if no credentials given providers: addic7ed: actually try three times to log in	2019-10-20 05:04:02 +02:00
panni	1841a72ca7	bump dev	2019-10-20 04:00:23 +02:00
panni	997d4aa1cf	core: don't process any further if stream info is missing	2019-10-20 03:59:39 +02:00
panni	d517e86333	core: don't fall back to default providers if none enabled	2019-10-20 03:45:16 +02:00
panni	8bcfc712fb	updated dev	2019-10-19 23:21:03 +02:00
panni	c0cf2fd78e	providers: argenteam: bazarr-backport: use new url; fixes	2019-10-19 23:19:19 +02:00
panni	0a7de0e9b6	core: bazarr-backport: generic 10 minute throttling if uncaught exception occurs; also for downloads	2019-10-19 23:17:09 +02:00
panni	1e2a127dac	core: bazarr-backport: generic 10 minute throttling if uncaught exception occurs	2019-10-19 23:13:57 +02:00
panni	5b8cd215e4	core: backport removal of existing subtitle file from bazarr, to support MergerFS	2019-10-19 23:02:57 +02:00
panni	7583edf3fe	providers: addic7ed: move re to correct place; fix show match; #686	2019-10-19 15:36:46 +02:00
panni	2f219a1a81	providers: addic7ed: fix show match	2019-10-19 15:35:19 +02:00
panni	9127c38297	bump dev	2019-10-19 07:15:45 +02:00
panni	0c379f8b9f	providers: addic7ed: add timeout on authentication error	2019-10-19 07:15:07 +02:00
panni	d2b617bdf4	addicted; fix #686	2019-10-19 06:55:32 +02:00
panni	6d6f6d9356	forgot to fix #681	2019-10-19 06:54:38 +02:00
panni	8ffb20ebe3	try fixing #681	2019-10-07 15:13:00 +02:00
panni	eed7b9da0c	core: support using mediainfo for retrieving MP4 MOV_TEXT subtitle stream titles (PMS bug)	2019-10-05 16:47:45 +02:00
panni	802381b2bc	back to dev	2019-10-05 04:25:48 +02:00
panni	f265c861d2	back to dev	2019-10-05 04:24:20 +02:00
panni	1dc7b4b5e4	back to dev	2019-10-05 04:15:11 +02:00
panni	c48aa2b255	release 2.6.5.3152	2019-10-05 04:14:28 +02:00
panni	66859802f9	update readme	2019-10-05 04:13:50 +02:00
panni	433c8e987b	back from dev	2019-10-05 04:07:52 +02:00
panni	aa477ca48c	Merge branch 'develop-2.6'	2019-10-05 04:07:33 +02:00
pannal	65b502afa4	bump dev	2019-09-21 16:25:33 +02:00
pannal	06c0b44589	fix Dicked.get	2019-09-21 16:24:21 +02:00
pannal	d651f2cbb7	bump dev	2019-09-20 18:24:16 +02:00
pannal	8b5be8ea4b	#676 improve	2019-09-20 18:14:14 +02:00
pannal	f4e82c560d	core: fix default values of opensubtitles-skip-wrong-fps, use_https; fix #676	2019-09-20 18:10:16 +02:00
panni	c23b3e93a6	bump dev	2019-08-31 14:33:04 +02:00
panni	de447d2d0b	core: fix for determining whether to search under certain circumstances; fixes #666	2019-08-31 14:32:42 +02:00
panni	95b1272018	explicit language=None check	2019-08-25 06:12:48 +02:00
pannal	11d111da7c	Update README.md	2019-08-24 04:51:26 +02:00
pannal	638dec0f04	Update README.md	2019-08-24 04:45:45 +02:00
panni	a0ab6e406a	providers: titlovi: raise ConfigurationError if credentials aren't given	2019-08-22 15:46:38 +02:00
panni	23242c0f52	bump dev	2019-08-22 15:34:45 +02:00
pannal	48bf70e825	Merge pull request #660 from viking1304/develop-2.6 New implementation of Titlovi using API	2019-08-22 15:32:43 +02:00
panni	ada0b96872	#664 fix missing language processing of multiple videos refreshed at once	2019-08-22 14:58:28 +02:00
panni	0e4917bba9	#661 further improvements	2019-08-13 18:06:06 +02:00
panni	8169d31e86	#661 fix bad condition	2019-08-13 13:01:05 +02:00
panni	75b83aa163	#661 fix match strictness when determining preexisting external subtitles	2019-08-13 12:56:08 +02:00
viking1304	d2022de970	Removed titlovi from AntiCaptcha lablel	2019-08-12 21:02:51 +02:00
viking1304	8db1cdacb4	Revert "Disable provider Titlovi if user and password are not set" This reverts commit `527d171a6a`.	2019-08-09 19:01:08 +02:00
viking1304	527d171a6a	Disable provider Titlovi if user and password are not set	2019-08-09 18:54:35 +02:00
viking1304	20620cfa7e	New implentation of Titlovi using API	2019-08-09 18:22:42 +02:00
panni	4d03ca078d	back to dev	2019-08-09 03:38:54 +02:00
panni	775e2cca47	Merge remote-tracking branch 'origin/master'	2019-08-09 03:13:46 +02:00
panni	7cb2486d3e	release 2.6.5.3124	2019-08-09 03:13:34 +02:00
panni	02a3ecc9fe	prepare for release	2019-08-09 03:11:55 +02:00
panni	54435398af	bump dev	2019-08-08 15:09:04 +02:00
panni	ffc42883de	core: extract embedded/menu: fix detection of unknown streams; don't use unknown streams if a known language was previously found	2019-08-08 14:30:29 +02:00
panni	0cf0371a43	core: language: use replacement map from bazarr	2019-08-06 18:04:34 +02:00
panni	f5156bcea7	providers: titlovi: fix matching	2019-07-27 03:10:08 +02:00
panni	efdf3b2c9d	core: http: fallback to default DNS when normal resolving fails; fixes #657	2019-07-27 02:57:35 +02:00
panni	c3d3163392	providers: subscene: fix unknown language code error when "empty" result is returned	2019-07-05 13:56:21 +02:00
panni	c91d5ca483	providers: subscene: add support for pt-BR (based on https://github.com/Diaoul/subliminal/pull/740/commits/b22cf08a5d0e7082b0dc6c0de8cc764f01233625 )	2019-07-05 13:55:52 +02:00
panni	5f0982970d	docker/bazarr compat	2019-07-05 03:22:06 +02:00
panni	ee05da70f4	providers: subscene: explicitly set account filters for languages	2019-06-23 15:26:41 +02:00
panni	04c283c48d	providers: subscene: limit alternative searches to 3; set throttle to 8	2019-06-23 04:24:18 +02:00
panni	836945c95c	providers: subscene: move login/cookies to initialization sequence	2019-06-22 16:45:31 +02:00
panni	bd4c180c07	submod: generic: en: fix ";='s	2019-06-22 04:29:03 +02:00
pannal	e1f5290365	Update README.md	2019-06-22 04:07:30 +02:00
panni	eefffcfb1b	back to dev	2019-06-22 04:06:10 +02:00
panni	9e088a5e9d	release 2.6.5.3109	2019-06-22 04:05:37 +02:00
panni	317c02bf06	prepare next release	2019-06-22 04:04:22 +02:00
panni	22724c269c	core: bazarr compat	2019-06-21 15:04:00 +02:00
panni	2a48782b6b	core: bazarr compat	2019-06-21 15:00:35 +02:00
panni	e7c3039fde	providers: subscene: detect login availability; fallback to non year results if none found with year	2019-06-21 04:49:54 +02:00
panni	2afba02b59	bump dev	2019-06-21 04:17:15 +02:00
panni	94928c2930	providers: add Napisy24 (polish)	2019-06-21 04:16:46 +02:00
panni	2c25191291	providers: subscene: support logging in	2019-06-20 16:11:21 +02:00
panni	ba2f3f2172	back to dev	2019-06-06 02:18:34 +02:00
panni	aa5cba9347	release 2.6.5.3099	2019-06-06 02:15:23 +02:00
panni	5f40452f57	release 2.6.5.3099	2019-06-06 02:14:50 +02:00
panni	2dd9b1723b	core: allow system DNS again by putting "system" as the DNS	2019-06-06 02:13:20 +02:00
panni	ee54839f28	back to dev	2019-06-06 01:45:16 +02:00
panni	c2f054a25e	release 2.6.5.3096	2019-06-06 01:41:57 +02:00
panni	f095d5c99c	providers: subscene: remove obsolete exception handling	2019-06-06 01:38:28 +02:00
panni	ab93f9809a	providers: subscene: dumb down endpoint detection; adapt	2019-06-06 01:31:27 +02:00
panni	bbb9a62357	back to dev	2019-05-30 04:23:49 +02:00
panni	82ffed699f	release 2.6.5.3092	2019-05-30 04:23:07 +02:00
panni	4751ea8396	bump dev	2019-05-30 04:21:50 +02:00
panni	c15d8fbe58	bump dev	2019-05-30 04:14:38 +02:00
panni	b379468b47	properly re-raise	2019-05-30 04:13:00 +02:00
panni	0deb3eae21	providers: subscene: react to new endpoint; store and use new endpoint	2019-05-30 04:11:12 +02:00
panni	0c1042ec5c	bump dev	2019-05-27 12:39:04 +02:00
panni	05d0de5120	core: providers: argenteam: backport fixes from bazarr	2019-05-27 12:34:37 +02:00
panni	2fa217d5d9	core: subtitle: encoding: re-revert 1ed4f11	2019-05-27 12:27:18 +02:00
panni	a65b5a5d82	core: missed forced utf-8 instance	2019-05-24 18:10:09 +02:00
panni	7bb42e95d8	core: add env var SZ_KEEP_ENCODING to keep encoding of subtitles	2019-05-24 18:06:00 +02:00
panni	db536502a1	bump dev	2019-05-19 06:06:43 +02:00
panni	47c8f1a2e6	Merge branch 'submod_opt' into develop-2.6	2019-05-19 06:04:17 +02:00
panni	30a0f11515	providers: subscene: don't calculate video fn for now	2019-05-19 04:27:20 +02:00
panni	9bf5123a00	providers: subscene: don't search for season packs (broken); fix endpoint error handling	2019-05-18 15:03:53 +02:00
panni	f337b53ae3	submod: HI: remove music submod: common: be less aggressive about music symbols submod: HI: be less aggressive about brackets submod: HI: be less aggressive about MAN	2019-05-18 06:23:04 +02:00
panni	aea6050d71	subtitle: try decoding with utf-16 by default as well	2019-05-17 23:45:06 +02:00
panni	13d5e0761e	providers: subscene: fix endpoint once again	2019-05-13 16:14:26 +02:00
panni	ce28d0284c	back from dev	2019-05-12 06:17:08 +02:00
panni	1a0bb9c3e4	release 2.6.5.3074	2019-05-12 06:05:16 +02:00
panni	d0c71b4b67	bump dev	2019-05-12 05:12:58 +02:00
panni	b3f062956d	core: re-fix ass/ssa tags in srt in pysubs2 0.2.3	2019-05-12 05:12:34 +02:00
panni	1a853a780c	core: update pysubs2 to 0.2.3	2019-05-12 05:01:38 +02:00
panni	5c47ddeb2d	core: update chinese encodings; #646	2019-05-12 04:49:30 +02:00
panni	b51deb5d01	core: subliminal: don't replace \r with \n by default; fixes utf-16 character transformation issues; fixes #646	2019-05-12 04:48:23 +02:00
panni	cbf5ea69be	core: cf: update cloudscraper to 1.1.9; fix keyerror	2019-05-08 15:57:33 +02:00
panni	e139ffefe6	bump dev	2019-05-08 04:18:25 +02:00
panni	dc0a8deb40	core: cf: testing providers: subscene: testing	2019-05-08 04:14:04 +02:00
panni	97e93cd10a	core: cf: update js2py; update cloudscraper to 1.1.5;	2019-05-08 01:31:21 +02:00
panni	03c934cf21	back to dev	2019-05-01 15:39:23 +02:00
				`@@ -0,0 +1 @@`
				`__path__ = __import__('pkgutil').extend_path(__path__, __name__)`