Release 1.4.27.974

(cherry picked from commit 8b91093)

CHANGELOG.md

@@ -1,3 +1,166 @@
+1.4.27.957
+- core: correctly fall back to the next best subtitle if the current one couldn't be downloaded; hopefully fixes #231
+- core: add "Scan: which external subtitles should be picked up?"-setting
+- core: add optional on_playing activities. refresh currently playing movie, refresh next episode in season, both or none; fixes #259 #33
+- core: skip to next best subtitle if findbettersubtitles failed
+- core: add setting to treat undefined-language embedded subtitle as configured language1 #239
+- core: fix handling of inexistant addic7ed show id
+- core: fix regression issue breaking relative custom subtitle folder handling
+- core: fix loading of stored subtitle info data of now-non-existant items
+- core: re-add separate global subtitle folder handling
+- menu: remove obsolete actions from the advanced menu
+
+
+1.4.23.920
+- core: handle undecodable paths better #255
+- core: don't fail on unrecoverable data #257
+- core: increase default scores from 110 (series) and 23 (movies) to 116 and 33
+- core: fix global subtitle folder handling #234
+- core: better invoking of configured executable after subtitle addition #247
+
+
+1.4.22.908
+- core: hotfix for more robust migrations
+
+
+1.4.22.898
+- core: migrate history and subtitle storage to a better implementation, making it far more stable. subtitle storage now also stores the downloaded subtitle data for future usage, so it will be possible to switch between them
+- core/menu: manual subtitle download and the FindBetterSubtitles-task now also work with metadata storage (hi @ shield users)
+- core: optimize FindBetterSubtitles-task
+
+
+1.4.19.882
+- core: fix tasks for new users
+- core: double check pin correctness/existance when pin is enabled
+
+
+1.4.19.878
+- core/menu: fix a task's last runtime display
+- core: task optimizations
+- core: fix leftover subtitles cleanup handling in case of a custom subtitle folder #234
+- core: run the scheduler even if permissions for libraries are wrong ("fixes" #236)
+- core: store subtitle history data in a different data format; reduce used storage size drastically (#233)
+
+
+1.4.19.866
+- core: fix wrong usage of LogKit
+
+
+1.4.19.857
+
+- core: add option to enable/disable channel and/or agent modes (fixes #220)
+- core: skip inexistent internal streams when scanning for internal subtitles (fixes #222)
+- core: fix filename encoding (fixes #223)
+- core: storage optimizations
+- menu: add pin-based channel menu locking (the whole channel or only the advanced menu)
+
+
+1.4.17.836
+- core: support for any media file that PMS supports (internal subtitles on mp4 for example)
+- core: fix broken ignore folders containing "subzero.ignore/.subzero.ignore/.nosz"
+- core: fix duplicate subtitles (lowercase/default case)
+- core: fix broken tasks queue due to oversight
+
+
+1.4.16.822
+- menu: add per-section recently added menu
+- menu: fix accidentally double-triggering a just triggered force-refresh
+- core: reorder settings in a more logical, grouped way
+- core: add simple automatic filesystem/external leftover subtitle cleaning (#133, #152)
+- core: fix force-refresh for big seasons/series
+- core: add setting to look for forced/foreign-only subtitles only (only works for opensubtitles and podnapisi)
+- core: fix custom subtitle folder was being ignored (#211)
+- core: only trust PMS for its movie name, not the series title (fixes #210)
+- core: full support (in filesystem/external mode) for forced/default/normal subtitle tags
+- core: ignore "non-standard" external subtitle files when scanning by default (everything but .srt, .ass, .ssa, fixes #192)
+- core: lower default max_recent_items_per_library to 500
+- core: skip forced/foreign-only subtitles if not specifically wanted
+- core: modify the task queue, hopefully helping #206
+- core: update anonymous usage collection
+
+
+1.4.11.781
+- core: cleanup, logging
+- core/menu: fix addic7ed display in manual subtitle list
+- core: use HTTP for OpenSubtitles instead of HTTPS because of current certificate errors
+- core: find better subtitles should now run smoothly even with replaced files (newer parts)
+
+
+1.4.10.769
+- core: hotfix for legacy intent storage regression
+
+1.4.10.768
+- core: automatically find better subtitles (configurable)
+- menu: display how the subtitle was downloaded (auto, manual, auto-better), in history menu
+- menu/core: correctly handle subtitle list for multiple languages
+- core: lower minimum series score to list subtitles for to 66
+- core: better matching of garbage filenames; we trust Plex now for the series name/movie title fully
+- core: add setting to specifically set the file permissions (chmod)
+
+
+1.4.5.742
+- core: fix force-refresh in certain situations
+- menu: add history
+- menu: add manual subtitle selection
+- menu: run Items with missing subtitles in separate thread for big libraries
+- settings: add history list size option (default: 100)
+- settings: add new default scores (TV: 110); use input instead of dropdown
+- settings: increase default missing subtitles amount per library to 2000
+- core: generic rewrites and optimizations
+- core: better hash verification
+- core: add anonymous usage data (opt-out in settings)
+- core: fix pt-BR display (IETF) again
+- wiki: update (thanks @dane22!) - quick URL: http://v.ht/szwiki
+- wiki: add score explanation - quick URL: http://v.ht/szscores
+- core: add persian/farsi encoding support
+
+
+1.3.49.636
+- core/menu: fix force refreshing (again)
+- core/menu: fix redundant route calls
+
+
+1.3.49.630 (backported some changes of the develop-1.4 branch to 1.3)
+- core/menu: make addic7ed boost configurable; lower the default boost value massively (to 10)
+- core: fix force refreshing (hopefully)
+- core: add (thai) tis-620 subtitle encoding support
+- menu: lower letter based menu browsing from 200 to 80 items
+- core: support greek encodings (windows-1253, cp1253, cp737, iso8859_7, cp875, cp869, iso2022_jp_2, mac_greek); hopefully fixes badly saved greek subs
+- menu: add generic back-to-home button to the top of every container view
+- menu: warn the user when SZ isn't enabled for any sections/libraries
+- menu: always re-check permissions status and enabled sections when opening the main menu; no server restart necessary anymore
+
+
+1.3.46.606
+- core: hotfix for new users (who've never downloaded a subtitle with SZ before); fixes #169
+
+
+1.3.46.605
+
+- add wiki (thanks @ukdtom / @dane22)
+- core: remove necessity of Plex credentials; fixes #148
+- core: fix non-SRT subtitle support; fixes #138
+- core: generic source overhaul in preparation for release 1.4
+- core: better filesystem encoding detection; may fix #159
+- core: add encoding handling for windows-1250 and windows-1251 encoding (eastern europe); fixes #162
+- core: overhaul ignore handling; fixes #164
+- core: implement ignore by path setting; fixes #134
+- core: add setting for optional fallback to metadata storage, if filesystem storage failed; fixes #100
+- core: add setting for notifying an executable after a subtitle has been downloaded (see Wiki); fixes #65
+- core: only handle sections for which Sub-Zero is enabled (in PMS agent settings); fixes #167
+- menu: add series/season force-refresh
+- menu: show item thumbnail/art where applicable
+- menu: mitigate PlexWeb behaviour of calling our handlers twice; fixes #168
+
+
+1.3.33.522
+
+- core: fix library permission detection on windows; fixes #151
+- core: "Restrict to one language" now behaves like it should (one found subtitle of any language is treated as sufficient); fixes #149
+- core: add support for other subtitle formats such as ssa/ass/microdvd, convert to srt; fixes #138
+- core: hopefully more consistent force-refresh handling (intent); fixes #118
+
+
 1.3.31.513
 
 - core: add option to only download one language again (and skip the addition of .lang to the subtitle filename) (default: off); fixes #126 

Contents/Code/__init__.py

@@ -1,8 +1,8 @@
 # coding=utf-8
-import os
+import datetime
 import sys
+import traceback
 
-# just some slight modifications to support sum and iter again
 from subzero.sandbox import restore_builtins
 
 module = sys.modules['__main__']
@@ -14,30 +14,27 @@ for key, value in getattr(module, "__builtins__").iteritems():
         globals()[key] = value
 
 import logger
-import logging
-
-# temporarily add the console handler and set it to DEBUG to catch errors upon imports
-Core.log.addHandler(logger.console_handler)
-Core.log.setLevel(logging.DEBUG)
 
 sys.modules["logger"] = logger
 
 import subliminal
-import subliminal_patch
 import support
 
 import interface
 sys.modules["interface"] = interface
 
 from subzero.constants import OS_PLEX_USERAGENT, PERSONAL_MEDIA_IDENTIFIER
-from subzero import intent
 from interface.menu import *
-from support.plex_media import convert_media_to_parts, get_media_item_ids, scan_parts
-from support.subtitlehelpers import get_subtitles_from_metadata, force_utf8
-from support.helpers import notify_executable
-from support.storage import store_subtitle_info, whack_missing_parts
+from support.plex_media import media_to_videos, get_media_item_ids, scan_videos
+from support.subtitlehelpers import get_subtitles_from_metadata
+from support.storage import whack_missing_parts, save_subtitles
 from support.items import is_ignored
 from support.config import config
+from support.lib import get_intent
+from support.helpers import track_usage, get_title_for_video_metadata, get_identifier, cast_bool
+from support.history import get_history
+from support.data import dispatch_migrate
+from support.activities import activity
 
 
 def Start():
@@ -47,6 +44,24 @@ def Start():
     # configured cache to be in memory as per https://github.com/Diaoul/subliminal/issues/303
     subliminal.region.configure('dogpile.cache.memory')
 
+    # clear expired intents
+    intent = get_intent()
+    intent.cleanup()
+
+    # clear expired menu history items
+    now = datetime.datetime.now()
+    if "menu_history" in Dict:
+        for key, timeout in Dict["menu_history"].items():
+            if now > timeout:
+                del Dict["menu_history"][key]
+
+    # run migrations
+    if "subs" in Dict or "history" in Dict:
+        Thread.Create(dispatch_migrate)
+
+    # clear old task data
+    scheduler.clear_task_data()
+
     # init defaults; perhaps not the best idea to use ValidatePrefs here, but we'll see
     ValidatePrefs()
     Log.Debug(config.full_version)
@@ -55,17 +70,23 @@ def Start():
         Log.Error("Insufficient permissions on library folders:")
         for title, path in config.missing_permissions:
             Log.Error("Insufficient permissions on library %s, folder: %s" % (title, path))
-        return
 
+    # run task scheduler
     scheduler.run()
 
+    # bind activities
+    Thread.Create(activity.start)
 
-def init_subliminal_patches():
-    # configure custom subtitle destination folders for scanning pre-existing subs
-    dest_folder = config.subtitle_destination_folder
-    subliminal_patch.patch_video.CUSTOM_PATHS = [dest_folder] if dest_folder else []
-    subliminal_patch.patch_provider_pool.DOWNLOAD_TRIES = int(Prefs['subtitles.try_downloads'])
-    subliminal_patch.patch_providers.addic7ed.USE_BOOST = bool(Prefs['provider.addic7ed.boost'])
+    if "anon_id" not in Dict:
+        Dict["anon_id"] = get_identifier()
+
+    # track usage
+    if cast_bool(Prefs["track_usage"]):
+        if "first_use" not in Dict:
+            Dict["first_use"] = datetime.datetime.utcnow()
+            Dict.Save()
+            track_usage("General", "plugin", "first_start", config.version)
+        track_usage("General", "plugin", "start", config.version)
 
 
 def download_best_subtitles(video_part_map, min_score=0):
@@ -106,71 +127,6 @@ def download_best_subtitles(video_part_map, min_score=0):
     Log.Debug("All languages for all requested videos exist. Doing nothing.")
 
 
-def save_subtitles(videos, subtitles):
-    meta_fallback = False
-    save_successful = False
-    storage = "metadata"
-    if Prefs['subtitles.save.filesystem']:
-        storage = "filesystem"
-        try:
-            Log.Debug("Using filesystem as subtitle storage")
-            save_subtitles_to_file(subtitles)
-        except OSError:
-            if Prefs["subtitles.save.metadata_fallback"]:
-                meta_fallback = True
-            else:
-                raise
-        else:
-            save_successful = True
-
-    if not Prefs['subtitles.save.filesystem'] or meta_fallback:
-        if meta_fallback:
-            Log.Debug("Using metadata as subtitle storage, because filesystem storage failed")
-        else:
-            Log.Debug("Using metadata as subtitle storage")
-        save_successful = save_subtitles_to_metadata(videos, subtitles)
-
-    if save_successful and config.notify_executable:
-        notify_executable(config.notify_executable, videos, subtitles, storage)
-
-    store_subtitle_info(videos, subtitles, storage)
-
-
-def save_subtitles_to_file(subtitles):
-    fld_custom = Prefs["subtitles.save.subFolder.Custom"].strip() if bool(Prefs["subtitles.save.subFolder.Custom"]) else None
-
-    for video, video_subtitles in subtitles.items():
-        if not video_subtitles:
-            continue
-
-        fld = None
-        if fld_custom or Prefs["subtitles.save.subFolder"] != "current folder":
-            # specific subFolder requested, create it if it doesn't exist
-            fld_base = os.path.split(video.name)[0]
-            if fld_custom:
-                if fld_custom.startswith("/"):
-                    # absolute folder
-                    fld = fld_custom
-                else:
-                    fld = os.path.join(fld_base, fld_custom)
-            else:
-                fld = os.path.join(fld_base, Prefs["subtitles.save.subFolder"])
-            if not os.path.exists(fld):
-                os.makedirs(fld)
-        subliminal.api.save_subtitles(video, video_subtitles, directory=fld, single=Prefs['subtitles.only_one'],
-                                      encode_with=force_utf8 if Prefs['subtitles.enforce_encoding'] else None)
-    return True
-
-
-def save_subtitles_to_metadata(videos, subtitles):
-    for video, video_subtitles in subtitles.items():
-        mediaPart = videos[video]
-        for subtitle in video_subtitles:
-            content = force_utf8(subtitle.text) if Prefs['subtitles.enforce_encoding'] else subtitle.content
-            mediaPart.subtitles[Locale.Language.Match(subtitle.language.alpha2)][subtitle.page_link] = Proxy.Media(content, ext="srt")
-    return True
-
-
 def update_local_media(metadata, media, media_type="movies"):
     # Look for subtitles
     if media_type == "movies":
@@ -211,24 +167,27 @@ class SubZeroAgent(object):
         results.Append(MetadataSearchResult(id='null', score=100))
 
     def update(self, metadata, media, lang):
+        if not config.enable_agent:
+            Log.Debug("Skipping Sub-Zero agent(s)")
+            return
+
         Log.Debug("Sub-Zero %s, %s update called" % (config.version, self.agent_type))
+        intent = get_intent()
 
         if not media:
             Log.Error("Called with empty media, something is really wrong with your setup!")
             return
 
-        set_refresh_menu_state(media, media_type=self.agent_type)
-
         item_ids = []
         try:
-            init_subliminal_patches()
-            parts = convert_media_to_parts(media, kind=self.agent_type)
+            config.init_subliminal_patches()
+            videos = media_to_videos(media, kind=self.agent_type)
 
             # media ignored?
             use_any_parts = False
-            for part in parts:
-                if is_ignored(part["id"]):
-                    Log.Debug(u"Ignoring %s" % part)
+            for video in videos:
+                if is_ignored(video["id"]):
+                    Log.Debug(u"Ignoring %s" % video)
                     continue
                 use_any_parts = True
 
@@ -236,15 +195,37 @@ class SubZeroAgent(object):
                 Log.Debug(u"Nothing to do.")
                 return
 
-            use_score = Prefs[self.score_prefs_key]
-            scanned_parts = scan_parts(parts, kind=self.agent_type)
-            subtitles = download_best_subtitles(scanned_parts, min_score=int(use_score))
+            try:
+                use_score = int(Prefs[self.score_prefs_key].strip())
+            except ValueError:
+                Log.Error("Please only put numbers into the scores setting. Exiting")
+                return
+
+            set_refresh_menu_state(media, media_type=self.agent_type)
+
+            # find local media
+            update_local_media(metadata, media, media_type=self.agent_type)
+
+            # scanned_video_part_map = {subliminal.Video: plex_part, ...}
+            scanned_video_part_map = scan_videos(videos, kind=self.agent_type)
+
+            # downloaded_subtitles = {subliminal.Video: [subtitle, subtitle, ...]}
+            downloaded_subtitles = download_best_subtitles(scanned_video_part_map, min_score=use_score)
             item_ids = get_media_item_ids(media, kind=self.agent_type)
 
-            whack_missing_parts(scanned_parts)
+            whack_missing_parts(scanned_video_part_map)
 
-            if subtitles:
-                save_subtitles(scanned_parts, subtitles)
+            if downloaded_subtitles:
+                save_subtitles(scanned_video_part_map, downloaded_subtitles)
+                track_usage("Subtitle", "refreshed", "download", 1)
+
+                for video, video_subtitles in downloaded_subtitles.items():
+                    # store item(s) in history
+                    for subtitle in video_subtitles:
+                        item_title = get_title_for_video_metadata(video.plexapi_metadata, add_section_title=False)
+                        history = get_history()
+                        history.add(item_title, video.id, section_title=video.plexapi_metadata["section"],
+                                    subtitle=subtitle)
 
             update_local_media(metadata, media, media_type=self.agent_type)
 
@@ -258,17 +239,18 @@ class SubZeroAgent(object):
 
                 # resolve existing intent for that id
                 intent.resolve("force", item_id)
+
             Dict.Save()
 
 
 class SubZeroSubtitlesAgentMovies(SubZeroAgent, Agent.Movies):
     contributes_to = ['com.plexapp.agents.imdb', 'com.plexapp.agents.xbmcnfo', 'com.plexapp.agents.themoviedb', 'com.plexapp.agents.hama']
-    score_prefs_key = "subtitles.search.minimumMovieScore"
+    score_prefs_key = "subtitles.search.minimumMovieScore1"
     agent_type_verbose = "Movies"
 
 
 class SubZeroSubtitlesAgentTvShows(SubZeroAgent, Agent.TV_Shows):
     contributes_to = ['com.plexapp.agents.thetvdb', 'com.plexapp.agents.themoviedb',
                       'com.plexapp.agents.thetvdbdvdorder', 'com.plexapp.agents.xbmcnfotv', 'com.plexapp.agents.hama']
-    score_prefs_key = "subtitles.search.minimumTVScore"
+    score_prefs_key = "subtitles.search.minimumTVScore1"
     agent_type_verbose = "TV"

Contents/Code/interface/menu.py

@@ -1,19 +1,25 @@
 # coding=utf-8
 import logging
+
+import datetime
+
 import logger
+import os
 
 from menu_helpers import add_ignore_options, dig_tree, set_refresh_menu_state, \
-    should_display_ignore, enable_channel_wrapper, default_thumb, debounce
+    should_display_ignore, enable_channel_wrapper, default_thumb, debounce, ObjectContainer, SubFolderObjectContainer
 from subzero.constants import TITLE, ART, ICON, PREFIX, PLUGIN_IDENTIFIER, DEPENDENCY_MODULE_NAMES
+from subzero.history_storage import mode_map
 from support.background import scheduler
 from support.config import config
-from support.helpers import pad_title, timestamp
+from support.helpers import pad_title, timestamp, get_language, df, cast_bool
 from support.ignore import ignore_list
-from support.items import get_item, get_on_deck_items, refresh_item, get_all_items, get_recent_items, get_items_info, get_item_thumb
+from support.items import get_item, get_on_deck_items, refresh_item, get_all_items, get_items_info, \
+    get_item_thumb, get_item_kind_from_rating_key
 from support.lib import Plex
 from support.missing_subtitles import items_get_all_missing_subs
-from support.storage import reset_storage, log_storage, get_subtitle_info
-from support.plex_media import scan_parts
+from support.plex_media import get_plex_metadata, scan_videos
+from support.storage import reset_storage, log_storage, get_subtitle_storage
 
 # init GUI
 ObjectContainer.art = R(ART)
@@ -35,10 +41,24 @@ def fatality(randomize=None, force_title=None, header=None, message=None, only_r
     """
     subzero main menu
     """
-    title = force_title if force_title is not None else config.full_version
-    oc = ObjectContainer(title1=title, title2=None, header=unicode(header) if header else header, message=message, no_history=no_history,
+    title = config.full_version#force_title if force_title is not None else config.full_version
+    oc = ObjectContainer(title1=title, title2=title, header=unicode(header) if header else title, message=message, no_history=no_history,
                          replace_parent=replace_parent, no_cache=True)
 
+    # always re-check permissions
+    config.refresh_permissions_status()
+
+    # always re-check enabled sections
+    config.refresh_enabled_sections()
+
+    if config.lock_menu and not config.pin_correct:
+        oc.add(DirectoryObject(
+            key=Callback(PinMenu, randomize=timestamp()),
+            title=pad_title("Enter PIN"),
+            summary="The owner has restricted the access to this menu. Please enter the correct pin",
+        ))
+        return oc
+
     if not config.permissions_ok and config.missing_permissions:
         for title, path in config.missing_permissions:
             oc.add(DirectoryObject(
@@ -48,6 +68,14 @@ def fatality(randomize=None, force_title=None, header=None, message=None, only_r
             ))
         return oc
 
+    if not config.enabled_sections:
+        oc.add(DirectoryObject(
+            key=Callback(fatality, randomize=timestamp()),
+            title=pad_title("I'm not enabled!"),
+            summary="Please enable me for some of your libraries in your server settings; currently I do nothing",
+        ))
+        return oc
+
     if not only_refresh:
         if Dict["current_refresh_state"]:
             oc.add(DirectoryObject(
@@ -62,13 +90,20 @@ def fatality(randomize=None, force_title=None, header=None, message=None, only_r
         oc.add(DirectoryObject(
             key=Callback(OnDeckMenu),
             title="On Deck items",
-            summary="Shows the current on deck items and allows you to individually (force-) refresh their metadata/subtitles."
+            summary="Shows the current on deck items and allows you to individually (force-) refresh their metadata/"
+                    "subtitles."
         ))
         oc.add(DirectoryObject(
             key=Callback(RecentlyAddedMenu),
+            title="Recently Added items",
+            summary="Shows the recently added items per section."
+        ))
+        oc.add(DirectoryObject(
+            key=Callback(RecentMissingSubtitlesMenu, randomize=timestamp()),
             title="Items with missing subtitles",
             summary="Shows the items honoring the configured 'Item age to be considered recent'-setting (%s)"
-                    " and allowing you to individually (force-) refresh their metadata/subtitles. " % Prefs["scheduler.item_is_recent_age"]
+                    " and allowing you to individually (force-) refresh their metadata/subtitles. " %
+                    Prefs["scheduler.item_is_recent_age"]
         ))
         oc.add(DirectoryObject(
             key=Callback(SectionsMenu),
@@ -77,14 +112,14 @@ def fatality(randomize=None, force_title=None, header=None, message=None, only_r
                     "(force-) refresh the metadata/subtitles of individual items."
         ))
 
-        task_name = "searchAllRecentlyAddedMissing"
+        task_name = "SearchAllRecentlyAddedMissing"
         task = scheduler.task(task_name)
 
         if task.ready_for_display:
             task_state = "Running: %s/%s (%s%%)" % (len(task.items_done), len(task.items_searching), task.percentage)
         else:
-            task_state = "Last scheduler run: %s; Next scheduled run: %s; Last runtime: %s" % (scheduler.last_run(task_name) or "never",
-                                                                                               scheduler.next_run(task_name) or "never",
+            task_state = "Last scheduler run: %s; Next scheduled run: %s; Last runtime: %s" % (df(scheduler.last_run(task_name)) or "never",
+                                                                                               df(scheduler.next_run(task_name)) or "never",
                                                                                                str(task.last_run_time).split(".")[0])
 
         oc.add(DirectoryObject(
@@ -99,6 +134,12 @@ def fatality(randomize=None, force_title=None, header=None, message=None, only_r
             summary="Show the current ignore list (mainly used for the automatic tasks)"
         ))
 
+        oc.add(DirectoryObject(
+            key=Callback(HistoryMenu),
+            title="History",
+            summary="Show the last %i downloaded subtitles" % int(Prefs["history_size"])
+        ))
+
     oc.add(DirectoryObject(
         key=Callback(fatality, force_title=" ", randomize=timestamp()),
         title=pad_title("Refresh"),
@@ -108,6 +149,14 @@ def fatality(randomize=None, force_title=None, header=None, message=None, only_r
         )
     ))
 
+    # add re-lock after pin unlock
+    if config.pin:
+        oc.add(DirectoryObject(
+            key=Callback(ClearPin, randomize=timestamp()),
+            title=pad_title("Re-lock menu(s)"),
+            summary="Enabled the PIN again for menu(s)"
+        ))
+
     if not only_refresh:
         oc.add(DirectoryObject(
             key=Callback(AdvancedMenu),
@@ -118,6 +167,38 @@ def fatality(randomize=None, force_title=None, header=None, message=None, only_r
     return oc
 
 
+@route(PREFIX + '/pin')
+def PinMenu(pin="", randomize=None, success_go_to="channel"):
+    oc = ObjectContainer(title2="Enter PIN number %s" % (len(pin) + 1), no_cache=True, no_history=True,
+                         skip_pin_lock=True)
+
+    if pin == config.pin:
+        Dict["pin_correct_time"] = datetime.datetime.now()
+        config.locked = False
+        if success_go_to == "channel":
+            return fatality(force_title="PIN correct", header="PIN correct", no_history=True)
+        elif success_go_to == "advanced":
+            return AdvancedMenu(randomize=timestamp())
+
+    for i in range(10):
+        oc.add(DirectoryObject(
+            key=Callback(PinMenu, randomize=timestamp(), pin=pin + str(i),success_go_to=success_go_to),
+            title=pad_title(str(i)),
+        ))
+    oc.add(DirectoryObject(
+        key=Callback(PinMenu, randomize=timestamp(),success_go_to=success_go_to),
+        title=pad_title("Reset"),
+    ))
+    return oc
+
+
+@route(PREFIX + '/pin_lock')
+def ClearPin(randomize=None):
+    Dict["pin_correct_time"] = None
+    config.locked = True
+    return fatality(force_title="Menu locked", header=" ", no_history=True)
+
+
 @route(PREFIX + '/on_deck')
 def OnDeckMenu(message=None):
     """
@@ -128,28 +209,53 @@ def OnDeckMenu(message=None):
     return mergedItemsMenu(title="Items On Deck", base_title="Items On Deck", itemGetter=get_on_deck_items)
 
 
-@route(PREFIX + '/recent')
+@route(PREFIX + '/recently_added')
 def RecentlyAddedMenu(message=None):
     """
-    displays the recently added items with missing subtitles
+    displays the items recently added per section
     :param message:
     :return:
     """
-    return recentItemsMenu(title="Missing Subtitles", base_title="Missing Subtitles")
+    return SectionsMenu(base_title="Recently added", section_items_key="recently_added", ignore_options=False)
 
 
-def recentItemsMenu(title, base_title=None):
-    oc = ObjectContainer(title2=title, no_cache=True, no_history=True)
-    recent_items = get_recent_items()
-    if recent_items:
-        missing_items = items_get_all_missing_subs(recent_items)
-        if missing_items:
-            for added_at, item_id, title, item in missing_items:
-                oc.add(DirectoryObject(
-                    key=Callback(ItemDetailsMenu, title=base_title + " > " + title, item_title=title, rating_key=item_id),
-                    title=title,
-                    thumb=get_item_thumb(item) or default_thumb
-                ))
+@route(PREFIX + '/recent', force=bool)
+@debounce
+def RecentMissingSubtitlesMenu(force=False, randomize=None):
+    title="Items with missing subtitles"
+    oc = SubFolderObjectContainer(title2=title, no_cache=True, no_history=True)
+
+    running = scheduler.is_task_running("MissingSubtitles")
+    task_data = scheduler.get_task_data("MissingSubtitles")
+    missing_items = task_data["missing_subtitles"] if task_data else None
+
+    if ((missing_items is None) or force) and not running:
+        scheduler.dispatch_task("MissingSubtitles")
+        running = True
+
+    if not running:
+        oc.add(DirectoryObject(
+            key=Callback(RecentMissingSubtitlesMenu, force=True, randomize=timestamp()),
+            title=u"Get items with missing subtitles",
+            thumb=default_thumb
+        ))
+    else:
+        oc.add(DirectoryObject(
+            key=Callback(RecentMissingSubtitlesMenu, force=False, randomize=timestamp()),
+            title=u"Updating, refresh here ...",
+            thumb=default_thumb
+        ))
+
+    if missing_items is not None:
+        for added_at, item_id, item_title, item, missing_languages in missing_items:
+            oc.add(DirectoryObject(
+                key=Callback(ItemDetailsMenu, title=title + " > " + item_title, item_title=item_title, rating_key=item_id),
+                title=item_title,
+                summary="Missing: %s" % ", ".join(l.name for l in missing_languages),
+                thumb=get_item_thumb(item) or default_thumb
+            ))
+
+        scheduler.clear_task_data("MissingSubtitles")
 
     return oc
 
@@ -165,7 +271,7 @@ def mergedItemsMenu(title, itemGetter, itemGetterKwArgs=None, base_title=None, *
     :param kwargs:
     :return:
     """
-    oc = ObjectContainer(title2=title, no_cache=True, no_history=True)
+    oc = SubFolderObjectContainer(title2=title, no_cache=True, no_history=True)
     items = itemGetter(*args, **kwargs)
 
     for kind, title, item_id, deeper, item in items:
@@ -178,14 +284,16 @@ def mergedItemsMenu(title, itemGetter, itemGetterKwArgs=None, base_title=None, *
     return oc
 
 
-def determine_section_display(kind, item):
+def determine_section_display(kind, item, pass_kwargs=None):
     """
     returns the menu function for a section based on the size of it (amount of items)
     :param kind:
     :param item:
     :return:
     """
-    if item.size > 200:
+    if pass_kwargs and pass_kwargs.get("section_items_key", "all") != "all":
+        return SectionMenu
+    if item.size > 80:
         return SectionFirstLetterMenu
     return SectionMenu
 
@@ -203,7 +311,7 @@ def IgnoreMenu(kind, rating_key, title=None, sure=False, todo="not_set"):
     """
     is_ignored = rating_key in ignore_list[kind]
     if not sure:
-        oc = ObjectContainer(no_history=True, replace_parent=True, title1="%s %s %s %s the ignore list" % (
+        oc = SubFolderObjectContainer(no_history=True, replace_parent=True, title1="%s %s %s %s the ignore list" % (
             "Add" if not is_ignored else "Remove", ignore_list.verbose(kind), title, "to" if not is_ignored else "from"), title2="Are you sure?")
         oc.add(DirectoryObject(
             key=Callback(IgnoreMenu, kind=kind, rating_key=rating_key, title=title, sure=True, todo="add" if not is_ignored else "remove"),
@@ -241,22 +349,26 @@ def IgnoreMenu(kind, rating_key, title=None, sure=False, todo="not_set"):
 
 
 @route(PREFIX + '/sections')
-def SectionsMenu():
+def SectionsMenu(base_title="Sections", section_items_key="all", ignore_options=True):
     """
     displays the menu for all sections
     :return:
     """
     items = get_all_items("sections")
 
-    return dig_tree(ObjectContainer(title2="Sections", no_cache=True, no_history=True), items, None,
-                    menu_determination_callback=determine_section_display, pass_kwargs={"base_title": "Sections"},
+    return dig_tree(SubFolderObjectContainer(title2="Sections", no_cache=True, no_history=True), items, None,
+                    menu_determination_callback=determine_section_display, pass_kwargs={"base_title": base_title,
+                                                                                        "section_items_key": section_items_key,
+                                                                                        "ignore_options": ignore_options},
                     fill_args={"title": "section_title"})
 
 
 @route(PREFIX + '/section', ignore_options=bool)
-def SectionMenu(rating_key, title=None, base_title=None, section_title=None, ignore_options=True):
+def SectionMenu(rating_key, title=None, base_title=None, section_title=None, ignore_options=True,
+                section_items_key="all"):
     """
     displays the contents of a section
+    :param section_items_key:
     :param rating_key:
     :param title:
     :param base_title:
@@ -264,14 +376,14 @@ def SectionMenu(rating_key, title=None, base_title=None, section_title=None, ign
     :param ignore_options:
     :return:
     """
-    items = get_all_items(key="all", value=rating_key, base="library/sections")
+    items = get_all_items(key=section_items_key, value=rating_key, base="library/sections")
 
     kind, deeper = get_items_info(items)
     title = unicode(title)
 
     section_title = title
     title = base_title + " > " + title
-    oc = ObjectContainer(title2=title, no_cache=True, no_history=True)
+    oc = SubFolderObjectContainer(title2=title, no_cache=True, no_history=True)
     if ignore_options:
         add_ignore_options(oc, "sections", title=section_title, rating_key=rating_key, callback_menu=IgnoreMenu)
 
@@ -281,9 +393,12 @@ def SectionMenu(rating_key, title=None, base_title=None, section_title=None, ign
 
 
 @route(PREFIX + '/section/firstLetter', deeper=bool)
-def SectionFirstLetterMenu(rating_key, title=None, base_title=None, section_title=None):
+def SectionFirstLetterMenu(rating_key, title=None, base_title=None, section_title=None, ignore_options=True,
+                           section_items_key="all"):
     """
     displays the contents of a section indexed by its first char (A-Z, 0-9...)
+    :param ignore_options: ignored
+    :param section_items_key: ignored
     :param rating_key:
     :param title:
     :param base_title:
@@ -295,7 +410,7 @@ def SectionFirstLetterMenu(rating_key, title=None, base_title=None, section_titl
     kind, deeper = get_items_info(items)
 
     title = unicode(title)
-    oc = ObjectContainer(title2=section_title, no_cache=True, no_history=True)
+    oc = SubFolderObjectContainer(title2=section_title, no_cache=True, no_history=True)
     title = base_title + " > " + title
     add_ignore_options(oc, "sections", title=section_title, rating_key=rating_key, callback_menu=IgnoreMenu)
 
@@ -320,7 +435,7 @@ def FirstLetterMetadataMenu(rating_key, key, title=None, base_title=None, displa
     :return:
     """
     title = base_title + " > " + unicode(title)
-    oc = ObjectContainer(title2=title, no_cache=True, no_history=True)
+    oc = SubFolderObjectContainer(title2=title, no_cache=True, no_history=True)
 
     items = get_all_items(key="first_character", value=[rating_key, key], base="library/sections", flat=False)
     kind, deeper = get_items_info(items)
@@ -330,7 +445,8 @@ def FirstLetterMetadataMenu(rating_key, key, title=None, base_title=None, displa
 
 
 @route(PREFIX + '/section/contents', display_items=bool)
-def MetadataMenu(rating_key, title=None, base_title=None, display_items=False, previous_item_type=None, previous_rating_key=None):
+def MetadataMenu(rating_key, title=None, base_title=None, display_items=False, previous_item_type=None,
+                 previous_rating_key=None):
     """
     displays the contents of a section based on whether it has a deeper tree or not (movies->movie (item) list; series->series list)
     :param rating_key:
@@ -344,7 +460,9 @@ def MetadataMenu(rating_key, title=None, base_title=None, display_items=False, p
     title = unicode(title)
     item_title = title
     title = base_title + " > " + title
-    oc = ObjectContainer(title2=title, no_cache=True, no_history=True)
+    oc = SubFolderObjectContainer(title2=title, no_cache=True, no_history=True)
+
+    current_kind = get_item_kind_from_rating_key(rating_key)
 
     if display_items:
         items = get_all_items(key="children", value=rating_key, base="library/metadata")
@@ -355,17 +473,24 @@ def MetadataMenu(rating_key, title=None, base_title=None, display_items=False, p
         if should_display_ignore(items, previous=previous_item_type):
             add_ignore_options(oc, "series", title=item_title, rating_key=rating_key, callback_menu=IgnoreMenu)
 
+        timeout = 30
+        if current_kind == "season":
+            timeout = 360
+        elif current_kind == "series":
+            timeout = 1800
+
         # add refresh
         oc.add(DirectoryObject(
-            key=Callback(RefreshItem, rating_key=rating_key, item_title=item_title, refresh_kind=kind, previous_rating_key=previous_rating_key,
-                         timeout=16000, randomize=timestamp()),
+            key=Callback(RefreshItem, rating_key=rating_key, item_title=title, refresh_kind=current_kind,
+                         previous_rating_key=previous_rating_key, timeout=timeout*1000, randomize=timestamp()),
             title=u"Refresh: %s" % item_title,
-            summary="Refreshes the item, possibly picking up new subtitles on disk"
+            summary="Refreshes the %s, possibly searching for missing and picking up new subtitles on disk" % current_kind
         ))
         oc.add(DirectoryObject(
-            key=Callback(RefreshItem, rating_key=rating_key, item_title=item_title, force=True, refresh_kind=kind,
-                         previous_rating_key=previous_rating_key, timeout=16000),
-            title=u"Force-Refresh: %s" % item_title,
+            key=Callback(RefreshItem, rating_key=rating_key, item_title=title, force=True,
+                         refresh_kind=current_kind, previous_rating_key=previous_rating_key, timeout=timeout*1000,
+                         randomize=timestamp()),
+            title=u"Auto-Find subtitles: %s" % item_title,
             summary="Issues a forced refresh, ignoring known subtitles and searching for new ones"
         ))
     else:
@@ -376,7 +501,7 @@ def MetadataMenu(rating_key, title=None, base_title=None, display_items=False, p
 
 @route(PREFIX + '/ignore_list')
 def IgnoreListMenu():
-    oc = ObjectContainer(title2="Ignore list", replace_parent=True)
+    oc = SubFolderObjectContainer(title2="Ignore list", replace_parent=True)
     for key in ignore_list.key_order:
         values = ignore_list[key]
         for value in values:
@@ -384,7 +509,26 @@ def IgnoreListMenu():
     return oc
 
 
+@route(PREFIX + '/history')
+def HistoryMenu():
+    from support.history import get_history
+    history = get_history()
+    oc = SubFolderObjectContainer(title2="History", replace_parent=True)
+
+    for item in history.history_items:
+        oc.add(DirectoryObject(
+            key=Callback(ItemDetailsMenu, title=item.title, item_title=item.item_title,
+                         rating_key=item.rating_key),
+            title=u"%s (%s)" % (item.item_title, item.mode_verbose),
+            summary=u"%s in %s (%s, score: %s), %s" % (item.lang_name, item.section_title,
+                                                       item.provider_name, item.score, df(item.time))
+        ))
+
+    return oc
+
+
 @route(PREFIX + '/item/{rating_key}/actions')
+@debounce
 def ItemDetailsMenu(rating_key, title=None, base_title=None, item_title=None, randomize=None):
     """
     displays the item details menu of an item that doesn't contain any deeper tree, such as a movie or an episode
@@ -397,64 +541,228 @@ def ItemDetailsMenu(rating_key, title=None, base_title=None, item_title=None, ra
     """
     title = unicode(base_title) + " > " + unicode(title) if base_title else unicode(title)
     item = get_item(rating_key)
+    current_kind = get_item_kind_from_rating_key(rating_key)
 
-    oc = ObjectContainer(title2=title, replace_parent=True)
+    timeout = 30
+
+    oc = SubFolderObjectContainer(title2=title, replace_parent=True)
     oc.add(DirectoryObject(
-        key=Callback(RefreshItem, rating_key=rating_key, item_title=item_title, randomize=timestamp()),
+        key=Callback(RefreshItem, rating_key=rating_key, item_title=item_title, randomize=timestamp(),
+                     timeout=timeout*1000),
         title=u"Refresh: %s" % item_title,
-        summary="Refreshes the item, possibly picking up new subtitles on disk",
+        summary="Refreshes the %s, possibly searching for missing and picking up new subtitles on disk" % current_kind,
         thumb=item.thumb or default_thumb
     ))
     oc.add(DirectoryObject(
-        key=Callback(RefreshItem, rating_key=rating_key, item_title=item_title, force=True, randomize=timestamp()),
-        title=u"Force-Refresh: %s" % item_title,
+        key=Callback(RefreshItem, rating_key=rating_key, item_title=item_title, force=True, randomize=timestamp(),
+                     timeout=timeout*1000),
+        title=u"Auto-search: %s" % item_title,
         summary="Issues a forced refresh, ignoring known subtitles and searching for new ones",
         thumb=item.thumb or default_thumb
     ))
+
+    # get stored subtitle info for item id
+    subtitle_storage = get_subtitle_storage()
+    stored_subs = subtitle_storage.load_or_new(item)
+
+    # get the plex item
+    plex_item = list(Plex["library"].metadata(rating_key))[0]
+
+    # get current media info for that item
+    media = plex_item.media
+
+    # look for subtitles for all available media parts and all of their languages
+    for part in media.parts:
+        filename = os.path.basename(part.file)
+        part_id = str(part.id)
+
+        # iterate through all configured languages
+        for lang in config.lang_list:
+            lang_a2 = lang.alpha2
+            # ietf lang?
+            if cast_bool(Prefs["subtitles.language.ietf"]) and "-" in lang_a2:
+                lang_a2 = lang_a2.split("-")[0]
+
+            # get corresponding stored subtitle data for that media part (physical media item), for language
+            current_sub = stored_subs.get_any(part_id, lang_a2)
+            current_sub_id = None
+            current_sub_provider_name = None
+
+            summary = u"No current subtitle in storage"
+            current_score = None
+            if current_sub:
+                current_sub_id = current_sub.id
+                current_sub_provider_name = current_sub.provider_name
+                current_score = current_sub.score
+
+                summary = u"Current subtitle: %s (added: %s, %s), Language: %s, Score: %i, Storage: %s" % \
+                          (current_sub.provider_name, df(current_sub.date_added), current_sub.mode_verbose, lang,
+                           current_sub.score, current_sub.storage_type)
+
+            oc.add(DirectoryObject(
+                key=Callback(ListAvailableSubsForItemMenu, rating_key=rating_key, part_id=part_id, title=title,
+                             item_title=item_title, language=lang, current_id=current_sub_id,
+                             item_type=plex_item.type, filename=filename, current_data=summary,
+                             randomize=timestamp(), current_provider=current_sub_provider_name,
+                             current_score=current_score),
+                title=u"List %s subtitles" % lang.name,
+                summary=summary
+            ))
+
     add_ignore_options(oc, "videos", title=item_title, rating_key=rating_key, callback_menu=IgnoreMenu)
 
     return oc
 
 
+def get_item_task_data(task_name, rating_key, language):
+    task_data = scheduler.get_task_data(task_name)
+    search_results = task_data.get(rating_key, {}) if task_data else {}
+    return search_results.get(language)
+
+
+@route(PREFIX + '/item/search/{rating_key}/{part_id}', force=bool)
+@debounce
+def ListAvailableSubsForItemMenu(rating_key=None, part_id=None, title=None, item_title=None, filename=None,
+                                 item_type="episode", language=None, force=False, current_id=None, current_data=None,
+                                 current_provider=None, current_score=None, randomize=None):
+    assert rating_key, part_id
+
+    running = scheduler.is_task_running("AvailableSubsForItem")
+    search_results = get_item_task_data("AvailableSubsForItem", rating_key, language)
+
+    if (search_results is None or force) and not running:
+        scheduler.dispatch_task("AvailableSubsForItem", rating_key=rating_key, item_type=item_type, part_id=part_id,
+                                language=language)
+        running = True
+
+    oc = SubFolderObjectContainer(title2=unicode(title), replace_parent=True)
+    oc.add(DirectoryObject(
+        key=Callback(ItemDetailsMenu, rating_key=rating_key, item_title=item_title, title=title, randomize=timestamp()),
+        title=u"Back to: %s" % title,
+        summary=current_data,
+        thumb=default_thumb
+    ))
+
+    metadata = get_plex_metadata(rating_key, part_id, item_type)
+    scanned_parts = scan_videos([metadata], kind="series" if item_type == "episode" else "movie", ignore_all=True)
+
+    if not scanned_parts:
+        Log.Error("Couldn't list available subtitles for %s", rating_key)
+        return oc
+
+    video, plex_part = scanned_parts.items()[0]
+
+    video_display_data = [video.format] if video.format else []
+    if video.release_group:
+        video_display_data.append(u"by %s" % video.release_group)
+    video_display_data = " ".join(video_display_data)
+
+    current_display = (u"Current: %s (%s) " % (current_provider, current_score) if current_provider else "")
+    if not running:
+        oc.add(DirectoryObject(
+            key=Callback(ListAvailableSubsForItemMenu, rating_key=rating_key, item_title=item_title, language=language,
+                         filename=filename, part_id=part_id, title=title, current_id=current_id, force=True,
+                         current_provider=current_provider, current_score=current_score,
+                         current_data=current_data, item_type=item_type, randomize=timestamp()),
+            title=u"Search for %s subs (%s)" % (get_language(language).name, video_display_data),
+            summary=u"%sFilename: %s" % (current_display, filename),
+            thumb=default_thumb
+        ))
+    else:
+        oc.add(DirectoryObject(
+            key=Callback(ListAvailableSubsForItemMenu, rating_key=rating_key, item_title=item_title,
+                         language=language, filename=filename, current_data=current_data,
+                         part_id=part_id, title=title, current_id=current_id, item_type=item_type,
+                         current_provider=current_provider, current_score=current_score,
+                         randomize=timestamp()),
+            title=u"Searching for %s subs (%s), refresh here ..." % (get_language(language).name, video_display_data),
+            summary=u"%sFilename: %s" % (current_display, filename),
+            thumb=default_thumb
+        ))
+
+    if not search_results:
+        return oc
+
+    for subtitle in search_results:
+        oc.add(DirectoryObject(
+            key=Callback(TriggerDownloadSubtitle, rating_key=rating_key, randomize=timestamp(), item_title=item_title,
+                         subtitle_id=str(subtitle.id), language=language),
+            title=u"%s: %s, score: %s" % ("Available" if current_id != subtitle.id else "Current",
+                                    subtitle.provider_name, subtitle.score),
+            summary=u"Release: %s, Matches: %s" % (subtitle.release_info, ", ".join(subtitle.matches)),
+            thumb=default_thumb
+        ))
+
+    return oc
+
+
+@route(PREFIX + '/download_subtitle/{rating_key}')
+@debounce
+def TriggerDownloadSubtitle(rating_key=None, subtitle_id=None, item_title=None, language=None, randomize=None):
+    set_refresh_menu_state("Downloading subtitle for %s" % item_title or rating_key)
+    search_results = get_item_task_data("AvailableSubsForItem", rating_key, language)
+
+    download_subtitle = None
+    for subtitle in search_results:
+        if str(subtitle.id) == subtitle_id:
+            download_subtitle = subtitle
+            break
+    if not download_subtitle:
+        Log.Error(u"Something went horribly wrong")
+
+    else:
+        scheduler.dispatch_task("DownloadSubtitleForItem", rating_key=rating_key, subtitle=download_subtitle)
+
+    return fatality(randomize=timestamp(), header=" ", replace_parent=True)
+
+
 @route(PREFIX + '/item/{rating_key}')
 @debounce
-def RefreshItem(rating_key=None, came_from="/recent", item_title=None, force=False, refresh_kind=None, previous_rating_key=None, timeout=8000, randomize=None, trigger=True):
+def RefreshItem(rating_key=None, came_from="/recent", item_title=None, force=False, refresh_kind=None,
+                previous_rating_key=None, timeout=8000, randomize=None, trigger=True):
     assert rating_key
     header = " "
     if trigger:
         set_refresh_menu_state(u"Triggering %sRefresh for %s" % ("Force-" if force else "", item_title))
-        Thread.Create(refresh_item, rating_key=rating_key, force=force, refresh_kind=refresh_kind, parent_rating_key=previous_rating_key,
-                      timeout=int(timeout))
+        Thread.Create(refresh_item, rating_key=rating_key, force=force, refresh_kind=refresh_kind,
+                      parent_rating_key=previous_rating_key, timeout=int(timeout))
+
         header = u"%s of item %s triggered" % ("Refresh" if not force else "Forced-refresh", rating_key)
     return fatality(randomize=timestamp(), header=header, replace_parent=True)
 
 
 @route(PREFIX + '/missing/refresh')
 @debounce
-def RefreshMissing(randomize=None, trigger=True):
-    header = " "
-    if trigger:
-        Thread.CreateTimer(1.0, lambda: scheduler.run_task("searchAllRecentlyAddedMissing"))
-        header = "Refresh of recently added items with missing subtitles triggered"
+def RefreshMissing(randomize=None):
+    scheduler.dispatch_task("SearchAllRecentlyAddedMissing")
+    header = "Refresh of recently added items with missing subtitles triggered"
     return fatality(header=header, replace_parent=True)
 
 
 @route(PREFIX + '/advanced')
 def AdvancedMenu(randomize=None, header=None, message=None):
-    oc = ObjectContainer(header=header or "Internal stuff, pay attention!", message=message, no_cache=True, no_history=True,
-                         replace_parent=True, title2="Advanced")
+    oc = SubFolderObjectContainer(header=header or "Internal stuff, pay attention!", message=message, no_cache=True, no_history=True,
+                                  replace_parent=False, title2="Advanced")
+
+    if config.lock_advanced_menu and not config.pin_correct:
+        oc.add(DirectoryObject(
+            key=Callback(PinMenu, randomize=timestamp(), success_go_to="advanced"),
+            title=pad_title("Enter PIN"),
+            summary="The owner has restricted the access to this menu. Please enter the correct pin",
+        ))
+        return oc
 
     oc.add(DirectoryObject(
         key=Callback(TriggerRestart, randomize=timestamp()),
         title=pad_title("Restart the plugin"),
     ))
     oc.add(DirectoryObject(
-        key=Callback(LogStorage, key="tasks", randomize=timestamp()),
-        title=pad_title("Log the plugin's scheduled tasks state storage"),
+        key=Callback(TriggerBetterSubtitles, randomize=timestamp()),
+        title=pad_title("Trigger find better subtitles"),
     ))
     oc.add(DirectoryObject(
-        key=Callback(LogStorage, key="subs", randomize=timestamp()),
-        title=pad_title("Log the plugin's internal subtitle information storage"),
+        key=Callback(LogStorage, key="tasks", randomize=timestamp()),
+        title=pad_title("Log the plugin's scheduled tasks state storage"),
     ))
     oc.add(DirectoryObject(
         key=Callback(LogStorage, key="ignore", randomize=timestamp()),
@@ -464,10 +772,6 @@ def AdvancedMenu(randomize=None, header=None, message=None):
         key=Callback(ResetStorage, key="tasks", randomize=timestamp()),
         title=pad_title("Reset the plugin's scheduled tasks state storage"),
     ))
-    oc.add(DirectoryObject(
-        key=Callback(ResetStorage, key="subs", randomize=timestamp()),
-        title=pad_title("Reset the plugin's internal subtitle information storage"),
-    ))
     oc.add(DirectoryObject(
         key=Callback(ResetStorage, key="ignore", randomize=timestamp()),
         title=pad_title("Reset the plugin's internal ignorelist storage"),
@@ -483,22 +787,26 @@ def ValidatePrefs():
     # cache the channel state
     update_dict = False
     restart = False
+
+    # reset pin
+    Dict["pin_correct_time"] = None
+
+    config.initialize()
     if "channel_enabled" not in Dict:
         update_dict = True
 
-    elif Dict["channel_enabled"] != Prefs["enable_channel"]:
-        Log.Debug("Channel features %s, restarting plugin", "enabled" if Prefs["enable_channel"] else "disabled")
+    elif Dict["channel_enabled"] != config.enable_channel:
+        Log.Debug("Channel features %s, restarting plugin", "enabled" if config.enable_channel else "disabled")
         update_dict = True
         restart = True
 
     if update_dict:
-        Dict["channel_enabled"] = Prefs["enable_channel"]
+        Dict["channel_enabled"] = config.enable_channel
         Dict.Save()
 
     if restart:
         DispatchRestart()
 
-    config.initialize()
     scheduler.setup_tasks()
     set_refresh_menu_state(None)
 
@@ -522,10 +830,9 @@ def DispatchRestart():
 
 @route(PREFIX + '/advanced/restart/trigger')
 @debounce
-def TriggerRestart(randomize=None, trigger=True):
-    if trigger:
-        set_refresh_menu_state("Restarting the plugin")
-        DispatchRestart()
+def TriggerRestart(randomize=None):
+    set_refresh_menu_state("Restarting the plugin")
+    DispatchRestart()
     return fatality(header="Restart triggered, please wait about 5 seconds", force_title=" ", only_refresh=True, replace_parent=True,
                     no_history=True, randomize=timestamp())
 
@@ -538,7 +845,7 @@ def Restart():
 @route(PREFIX + '/storage/reset', sure=bool)
 def ResetStorage(key, randomize=None, sure=False):
     if not sure:
-        oc = ObjectContainer(no_history=True, title1="Reset subtitle storage", title2="Are you sure?")
+        oc = SubFolderObjectContainer(no_history=True, title1="Reset subtitle storage", title2="Are you sure?")
         oc.add(DirectoryObject(
             key=Callback(ResetStorage, key=key, sure=True, randomize=timestamp()),
             title=pad_title("Are you really sure?"),
@@ -568,3 +875,13 @@ def LogStorage(key, randomize=None):
         header='Success',
         message='Information Storage (%s) logged' % key
     )
+
+
+@route(PREFIX + '/triggerbetter')
+def TriggerBetterSubtitles(randomize=None):
+    scheduler.dispatch_task("FindBetterSubtitles")
+    return AdvancedMenu(
+        randomize=timestamp(),
+        header='Success',
+        message='FindBetterSubtitles triggered'
+    )

Contents/Code/interface/menu_helpers.py

@@ -1,12 +1,13 @@
 # coding=utf-8
 import types
+import datetime
 
 from support.items import get_kind, get_item_thumb
-from subzero import intent
-from support.helpers import format_video
+from support.helpers import get_video_display_title
 from support.ignore import ignore_list
+from support.lib import get_intent
+from support.config import config
 from subzero.constants import ICON
-from subzero.func import debouncer
 
 default_thumb = R(ICON)
 
@@ -46,8 +47,8 @@ def add_ignore_options(oc, kind, callback_menu=None, title=None, rating_key=None
     )
 
 
-def dig_tree(oc, items, menu_callback, menu_determination_callback=None, force_rating_key=None, fill_args=None, pass_kwargs=None,
-             thumb=default_thumb):
+def dig_tree(oc, items, menu_callback, menu_determination_callback=None, force_rating_key=None, fill_args=None,
+             pass_kwargs=None, thumb=default_thumb):
     for kind, title, key, dig_deeper, item in items:
         thumb = get_item_thumb(item) or thumb
 
@@ -57,10 +58,13 @@ def dig_tree(oc, items, menu_callback, menu_determination_callback=None, force_r
         if pass_kwargs:
             add_kwargs.update(pass_kwargs)
 
+        # force details view for show/season
+        summary = " " if kind in ("show", "season") else None
+
         oc.add(DirectoryObject(
-            key=Callback(menu_callback or menu_determination_callback(kind, item), title=title, rating_key=force_rating_key or key,
-                         **add_kwargs),
-            title=title, thumb=thumb
+            key=Callback(menu_callback or menu_determination_callback(kind, item, pass_kwargs=pass_kwargs), title=title,
+                         rating_key=force_rating_key or key, **add_kwargs),
+            title=title, thumb=thumb, summary=summary
         ))
     return oc
 
@@ -90,9 +94,11 @@ def set_refresh_menu_state(state_or_media, media_type="movies"):
             for episode in media.seasons[season].episodes:
                 ep = media.seasons[season].episodes[episode]
                 media_id = ep.id
-                title = format_video("show", ep.title, parent_title=media.title, season=int(season), episode=int(episode))
+                title = get_video_display_title("show", ep.title, parent_title=media.title, season=int(season), episode=int(episode))
     else:
-        title = format_video("movie", media.title)
+        title = get_video_display_title("movie", media.title)
+
+    intent = get_intent()
     force_refresh = intent.get("force", media_id)
 
     Dict["current_refresh_state"] = u"%sRefreshing %s" % ("Force-" if force_refresh else "", unicode(title))
@@ -117,7 +123,7 @@ def enable_channel_wrapper(func):
 
     def wrap(*args, **kwargs):
         enforce_route = kwargs.pop("enforce_route", None)
-        return (func if Prefs["enable_channel"] or enforce_route else noop)(*args, **kwargs)
+        return (func if config.enable_channel or enforce_route else noop)(*args, **kwargs)
 
     return wrap
 
@@ -128,13 +134,61 @@ def debounce(func):
     :param func:
     :return:
     """
+    def get_lookup_key(args, kwargs):
+        func_name = list(args).pop(0).__name__
+        return tuple([func_name] + [(key, value) for key, value in kwargs.iteritems()])
+
     def wrap(*args, **kwargs):
         if "randomize" in kwargs:
-            if ([func] + list(args), kwargs) in debouncer:
-                kwargs["trigger"] = False
+            if not "menu_history" in Dict:
+                Dict["menu_history"] = {}
+
+            key = get_lookup_key([func] + list(args), kwargs)
+            if key in Dict["menu_history"]:
                 Log.Debug("not triggering %s twice with %s, %s" % (func, args, kwargs))
+                return ObjectContainer()
             else:
-                debouncer.add([func] + list(args), kwargs)
+                Dict["menu_history"][key] = datetime.datetime.now() + datetime.timedelta(days=1)
+                Dict.Save()
         return func(*args, **kwargs)
 
     return wrap
+
+
+class SZObjectContainer(ObjectContainer):
+    def __init__(self, *args, **kwargs):
+        skip_pin_lock = kwargs.pop("skip_pin_lock", False)
+
+        super(SZObjectContainer, self).__init__(*args, **kwargs)
+
+        if (config.lock_menu or config.lock_advanced_menu) and not config.pin_correct and not skip_pin_lock:
+            config.locked = True
+
+    def add(self, *args, **kwargs):
+        # disable self.add if we're in lockdown
+        container = args[0]
+        current_menu_target = container.key.split("?")[0]
+        is_pin_menu = current_menu_target.endswith("/pin")
+
+        if config.locked and config.lock_menu and not is_pin_menu:
+            return
+        return super(SZObjectContainer, self).add(*args, **kwargs)
+
+
+OriginalObjectContainer = ObjectContainer
+ObjectContainer = SZObjectContainer
+
+
+class SubFolderObjectContainer(ObjectContainer):
+    def __init__(self, *args, **kwargs):
+        super(SubFolderObjectContainer, self).__init__(*args, **kwargs)
+        from interface.menu import fatality
+        from support.helpers import pad_title, timestamp
+        self.add(DirectoryObject(
+            key=Callback(fatality, force_title=" ", randomize=timestamp()),
+            title=pad_title("<< Back to home"),
+            summary="Current state: %s; Last state: %s" % (
+                (Dict["current_refresh_state"] or "Idle") if "current_refresh_state" in Dict else "Idle",
+                (Dict["last_refresh_state"] or "None") if "last_refresh_state" in Dict else "None"
+            )
+        ))

Contents/Code/support/__init__.py

@@ -47,3 +47,14 @@ sys.modules["support.storage"] = storage
 import ignore
 
 sys.modules["support.ignore"] = ignore
+
+import history
+
+sys.modules["support.history"] = history
+
+import data
+
+sys.modules["support.data"] = data
+
+import activities
+sys.modules["support.activities"] = activities

Contents/Code/support/activities.py

@@ -0,0 +1,111 @@
+# coding=utf-8
+from wraptor.decorators import throttle
+from config import config
+from items import get_item, get_item_kind_from_item, refresh_item
+
+from plex_activity import Activity
+from plex_activity.sources.s_logging.main import Logging as Activity_Logging
+
+
+class PlexActivityManager(object):
+    def start(self):
+        activity_sources_enabled = None
+
+        if config.universal_plex_token:
+            from plex import Plex
+            Plex.configuration.defaults.authentication(config.universal_plex_token)
+            activity_sources_enabled = ["websocket"]
+            Activity.on('websocket.playing', self.on_playing)
+
+        elif config.server_log_path:
+            Activity_Logging.add_hint(config.server_log_path, None)
+            activity_sources_enabled = ["logging"]
+            Activity.on('logging.playing', self.on_playing)
+
+        if activity_sources_enabled:
+            Activity.start(activity_sources_enabled)
+
+    @throttle(5, instance_method=True)
+    def on_playing(self, info):
+        if not config.use_activities:
+            return
+
+        # ignore non-playing states and anything too far in
+        if info["state"] != "playing" or info["viewOffset"] > 60000:
+            return
+
+        # don't trigger on the first hit ever
+        if "last_played_items" not in Dict:
+            Dict["last_played_items"] = []
+            Dict.Save()
+            return
+
+        rating_key = info["ratingKey"]
+        if rating_key not in Dict["last_played_items"]:
+            # new playing; store last 10 recently played items
+            Dict["last_played_items"].insert(0, rating_key)
+            Dict["last_played_items"] = Dict["last_played_items"][:10]
+
+            Dict.Save()
+
+            debug_msg = "Started playing %s. Refreshing it." % rating_key
+
+            key_to_refresh = None
+            if config.activity_mode in ["refresh", "next_episode", "hybrid"]:
+                # next episode or next episode and current movie
+                if config.activity_mode in ["next_episode", "hybrid"]:
+                    plex_item = get_item(rating_key)
+                    if not plex_item:
+                        Log.Warn("Can't determine media type of %s, skipping" % rating_key)
+                        return
+
+                    if get_item_kind_from_item(plex_item) == "episode":
+                        next_ep = self.get_next_episode(rating_key)
+                        if next_ep:
+                            key_to_refresh = next_ep.rating_key
+                            debug_msg = "Started playing %s. Refreshing next episode (%s, S%02iE%02i)." % \
+                                        (rating_key, next_ep.rating_key, int(next_ep.season.index), int(next_ep.index))
+
+                    else:
+                        if config.activity_mode == "hybrid":
+                            key_to_refresh = rating_key
+                elif config.activity_mode == "refresh":
+                    key_to_refresh = rating_key
+
+                if key_to_refresh:
+                    Log.Debug(debug_msg)
+                    refresh_item(key_to_refresh)
+
+    def get_next_episode(self, rating_key):
+        plex_item = get_item(rating_key)
+        if not plex_item:
+            return
+
+        if get_item_kind_from_item(plex_item) == "episode":
+            # get season
+            season = get_item(plex_item.season.rating_key)
+            if not season:
+                return
+
+            # determine next episode
+            # next episode is in the same season
+            if plex_item.index < season.episode_count:
+                # get next ep
+                for ep in season.children():
+                    if ep.index == plex_item.index + 1:
+                        return ep
+
+            # it's not, try getting the first episode of the next season
+            else:
+                # get show
+                show = get_item(plex_item.show.rating_key)
+                # is there a next season?
+                if season.index < show.season_count:
+                    for other_season in show.children():
+                        if other_season.index == season.index + 1:
+                            next_season = other_season
+                            for ep in next_season.children():
+                                if ep.index == 1:
+                                    return ep
+
+activity = PlexActivityManager()

Contents/Code/support/background.py

@@ -6,7 +6,7 @@ import traceback
 
 
 def parse_frequency(s):
-    if s == "never":
+    if s == "never" or s == None:
         return None, None
     kind, num, unit = s.split()
     return int(num), unit
@@ -27,9 +27,40 @@ class DefaultScheduler(object):
 
     def init_storage(self):
         if "tasks" not in Dict:
-            Dict["tasks"] = {}
+            Dict["tasks"] = {"queue": []}
             Dict.Save()
 
+        if "queue" not in Dict["tasks"]:
+            Dict["tasks"]["queue"] = []
+
+    def get_task_data(self, name):
+        if name not in Dict["tasks"]:
+            raise NotImplementedError("Task missing! %s" % name)
+
+        if "data" in Dict["tasks"][name]:
+            return Dict["tasks"][name]["data"]
+
+    def clear_task_data(self, name=None):
+        if name is None:
+            # full clean
+            Log.Debug("Clearing previous task data")
+            if Dict["tasks"]:
+                for task_name in Dict["tasks"].keys():
+                    if task_name == "queue":
+                        continue
+
+                    Dict["tasks"][task_name]["data"] = {}
+                    Dict["tasks"][task_name]["running"] = False
+                Dict.Save()
+            return
+
+        if name not in Dict["tasks"]:
+            raise NotImplementedError("Task missing! %s" % name)
+
+        Dict["tasks"][name]["data"] = {}
+        Dict.Save()
+        Log.Debug("Task data cleared: %s", name)
+
     def register(self, task):
         self.registry.append(task)
 
@@ -38,7 +69,12 @@ class DefaultScheduler(object):
         self.tasks = {}
         for cls in self.registry:
             task = cls(self)
-            self.tasks[task.name] = {"task": task, "frequency": parse_frequency(Prefs["scheduler.tasks.%s" % task.name])}
+            try:
+                task_frequency = Prefs["scheduler.tasks.%s.frequency" % task.name]
+            except KeyError:
+                task_frequency = None
+
+            self.tasks[task.name] = {"task": task, "frequency": parse_frequency(task_frequency)}
 
     def run(self):
         self.running = True
@@ -52,13 +88,18 @@ class DefaultScheduler(object):
             return None
         return self.tasks[name]["task"]
 
+    def is_task_running(self, name):
+        task = self.task(name)
+        if task:
+            return task.running
+
     def last_run(self, task):
         if task not in self.tasks:
             return None
         return self.tasks[task]["task"].last_run
 
     def next_run(self, task):
-        if task not in self.tasks:
+        if task not in self.tasks or not self.tasks[task]["task"].periodic:
             return None
         frequency_num, frequency_key = self.tasks[task]["frequency"]
         if not frequency_num:
@@ -70,24 +111,34 @@ class DefaultScheduler(object):
             use_date = now
         return max(use_date + datetime.timedelta(**{frequency_key: frequency_num}), now)
 
-    def run_task(self, name):
+    def run_task(self, name, *args, **kwargs):
         task = self.tasks[name]["task"]
         if task.running:
             Log.Debug("Scheduler: Not running %s, as it's currently running.", name)
-            return
+            return False
 
         Log.Debug("Scheduler: Running task %s", name)
         try:
-            task.prepare()
+            task.prepare(*args, **kwargs)
             task.run()
         except Exception, e:
             Log.Error("Scheduler: Something went wrong when running %s: %s", name, traceback.format_exc())
         finally:
-            task.post_run()
+            task.post_run(Dict["tasks"][name]["data"])
+            Dict.Save()
+
+    def dispatch_task(self, *args, **kwargs):
+        if "queue" not in Dict["tasks"]:
+            Dict["tasks"]["queue"] = []
+
+        Dict["tasks"]["queue"].append((args, kwargs))
 
     def signal(self, name, *args, **kwargs):
         for task_name, info in self.tasks.iteritems():
             task = info["task"]
+            if not task.periodic:
+                continue
+
             if task.running:
                 Log.Debug("Scheduler: Sending signal %s to task %s (%s, %s)", name, task_name, args, kwargs)
                 status = task.signal(name, *args, **kwargs)
@@ -104,11 +155,22 @@ class DefaultScheduler(object):
             if not self.running:
                 break
 
+            # single dispatch requested?
+            if Dict["tasks"]["queue"]:
+                # work queue off
+                queue = Dict["tasks"]["queue"][:]
+                Dict["tasks"]["queue"] = []
+                Dict.Save()
+                for args, kwargs in queue:
+                    Log.Debug("Dispatching single task: %s, %s", args, kwargs)
+                    Thread.Create(self.run_task, True, *args, **kwargs)
+
+            # scheduled tasks
             for name, info in self.tasks.iteritems():
                 now = datetime.datetime.now()
                 task = info["task"]
 
-                if name not in Dict["tasks"]:
+                if name not in Dict["tasks"] or not task.periodic:
                     continue
 
                 if task.running:
@@ -118,10 +180,10 @@ class DefaultScheduler(object):
                 if not frequency_num:
                     continue
 
-                if not task.last_run or task.last_run + datetime.timedelta(**{frequency_key: frequency_num}) <= now:
+                if not task.last_run or (task.last_run + datetime.timedelta(**{frequency_key: frequency_num}) <= now):
                     self.run_task(name)
 
-            Thread.Sleep(10.0)
+            Thread.Sleep(5.0)
 
 
 scheduler = DefaultScheduler()

Contents/Code/support/config.py

@@ -3,16 +3,23 @@
 import os
 import re
 import inspect
+
+import datetime
+
+import subliminal
+import subliminal_patch
 from babelfish import Language
 from subzero.lib.io import FileIO, get_viable_encoding
 from subzero.constants import PLUGIN_NAME, PLUGIN_IDENTIFIER, MOVIE, SHOW
 from lib import Plex
-from helpers import check_write_permissions
+from helpers import check_write_permissions, cast_bool
 
 SUBTITLE_EXTS = ['utf', 'utf8', 'utf-8', 'srt', 'smi', 'rt', 'ssa', 'aqt', 'jss', 'ass', 'idx', 'sub', 'txt', 'psb']
-VIDEO_EXTS = ['3g2', '3gp', 'asf', 'asx', 'avc', 'avi', 'avs', 'bivx', 'bup', 'divx', 'dv', 'dvr-ms', 'evo', 'fli', 'flv',
+VIDEO_EXTS = ['3g2', '3gp', 'asf', 'asx', 'avc', 'avi', 'avs', 'bivx', 'bup', 'divx', 'dv', 'dvr-ms', 'evo', 'fli',
+              'flv',
               'm2t', 'm2ts', 'm2v', 'm4v', 'mkv', 'mov', 'mp4', 'mpeg', 'mpg', 'mts', 'nsv', 'nuv', 'ogm', 'ogv', 'tp',
-              'pva', 'qt', 'rm', 'rmvb', 'sdp', 'svq3', 'strm', 'ts', 'ty', 'vdr', 'viv', 'vob', 'vp3', 'wmv', 'wpl', 'wtv', 'xsp', 'xvid',
+              'pva', 'qt', 'rm', 'rmvb', 'sdp', 'svq3', 'strm', 'ts', 'ty', 'vdr', 'viv', 'vob', 'vp3', 'wmv', 'wpl',
+              'wtv', 'xsp', 'xvid',
               'webm']
 
 IGNORE_FN = ("subzero.ignore", ".subzero.ignore", ".nosz")
@@ -30,6 +37,17 @@ def int_or_default(s, default):
 class Config(object):
     version = None
     full_version = None
+    server_log_path = None
+    app_support_path = None
+    universal_plex_token = None
+
+    enable_channel = True
+    enable_agent = True
+    pin = None
+    lock_menu = False
+    lock_advanced_menu = False
+    locked = False
+    pin_valid_minutes = 10
     lang_list = None
     subtitle_destination_folder = None
     providers = None
@@ -37,11 +55,20 @@ class Config(object):
     max_recent_items_per_library = 200
     permissions_ok = False
     missing_permissions = None
+    ignore_sz_files = False
     ignore_paths = None
     fs_encoding = None
     notify_executable = None
     sections = None
     enabled_sections = None
+    enforce_encoding = False
+    chmod = None
+    forced_only = False
+    exotic_ext = False
+    treat_und_as_first = False
+    ext_match_strictness = False
+    use_activities = False
+    activity_mode = None
 
     initialized = False
 
@@ -49,26 +76,107 @@ class Config(object):
         self.fs_encoding = get_viable_encoding()
         self.version = self.get_version()
         self.full_version = u"%s %s" % (PLUGIN_NAME, self.version)
+        self.server_log_path = self.get_server_log_path()
+        self.app_support_path = Core.app_support_path
+        self.universal_plex_token = self.get_universal_plex_token()
+
+        self.set_plugin_mode()
+        self.set_plugin_lock()
+        self.set_activity_modes()
+
         self.lang_list = self.get_lang_list()
         self.subtitle_destination_folder = self.get_subtitle_destination_folder()
         self.providers = self.get_providers()
         self.provider_settings = self.get_provider_settings()
-        self.max_recent_items_per_library = int_or_default(Prefs["scheduler.max_recent_items_per_library"], 200)
+        self.max_recent_items_per_library = int_or_default(Prefs["scheduler.max_recent_items_per_library"], 2000)
         self.sections = list(Plex["library"].sections())
         self.missing_permissions = []
+        self.ignore_sz_files = cast_bool(Prefs["subtitles.ignore_fs"])
         self.ignore_paths = self.parse_ignore_paths()
+        self.enabled_sections = self.check_enabled_sections()
         self.permissions_ok = self.check_permissions()
         self.notify_executable = self.check_notify_executable()
-        self.enabled_sections = self.check_enabled_sections()
+        self.enforce_encoding = cast_bool(Prefs['subtitles.enforce_encoding'])
+        self.chmod = self.check_chmod()
+        self.forced_only = cast_bool(Prefs["subtitles.only_foreign"])
+        self.exotic_ext = cast_bool(Prefs["subtitles.scan.exotic_ext"])
+        self.treat_und_as_first = cast_bool(Prefs["subtitles.language.treat_und_as_first"])
+        self.ext_match_strictness = self.determine_ext_sub_strictness()
         self.initialized = True
 
+    def get_server_log_path(self):
+        # find log handler
+        for handler in Core.log.handlers:
+            if getattr(getattr(handler, "__class__"), "__name__") in (
+                    'FileHandler', 'RotatingFileHandler', 'TimedRotatingFileHandler'):
+                plugin_log_file = handler.baseFilename
+
+                if plugin_log_file:
+                    server_log_file = os.path.realpath(os.path.join(plugin_log_file, "../../Plex Media Server.log"))
+                    if os.path.isfile(server_log_file):
+                        return server_log_file
+
+    def get_universal_plex_token(self):
+        # thanks to: https://forums.plex.tv/discussion/247136/read-current-x-plex-token-in-an-agent-ensure-that-a-http-request-gets-executed-exactly-once#latest
+        pref_path = os.path.join(self.app_support_path, "Preferences.xml")
+        if os.path.exists(pref_path):
+            try:
+                global_prefs = Core.storage.load(pref_path)
+                return XML.ElementFromString(global_prefs).xpath('//Preferences/@PlexOnlineToken')[0]
+            except:
+                Log.Warn("Couldn't determine Plex Token")
+        else:
+            Log("Did NOT find Preferences file - please check logfile and hierarchy. Aborting!")
+
+    def set_plugin_mode(self):
+        if Prefs["plugin_mode"] == "only agent":
+            self.enable_channel = False
+        elif Prefs["plugin_mode"] == "only channel":
+            self.enable_agent = False
+
+    def set_plugin_lock(self):
+        if Prefs["plugin_pin_mode"] in ("channel menu", "advanced menu"):
+            # check pin
+            pin = Prefs["plugin_pin"]
+            if not pin or not len(pin):
+                Log.Warn("PIN enabled but not set, disabling PIN!")
+                return
+
+            pin = pin.strip()
+            try:
+                int(pin)
+            except ValueError:
+                Log.Warn("PIN has to be an integer (0-9)")
+            self.pin = pin
+            self.lock_advanced_menu = Prefs["plugin_pin_mode"] == "advanced menu"
+            self.lock_menu = Prefs["plugin_pin_mode"] == "channel menu"
+
+            try:
+                self.pin_valid_minutes = int(Prefs["plugin_pin_valid_for"].strip())
+            except ValueError:
+                pass
+
+    @property
+    def pin_correct(self):
+        if isinstance(Dict["pin_correct_time"], datetime.datetime) \
+                and Dict["pin_correct_time"] + datetime.timedelta(
+                    minutes=self.pin_valid_minutes) > datetime.datetime.now():
+            return True
+
+    def refresh_permissions_status(self):
+        self.permissions_ok = self.check_permissions()
+
     def check_permissions(self):
         if not Prefs["subtitles.save.filesystem"] or not Prefs["check_permissions"]:
             return True
 
+        self.missing_permissions = []
         use_ignore_fs = Prefs["subtitles.ignore_fs"]
         all_permissions_ok = True
         for section in self.sections:
+            if section.key not in self.enabled_sections:
+                continue
+
             title = section.title
             for location in section:
                 path_str = location.path
@@ -137,6 +245,9 @@ class Config(object):
             return exe_fn, arguments
         Log.Error("Notify executable not existing or not executable: %s" % exe_fn)
 
+    def refresh_enabled_sections(self):
+        self.enabled_sections = self.check_enabled_sections()
+
     def check_enabled_sections(self):
         enabled_for_primary_agents = []
         enabled_sections = {}
@@ -193,30 +304,91 @@ class Config(object):
         if not Prefs["subtitles.save.filesystem"]:
             return
 
-        fld_custom = Prefs["subtitles.save.subFolder.Custom"].strip() if bool(Prefs["subtitles.save.subFolder.Custom"]) else None
-        return fld_custom or (Prefs["subtitles.save.subFolder"] if Prefs["subtitles.save.subFolder"] != "current folder" else None)
+        fld_custom = Prefs["subtitles.save.subFolder.Custom"].strip() if cast_bool(
+            Prefs["subtitles.save.subFolder.Custom"]) else None
+        return fld_custom or (
+        Prefs["subtitles.save.subFolder"] if Prefs["subtitles.save.subFolder"] != "current folder" else None)
 
     def get_providers(self):
-        providers = {'opensubtitles': Prefs['provider.opensubtitles.enabled'],
-                     #'thesubdb': Prefs['provider.thesubdb.enabled'],
-                     'podnapisi': Prefs['provider.podnapisi.enabled'],
-                     'addic7ed': Prefs['provider.addic7ed.enabled'],
-                     'tvsubtitles': Prefs['provider.tvsubtitles.enabled']
+        providers = {'opensubtitles': cast_bool(Prefs['provider.opensubtitles.enabled']),
+                     # 'thesubdb': Prefs['provider.thesubdb.enabled'],
+                     'podnapisi': cast_bool(Prefs['provider.podnapisi.enabled']),
+                     'addic7ed': cast_bool(Prefs['provider.addic7ed.enabled']),
+                     'tvsubtitles': cast_bool(Prefs['provider.tvsubtitles.enabled'])
                      }
+
+        # ditch non-forced-subtitles-reporting providers
+        if cast_bool(Prefs['subtitles.only_foreign']):
+            providers["addic7ed"] = False
+            providers["tvsubtitles"] = False
+
         return filter(lambda prov: providers[prov], providers)
 
     def get_provider_settings(self):
         provider_settings = {'addic7ed': {'username': Prefs['provider.addic7ed.username'],
                                           'password': Prefs['provider.addic7ed.password'],
-                                          'use_random_agents': Prefs['provider.addic7ed.use_random_agents'],
+                                          'use_random_agents': cast_bool(Prefs['provider.addic7ed.use_random_agents']),
                                           },
                              'opensubtitles': {'username': Prefs['provider.opensubtitles.username'],
                                                'password': Prefs['provider.opensubtitles.password'],
-                                               'use_tag_search': Prefs['provider.opensubtitles.use_tags']
+                                               'use_tag_search': cast_bool(Prefs['provider.opensubtitles.use_tags']),
+                                               'only_foreign': cast_bool(Prefs['subtitles.only_foreign'])
                                                },
+                             'podnapisi': {
+                                 'only_foreign': cast_bool(Prefs['subtitles.only_foreign'])
+                             },
                              }
 
         return provider_settings
 
+    def check_chmod(self):
+        val = Prefs["subtitles.save.chmod"]
+        if not val or not len(val):
+            return
+
+        wrong_chmod = False
+        if len(val) != 4:
+            wrong_chmod = True
+
+        try:
+            return int(val, 8)
+        except ValueError:
+            wrong_chmod = True
+
+        if wrong_chmod:
+            Log.Warn("Chmod setting ignored, please use only 4-digit integers with leading 0 (e.g.: 775)")
+
+    def determine_ext_sub_strictness(self):
+        val = Prefs["subtitles.scan.filename_strictness"]
+        if val == "any":
+            return "any"
+        elif val.startswith("loose"):
+            return "loose"
+        return "strict"
+
+    def set_activity_modes(self):
+        val = Prefs["activity.on_playback"]
+        if val == "never":
+            self.use_activities = False
+            return
+
+        self.use_activities = True
+        if val == "current media item":
+            self.activity_mode = "refresh"
+        elif val == "hybrid: current item or next episode":
+            self.activity_mode = "hybrid"
+        else:
+            self.activity_mode = "next_episode"
+
+    def init_subliminal_patches(self):
+        # configure custom subtitle destination folders for scanning pre-existing subs
+        Log.Debug("Patching subliminal ...")
+        dest_folder = self.subtitle_destination_folder
+        subliminal_patch.patch_video.CUSTOM_PATHS = [dest_folder] if dest_folder else []
+        subliminal_patch.patch_video.INCLUDE_EXOTIC_SUBS = self.exotic_ext
+        subliminal_patch.patch_provider_pool.DOWNLOAD_TRIES = int(Prefs['subtitles.try_downloads'])
+        subliminal.video.Episode.scores["addic7ed_boost"] = int(Prefs['provider.addic7ed.boost_by'])
+
 
 config = Config()
+config.initialize()

Contents/Code/support/data.py

@@ -0,0 +1,84 @@
+# coding=utf-8
+
+
+def dispatch_migrate():
+    try:
+        migrate()
+    except:
+        Log.Error("Migration failed: %s" % traceback.format_exc())
+
+
+def migrate():
+    """
+    some Dict/Data migrations here, no need for a more in-depth migration path for now
+    :return:
+    """
+
+    # migrate subtitle history from Dict to Data
+    if "history" in Dict and Dict["history"].get("history_items"):
+        Log.Debug("Running migration for history data")
+        from support.history import get_history
+        history = get_history()
+
+        for item in reversed(Dict["history"]["history_items"]):
+            history.add(item.item_title, item.rating_key, item.section_title, subtitle=item.subtitle, mode=item.mode,
+                        time=item.time)
+
+        del Dict["history"]
+        Dict.Save()
+
+    # migrate subtitle storage from Dict to Data
+    if "subs" in Dict:
+        from support.storage import get_subtitle_storage
+        from subzero.subtitle_storage import StoredSubtitle
+        from support.plex_media import get_item
+
+        subtitle_storage = get_subtitle_storage()
+
+        for video_id, parts in Dict["subs"].iteritems():
+            try:
+                item = get_item(video_id)
+            except:
+                continue
+
+            if not item:
+                continue
+
+            stored_subs = subtitle_storage.load_or_new(item)
+            stored_subs.version = 1
+
+            Log.Debug(u"Migrating %s" % video_id)
+
+            stored_any = False
+            for part_id, lang_dict in parts.iteritems():
+                part_id = str(part_id)
+                Log.Debug(u"Migrating %s, %s" % (video_id, part_id))
+
+                for lang, subs in lang_dict.iteritems():
+                    lang = str(lang)
+                    if "current" in subs:
+                        current_key = subs["current"]
+                        provider_name, subtitle_id = current_key
+                        sub = subs.get(current_key)
+                        if sub and sub.get("title") and sub.get("mode"):  # ditch legacy data without sufficient info
+                            stored_subs.title = sub["title"]
+                            new_sub = StoredSubtitle(sub["score"], sub["storage"], sub["hash"], provider_name,
+                                                     subtitle_id, date_added=sub["date_added"], mode=sub["mode"])
+
+                            if part_id not in stored_subs.parts:
+                                stored_subs.parts[part_id] = {}
+
+                            if lang not in stored_subs.parts[part_id]:
+                                stored_subs.parts[part_id][lang] = {}
+
+                            Log.Debug(u"Migrating %s, %s, %s" % (video_id, part_id, current_key))
+
+                            stored_subs.parts[part_id][lang][current_key] = new_sub
+                            stored_subs.parts[part_id][lang]["current"] = current_key
+                            stored_any = True
+
+            if stored_any:
+                subtitle_storage.save(stored_subs)
+
+        del Dict["subs"]
+        Dict.Save()

Contents/Code/support/helpers.py

@@ -1,6 +1,7 @@
 # coding=utf-8
 import os
 import traceback
+import types
 import unicodedata
 import datetime
 import urllib
@@ -9,6 +10,14 @@ import re
 import platform
 import subprocess
 
+from bs4 import UnicodeDammit
+
+import chardet
+
+from babelfish import Language
+
+from subzero.analytics import track_event
+
 # Unicode control characters can appear in ID3v2 tags but are not legal in XML.
 RE_UNICODE_CONTROL = u'([\u0000-\u0008\u000b-\u000c\u000e-\u001f\ufffe-\uffff])' + \
                      u'|' + \
@@ -20,6 +29,10 @@ RE_UNICODE_CONTROL = u'([\u0000-\u0008\u000b-\u000c\u000e-\u001f\ufffe-\uffff])'
                      )
 
 
+def cast_bool(value):
+    return str(value) in ("true", "True")
+
+
 # A platform independent way to split paths which might come in with different separators.
 def split_path(str):
     if str.find('\\') != -1:
@@ -33,14 +46,27 @@ def unicodize(s):
     try:
         filename = unicodedata.normalize('NFC', unicode(s.decode('utf-8')))
     except:
-        Log('Failed to unicodize: ' + filename)
+        Log('Failed to unicodize: ' + repr(filename))
     try:
         filename = re.sub(RE_UNICODE_CONTROL, '', filename)
     except:
-        Log('Couldn\'t strip control characters: ' + filename)
+        Log('Couldn\'t strip control characters: ' + repr(filename))
     return filename
 
 
+def force_unicode(s):
+    if not isinstance(s, types.UnicodeType):
+        try:
+            s = s.decode("utf-8")
+        except UnicodeDecodeError:
+            t = chardet.detect(s)
+            try:
+                s = s.decode(t["encoding"])
+            except UnicodeDecodeError:
+                s = UnicodeDammit(s).unicode_markup
+    return s
+
+
 def clean_filename(filename):
     # this will remove any whitespace and punctuation chars and replace them with spaces, strip and return as lowercase
     return string.translate(filename.encode('utf-8'), string.maketrans(string.punctuation + string.whitespace,
@@ -89,7 +115,8 @@ def pad_title(value):
     return str_pad(value, 30, pad_char=' ')
 
 
-def format_item(item, kind, parent=None, parent_title=None, section_title=None, add_section_title=False):
+def get_plex_item_display_title(item, kind, parent=None, parent_title=None, section_title=None,
+                                add_section_title=False):
     """
     :param item: plex item
     :param kind: show or movie
@@ -97,28 +124,64 @@ def format_item(item, kind, parent=None, parent_title=None, section_title=None,
     :param parent_title: parentTitle or None
     :return:
     """
-    return format_video(kind, item.title,
-                        section_title=(
-                            section_title or (parent.section.title if parent and getattr(parent, "section") else None)),
-                        parent_title=(parent_title or (parent.show.title if parent else None)),
-                        season=parent.index if parent else None,
-                        episode=item.index if kind == "show" else None,
-                        add_section_title=add_section_title)
+    return get_video_display_title(kind, item.title,
+                                   section_title=(
+                                       section_title or (parent.section.title if parent and getattr(parent, "section")
+                                                         else None)),
+                                   parent_title=(parent_title or (parent.show.title if parent else None)),
+                                   season=parent.index if parent else None,
+                                   episode=item.index if kind == "show" else None,
+                                   add_section_title=add_section_title)
 
 
-def format_video(kind, title, section_title=None, parent_title=None, season=None, episode=None,
-                 add_section_title=False):
+def get_video_display_title(kind, title, section_title=None, parent_title=None, season=None, episode=None,
+                            add_section_title=False):
     section_add = ""
     if add_section_title:
         section_add = ("%s: " % section_title) if section_title else ""
 
     if kind == "show" and parent_title:
         if season and episode:
-            return '%s%s S%02dE%02d, %s' % (section_add, parent_title, season or 0, episode or 0, title)
-        return '%s%s, %s' % (section_add, parent_title, title)
+            return '%s%s S%02dE%02d%s' % (section_add, parent_title, season or 0, episode or 0,
+                                          (", %s" % title if title else ""))
+        return '%s%s%s' % (section_add, parent_title, (", %s" % title if title else ""))
     return "%s%s" % (section_add, title)
 
 
+def get_title_for_video_metadata(metadata, add_section_title=True, add_episode_title=False):
+    """
+
+    :param metadata:
+    :param add_section_title:
+    :param add_episode_title: add the episode's title if its an episode else always add title
+    :return:
+    """
+    # compute item title
+    add_title = (add_episode_title and metadata["series_id"]) or not metadata["series_id"]
+    return get_video_display_title(
+        "show" if metadata["series_id"] else "movie",
+        metadata["title"] if add_title else "",
+        parent_title=metadata.get("series", None),
+        season=metadata.get("season", None),
+        episode=metadata.get("episode", None),
+        section_title=metadata.get("section", None),
+        add_section_title=add_section_title
+    )
+
+
+def get_identifier():
+    identifier = None
+    try:
+        identifier = Platform.MachineIdentifier
+    except:
+        pass
+
+    if not identifier:
+        identifier = String.UUID()
+
+    return Hash.SHA1(identifier + "SUBZEROOOOOOOOOO")
+
+
 def encode_message(base, s):
     return "%s?message=%s" % (base, urllib.quote_plus(s))
 
@@ -131,6 +194,10 @@ def timestamp():
     return int(time.time())
 
 
+def df(d):
+    return d.strftime("%Y-%m-%d %H:%M:%S") if d else "legacy data"
+
+
 def query_plex(url, args):
     """
     simple http query to the plex API without parsing anything too complicated
@@ -196,9 +263,32 @@ def notify_executable(exe_info, videos, subtitles, storage):
 
             Log.Debug(u"Calling %s with arguments: %s" % (exe, prepared_arguments))
             try:
-                output = subprocess.check_output([exe] + prepared_arguments, stderr=subprocess.STDOUT)
+                output = subprocess.check_output(subprocess.list2cmdline([exe] + prepared_arguments),
+                                                 stderr=subprocess.STDOUT, shell=True)
             except subprocess.CalledProcessError:
                 Log.Error(u"Calling %s failed: %s" % (exe, traceback.format_exc()))
             else:
                 Log.Debug(u"Process output: %s" % output)
 
+
+def track_usage(category=None, action=None, label=None, value=None):
+    if not cast_bool(Prefs["track_usage"]):
+        return
+
+    Thread.Create(dispatch_track_usage, category, action, label, value,
+                  identifier=Dict["anon_id"], first_use=Dict["first_use"],
+                  add=Network.PublicAddress)
+
+
+def dispatch_track_usage(*args, **kwargs):
+    identifier = kwargs.pop("identifier")
+    first_use = kwargs.pop("first_use")
+    add = kwargs.pop("add")
+    try:
+        track_event(identifier=identifier, first_use=first_use, add=add, *[str(a) for a in args])
+    except:
+        Log.Debug("Something went wrong when reporting anonymous user statistics: %s", traceback.format_exc())
+
+
+def get_language(lang_short):
+    return Language.fromietf(lang_short)

Contents/Code/support/history.py

@@ -0,0 +1,4 @@
+# coding=utf-8
+from subzero.history_storage import SubtitleHistory
+
+get_history = lambda: SubtitleHistory(Data, int(Prefs["history_size"]))

Contents/Code/support/items.py

@@ -5,9 +5,8 @@ import re
 import types
 import os
 from ignore import ignore_list
-from helpers import is_recent, format_item, query_plex
-from subzero import intent
-from lib import Plex
+from helpers import is_recent, get_plex_item_display_title, query_plex
+from lib import Plex, get_intent
 from config import config, IGNORE_FN
 
 logger = logging.getLogger(__name__)
@@ -21,14 +20,33 @@ def get_item(key):
     item_id = int(key)
     item_container = Plex["library"].metadata(item_id)
 
-    item = list(item_container)[0]
-    return item
+    try:
+        return list(item_container)[0]
+    except IndexError:
+        pass
 
 
 def get_item_kind(item):
     return type(item).__name__
 
 
+PLEX_API_TYPE_MAP = {
+    "Show": "series",
+    "Season": "season",
+    "Episode": "episode",
+    "Movie": "movie",
+}
+
+
+def get_item_kind_from_rating_key(key):
+    item = get_item(key)
+    return PLEX_API_TYPE_MAP[get_item_kind(item)]
+
+
+def get_item_kind_from_item(item):
+    return PLEX_API_TYPE_MAP[get_item_kind(item)]
+
+
 def get_item_thumb(item):
     kind = get_item_kind(item)
     if kind == "Episode":
@@ -104,7 +122,7 @@ def get_items(key="recently_added", base="library", value=None, flat=False, add_
             if flat:
                 # return episodes
                 for child in item.children():
-                    items.append(("episode", format_item(child, "show", parent=item, add_section_title=add_section_title), int(item.rating_key),
+                    items.append(("episode", get_plex_item_display_title(child, "show", parent=item, add_section_title=add_section_title), int(item.rating_key),
                                   False, child))
             else:
                 # return seasons
@@ -120,26 +138,21 @@ def get_items(key="recently_added", base="library", value=None, flat=False, add_
 
         elif kind == "episode":
             items.append(
-                (kind, format_item(item, "show", parent=item.season, parent_title=item.show.title, section_title=item.section.title,
-                                   add_section_title=add_section_title), int(item.rating_key), False, item))
+                (kind, get_plex_item_display_title(item, "show", parent=item.season, parent_title=item.show.title, section_title=item.section.title,
+                                                   add_section_title=add_section_title), int(item.rating_key), False, item))
 
         elif kind in ("movie", "artist", "photo"):
-            items.append((kind, format_item(item, kind, section_title=item.section.title, add_section_title=add_section_title),
+            items.append((kind, get_plex_item_display_title(item, kind, section_title=item.section.title, add_section_title=add_section_title),
                           int(item.rating_key), False, item))
 
         elif kind == "show":
             items.append((
-                kind, format_item(item, kind, section_title=item.section.title, add_section_title=add_section_title), int(item.rating_key), True,
+                kind, get_plex_item_display_title(item, kind, section_title=item.section.title, add_section_title=add_section_title), int(item.rating_key), True,
                 item))
 
     return items
 
 
-def get_recently_added_items():
-    items = get_items(key="recently_added")
-    return filter(lambda x: is_recent(x[MI_ITEM].added_at), items)
-
-
 def get_recent_items():
     """
     actually get the recent items, not limited like /library/recentlyAdded
@@ -195,6 +208,10 @@ def get_on_deck_items():
     return get_items(key="on_deck", add_section_title=True)
 
 
+def get_recently_added_items():
+    return get_items(key="recently_added", add_section_title=True, flat=False)
+
+
 def get_all_items(key, base="library", value=None, flat=False):
     return get_items(key, base=base, value=value, flat=flat)
 
@@ -225,7 +242,7 @@ def is_ignored(rating_key, item=None):
         return True
 
     # physical/path ignore
-    if Prefs["subtitles.ignore_fs"] or config.ignore_paths:
+    if config.ignore_sz_files or config.ignore_paths:
         # normally check current item folder and the library
         check_ignore_paths = [".", "../"]
         if kind == "Episode":
@@ -237,7 +254,7 @@ def is_ignored(rating_key, item=None):
                 Log.Debug("Item %s's path is manually ignored" % rating_key)
                 return True
 
-            if Prefs["subtitles.ignore_fs"]:
+            if config.ignore_sz_files:
                 for sub_path in check_ignore_paths:
                     if config.is_physically_ignored(os.path.abspath(os.path.join(os.path.dirname(part.file), sub_path))):
                         Log.Debug("An ignore file exists in either the items or its parent folders")
@@ -247,13 +264,22 @@ def is_ignored(rating_key, item=None):
 
 
 def refresh_item(rating_key, force=False, timeout=8000, refresh_kind=None, parent_rating_key=None):
+    intent = get_intent()
+
     # timeout actually is the time for which the intent will be valid
     if force:
+        Log.Debug("Setting intent for force-refresh of %s to timeout: %s", rating_key, timeout)
         intent.set("force", rating_key, timeout=timeout)
 
-    if refresh_kind == "episode":
-        # season refresh
-        rating_key = parent_rating_key
+        # force Dict.Save()
+        intent.store.save()
 
-    Log.Info("%s item %s", "Refreshing" if not force else "Forced-refreshing", rating_key)
-    Plex["library/metadata"].refresh(rating_key)
+    refresh = [rating_key]
+
+    if refresh_kind == "season":
+        # season refresh, needs explicit per-episode refresh
+        refresh = [item.rating_key for item in list(Plex["library/metadata"].children(int(rating_key)))]
+
+    for key in refresh:
+        Log.Info("%s item %s", "Refreshing" if not force else "Forced-refreshing", key)
+        Plex["library/metadata"].refresh(key)

Contents/Code/support/lib.py

@@ -1,7 +1,10 @@
 # coding=utf-8
 
 import plex
+from subzero.intent import TempIntent
+from subzero.lib.dict import DictProxy
 from subzero.lib.httpfake import PlexPyNativeResponseProxy
+from subzero.constants import DEFAULT_TIMEOUT
 
 
 class PlexPyNativeRequestProxy(object):
@@ -26,7 +29,8 @@ class PlexPyNativeRequestProxy(object):
         data = None
         status_code = 200
         try:
-            data = HTTP.Request(self.url, headers=self.headers, immediate=True, method=self.method)
+            data = HTTP.Request(self.url, headers=self.headers, immediate=True, method=self.method,
+                                timeout=DEFAULT_TIMEOUT)
         except Ex.HTTPError as e:
             status_code = e.code
         return PlexPyNativeResponseProxy(data, status_code, self)
@@ -35,3 +39,18 @@ class PlexPyNativeRequestProxy(object):
 plex.request.Request = PlexPyNativeRequestProxy
 
 Plex = plex.Plex
+
+
+class IntentDictStorage(DictProxy):
+    store = "intent"
+
+    def setup_defaults(self):
+        return {"force": {}}
+
+
+def get_intent():
+    """
+    use this to get an intent from inside a separate thread
+    :return:
+    """
+    return TempIntent(store=IntentDictStorage(Dict))

Contents/Code/support/localmedia.py

@@ -1,6 +1,7 @@
 # coding=utf-8
 
 import os
+
 import config
 import helpers
 import subtitlehelpers
@@ -12,11 +13,13 @@ def find_subtitles(part):
     lang_sub_map = {}
     part_filename = helpers.unicodize(part.file)
     part_basename = os.path.splitext(os.path.basename(part_filename))[0]
-    use_filesystem = bool(Prefs["subtitles.save.filesystem"])
+    use_filesystem = helpers.cast_bool(Prefs["subtitles.save.filesystem"])
     paths = [os.path.dirname(part_filename)] if use_filesystem else []
 
     global_subtitle_folder = None
 
+    global_folders = []
+
     if use_filesystem:
         # Check for local subtitles subdirectory
         sub_dir_base = paths[0]
@@ -27,15 +30,20 @@ def find_subtitles(part):
             # got selected subfolder
             sub_dir_list.append(os.path.join(sub_dir_base, Prefs["subtitles.save.subFolder"]))
 
-        sub_dir_custom = Prefs["subtitles.save.subFolder.Custom"].strip() if bool(Prefs["subtitles.save.subFolder.Custom"]) else None
+        sub_dir_custom = Prefs["subtitles.save.subFolder.Custom"].strip() \
+            if Prefs["subtitles.save.subFolder.Custom"] else None
+
         if sub_dir_custom:
             # got custom subfolder
-            if sub_dir_custom.startswith("/"):
+            sub_dir_custom = os.path.normpath(sub_dir_custom)
+            if os.path.isdir(sub_dir_custom) and os.path.isabs(sub_dir_custom):
                 # absolute folder
                 sub_dir_list.append(sub_dir_custom)
+                global_folders.append(sub_dir_custom)
             else:
                 # relative folder
-                sub_dir_list.append(os.path.join(sub_dir_base, sub_dir_custom))
+                fld = os.path.join(sub_dir_base, sub_dir_custom)
+                sub_dir_list.append(fld)
 
         for sub_dir in sub_dir_list:
             if os.path.isdir(sub_dir):
@@ -45,6 +53,10 @@ def find_subtitles(part):
         global_subtitle_folder = os.path.join(Core.app_support_path, 'Subtitles')
         if os.path.exists(global_subtitle_folder):
             paths.append(global_subtitle_folder)
+            global_folders.append(global_subtitle_folder)
+
+    # normalize all paths
+    paths = [os.path.normpath(helpers.unicodize(path)) for path in paths]
 
     # We start by building a dictionary of files to their absolute paths. We also need to know
     # the number of media files that are actually present, in case the found local media asset
@@ -52,10 +64,9 @@ def find_subtitles(part):
     #
     file_paths = {}
     total_media_files = 0
+    media_files = []
     for path in paths:
-        path = helpers.unicodize(path)
         for file_path_listing in os.listdir(path.encode(sz_config.fs_encoding)):
-
             # When using os.listdir with a unicode path, it will always return a string using the
             # NFD form. However, we internally are using the form NFC and therefore need to convert
             # it to allow correct regex / comparisons to be performed.
@@ -69,29 +80,92 @@ def find_subtitles(part):
             if ext.lower()[1:] in config.VIDEO_EXTS:
                 total_media_files += 1
 
+                # collect found media files
+                media_files.append(root)
+
+    # cleanup any leftover subtitle if no associated media file was found
+    if helpers.cast_bool(Prefs["subtitles.autoclean"]):
+        for path in paths:
+            # we can't housekeep the global subtitle folders as we don't know about *all* media files
+            # in a library; skip them
+            skip_path = False
+            for fld in global_folders:
+                if path.startswith(fld):
+                    Log.Info("Skipping housekeeping of folder: %s", path)
+                    skip_path = True
+                    break
+
+            if skip_path:
+                continue
+
+            for file_path_listing in os.listdir(path.encode(sz_config.fs_encoding)):
+                file_path_listing = helpers.unicodize(file_path_listing)
+                enc_fn = os.path.join(path, file_path_listing).encode(sz_config.fs_encoding)
+
+                if os.path.isfile(enc_fn):
+                    (root, ext) = os.path.splitext(file_path_listing)
+                    # it's a subtitle file
+                    if ext.lower()[1:] in config.SUBTITLE_EXTS:
+                        # get fn without forced/default/normal tag
+                        split_tag = root.rsplit(".", 1)
+                        if len(split_tag) > 1 and split_tag[1].lower() in ['forced', 'normal', 'default']:
+                            root = split_tag[0]
+
+                        # get associated media file name without language
+                        sub_fn = subtitlehelpers.ENDSWITH_LANGUAGECODE_RE.sub("", root)
+
+                        # subtitle basename and basename without possible language tag  not found in collected
+                        # media files? kill.
+                        if root not in media_files and sub_fn not in media_files:
+                            Log.Info("Removing leftover subtitle: %s", os.path.join(path, file_path_listing))
+                            try:
+                                os.remove(enc_fn)
+                            except (OSError, IOError):
+                                Log.Error("Removing failed")
+
     Log('Looking for subtitle media in %d paths with %d media files.', len(paths), total_media_files)
     Log('Paths: %s', ", ".join([helpers.unicodize(p) for p in paths]))
 
     for file_path in file_paths.values():
+        local_filename = os.path.basename(file_path)
+        bn, ext = os.path.splitext(local_filename)
+        local_basename = helpers.unicodize(bn)
 
-        local_basename = helpers.unicodize(os.path.splitext(os.path.basename(file_path))[0])
+        # get fn without forced/default/normal tag
+        split_tag = local_basename.rsplit(".", 1)
+        if len(split_tag) > 1 and split_tag[1].lower() in ['forced', 'normal', 'default']:
+            local_basename = split_tag[0]
+
+        # split off possible language tag
         local_basename2 = local_basename.rsplit('.', 1)[0]
         filename_matches_part = local_basename == part_basename or local_basename2 == part_basename
+        filename_contains_part = part_basename in local_basename
 
-        # If the file is located within the global subtitle folder and it's name doesn't match exactly
-        # then we should simply ignore it.
-        #
-        if global_subtitle_folder and file_path.count(global_subtitle_folder) and not filename_matches_part:
+        if not ext.lower()[1:] in config.SUBTITLE_EXTS:
             continue
 
-        # If we have more than one media file within the folder and located filename doesn't match
-        # exactly then we should simply ignore it.
-        #
-        if total_media_files > 1 and not filename_matches_part:
-            continue
+        # if the file is located within the global subtitle folders and its name doesn't match exactly, ignore it
+        if global_folders and not filename_matches_part:
+            skip_path = False
+            for fld in global_folders:
+                if file_path.startswith(fld):
+                    skip_path = True
+                    break
+
+            if skip_path:
+                continue
+
+        # determine whether to pick up the subtitle based on our match strictness
+        elif not filename_matches_part:
+            if sz_config.ext_match_strictness == "strict" or (
+                    sz_config.ext_match_strictness == "loose" and not filename_contains_part):
+
+                Log.Debug("%s doesn't match %s, skipping" % (helpers.unicodize(local_filename),
+                                                             helpers.unicodize(part_basename)))
+                continue
 
         subtitle_helper = subtitlehelpers.subtitle_helpers(file_path)
-        if subtitle_helper != None:
+        if subtitle_helper is not None:
             local_lang_map = subtitle_helper.process_subtitles(part)
             for new_language, subtitles in local_lang_map.items():
 

Contents/Code/support/missing_subtitles.py

@@ -2,7 +2,7 @@
 import traceback
 
 from support.config import config
-from support.helpers import format_item
+from support.helpers import get_plex_item_display_title, cast_bool
 from support.items import get_item
 from support.lib import Plex
 
@@ -14,9 +14,9 @@ def item_discover_missing_subs(rating_key, kind="show", added_at=None, section_t
     item = get_item(rating_key)
 
     if kind == "show":
-        item_title = format_item(item, kind, parent=item.season, section_title=section_title, parent_title=item.show.title)
+        item_title = get_plex_item_display_title(item, kind, parent=item.season, section_title=section_title, parent_title=item.show.title)
     else:
-        item_title = format_item(item, kind, section_title=section_title)
+        item_title = get_plex_item_display_title(item, kind, section_title=section_title)
 
     video = item.media
 
@@ -44,7 +44,7 @@ def item_discover_missing_subs(rating_key, kind="show", added_at=None, section_t
         Log.Info(u"Subs still missing for '%s': %s", item_title, missing)
 
     if missing:
-        return added_at, item_id, item_title, item
+        return added_at, item_id, item_title, item, missing
 
 
 def items_get_all_missing_subs(items):
@@ -57,21 +57,21 @@ def items_get_all_missing_subs(items):
                 added_at=added_at,
                 section_title=section_title,
                 languages=config.lang_list,
-                internal=bool(Prefs["subtitles.scan.embedded"]),
-                external=bool(Prefs["subtitles.scan.external"])
+                internal=cast_bool(Prefs["subtitles.scan.embedded"]),
+                external=cast_bool(Prefs["subtitles.scan.external"])
             )
             if state:
-                # (added_at, item_id, title)
+                # (added_at, item_id, title, item, missing_languages)
                 missing.append(state)
         except:
             Log.Error("Something went wrong when getting the state of item %s: %s", key, traceback.format_exc())
     return missing
 
 
-def refresh_item(item, title):
+def refresh_item(item):
     Plex["library/metadata"].refresh(item)
 
 
 def refresh_items(items):
     for item, title in items:
-        refresh_item(item, title)
+        refresh_item(item)

Contents/Code/support/plex_media.py

@@ -1,31 +1,35 @@
 # coding=utf-8
 
 import os
+
 import subliminal
 import helpers
 
 from items import get_item
-from subzero import intent
+from lib import get_intent, Plex
+from config import config
 
 
-def flatten_media(media, kind="series"):
+def get_metadata_dict(item, part, add):
+    data = {
+        "item": item,
+        "section": item.section.title,
+        "path": part.file,
+        "folder": os.path.dirname(part.file),
+        "filename": os.path.basename(part.file)
+    }
+    data.update(add)
+    return data
+
+
+def media_to_videos(media, kind="series"):
     """
     iterates through media and returns the associated parts (videos)
     :param media:
     :param kind:
     :return:
     """
-    parts = []
-
-    def get_metadata_dict(item, part, add):
-        data = {
-            "section": item.section.title,
-            "path": part.file,
-            "folder": os.path.dirname(part.file),
-            "filename": os.path.basename(part.file)
-        }
-        data.update(add)
-        return data
+    videos = []
 
     if kind == "series":
         for season in media.seasons:
@@ -38,41 +42,32 @@ def flatten_media(media, kind="series"):
 
                 for item in media.seasons[season].episodes[episode].items:
                     for part in item.parts:
-                        parts.append(
+                        videos.append(
                             get_metadata_dict(plex_episode, part,
-                                              {"video": part, "type": "episode", "title": ep.title,
+                                              {"plex_part": part, "type": "episode", "title": ep.title,
                                                "series": media.title, "id": ep.id,
                                                "series_id": media.id, "season_id": season_object.id,
-                                               "season": plex_episode.season.index,
+                                               "episode": plex_episode.index, "season": plex_episode.season.index,
+                                               "section": plex_episode.section.title
                                                })
                         )
     else:
         plex_item = get_item(media.id)
         for item in media.items:
             for part in item.parts:
-                parts.append(
-                    get_metadata_dict(plex_item, part, {"video": part, "type": "movie",
+                videos.append(
+                    get_metadata_dict(plex_item, part, {"plex_part": part, "type": "movie",
                                                         "title": media.title, "id": media.id,
                                                         "series_id": None,
                                                         "season_id": None,
                                                         "section": plex_item.section.title})
                 )
-    return parts
+    return videos
 
 
 IGNORE_FN = ("subzero.ignore", ".subzero.ignore", ".nosz")
 
 
-def convert_media_to_parts(media, kind="series"):
-    """
-    returns a list of parts to be used later on; ignores folders with an existing "subzero.ignore" file
-    :param media:
-    :param kind:
-    :return:
-    """
-    return flatten_media(media, kind=kind)
-
-
 def get_stream_fps(streams):
     """
     accepts a list of plex streams or a list of the plex api streams
@@ -97,43 +92,157 @@ def get_media_item_ids(media, kind="series"):
     return ids
 
 
-def scan_video(plex_video, ignore_all=False, hints=None):
+def scan_video(plex_part, ignore_all=False, hints=None, rating_key=None):
     embedded_subtitles = not ignore_all and Prefs['subtitles.scan.embedded']
     external_subtitles = not ignore_all and Prefs['subtitles.scan.external']
 
     if ignore_all:
         Log.Debug("Force refresh intended.")
 
-    Log.Debug("Scanning video: %s, subtitles=%s, embedded_subtitles=%s" % (plex_video.file, external_subtitles, embedded_subtitles))
+    Log.Debug("Scanning video: %s, subtitles=%s, embedded_subtitles=%s" % (plex_part.file, external_subtitles, embedded_subtitles))
+
+    known_embedded = []
+    parts = list(Plex["library"].metadata(rating_key))[0].media.parts
+    plexpy_part = None
+    for part in parts:
+        if int(part.id) == int(plex_part.id):
+            plexpy_part = part
+
+    # embedded subtitles
+    if plexpy_part:
+        for stream in plexpy_part.streams:
+            # subtitle stream
+            if stream.stream_type == 3:
+                if (config.forced_only and getattr(stream, "forced")) or \
+                        (not config.forced_only and not getattr(stream, "forced")):
+
+                    # embedded subtitle
+                    if not stream.stream_key:
+                        if config.exotic_ext or stream.codec in ("srt", "ass", "ssa"):
+                            lang_code = stream.language_code
+
+                            # treat unknown language as lang1?
+                            if not lang_code and config.treat_und_as_first:
+                                lang_code = list(config.lang_list)[0].alpha3
+                            known_embedded.append(lang_code)
+    else:
+        Log.Warn("Part %s missing of %s, not able to scan internal streams", plex_part.id, rating_key)
 
     try:
-        return subliminal.video.scan_video(plex_video.file, subtitles=external_subtitles, embedded_subtitles=embedded_subtitles,
-                                           hints=hints or {}, video_fps=plex_video.fps)
+        return subliminal.video.scan_video(plex_part.file, subtitles=external_subtitles,
+                                           embedded_subtitles=embedded_subtitles, hints=hints or {},
+                                           video_fps=plex_part.fps, forced_tag=config.forced_only,
+                                           known_embedded_subtitle_streams=known_embedded)
 
     except ValueError:
         Log.Warn("File could not be guessed by subliminal")
 
 
-def scan_parts(parts, kind="series"):
+def scan_videos(videos, kind="series", ignore_all=False):
     """
-    receives a list of parts containing dictionaries returned by flattenToParts
-    :param parts:
+    receives a list of videos containing dictionaries returned by media_to_videos
+    :param videos:
     :param kind: series or movies
     :return: dictionary of subliminal.video.scan_video, key=subliminal scanned video, value=plex file part
     """
     ret = {}
-    for part in parts:
-        force_refresh = intent.get("force", part["id"], part["series_id"], part["season_id"])
+    for video in videos:
+        intent = get_intent()
+        force_refresh = intent.get("force", video["id"], video["series_id"], video["season_id"])
+        Log.Debug("Determining force-refresh (video: %s, series: %s, season: %s), result: %s"
+                  % (video["id"], video["series_id"], video["season_id"], force_refresh))
+
+        hints = helpers.get_item_hints(video["title"], kind, series=video["series"] if kind == "series" else None)
+        video["plex_part"].fps = get_stream_fps(video["plex_part"].streams)
+        scanned_video = scan_video(video["plex_part"], ignore_all=force_refresh or ignore_all, hints=hints,
+                                   rating_key=video["id"])
 
-        hints = helpers.get_item_hints(part["title"], kind, series=part["series"] if kind == "series" else None)
-        part["video"].fps = get_stream_fps(part["video"].streams)
-        scanned_video = scan_video(part["video"], ignore_all=force_refresh, hints=hints)
         if not scanned_video:
             continue
 
-        scanned_video.id = part["id"]
-        part_metadata = part.copy()
-        del part_metadata["video"]
+        scanned_video.id = video["id"]
+        part_metadata = video.copy()
+        del part_metadata["plex_part"]
         scanned_video.plexapi_metadata = part_metadata
-        ret[scanned_video] = part["video"]
-    return ret
+        ret[scanned_video] = video["plex_part"]
+    return ret
+
+
+class PartUnknownException(Exception):
+    pass
+
+
+def get_plex_metadata(rating_key, part_id, item_type):
+    """
+    uses the Plex 3rd party API accessor to get metadata information
+
+    :param rating_key:
+    :param part_id:
+    :param item_type:
+    :return:
+    """
+
+    plex_item = list(Plex["library"].metadata(rating_key))[0]
+
+    # find current part
+    current_part = None
+    for part in plex_item.media.parts:
+        if str(part.id) == part_id:
+            current_part = part
+
+    if not current_part:
+        raise PartUnknownException("Part unknown")
+
+    # get normalized metadata
+    if item_type == "episode":
+        metadata = get_metadata_dict(plex_item, current_part,
+                                     {"plex_part": current_part, "type": "episode", "title": plex_item.title,
+                                      "series": plex_item.show.title, "id": plex_item.rating_key,
+                                      "series_id": plex_item.show.rating_key,
+                                      "season_id": plex_item.season.rating_key,
+                                      "season": plex_item.season.index,
+                                      "episode": plex_item.index
+                                      })
+    else:
+        metadata = get_metadata_dict(plex_item, current_part, {"plex_part": current_part, "type": "movie",
+                                                               "title": plex_item.title, "id": plex_item.rating_key,
+                                                               "series_id": None,
+                                                               "season_id": None,
+                                                               "season": None,
+                                                               "episode": None,
+                                                               "section": plex_item.section.title})
+    return metadata
+
+
+class PMSMediaProxy(object):
+    """
+    Proxy object for getting data from a mediatree items "internally" via the PMS
+
+    note: this could be useful later on: Media.TV_Show(getattr(Metadata, "_access_point"), id=XXXXXX)
+    """
+
+    def __init__(self, media_id):
+        self.mediatree = Media.TreeForDatabaseID(media_id)
+
+    def get_part(self, part_id=None):
+        """
+        walk the mediatree until the given part was found; if no part was given, return the first one
+        :param part_id:
+        :return:
+        """
+        m = self.mediatree
+        while 1:
+            if m.items:
+                media_item = m.items[0]
+                if not part_id:
+                    return media_item.parts[0] if media_item.parts else None
+
+                for part in media_item.parts:
+                    if str(part.id) == str(part_id):
+                        return part
+                break
+
+            if not m.children:
+                break
+
+            m = m.children[0]

Contents/Code/support/storage.py

@@ -1,78 +1,97 @@
 # coding=utf-8
 
 import datetime
+import os
 import pprint
+import copy
+
+import subliminal
+from items import get_item
+from subzero.subtitle_storage import StoredSubtitlesManager
+
+from subtitlehelpers import force_utf8
+from config import config
+from helpers import notify_executable, get_title_for_video_metadata, cast_bool, force_unicode
+from plex_media import PMSMediaProxy
 
 
-def get_subtitle_info(rating_key):
-    return Dict["subs"].get(rating_key)
+get_subtitle_storage = lambda: StoredSubtitlesManager(Data, get_item)
 
 
-def whack_missing_parts(videos, existing_parts=None):
+def whack_missing_parts(scanned_video_part_map, existing_parts=None):
     """
     cleans out our internal storage's video parts (parts may get updated/deleted/whatever)
     :param existing_parts: optional list of part ids known
-    :param videos: videos to check for
+    :param scanned_video_part_map: videos to check for
     :return:
     """
     # shortcut
 
+    if "subs" not in Dict:
+        return
+
     if not existing_parts:
         existing_parts = []
-        for part in videos.viewvalues():
-            existing_parts.append(part.id)
+        for part in scanned_video_part_map.viewvalues():
+            existing_parts.append(str(part.id))
 
     whacked_parts = False
-    for video in videos.keys():
-        if video.id not in Dict["subs"]:
+    for video in scanned_video_part_map.keys():
+        video_id = str(video.id)
+        if video_id not in Dict["subs"]:
             continue
 
-        for part_id in Dict["subs"][video.id].keys():
+        parts = Dict["subs"][video_id].keys()
+
+        for part_id in parts:
+            part_id = str(part_id)
             if part_id not in existing_parts:
-                del Dict["subs"][video.id][part_id]
-                Log.Info("Whacking part %s in internal storage of video %s", part_id, video.id)
+                Log.Info("Whacking part %s in internal storage of video %s (%s, %s)", part_id, video_id,
+                         repr(existing_parts), repr(parts))
+                del Dict["subs"][video_id][part_id]
                 whacked_parts = True
 
     if whacked_parts:
         Dict.Save()
 
 
-def store_subtitle_info(videos, subtitles, storage_type):
+def store_subtitle_info(scanned_video_part_map, downloaded_subtitles, storage_type, mode="a"):
     """
     stores information about downloaded subtitles in plex's Dict()
     """
-    if "subs" not in Dict:
-        Dict["subs"] = {}
-
-    storage = Dict["subs"]
-
     existing_parts = []
-    for video, video_subtitles in subtitles.items():
-        part = videos[video]
+    for video, video_subtitles in downloaded_subtitles.items():
+        part = scanned_video_part_map[video]
+        part_id = str(part.id)
+        video_id = str(video.id)
+        plex_item = get_item(video_id)
+        metadata = video.plexapi_metadata
+        title = get_title_for_video_metadata(metadata)
 
-        if video.id not in storage:
-            storage[video.id] = {}
+        subtitle_storage = get_subtitle_storage()
+        stored_subs = subtitle_storage.load_or_new(plex_item)
 
-        video_dict = storage[video.id]
-        if part.id not in video_dict:
-            video_dict[part.id] = {}
+        existing_parts.append(part_id)
 
-        existing_parts.append(part.id)
-
-        part_dict = video_dict[part.id]
+        stored_any = False
         for subtitle in video_subtitles:
             lang = Locale.Language.Match(subtitle.language.alpha2)
-            if lang not in part_dict:
-                part_dict[lang] = {}
-            lang_dict = part_dict[lang]
-            sub_key = (subtitle.provider_name, subtitle.id)
-            lang_dict[sub_key] = dict(score=subtitle.score, link=subtitle.page_link, storage=storage_type, hash=Hash.MD5(subtitle.content),
-                                      date_added=datetime.datetime.now())
-            lang_dict["current"] = sub_key
+            Log.Debug(u"Adding subtitle to storage: %s, %s, %s" % (video_id, part_id, title))
+            ret_val = stored_subs.add(part_id, lang, subtitle, storage_type, mode=mode)
 
-    if existing_parts:
-        whack_missing_parts(videos, existing_parts=existing_parts)
-    Dict.Save()
+            if ret_val:
+                Log.Debug("Subtitle stored")
+                stored_any = True
+
+            else:
+                Log.Debug("Subtitle already existing in storage")
+
+        if stored_any:
+            Log.Debug("Saving subtitle storage for %s" % video_id)
+            subtitle_storage.save(stored_subs)
+
+    #if existing_parts:
+    #    whack_missing_parts(scanned_video_part_map, existing_parts=existing_parts)
 
 
 def reset_storage(key):
@@ -90,3 +109,78 @@ def reset_storage(key):
 def log_storage(key):
     if key in Dict:
         Log.Debug(pprint.pformat(Dict[key]))
+
+
+def save_subtitles_to_file(subtitles):
+    fld_custom = Prefs["subtitles.save.subFolder.Custom"].strip() \
+        if Prefs["subtitles.save.subFolder.Custom"] else None
+
+    for video, video_subtitles in subtitles.items():
+        if not video_subtitles:
+            continue
+
+        fld = None
+        if fld_custom or Prefs["subtitles.save.subFolder"] != "current folder":
+            # specific subFolder requested, create it if it doesn't exist
+            fld_base = os.path.split(video.name)[0]
+            if fld_custom:
+                if fld_custom.startswith("/"):
+                    # absolute folder
+                    fld = fld_custom
+                else:
+                    fld = os.path.join(fld_base, fld_custom)
+            else:
+                fld = os.path.join(fld_base, Prefs["subtitles.save.subFolder"])
+            fld = force_unicode(fld)
+            if not os.path.exists(fld):
+                os.makedirs(fld)
+        subliminal.api.save_subtitles(video, video_subtitles, directory=fld, single=cast_bool(Prefs['subtitles.only_one']),
+                                      encode_with=force_utf8 if config.enforce_encoding else None,
+                                      chmod=config.chmod, forced_tag=config.forced_only, path_decoder=force_unicode)
+    return True
+
+
+def save_subtitles_to_metadata(videos, subtitles):
+    for video, video_subtitles in subtitles.items():
+        mediaPart = videos[video]
+        for subtitle in video_subtitles:
+            content = force_utf8(subtitle.text) if config.enforce_encoding else subtitle.content
+
+            if not isinstance(mediaPart, Framework.api.agentkit.MediaPart):
+                # we're being handed a Plex.py model instance here, not an internal PMS MediaPart object.
+                # get the correct one
+                mp = PMSMediaProxy(video.id).get_part(mediaPart.id)
+            else:
+                mp = mediaPart
+            mp.subtitles[Locale.Language.Match(subtitle.language.alpha2)][subtitle.id] = Proxy.Media(content, ext="srt")
+    return True
+
+
+def save_subtitles(scanned_video_part_map, downloaded_subtitles, mode="a"):
+    meta_fallback = False
+    save_successful = False
+    storage = "metadata"
+    if Prefs['subtitles.save.filesystem']:
+        storage = "filesystem"
+        try:
+            Log.Debug("Using filesystem as subtitle storage")
+            save_subtitles_to_file(downloaded_subtitles)
+        except OSError:
+            if Prefs["subtitles.save.metadata_fallback"]:
+                meta_fallback = True
+            else:
+                raise
+        else:
+            save_successful = True
+
+    if not Prefs['subtitles.save.filesystem'] or meta_fallback:
+        if meta_fallback:
+            Log.Debug("Using metadata as subtitle storage, because filesystem storage failed")
+        else:
+            Log.Debug("Using metadata as subtitle storage")
+        save_successful = save_subtitles_to_metadata(scanned_video_part_map, downloaded_subtitles)
+
+    if save_successful and config.notify_executable:
+        notify_executable(config.notify_executable, scanned_video_part_map, downloaded_subtitles, storage)
+
+    store_subtitle_info(scanned_video_part_map, downloaded_subtitles, storage, mode=mode)

Contents/Code/support/subtitlehelpers.py

@@ -14,7 +14,12 @@ class SubtitleHelper(object):
 
 def subtitle_helpers(filename):
     filename = helpers.unicodize(filename)
-    for cls in [VobSubSubtitleHelper, DefaultSubtitleHelper]:
+    helper_classes = [DefaultSubtitleHelper]
+
+    if helpers.cast_bool(Prefs["subtitles.scan.exotic_ext"]):
+        helper_classes.insert(0, VobSubSubtitleHelper)
+
+    for cls in helper_classes:
         if cls.is_helper_for(filename):
             return cls(filename)
     return None
@@ -79,6 +84,20 @@ class VobSubSubtitleHelper(SubtitleHelper):
 
 #####################################################################################################################
 
+
+IETF_MATCH = ".+\.([^-.]+)(?:-[A-Za-z]+)?$"
+ENDSWITH_LANGUAGECODE_RE = re.compile("\.([^-.]{2,3})(?:-[A-Za-z]{2})?$")
+
+
+def match_ietf_language(s):
+    language_match = re.match(".+\.([^\.]+)$" if not helpers.cast_bool(Prefs["subtitles.language.ietf"])
+                              else IETF_MATCH, s)
+    if language_match and len(language_match.groups()) == 1:
+        language = language_match.groups()[0]
+        return language
+    return s
+
+
 class DefaultSubtitleHelper(SubtitleHelper):
     @classmethod
     def is_helper_for(cls, filename):
@@ -89,20 +108,35 @@ class DefaultSubtitleHelper(SubtitleHelper):
 
         lang_sub_map = {}
 
+        if not os.path.exists(self.filename):
+            return lang_sub_map
+
         basename = os.path.basename(self.filename)
         (file, ext) = os.path.splitext(self.filename)
 
         # Remove the initial '.' from the extension
         ext = ext[1:]
 
+        forced = ''
+        default = ''
+        split_tag = file.rsplit('.', 1)
+        if len(split_tag) > 1 and split_tag[1].lower() in ['forced', 'normal', 'default']:
+            file = split_tag[0]
+            # don't do anything with 'normal', we don't need it
+            if 'forced' == split_tag[1].lower():
+                forced = '1'
+            if 'default' == split_tag[1].lower():
+                default = '1'
+
         # Attempt to extract the language from the filename (e.g. Avatar (2009).eng)
         language = ""
 
         # IETF support thanks to https://github.com/hpsbranco/LocalMedia.bundle/commit/4fad9aefedece78a1fa96401304351347f644369
-        language_match = re.match(".+\.([^\.]+)$" if not Prefs["subtitles.language.ietf"] else ".+\.([^-.]+)(?:-[A-Za-z]+)?$", file)
-        if language_match and len(language_match.groups()) == 1:
-            language = language_match.groups()[0]
-        language = Locale.Language.Match(language)
+        language = Locale.Language.Match(match_ietf_language(file))
+
+        # skip non-SRT if wanted
+        if not helpers.cast_bool(Prefs["subtitles.scan.exotic_ext"]) and ext not in ["srt", "ass", "ssa"]:
+            return lang_sub_map
 
         codec = None
         format = None
@@ -130,8 +164,10 @@ class DefaultSubtitleHelper(SubtitleHelper):
         if format is None:
             format = codec
 
-        Log('Found subtitle file: ' + self.filename + ' language: ' + language + ' codec: ' + str(codec) + ' format: ' + str(format))
-        part.subtitles[language][basename] = Proxy.LocalFile(self.filename, codec=codec, format=format)
+        Log('Found subtitle file: ' + self.filename + ' language: ' + language + ' codec: ' + str(
+            codec) + ' format: ' + str(format) + ' default: ' + default + ' forced: ' + forced)
+        part.subtitles[language][basename] = Proxy.LocalFile(self.filename, codec=codec, format=format, default=default,
+                                                             forced=forced)
 
         lang_sub_map[language] = [basename]
         return lang_sub_map

Contents/Code/support/tasks.py

@@ -3,29 +3,50 @@
 import datetime
 import time
 
+import operator
+import traceback
+
+import subliminal
+import subliminal_patch
+
+from subliminal_patch.patch_api import list_all_subtitles, download_subtitles
+from babelfish import Language
+from subliminal_patch.patch_subtitle import compute_score
 from missing_subtitles import items_get_all_missing_subs, refresh_item
 from background import scheduler
-from support.items import get_recent_items, is_ignored
+from storage import save_subtitles, whack_missing_parts, get_subtitle_storage
+from support.config import config
+from support.items import get_recent_items, is_ignored, get_item
+from support.lib import Plex
+from support.helpers import track_usage, get_title_for_video_metadata, cast_bool
+from support.plex_media import scan_videos, get_plex_metadata, PartUnknownException
 
 
 class Task(object):
     name = None
     scheduler = None
+    periodic = False
     running = False
     time_start = None
+    data = None
 
-    stored_attributes = ("last_run", "last_run_time")
+    stored_attributes = ("last_run", "last_run_time", "running")
+    default_data = {"last_run": None, "last_run_time": None, "running": False, "data": {}}
 
     # task ready for being status-displayed?
     ready_for_display = False
 
     def __init__(self, scheduler):
+        self.name = self.get_class_name()
         self.ready_for_display = False
-        self.running = False
         self.time_start = None
         self.scheduler = scheduler
-        if self.name not in Dict["tasks"]:
-            Dict["tasks"][self.name] = {"last_run": None, "last_run_time": None}
+        self.setup_defaults()
+
+        self.running = False
+
+    def get_class_name(self):
+        return getattr(getattr(self, "__class__"), "__name__")
 
     def __getattribute__(self, name):
         if name in object.__getattribute__(self, "stored_attributes"):
@@ -41,18 +62,38 @@ class Task(object):
 
         object.__setattr__(self, name, value)
 
+    def setup_defaults(self):
+        if self.name not in Dict["tasks"]:
+            Dict["tasks"][self.name] = self.default_data.copy()
+            return
+
+        sd = Dict["tasks"][self.name]
+
+        # forward-migration
+        for key, def_value in self.default_data.iteritems():
+            hasval = key in sd
+            if not hasval:
+                sd[key] = def_value
+
     def signal(self, *args, **kwargs):
         raise NotImplementedError
 
-    def prepare(self):
-        raise NotImplementedError
+    def prepare(self, *args, **kwargs):
+        return
 
     def run(self):
-        raise NotImplementedError
+        self.time_start = datetime.datetime.now()
+
+    def post_run(self, data_holder):
+        self.running = False
+        self.last_run = datetime.datetime.now()
+        if self.time_start:
+            self.last_run_time = self.last_run - self.time_start
+        self.time_start = None
 
 
 class SearchAllRecentlyAddedMissing(Task):
-    name = "searchAllRecentlyAddedMissing"
+    periodic = True
     items_done = None
     items_searching = None
     items_searching_ids = None
@@ -80,26 +121,26 @@ class SearchAllRecentlyAddedMissing(Task):
             self.items_done.append(item_id)
             return True
 
-    def prepare(self):
+    def prepare(self, *args, **kwargs):
         self.items_done = []
         recent_items = get_recent_items()
         missing = items_get_all_missing_subs(recent_items)
-        ids = set([id for added_at, id, title, item in missing if not is_ignored(id, item=item)])
+        ids = set([id for added_at, id, title, item, missing_languages in missing if not is_ignored(id, item=item)])
         self.items_searching = missing
         self.items_searching_ids = ids
         self.items_failed = []
         self.percentage = 0
-        self.time_start = datetime.datetime.now()
         self.ready_for_display = True
 
     def run(self):
+        super(SearchAllRecentlyAddedMissing, self).run()
         self.running = True
         missing_count = len(self.items_searching)
         items_done_count = 0
 
-        for added_at, item_id, title, item in self.items_searching:
+        for added_at, item_id, title, item, missing_languages in self.items_searching:
             Log.Debug(u"Task: %s, triggering refresh for %s (%s)", self.name, title, item_id)
-            refresh_item(item_id, title)
+            refresh_item(item_id)
             search_started = datetime.datetime.now()
             tries = 1
             while 1:
@@ -116,9 +157,10 @@ class SearchAllRecentlyAddedMissing(Task):
                         Log.Debug(u"Task: %s, item stalled for %s times: %s, skipping", self.name, tries, item_id)
                         break
 
-                    Log.Debug(u"Task: %s, item stalled for %s seconds: %s, retrying", self.name, self.stall_time, item_id)
+                    Log.Debug(u"Task: %s, item stalled for %s seconds: %s, retrying", self.name, self.stall_time,
+                              item_id)
                     tries += 1
-                    refresh_item(item_id, title)
+                    refresh_item(item_id)
                     search_started = datetime.datetime.now()
                     time.sleep(1)
                 time.sleep(0.1)
@@ -128,12 +170,9 @@ class SearchAllRecentlyAddedMissing(Task):
         Log.Debug("Task: %s, done. Failed items: %s", self.name, self.items_failed)
         self.running = False
 
-    def post_run(self):
+    def post_run(self, task_data):
+        super(SearchAllRecentlyAddedMissing, self).post_run(task_data)
         self.ready_for_display = False
-        self.last_run = datetime.datetime.now()
-        if self.time_start:
-            self.last_run_time = self.last_run - self.time_start
-        self.time_start = None
         self.percentage = 0
         self.items_done = None
         self.items_failed = None
@@ -141,4 +180,275 @@ class SearchAllRecentlyAddedMissing(Task):
         self.items_searching_ids = None
 
 
+class SubtitleListingMixin(object):
+    def list_subtitles(self, rating_key, item_type, part_id, language):
+        metadata = get_plex_metadata(rating_key, part_id, item_type)
+
+        if item_type == "episode":
+            min_score = 66
+        else:
+            min_score = 23
+
+        scanned_parts = scan_videos([metadata], kind="series" if item_type == "episode" else "movie", ignore_all=True)
+        if not scanned_parts:
+            Log.Error("Couldn't list available subtitles for %s", rating_key)
+            return
+
+        video, plex_part = scanned_parts.items()[0]
+        config.init_subliminal_patches()
+
+        available_subs = list_all_subtitles(scanned_parts, {Language.fromietf(language)},
+                                            providers=config.providers,
+                                            provider_configs=config.provider_settings)
+
+        use_hearing_impaired = Prefs['subtitles.search.hearingImpaired'] in ("prefer", "force HI")
+
+        # sort subtitles by score
+        unsorted_subtitles = []
+        for s in available_subs[video]:
+            Log.Debug("Starting score computation for %s", s)
+            try:
+                matches = s.get_matches(video, hearing_impaired=use_hearing_impaired)
+            except AttributeError:
+                Log.Error("Match computation failed for %s: %s", s, traceback.format_exc())
+                continue
+
+            unsorted_subtitles.append((s, compute_score(matches, video), matches))
+        scored_subtitles = sorted(unsorted_subtitles, key=operator.itemgetter(1), reverse=True)
+
+        subtitles = []
+        for subtitle, score, matches in scored_subtitles:
+            # check score
+            if score < min_score:
+                Log.Info('Score %d is below min_score (%d)', score, min_score)
+                continue
+            subtitle.score = score
+            subtitle.matches = matches
+            subtitle.part_id = part_id
+            subtitle.item_type = item_type
+            subtitles.append(subtitle)
+        return subtitles
+
+
+class DownloadSubtitleMixin(object):
+    def download_subtitle(self, subtitle, rating_key, mode="m"):
+        from interface.menu_helpers import set_refresh_menu_state
+
+        item_type = subtitle.item_type
+        part_id = subtitle.part_id
+        metadata = get_plex_metadata(rating_key, part_id, item_type)
+        scanned_parts = scan_videos([metadata], kind="series" if item_type == "episode" else "movie", ignore_all=True)
+        video, plex_part = scanned_parts.items()[0]
+
+        # downloaded_subtitles = {subliminal.Video: [subtitle, subtitle, ...]}
+        download_subtitles([subtitle], providers=config.providers, provider_configs=config.provider_settings)
+        download_successful = False
+
+        if subtitle.content:
+            try:
+                whack_missing_parts(scanned_parts)
+                save_subtitles(scanned_parts, {video: [subtitle]}, mode=mode)
+                Log.Debug("Manually downloaded subtitle for: %s", rating_key)
+                download_successful = True
+                refresh_item(rating_key)
+                track_usage("Subtitle", "manual", "download", 1)
+            except:
+                Log.Error("Something went wrong when downloading specific subtitle: %s", traceback.format_exc())
+            finally:
+                set_refresh_menu_state(None)
+
+                if download_successful:
+                    # store item in history
+                    from support.history import get_history
+                    item_title = get_title_for_video_metadata(metadata, add_section_title=False)
+                    history = get_history()
+                    history.add(item_title, video.id, section_title=video.plexapi_metadata["section"], subtitle=subtitle,
+                                mode=mode)
+        return download_successful
+
+
+class AvailableSubsForItem(SubtitleListingMixin, Task):
+    item_type = None
+    part_id = None
+    language = None
+    rating_key = None
+
+    def prepare(self, *args, **kwargs):
+        self.item_type = kwargs.get("item_type")
+        self.part_id = kwargs.get("part_id")
+        self.language = kwargs.get("language")
+        self.rating_key = kwargs.get("rating_key")
+
+    def setup_defaults(self):
+        super(AvailableSubsForItem, self).setup_defaults()
+
+        # reset any previous data
+        Dict["tasks"][self.name]["data"] = {}
+
+    def run(self):
+        super(AvailableSubsForItem, self).run()
+        self.running = True
+        track_usage("Subtitle", "manual", "list", 1)
+        self.data = self.list_subtitles(self.rating_key, self.item_type, self.part_id, self.language)
+
+    def post_run(self, task_data):
+        super(AvailableSubsForItem, self).post_run(task_data)
+        if self.rating_key not in task_data:
+            task_data[self.rating_key] = {}
+
+        task_data[self.rating_key][self.language] = self.data
+
+
+class DownloadSubtitleForItem(DownloadSubtitleMixin, Task):
+    subtitle = None
+    rating_key = None
+
+    def prepare(self, *args, **kwargs):
+        self.subtitle = kwargs["subtitle"]
+        self.rating_key = kwargs["rating_key"]
+
+    def run(self):
+        super(DownloadSubtitleForItem, self).run()
+        self.running = True
+        self.download_subtitle(self.subtitle, self.rating_key)
+        self.running = False
+
+
+class MissingSubtitles(Task):
+    rating_key = None
+    item_type = None
+    part_id = None
+    language = None
+
+    def run(self):
+        super(MissingSubtitles, self).run()
+        self.running = True
+        self.data = []
+        recent_items = get_recent_items()
+        if recent_items:
+            self.data = items_get_all_missing_subs(recent_items)
+
+    def post_run(self, task_data):
+        super(MissingSubtitles, self).post_run(task_data)
+        task_data["missing_subtitles"] = self.data
+
+
+class FindBetterSubtitles(DownloadSubtitleMixin, SubtitleListingMixin, Task):
+    periodic = True
+
+    # TV: episode, format, series, year, season, video_codec, release_group, hearing_impaired
+    series_cutoff = 132
+
+    # movies: format, title, release_group, year, video_codec, resolution, hearing_impaired
+    movies_cutoff = 61
+
+    def signal_updated_metadata(self, *args, **kwargs):
+        return True
+
+    def run(self):
+        super(FindBetterSubtitles, self).run()
+        self.running = True
+        better_found = 0
+        try:
+            max_search_days = int(Prefs["scheduler.tasks.FindBetterSubtitles.max_days_after_added"].strip())
+        except ValueError:
+            Log.Error("Please only put numbers into the FindBetterSubtitles.max_days_after_added setting. Exiting")
+            return
+        else:
+            if max_search_days > 30:
+                Log.Error("FindBetterSubtitles.max_days_after_added is too big. Max is 30 days.")
+                return
+
+        now = datetime.datetime.now()
+
+        subtitle_storage = get_subtitle_storage()
+        recent_subs = subtitle_storage.load_recent_files(age_days=max_search_days)
+
+        for fn, stored_subs in recent_subs.iteritems():
+            video_id = stored_subs.video_id
+            cutoff = self.series_cutoff if stored_subs.item_type == "episode" else self.movies_cutoff
+
+            # don't search for better subtitles until at least 30 minutes have passed
+            if stored_subs.added_at + datetime.timedelta(minutes=30) > now:
+                Log.Debug("Item %s too new, skipping", video_id)
+                continue
+
+            # added_date <= max_search_days?
+            if stored_subs.added_at + datetime.timedelta(days=max_search_days) <= now:
+                continue
+
+            ditch_parts = []
+
+            # look through all stored subtitle data
+            for part_id, languages in stored_subs.parts.iteritems():
+                part_id = str(part_id)
+
+                # all languages
+                for language, current_subs in languages.iteritems():
+                    current_key = current_subs.get("current")
+                    current = current_subs.get(current_key)
+
+                    # currently got subtitle?
+                    if not current:
+                        continue
+                    current_score = current.score
+                    current_mode = current.mode
+
+                    # late cutoff met? skip
+                    if current_score >= cutoff:
+                        Log.Debug(u"Skipping finding better subs, cutoff met (current: %s, cutoff: %s): %s",
+                                  current_score, cutoff, stored_subs.title)
+                        continue
+
+                    # got manual subtitle but don't want to touch those?
+                    if current_mode == "m" and \
+                            not cast_bool(Prefs["scheduler.tasks.FindBetterSubtitles.overwrite_manually_selected"]):
+                        Log.Debug(u"Skipping finding better subs, had manual: %s", stored_subs.title)
+                        continue
+
+                    try:
+                        subs = self.list_subtitles(video_id, stored_subs.item_type, part_id, language)
+                    except PartUnknownException:
+                        Log.Info("Part %s unknown/gone; ditching subtitle info", part_id)
+                        ditch_parts.append(part_id)
+                        continue
+
+                    if subs:
+                        # subs are already sorted by score
+                        better_downloaded = False
+                        better_tried_download = 0
+                        for sub in subs:
+                            if sub.score > current_score:
+                                Log.Debug("Better subtitle found for %s, downloading", video_id)
+                                better_tried_download += 1
+                                ret = self.download_subtitle(sub, video_id, mode="b")
+                                if ret:
+                                    better_found += 1
+                                    better_downloaded = True
+                                    break
+                                else:
+                                    Log.Debug("Couldn't download/save subtitle. Continuing to the next one")
+                        if better_tried_download and not better_downloaded:
+                            Log.Debug("Tried downloading better subtitle for %s, but every try failed.", video_id)
+
+                        elif better_downloaded:
+                            Log.Debug("Better subtitle downloaded for %s", video_id)
+
+            if ditch_parts:
+                for part_id in ditch_parts:
+                    try:
+                        del stored_subs.parts[part_id]
+                    except KeyError:
+                        pass
+                subtitle_storage.save(stored_subs)
+
+        if better_found:
+            Log.Debug("Task: %s, done. Better subtitles found for %s items", self.name, better_found)
+        self.running = False
+
+
 scheduler.register(SearchAllRecentlyAddedMissing)
+scheduler.register(AvailableSubsForItem)
+scheduler.register(DownloadSubtitleForItem)
+scheduler.register(MissingSubtitles)
+scheduler.register(FindBetterSubtitles)

Contents/DefaultPrefs.json

@@ -1,56 +1,4 @@
 [
-  {
-    "id": "enable_channel",
-    "label": "Enable Sub-Zero channel (disabling doesn't affect the subtitle features)?",
-    "type": "bool",
-    "default": "true"
-  },
-  {
-    "id": "subtitles.try_downloads",
-    "label": "How many download tries per subtitle (on timeout or error)",
-    "type": "enum",
-    "values": [
-      "1",
-      "2",
-      "3",
-      "4"
-    ],
-    "default": "2"
-  },
-  {
-    "id": "provider.addic7ed.username",
-    "label": "Addic7ed Username",
-    "type": "text",
-    "default": ""
-  },
-  {
-    "id": "provider.addic7ed.password",
-    "label": "Addic7ed Password",
-    "type": "text",
-    "option": "hidden",
-    "default": "",
-    "secure": "true"
-  },
-  {
-    "id": "provider.opensubtitles.username",
-    "label": "Opensubtitles Username (VIP)",
-    "type": "text",
-    "default": ""
-  },
-  {
-    "id": "provider.opensubtitles.password",
-    "label": "Opensubtitles Password",
-    "type": "text",
-    "option": "hidden",
-    "default": "",
-    "secure": "true"
-  },
-  {
-    "id": "provider.addic7ed.use_random_agents",
-    "label": "Addic7ed: Use random user agents (should not be necessary)",
-    "type": "bool",
-    "default": "false"
-  },
   {
     "id": "langPref1",
     "label": "Subtitle Language (1)",
@@ -218,6 +166,18 @@
     "type": "text",
     "default": "None"
   },
+  {
+    "id": "subtitles.only_foreign",
+    "label": "Only download foreign/forced subtitles",
+    "type": "bool",
+    "default": "false"
+  },
+  {
+    "id": "subtitles.language.ietf",
+    "label": "Treat IETF language tags as ISO 639-1 (e.g. pt-BR = pt)",
+    "type": "bool",
+    "default": "true"
+  },
   {
     "id": "subtitles.only_one",
     "label": "Restrict to one language (skips adding \".lang.\" to the subtitle filename; only uses \"Subtitle Language (1)\")",
@@ -225,8 +185,8 @@
     "default": "false"
   },
   {
-    "id": "subtitles.enforce_encoding",
-    "label": "Normalize subtitle encoding to UTF-8",
+    "id": "subtitles.language.treat_und_as_first",
+    "label": "Embedded subtitles: Treat \"Undefined\" (und) as language 1",
     "type": "bool",
     "default": "true"
   },
@@ -237,10 +197,18 @@
     "default": "true"
   },
   {
-    "id": "provider.thesubdb.enabled",
-    "label": "Provider: Enable TheSubDB",
-    "type": "bool",
-    "default": "true"
+    "id": "provider.opensubtitles.username",
+    "label": "Opensubtitles Username (VIP)",
+    "type": "text",
+    "default": ""
+  },
+  {
+    "id": "provider.opensubtitles.password",
+    "label": "Opensubtitles Password",
+    "type": "text",
+    "option": "hidden",
+    "default": "",
+    "secure": "true"
   },
   {
     "id": "provider.podnapisi.enabled",
@@ -255,8 +223,52 @@
     "default": "true"
   },
   {
-    "id": "provider.addic7ed.boost",
-    "label": "Addic7ed: prefer over other providers (if requirements met)",
+    "id": "provider.addic7ed.username",
+    "label": "Addic7ed Username",
+    "type": "text",
+    "default": ""
+  },
+  {
+    "id": "provider.addic7ed.password",
+    "label": "Addic7ed Password",
+    "type": "text",
+    "option": "hidden",
+    "default": "",
+    "secure": "true"
+  },
+  {
+    "id": "provider.addic7ed.boost_by",
+    "label": "Addic7ed: boost score (if requirements met)",
+    "type": "enum",
+    "values": [
+      "100",
+      "95",
+      "90",
+      "85",
+      "80",
+      "75",
+      "70",
+      "67",
+      "65",
+      "60",
+      "55",
+      "50",
+      "45",
+      "40",
+      "35",
+      "30",
+      "25",
+      "20",
+      "15",
+      "10",
+      "5",
+      "0"
+    ],
+    "default": "10"
+  },
+  {
+    "id": "provider.addic7ed.use_random_agents",
+    "label": "Addic7ed: Use random user agents",
     "type": "bool",
     "default": "false"
   },
@@ -285,64 +297,33 @@
     "default": "true"
   },
   {
-    "id": "subtitles.search.minimumTVScore",
-    "label": "Minimum score for TV subtitles to download",
-    "type": "enum",
-    "values": [
-      "100",
-      "95",
-      "90",
-      "85",
-      "80",
-      "75",
-      "70",
-      "67",
-      "65",
-      "60",
-      "55",
-      "50",
-      "45",
-      "40",
-      "35",
-      "30",
-      "25",
-      "20",
-      "15",
-      "10",
-      "5",
-      "0"
-    ],
-    "default": "85"
+    "id": "subtitles.scan.exotic_ext",
+    "label": "Scan: include \"exotic\" external subtitle formats (anything else than .srt/.ssa/.ass)",
+    "type": "bool",
+    "default": "false"
   },
   {
-    "id": "subtitles.search.minimumMovieScore",
-    "label": "Minimum score for movie subtitles to download",
+    "id": "subtitles.scan.filename_strictness",
+    "label": "Scan: which external subtitles should be picked up?",
     "type": "enum",
     "values": [
-      "100",
-      "95",
-      "90",
-      "85",
-      "80",
-      "75",
-      "70",
-      "65",
-      "60",
-      "55",
-      "50",
-      "45",
-      "40",
-      "35",
-      "30",
-      "25",
-      "23",
-      "20",
-      "15",
-      "10",
-      "5",
-      "0"
+      "exact: media filename match",
+      "loose: filename contains media filename",
+      "any"
     ],
-    "default": "23"
+    "default": "loose: filename contains media filename"
+  },
+  {
+    "id": "subtitles.search.minimumTVScore1",
+    "label": "Minimum score for TV (min: 77, sane: 110; min-ideal: 116; see http://v.ht/szscores)",
+    "type": "text",
+    "default": "116"
+  },
+  {
+    "id": "subtitles.search.minimumMovieScore1",
+    "label": "Minimum score for movies (min: 23, def/sane/min-ideal: 33; see http://v.ht/szscores)",
+    "type": "text",
+    "default": "33"
   },
   {
     "id": "subtitles.search.hearingImpaired",
@@ -356,6 +337,12 @@
     ],
     "default": "don't prefer"
   },
+  {
+    "id": "subtitles.enforce_encoding",
+    "label": "Normalize subtitle encoding to UTF-8",
+    "type": "bool",
+    "default": "true"
+  },
   {
     "id": "subtitles.save.filesystem",
     "label": "Store subtitles next to media files (instead of metadata)",
@@ -388,31 +375,31 @@
     "default": "false"
   },
   {
-    "id": "subtitles.language.ietf",
-    "label": "Treat IETF language tags as ISO 639-1 (e.g. pt-BR = pt)",
+    "id": "subtitles.save.chmod",
+      "label": "Set subtitle file permissions to (integer, e.g.: 0775)",
+      "type": "text",
+      "default": ""
+  },
+  {
+    "id": "subtitles.autoclean",
+    "label": "Automatically delete leftover/unused (externally saved) subtitles",
     "type": "bool",
     "default": "true"
   },
   {
-    "id": "subtitles.ignore_fs",
-    "label": "Ignore folders (with \"subzero.ignore/.subzero.ignore/.nosz\" files in them)",
-    "type": "bool",
-    "default": "false"
+    "id": "activity.on_playback",
+    "label": "On media playback: search for missing subtitles (refresh item)",
+    "type": "enum",
+    "values": [
+      "never",
+      "current media item",
+      "next episode (series)",
+      "hybrid: current item or next episode"
+    ],
+    "default": "never"
   },
   {
-      "id": "subtitles.ignore_paths",
-      "label": "Ignore anything in the following paths (comma-separated)",
-      "type": "text",
-      "default": ""
-  },
-  {
-      "id": "notify_executable",
-      "label": "Call this executable upon successful subtitle download",
-      "type": "text",
-      "default": ""
-  },
-  {
-    "id": "scheduler.tasks.searchAllRecentlyAddedMissing",
+    "id": "scheduler.tasks.SearchAllRecentlyAddedMissing.frequency",
     "label": "Scheduler: Periodically search for recent items with missing subtitles",
     "type": "enum",
     "values": [
@@ -447,7 +434,110 @@
     "id": "scheduler.max_recent_items_per_library",
     "label": "Scheduler: Recent items to consider per library",
     "type": "text",
-    "default": "200"
+    "default": "500"
+  },
+  {
+    "id": "scheduler.tasks.FindBetterSubtitles.frequency",
+    "label": "Scheduler: Periodically search for better subtitles",
+    "type": "enum",
+    "values": [
+      "never",
+      "every 6 hours",
+      "every 12 hours",
+      "every 24 hours"
+    ],
+    "default": "every 12 hours"
+  },
+  {
+    "id": "scheduler.tasks.FindBetterSubtitles.max_days_after_added",
+    "label": "Scheduler: Days to search for better subtitles (max: 30 days)",
+    "type": "text",
+    "default": "7"
+  },
+  {
+    "id": "scheduler.tasks.FindBetterSubtitles.overwrite_manually_selected",
+    "label": "Scheduler: Overwrite manually selected subtitles when better found",
+    "type": "bool",
+    "default": "true"
+  },
+  {
+    "id": "history_size",
+    "label": "History: amount of items to store historical data for",
+    "type": "enum",
+    "values": [
+      "50",
+      "100",
+      "150",
+      "250",
+      "500"
+    ],
+    "default": "100"
+  },
+  {
+    "id": "subtitles.try_downloads",
+    "label": "How many download tries per subtitle (on timeout or error)",
+    "type": "enum",
+    "values": [
+      "1",
+      "2",
+      "3",
+      "4"
+    ],
+    "default": "2"
+  },
+  {
+    "id": "subtitles.ignore_fs",
+    "label": "Ignore folders (with \"subzero.ignore/.subzero.ignore/.nosz\" files in them)",
+    "type": "bool",
+    "default": "false"
+  },
+  {
+    "id": "subtitles.ignore_paths",
+    "label": "Ignore anything in the following paths (comma-separated)",
+    "type": "text",
+    "default": ""
+  },
+  {
+    "id": "plugin_mode",
+    "label": "Sub-Zero mode",
+    "type": "enum",
+    "values": [
+      "agent + channel",
+      "only agent",
+      "only channel"
+    ],
+    "default": "agent + channel"
+  },
+  {
+    "id": "plugin_pin",
+    "label": "Access PIN (any amount of numbers, 0-9)",
+    "type": "text",
+    "option": "hidden",
+    "default": "",
+    "secure": "true"
+  },
+  {
+    "id": "plugin_pin_valid_for",
+    "label": "Access PIN valid for minutes",
+    "type": "text",
+    "default": "10"
+  },
+  {
+    "id": "plugin_pin_mode",
+    "label": "Use PIN to restrict access to (needs plugin or PMS restart)",
+    "type": "enum",
+    "values": [
+      "disabled",
+      "channel menu",
+      "advanced menu"
+    ],
+    "default": "disabled"
+  },
+  {
+    "id": "notify_executable",
+    "label": "Call this executable upon successful subtitle download",
+    "type": "text",
+    "default": ""
   },
   {
     "id": "check_permissions",
@@ -473,5 +563,11 @@
     "label": "Log to console (for development/debugging)",
     "type": "bool",
     "default": "false"
+  },
+  {
+    "id": "track_usage",
+    "label": "Collect anonymous usage statistics",
+    "type": "bool",
+    "default": "true"
   }
 ]

Contents/Info.plist

@@ -9,11 +9,11 @@
         <key>CFBundleInfoDictionaryVersion</key>
         <string>6.0</string>
         <key>CFBundleShortVersionString</key>
-        <string>1.3.31</string>
+        <string>1.4.27</string>
         <key>CFBundleSignature</key>
         <string>????</string>
         <key>CFBundleVersion</key>
-        <string>1.3.33.522</string>
+        <string>1.4.27.974</string>
         <key>PlexFrameworkVersion</key>
         <string>2</string>
         <key>PlexPluginClass</key>
@@ -25,20 +25,22 @@
         <key>PlexPluginDevMode</key>
         <string>0</string>
          <key>PlexPluginCodePolicy</key>
-            <!-- this allows channels to access some python methods which are otherwise blocked, as well as import external code libraries, and interact with the PMS HTTP API -->     
+            <!-- this allows channels to access some python methods which are otherwise blocked, as well as import external code libraries, and interact with the PMS HTTP API -->
             <string>Elevated</string>
 	<key>PlexAgentAttributionText</key>
 	<string>&lt;div style=&quot;white-space: pre;&quot;&gt;&lt;img src=&quot;https://raw.githubusercontent.com/pannal/Sub-Zero.bundle/master/Contents/Resources/subzero.gif&quot; /&gt;
 
 &lt;h1&gt;Sub-Zero for Plex&lt;/h1&gt;&lt;i&gt;Subtitles done right&lt;/i&gt;
 
-Version 1.3.33.522
+Version 1.4.27.974
 
 Originally based on @bramwalet's awesome &lt;a href=&quot;https://github.com/bramwalet/Subliminal.bundle&quot;&gt;Subliminal.bundle&lt;/a&gt;
 
 If you like this, buy me a beer: &lt;a href=&quot;https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&amp;hosted_button_id=G9VKR2B8PMNKG&quot; target=&quot;_blank&quot; title=&quot;donate&quot;&gt;&lt;img src=&quot;https://www.paypalobjects.com/en_US/i/btn/btn_donate_LG.gif&quot; alt=&quot;donate&quot; title=&quot;donate&quot; /&gt;&lt;/a&gt;
 
 &lt;strong&gt;Need help?&lt;/strong&gt;
+Wiki: &lt;a href=&quot;http://v.ht/szwiki&quot;&gt;http://v.ht/szwiki&lt;/a&gt;
+Score info: &lt;a href=&quot;http://v.ht/szscores&quot;&gt;http://v.ht/szscores&lt;/a&gt;
 Plex thread: &lt;a href=&quot;https://forums.plex.tv/discussion/186575&quot;>https://forums.plex.tv/discussion/186575&lt;/a&gt;
 Github: &lt;a href=&quot;https://github.com/pannal/Sub-Zero.bundle&quot;&gt;https://github.com/pannal/Sub-Zero&lt;/a&gt;
 

Contents/Libraries/Shared/asio/__init__.py

@@ -0,0 +1,61 @@
+# Copyright 2013 Dean Gardiner <gardiner91@gmail.com>
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from asio.file import SEEK_ORIGIN_CURRENT
+from asio.file_opener import FileOpener
+from asio.open_parameters import OpenParameters
+from asio.interfaces.posix import PosixInterface
+from asio.interfaces.windows import WindowsInterface
+
+import os
+
+
+class ASIO(object):
+    platform_handler = None
+
+    @classmethod
+    def get_handler(cls):
+        if cls.platform_handler:
+            return cls.platform_handler
+
+        if os.name == 'nt':
+            cls.platform_handler = WindowsInterface
+        elif os.name == 'posix':
+            cls.platform_handler = PosixInterface
+        else:
+            raise NotImplementedError()
+
+        return cls.platform_handler
+
+    @classmethod
+    def open(cls, file_path, opener=True, parameters=None):
+        """Open file
+
+        :type file_path: str
+
+        :param opener: Use FileOpener, for use with the 'with' statement
+        :type opener: bool
+
+        :rtype: asio.file.File
+        """
+        if not parameters:
+            parameters = OpenParameters()
+
+        if opener:
+            return FileOpener(file_path, parameters)
+
+        return ASIO.get_handler().open(
+            file_path,
+            parameters=parameters.handlers.get(ASIO.get_handler())
+        )

Contents/Libraries/Shared/asio/file.py

@@ -0,0 +1,92 @@
+# Copyright 2013 Dean Gardiner <gardiner91@gmail.com>
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from io import RawIOBase
+import time
+
+DEFAULT_BUFFER_SIZE = 4096
+
+SEEK_ORIGIN_BEGIN = 0
+SEEK_ORIGIN_CURRENT = 1
+SEEK_ORIGIN_END = 2
+
+
+class ReadTimeoutError(Exception):
+    pass
+
+
+class File(RawIOBase):
+    platform_handler = None
+
+    def __init__(self, *args, **kwargs):
+        super(File, self).__init__(*args, **kwargs)
+
+    def get_handler(self):
+        """
+        :rtype: asio.interfaces.base.Interface
+        """
+        if not self.platform_handler:
+            raise ValueError()
+
+        return self.platform_handler
+
+    def get_size(self):
+        """Get the current file size
+
+        :rtype: int
+        """
+        return self.get_handler().get_size(self)
+
+    def get_path(self):
+        """Get the path of this file
+
+        :rtype: str
+        """
+        return self.get_handler().get_path(self)
+
+    def seek(self, offset, origin):
+        """Sets a reference point of a file to the given value.
+
+        :param offset: The point relative to origin to move
+        :type offset: int
+
+        :param origin: Reference point to seek (SEEK_ORIGIN_BEGIN, SEEK_ORIGIN_CURRENT, SEEK_ORIGIN_END)
+        :type origin: int
+        """
+        return self.get_handler().seek(self, offset, origin)
+
+    def read(self, n=-1):
+        """Read up to n bytes from the object and return them.
+
+        :type n: int
+        :rtype: str
+        """
+        return self.get_handler().read(self, n)
+
+    def readinto(self, b):
+        """Read up to len(b) bytes into bytearray b and return the number of bytes read."""
+        data = self.read(len(b))
+
+        if data is None:
+            return None
+
+        b[:len(data)] = data
+        return len(data)
+
+    def close(self):
+        """Close the file handle"""
+        return self.get_handler().close(self)
+
+    def readable(self, *args, **kwargs):
+        return True

Contents/Libraries/Shared/asio/file_opener.py

@@ -0,0 +1,21 @@
+class FileOpener(object):
+    def __init__(self, file_path, parameters=None):
+        self.file_path = file_path
+        self.parameters = parameters
+
+        self.file = None
+
+    def __enter__(self):
+        self.file = ASIO.get_handler().open(
+            self.file_path,
+            self.parameters.handlers.get(ASIO.get_handler())
+        )
+
+        return self.file
+
+    def __exit__(self, exc_type, exc_val, exc_tb):
+        if not self.file:
+            return
+
+        self.file.close()
+        self.file = None

Contents/Libraries/Shared/asio/interfaces/base.py

@@ -0,0 +1,41 @@
+# Copyright 2013 Dean Gardiner <gardiner91@gmail.com>
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from asio.file import DEFAULT_BUFFER_SIZE
+
+
+class Interface(object):
+    @classmethod
+    def open(cls, file_path, parameters=None):
+        raise NotImplementedError()
+
+    @classmethod
+    def get_size(cls, fp):
+        raise NotImplementedError()
+
+    @classmethod
+    def get_path(cls, fp):
+        raise NotImplementedError()
+
+    @classmethod
+    def seek(cls, fp, pointer, distance):
+        raise NotImplementedError()
+
+    @classmethod
+    def read(cls, fp, n=DEFAULT_BUFFER_SIZE):
+        raise NotImplementedError()
+
+    @classmethod
+    def close(cls, fp):
+        raise NotImplementedError()

Contents/Libraries/Shared/asio/interfaces/posix.py

@@ -0,0 +1,123 @@
+# Copyright 2013 Dean Gardiner <gardiner91@gmail.com>
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from asio.file import File, DEFAULT_BUFFER_SIZE
+from asio.interfaces.base import Interface
+
+import sys
+import os
+
+if os.name == 'posix':
+    import select
+
+    # fcntl is only required on darwin
+    if sys.platform == 'darwin':
+        import fcntl
+
+F_GETPATH = 50
+
+
+class PosixInterface(Interface):
+    @classmethod
+    def open(cls, file_path, parameters=None):
+        """
+        :type file_path: str
+        :rtype: asio.interfaces.posix.PosixFile
+        """
+        if not parameters:
+            parameters = {}
+
+        if not parameters.get('mode'):
+            parameters.pop('mode')
+
+        if not parameters.get('buffering'):
+            parameters.pop('buffering')
+
+        fd = os.open(file_path, os.O_RDONLY | os.O_NONBLOCK)
+
+        return PosixFile(fd)
+
+    @classmethod
+    def get_size(cls, fp):
+        """
+        :type fp: asio.interfaces.posix.PosixFile
+        :rtype: int
+        """
+        return os.fstat(fp.fd).st_size
+
+    @classmethod
+    def get_path(cls, fp):
+        """
+        :type fp: asio.interfaces.posix.PosixFile
+        :rtype: int
+        """
+
+        # readlink /dev/fd fails on darwin, so instead use fcntl F_GETPATH
+        if sys.platform == 'darwin':
+            return fcntl.fcntl(fp.fd, F_GETPATH, '\0' * 1024).rstrip('\0')
+
+        # Use /proc/self/fd if available
+        if os.path.lexists("/proc/self/fd/"):
+            return os.readlink("/proc/self/fd/%s" % fp.fd)
+
+        # Fallback to /dev/fd
+        if os.path.lexists("/dev/fd/"):
+            return os.readlink("/dev/fd/%s" % fp.fd)
+
+        raise NotImplementedError('Environment not supported (fdescfs not mounted?)')
+
+    @classmethod
+    def seek(cls, fp, offset, origin):
+        """
+        :type fp: asio.interfaces.posix.PosixFile
+        :type offset: int
+        :type origin: int
+        """
+        os.lseek(fp.fd, offset, origin)
+
+    @classmethod
+    def read(cls, fp, n=DEFAULT_BUFFER_SIZE):
+        """
+        :type fp: asio.interfaces.posix.PosixFile
+        :type n: int
+        :rtype: str
+        """
+        r, w, x = select.select([fp.fd], [], [], 5)
+
+        if r:
+            return os.read(fp.fd, n)
+
+        return None
+
+    @classmethod
+    def close(cls, fp):
+        """
+        :type fp: asio.interfaces.posix.PosixFile
+        """
+        os.close(fp.fd)
+
+
+class PosixFile(File):
+    platform_handler = PosixInterface
+
+    def __init__(self, fd, *args, **kwargs):
+        """
+        :type fd: asio.file.File
+        """
+        super(PosixFile, self).__init__(*args, **kwargs)
+
+        self.fd = fd
+
+    def __str__(self):
+        return "<asio_posix.PosixFile file: %s>" % self.fd

Contents/Libraries/Shared/asio/interfaces/windows/__init__.py

@@ -0,0 +1,201 @@
+# Copyright 2013 Dean Gardiner <gardiner91@gmail.com>
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from asio.file import File, DEFAULT_BUFFER_SIZE
+from asio.interfaces.base import Interface
+
+import os
+
+
+NULL = 0
+
+if os.name == 'nt':
+    from asio.interfaces.windows.interop import WindowsInterop
+
+
+class WindowsInterface(Interface):
+    @classmethod
+    def open(cls, file_path, parameters=None):
+        """
+        :type file_path: str
+        :rtype: asio.interfaces.windows.WindowsFile
+        """
+        if not parameters:
+            parameters = {}
+
+        return WindowsFile(WindowsInterop.create_file(
+            file_path,
+            parameters.get('desired_access', WindowsInterface.GenericAccess.READ),
+            parameters.get('share_mode', WindowsInterface.ShareMode.ALL),
+            parameters.get('creation_disposition', WindowsInterface.CreationDisposition.OPEN_EXISTING),
+            parameters.get('flags_and_attributes', NULL)
+        ))
+
+    @classmethod
+    def get_size(cls, fp):
+        """
+        :type fp: asio.interfaces.windows.WindowsFile
+        :rtype: int
+        """
+        return WindowsInterop.get_file_size(fp.handle)
+
+    @classmethod
+    def get_path(cls, fp):
+        """
+        :type fp: asio.interfaces.windows.WindowsFile
+        :rtype: str
+        """
+
+        if not fp.file_map:
+            fp.file_map = WindowsInterop.create_file_mapping(fp.handle, WindowsInterface.Protection.READONLY)
+
+        if not fp.map_view:
+            fp.map_view = WindowsInterop.map_view_of_file(fp.file_map, WindowsInterface.FileMapAccess.READ, 1)
+
+        file_name = WindowsInterop.get_mapped_file_name(fp.map_view)
+
+        return file_name
+
+    @classmethod
+    def seek(cls, fp, offset, origin):
+        """
+        :type fp: asio.interfaces.windows.WindowsFile
+        :type offset: int
+        :type origin: int
+        :rtype: int
+        """
+
+        return WindowsInterop.set_file_pointer(
+            fp.handle,
+            offset,
+            origin
+        )
+
+    @classmethod
+    def read(cls, fp, n=DEFAULT_BUFFER_SIZE):
+        """
+        :type fp: asio.interfaces.windows.WindowsFile
+        :type n: int
+        :rtype: str
+        """
+        return WindowsInterop.read(fp.handle, n)
+
+    @classmethod
+    def read_into(cls, fp, b):
+        """
+        :type fp: asio.interfaces.windows.WindowsFile
+        :type b: str
+        :rtype: int
+        """
+        return WindowsInterop.read_into(fp.handle, b)
+
+    @classmethod
+    def close(cls, fp):
+        """
+        :type fp: asio.interfaces.windows.WindowsFile
+        :rtype: bool
+        """
+        if fp.map_view:
+            WindowsInterop.unmap_view_of_file(fp.map_view)
+
+        if fp.file_map:
+            WindowsInterop.close_handle(fp.file_map)
+
+        return bool(WindowsInterop.close_handle(fp.handle))
+
+    class GenericAccess(object):
+        READ = 0x80000000
+        WRITE = 0x40000000
+        EXECUTE = 0x20000000
+        ALL = 0x10000000
+
+    class ShareMode(object):
+        READ = 0x00000001
+        WRITE = 0x00000002
+        DELETE = 0x00000004
+        ALL = READ | WRITE | DELETE
+
+    class CreationDisposition(object):
+        CREATE_NEW = 1
+        CREATE_ALWAYS = 2
+        OPEN_EXISTING = 3
+        OPEN_ALWAYS = 4
+        TRUNCATE_EXISTING = 5
+
+    class Attribute(object):
+        READONLY = 0x00000001
+        HIDDEN = 0x00000002
+        SYSTEM = 0x00000004
+        DIRECTORY = 0x00000010
+        ARCHIVE = 0x00000020
+        DEVICE = 0x00000040
+        NORMAL = 0x00000080
+        TEMPORARY = 0x00000100
+        SPARSE_FILE = 0x00000200
+        REPARSE_POINT = 0x00000400
+        COMPRESSED = 0x00000800
+        OFFLINE = 0x00001000
+        NOT_CONTENT_INDEXED = 0x00002000
+        ENCRYPTED = 0x00004000
+
+    class Flag(object):
+        WRITE_THROUGH = 0x80000000
+        OVERLAPPED = 0x40000000
+        NO_BUFFERING = 0x20000000
+        RANDOM_ACCESS = 0x10000000
+        SEQUENTIAL_SCAN = 0x08000000
+        DELETE_ON_CLOSE = 0x04000000
+        BACKUP_SEMANTICS = 0x02000000
+        POSIX_SEMANTICS = 0x01000000
+        OPEN_REPARSE_POINT = 0x00200000
+        OPEN_NO_RECALL = 0x00100000
+        FIRST_PIPE_INSTANCE = 0x00080000
+
+    class Protection(object):
+        NOACCESS = 0x01
+        READONLY = 0x02
+        READWRITE = 0x04
+        WRITECOPY = 0x08
+        EXECUTE = 0x10
+        EXECUTE_READ = 0x20,
+        EXECUTE_READWRITE = 0x40
+        EXECUTE_WRITECOPY = 0x80
+        GUARD = 0x100
+        NOCACHE = 0x200
+        WRITECOMBINE = 0x400
+
+    class FileMapAccess(object):
+        COPY = 0x0001
+        WRITE = 0x0002
+        READ = 0x0004
+        ALL_ACCESS = 0x001f
+        EXECUTE = 0x0020
+
+
+class WindowsFile(File):
+    platform_handler = WindowsInterface
+
+    def __init__(self, handle, *args, **kwargs):
+        super(WindowsFile, self).__init__(*args, **kwargs)
+
+        self.handle = handle
+
+        self.file_map = None
+        self.map_view = None
+
+    def readinto(self, b):
+        return self.get_handler().read_into(self, b)
+
+    def __str__(self):
+        return "<asio_windows.WindowsFile file: %s>" % self.handle

Contents/Libraries/Shared/asio/interfaces/windows/interop.py

@@ -0,0 +1,230 @@
+# Copyright 2013 Dean Gardiner <gardiner91@gmail.com>
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from ctypes.wintypes import *
+from ctypes import *
+import logging
+
+log = logging.getLogger(__name__)
+
+
+CreateFileW = windll.kernel32.CreateFileW
+CreateFileW.argtypes = (LPCWSTR, DWORD, DWORD, c_void_p, DWORD, DWORD, HANDLE)
+CreateFileW.restype = HANDLE
+
+ReadFile = windll.kernel32.ReadFile
+ReadFile.argtypes = (HANDLE, c_void_p, DWORD, POINTER(DWORD), HANDLE)
+ReadFile.restype = BOOL
+
+
+NULL = 0
+MAX_PATH = 260
+DEFAULT_BUFFER_SIZE = 4096
+LPSECURITY_ATTRIBUTES = c_void_p
+
+
+class WindowsInterop(object):
+    ri_buffer = None
+
+    @classmethod
+    def create_file(cls, path, desired_access, share_mode, creation_disposition, flags_and_attributes):
+        h = CreateFileW(
+            path,
+            desired_access,
+            share_mode,
+            NULL,
+            creation_disposition,
+            flags_and_attributes,
+            NULL
+        )
+
+        error = GetLastError()
+        if error != 0:
+            raise Exception('[WindowsASIO.open] "%s"' % FormatError(error))
+
+        return h
+
+    @classmethod
+    def read(cls, handle, buf_size=DEFAULT_BUFFER_SIZE):
+        buf = create_string_buffer(buf_size)
+        bytes_read = c_ulong(0)
+
+        success = ReadFile(handle, buf, buf_size, byref(bytes_read), NULL)
+
+        error = GetLastError()
+        if error:
+            log.debug('read_file - error: (%s) "%s"', error, FormatError(error))
+
+        if not success and error:
+            raise Exception('[WindowsInterop.read_file] (%s) "%s"' % (error, FormatError(error)))
+
+        # Return if we have a valid buffer
+        if success and bytes_read.value:
+            return buf.value
+
+        return None
+
+    @classmethod
+    def read_into(cls, handle, b):
+        if cls.ri_buffer is None or len(cls.ri_buffer) < len(b):
+            cls.ri_buffer = create_string_buffer(len(b))
+
+        bytes_read = c_ulong(0)
+
+        success = ReadFile(handle, cls.ri_buffer, len(b), byref(bytes_read), NULL)
+        bytes_read = int(bytes_read.value)
+
+        b[:bytes_read] = cls.ri_buffer[:bytes_read]
+
+        error = GetLastError()
+        
+        if not success and error:
+            raise Exception('[WindowsInterop.read_file] (%s) "%s"' % (error, FormatError(error)))
+
+        # Return if we have a valid buffer
+        if success and bytes_read:
+            return bytes_read
+
+        return None
+
+    @classmethod
+    def set_file_pointer(cls, handle, distance, method):
+        pos_high = DWORD(NULL)
+
+        result = windll.kernel32.SetFilePointer(
+            handle,
+            c_ulong(distance),
+            byref(pos_high),
+            DWORD(method)
+        )
+
+        if result == -1:
+            raise Exception('[WindowsASIO.seek] INVALID_SET_FILE_POINTER: "%s"' % FormatError(GetLastError()))
+
+        return result
+
+    @classmethod
+    def get_file_size(cls, handle):
+        return windll.kernel32.GetFileSize(
+            handle,
+            DWORD(NULL)
+        )
+
+    @classmethod
+    def close_handle(cls, handle):
+        return windll.kernel32.CloseHandle(handle)
+
+    @classmethod
+    def create_file_mapping(cls, handle, protect, maximum_size_high=0, maximum_size_low=1):
+        return HANDLE(windll.kernel32.CreateFileMappingW(
+            handle,
+            LPSECURITY_ATTRIBUTES(NULL),
+            DWORD(protect),
+            DWORD(maximum_size_high),
+            DWORD(maximum_size_low),
+            LPCSTR(NULL)
+        ))
+
+    @classmethod
+    def map_view_of_file(cls, map_handle, desired_access, num_bytes, file_offset_high=0, file_offset_low=0):
+        return HANDLE(windll.kernel32.MapViewOfFile(
+            map_handle,
+            DWORD(desired_access),
+            DWORD(file_offset_high),
+            DWORD(file_offset_low),
+            num_bytes
+        ))
+
+    @classmethod
+    def unmap_view_of_file(cls, view_handle):
+        return windll.kernel32.UnmapViewOfFile(view_handle)
+
+    @classmethod
+    def get_mapped_file_name(cls, view_handle, translate_device_name=True):
+        buf = create_string_buffer(MAX_PATH + 1)
+
+        result = windll.psapi.GetMappedFileNameW(
+            cls.get_current_process(),
+            view_handle,
+            buf,
+            MAX_PATH
+        )
+
+        # Raise exception on error
+        error = GetLastError()
+        if result == 0:
+            raise Exception(FormatError(error))
+
+        # Retrieve a clean file name (skipping over NUL bytes)
+        file_name = cls.clean_buffer_value(buf)
+
+        # If we are not translating the device name return here
+        if not translate_device_name:
+            return file_name
+
+        drives = cls.get_logical_drive_strings()
+
+        # Find the drive matching the file_name device name
+        translated = False
+        for drive in drives:
+            device_name = cls.query_dos_device(drive)
+
+            if file_name.startswith(device_name):
+                file_name = drive + file_name[len(device_name):]
+                translated = True
+                break
+
+        if not translated:
+            raise Exception('Unable to translate device name')
+
+        return file_name
+
+    @classmethod
+    def get_logical_drive_strings(cls, buf_size=512):
+        buf = create_string_buffer(buf_size)
+
+        result = windll.kernel32.GetLogicalDriveStringsW(buf_size, buf)
+
+        error = GetLastError()
+        if result == 0:
+            raise Exception(FormatError(error))
+
+        drive_strings = cls.clean_buffer_value(buf)
+        return [dr for dr in drive_strings.split('\\') if dr != '']
+
+    @classmethod
+    def query_dos_device(cls, drive, buf_size=MAX_PATH):
+        buf = create_string_buffer(buf_size)
+
+        result = windll.kernel32.QueryDosDeviceA(
+            drive,
+            buf,
+            buf_size
+        )
+
+        return cls.clean_buffer_value(buf)
+
+    @classmethod
+    def get_current_process(cls):
+        return HANDLE(windll.kernel32.GetCurrentProcess())
+
+    @classmethod
+    def clean_buffer_value(cls, buf):
+        value = ""
+
+        for ch in buf.raw:
+            if ord(ch) != 0:
+                value += ch
+
+        return value

Contents/Libraries/Shared/asio/open_parameters.py

@@ -0,0 +1,47 @@
+from asio.interfaces.posix import PosixInterface
+from asio.interfaces.windows import WindowsInterface
+
+
+class OpenParameters(object):
+    def __init__(self):
+        self.handlers = {}
+
+        # Update handler_parameters with defaults
+        self.posix()
+        self.windows()
+
+    def posix(self, mode=None, buffering=None):
+        """
+        :type mode: str
+        :type buffering: int
+        """
+        self.handlers.update({PosixInterface: {
+            'mode': mode,
+            'buffering': buffering
+        }})
+
+    def windows(self, desired_access=WindowsInterface.GenericAccess.READ,
+                share_mode=WindowsInterface.ShareMode.ALL,
+                creation_disposition=WindowsInterface.CreationDisposition.OPEN_EXISTING,
+                flags_and_attributes=0):
+
+        """
+        :param desired_access: WindowsInterface.DesiredAccess
+        :type desired_access: int
+
+        :param share_mode: WindowsInterface.ShareMode
+        :type share_mode: int
+
+        :param creation_disposition: WindowsInterface.CreationDisposition
+        :type creation_disposition: int
+
+        :param flags_and_attributes: WindowsInterface.Attribute, WindowsInterface.Flag
+        :type flags_and_attributes: int
+        """
+
+        self.handlers.update({WindowsInterface: {
+            'desired_access': desired_access,
+            'share_mode': share_mode,
+            'creation_disposition': creation_disposition,
+            'flags_and_attributes': flags_and_attributes
+        }})

Contents/Libraries/Shared/plex/interfaces/section.py

@@ -20,6 +20,23 @@ class SectionInterface(Interface):
             }))
         }))
 
+    def recently_added(self, key):
+        response = self.http.get(key, 'recentlyAdded')
+
+        return self.parse(response, idict({
+            'MediaContainer': ('MediaContainer', idict({
+                'Directory': {
+                    'artist':   'Artist',
+                    'show':     'Show'
+                },
+                'Video': {
+                    'movie':    'Movie',
+                    'episode':  'Episode',
+                    'clip':     'Clip',
+                }
+            }))
+        }))
+
     def first_character(self, key, character=None):
         if character:
             response = self.http.get(key, ['firstCharacter', character])

Contents/Libraries/Shared/plex/objects/library/stream.py

@@ -5,9 +5,14 @@ class Stream(Descriptor):
     id = Property(type=int)
     index = Property(type=int)
 
+    stream_key = Property('key')
+
     stream_type = Property('streamType', type=int)
     selected = Property(type=bool)
 
+    forced = Property(type=bool)
+    default = Property(type=bool)
+
     title = Property
     duration = Property(type=int)
 

Contents/Libraries/Shared/plex_activity/__init__.py

@@ -0,0 +1,15 @@
+import logging
+import traceback
+
+log = logging.getLogger(__name__)
+
+__version__ = '0.7.1'
+
+
+try:
+    from plex_activity import activity
+
+    # Global objects (using defaults)
+    Activity = activity.Activity()
+except Exception as ex:
+    log.warn('Unable to import submodules: %s - %s', ex, traceback.format_exc())

Contents/Libraries/Shared/plex_activity/activity.py

@@ -0,0 +1,96 @@
+from plex.lib import six as six
+from plex.lib.six.moves import xrange
+from plex_activity.sources import Logging, WebSocket
+
+from pyemitter import Emitter
+import logging
+
+log = logging.getLogger(__name__)
+
+
+class ActivityMeta(type):
+    def __getitem__(self, key):
+        for (weight, source) in self.registered:
+            if source.name == key:
+                return source
+
+        return None
+
+
+@six.add_metaclass(ActivityMeta)
+class Activity(Emitter):
+    registered = []
+
+    def __init__(self, sources=None):
+        self.available = self.get_available(sources)
+        self.enabled = []
+
+    def start(self, sources=None):
+        # TODO async start
+
+        if sources is not None:
+            self.available = self.get_available(sources)
+
+        # Test methods until an available method is found
+        for weight, source in self.available:
+            if weight is None:
+                # None = always start
+                self.start_source(source)
+            elif source.test():
+                # Test passed
+                self.start_source(source)
+            else:
+                log.info('activity source "%s" is not available', source.name)
+
+        log.info(
+            'Finished starting %s method(s): %s',
+            len(self.enabled),
+            ', '.join([('"%s"' % source.name) for source in self.enabled])
+        )
+
+    def start_source(self, source):
+        instance = source(self)
+        instance.start()
+
+        self.enabled.append(instance)
+
+    def __getitem__(self, key):
+        for (weight, source) in self.registered:
+            if source.name == key:
+                return source
+
+        return None
+
+    @classmethod
+    def get_available(cls, sources):
+        if sources:
+            return [
+                (weight, source) for (weight, source) in cls.registered
+                if source.name in sources
+            ]
+
+        return cls.registered
+
+    @classmethod
+    def register(cls, source, weight=None):
+        item = (weight, source)
+
+        # weight = None, highest priority
+        if weight is None:
+            cls.registered.insert(0, item)
+            return
+
+        # insert in DESC order
+        for x in xrange(len(cls.registered)):
+            w, _ = cls.registered[x]
+
+            if w is not None and w < weight:
+                cls.registered.insert(x, item)
+                return
+
+        # otherwise append
+        cls.registered.append(item)
+
+# Register activity sources
+Activity.register(WebSocket)
+Activity.register(Logging, weight=1)

Contents/Libraries/Shared/plex_activity/core/helpers.py

@@ -0,0 +1,44 @@
+def str_format(s, *args, **kwargs):
+    """Return a formatted version of S, using substitutions from args and kwargs.
+
+    (Roughly matches the functionality of str.format but ensures compatibility with Python 2.5)
+    """
+
+    args = list(args)
+
+    x = 0
+    while x < len(s):
+        # Skip non-start token characters
+        if s[x] != '{':
+            x += 1
+            continue
+
+        end_pos = s.find('}', x)
+
+        # If end character can't be found, move to next character
+        if end_pos == -1:
+            x += 1
+            continue
+
+        name = s[x + 1:end_pos]
+
+        # Ensure token name is alpha numeric
+        if not name.isalnum():
+            x += 1
+            continue
+
+        # Try find value for token
+        value = args.pop(0) if args else kwargs.get(name)
+
+        if value:
+            value = str(value)
+
+            # Replace token with value
+            s = s[:x] + value + s[end_pos + 1:]
+
+            # Update current position
+            x = x + len(value) - 1
+
+        x += 1
+
+    return s

Contents/Libraries/Shared/plex_activity/sources/__init__.py

@@ -0,0 +1,4 @@
+from plex_activity.sources.s_logging import Logging
+from plex_activity.sources.s_websocket import WebSocket
+
+__all__ = ['Logging', 'WebSocket']

Contents/Libraries/Shared/plex_activity/sources/base.py

@@ -0,0 +1,24 @@
+from pyemitter import Emitter
+from threading import Thread
+import logging
+
+log = logging.getLogger(__name__)
+
+
+class Source(Emitter):
+    name = None
+
+    def __init__(self):
+        self.thread = Thread(target=self._run_wrapper)
+
+    def start(self):
+        self.thread.start()
+
+    def run(self):
+        pass
+
+    def _run_wrapper(self):
+        try:
+            self.run()
+        except Exception as ex:
+            log.error('Exception raised in "%s" activity source: %s', self.name, ex, exc_info=True)

Contents/Libraries/Shared/plex_activity/sources/s_logging/__init__.py

@@ -0,0 +1,3 @@
+from plex_activity.sources.s_logging.main import Logging
+
+__all__ = ['Logging']

Contents/Libraries/Shared/plex_activity/sources/s_logging/main.py

@@ -0,0 +1,249 @@
+from plex import Plex
+from plex_activity.sources.base import Source
+from plex_activity.sources.s_logging.parsers import NowPlayingParser, ScrobbleParser
+
+from asio import ASIO
+from asio.file import SEEK_ORIGIN_CURRENT
+from io import BufferedReader
+import inspect
+import logging
+import os
+import platform
+import time
+
+log = logging.getLogger(__name__)
+
+PATH_HINTS = {
+    'Darwin': [
+        lambda: os.path.join(os.getenv('HOME'), 'Library/Logs/Plex Media Server.log')
+    ],
+    'FreeBSD': [
+        # FreeBSD
+        '/usr/local/plexdata/Plex Media Server/Logs/Plex Media Server.log',
+        '/usr/local/plexdata-plexpass/Plex Media Server/Logs/Plex Media Server.log',
+
+        # FreeNAS
+        '/usr/pbi/plexmediaserver-amd64/plexdata/Plex Media Server/Logs/Plex Media Server.log',
+        '/var/db/plexdata/Plex Media Server/Logs/Plex Media Server.log',
+        '/var/db/plexdata-plexpass/Plex Media Server/Logs/Plex Media Server.log'
+    ],
+    'Linux': [
+        # QNAP
+        '/share/HDA_DATA/.qpkg/PlexMediaServer/Library/Plex Media Server/Logs/Plex Media Server.log',
+
+        # Debian
+        '/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Logs/Plex Media Server.log'
+    ],
+    'Windows': [
+        lambda: os.path.join(os.getenv('LOCALAPPDATA'), 'Plex Media Server\\Logs\\Plex Media Server.log')
+    ]
+}
+
+
+class Logging(Source):
+    name = 'logging'
+    events = [
+        'logging.playing',
+        'logging.action.played',
+        'logging.action.unplayed'
+    ]
+
+    parsers = []
+
+    path = None
+    path_hints = PATH_HINTS
+
+    def __init__(self, activity):
+        super(Logging, self).__init__()
+
+        self.parsers = [p(self) for p in Logging.parsers]
+
+        self.file = None
+        self.reader = None
+
+        self.path = None
+
+        # Pipe events to the main activity instance
+        self.pipe(self.events, activity)
+
+    def run(self):
+        line = self.read_line_retry(ping=True, stale_sleep=0.5)
+        if not line:
+            log.info('Unable to read log file')
+            return
+
+        log.debug('Ready')
+
+        while True:
+            # Grab the next line of the log
+            line = self.read_line_retry(ping=True)
+
+            if line:
+                self.process(line)
+            else:
+                log.info('Unable to read log file')
+
+    def process(self, line):
+        for parser in self.parsers:
+            if parser.process(line):
+                return True
+
+        return False
+
+    def read_line(self):
+        if not self.file:
+            path = self.get_path()
+            if not path:
+                raise Exception('Unable to find the location of "Plex Media Server.log"')
+
+            # Open file
+            self.file = ASIO.open(path, opener=False)
+            self.file.seek(self.file.get_size(), SEEK_ORIGIN_CURRENT)
+
+            # Create buffered reader
+            self.reader = BufferedReader(self.file)
+
+            self.path = self.file.get_path()
+            log.info('Opened file path: "%s"' % self.path)
+
+        return self.reader.readline()
+
+    def read_line_retry(self, timeout=60, ping=False, stale_sleep=1.0):
+        line = None
+        stale_since = None
+
+        while not line:
+            line = self.read_line()
+
+            if line:
+                stale_since = None
+                time.sleep(0.05)
+                break
+
+            if stale_since is None:
+                stale_since = time.time()
+                time.sleep(stale_sleep)
+                continue
+            elif (time.time() - stale_since) > timeout:
+                return None
+            elif (time.time() - stale_since) > timeout / 2:
+                # Nothing returned for 5 seconds
+                if self.file.get_path() != self.path:
+                    log.debug("Log file moved (probably rotated), closing")
+                    self.close()
+                elif ping:
+                    # Ping server to see if server is still active
+                    Plex.detail()
+                    ping = False
+
+            time.sleep(stale_sleep)
+
+        return line
+
+    def close(self):
+        if not self.file:
+            return
+
+        try:
+            # Close the buffered reader
+            self.reader.close()
+        except Exception as ex:
+            log.error('reader.close() - raised exception: %s', ex, exc_info=True)
+        finally:
+            self.reader = None
+
+        try:
+            # Close the file handle
+            self.file.close()
+        except OSError as ex:
+            if ex.errno == 9:
+                # Bad file descriptor, already closed?
+                log.info('file.close() - ignoring raised exception: %s (already closed)', ex)
+            else:
+                log.error('file.close() - raised exception: %s', ex, exc_info=True)
+        except Exception as ex:
+            log.error('file.close() - raised exception: %s', ex, exc_info=True)
+        finally:
+            self.file = None
+
+    @classmethod
+    def get_path(cls):
+        if cls.path:
+            return cls.path
+
+        hints = cls.get_hints()
+
+        log.debug('hints: %r', hints)
+
+        if not hints:
+            log.error('Unable to find any hints for "%s", operating system not supported', platform.system())
+            return None
+
+        for hint in hints:
+            log.debug('Testing if "%s" exists', hint)
+
+            if os.path.exists(hint):
+                cls.path = hint
+                break
+
+        if cls.path:
+            log.debug('Using the path: %r', cls.path)
+        else:
+            log.error('Unable to find a valid path for "Plex Media Server.log"', extra={
+                'data': {
+                    'hints': hints
+                }
+            })
+
+        return cls.path
+
+    @classmethod
+    def add_hint(cls, path, system=None):
+        if system not in cls.path_hints:
+            cls.path_hints[system] = []
+
+        cls.path_hints[system].append(path)
+
+    @classmethod
+    def get_hints(cls):
+        # Retrieve system hints
+        hints_system = PATH_HINTS.get(platform.system(), [])
+
+        # Retrieve global hints
+        hints_global = PATH_HINTS.get(None, [])
+
+        # Retrieve hint from server preferences (if available)
+        data_path = Plex[':/prefs'].get('LocalAppDataPath')
+
+        if data_path:
+            hints_global.append(os.path.join(data_path.value, "Plex Media Server", "Logs", "Plex Media Server.log"))
+        else:
+            log.info('Unable to retrieve "LocalAppDataPath" from server')
+
+        hints = []
+
+        for hint in (hints_global + hints_system):
+            # Resolve hint function
+            if inspect.isfunction(hint):
+                hint = hint()
+
+            # Check for duplicate
+            if hint in hints:
+                continue
+
+            hints.append(hint)
+
+        return hints
+
+    @classmethod
+    def test(cls):
+        # TODO "Logging" source testing
+        return True
+
+    @classmethod
+    def register(cls, parser):
+        cls.parsers.append(parser)
+
+
+Logging.register(NowPlayingParser)
+Logging.register(ScrobbleParser)

Contents/Libraries/Shared/plex_activity/sources/s_logging/parsers/__init__.py

@@ -0,0 +1,4 @@
+from plex_activity.sources.s_logging.parsers.now_playing import NowPlayingParser
+from plex_activity.sources.s_logging.parsers.scrobble import ScrobbleParser
+
+__all__ = ['NowPlayingParser', 'ScrobbleParser']

Contents/Libraries/Shared/plex_activity/sources/s_logging/parsers/base.py

@@ -0,0 +1,96 @@
+from plex.lib.six.moves import urllib_parse as urlparse
+from plex_activity.core.helpers import str_format
+
+from pyemitter import Emitter
+import logging
+import re
+
+log = logging.getLogger(__name__)
+
+LOG_PATTERN = r'^.*?\[\w+\]\s\w+\s-\s{message}$'
+REQUEST_HEADER_PATTERN = str_format(LOG_PATTERN, message=r"Request: (\[(?P<address>.*?):(?P<port>\d+)[^]]*\]\s)?{method} {path}.*?")
+
+IGNORE_PATTERNS = [
+    r'error parsing allowedNetworks.*?',
+    r'Comparing request from.*?',
+    r'(Auth: )?We found auth token (.*?), enabling token-based authentication\.',
+    r'(Auth: )?Came in with a super-token, authorization succeeded\.',
+    r'(Auth: )?Refreshing tokens inside the token-based authentication filter\.',
+    r'\[Now\] Updated play state for .*?',
+    r'Play progress on .*? - got played .*? ms by account .*?!',
+    r'(Statistics: )?\(.*?\) Reporting active playback in state \d+ of type \d+ \(.*?\) for account \d+',
+    r'Request: \[.*?\] (GET|PUT) /video/:/transcode/.*?',
+    r'Received transcode session ping for session .*?'
+]
+
+IGNORE_REGEX = re.compile(str_format(LOG_PATTERN, message='(%s)' % ('|'.join('(%s)' % x for x in IGNORE_PATTERNS))), re.IGNORECASE)
+
+
+PARAM_REGEX = re.compile(str_format(LOG_PATTERN, message=r' \* (?P<key>.*?) =\> (?P<value>.*?)'), re.IGNORECASE)
+
+
+class Parser(Emitter):
+    def __init__(self, core):
+        self.core = core
+
+    def read_parameters(self, *match_functions):
+        match_functions = [self.parameter_match] + list(match_functions)
+
+        info = {}
+
+        while True:
+            line = self.core.read_line_retry(timeout=5)
+            if not line:
+                log.info('Unable to read log file')
+                return {}
+
+            # Run through each match function to find a result
+            match = None
+            for func in match_functions:
+                match = func(line)
+
+                if match is not None:
+                    break
+
+            # Update info dict with result, otherwise finish reading
+            if match:
+                info.update(match)
+            elif match is None and IGNORE_REGEX.match(line.strip()) is None:
+                log.debug('break on "%s"', line.strip())
+                break
+
+        return info
+
+    def process(self, line):
+        raise NotImplementedError()
+
+    @staticmethod
+    def parameter_match(line):
+        match = PARAM_REGEX.match(line.strip())
+        if not match:
+            return None
+
+        match = match.groupdict()
+
+        return {match['key']: match['value']}
+
+    @staticmethod
+    def regex_match(regex, line):
+        match = regex.match(line.strip())
+        if not match:
+            return None
+
+        return match.groupdict()
+
+    @staticmethod
+    def query(match, value):
+        if not value:
+            return
+
+        try:
+            parameters = urlparse.parse_qsl(value, strict_parsing=True)
+        except ValueError:
+            return
+
+        for key, value in parameters:
+            match.setdefault(key, value)

Contents/Libraries/Shared/plex_activity/sources/s_logging/parsers/now_playing.py

@@ -0,0 +1,116 @@
+from plex_activity.core.helpers import str_format
+from plex_activity.sources.s_logging.parsers.base import Parser, LOG_PATTERN, REQUEST_HEADER_PATTERN
+
+import logging
+import re
+
+log = logging.getLogger(__name__)
+
+PLAYING_HEADER_PATTERN = str_format(REQUEST_HEADER_PATTERN, method="GET", path="/:/(?P<type>timeline|progress)/?(?:\?(?P<query>.*?))?\s")
+PLAYING_HEADER_REGEX = re.compile(PLAYING_HEADER_PATTERN, re.IGNORECASE)
+
+RANGE_REGEX = re.compile(str_format(LOG_PATTERN, message=r'Request range: \d+ to \d+'), re.IGNORECASE)
+CLIENT_REGEX = re.compile(str_format(LOG_PATTERN, message=r'Client \[(?P<machineIdentifier>.*?)\].*?'), re.IGNORECASE)
+
+NOW_USER_REGEX = re.compile(str_format(LOG_PATTERN, message=r'\[Now\] User is (?P<user_name>.+) \(ID: (?P<user_id>\d+)\)'), re.IGNORECASE)
+NOW_CLIENT_REGEX = re.compile(str_format(LOG_PATTERN, message=r'\[Now\] Device is (?P<product>.+?) \((?P<client>.+)\)\.'), re.IGNORECASE)
+
+
+class NowPlayingParser(Parser):
+    required_info = [
+        'ratingKey',
+        'state', 'time'
+    ]
+
+    extra_info = [
+        'duration',
+
+        'user_name', 'user_id',
+        'machineIdentifier', 'client'
+    ]
+
+    events = [
+        'logging.playing'
+    ]
+
+    def __init__(self, main):
+        super(NowPlayingParser, self).__init__(main)
+
+        # Pipe events to the main logging activity instance
+        self.pipe(self.events, main)
+
+    def process(self, line):
+        header_match = PLAYING_HEADER_REGEX.match(line)
+        if not header_match:
+            return False
+
+        activity_type = header_match.group('type')
+
+        # Get a match from the activity entries
+        if activity_type == 'timeline':
+            match = self.timeline()
+        elif activity_type == 'progress':
+            match = self.progress()
+        else:
+            log.warn('Unknown activity type "%s"', activity_type)
+            return True
+
+        print match, activity_type
+
+        if match is None:
+            match = {}
+
+        # Extend match with query info
+        self.query(match, header_match.group('query'))
+
+        # Ensure we successfully matched a result
+        if not match:
+            return True
+
+        # Sanitize the activity result
+        info = {
+            'address': header_match.group('address'),
+            'port': header_match.group('port')
+        }
+
+        # - Get required info parameters
+        for key in self.required_info:
+            if key in match and match[key] is not None:
+                info[key] = match[key]
+            else:
+                log.info('Invalid activity match, missing key %s (matched keys: %s)', key, match.keys())
+                return True
+
+        # - Add in any extra info parameters
+        for key in self.extra_info:
+            if key in match:
+                info[key] = match[key]
+            else:
+                info[key] = None
+
+        # Update the scrobbler with the current state
+        self.emit('logging.playing', info)
+        return True
+
+    def timeline(self):
+        return self.read_parameters(
+            lambda line: self.regex_match(CLIENT_REGEX, line),
+            lambda line: self.regex_match(RANGE_REGEX, line),
+
+            # [Now]* entries
+            lambda line: self.regex_match(NOW_USER_REGEX, line),
+            lambda line: self.regex_match(NOW_CLIENT_REGEX, line),
+        )
+
+    def progress(self):
+        data = self.read_parameters()
+
+        if not data:
+            return {}
+
+        # Translate parameters into timeline-style form
+        return {
+            'state': data.get('state'),
+            'ratingKey': data.get('key'),
+            'time': data.get('time')
+        }

Contents/Libraries/Shared/plex_activity/sources/s_logging/parsers/scrobble.py

@@ -0,0 +1,38 @@
+from plex_activity.core.helpers import str_format
+from plex_activity.sources.s_logging.parsers.base import Parser, LOG_PATTERN
+
+import re
+
+
+class ScrobbleParser(Parser):
+    pattern = str_format(LOG_PATTERN, message=r'Library item (?P<rating_key>\d+) \'(?P<title>.*?)\' got (?P<action>(?:un)?played) by account (?P<account_key>\d+)!.*?')
+    regex = re.compile(pattern, re.IGNORECASE)
+
+    events = [
+        'logging.action.played',
+        'logging.action.unplayed'
+    ]
+
+    def __init__(self, main):
+        super(ScrobbleParser, self).__init__(main)
+
+        # Pipe events to the main logging activity instance
+        self.pipe(self.events, main)
+
+    def process(self, line):
+        match = self.regex.match(line)
+        if not match:
+            return False
+
+        action = match.group('action')
+        if not action:
+            return False
+
+        self.emit('logging.action.%s' % action, {
+            'account_key': match.group('account_key'),
+            'rating_key': match.group('rating_key'),
+
+            'title': match.group('title')
+        })
+
+        return True

Contents/Libraries/Shared/plex_activity/sources/s_websocket/__init__.py

@@ -0,0 +1,3 @@
+from plex_activity.sources.s_websocket.main import WebSocket
+
+__all__ = ['WebSocket']

Contents/Libraries/Shared/plex_activity/sources/s_websocket/main.py

@@ -0,0 +1,298 @@
+from plex import Plex
+from plex.lib.six.moves.urllib_parse import urlencode
+from plex_activity.sources.base import Source
+
+import json
+import logging
+import re
+import time
+import websocket
+
+log = logging.getLogger(__name__)
+
+SCANNING_REGEX = re.compile('Scanning the "(?P<section>.*?)" section', re.IGNORECASE)
+SCAN_COMPLETE_REGEX = re.compile('Library scan complete', re.IGNORECASE)
+
+TIMELINE_STATES = {
+    0: 'created',
+    2: 'matching',
+    3: 'downloading',
+    4: 'loading',
+    5: 'finished',
+    6: 'analyzing',
+    9: 'deleted'
+}
+
+
+class WebSocket(Source):
+    name = 'websocket'
+    events = [
+        'websocket.playing',
+
+        'websocket.scanner.started',
+        'websocket.scanner.progress',
+        'websocket.scanner.finished',
+
+        'websocket.timeline.created',
+        'websocket.timeline.matching',
+        'websocket.timeline.downloading',
+        'websocket.timeline.loading',
+        'websocket.timeline.finished',
+        'websocket.timeline.analyzing',
+        'websocket.timeline.deleted'
+    ]
+
+    opcode_data = (websocket.ABNF.OPCODE_TEXT, websocket.ABNF.OPCODE_BINARY)
+
+    def __init__(self, activity):
+        super(WebSocket, self).__init__()
+
+        self.ws = None
+        self.reconnects = 0
+
+        # Pipe events to the main activity instance
+        self.pipe(self.events, activity)
+
+    def connect(self):
+        uri = 'ws://%s:%s/:/websockets/notifications' % (
+            Plex.configuration.get('server.host', '127.0.0.1'),
+            Plex.configuration.get('server.port', 32400)
+        )
+
+        params = {}
+
+        # Set authentication token (if one is available)
+        if Plex.configuration['authentication.token']:
+            params['X-Plex-Token'] = Plex.configuration['authentication.token']
+
+        # Append parameters to uri
+        if params:
+            uri += '?' + urlencode(params)
+
+        # Create websocket connection
+        self.ws = websocket.create_connection(uri)
+
+    def run(self):
+        self.connect()
+
+        log.debug('Ready')
+
+        while True:
+            try:
+                self.process(*self.receive())
+
+                # successfully received data, reset reconnects counter
+                self.reconnects = 0
+            except websocket.WebSocketConnectionClosedException:
+                if self.reconnects <= 5:
+                    self.reconnects += 1
+
+                    # Increasing sleep interval between reconnections
+                    if self.reconnects > 1:
+                        time.sleep(2 * (self.reconnects - 1))
+
+                    log.info('WebSocket connection has closed, reconnecting...')
+                    self.connect()
+                else:
+                    log.error('WebSocket connection unavailable, activity monitoring not available')
+                    break
+
+    def receive(self):
+        frame = self.ws.recv_frame()
+
+        if not frame:
+            raise websocket.WebSocketException("Not a valid frame %s" % frame)
+        elif frame.opcode in self.opcode_data:
+            return frame.opcode, frame.data
+        elif frame.opcode == websocket.ABNF.OPCODE_CLOSE:
+            self.ws.send_close()
+            return frame.opcode, None
+        elif frame.opcode == websocket.ABNF.OPCODE_PING:
+            self.ws.pong("Hi!")
+
+        return None, None
+
+    def process(self, opcode, data):
+        if opcode not in self.opcode_data:
+            return False
+
+        try:
+            info = json.loads(data)
+        except UnicodeDecodeError as ex:
+            log.warn('Error decoding message from websocket: %s' % ex, extra={
+                'event': {
+                    'module': __name__,
+                    'name': 'process.loads.unicode_decode_error',
+                    'key': '%s:%s' % (ex.encoding, ex.reason)
+                }
+            })
+            log.debug(data)
+            return False
+        except Exception as ex:
+            log.warn('Error decoding message from websocket: %s' % ex, extra={
+                'event': {
+                    'module': __name__,
+                    'name': 'process.load_exception',
+                    'key': ex.message
+                }
+            })
+            log.debug(data)
+            return False
+
+        # Handle modern messages (PMS 1.3.0+)
+        if type(info.get('NotificationContainer')) is dict:
+            info = info['NotificationContainer']
+
+        # Process  message
+        m_type = info.get('type')
+
+        if not m_type:
+            log.debug('Received message with no "type" parameter: %r', info)
+            return False
+
+        # Pre-process message (if function exists)
+        process_func = getattr(self, 'process_%s' % m_type, None)
+
+        if process_func and process_func(info):
+            return True
+
+        # Emit raw message
+        return self.emit_notification('%s.notification.%s' % (self.name, m_type), info)
+
+    def process_playing(self, info):
+        children = info.get('_children') or info.get('PlaySessionStateNotification')
+
+        if not children:
+            log.debug('Received "playing" message with no children: %r', info)
+            return False
+
+        return self.emit_notification('%s.playing' % self.name, children)
+
+    def process_progress(self, info):
+        children = info.get('_children') or info.get('ProgressNotification')
+
+        if not children:
+            log.debug('Received "progress" message with no children: %r', info)
+            return False
+
+        for notification in children:
+            self.emit('%s.scanner.progress' % self.name, {
+                'message': notification.get('message')
+            })
+
+        return True
+
+    def process_status(self, info):
+        children = info.get('_children') or info.get('StatusNotification')
+
+        if not children:
+            log.debug('Received "status" message with no children: %r', info)
+            return False
+
+        # Process children
+        count = 0
+
+        for notification in children:
+            title = notification.get('title')
+
+            if not title:
+                continue
+
+            # Scan complete message
+            if SCAN_COMPLETE_REGEX.match(title):
+                self.emit('%s.scanner.finished' % self.name)
+                count += 1
+                continue
+
+            # Scanning message
+            match = SCANNING_REGEX.match(title)
+
+            if not match:
+                continue
+
+            section = match.group('section')
+
+            if not section:
+                continue
+
+            self.emit('%s.scanner.started' % self.name, {'section': section})
+            count += 1
+
+        # Validate result
+        if count < 1:
+            log.debug('Received "status" message with no valid children: %r', info)
+            return False
+
+        return True
+
+    def process_timeline(self, info):
+        children = info.get('_children') or info.get('TimelineEntry')
+
+        if not children:
+            log.debug('Received "timeline" message with no children: %r', info)
+            return False
+
+        # Process children
+        count = 0
+
+        for entry in children:
+            state = TIMELINE_STATES.get(entry.get('state'))
+
+            if not state:
+                continue
+
+            self.emit('%s.timeline.%s' % (self.name, state), entry)
+            count += 1
+
+        # Validate result
+        if count < 1:
+            log.debug('Received "timeline" message with no valid children: %r', info)
+            return False
+
+        return True
+
+    #
+    # Helpers
+    #
+
+    def emit_notification(self, name, info=None):
+        if info is None:
+            info = {}
+
+        # Emit children
+        children = self._get_children(info)
+
+        if children:
+            for child in children:
+                self.emit(name, child)
+
+            return True
+
+        # Emit objects
+        if info:
+            self.emit(name, info)
+        else:
+            self.emit(name)
+
+        return True
+
+    @staticmethod
+    def _get_children(info):
+        if type(info) is list:
+            return info
+
+        if type(info) is not dict:
+            return None
+
+        # Return legacy children
+        if info.get('_children'):
+            return info['_children']
+
+        # Search for modern children container
+        for key, value in info.items():
+            key = key.lower()
+
+            if (key.endswith('entry') or key.endswith('notification')) and type(value) is list:
+                return value
+
+        return None

Contents/Libraries/Shared/pyemitter.py

@@ -0,0 +1,235 @@
+import logging
+
+# concurrent.futures is optional
+try:
+    from concurrent.futures import ThreadPoolExecutor
+except ImportError:
+    ThreadPoolExecutor = None
+
+
+log = logging.getLogger(__name__)
+
+
+class Emitter(object):
+    threading = False
+    threading_workers = 2
+
+    __constructed = False
+    __name = None
+
+    __callbacks = None
+    __threading_pool = None
+
+    def __ensure_constructed(self):
+        if self.__constructed:
+            return
+
+        self.__callbacks = {}
+        self.__constructed = True
+
+        if self.threading:
+            if ThreadPoolExecutor is None:
+                raise Exception('concurrent.futures is required for threading')
+
+            self.__threading_pool = ThreadPoolExecutor(max_workers=self.threading_workers)
+
+    def __log(self, message, *args, **kwargs):
+        if self.__name is None:
+            self.__name = '%s.%s' % (
+                self.__module__,
+                self.__class__.__name__
+            )
+
+        log.debug(
+            ('[%s]:' % self.__name.ljust(34)) + str(message),
+            *args, **kwargs
+        )
+
+    def __wrap(self, callback, *args, **kwargs):
+        def wrap(func):
+            callback(func=func, *args, **kwargs)
+            return func
+
+        return wrap
+
+    def on(self, events, func=None, on_bound=None):
+        if not func:
+            # assume decorator, wrap
+            return self.__wrap(self.on, events, on_bound=on_bound)
+
+        if not isinstance(events, (list, tuple)):
+            events = [events]
+
+        self.__log('on(events: %s, func: %s)', repr(events), repr(func))
+
+        self.__ensure_constructed()
+
+        for event in events:
+            if event not in self.__callbacks:
+                self.__callbacks[event] = []
+
+            # Bind callback to event
+            self.__callbacks[event].append(func)
+
+        # Call 'on_bound' callback
+        if on_bound:
+            self.__call(on_bound, kwargs={
+                'func': func
+            })
+
+        return self
+
+    def once(self, event, func=None):
+        if not func:
+            # assume decorator, wrap
+            return self.__wrap(self.once, event)
+
+        self.__log('once(event: %s, func: %s)', repr(event), repr(func))
+
+        def once_callback(*args, **kwargs):
+            self.off(event, once_callback)
+            func(*args, **kwargs)
+
+        self.on(event, once_callback)
+
+        return self
+
+    def off(self, event=None, func=None):
+        self.__log('off(event: %s, func: %s)', repr(event), repr(func))
+
+        self.__ensure_constructed()
+
+        if event and event not in self.__callbacks:
+            return self
+
+        if func and func not in self.__callbacks[event]:
+            return self
+
+        if event and func:
+            self.__callbacks[event].remove(func)
+        elif event:
+            self.__callbacks[event] = []
+        elif func:
+            raise ValueError('"event" is required if "func" is specified')
+        else:
+            self.__callbacks = {}
+
+        return self
+
+    def emit(self, event, *args, **kwargs):
+        suppress = kwargs.pop('__suppress', False)
+
+        if not suppress:
+            self.__log('emit(event: %s, args: %s, kwargs: %s)', repr(event), repr_trim(args), repr_trim(kwargs))
+
+        self.__ensure_constructed()
+
+        if event not in self.__callbacks:
+            return
+
+        for callback in list(self.__callbacks[event]):
+            self.__call(callback, args, kwargs, event)
+
+        return self
+
+    def emit_on(self, event, *args, **kwargs):
+        func = kwargs.pop('func', None)
+
+        if not func:
+            # assume decorator, wrap
+            return self.__wrap(self.emit_on, event, *args, **kwargs)
+
+        self.__log('emit_on(event: %s, func: %s, args: %s, kwargs: %s)', repr(event), repr(func), repr(args), repr(kwargs))
+
+        # Bind func from wrapper
+        self.on(event, func)
+
+        # Emit event (calling 'func')
+        self.emit(event, *args, **kwargs)
+
+    def pipe(self, events, other):
+        if type(events) is not list:
+            events = [events]
+
+        self.__log('pipe(events: %s, other: %s)', repr(events), repr(other))
+
+        self.__ensure_constructed()
+
+        for event in events:
+            self.on(event, PipeHandler(event, other.emit))
+
+        return self
+
+    def __call(self, callback, args=None, kwargs=None, event=None):
+        args = args or ()
+        kwargs = kwargs or {}
+
+        if self.threading:
+            return self.__call_async(callback, args, kwargs, event)
+
+        return self.__call_sync(callback, args, kwargs, event)
+
+    @classmethod
+    def __call_sync(cls, callback, args=None, kwargs=None, event=None):
+        try:
+            callback(*args, **kwargs)
+            return True
+        except Exception as ex:
+            log.warn('[%s] Exception raised in: %s - %s' % (event, cls.__function_name(callback), ex), exc_info=True)
+            return False
+
+    def __call_async(self, callback, args=None, kwargs=None, event=None):
+        self.__threading_pool.submit(self.__call_sync, callback, args, kwargs, event)
+
+    @staticmethod
+    def __function_name(func):
+        fragments = []
+
+        # Try append class name
+        cls = getattr(func, 'im_class', None)
+
+        if cls and hasattr(cls, '__name__'):
+            fragments.append(cls.__name__)
+
+        # Append function name
+        fragments.append(func.__name__)
+
+        return '.'.join(fragments)
+
+
+class PipeHandler(object):
+    def __init__(self, event, callback):
+        self.event = event
+        self.callback = callback
+
+    def __call__(self, *args, **kwargs):
+        self.callback(self.event, *args, **kwargs)
+
+
+def on(emitter, event, func=None):
+    emitter.on(event, func)
+
+    return {
+        'destroy': lambda: emitter.off(event, func)
+    }
+
+
+def once(emitter, event, func=None):
+    return emitter.once(event, func)
+
+
+def off(emitter, event, func=None):
+    return emitter.off(event, func)
+
+
+def emit(emitter, event, *args, **kwargs):
+    return emitter.emit(event, *args, **kwargs)
+
+
+def repr_trim(value, length=1000):
+    value = repr(value)
+
+    if len(value) < length:
+        return value
+
+    return '<%s - %s characters>' % (type(value).__name__, len(value))

Contents/Libraries/Shared/pyga/__init__.py

@@ -0,0 +1,8 @@
+from pyga.requests import Q
+
+def shutdown():
+    '''
+    Fire all stored GIF requests One by One.
+    You should call this if you set Config.queue_requests = True
+    '''
+    map(lambda func: func(), Q.REQ_ARRAY)

Contents/Libraries/Shared/pyga/entities.py

@@ -0,0 +1,506 @@
+# -*- coding: utf-8 -*-
+
+from datetime import datetime
+from operator import itemgetter
+import six
+from pyga import utils
+from pyga import exceptions
+
+__author__ = "Arun KR (kra3) <the1.arun@gmail.com>"
+__license__ = "Simplified BSD"
+
+
+class Campaign(object):
+    '''
+    A representation of Campaign
+
+    Properties:
+    _type -- See TYPE_* constants, will be mapped to "__utmz" parameter.
+    creation_time --  Time of the creation of this campaign, will be mapped to "__utmz" parameter.
+    response_count -- Response Count, will be mapped to "__utmz" parameter.
+        Is also used to determine whether the campaign is new or repeated,
+        which will be mapped to "utmcn" and "utmcr" parameters.
+    id -- Campaign ID, a.k.a. "utm_id" query parameter for ga.js
+           Will be mapped to "__utmz" parameter.
+    source -- Source, a.k.a. "utm_source" query parameter for ga.js.
+              Will be mapped to "utmcsr" key in "__utmz" parameter.
+    g_click_id -- Google AdWords Click ID, a.k.a. "gclid" query parameter for ga.js.
+                  Will be mapped to "utmgclid" key in "__utmz" parameter.
+    d_click_id -- DoubleClick (?) Click ID. Will be mapped to "utmdclid" key in "__utmz" parameter.
+    name --  Name, a.k.a. "utm_campaign" query parameter for ga.js.
+             Will be mapped to "utmccn" key in "__utmz" parameter.
+    medium -- Medium, a.k.a. "utm_medium" query parameter for ga.js.
+              Will be mapped to "utmcmd" key in "__utmz" parameter.
+    term -- Terms/Keywords, a.k.a. "utm_term" query parameter for ga.js.
+            Will be mapped to "utmctr" key in "__utmz" parameter.
+    content -- Ad Content Description, a.k.a. "utm_content" query parameter for ga.js.
+               Will be mapped to "utmcct" key in "__utmz" parameter.
+
+    '''
+
+    TYPE_DIRECT = 'direct'
+    TYPE_ORGANIC = 'organic'
+    TYPE_REFERRAL = 'referral'
+
+    CAMPAIGN_DELIMITER = '|'
+
+    UTMZ_PARAM_MAP = {
+        'utmcid': 'id',
+        'utmcsr': 'source',
+        'utmgclid': 'g_click_id',
+        'utmdclid': 'd_click_id',
+        'utmccn': 'name',
+        'utmcmd': 'medium',
+        'utmctr': 'term',
+        'utmcct': 'content',
+    }
+
+    def __init__(self, typ):
+        self._type = None
+        self.creation_time = None
+        self.response_count = 0
+        self.id = None
+        self.source = None
+        self.g_click_id = None
+        self.d_click_id = None
+        self.name = None
+        self.medium = None
+        self.term = None
+        self.content = None
+
+        if typ:
+            if typ not in ('direct', 'organic', 'referral'):
+                raise ValueError('Campaign type has to be one of the Campaign::TYPE_* constant values.')
+
+            self._type = typ
+            if typ == Campaign.TYPE_DIRECT:
+                self.name = '(direct)'
+                self.source = '(direct)'
+                self.medium = '(none)'
+            elif typ == Campaign.TYPE_REFERRAL:
+                self.name = '(referral)'
+                self.medium = 'referral'
+            elif typ == Campaign.TYPE_ORGANIC:
+                self.name = '(organic)'
+                self.medium = 'organic'
+            else:
+                self._type = None
+
+        self.creation_time = datetime.utcnow()
+
+    def validate(self):
+        if not self.source:
+            raise exceptions.ValidationError('Campaigns need to have at least the "source" attribute defined.')
+
+    @staticmethod
+    def create_from_referrer(url):
+        obj = Campaign(Campaign.TYPE_REFERRAL)
+        parse_rslt = six.moves.urllib.parse.urlparse(url)
+        obj.source = parse_rslt.netloc
+        obj.content = parse_rslt.path
+        return obj
+
+    def extract_from_utmz(self, utmz):
+        parts = utmz.split('.', 4)
+
+        if len(parts) != 5:
+            raise ValueError('The given "__utmz" cookie value is invalid.')
+
+        self.creation_time = utils.convert_ga_timestamp(parts[1])
+        self.response_count = int(parts[3])
+        params = parts[4].split(Campaign.CAMPAIGN_DELIMITER)
+
+        for param in params:
+            key, val = param.split('=')
+
+            try:
+                setattr(self, self.UTMZ_PARAM_MAP[key], six.moves.urllib.parse.unquote_plus(val))
+            except KeyError:
+                continue
+
+        return self
+
+
+class CustomVariable(object):
+    '''
+    Represent a Custom Variable
+
+    Properties:
+    index -- Is the slot, you have 5 slots
+    name -- Name given to custom variable
+    value -- Value for the variable
+    scope -- Scope can be any one of 1, 2 or 3.
+
+    WATCH OUT: It's a known issue that GA will not decode URL-encoded
+    characters in custom variable names and values properly, so spaces
+    will show up as "%20" in the interface etc. (applicable to name & value)
+    http://www.google.com/support/forum/p/Google%20Analytics/thread?tid=2cdb3ec0be32e078
+
+    '''
+
+    SCOPE_VISITOR = 1
+    SCOPE_SESSION = 2
+    SCOPE_PAGE = 3
+
+    def __init__(self, index=None, name=None, value=None, scope=3):
+        self.index = index
+        self.name = name
+        self.value = value
+        self.scope = CustomVariable.SCOPE_PAGE
+        if scope:
+            self.scope = scope
+
+    def __setattr__(self, name, value):
+        if name == 'scope':
+            if value and value not in range(1, 4):
+                raise ValueError('Custom Variable scope has to be one of the 1,2 or 3')
+
+        if name == 'index':
+            # Custom Variables are limited to five slots officially, but there seems to be a
+            # trick to allow for more of them which we could investigate at a later time (see
+            # http://analyticsimpact.com/2010/05/24/get-more-than-5-custom-variables-in-google-analytics/
+            if value and (value < 0 or value > 5):
+                raise ValueError('Custom Variable index has to be between 1 and 5.')
+
+        object.__setattr__(self, name, value)
+
+    def validate(self):
+        '''
+        According to the GA documentation, there is a limit to the combined size of
+        name and value of 64 bytes after URL encoding,
+        see http://code.google.com/apis/analytics/docs/tracking/gaTrackingCustomVariables.html#varTypes
+        and http://xahlee.org/js/google_analytics_tracker_2010-07-01_expanded.js line 563
+        This limit was increased to 128 bytes BEFORE encoding with the 2012-01 release of ga.js however,
+        see http://code.google.com/apis/analytics/community/gajs_changelog.html
+        '''
+        if len('%s%s' % (self.name, self.value)) > 128:
+            raise exceptions.ValidationError('Custom Variable combined name and value length must not be larger than 128 bytes.')
+
+
+class Event(object):
+    '''
+    Represents an Event
+    https://developers.google.com/analytics/devguides/collection/gajs/eventTrackerGuide
+
+    Properties:
+    category -- The general event category
+    action -- The action for the event
+    label -- An optional descriptor for the event
+    value -- An optional value associated with the event. You can see your
+             event values in the Overview, Categories, and Actions reports,
+             where they are listed by event or aggregated across events,
+             depending upon your report view.
+    noninteraction -- By default, event hits will impact a visitor's bounce rate.
+                      By setting this parameter to true, this event hit
+                      will not be used in bounce rate calculations.
+                      (default False)
+    '''
+
+    def __init__(self, category=None, action=None, label=None, value=None, noninteraction=False):
+        self.category = category
+        self.action = action
+        self.label = label
+        self.value = value
+        self.noninteraction = bool(noninteraction)
+
+        if self.noninteraction and not self.value:
+            self.value = 0
+
+    def validate(self):
+        if not(self.category and self.action):
+            raise exceptions.ValidationError('Events, at least need to have a category and action defined.')
+
+
+class Item(object):
+    '''
+    Represents an Item in Transaction
+
+    Properties:
+    order_id -- Order ID, will be mapped to "utmtid" parameter
+    sku -- Product Code. This is the sku code for a given product, will be mapped to "utmipc" parameter
+    name -- Product Name, will be mapped to "utmipn" parameter
+    variation -- Variations on an item, will be mapped to "utmiva" parameter
+    price -- Unit Price. Value is set to numbers only, will be mapped to "utmipr" parameter
+    quantity -- Unit Quantity, will be mapped to "utmiqt" parameter
+
+    '''
+
+    def __init__(self):
+        self.order_id = None
+        self.sku = None
+        self.name = None
+        self.variation = None
+        self.price = None
+        self.quantity = 1
+
+    def validate(self):
+        if not self.sku:
+            raise exceptions.ValidationError('sku/product is a required parameter')
+
+
+class Page(object):
+    '''
+    Contains all parameters needed for tracking a page
+
+    Properties:
+    path -- Page request URI, will be mapped to "utmp" parameter
+    title -- Page title, will be mapped to "utmdt" parameter
+    charset -- Charset encoding, will be mapped to "utmcs" parameter
+    referrer -- Referer URL, will be mapped to "utmr" parameter
+    load_time -- Page load time in milliseconds, will be encoded into "utme" parameter.
+
+    '''
+    REFERRER_INTERNAL = '0'
+
+    def __init__(self, path):
+        self.path = None
+        self.title = None
+        self.charset = None
+        self.referrer = None
+        self.load_time = None
+
+        if path:
+            self.path = path
+
+    def __setattr__(self, name, value):
+        if name == 'path':
+            if value and value != '':
+                if value[0] != '/':
+                    raise ValueError('The page path should always start with a slash ("/").')
+        elif name == 'load_time':
+            if value and not isinstance(value, int):
+                raise ValueError('Page load time must be specified in integer milliseconds.')
+
+        object.__setattr__(self, name, value)
+
+
+class Session(object):
+    '''
+    You should serialize this object and store it in the user session to keep it
+    persistent between requests (similar to the "__umtb" cookie of the GA Javascript client).
+
+    Properties:
+    session_id -- A unique per-session ID, will be mapped to "utmhid" parameter
+    track_count -- The amount of pageviews that were tracked within this session so far,
+                   will be part of the "__utmb" cookie parameter.
+                   Will get incremented automatically upon each request
+    start_time -- Timestamp of the start of this new session, will be part of the "__utmb" cookie parameter
+
+    '''
+    def __init__(self):
+        self.session_id = utils.get_32bit_random_num()
+        self.track_count = 0
+        self.start_time = datetime.utcnow()
+
+    @staticmethod
+    def generate_session_id():
+        return utils.get_32bit_random_num()
+
+    def extract_from_utmb(self, utmb):
+        '''
+        Will extract information for the "trackCount" and "startTime"
+        properties from the given "__utmb" cookie value.
+        '''
+        parts = utmb.split('.')
+        if len(parts) != 4:
+            raise ValueError('The given "__utmb" cookie value is invalid.')
+
+        self.track_count = int(parts[1])
+        self.start_time = utils.convert_ga_timestamp(parts[3])
+
+        return self
+
+
+class SocialInteraction(object):
+    '''
+
+    Properties:
+    action -- Required. A string representing the social action being tracked,
+              will be mapped to "utmsa" parameter
+    network -- Required. A string representing the social network being tracked,
+               will be mapped to "utmsn" parameter
+    target -- Optional. A string representing the URL (or resource) which receives the action.
+
+    '''
+
+    def __init__(self, action=None, network=None, target=None):
+        self.action = action
+        self.network = network
+        self.target = target
+
+    def validate(self):
+        if not(self.action and self.network):
+            raise exceptions.ValidationError('Social interactions need to have at least the "network" and "action" attributes defined.')
+
+
+class Transaction(object):
+    '''
+    Represents parameters for a Transaction call
+
+    Properties:
+    order_id -- Order ID, will be mapped to "utmtid" parameter
+    affiliation -- Affiliation, Will be mapped to "utmtst" parameter
+    total -- Total Cost, will be mapped to "utmtto" parameter
+    tax -- Tax Cost, will be mapped to "utmttx" parameter
+    shipping -- Shipping Cost, values as for unit and price, will be mapped to "utmtsp" parameter
+    city -- Billing City, will be mapped to "utmtci" parameter
+    state -- Billing Region, will be mapped to "utmtrg" parameter
+    country -- Billing Country, will be mapped to "utmtco" parameter
+    items -- @entity.Items in a transaction
+
+    '''
+    def __init__(self):
+        self.items = []
+        self.order_id = None
+        self.affiliation = None
+        self.total = None
+        self.tax = None
+        self.shipping = None
+        self.city = None
+        self.state = None
+        self.country = None
+
+    def __setattr__(self, name, value):
+        if name == 'order_id':
+            for itm in self.items:
+                itm.order_id = value
+        object.__setattr__(self, name, value)
+
+    def validate(self):
+        if len(self.items) == 0:
+            raise exceptions.ValidationError('Transaction need to consist of at least one item')
+
+    def add_item(self, item):
+        ''' item of type entities.Item '''
+        if isinstance(item, Item):
+            item.order_id = self.order_id
+            self.items.append(item)
+
+
+class Visitor(object):
+    '''
+    You should serialize this object and store it in the user database to keep it
+    persistent for the same user permanently (similar to the "__umta" cookie of
+    the GA Javascript client).
+
+    Properties:
+    unique_id -- Unique user ID, will be part of the "__utma" cookie parameter
+    first_visit_time -- Time of the very first visit of this user, will be part of the "__utma" cookie parameter
+    previous_visit_time -- Time of the previous visit of this user, will be part of the "__utma" cookie parameter
+    current_visit_time -- Time of the current visit of this user, will be part of the "__utma" cookie parameter
+    visit_count -- Amount of total visits by this user, will be part of the "__utma" cookie parameter
+    ip_address -- IP Address of the end user, will be mapped to "utmip" parameter and "X-Forwarded-For" request header
+    user_agent -- User agent string of the end user, will be mapped to "User-Agent" request header
+    locale -- Locale string (country part optional) will be mapped to "utmul" parameter
+    flash_version -- Visitor's Flash version, will be maped to "utmfl" parameter
+    java_enabled -- Visitor's Java support, will be mapped to "utmje" parameter
+    screen_colour_depth -- Visitor's screen color depth, will be mapped to "utmsc" parameter
+    screen_resolution -- Visitor's screen resolution, will be mapped to "utmsr" parameter
+    '''
+    def __init__(self):
+        now = datetime.utcnow()
+
+        self.unique_id = None
+        self.first_visit_time = now
+        self.previous_visit_time = now
+        self.current_visit_time = now
+        self.visit_count = 1
+        self.ip_address = None
+        self.user_agent = None
+        self.locale = None
+        self.flash_version = None
+        self.java_enabled = None
+        self.screen_colour_depth = None
+        self.screen_resolution = None
+
+    def __setattr__(self, name, value):
+        if name == 'unique_id':
+            if value and (value < 0 or value > 0x7fffffff):
+                raise ValueError('Visitor unique ID has to be a 32-bit integer between 0 and 0x7fffffff')
+        object.__setattr__(self, name, value)
+
+    def __getattribute__(self, name):
+        if name == 'unique_id':
+            tmp = object.__getattribute__(self, name)
+            if tmp is None:
+                self.unique_id = self.generate_unique_id()
+        return object.__getattribute__(self, name)
+
+    def __getstate__(self):
+        state = self.__dict__
+        if state.get('user_agent') is None:
+            state['unique_id'] = self.generate_unique_id()
+
+        return state
+
+    def extract_from_utma(self, utma):
+        '''
+        Will extract information for the "unique_id", "first_visit_time", "previous_visit_time",
+        "current_visit_time" and "visit_count" properties from the given "__utma" cookie value.
+        '''
+        parts = utma.split('.')
+        if len(parts) != 6:
+            raise ValueError('The given "__utma" cookie value is invalid.')
+
+        self.unique_id = int(parts[1])
+        self.first_visit_time = utils.convert_ga_timestamp(parts[2])
+        self.previous_visit_time = utils.convert_ga_timestamp(parts[3])
+        self.current_visit_time = utils.convert_ga_timestamp(parts[4])
+        self.visit_count = int(parts[5])
+
+        return self
+
+    def extract_from_server_meta(self, meta):
+        '''
+        Will extract information for the "ip_address", "user_agent" and "locale"
+        properties from the given WSGI REQUEST META variable or equivalent.
+        '''
+        if 'REMOTE_ADDR' in meta and meta['REMOTE_ADDR']:
+            ip = None
+            for key in ('HTTP_X_FORWARDED_FOR', 'REMOTE_ADDR'):
+                if key in meta and not ip:
+                    ips = meta.get(key, '').split(',')
+                    ip = ips[-1].strip()
+                    if not utils.is_valid_ip(ip):
+                        ip = ''
+                    if utils.is_private_ip(ip):
+                        ip = ''
+            if ip:
+                self.ip_address = ip
+
+        if 'HTTP_USER_AGENT' in meta and meta['HTTP_USER_AGENT']:
+            self.user_agent = meta['HTTP_USER_AGENT']
+
+        if 'HTTP_ACCEPT_LANGUAGE' in meta and meta['HTTP_ACCEPT_LANGUAGE']:
+            user_locals = []
+            matched_locales = utils.validate_locale(meta['HTTP_ACCEPT_LANGUAGE'])
+            if matched_locales:
+                lang_lst = map((lambda x: x.replace('-', '_')), (i[1] for i in matched_locales))
+                quality_lst = map((lambda x: x and x or 1), (float(i[4] and i[4] or '0') for i in matched_locales))
+                lang_quality_map = map((lambda x, y: (x, y)), lang_lst, quality_lst)
+                user_locals = [x[0] for x in sorted(lang_quality_map, key=itemgetter(1), reverse=True)]
+
+            if user_locals:
+                self.locale = user_locals[0]
+
+        return self
+
+    def generate_hash(self):
+        '''Generates a hashed value from user-specific properties.'''
+        tmpstr = "%s%s%s" % (self.user_agent, self.screen_resolution, self.screen_colour_depth)
+        return utils.generate_hash(tmpstr)
+
+    def generate_unique_id(self):
+        '''Generates a unique user ID from the current user-specific properties.'''
+        return ((utils.get_32bit_random_num() ^ self.generate_hash()) & 0x7fffffff)
+
+    def add_session(self, session):
+        '''
+        Updates the "previousVisitTime", "currentVisitTime" and "visitCount"
+        fields based on the given session object.
+        '''
+        start_time = session.start_time
+        if start_time != self.current_visit_time:
+            self.previous_visit_time = self.current_visit_time
+            self.current_visit_time = start_time
+            self.visit_count = self.visit_count + 1

Contents/Libraries/Shared/pyga/exceptions.py

@@ -0,0 +1,2 @@
+class ValidationError(Exception):
+    pass

Contents/Libraries/Shared/pyga/utils.py

@@ -0,0 +1,116 @@
+# -*- coding: utf-8 -*-
+
+import logging
+from random import randint
+import re
+import six
+import os
+from datetime import datetime
+
+__author__ = "Arun KR (kra3) <the1.arun@gmail.com>"
+__license__ = "Simplified BSD"
+
+RE_IP = re.compile(r'^[\d+]{1,3}\.[\d+]{1,3}\.[\d+]{1,3}\.[\d+]{1,3}$', re.I)
+RE_PRIV_IP = re.compile(r'^(?:127\.0\.0\.1|10\.|192\.168\.|172\.(?:1[6-9]|2[0-9]|3[0-1])\.)')
+RE_LOCALE = re.compile(r'(^|\s*,\s*)([a-zA-Z]{1,8}(-[a-zA-Z]{1,8})*)\s*(;\s*q\s*=\s*(1(\.0{0,3})?|0(\.[0-9]{0,3})))?', re.I)
+RE_GA_ACCOUNT_ID = re.compile(r'^(UA|MO)-[0-9]*-[0-9]*$')
+RE_FIRST_THREE_OCTETS_OF_IP = re.compile(r'^((\d{1,3}\.){3})\d{1,3}$')
+
+def convert_ga_timestamp(timestamp_string):
+    timestamp = float(timestamp_string)
+    if timestamp > ((2 ** 31) - 1):
+        timestamp /= 1000
+    return datetime.utcfromtimestamp(timestamp)
+
+def get_32bit_random_num():
+    return randint(0, 0x7fffffff)
+
+def is_valid_ip(ip):
+    return True if RE_IP.match(str(ip)) else False
+
+def is_private_ip(ip):
+    return True if RE_PRIV_IP.match(str(ip)) else False
+
+def validate_locale(locale):
+    return RE_LOCALE.findall(str(locale))
+
+def is_valid_google_account(account):
+    return True if RE_GA_ACCOUNT_ID.match(str(account)) else False
+
+def generate_hash(tmpstr):
+    hash_val = 1
+
+    if tmpstr:
+        hash_val = 0
+        for ordinal in map(ord, tmpstr[::-1]):
+            hash_val = ((hash_val << 6) & 0xfffffff) + ordinal + (ordinal << 14)
+            left_most_7 = hash_val & 0xfe00000
+            if left_most_7 != 0:
+                hash_val ^= left_most_7 >> 21
+
+    return hash_val
+
+def anonymize_ip(ip):
+    if ip:
+        match = RE_FIRST_THREE_OCTETS_OF_IP.findall(str(ip))
+        if match:
+            return '%s%s' % (match[0][0], '0')
+
+    return ''
+
+def encode_uri_components(value):
+    '''Mimics Javascript's encodeURIComponent() function for consistency with the GA Javascript client.'''
+    return convert_to_uri_component_encoding(six.moves.urllib.parse.quote(value))
+
+def convert_to_uri_component_encoding(value):
+    return value.replace('%21', '!').replace('%2A', '*').replace('%27', "'").replace('%28', '(').replace('%29', ')')
+
+# Taken from expicient.com BJs repo.
+def stringify(s, stype=None, fn=None):
+    ''' Converts elements of a complex data structure to strings
+
+    The data structure can be a multi-tiered one - with tuples and lists etc
+    This method will loop through each and convert everything to string.
+    For example - it can be -
+    [[{'a1': {'a2': {'a3': ('a4', timedelta(0, 563)), 'a5': {'a6': datetime()}}}}]]
+    which will be converted to -
+    [[{'a1': {'a2': {'a3': ('a4', '0:09:23'), 'a5': {'a6': '2009-05-27 16:19:52.401500' }}}}]]
+
+    @param stype: If only one type of data element needs to be converted to
+        string without affecting others, stype can be used.
+        In the earlier example, if it is called with stringify(s, stype=datetime.timedelta)
+        the result would be
+        [[{'a1': {'a2': {'a3': ('a4', '0:09:23'), 'a5': {'a6': datetime() }}}}]]
+
+    Also, even though the name is stringify, any function can be run on it, based on
+    parameter fn. If fn is None, it will be stringified.
+
+    '''
+
+    if type(s) in [list, set, dict, tuple]:
+        if isinstance(s, dict):
+            for k in s:
+                s[k] = stringify(s[k], stype, fn)
+        elif type(s) in [list, set]:
+            for i, k in enumerate(s):
+                s[i] = stringify(k, stype, fn)
+        else: #tuple
+            tmp = []
+            for k in s:
+                tmp.append(stringify(k, stype, fn))
+            s = tuple(tmp)
+    else:
+        if fn:
+            if not stype or (stype == type(s)):
+                return fn(s)
+        else:
+            # To do str(s). But, str() can fail on unicode. So, use .encode instead
+            if not stype or (stype == type(s)):
+                try:
+                    return six.text_type(s)
+                    #return s.encode('ascii', 'replace')
+                except AttributeError:
+                    return str(s)
+                except UnicodeDecodeError:
+                    return s.decode('ascii', 'replace')
+    return s

Contents/Libraries/Shared/subliminal_patch/__init__.py

@@ -4,28 +4,38 @@ import subliminal
 import babelfish
 import logging
 
-# patch subliminal's subtitle encoding detection
+# patch subliminal's subtitle and provider base
 from .patch_subtitle import PatchedSubtitle
+from .patch_providers import PatchedProvider
 subliminal.subtitle.Subtitle = PatchedSubtitle
-from subliminal.providers.addic7ed import Addic7edSubtitle
-from subliminal.providers.podnapisi import PodnapisiSubtitle
-from subliminal.providers.tvsubtitles import TVsubtitlesSubtitle
-from subliminal.providers.opensubtitles import OpenSubtitlesSubtitle
+subliminal.providers.Provider = PatchedProvider
+from subliminal.providers.addic7ed import Addic7edSubtitle, Addic7edProvider
+from subliminal.providers.podnapisi import PodnapisiSubtitle, PodnapisiProvider
+from subliminal.providers.tvsubtitles import TVsubtitlesSubtitle, TVsubtitlesProvider
+from subliminal.providers.opensubtitles import OpenSubtitlesSubtitle, OpenSubtitlesProvider
+
+# add our patched base classes
 setattr(Addic7edSubtitle, "__bases__", (PatchedSubtitle,))
 setattr(PodnapisiSubtitle, "__bases__", (PatchedSubtitle,))
 setattr(TVsubtitlesSubtitle, "__bases__", (PatchedSubtitle,))
 setattr(OpenSubtitlesSubtitle, "__bases__", (PatchedSubtitle,))
+setattr(Addic7edProvider, "__bases__", (PatchedProvider,))
+setattr(PodnapisiProvider, "__bases__", (PatchedProvider,))
+setattr(TVsubtitlesProvider, "__bases__", (PatchedProvider,))
+setattr(OpenSubtitlesProvider, "__bases__", (PatchedProvider,))
 
 from .patch_provider_pool import PatchedProviderPool
 from .patch_video import patched_search_external_subtitles, scan_video
 from .patch_providers import addic7ed, podnapisi, tvsubtitles, opensubtitles
-from .patch_api import save_subtitles
+from .patch_api import save_subtitles, list_all_subtitles, download_subtitles
 
 # patch subliminal's ProviderPool
 subliminal.api.ProviderPool = PatchedProviderPool
 
-# patch subliminal's save_subtitles function
+# patch subliminal's functions
 subliminal.api.save_subtitles = save_subtitles
+subliminal.api.list_all_subtitles = list_all_subtitles
+subliminal.api.download_subtitles = download_subtitles
 
 # patch subliminal's subtitle classes
 def subtitleRepr(self):
@@ -55,6 +65,4 @@ subliminal.video.search_external_subtitles = patched_search_external_subtitles
 # patch subliminal's scan_video function
 subliminal.video.scan_video = scan_video
 
-subliminal.video.Episode.scores["boost"] = 40
-
 subliminal.video.Episode.scores["title"] = 0

Contents/Libraries/Shared/subliminal_patch/patch_api.py

@@ -2,13 +2,83 @@
 import os
 import logging
 from bs4 import UnicodeDammit
-from subliminal.api import get_subtitle_path, io
-from subzero.lib.io import get_viable_encoding
+from subliminal.api import io, defaultdict
+from subliminal_patch.patch_provider_pool import PatchedProviderPool
 
 logger = logging.getLogger(__name__)
 
 
-def save_subtitles(video, subtitles, single=False, directory=None, encoding=None, encode_with=None):
+def download_subtitles(subtitles, **kwargs):
+    """Download :attr:`~subliminal.subtitle.Subtitle.content` of `subtitles`.
+
+    All other parameters are passed onwards to the :class:`ProviderPool` constructor.
+
+    :param subtitles: subtitles to download.
+    :type subtitles: list of :class:`~subliminal.subtitle.Subtitle`
+
+    """
+    with PatchedProviderPool(**kwargs) as pool:
+        for subtitle in subtitles:
+            logger.info('Downloading subtitle %r', subtitle)
+            pool.download_subtitle(subtitle)
+
+
+def list_all_subtitles(videos, languages, **kwargs):
+    """List all available subtitles.
+
+    The `videos` must pass the `languages` check of :func:`check_video`.
+
+    All other parameters are passed onwards to the :class:`ProviderPool` constructor.
+
+    :param videos: videos to list subtitles for.
+    :type videos: set of :class:`~subliminal.video.Video`
+    :param languages: languages to search for.
+    :type languages: set of :class:`~babelfish.language.Language`
+    :return: found subtitles per video.
+    :rtype: dict of :class:`~subliminal.video.Video` to list of :class:`~subliminal.subtitle.Subtitle`
+
+    """
+    listed_subtitles = defaultdict(list)
+
+    # return immediatly if no video passed the checks
+    if not videos:
+        return listed_subtitles
+
+    # list subtitles
+    with PatchedProviderPool(**kwargs) as pool:
+        for video in videos:
+            logger.info('Listing subtitles for %r', video)
+            subtitles = pool.list_subtitles(video, languages - video.subtitle_languages)
+            listed_subtitles[video].extend(subtitles)
+            logger.info('Found %d subtitle(s)', len(subtitles))
+
+    return listed_subtitles
+
+
+def get_subtitle_path(video_path, language=None, extension='.srt', forced_tag=False):
+    """Get the subtitle path using the `video_path` and `language`.
+
+    :param str video_path: path to the video.
+    :param language: language of the subtitle to put in the path.
+    :type language: :class:`~babelfish.language.Language`
+    :param str extension: extension of the subtitle.
+    :return: path of the subtitle.
+    :rtype: str
+
+    """
+    subtitle_root = os.path.splitext(video_path)[0]
+
+    if language:
+        subtitle_root += '.' + str(language)
+
+    if forced_tag:
+        subtitle_root += ".forced"
+
+    return subtitle_root + extension
+
+
+def save_subtitles(video, subtitles, single=False, directory=None, encoding=None, encode_with=None, chmod=None,
+                   forced_tag=False, path_decoder=None):
     """Save subtitles on filesystem.
 
     Subtitles are saved in the order of the list. If a subtitle with a language has already been saved, other subtitles
@@ -42,10 +112,13 @@ def save_subtitles(video, subtitles, single=False, directory=None, encoding=None
             continue
 
         # create subtitle path
-        subtitle_path = get_subtitle_path(video.name, None if single else subtitle.language)
+        subtitle_path = get_subtitle_path(video.name, None if single else subtitle.language, forced_tag=forced_tag)
         if directory is not None:
             subtitle_path = os.path.join(directory, os.path.split(subtitle_path)[1])
 
+        if path_decoder:
+            subtitle_path = path_decoder(subtitle_path)
+
         # force unicode
         subtitle_path = UnicodeDammit(subtitle_path).unicode_markup
 
@@ -64,6 +137,10 @@ def save_subtitles(video, subtitles, single=False, directory=None, encoding=None
             with io.open(subtitle_path, 'wb') as f:
                 f.write(content)
 
+            # change chmod if requested
+            if chmod:
+                os.chmod(subtitle_path, chmod)
+
             if single:
                 break
             continue
@@ -73,6 +150,10 @@ def save_subtitles(video, subtitles, single=False, directory=None, encoding=None
             with io.open(subtitle_path, 'w', encoding=encoding) as f:
                 f.write(subtitle.text)
 
+        # change chmod if requested
+        if chmod:
+            os.chmod(subtitle_path, chmod)
+
         saved_subtitles.append(subtitle)
 
         # check single

Contents/Libraries/Shared/subliminal_patch/patch_provider_pool.py

@@ -212,15 +212,13 @@ class PatchedProviderPool(ProviderPool):
             tries += 1
             try:
                 self[subtitle.provider_name].download_subtitle(subtitle)
+                break
             except (requests.Timeout, socket.timeout):
                 logger.error('Provider %r timed out', subtitle.provider_name)
             except ProviderError:
                 logger.error('Unexpected error in provider %r, Traceback: %s', subtitle.provider_name, traceback.format_exc())
-                break
             except:
                 logger.exception('Unexpected error in provider %r, Traceback: %s', subtitle.provider_name, traceback.format_exc())
-            else:
-                break
 
             if tries == DOWNLOAD_TRIES:
                 self.discarded_providers.add(subtitle.provider_name)

Contents/Libraries/Shared/subliminal_patch/patch_providers/__init__.py

@@ -0,0 +1,6 @@
+# coding=utf-8
+from subliminal import Provider
+
+
+class PatchedProvider(Provider):
+    pass

Contents/Libraries/Shared/subliminal_patch/patch_providers/addic7ed.py

@@ -2,10 +2,11 @@
 
 import logging
 import re
+import subliminal
 from random import randint
 from subliminal.providers.addic7ed import Addic7edProvider, Addic7edSubtitle, ParserBeautifulSoup, Language
 from subliminal.cache import SHOW_EXPIRATION_TIME, region
-from .mixins import PunctuationMixin
+from .mixins import PunctuationMixin, ProviderRetryMixin
 
 logger = logging.getLogger(__name__)
 
@@ -16,21 +17,24 @@ USE_BOOST = False
 
 
 class PatchedAddic7edSubtitle(Addic7edSubtitle):
-    def __init__(self, *args, **kwargs):
-        super(PatchedAddic7edSubtitle, self).__init__(*args, **kwargs)
+    def __init__(self, language, hearing_impaired, page_link, series, season, episode, title, year, version,
+                 download_link):
+        super(PatchedAddic7edSubtitle, self).__init__(language, hearing_impaired, page_link, series, season, episode,
+                                                      title, year, version, download_link)
+        self.release_info = version
 
     def get_matches(self, video, hearing_impaired=False):
         matches = super(PatchedAddic7edSubtitle, self).get_matches(video, hearing_impaired=hearing_impaired)
-        if not USE_BOOST:
+        if not subliminal.video.Episode.scores["addic7ed_boost"]:
             return matches
 
         if {"series", "season", "episode", "year"}.issubset(matches) and "format" in matches:
-            matches.add("boost")
-            logger.info("Boosting Addic7ed subtitle")
+            matches.add("addic7ed_boost")
+            logger.info("Boosting Addic7ed subtitle by %s" % subliminal.video.Episode.scores["addic7ed_boost"])
         return matches
 
 
-class PatchedAddic7edProvider(PunctuationMixin, Addic7edProvider):
+class PatchedAddic7edProvider(PunctuationMixin, ProviderRetryMixin, Addic7edProvider):
     USE_ADDICTED_RANDOM_AGENTS = False
 
     def __init__(self, username=None, password=None, use_random_agents=False):
@@ -58,7 +62,7 @@ class PatchedAddic7edProvider(PunctuationMixin, Addic7edProvider):
         """
         # get the show page
         logger.info('Getting show ids')
-        r = self.session.get(self.server_url + 'shows.php', timeout=10)
+        r = self.retry(lambda: self.session.get(self.server_url + 'shows.php', timeout=10))
         r.raise_for_status()
         soup = ParserBeautifulSoup(r.content, ['lxml', 'html.parser'])
 
@@ -66,7 +70,11 @@ class PatchedAddic7edProvider(PunctuationMixin, Addic7edProvider):
         show_ids = {}
         for show in soup.select('td.version > h3 > a[href^="/show/"]'):
             show_clean = self.clean_punctuation(show.text.lower())
-            show_id = int(show['href'][6:])
+            try:
+                show_id = int(show['href'][6:])
+            except ValueError:
+                continue
+
             show_ids[show_clean] = show_id
             match = series_year_re.match(show_clean)
             if match.group(2) and match.group(1) not in show_ids:
@@ -140,7 +148,7 @@ class PatchedAddic7edProvider(PunctuationMixin, Addic7edProvider):
 
         # make the search
         logger.info('Searching show ids with %r', params)
-        r = self.session.get(self.server_url + 'search.php', params=params, timeout=10)
+        r = self.retry(lambda: self.session.get(self.server_url + 'search.php', params=params, timeout=10))
         r.raise_for_status()
         soup = ParserBeautifulSoup(r.content, ['lxml', 'html.parser'])
 
@@ -167,7 +175,8 @@ class PatchedAddic7edProvider(PunctuationMixin, Addic7edProvider):
 
         # get the page of the season of the show
         logger.info('Getting the page of show id %d, season %d', show_id, season)
-        r = self.session.get(self.server_url + 'show/%d' % show_id, params={'season': season}, timeout=10)
+        r = self.retry(lambda: self.session.get(self.server_url + 'show/%d' % show_id,
+                                                params={'season': season}, timeout=10))
         r.raise_for_status()
         soup = ParserBeautifulSoup(r.content, ['lxml', 'html.parser'])
 

Contents/Libraries/Shared/subliminal_patch/patch_providers/mixins.py

@@ -1,6 +1,10 @@
 # coding=utf-8
 
 import re
+import time
+import logging
+
+logger = logging.getLogger(__name__)
 
 clean_whitespace_re = re.compile(r'\s+')
 
@@ -20,3 +24,18 @@ class PunctuationMixin(object):
 
     def full_clean(self, s):
         return self.clean_whitespace(self.clean_punctuation(s))
+
+
+class ProviderRetryMixin(object):
+    def retry(self, f, amount=3, exc=Exception, retry_timeout=1):
+        i = 0
+        while i <= amount:
+            try:
+                return f()
+            except exc, e:
+                i += 1
+                if i == amount:
+                    raise
+
+            logger.debug(u"Retrying %s, try: %i/%i, exception: %s" % (self.__class__.__name__, i, amount, e))
+            time.sleep(retry_timeout)

Contents/Libraries/Shared/subliminal_patch/patch_providers/opensubtitles.py

@@ -5,7 +5,10 @@ import os
 
 from babelfish import Language
 from subliminal.exceptions import ConfigurationError
-from subliminal.providers.opensubtitles import OpenSubtitlesProvider, checked, get_version, __version__, OpenSubtitlesSubtitle, Episode
+from subliminal.providers.opensubtitles import OpenSubtitlesProvider, checked, get_version, __version__, \
+    OpenSubtitlesSubtitle, Episode, ServerProxy
+from mixins import ProviderRetryMixin
+from six.moves.xmlrpc_client import Transport
 
 logger = logging.getLogger(__name__)
 
@@ -18,6 +21,7 @@ class PatchedOpenSubtitlesSubtitle(OpenSubtitlesSubtitle):
                                                            movie_release_name, movie_year, movie_imdb_id, series_season, series_episode)
         self.query_parameters = query_parameters or {}
         self.fps = fps
+        self.release_info = movie_release_name
 
     def get_matches(self, video, hearing_impaired=False):
         matches = super(PatchedOpenSubtitlesSubtitle, self).get_matches(video, hearing_impaired=hearing_impaired)
@@ -39,26 +43,52 @@ class PatchedOpenSubtitlesSubtitle(OpenSubtitlesSubtitle):
             # treat a tag match equally to a hash match
             logger.debug("Subtitle matched by tag, treating it as a hash-match. Tag: '%s'", self.query_parameters.get("tag", None))
             matches.add("hash")
+
         return matches
 
 
-class PatchedOpenSubtitlesProvider(OpenSubtitlesProvider):
-    def __init__(self, username=None, password=None, use_tag_search=False):
+class TimeoutTransport(Transport):
+    """Timeout support for ``xmlrpc.client.SafeTransport``."""
+    def __init__(self, timeout, *args, **kwargs):
+        Transport.__init__(self, *args, **kwargs)
+        self.timeout = timeout
+
+    def make_connection(self, host):
+        c = Transport.make_connection(self, host)
+        c.timeout = self.timeout
+
+        return c
+
+
+class PatchedOpenSubtitlesProvider(ProviderRetryMixin, OpenSubtitlesProvider):
+    only_foreign = True
+
+    def __init__(self, username=None, password=None, use_tag_search=False, only_foreign=False):
         if username is not None and password is None or username is None and password is not None:
             raise ConfigurationError('Username and password must be specified')
 
         self.username = username or ''
         self.password = password or ''
         self.use_tag_search = use_tag_search
+        self.only_foreign = only_foreign
 
         if use_tag_search:
             logger.info("Using tag/exact filename search")
 
+        if only_foreign:
+            logger.info("Only searching for foreign/forced subtitles")
+
         super(PatchedOpenSubtitlesProvider, self).__init__()
+        self.server = ServerProxy('http://api.opensubtitles.org/xml-rpc', TimeoutTransport(10))
 
     def initialize(self):
         logger.info('Logging in')
-        response = checked(self.server.LogIn(self.username, self.password, 'eng', 'subliminal v%s' % get_version(__version__)))
+        # fixme: retry on SSLError
+        response = self.retry(
+            lambda: checked(
+                self.server.LogIn(self.username, self.password, 'eng', 'subliminal v%s' % get_version(__version__))
+            )
+        )
         self.token = response['token']
         logger.debug('Logged in with token %r', self.token)
 
@@ -70,6 +100,7 @@ class PatchedOpenSubtitlesProvider(OpenSubtitlesProvider):
 
          patch: query movies even if hash is known; add tag parameter
         """
+
         season = episode = None
         if isinstance(video, Episode):
             query = video.series
@@ -81,9 +112,11 @@ class PatchedOpenSubtitlesProvider(OpenSubtitlesProvider):
             query = video.title
 
         return self.query(languages, hash=video.hashes.get('opensubtitles'), size=video.size, imdb_id=video.imdb_id,
-                          query=query, season=season, episode=episode, tag=os.path.basename(video.name), use_tag_search=self.use_tag_search)
+                          query=query, season=season, episode=episode, tag=os.path.basename(video.name),
+                          use_tag_search=self.use_tag_search, only_foreign=self.only_foreign)
 
-    def query(self, languages, hash=None, size=None, imdb_id=None, query=None, season=None, episode=None, tag=None, use_tag_search=False):
+    def query(self, languages, hash=None, size=None, imdb_id=None, query=None, season=None, episode=None, tag=None,
+              use_tag_search=False, only_foreign=False):
         # fill the search criteria
         criteria = []
         if hash and size:
@@ -105,7 +138,7 @@ class PatchedOpenSubtitlesProvider(OpenSubtitlesProvider):
 
         # query the server
         logger.info('Searching subtitles %r', criteria)
-        response = checked(self.server.SearchSubtitles(self.token, criteria))
+        response = self.retry(lambda: checked(self.server.SearchSubtitles(self.token, criteria)))
         subtitles = []
 
         # exit if no data
@@ -130,6 +163,17 @@ class PatchedOpenSubtitlesProvider(OpenSubtitlesProvider):
             movie_fps = subtitle_item.get('MovieFPS')
             series_season = int(subtitle_item['SeriesSeason']) if subtitle_item['SeriesSeason'] else None
             series_episode = int(subtitle_item['SeriesEpisode']) if subtitle_item['SeriesEpisode'] else None
+            sub_file_name = subtitle_item.get('SubFileName')
+            foreign_parts_only = bool(int(subtitle_item.get('SubForeignPartsOnly', 0)))
+
+            # foreign/forced subtitles only wanted
+            if only_foreign and not foreign_parts_only:
+                continue
+
+            # foreign/forced not wanted
+            if not only_foreign and foreign_parts_only:
+                continue
+
             query_parameters = subtitle_item.get("QueryParameters")
 
             subtitle = PatchedOpenSubtitlesSubtitle(language, hearing_impaired, page_link, subtitle_id, matched_by, movie_kind,

Contents/Libraries/Shared/subliminal_patch/patch_providers/podnapisi.py

@@ -2,17 +2,60 @@
 
 import logging
 import io
+import re
+
+try:
+    from lxml import etree
+except ImportError:
+    try:
+        import xml.etree.cElementTree as etree
+    except ImportError:
+        import xml.etree.ElementTree as etree
+from babelfish import Language
 from zipfile import ZipFile
-from subliminal.providers.podnapisi import PodnapisiProvider, fix_line_ending, ProviderError
+from subliminal import Episode
+from subliminal import Movie
+from subliminal.providers.podnapisi import PodnapisiProvider, PodnapisiSubtitle, fix_line_ending, ProviderError
+from mixins import ProviderRetryMixin
 
 logger = logging.getLogger(__name__)
 
 
-class PatchedPodnapisiProvider(PodnapisiProvider):
+class PatchedPodnapisiSubtitle(PodnapisiSubtitle):
+    provider_name = 'podnapisi'
+
+    def __init__(self, language, hearing_impaired, page_link, pid, releases, title, season=None, episode=None,
+                 year=None):
+        super(PatchedPodnapisiSubtitle, self).__init__(language, hearing_impaired, page_link, pid, releases, title,
+                                                       season=season, episode=episode, year=year)
+        self.release_info = u", ".join(releases)
+
+
+class PatchedPodnapisiProvider(ProviderRetryMixin, PodnapisiProvider):
+    only_foreign = False
+
+    def __init__(self, only_foreign=False):
+        self.only_foreign = only_foreign
+
+        if only_foreign:
+            logger.info("Only searching for foreign/forced subtitles")
+
+        super(PatchedPodnapisiProvider, self).__init__()
+
+    def list_subtitles(self, video, languages):
+        if isinstance(video, Episode):
+            return [s for l in languages for s in self.query(l, video.series, season=video.season,
+                                                             episode=video.episode, year=video.year,
+                                                             only_foreign=self.only_foreign)]
+        elif isinstance(video, Movie):
+            return [s for l in languages for s in self.query(l, video.title, year=video.year,
+                                                             only_foreign=self.only_foreign)]
+
     def download_subtitle(self, subtitle):
         # download as a zip
         logger.info('Downloading subtitle %r', subtitle)
-        r = self.session.get(self.server_url + subtitle.pid + '/download', params={'container': 'zip'}, timeout=10)
+        r = self.retry(lambda: self.session.get(self.server_url + subtitle.pid + '/download',
+                                                params={'container': 'zip'}, timeout=10))
         r.raise_for_status()
 
         # open the zip
@@ -21,3 +64,76 @@ class PatchedPodnapisiProvider(PodnapisiProvider):
                 raise ProviderError('More than one file to unzip')
 
             subtitle.content = fix_line_ending(zf.read(zf.namelist()[0]))
+
+    def query(self, language, keyword, season=None, episode=None, year=None, only_foreign=False):
+        # set parameters, see http://www.podnapisi.net/forum/viewtopic.php?f=62&t=26164#p212652
+        params = {'sXML': 1, 'sL': str(language), 'sK': keyword}
+        is_episode = False
+        if season and episode:
+            is_episode = True
+            params['sTS'] = season
+            params['sTE'] = episode
+        if year:
+            params['sY'] = year
+
+        # loop over paginated results
+        logger.info('Searching subtitles %r', params)
+        subtitles = []
+        pids = set()
+        while True:
+            # query the server
+            xml = etree.fromstring(self.retry(lambda: self.session.get(self.server_url + 'search/old',
+                                                                       params=params, timeout=10).content))
+
+            # exit if no results
+            if not int(xml.find('pagination/results').text):
+                logger.debug('No subtitles found')
+                break
+
+            # loop over subtitles
+            for subtitle_xml in xml.findall('subtitle'):
+                # read xml elements
+                language = Language.fromietf(subtitle_xml.find('language').text)
+                hearing_impaired = 'n' in (subtitle_xml.find('flags').text or '')
+                foreign = 'f' in (subtitle_xml.find('flags').text or '')
+                if only_foreign and not foreign:
+                    continue
+
+                if not only_foreign and foreign:
+                    continue
+
+                page_link = subtitle_xml.find('url').text
+                pid = subtitle_xml.find('pid').text
+                releases = []
+                if subtitle_xml.find('release').text:
+                    for release in subtitle_xml.find('release').text.split():
+                        releases.append(re.sub(r'\.+$', '', release))  # remove trailing dots
+                title = subtitle_xml.find('title').text
+                season = int(subtitle_xml.find('tvSeason').text)
+                episode = int(subtitle_xml.find('tvEpisode').text)
+                year = int(subtitle_xml.find('year').text)
+
+                if is_episode:
+                    subtitle = PatchedPodnapisiSubtitle(language, hearing_impaired, page_link, pid, releases, title,
+                                                 season=season, episode=episode, year=year)
+                else:
+                    subtitle = PatchedPodnapisiSubtitle(language, hearing_impaired, page_link, pid, releases, title,
+                                                 year=year)
+
+                # ignore duplicates, see http://www.podnapisi.net/forum/viewtopic.php?f=62&t=26164&start=10#p213321
+                if pid in pids:
+                    continue
+
+                logger.debug('Found subtitle %r', subtitle)
+                subtitles.append(subtitle)
+                pids.add(pid)
+
+            # stop on last page
+            if int(xml.find('pagination/current').text) >= int(xml.find('pagination/count').text):
+                break
+
+            # increment current page
+            params['page'] = int(xml.find('pagination/current').text) + 1
+            logger.debug('Getting page %d', params['page'])
+
+        return subtitles

Contents/Libraries/Shared/subliminal_patch/patch_providers/tvsubtitles.py

@@ -2,10 +2,11 @@
 
 import re
 import logging
+from babelfish import Language
 from subliminal.providers import ParserBeautifulSoup
 from subliminal.cache import SHOW_EXPIRATION_TIME, region
-from subliminal.providers.tvsubtitles import TVsubtitlesProvider
-from .mixins import PunctuationMixin
+from subliminal.providers.tvsubtitles import TVsubtitlesProvider, TVsubtitlesSubtitle
+from .mixins import PunctuationMixin, ProviderRetryMixin
 
 logger = logging.getLogger(__name__)
 
@@ -14,7 +15,14 @@ logger = logging.getLogger(__name__)
 link_re = re.compile('^(?P<series>.+)(?: \(?\d{4}\)?| \((?:US|UK)\))? \((?P<first_year>\d{4})\d{4}\)$')
 
 
-class PatchedTVsubtitlesProvider(PunctuationMixin, TVsubtitlesProvider):
+class PatchedTVsubtitlesSubtitle(TVsubtitlesSubtitle):
+    def __init__(self, language, page_link, subtitle_id, series, season, episode, year, rip, release):
+        super(PatchedTVsubtitlesSubtitle, self).__init__(language, page_link, subtitle_id, series, season, episode,
+                                                         year, rip, release)
+        self.release_info = u"%s, %s" % (rip, release)
+
+
+class PatchedTVsubtitlesProvider(PunctuationMixin, ProviderRetryMixin, TVsubtitlesProvider):
     @region.cache_on_arguments(expiration_time=SHOW_EXPIRATION_TIME)
     def search_show_id(self, series, year=None):
         """Search the show id from the `series` and `year`.
@@ -27,7 +35,7 @@ class PatchedTVsubtitlesProvider(PunctuationMixin, TVsubtitlesProvider):
         # make the search
         series_clean = self.clean_punctuation(series).lower()
         logger.info('Searching show id for %r', series_clean)
-        r = self.session.post(self.server_url + 'search.php', data={'q': series_clean}, timeout=10)
+        r = self.retry(lambda: self.session.post(self.server_url + 'search.php', data={'q': series_clean}, timeout=10))
         r.raise_for_status()
 
         # get the series out of the suggestions
@@ -48,3 +56,38 @@ class PatchedTVsubtitlesProvider(PunctuationMixin, TVsubtitlesProvider):
                 break
 
         return show_id
+
+    def query(self, series, season, episode, year=None):
+        # search the show id
+        show_id = self.search_show_id(series, year)
+        if show_id is None:
+            logger.error('No show id found for %r (%r)', series, {'year': year})
+            return []
+
+        # get the episode ids
+        episode_ids = self.retry(lambda: self.get_episode_ids(show_id, season))
+        if episode not in episode_ids:
+            logger.error('Episode %d not found', episode)
+            return []
+
+        # get the episode page
+        logger.info('Getting the page for episode %d', episode_ids[episode])
+        r = self.retry(lambda: self.session.get(self.server_url + 'episode-%d.html' % episode_ids[episode], timeout=10))
+        soup = ParserBeautifulSoup(r.content, ['lxml', 'html.parser'])
+
+        # loop over subtitles rows
+        subtitles = []
+        for row in soup.select('.subtitlen'):
+            # read the item
+            language = Language.fromtvsubtitles(row.h5.img['src'][13:-4])
+            subtitle_id = int(row.parent['href'][10:-5])
+            page_link = self.server_url + 'subtitle-%d.html' % subtitle_id
+            rip = row.find('p', title='rip').text.strip() or None
+            release = row.find('p', title='release').text.strip() or None
+
+            subtitle = PatchedTVsubtitlesSubtitle(language, page_link, subtitle_id, series, season, episode, year, rip,
+                                           release)
+            logger.info('Found subtitle %s', subtitle)
+            subtitles.append(subtitle)
+
+        return subtitles

Contents/Libraries/Shared/subliminal_patch/patch_subtitle.py

@@ -35,8 +35,8 @@ def compute_score(matches, video, scores=None):
 
     is_episode = isinstance(video, Episode)
 
-    episode_hash_valid_if = {"series", "season", "episode"}
-    movie_hash_valid_if = {"title", "video_codec"}
+    episode_hash_valid_if = {"series", "season", "episode", "format"}
+    movie_hash_valid_if = {"video_codec", "format"}
 
     # remove equivalent match combinations
     if 'hash' in final_matches:
@@ -68,6 +68,8 @@ def compute_score(matches, video, scores=None):
 
 class PatchedSubtitle(Subtitle):
     storage_path = None
+    release_info = None
+    matches = None
 
     def guess_encoding(self):
         """Guess encoding using the language, falling back on chardet.
@@ -76,9 +78,8 @@ class PatchedSubtitle(Subtitle):
         :rtype: str
 
         """
-        logger.info('Guessing encoding for language %s', self.language)
+        logger.info('Guessing encoding for language %s', self.language.alpha3)
 
-        # always try utf-8 first
         encodings = ['utf-8']
 
         # add language-specific encodings
@@ -86,23 +87,33 @@ class PatchedSubtitle(Subtitle):
             encodings.extend(['gb18030', 'big5'])
         elif self.language.alpha3 == 'jpn':
             encodings.append('shift-jis')
-        elif self.language.alpha3 == 'ara':
+        elif self.language.alpha3 == 'tha':
+            encodings.append('tis-620')
+
+        # arabian/farsi
+        elif self.language.alpha3 in ('ara', 'fas', 'per'):
             encodings.append('windows-1256')
         elif self.language.alpha3 == 'heb':
             encodings.append('windows-1255')
         elif self.language.alpha3 == 'tur':
             encodings.extend(['iso-8859-9', 'windows-1254'])
 
+        # Greek
+        elif self.language.alpha3 in ('grc', 'gre', 'ell'):
+            encodings.extend(['windows-1253', 'cp1253', 'cp737', 'iso8859_7', 'cp875', 'cp869', 'iso2022_jp_2',
+                              'mac_greek'])
+
         # Polish, Czech, Slovak, Hungarian, Slovene, Bosnian, Croatian, Serbian (Latin script),
         # Romanian (before 1993 spelling reform) and Albanian
-        elif self.language.alpha3 in ('pol', 'cze', 'svk', 'hun', 'svn', 'bih', 'hrv', 'srb', 'rou', 'alb'):
+        elif self.language.alpha3 in ('pol', 'cze', 'ces', 'slk', 'slo', 'slv', 'hun', 'bos', 'hbs', 'hrv', 'rsb',
+                                      'ron', 'rum', 'sqi', 'alb'):
             # Eastern European Group 1
-            encodings.extend(['windows-1250'])
+            encodings.append('windows-1250')
 
         # Bulgarian, Serbian and Macedonian
-        elif self.language.alpha3 in ('bul', 'srb', 'mkd'):
+        elif self.language.alpha3 in ('bul', 'srp', 'mkd', 'mac'):
             # Eastern European Group 2
-            encodings.extend(['windows-1251'])
+            encodings.append('windows-1251')
         else:
             # Western European (windows-1252)
             encodings.append('latin-1')

Contents/Libraries/Shared/subliminal_patch/patch_video.py

@@ -1,20 +1,24 @@
 # coding=utf-8
 
 import os
+import re
 import logging
 import traceback
 
 from babelfish import Error as BabelfishError
-from subliminal.video import SUBTITLE_EXTENSIONS, VIDEO_EXTENSIONS, Language, Video, EnzymeError, MKV, guess_file_info, hash_opensubtitles, \
-    hash_thesubdb
+from subliminal.video import SUBTITLE_EXTENSIONS, VIDEO_EXTENSIONS, Language, Video, EnzymeError, MKV, \
+    guess_file_info, hash_opensubtitles, hash_thesubdb
 
 logger = logging.getLogger(__name__)
 
 # may be absolute or relative paths; set to selected options
 CUSTOM_PATHS = []
+INCLUDE_EXOTIC_SUBS = True
+
+REMOVE_CRAP_FROM_FILENAME = re.compile("(?i)[_-](obfuscated|scrambled)(\.[\w]+)$")
 
 
-def _search_external_subtitles(path):
+def _search_external_subtitles(path, forced_tag=False):
     dirpath, filename = os.path.split(path)
     dirpath = dirpath or '.'
     fileroot, fileext = os.path.splitext(filename)
@@ -24,8 +28,25 @@ def _search_external_subtitles(path):
         if not p.startswith(fileroot) or not p.endswith(SUBTITLE_EXTENSIONS):
             continue
 
+        p_root, p_ext = os.path.splitext(p)
+        if not INCLUDE_EXOTIC_SUBS and p_ext not in (".srt", ".ass", ".ssa"):
+            continue
+
+        # extract potential forced/normal/default tag
+        # fixme: duplicate from subtitlehelpers
+        split_tag = p_root.rsplit('.', 1)
+        adv_tag = None
+        if len(split_tag) > 1:
+            adv_tag = split_tag[1].lower()
+            if adv_tag in ['forced', 'normal', 'default']:
+                p_root = split_tag[0]
+
+        # forced wanted but NIL
+        if forced_tag and adv_tag != "forced":
+            continue
+
         # extract the potential language code
-        language_code = p[len(fileroot):-len(os.path.splitext(p)[1])].replace(fileext, '').replace('_', '-')[1:]
+        language_code = p_root[len(fileroot):].replace('_', '-')[1:]
 
         # default language is undefined
         language = Language('und')
@@ -44,7 +65,7 @@ def _search_external_subtitles(path):
     return subtitles
 
 
-def patched_search_external_subtitles(path):
+def patched_search_external_subtitles(path, forced_tag=False):
     """
     wrap original search_external_subtitles function to search multiple paths for one given video
     # todo: cleanup and merge with _search_external_subtitles
@@ -62,12 +83,13 @@ def patched_search_external_subtitles(path):
         logger.debug("external subs: scanning path %s", abspath)
 
         if os.path.isdir(os.path.dirname(abspath)):
-            subtitles.update(_search_external_subtitles(abspath))
+            subtitles.update(_search_external_subtitles(abspath, forced_tag=forced_tag))
     logger.debug("external subs: found %s", subtitles)
     return subtitles
 
 
-def scan_video(path, subtitles=True, embedded_subtitles=True, hints=None, video_fps=None, dont_use_actual_file=False):
+def scan_video(path, subtitles=True, embedded_subtitles=True, hints=None, video_fps=None, dont_use_actual_file=False,
+               forced_tag=False, known_embedded_subtitle_streams=None):
     """Scan a video and its subtitle languages from a video `path`.
     :param dont_use_actual_file: guess on filename, but don't use the actual file itself
     :param str path: existing path to the video.
@@ -80,6 +102,7 @@ def scan_video(path, subtitles=True, embedded_subtitles=True, hints=None, video_
     # patch: suggest video type to guessit beforehand
     """
     hints = hints or {}
+    video_type = hints.get("type")
 
     # check for non-existing path
     if not dont_use_actual_file and not os.path.exists(path):
@@ -92,34 +115,49 @@ def scan_video(path, subtitles=True, embedded_subtitles=True, hints=None, video_
     dirpath, filename = os.path.split(path)
 
     # hint guessit the filename itself and its 2 parent directories if we're an episode (most likely Series name/Season/filename), else only one
-    guess_from = os.path.join(*os.path.normpath(path).split(os.path.sep)[-3 if hints.get("type") == "episode" else -2:])
+    guess_from = os.path.join(*os.path.normpath(path).split(os.path.sep)[-3 if video_type == "episode" else -2:])
+    guess_from = REMOVE_CRAP_FROM_FILENAME.sub(r"\2", guess_from)
+
     hints = hints or {}
     logger.info('Scanning video (hints: %s) %r', hints, guess_from)
 
     # guess
-    try:
-        video = Video.fromguess(path, guess_file_info(guess_from, options=hints))
-        video.fps = video_fps
+    video = Video.fromguess(path, guess_file_info(guess_from, options=hints))
+    video.fps = video_fps
 
-        if dont_use_actual_file:
-            return video
+    # trust plex's movie name
+    if video_type == "movie" and hints.get("expected_title"):
+        video.title = hints.get("expected_title")[0]
 
-        # size and hashes
-        video.size = os.path.getsize(path)
-        if video.size > 10485760:
-            logger.debug('Size is %d', video.size)
-            video.hashes['opensubtitles'] = hash_opensubtitles(path)
-            video.hashes['thesubdb'] = hash_thesubdb(path)
-            logger.debug('Computed hashes %r', video.hashes)
-        else:
-            logger.warning('Size is lower than 10MB: hashes not computed')
+    if dont_use_actual_file:
+        return video
 
-        # external subtitles
-        if subtitles:
-            video.subtitle_languages |= set(patched_search_external_subtitles(path).values())
-    except Exception:
-        logger.error("Something went wrong when running guessit: %s", traceback.format_exc())
-        return
+    # size and hashes
+    video.size = os.path.getsize(path)
+    if video.size > 10485760:
+        logger.debug('Size is %d', video.size)
+        video.hashes['opensubtitles'] = hash_opensubtitles(path)
+        video.hashes['thesubdb'] = hash_thesubdb(path)
+        logger.debug('Computed hashes %r', video.hashes)
+    else:
+        logger.warning('Size is lower than 10MB: hashes not computed')
+
+    # external subtitles
+    if subtitles:
+        video.subtitle_languages |= set(patched_search_external_subtitles(path, forced_tag=forced_tag).values())
+
+    if embedded_subtitles and known_embedded_subtitle_streams:
+        embedded_subtitle_languages = set()
+        # mp4 and stuff, check burned in
+        for language in known_embedded_subtitle_streams:
+            try:
+                embedded_subtitle_languages.add(Language.fromalpha3b(language))
+            except BabelfishError:
+                logger.error('Embedded subtitle track language %r is not a valid language', language)
+                embedded_subtitle_languages.add(Language('und'))
+
+            logger.debug('Found embedded subtitle %r', embedded_subtitle_languages)
+            video.subtitle_languages |= embedded_subtitle_languages
 
     # video metadata with enzyme
     try:
@@ -168,33 +206,6 @@ def scan_video(path, subtitles=True, embedded_subtitles=True, hints=None, video_
             else:
                 logger.warning('MKV has no audio track')
 
-            # subtitle tracks
-            if mkv.subtitle_tracks:
-                if embedded_subtitles:
-                    embedded_subtitle_languages = set()
-                    for st in mkv.subtitle_tracks:
-                        if st.forced:
-                            logger.debug("Ignoring forced subtitle track %r", st)
-                            continue
-                        if st.language:
-                            try:
-                                embedded_subtitle_languages.add(Language.fromalpha3b(st.language))
-                            except BabelfishError:
-                                logger.error('Embedded subtitle track language %r is not a valid language', st.language)
-                                embedded_subtitle_languages.add(Language('und'))
-                        elif st.name:
-                            try:
-                                embedded_subtitle_languages.add(Language.fromname(st.name))
-                            except BabelfishError:
-                                logger.debug('Embedded subtitle track name %r is not a valid language', st.name)
-                                embedded_subtitle_languages.add(Language('und'))
-                        else:
-                            embedded_subtitle_languages.add(Language('und'))
-                    logger.debug('Found embedded subtitle %r with enzyme', embedded_subtitle_languages)
-                    video.subtitle_languages |= embedded_subtitle_languages
-            else:
-                logger.debug('MKV has no subtitle track')
-
     except EnzymeError:
         logger.error('Parsing video metadata with enzyme failed')
 

Contents/Libraries/Shared/subzero/__init__.py

@@ -1,11 +1,2 @@
 # coding=utf-8
 
-from intent import intent
-
-OS_PLEX_USERAGENT = 'plexapp.com v9.0'
-
-DEPENDENCY_MODULE_NAMES = ['subliminal', 'subliminal_patch', 'enzyme', 'guessit', 'requests']
-PERSONAL_MEDIA_IDENTIFIER = "com.plexapp.agents.none"
-PREFIX = "/video/subzero"
-
-

Contents/Libraries/Shared/subzero/analytics.py

@@ -0,0 +1,33 @@
+# coding=utf-8
+
+import struct
+import binascii
+
+from pyga.requests import Event, Page, Tracker, Session, Visitor, Config
+
+
+def track_event(category=None, action=None, label=None, value=None, identifier=None, first_use=None, add=None,
+                noninteraction=True):
+    anonymousConfig = Config()
+    anonymousConfig.anonimize_ip_address = True
+
+    tracker = Tracker('UA-86466078-1', 'none', conf=anonymousConfig)
+    visitor = Visitor()
+
+    # convert the last 8 bytes of the machine identifier to an integer to get a "unique" user
+    visitor.unique_id = struct.unpack("!I", binascii.unhexlify(identifier[32:]))[0]/2
+
+    if add:
+        # add visitor's ip address (will be anonymized)
+        visitor.ip_address = add
+
+    if first_use:
+        visitor.first_visit_time = first_use
+
+    session = Session()
+    event = Event(category=category, action=action, label=label, value=value, noninteraction=noninteraction)
+    path = u"/" + u"/".join([category, action, label])
+    page = Page(path.lower())
+
+    tracker.track_event(event, session, visitor)
+    tracker.track_pageview(page, session, visitor)

Contents/Libraries/Shared/subzero/constants.py

@@ -2,7 +2,7 @@
 
 OS_PLEX_USERAGENT = 'plexapp.com v9.0'
 
-DEPENDENCY_MODULE_NAMES = ['subliminal', 'subliminal_patch', 'enzyme', 'guessit']
+DEPENDENCY_MODULE_NAMES = ['subliminal', 'subliminal_patch', 'enzyme', 'guessit', 'subzero', 'plex_activity']
 PERSONAL_MEDIA_IDENTIFIER = "com.plexapp.agents.none"
 PLUGIN_IDENTIFIER_SHORT = "subzero"
 PLUGIN_IDENTIFIER = "com.plexapp.agents.%s" % PLUGIN_IDENTIFIER_SHORT
@@ -13,6 +13,8 @@ TITLE = "%s Subtitles" % PLUGIN_NAME
 ART      = 'art-default.jpg'
 ICON     = 'icon-default.jpg'
 
+DEFAULT_TIMEOUT = 10
+
 
 # media types as on https://github.com/Arcanemagus/plex-api/wiki/MediaTypes
 MOVIE = 1
@@ -30,3 +32,10 @@ PICTURE = 12
 PHOTO = 13
 CLIP = 14
 PLAYLIST_ITEM = 15
+
+
+mode_map = {
+    "a": "auto",
+    "m": "manual",
+    "b": "auto-better"
+}

Contents/Libraries/Shared/subzero/func.py

@@ -1,24 +0,0 @@
-# coding=utf-8
-import threading
-
-lock = threading.Lock()
-
-
-class Debouncer(object):
-    call_history = set()
-
-    def get_lookup_key(self, args, kwargs):
-        func_name = list(args).pop(0).__name__
-        return tuple([func_name] + [(key, value) for key, value in kwargs.iteritems()])
-
-    def __contains__(self, item):
-        args, kwargs = item
-        lookup = self.get_lookup_key(args, kwargs)
-        with lock:
-            return lookup in self.call_history
-
-    def add(self, args, kwargs):
-        with lock:
-            self.call_history.add(self.get_lookup_key(args, kwargs))
-
-debouncer = Debouncer()

Contents/Libraries/Shared/subzero/history_storage.py

@@ -0,0 +1,89 @@
+# coding=utf-8
+
+import datetime
+import logging
+import traceback
+
+from constants import mode_map
+
+logger = logging.getLogger(__name__)
+
+
+class SubtitleHistoryItem(object):
+    item_title = None
+    section_title = None
+    rating_key = None
+    provider_name = None
+    lang_name = None
+    score = None
+    time = None
+    mode = "a"
+
+    def __init__(self, item_title, rating_key, section_title=None, subtitle=None, mode="a", time=None):
+        self.item_title = item_title
+        self.section_title = section_title
+        self.rating_key = str(rating_key)
+        self.provider_name = subtitle.provider_name
+        self.lang_name = subtitle.language.name
+        self.score = subtitle.score
+        self.time = time or datetime.datetime.now()
+        self.mode = mode
+
+    @property
+    def title(self):
+        return u"%s: %s" % (self.section_title, self.item_title)
+
+    @property
+    def mode_verbose(self):
+        return mode_map.get(self.mode, "Unknown")
+
+    def __repr__(self):
+        return unicode(self)
+
+    def __unicode__(self):
+        return u"%s (Score: %s)" % (unicode(self.item_title), self.score)
+
+    def __str__(self):
+        return str(self.rating_key)
+
+    def __hash__(self):
+        return hash((self.rating_key, self.score))
+
+    def __eq__(self, other):
+        return (self.rating_key, self.score) == (other.rating_key, other.score)
+
+    def __ne__(self, other):
+        # Not strictly necessary, but to avoid having both x==y and x!=y
+        # True at the same time
+        return not (self == other)
+
+
+class SubtitleHistory(object):
+    size = 100
+    history_items = None
+    storage = None
+
+    def __init__(self, storage, size=100):
+        self.size = size
+        self.storage = storage
+        self.history_items = []
+        try:
+            self.history_items = storage.LoadObject("subtitle_history") or []
+        except:
+            logger.error("Failed to load history storage: %s" % traceback.format_exc())
+
+    def add(self, item_title, rating_key, section_title=None, subtitle=None, mode="a", time=None):
+        # create copy
+        items = self.history_items
+        item = SubtitleHistoryItem(item_title, rating_key, section_title=section_title, subtitle=subtitle, mode=mode, time=time)
+
+        # insert item
+        items.insert(0, item)
+
+        # clamp item amount
+        self.history_items = items[:self.size]
+
+        # store items
+        self.storage.SaveObject("subtitle_history", self.history_items)
+
+

Contents/Libraries/Shared/subzero/intent.py

@@ -6,25 +6,16 @@ import threading
 lock = threading.Lock()
 
 
-class TempIntent(dict):
+class TempIntent(object):
     timeout = 1000  # milliseconds
     store = None
 
-    def __init__(self, timeout=1000):
+    def __init__(self, timeout=1000, store=None):
         self.timeout = timeout
-        with lock:
-            self.store = {}
+        if store is None:
+            raise NotImplementedError
 
-    def __getattr__(self, name):
-        if name in self:
-            return self[name]
-
-    def __setattr__(self, name, value):
-        self[name] = value
-
-    def __delattr__(self, name):
-        if name in self:
-            del self[name]
+        self.store = store
 
     def get(self, kind, *keys):
         with lock:
@@ -37,13 +28,15 @@ class TempIntent(dict):
                     continue
 
                 # valid kind?
-                if kind in self["store"]:
+                if kind in self.store:
                     now = datetime.datetime.now()
+                    key = str(key)
 
                     # iter all known kinds (previously created)
-                    for known_key in self["store"][kind].keys():
+                    for known_key in self.store[kind].keys():
                         # may need locking, for now just play it safe
-                        ends = self["store"][kind].get(known_key, None)
+                        data = self.store[kind].get(known_key, {})
+                        ends = data.get("timeout")
                         if not ends:
                             continue
 
@@ -57,7 +50,7 @@ class TempIntent(dict):
 
                         if timed_out:
                             try:
-                                del self["store"][kind][key]
+                                del self.store[kind][key]
                             except:
                                 continue
 
@@ -67,22 +60,42 @@ class TempIntent(dict):
 
     def resolve(self, kind, key):
         with lock:
-            if kind in self["store"] and key in self["store"][kind]:
-                del self["store"][kind][key]
+            if kind in self.store and key in self.store[kind]:
+                del self.store[kind][key]
                 return True
             return False
 
-    def set(self, kind, key, timeout=None):
+    def set(self, kind, key, data=None, timeout=None):
         with lock:
-            if kind not in self["store"]:
-                self["store"][kind] = {}
-            self["store"][kind][key] = datetime.datetime.now() + datetime.timedelta(milliseconds=timeout or self.timeout)
+            if kind not in self.store:
+                self.store[kind] = {}
+
+            key = str(key)
+            self.store[kind][key] = {
+                "data": data,
+                "timeout": datetime.datetime.now() + datetime.timedelta(milliseconds=timeout or self.timeout)
+            }
 
     def has(self, kind, key):
         with lock:
-            if kind not in self["store"]:
+            if kind not in self.store:
                 return False
-            return key in self["store"][kind]
+            return key in self.store[kind]
 
+    def cleanup(self):
+        now = datetime.datetime.now()
+        clear_all = False
+        for kind, info in self.store.items():
+            for key, intent_data in info.items():
+                # legacy intent data, clear everything
+                if not isinstance(intent_data, dict):
+                    clear_all = True
+                    continue
+
+                if now > intent_data["timeout"]:
+                    del self.store[kind][key]
+        if clear_all:
+            self.store.clear()
+
+        self.store.save()
 
-intent = TempIntent()

Contents/Libraries/Shared/subzero/lib/dict.py

@@ -10,11 +10,27 @@ class DictProxy(object):
 
         if self.store not in self.Dict or not self.Dict[self.store]:
             self.Dict[self.store] = self.setup_defaults()
+        self.save()
+        self.__initialized = True
 
     def __getattr__(self, name):
         if name in self.Dict[self.store]:
             return self.Dict[self.store][name]
-        return getattr(super(self.DictProxy, self), name)
+        return getattr(super(DictProxy, self), name)
+
+    def __setattr__(self, name, value):
+        if not self.__dict__.has_key(
+                '_DictProxy__initialized'):  # this test allows attributes to be set in the __init__ method
+            return object.__setattr__(self, name, value)
+
+        elif self.__dict__.has_key(name):  # any normal attributes are handled normally
+            object.__setattr__(self, name, value)
+
+        else:
+            if name in self.Dict[self.store]:
+                self.Dict[self.store][name] = value
+                return
+        object.__setattr__(self, name, value)
 
     def __cmp__(self, d):
         return cmp(self.Dict[self.store], d)
@@ -45,6 +61,9 @@ class DictProxy(object):
     def __delitem__(self, key):
         del self.Dict[self.store][key]
 
+    def save(self):
+        self.Dict.Save()
+
     def clear(self):
         del self.Dict[self.store]
         return None

Contents/Libraries/Shared/subzero/score_range.py

@@ -0,0 +1,22 @@
+# coding=utf-8
+
+import sys
+from itertools import chain, combinations, permutations
+
+from subliminal.video import Episode
+
+
+def permute(x):
+    return [base_score + i + j for i in x for j in x]
+
+if __name__ == "__main__":
+    scores = Episode.scores
+    base_score_keys = ["series", "season", "episode"]
+    leftover_keys = list(set(scores.keys()) - set(base_score_keys))
+    base_score = sum([val for key, val in scores.items() if key in base_score_keys])
+    leftover_scores = set([score for key, score in scores.items() if key in leftover_keys])
+    print "base score:", base_score
+    print "leftover:", sorted(set(leftover_scores))
+    # print sum_possible_scores(base_score, leftover_scores)
+    # print list(permutations(leftover_scores))
+    print ",\n".join(map(lambda x: '"%s"' % x, ["66"] + sorted(set(permute(leftover_scores)))))

Contents/Libraries/Shared/subzero/subtitle_storage.py

@@ -0,0 +1,213 @@
+# coding=utf-8
+import datetime
+import hashlib
+import os
+import logging
+import traceback
+
+from constants import mode_map
+
+logger = logging.getLogger(__name__)
+
+
+class StoredSubtitle(object):
+    score = None
+    storage_type = None
+    hash = None
+    provider_name = None
+    id = None
+    date_added = None
+    mode = "a"  # auto/manual/auto-better (a/m/b)
+    content = None
+
+    def __init__(self, score, storage_type, hash, provider_name, id, date_added=None, mode="a", content=None):
+        self.score = int(score)
+        self.storage_type = storage_type
+        self.hash = hash
+        self.provider_name = provider_name
+        self.id = id
+        self.date_added = date_added or datetime.datetime.now()
+        self.mode = mode
+        self.content = content
+
+    @property
+    def mode_verbose(self):
+        return mode_map.get(self.mode, "Unknown")
+
+
+class StoredVideoSubtitles(object):
+    """
+    manages stored subtitles for video_id per media_part/language combination
+    """
+    video_id = None  # rating_key
+    title = None
+    parts = None
+    version = None
+    item_type = None  # movie / episode
+    added_at = None
+
+    def __init__(self, plex_item, version=None):
+        self.video_id = str(plex_item.rating_key)
+
+        self.title = plex_item.title
+        self.parts = {}
+        self.version = version
+        self.item_type = plex_item.type
+        self.added_at = datetime.datetime.fromtimestamp(plex_item.added_at)
+
+    def add(self, part_id, lang, subtitle, storage_type, date_added=None, mode="a"):
+        part_id = str(part_id)
+        part = self.parts.get(part_id)
+        if not part:
+            self.parts[part_id] = {}
+            part = self.parts[part_id]
+
+        subs = part.get(lang)
+        if not subs:
+            part[lang] = {}
+            subs = part[lang]
+
+        sub_key = self.get_sub_key(subtitle.provider_name, subtitle.id)
+        if sub_key in subs:
+            return
+
+        subs[sub_key] = StoredSubtitle(subtitle.score, storage_type, hashlib.md5(subtitle.content).hexdigest(),
+                                       subtitle.provider_name, subtitle.id, date_added=date_added, mode=mode,
+                                       content=subtitle.content)
+        subs["current"] = sub_key
+
+        return True
+
+    def get_any(self, part_id, lang):
+        part_id = str(part_id)
+        part = self.parts.get(part_id)
+        if not part:
+            return
+
+        subs = part.get(lang)
+        if not subs:
+            return
+
+        if "current" in subs and subs["current"]:
+            return subs.get(subs["current"])
+
+    def get_sub_key(self, provider_name, id):
+        return provider_name, str(id)
+
+    def __repr__(self):
+        return unicode(self)
+
+    def __unicode__(self):
+        return u"%s (%s)" % (self.title, self.video_id)
+
+    def __str__(self):
+        return str(self.video_id)
+
+
+class StoredSubtitlesManager(object):
+    """
+    manages the storage and retrieval of StoredVideoSubtitles instances for a given video_id
+    """
+    storage = None
+    version = 2
+
+    def __init__(self, storage, plexapi_item_getter):
+        self.storage = storage
+        self.get_item = plexapi_item_getter
+
+    def get_storage_filename(self, video_id):
+        return "subs_%s" % video_id
+
+    @property
+    def dataitems_path(self):
+        return os.path.join(getattr(self.storage, "_core").storage.data_path, "DataItems")
+
+    def get_all_files(self):
+        return os.listdir(self.dataitems_path)
+
+    def get_recent_files(self, age_days=30):
+        fl = []
+        root = self.dataitems_path
+        recent_dt = datetime.datetime.now() - datetime.timedelta(days=age_days)
+        for fn in self.get_all_files():
+            if not fn.startswith("subs_"):
+                continue
+
+            finfo = os.stat(os.path.join(root, fn))
+            created = datetime.datetime.fromtimestamp(finfo.st_ctime)
+            if created > recent_dt:
+                fl.append(fn)
+        return fl
+
+    def load_recent_files(self, age_days=30):
+        fl = self.get_recent_files(age_days=age_days)
+        out = {}
+        for fn in fl:
+            data = self.load(filename=fn)
+            if data:
+                out[fn] = data
+        return out
+
+    def migrate_v2(self, subs_for_video):
+        plex_item = self.get_item(subs_for_video.video_id)
+        if not plex_item:
+            return False
+        subs_for_video.item_type = plex_item.type
+        subs_for_video.added_at = datetime.datetime.fromtimestamp(plex_item.added_at)
+        subs_for_video.version = 2
+        return True
+
+    def load(self, video_id=None, filename=None):
+        subs_for_video = None
+        fn = self.get_storage_filename(video_id) if video_id else filename
+        try:
+            subs_for_video = self.storage.LoadObject(fn)
+        except:
+            logger.error("Failed to load item %s: %s" % (fn, traceback.format_exc()))
+
+        if not subs_for_video:
+            return
+
+        # apply possible migrations
+        cur_ver = old_ver = subs_for_video.version
+
+        if cur_ver < self.version:
+            success = False
+            while cur_ver < self.version:
+                cur_ver += 1
+                mig_func = "migrate_v%s" % cur_ver
+                if hasattr(self, mig_func):
+                    logger.info("Migrating subtitle storage for %s %s>%s" % (subs_for_video.video_id, old_ver, cur_ver))
+                    success = getattr(self, mig_func)(subs_for_video)
+                    if success is False:
+                        logger.error("Couldn't migrate %s, removing data", subs_for_video.video_id)
+                        self.delete(fn)
+                        break
+
+            if cur_ver > old_ver and success:
+                logger.info("Storing migrated subtitle storage for %s" % subs_for_video.video_id)
+                self.save(subs_for_video)
+            elif not success:
+                logger.info("Migration of %s %s>%s failed" % (subs_for_video.video_id, old_ver, cur_ver))
+
+        return subs_for_video
+
+    def load_or_new(self, plex_item):
+        subs_for_video = self.load(plex_item.rating_key)
+        if not subs_for_video:
+            subs_for_video = StoredVideoSubtitles(plex_item, version=self.version)
+            self.save(subs_for_video)
+        return subs_for_video
+
+    def save(self, subs_for_video):
+        fn = self.get_storage_filename(subs_for_video.video_id)
+        try:
+            self.storage.SaveObject(fn, subs_for_video)
+        except:
+            logger.error("Failed to save item %s: %s" % (fn, traceback.format_exc()))
+
+    def delete(self, filename):
+        try:
+            self.storage.Remove(filename)
+        except:
+            logger.error("Failed to delete item %s: %s" % (filename, traceback.format_exc()))

Contents/Libraries/Shared/websocket/__init__.py

@@ -0,0 +1,29 @@
+"""
+websocket - WebSocket client library for Python
+
+Copyright (C) 2010 Hiroki Ohtani(liris)
+
+    This library is free software; you can redistribute it and/or
+    modify it under the terms of the GNU Lesser General Public
+    License as published by the Free Software Foundation; either
+    version 2.1 of the License, or (at your option) any later version.
+
+    This library is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+    Lesser General Public License for more details.
+
+    You should have received a copy of the GNU Lesser General Public
+    License along with this library; if not, write to the Free Software
+    Foundation, Inc., 51 Franklin Street, Fifth Floor,
+    Boston, MA  02110-1335  USA
+
+"""
+from ._abnf import *
+from ._app import WebSocketApp
+from ._core import *
+from ._exceptions import *
+from ._logging import *
+from ._socket import *
+
+__version__ = "0.39.0"

Contents/Libraries/Shared/websocket/_abnf.py

@@ -0,0 +1,422 @@
+"""
+websocket - WebSocket client library for Python
+
+Copyright (C) 2010 Hiroki Ohtani(liris)
+
+    This library is free software; you can redistribute it and/or
+    modify it under the terms of the GNU Lesser General Public
+    License as published by the Free Software Foundation; either
+    version 2.1 of the License, or (at your option) any later version.
+
+    This library is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+    Lesser General Public License for more details.
+
+    You should have received a copy of the GNU Lesser General Public
+    License along with this library; if not, write to the Free Software
+    Foundation, Inc., 51 Franklin Street, Fifth Floor,
+    Boston, MA  02110-1335  USA
+
+"""
+import array
+import os
+import struct
+
+import six
+
+from ._exceptions import *
+from ._utils import validate_utf8
+
+try:
+    # If wsaccel is available we use compiled routines to mask data.
+    from wsaccel.xormask import XorMaskerSimple
+
+    def _mask(_m, _d):
+        return XorMaskerSimple(_m).process(_d)
+
+except ImportError:
+    # wsaccel is not available, we rely on python implementations.
+    def _mask(_m, _d):
+        for i in range(len(_d)):
+            _d[i] ^= _m[i % 4]
+
+        if six.PY3:
+            return _d.tobytes()
+        else:
+            return _d.tostring()
+
+__all__ = [
+    'ABNF', 'continuous_frame', 'frame_buffer',
+    'STATUS_NORMAL',
+    'STATUS_GOING_AWAY',
+    'STATUS_PROTOCOL_ERROR',
+    'STATUS_UNSUPPORTED_DATA_TYPE',
+    'STATUS_STATUS_NOT_AVAILABLE',
+    'STATUS_ABNORMAL_CLOSED',
+    'STATUS_INVALID_PAYLOAD',
+    'STATUS_POLICY_VIOLATION',
+    'STATUS_MESSAGE_TOO_BIG',
+    'STATUS_INVALID_EXTENSION',
+    'STATUS_UNEXPECTED_CONDITION',
+    'STATUS_BAD_GATEWAY',
+    'STATUS_TLS_HANDSHAKE_ERROR',
+]
+
+# closing frame status codes.
+STATUS_NORMAL = 1000
+STATUS_GOING_AWAY = 1001
+STATUS_PROTOCOL_ERROR = 1002
+STATUS_UNSUPPORTED_DATA_TYPE = 1003
+STATUS_STATUS_NOT_AVAILABLE = 1005
+STATUS_ABNORMAL_CLOSED = 1006
+STATUS_INVALID_PAYLOAD = 1007
+STATUS_POLICY_VIOLATION = 1008
+STATUS_MESSAGE_TOO_BIG = 1009
+STATUS_INVALID_EXTENSION = 1010
+STATUS_UNEXPECTED_CONDITION = 1011
+STATUS_BAD_GATEWAY = 1014
+STATUS_TLS_HANDSHAKE_ERROR = 1015
+
+VALID_CLOSE_STATUS = (
+    STATUS_NORMAL,
+    STATUS_GOING_AWAY,
+    STATUS_PROTOCOL_ERROR,
+    STATUS_UNSUPPORTED_DATA_TYPE,
+    STATUS_INVALID_PAYLOAD,
+    STATUS_POLICY_VIOLATION,
+    STATUS_MESSAGE_TOO_BIG,
+    STATUS_INVALID_EXTENSION,
+    STATUS_UNEXPECTED_CONDITION,
+    STATUS_BAD_GATEWAY,
+)
+
+
+class ABNF(object):
+    """
+    ABNF frame class.
+    see http://tools.ietf.org/html/rfc5234
+    and http://tools.ietf.org/html/rfc6455#section-5.2
+    """
+
+    # operation code values.
+    OPCODE_CONT = 0x0
+    OPCODE_TEXT = 0x1
+    OPCODE_BINARY = 0x2
+    OPCODE_CLOSE = 0x8
+    OPCODE_PING = 0x9
+    OPCODE_PONG = 0xa
+
+    # available operation code value tuple
+    OPCODES = (OPCODE_CONT, OPCODE_TEXT, OPCODE_BINARY, OPCODE_CLOSE,
+               OPCODE_PING, OPCODE_PONG)
+
+    # opcode human readable string
+    OPCODE_MAP = {
+        OPCODE_CONT: "cont",
+        OPCODE_TEXT: "text",
+        OPCODE_BINARY: "binary",
+        OPCODE_CLOSE: "close",
+        OPCODE_PING: "ping",
+        OPCODE_PONG: "pong"
+    }
+
+    # data length threshold.
+    LENGTH_7 = 0x7e
+    LENGTH_16 = 1 << 16
+    LENGTH_63 = 1 << 63
+
+    def __init__(self, fin=0, rsv1=0, rsv2=0, rsv3=0,
+                 opcode=OPCODE_TEXT, mask=1, data=""):
+        """
+        Constructor for ABNF.
+        please check RFC for arguments.
+        """
+        self.fin = fin
+        self.rsv1 = rsv1
+        self.rsv2 = rsv2
+        self.rsv3 = rsv3
+        self.opcode = opcode
+        self.mask = mask
+        if data is None:
+            data = ""
+        self.data = data
+        self.get_mask_key = os.urandom
+
+    def validate(self, skip_utf8_validation=False):
+        """
+        validate the ABNF frame.
+        skip_utf8_validation: skip utf8 validation.
+        """
+        if self.rsv1 or self.rsv2 or self.rsv3:
+            raise WebSocketProtocolException("rsv is not implemented, yet")
+
+        if self.opcode not in ABNF.OPCODES:
+            raise WebSocketProtocolException("Invalid opcode %r", self.opcode)
+
+        if self.opcode == ABNF.OPCODE_PING and not self.fin:
+            raise WebSocketProtocolException("Invalid ping frame.")
+
+        if self.opcode == ABNF.OPCODE_CLOSE:
+            l = len(self.data)
+            if not l:
+                return
+            if l == 1 or l >= 126:
+                raise WebSocketProtocolException("Invalid close frame.")
+            if l > 2 and not skip_utf8_validation and not validate_utf8(self.data[2:]):
+                raise WebSocketProtocolException("Invalid close frame.")
+
+            code = 256 * \
+                six.byte2int(self.data[0:1]) + six.byte2int(self.data[1:2])
+            if not self._is_valid_close_status(code):
+                raise WebSocketProtocolException("Invalid close opcode.")
+
+    @staticmethod
+    def _is_valid_close_status(code):
+        return code in VALID_CLOSE_STATUS or (3000 <= code < 5000)
+
+    def __str__(self):
+        return "fin=" + str(self.fin) \
+            + " opcode=" + str(self.opcode) \
+            + " data=" + str(self.data)
+
+    @staticmethod
+    def create_frame(data, opcode, fin=1):
+        """
+        create frame to send text, binary and other data.
+
+        data: data to send. This is string value(byte array).
+            if opcode is OPCODE_TEXT and this value is unicode,
+            data value is converted into unicode string, automatically.
+
+        opcode: operation code. please see OPCODE_XXX.
+
+        fin: fin flag. if set to 0, create continue fragmentation.
+        """
+        if opcode == ABNF.OPCODE_TEXT and isinstance(data, six.text_type):
+            data = data.encode("utf-8")
+        # mask must be set if send data from client
+        return ABNF(fin, 0, 0, 0, opcode, 1, data)
+
+    def format(self):
+        """
+        format this object to string(byte array) to send data to server.
+        """
+        if any(x not in (0, 1) for x in [self.fin, self.rsv1, self.rsv2, self.rsv3]):
+            raise ValueError("not 0 or 1")
+        if self.opcode not in ABNF.OPCODES:
+            raise ValueError("Invalid OPCODE")
+        length = len(self.data)
+        if length >= ABNF.LENGTH_63:
+            raise ValueError("data is too long")
+
+        frame_header = chr(self.fin << 7
+                           | self.rsv1 << 6 | self.rsv2 << 5 | self.rsv3 << 4
+                           | self.opcode)
+        if length < ABNF.LENGTH_7:
+            frame_header += chr(self.mask << 7 | length)
+            frame_header = six.b(frame_header)
+        elif length < ABNF.LENGTH_16:
+            frame_header += chr(self.mask << 7 | 0x7e)
+            frame_header = six.b(frame_header)
+            frame_header += struct.pack("!H", length)
+        else:
+            frame_header += chr(self.mask << 7 | 0x7f)
+            frame_header = six.b(frame_header)
+            frame_header += struct.pack("!Q", length)
+
+        if not self.mask:
+            return frame_header + self.data
+        else:
+            mask_key = self.get_mask_key(4)
+            return frame_header + self._get_masked(mask_key)
+
+    def _get_masked(self, mask_key):
+        s = ABNF.mask(mask_key, self.data)
+
+        if isinstance(mask_key, six.text_type):
+            mask_key = mask_key.encode('utf-8')
+
+        return mask_key + s
+
+    @staticmethod
+    def mask(mask_key, data):
+        """
+        mask or unmask data. Just do xor for each byte
+
+        mask_key: 4 byte string(byte).
+
+        data: data to mask/unmask.
+        """
+        if data is None:
+            data = ""
+
+        if isinstance(mask_key, six.text_type):
+            mask_key = six.b(mask_key)
+
+        if isinstance(data, six.text_type):
+            data = six.b(data)
+
+        _m = array.array("B", mask_key)
+        _d = array.array("B", data)
+        return _mask(_m, _d)
+
+
+class frame_buffer(object):
+    _HEADER_MASK_INDEX = 5
+    _HEADER_LENGTH_INDEX = 6
+
+    def __init__(self, recv_fn, skip_utf8_validation):
+        self.recv = recv_fn
+        self.skip_utf8_validation = skip_utf8_validation
+        # Buffers over the packets from the layer beneath until desired amount
+        # bytes of bytes are received.
+        self.recv_buffer = []
+        self.clear()
+
+    def clear(self):
+        self.header = None
+        self.length = None
+        self.mask = None
+
+    def has_received_header(self):
+        return self.header is None
+
+    def recv_header(self):
+        header = self.recv_strict(2)
+        b1 = header[0]
+
+        if six.PY2:
+            b1 = ord(b1)
+
+        fin = b1 >> 7 & 1
+        rsv1 = b1 >> 6 & 1
+        rsv2 = b1 >> 5 & 1
+        rsv3 = b1 >> 4 & 1
+        opcode = b1 & 0xf
+        b2 = header[1]
+
+        if six.PY2:
+            b2 = ord(b2)
+
+        has_mask = b2 >> 7 & 1
+        length_bits = b2 & 0x7f
+
+        self.header = (fin, rsv1, rsv2, rsv3, opcode, has_mask, length_bits)
+
+    def has_mask(self):
+        if not self.header:
+            return False
+        return self.header[frame_buffer._HEADER_MASK_INDEX]
+
+    def has_received_length(self):
+        return self.length is None
+
+    def recv_length(self):
+        bits = self.header[frame_buffer._HEADER_LENGTH_INDEX]
+        length_bits = bits & 0x7f
+        if length_bits == 0x7e:
+            v = self.recv_strict(2)
+            self.length = struct.unpack("!H", v)[0]
+        elif length_bits == 0x7f:
+            v = self.recv_strict(8)
+            self.length = struct.unpack("!Q", v)[0]
+        else:
+            self.length = length_bits
+
+    def has_received_mask(self):
+        return self.mask is None
+
+    def recv_mask(self):
+        self.mask = self.recv_strict(4) if self.has_mask() else ""
+
+    def recv_frame(self):
+        # Header
+        if self.has_received_header():
+            self.recv_header()
+        (fin, rsv1, rsv2, rsv3, opcode, has_mask, _) = self.header
+
+        # Frame length
+        if self.has_received_length():
+            self.recv_length()
+        length = self.length
+
+        # Mask
+        if self.has_received_mask():
+            self.recv_mask()
+        mask = self.mask
+
+        # Payload
+        payload = self.recv_strict(length)
+        if has_mask:
+            payload = ABNF.mask(mask, payload)
+
+        # Reset for next frame
+        self.clear()
+
+        frame = ABNF(fin, rsv1, rsv2, rsv3, opcode, has_mask, payload)
+        frame.validate(self.skip_utf8_validation)
+
+        return frame
+
+    def recv_strict(self, bufsize):
+        shortage = bufsize - sum(len(x) for x in self.recv_buffer)
+        while shortage > 0:
+            # Limit buffer size that we pass to socket.recv() to avoid
+            # fragmenting the heap -- the number of bytes recv() actually
+            # reads is limited by socket buffer and is relatively small,
+            # yet passing large numbers repeatedly causes lots of large
+            # buffers allocated and then shrunk, which results in
+            # fragmentation.
+            bytes_ = self.recv(min(16384, shortage))
+            self.recv_buffer.append(bytes_)
+            shortage -= len(bytes_)
+
+        unified = six.b("").join(self.recv_buffer)
+
+        if shortage == 0:
+            self.recv_buffer = []
+            return unified
+        else:
+            self.recv_buffer = [unified[bufsize:]]
+            return unified[:bufsize]
+
+
+class continuous_frame(object):
+
+    def __init__(self, fire_cont_frame, skip_utf8_validation):
+        self.fire_cont_frame = fire_cont_frame
+        self.skip_utf8_validation = skip_utf8_validation
+        self.cont_data = None
+        self.recving_frames = None
+
+    def validate(self, frame):
+        if not self.recving_frames and frame.opcode == ABNF.OPCODE_CONT:
+            raise WebSocketProtocolException("Illegal frame")
+        if self.recving_frames and \
+                frame.opcode in (ABNF.OPCODE_TEXT, ABNF.OPCODE_BINARY):
+            raise WebSocketProtocolException("Illegal frame")
+
+    def add(self, frame):
+        if self.cont_data:
+            self.cont_data[1] += frame.data
+        else:
+            if frame.opcode in (ABNF.OPCODE_TEXT, ABNF.OPCODE_BINARY):
+                self.recving_frames = frame.opcode
+            self.cont_data = [frame.opcode, frame.data]
+
+        if frame.fin:
+            self.recving_frames = None
+
+    def is_fire(self, frame):
+        return frame.fin or self.fire_cont_frame
+
+    def extract(self, frame):
+        data = self.cont_data
+        self.cont_data = None
+        frame.data = data[1]
+        if not self.fire_cont_frame and data[0] == ABNF.OPCODE_TEXT and not self.skip_utf8_validation and not validate_utf8(frame.data):
+            raise WebSocketPayloadException(
+                "cannot decode: " + repr(frame.data))
+
+        return [data[0], frame]

Contents/Libraries/Shared/websocket/_app.py

@@ -0,0 +1,273 @@
+"""
+websocket - WebSocket client library for Python
+
+Copyright (C) 2010 Hiroki Ohtani(liris)
+
+    This library is free software; you can redistribute it and/or
+    modify it under the terms of the GNU Lesser General Public
+    License as published by the Free Software Foundation; either
+    version 2.1 of the License, or (at your option) any later version.
+
+    This library is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+    Lesser General Public License for more details.
+
+    You should have received a copy of the GNU Lesser General Public
+    License along with this library; if not, write to the Free Software
+    Foundation, Inc., 51 Franklin Street, Fifth Floor,
+    Boston, MA  02110-1335  USA
+
+"""
+
+"""
+WebSocketApp provides higher level APIs.
+"""
+import select
+import sys
+import threading
+import time
+import traceback
+
+import six
+
+from ._abnf import ABNF
+from ._core import WebSocket, getdefaulttimeout
+from ._exceptions import *
+from ._logging import *
+
+__all__ = ["WebSocketApp"]
+
+
+class WebSocketApp(object):
+    """
+    Higher level of APIs are provided.
+    The interface is like JavaScript WebSocket object.
+    """
+
+    def __init__(self, url, header=None,
+                 on_open=None, on_message=None, on_error=None,
+                 on_close=None, on_ping=None, on_pong=None,
+                 on_cont_message=None,
+                 keep_running=True, get_mask_key=None, cookie=None,
+                 subprotocols=None,
+                 on_data=None):
+        """
+        url: websocket url.
+        header: custom header for websocket handshake.
+        on_open: callable object which is called at opening websocket.
+          this function has one argument. The argument is this class object.
+        on_message: callable object which is called when received data.
+         on_message has 2 arguments.
+         The 1st argument is this class object.
+         The 2nd argument is utf-8 string which we get from the server.
+        on_error: callable object which is called when we get error.
+         on_error has 2 arguments.
+         The 1st argument is this class object.
+         The 2nd argument is exception object.
+        on_close: callable object which is called when closed the connection.
+         this function has one argument. The argument is this class object.
+        on_cont_message: callback object which is called when receive continued
+         frame data.
+         on_cont_message has 3 arguments.
+         The 1st argument is this class object.
+         The 2nd argument is utf-8 string which we get from the server.
+         The 3rd argument is continue flag. if 0, the data continue
+         to next frame data
+        on_data: callback object which is called when a message received.
+          This is called before on_message or on_cont_message,
+          and then on_message or on_cont_message is called.
+          on_data has 4 argument.
+          The 1st argument is this class object.
+          The 2nd argument is utf-8 string which we get from the server.
+          The 3rd argument is data type. ABNF.OPCODE_TEXT or ABNF.OPCODE_BINARY will be came.
+          The 4th argument is continue flag. if 0, the data continue
+        keep_running: a boolean flag indicating whether the app's main loop
+          should keep running, defaults to True
+        get_mask_key: a callable to produce new mask keys,
+          see the WebSocket.set_mask_key's docstring for more information
+        subprotocols: array of available sub protocols. default is None.
+        """
+        self.url = url
+        self.header = header if header is not None else []
+        self.cookie = cookie
+        self.on_open = on_open
+        self.on_message = on_message
+        self.on_data = on_data
+        self.on_error = on_error
+        self.on_close = on_close
+        self.on_ping = on_ping
+        self.on_pong = on_pong
+        self.on_cont_message = on_cont_message
+        self.keep_running = keep_running
+        self.get_mask_key = get_mask_key
+        self.sock = None
+        self.last_ping_tm = 0
+        self.last_pong_tm = 0
+        self.subprotocols = subprotocols
+
+    def send(self, data, opcode=ABNF.OPCODE_TEXT):
+        """
+        send message.
+        data: message to send. If you set opcode to OPCODE_TEXT,
+              data must be utf-8 string or unicode.
+        opcode: operation code of data. default is OPCODE_TEXT.
+        """
+
+        if not self.sock or self.sock.send(data, opcode) == 0:
+            raise WebSocketConnectionClosedException(
+                "Connection is already closed.")
+
+    def close(self, **kwargs):
+        """
+        close websocket connection.
+        """
+        self.keep_running = False
+        if self.sock:
+            self.sock.close(**kwargs)
+
+    def _send_ping(self, interval, event):
+        while not event.wait(interval):
+            self.last_ping_tm = time.time()
+            if self.sock:
+                try:
+                    self.sock.ping()
+                except Exception as ex:
+                    warning("send_ping routine terminated: {}".format(ex))
+                    break
+
+    def run_forever(self, sockopt=None, sslopt=None,
+                    ping_interval=0, ping_timeout=None,
+                    http_proxy_host=None, http_proxy_port=None,
+                    http_no_proxy=None, http_proxy_auth=None,
+                    skip_utf8_validation=False,
+                    host=None, origin=None):
+        """
+        run event loop for WebSocket framework.
+        This loop is infinite loop and is alive during websocket is available.
+        sockopt: values for socket.setsockopt.
+            sockopt must be tuple
+            and each element is argument of sock.setsockopt.
+        sslopt: ssl socket optional dict.
+        ping_interval: automatically send "ping" command
+            every specified period(second)
+            if set to 0, not send automatically.
+        ping_timeout: timeout(second) if the pong message is not received.
+        http_proxy_host: http proxy host name.
+        http_proxy_port: http proxy port. If not set, set to 80.
+        http_no_proxy: host names, which doesn't use proxy.
+        skip_utf8_validation: skip utf8 validation.
+        host: update host header.
+        origin: update origin header.
+        """
+
+        if not ping_timeout or ping_timeout <= 0:
+            ping_timeout = None
+        if ping_timeout and ping_interval and ping_interval <= ping_timeout:
+            raise WebSocketException("Ensure ping_interval > ping_timeout")
+        if sockopt is None:
+            sockopt = []
+        if sslopt is None:
+            sslopt = {}
+        if self.sock:
+            raise WebSocketException("socket is already opened")
+        thread = None
+        close_frame = None
+
+        try:
+            self.sock = WebSocket(
+                self.get_mask_key, sockopt=sockopt, sslopt=sslopt,
+                fire_cont_frame=self.on_cont_message and True or False,
+                skip_utf8_validation=skip_utf8_validation)
+            self.sock.settimeout(getdefaulttimeout())
+            self.sock.connect(
+                self.url, header=self.header, cookie=self.cookie,
+                http_proxy_host=http_proxy_host,
+                http_proxy_port=http_proxy_port, http_no_proxy=http_no_proxy,
+                http_proxy_auth=http_proxy_auth, subprotocols=self.subprotocols,
+                host=host, origin=origin)
+            self._callback(self.on_open)
+
+            if ping_interval:
+                event = threading.Event()
+                thread = threading.Thread(
+                    target=self._send_ping, args=(ping_interval, event))
+                thread.setDaemon(True)
+                thread.start()
+
+            while self.sock.connected:
+                r, w, e = select.select(
+                    (self.sock.sock, ), (), (), ping_timeout)
+                if not self.keep_running:
+                    break
+
+                if r:
+                    op_code, frame = self.sock.recv_data_frame(True)
+                    if op_code == ABNF.OPCODE_CLOSE:
+                        close_frame = frame
+                        break
+                    elif op_code == ABNF.OPCODE_PING:
+                        self._callback(self.on_ping, frame.data)
+                    elif op_code == ABNF.OPCODE_PONG:
+                        self.last_pong_tm = time.time()
+                        self._callback(self.on_pong, frame.data)
+                    elif op_code == ABNF.OPCODE_CONT and self.on_cont_message:
+                        self._callback(self.on_data, data,
+                                       frame.opcode, frame.fin)
+                        self._callback(self.on_cont_message,
+                                       frame.data, frame.fin)
+                    else:
+                        data = frame.data
+                        if six.PY3 and opcode == ABNF.OPCODE_TEXT:
+                            data = data.decode("utf-8")
+                        self._callback(self.on_data, data, frame.opcode, True)
+                        self._callback(self.on_message, data)
+
+                if ping_timeout and self.last_ping_tm \
+                        and time.time() - self.last_ping_tm > ping_timeout \
+                        and self.last_ping_tm - self.last_pong_tm > ping_timeout:
+                    raise WebSocketTimeoutException("ping/pong timed out")
+        except (Exception, KeyboardInterrupt, SystemExit) as e:
+            self._callback(self.on_error, e)
+            if isinstance(e, SystemExit):
+                # propagate SystemExit further
+                raise
+        finally:
+            if thread and thread.isAlive():
+                event.set()
+                thread.join()
+                self.keep_running = False
+            self.sock.close()
+            close_args = self._get_close_args(
+                close_frame.data if close_frame else None)
+            self._callback(self.on_close, *close_args)
+            self.sock = None
+
+    def _get_close_args(self, data):
+        """ this functions extracts the code, reason from the close body
+        if they exists, and if the self.on_close except three arguments """
+        import inspect
+        # if the on_close callback is "old", just return empty list
+        if sys.version_info < (3, 0):
+            if not self.on_close or len(inspect.getargspec(self.on_close).args) != 3:
+                return []
+        else:
+            if not self.on_close or len(inspect.getfullargspec(self.on_close).args) != 3:
+                return []
+
+        if data and len(data) >= 2:
+            code = 256 * six.byte2int(data[0:1]) + six.byte2int(data[1:2])
+            reason = data[2:].decode('utf-8')
+            return [code, reason]
+
+        return [None, None]
+
+    def _callback(self, callback, *args):
+        if callback:
+            try:
+                callback(self, *args)
+            except Exception as e:
+                error("error from callback {}: {}".format(callback, e))
+                if isEnabledForDebug():
+                    _, _, tb = sys.exc_info()
+                    traceback.print_tb(tb)

Contents/Libraries/Shared/websocket/_core.py

@@ -0,0 +1,488 @@
+"""
+websocket - WebSocket client library for Python
+
+Copyright (C) 2010 Hiroki Ohtani(liris)
+
+    This library is free software; you can redistribute it and/or
+    modify it under the terms of the GNU Lesser General Public
+    License as published by the Free Software Foundation; either
+    version 2.1 of the License, or (at your option) any later version.
+
+    This library is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+    Lesser General Public License for more details.
+
+    You should have received a copy of the GNU Lesser General Public
+    License along with this library; if not, write to the Free Software
+    Foundation, Inc., 51 Franklin Street, Fifth Floor,
+    Boston, MA  02110-1335  USA
+
+"""
+from __future__ import print_function
+
+import socket
+import struct
+import threading
+
+import six
+
+# websocket modules
+from ._abnf import *
+from ._exceptions import *
+from ._handshake import *
+from ._http import *
+from ._logging import *
+from ._socket import *
+from ._utils import *
+
+__all__ = ['WebSocket', 'create_connection']
+
+"""
+websocket python client.
+=========================
+
+This version support only hybi-13.
+Please see http://tools.ietf.org/html/rfc6455 for protocol.
+"""
+
+
+class WebSocket(object):
+    """
+    Low level WebSocket interface.
+    This class is based on
+      The WebSocket protocol draft-hixie-thewebsocketprotocol-76
+      http://tools.ietf.org/html/draft-hixie-thewebsocketprotocol-76
+
+    We can connect to the websocket server and send/receive data.
+    The following example is an echo client.
+
+    >>> import websocket
+    >>> ws = websocket.WebSocket()
+    >>> ws.connect("ws://echo.websocket.org")
+    >>> ws.send("Hello, Server")
+    >>> ws.recv()
+    'Hello, Server'
+    >>> ws.close()
+
+    get_mask_key: a callable to produce new mask keys, see the set_mask_key
+      function's docstring for more details
+    sockopt: values for socket.setsockopt.
+        sockopt must be tuple and each element is argument of sock.setsockopt.
+    sslopt: dict object for ssl socket option.
+    fire_cont_frame: fire recv event for each cont frame. default is False
+    enable_multithread: if set to True, lock send method.
+    skip_utf8_validation: skip utf8 validation.
+    """
+
+    def __init__(self, get_mask_key=None, sockopt=None, sslopt=None,
+                 fire_cont_frame=False, enable_multithread=False,
+                 skip_utf8_validation=False, **_):
+        """
+        Initialize WebSocket object.
+        """
+        self.sock_opt = sock_opt(sockopt, sslopt)
+        self.handshake_response = None
+        self.sock = None
+
+        self.connected = False
+        self.get_mask_key = get_mask_key
+        # These buffer over the build-up of a single frame.
+        self.frame_buffer = frame_buffer(self._recv, skip_utf8_validation)
+        self.cont_frame = continuous_frame(
+            fire_cont_frame, skip_utf8_validation)
+
+        if enable_multithread:
+            self.lock = threading.Lock()
+        else:
+            self.lock = NoLock()
+
+    def __iter__(self):
+        """
+        Allow iteration over websocket, implying sequential `recv` executions.
+        """
+        while True:
+            yield self.recv()
+
+    def __next__(self):
+        return self.recv()
+
+    def next(self):
+        return self.__next__()
+
+    def fileno(self):
+        return self.sock.fileno()
+
+    def set_mask_key(self, func):
+        """
+        set function to create musk key. You can customize mask key generator.
+        Mainly, this is for testing purpose.
+
+        func: callable object. the func takes 1 argument as integer.
+              The argument means length of mask key.
+              This func must return string(byte array),
+              which length is argument specified.
+        """
+        self.get_mask_key = func
+
+    def gettimeout(self):
+        """
+        Get the websocket timeout(second).
+        """
+        return self.sock_opt.timeout
+
+    def settimeout(self, timeout):
+        """
+        Set the timeout to the websocket.
+
+        timeout: timeout time(second).
+        """
+        self.sock_opt.timeout = timeout
+        if self.sock:
+            self.sock.settimeout(timeout)
+
+    timeout = property(gettimeout, settimeout)
+
+    def getsubprotocol(self):
+        """
+        get subprotocol
+        """
+        if self.handshake_response:
+            return self.handshake_response.subprotocol
+        else:
+            return None
+
+    subprotocol = property(getsubprotocol)
+
+    def getstatus(self):
+        """
+        get handshake status
+        """
+        if self.handshake_response:
+            return self.handshake_response.status
+        else:
+            return None
+
+    status = property(getstatus)
+
+    def getheaders(self):
+        """
+        get handshake response header
+        """
+        if self.handshake_response:
+            return self.handshake_response.headers
+        else:
+            return None
+
+    headers = property(getheaders)
+
+    def connect(self, url, **options):
+        """
+        Connect to url. url is websocket url scheme.
+        ie. ws://host:port/resource
+        You can customize using 'options'.
+        If you set "header" list object, you can set your own custom header.
+
+        >>> ws = WebSocket()
+        >>> ws.connect("ws://echo.websocket.org/",
+                ...     header=["User-Agent: MyProgram",
+                ...             "x-custom: header"])
+
+        timeout: socket timeout time. This value is integer.
+                 if you set None for this value,
+                 it means "use default_timeout value"
+
+        options: "header" -> custom http header list or dict.
+                 "cookie" -> cookie value.
+                 "origin" -> custom origin url.
+                 "host"   -> custom host header string.
+                 "http_proxy_host" - http proxy host name.
+                 "http_proxy_port" - http proxy port. If not set, set to 80.
+                 "http_no_proxy"   - host names, which doesn't use proxy.
+                 "http_proxy_auth" - http proxy auth information.
+                                     tuple of username and password.
+                                     default is None
+                 "subprotocols" - array of available sub protocols.
+                                  default is None.
+                 "socket" - pre-initialized stream socket.
+
+        """
+        self.sock, addrs = connect(url, self.sock_opt, proxy_info(**options),
+                                   options.pop('socket', None))
+
+        try:
+            self.handshake_response = handshake(self.sock, *addrs, **options)
+            self.connected = True
+        except:
+            if self.sock:
+                self.sock.close()
+                self.sock = None
+            raise
+
+    def send(self, payload, opcode=ABNF.OPCODE_TEXT):
+        """
+        Send the data as string.
+
+        payload: Payload must be utf-8 string or unicode,
+                  if the opcode is OPCODE_TEXT.
+                  Otherwise, it must be string(byte array)
+
+        opcode: operation code to send. Please see OPCODE_XXX.
+        """
+
+        frame = ABNF.create_frame(payload, opcode)
+        return self.send_frame(frame)
+
+    def send_frame(self, frame):
+        """
+        Send the data frame.
+
+        frame: frame data created  by ABNF.create_frame
+
+        >>> ws = create_connection("ws://echo.websocket.org/")
+        >>> frame = ABNF.create_frame("Hello", ABNF.OPCODE_TEXT)
+        >>> ws.send_frame(frame)
+        >>> cont_frame = ABNF.create_frame("My name is ", ABNF.OPCODE_CONT, 0)
+        >>> ws.send_frame(frame)
+        >>> cont_frame = ABNF.create_frame("Foo Bar", ABNF.OPCODE_CONT, 1)
+        >>> ws.send_frame(frame)
+
+        """
+        if self.get_mask_key:
+            frame.get_mask_key = self.get_mask_key
+        data = frame.format()
+        length = len(data)
+        trace("send: " + repr(data))
+
+        with self.lock:
+            while data:
+                l = self._send(data)
+                data = data[l:]
+
+        return length
+
+    def send_binary(self, payload):
+        return self.send(payload, ABNF.OPCODE_BINARY)
+
+    def ping(self, payload=""):
+        """
+        send ping data.
+
+        payload: data payload to send server.
+        """
+        if isinstance(payload, six.text_type):
+            payload = payload.encode("utf-8")
+        self.send(payload, ABNF.OPCODE_PING)
+
+    def pong(self, payload):
+        """
+        send pong data.
+
+        payload: data payload to send server.
+        """
+        if isinstance(payload, six.text_type):
+            payload = payload.encode("utf-8")
+        self.send(payload, ABNF.OPCODE_PONG)
+
+    def recv(self):
+        """
+        Receive string data(byte array) from the server.
+
+        return value: string(byte array) value.
+        """
+        opcode, data = self.recv_data()
+        if six.PY3 and opcode == ABNF.OPCODE_TEXT:
+            return data.decode("utf-8")
+        elif opcode == ABNF.OPCODE_TEXT or opcode == ABNF.OPCODE_BINARY:
+            return data
+        else:
+            return ''
+
+    def recv_data(self, control_frame=False):
+        """
+        Receive data with operation code.
+
+        control_frame: a boolean flag indicating whether to return control frame
+        data, defaults to False
+
+        return  value: tuple of operation code and string(byte array) value.
+        """
+        opcode, frame = self.recv_data_frame(control_frame)
+        return opcode, frame.data
+
+    def recv_data_frame(self, control_frame=False):
+        """
+        Receive data with operation code.
+
+        control_frame: a boolean flag indicating whether to return control frame
+        data, defaults to False
+
+        return  value: tuple of operation code and string(byte array) value.
+        """
+        while True:
+            frame = self.recv_frame()
+            if not frame:
+                # handle error:
+                # 'NoneType' object has no attribute 'opcode'
+                raise WebSocketProtocolException(
+                    "Not a valid frame %s" % frame)
+            elif frame.opcode in (ABNF.OPCODE_TEXT, ABNF.OPCODE_BINARY, ABNF.OPCODE_CONT):
+                self.cont_frame.validate(frame)
+                self.cont_frame.add(frame)
+
+                if self.cont_frame.is_fire(frame):
+                    return self.cont_frame.extract(frame)
+
+            elif frame.opcode == ABNF.OPCODE_CLOSE:
+                self.send_close()
+                return frame.opcode, frame
+            elif frame.opcode == ABNF.OPCODE_PING:
+                if len(frame.data) < 126:
+                    self.pong(frame.data)
+                else:
+                    raise WebSocketProtocolException(
+                        "Ping message is too long")
+                if control_frame:
+                    return frame.opcode, frame
+            elif frame.opcode == ABNF.OPCODE_PONG:
+                if control_frame:
+                    return frame.opcode, frame
+
+    def recv_frame(self):
+        """
+        receive data as frame from server.
+
+        return value: ABNF frame object.
+        """
+        return self.frame_buffer.recv_frame()
+
+    def send_close(self, status=STATUS_NORMAL, reason=six.b("")):
+        """
+        send close data to the server.
+
+        status: status code to send. see STATUS_XXX.
+
+        reason: the reason to close. This must be string or bytes.
+        """
+        if status < 0 or status >= ABNF.LENGTH_16:
+            raise ValueError("code is invalid range")
+        self.connected = False
+        self.send(struct.pack('!H', status) + reason, ABNF.OPCODE_CLOSE)
+
+    def close(self, status=STATUS_NORMAL, reason=six.b(""), timeout=3):
+        """
+        Close Websocket object
+
+        status: status code to send. see STATUS_XXX.
+
+        reason: the reason to close. This must be string.
+
+        timeout: timeout until receive a close frame.
+            If None, it will wait forever until receive a close frame.
+        """
+        if self.connected:
+            if status < 0 or status >= ABNF.LENGTH_16:
+                raise ValueError("code is invalid range")
+
+            try:
+                self.connected = False
+                self.send(struct.pack('!H', status) +
+                          reason, ABNF.OPCODE_CLOSE)
+                sock_timeout = self.sock.gettimeout()
+                self.sock.settimeout(timeout)
+                try:
+                    frame = self.recv_frame()
+                    if isEnabledForError():
+                        recv_status = struct.unpack("!H", frame.data)[0]
+                        if recv_status != STATUS_NORMAL:
+                            error("close status: " + repr(recv_status))
+                except:
+                    pass
+                self.sock.settimeout(sock_timeout)
+                self.sock.shutdown(socket.SHUT_RDWR)
+            except:
+                pass
+
+        self.shutdown()
+
+    def abort(self):
+        """
+        Low-level asynchronous abort, wakes up other threads that are waiting in recv_*
+        """
+        if self.connected:
+            self.sock.shutdown(socket.SHUT_RDWR)
+
+    def shutdown(self):
+        """close socket, immediately."""
+        if self.sock:
+            self.sock.close()
+            self.sock = None
+            self.connected = False
+
+    def _send(self, data):
+        return send(self.sock, data)
+
+    def _recv(self, bufsize):
+        try:
+            return recv(self.sock, bufsize)
+        except WebSocketConnectionClosedException:
+            if self.sock:
+                self.sock.close()
+            self.sock = None
+            self.connected = False
+            raise
+
+
+def create_connection(url, timeout=None, class_=WebSocket, **options):
+    """
+    connect to url and return websocket object.
+
+    Connect to url and return the WebSocket object.
+    Passing optional timeout parameter will set the timeout on the socket.
+    If no timeout is supplied,
+    the global default timeout setting returned by getdefauttimeout() is used.
+    You can customize using 'options'.
+    If you set "header" list object, you can set your own custom header.
+
+    >>> conn = create_connection("ws://echo.websocket.org/",
+         ...     header=["User-Agent: MyProgram",
+         ...             "x-custom: header"])
+
+
+    timeout: socket timeout time. This value is integer.
+             if you set None for this value,
+             it means "use default_timeout value"
+
+    class_: class to instantiate when creating the connection. It has to implement
+            settimeout and connect. It's __init__ should be compatible with
+            WebSocket.__init__, i.e. accept all of it's kwargs.
+    options: "header" -> custom http header list or dict.
+             "cookie" -> cookie value.
+             "origin" -> custom origin url.
+             "host"   -> custom host header string.
+             "http_proxy_host" - http proxy host name.
+             "http_proxy_port" - http proxy port. If not set, set to 80.
+             "http_no_proxy"   - host names, which doesn't use proxy.
+             "http_proxy_auth" - http proxy auth information.
+                                    tuple of username and password.
+                                    default is None
+             "enable_multithread" -> enable lock for multithread.
+             "sockopt" -> socket options
+             "sslopt" -> ssl option
+             "subprotocols" - array of available sub protocols.
+                              default is None.
+             "skip_utf8_validation" - skip utf8 validation.
+             "socket" - pre-initialized stream socket.
+    """
+    sockopt = options.pop("sockopt", [])
+    sslopt = options.pop("sslopt", {})
+    fire_cont_frame = options.pop("fire_cont_frame", False)
+    enable_multithread = options.pop("enable_multithread", False)
+    skip_utf8_validation = options.pop("skip_utf8_validation", False)
+    websock = class_(sockopt=sockopt, sslopt=sslopt,
+                     fire_cont_frame=fire_cont_frame,
+                     enable_multithread=enable_multithread,
+                     skip_utf8_validation=skip_utf8_validation, **options)
+    websock.settimeout(timeout if timeout is not None else getdefaulttimeout())
+    websock.connect(url, **options)
+    return websock

Contents/Libraries/Shared/websocket/_exceptions.py

@@ -0,0 +1,80 @@
+"""
+websocket - WebSocket client library for Python
+
+Copyright (C) 2010 Hiroki Ohtani(liris)
+
+    This library is free software; you can redistribute it and/or
+    modify it under the terms of the GNU Lesser General Public
+    License as published by the Free Software Foundation; either
+    version 2.1 of the License, or (at your option) any later version.
+
+    This library is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+    Lesser General Public License for more details.
+
+    You should have received a copy of the GNU Lesser General Public
+    License along with this library; if not, write to the Free Software
+    Foundation, Inc., 51 Franklin Street, Fifth Floor,
+    Boston, MA  02110-1335  USA
+
+"""
+
+
+"""
+define websocket exceptions
+"""
+
+
+class WebSocketException(Exception):
+    """
+    websocket exception class.
+    """
+    pass
+
+
+class WebSocketProtocolException(WebSocketException):
+    """
+    If the websocket protocol is invalid, this exception will be raised.
+    """
+    pass
+
+
+class WebSocketPayloadException(WebSocketException):
+    """
+    If the websocket payload is invalid, this exception will be raised.
+    """
+    pass
+
+
+class WebSocketConnectionClosedException(WebSocketException):
+    """
+    If remote host closed the connection or some network error happened,
+    this exception will be raised.
+    """
+    pass
+
+
+class WebSocketTimeoutException(WebSocketException):
+    """
+    WebSocketTimeoutException will be raised at socket timeout during read/write data.
+    """
+    pass
+
+
+class WebSocketProxyException(WebSocketException):
+    """
+    WebSocketProxyException will be raised when proxy error occurred.
+    """
+    pass
+
+
+class WebSocketBadStatusException(WebSocketException):
+    """
+    WebSocketBadStatusException will be raised when we get bad handshake status code.
+    """
+
+    def __init__(self, message, status_code):
+        super(WebSocketBadStatusException, self).__init__(
+            message % status_code)
+        self.status_code = status_code

Contents/Libraries/Shared/websocket/_handshake.py

@@ -0,0 +1,167 @@
+"""
+websocket - WebSocket client library for Python
+
+Copyright (C) 2010 Hiroki Ohtani(liris)
+
+    This library is free software; you can redistribute it and/or
+    modify it under the terms of the GNU Lesser General Public
+    License as published by the Free Software Foundation; either
+    version 2.1 of the License, or (at your option) any later version.
+
+    This library is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+    Lesser General Public License for more details.
+
+    You should have received a copy of the GNU Lesser General Public
+    License along with this library; if not, write to the Free Software
+    Foundation, Inc., 51 Franklin Street, Fifth Floor,
+    Boston, MA  02110-1335  USA
+
+"""
+import hashlib
+import hmac
+import os
+
+import six
+
+from ._exceptions import *
+from ._http import *
+from ._logging import *
+from ._socket import *
+
+if six.PY3:
+    from base64 import encodebytes as base64encode
+else:
+    from base64 import encodestring as base64encode
+
+__all__ = ["handshake_response", "handshake"]
+
+if hasattr(hmac, "compare_digest"):
+    compare_digest = hmac.compare_digest
+else:
+    def compare_digest(s1, s2):
+        return s1 == s2
+
+# websocket supported version.
+VERSION = 13
+
+
+class handshake_response(object):
+
+    def __init__(self, status, headers, subprotocol):
+        self.status = status
+        self.headers = headers
+        self.subprotocol = subprotocol
+
+
+def handshake(sock, hostname, port, resource, **options):
+    headers, key = _get_handshake_headers(resource, hostname, port, options)
+
+    header_str = "\r\n".join(headers)
+    send(sock, header_str)
+    dump("request header", header_str)
+
+    status, resp = _get_resp_headers(sock)
+    success, subproto = _validate(resp, key, options.get("subprotocols"))
+    if not success:
+        raise WebSocketException("Invalid WebSocket Header")
+
+    return handshake_response(status, resp, subproto)
+
+
+def _get_handshake_headers(resource, host, port, options):
+    headers = [
+        "GET %s HTTP/1.1" % resource,
+        "Upgrade: websocket",
+        "Connection: Upgrade"
+    ]
+    if port == 80 or port == 443:
+        hostport = host
+    else:
+        hostport = "%s:%d" % (host, port)
+
+    if "host" in options and options["host"]:
+        headers.append("Host: %s" % options["host"])
+    else:
+        headers.append("Host: %s" % hostport)
+
+    if "origin" in options and options["origin"]:
+        headers.append("Origin: %s" % options["origin"])
+    else:
+        headers.append("Origin: http://%s" % hostport)
+
+    key = _create_sec_websocket_key()
+    headers.append("Sec-WebSocket-Key: %s" % key)
+    headers.append("Sec-WebSocket-Version: %s" % VERSION)
+
+    subprotocols = options.get("subprotocols")
+    if subprotocols:
+        headers.append("Sec-WebSocket-Protocol: %s" % ",".join(subprotocols))
+
+    if "header" in options:
+        header = options["header"]
+        if isinstance(header, dict):
+            header = map(": ".join, header.items())
+        headers.extend(header)
+
+    cookie = options.get("cookie", None)
+
+    if cookie:
+        headers.append("Cookie: %s" % cookie)
+
+    headers.append("")
+    headers.append("")
+
+    return headers, key
+
+
+def _get_resp_headers(sock, success_status=101):
+    status, resp_headers = read_headers(sock)
+    if status != success_status:
+        raise WebSocketBadStatusException("Handshake status %d", status)
+    return status, resp_headers
+
+_HEADERS_TO_CHECK = {
+    "upgrade": "websocket",
+    "connection": "upgrade",
+}
+
+
+def _validate(headers, key, subprotocols):
+    subproto = None
+    for k, v in _HEADERS_TO_CHECK.items():
+        r = headers.get(k, None)
+        if not r:
+            return False, None
+        r = r.lower()
+        if v != r:
+            return False, None
+
+    if subprotocols:
+        subproto = headers.get("sec-websocket-protocol", None).lower()
+        if not subproto or subproto not in [s.lower() for s in subprotocols]:
+            error("Invalid subprotocol: " + str(subprotocols))
+            return False, None
+
+    result = headers.get("sec-websocket-accept", None)
+    if not result:
+        return False, None
+    result = result.lower()
+
+    if isinstance(result, six.text_type):
+        result = result.encode('utf-8')
+
+    value = (key + "258EAFA5-E914-47DA-95CA-C5AB0DC85B11").encode('utf-8')
+    hashed = base64encode(hashlib.sha1(value).digest()).strip().lower()
+    success = compare_digest(hashed, result)
+
+    if success:
+        return True, subproto
+    else:
+        return False, None
+
+
+def _create_sec_websocket_key():
+    randomness = os.urandom(16)
+    return base64encode(randomness).decode('utf-8').strip()

Contents/Libraries/Shared/websocket/_http.py

@@ -0,0 +1,244 @@
+"""
+websocket - WebSocket client library for Python
+
+Copyright (C) 2010 Hiroki Ohtani(liris)
+
+    This library is free software; you can redistribute it and/or
+    modify it under the terms of the GNU Lesser General Public
+    License as published by the Free Software Foundation; either
+    version 2.1 of the License, or (at your option) any later version.
+
+    This library is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+    Lesser General Public License for more details.
+
+    You should have received a copy of the GNU Lesser General Public
+    License along with this library; if not, write to the Free Software
+    Foundation, Inc., 51 Franklin Street, Fifth Floor,
+    Boston, MA  02110-1335  USA
+
+"""
+import errno
+import os
+import socket
+import sys
+
+import six
+
+from ._exceptions import *
+from ._logging import *
+from ._socket import*
+from ._ssl_compat import *
+from ._url import *
+
+if six.PY3:
+    from base64 import encodebytes as base64encode
+else:
+    from base64 import encodestring as base64encode
+
+__all__ = ["proxy_info", "connect", "read_headers"]
+
+
+class proxy_info(object):
+
+    def __init__(self, **options):
+        self.host = options.get("http_proxy_host", None)
+        if self.host:
+            self.port = options.get("http_proxy_port", 0)
+            self.auth = options.get("http_proxy_auth", None)
+            self.no_proxy = options.get("http_no_proxy", None)
+        else:
+            self.port = 0
+            self.auth = None
+            self.no_proxy = None
+
+
+def connect(url, options, proxy, socket):
+    hostname, port, resource, is_secure = parse_url(url)
+
+    if socket:
+        return socket, (hostname, port, resource)
+
+    addrinfo_list, need_tunnel, auth = _get_addrinfo_list(
+        hostname, port, is_secure, proxy)
+    if not addrinfo_list:
+        raise WebSocketException(
+            "Host not found.: " + hostname + ":" + str(port))
+
+    sock = None
+    try:
+        sock = _open_socket(addrinfo_list, options.sockopt, options.timeout)
+        if need_tunnel:
+            sock = _tunnel(sock, hostname, port, auth)
+
+        if is_secure:
+            if HAVE_SSL:
+                sock = _ssl_socket(sock, options.sslopt, hostname)
+            else:
+                raise WebSocketException("SSL not available.")
+
+        return sock, (hostname, port, resource)
+    except:
+        if sock:
+            sock.close()
+        raise
+
+
+def _get_addrinfo_list(hostname, port, is_secure, proxy):
+    phost, pport, pauth = get_proxy_info(
+        hostname, is_secure, proxy.host, proxy.port, proxy.auth, proxy.no_proxy)
+    if not phost:
+        addrinfo_list = socket.getaddrinfo(
+            hostname, port, 0, 0, socket.SOL_TCP)
+        return addrinfo_list, False, None
+    else:
+        pport = pport and pport or 80
+        addrinfo_list = socket.getaddrinfo(phost, pport, 0, 0, socket.SOL_TCP)
+        return addrinfo_list, True, pauth
+
+
+def _open_socket(addrinfo_list, sockopt, timeout):
+    err = None
+    for addrinfo in addrinfo_list:
+        family = addrinfo[0]
+        sock = socket.socket(family)
+        sock.settimeout(timeout)
+
+        for opts in DEFAULT_SOCKET_OPTION + sockopt:
+            try:
+                sock.setsockopt(*opts)
+            except socket.error:
+                info('Unable to set option: %r', opts)
+
+        address = addrinfo[4]
+        try:
+            sock.connect(address)
+        except socket.error as error:
+            error.remote_ip = str(address[0])
+            if error.errno in (errno.ECONNREFUSED, ):
+                err = error
+                continue
+            else:
+                raise
+        else:
+            break
+    else:
+        raise err
+
+    return sock
+
+
+def _can_use_sni():
+    return six.PY2 and sys.version_info >= (2, 7, 9) or sys.version_info >= (3, 2)
+
+
+def _wrap_sni_socket(sock, sslopt, hostname, check_hostname):
+    context = ssl.SSLContext(sslopt.get('ssl_version', ssl.PROTOCOL_SSLv23))
+
+    if sslopt.get('cert_reqs', ssl.CERT_NONE) != ssl.CERT_NONE:
+        context.load_verify_locations(cafile=sslopt.get('ca_certs', None))
+    if sslopt.get('certfile', None):
+        context.load_cert_chain(
+            sslopt['certfile'],
+            sslopt.get('keyfile', None),
+            sslopt.get('password', None),
+        )
+    # see
+    # https://github.com/liris/websocket-client/commit/b96a2e8fa765753e82eea531adb19716b52ca3ca#commitcomment-10803153
+    context.verify_mode = sslopt['cert_reqs']
+    if HAVE_CONTEXT_CHECK_HOSTNAME:
+        context.check_hostname = check_hostname
+    if 'ciphers' in sslopt:
+        context.set_ciphers(sslopt['ciphers'])
+    if 'cert_chain' in sslopt:
+        certfile, keyfile, password = sslopt['cert_chain']
+        context.load_cert_chain(certfile, keyfile, password)
+
+    return context.wrap_socket(
+        sock,
+        do_handshake_on_connect=sslopt.get('do_handshake_on_connect', True),
+        suppress_ragged_eofs=sslopt.get('suppress_ragged_eofs', True),
+        server_hostname=hostname,
+    )
+
+
+def _ssl_socket(sock, user_sslopt, hostname):
+    sslopt = dict(cert_reqs=ssl.CERT_REQUIRED)
+    sslopt.update(user_sslopt)
+
+    if os.environ.get('WEBSOCKET_CLIENT_CA_BUNDLE'):
+        certPath = os.environ.get('WEBSOCKET_CLIENT_CA_BUNDLE')
+    else:
+        certPath = os.path.join(
+            os.path.dirname(__file__), "cacert.pem")
+    if os.path.isfile(certPath) and user_sslopt.get('ca_certs', None) is None:
+        sslopt['ca_certs'] = certPath
+    check_hostname = sslopt["cert_reqs"] != ssl.CERT_NONE and sslopt.pop(
+        'check_hostname', True)
+
+    if _can_use_sni():
+        sock = _wrap_sni_socket(sock, sslopt, hostname, check_hostname)
+    else:
+        sslopt.pop('check_hostname', True)
+        sock = ssl.wrap_socket(sock, **sslopt)
+
+    if not HAVE_CONTEXT_CHECK_HOSTNAME and check_hostname:
+        match_hostname(sock.getpeercert(), hostname)
+
+    return sock
+
+
+def _tunnel(sock, host, port, auth):
+    debug("Connecting proxy...")
+    connect_header = "CONNECT %s:%d HTTP/1.0\r\n" % (host, port)
+    # TODO: support digest auth.
+    if auth and auth[0]:
+        auth_str = auth[0]
+        if auth[1]:
+            auth_str += ":" + auth[1]
+        encoded_str = base64encode(auth_str.encode()).strip().decode()
+        connect_header += "Proxy-Authorization: Basic %s\r\n" % encoded_str
+    connect_header += "\r\n"
+    dump("request header", connect_header)
+
+    send(sock, connect_header)
+
+    try:
+        status, resp_headers = read_headers(sock)
+    except Exception as e:
+        raise WebSocketProxyException(str(e))
+
+    if status != 200:
+        raise WebSocketProxyException(
+            "failed CONNECT via proxy status: %r" % status)
+
+    return sock
+
+
+def read_headers(sock):
+    status = None
+    headers = {}
+    trace("--- response header ---")
+
+    while True:
+        line = recv_line(sock)
+        line = line.decode('utf-8').strip()
+        if not line:
+            break
+        trace(line)
+        if not status:
+
+            status_info = line.split(" ", 2)
+            status = int(status_info[1])
+        else:
+            kv = line.split(":", 1)
+            if len(kv) == 2:
+                key, value = kv
+                headers[key.lower()] = value.strip()
+            else:
+                raise WebSocketException("Invalid header")
+
+    trace("-----------------------")
+
+    return status, headers

Contents/Libraries/Shared/websocket/_logging.py

@@ -0,0 +1,78 @@
+"""
+websocket - WebSocket client library for Python
+
+Copyright (C) 2010 Hiroki Ohtani(liris)
+
+    This library is free software; you can redistribute it and/or
+    modify it under the terms of the GNU Lesser General Public
+    License as published by the Free Software Foundation; either
+    version 2.1 of the License, or (at your option) any later version.
+
+    This library is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+    Lesser General Public License for more details.
+
+    You should have received a copy of the GNU Lesser General Public
+    License along with this library; if not, write to the Free Software
+    Foundation, Inc., 51 Franklin Street, Fifth Floor,
+    Boston, MA  02110-1335  USA
+
+"""
+import logging
+
+_logger = logging.getLogger('websocket')
+_traceEnabled = False
+
+__all__ = ["enableTrace", "dump", "error", "info", "debug", "trace",
+           "isEnabledForError", "isEnabledForDebug"]
+
+
+def enableTrace(traceable):
+    """
+    turn on/off the traceability.
+
+    traceable: boolean value. if set True, traceability is enabled.
+    """
+    global _traceEnabled
+    _traceEnabled = traceable
+    if traceable:
+        if not _logger.handlers:
+            _logger.addHandler(logging.StreamHandler())
+        _logger.setLevel(logging.DEBUG)
+
+
+def dump(title, message):
+    if _traceEnabled:
+        _logger.debug("--- " + title + " ---")
+        _logger.debug(message)
+        _logger.debug("-----------------------")
+
+
+def error(msg):
+    _logger.error(msg)
+
+
+def warning(msg):
+    _logger.warning(msg)
+
+
+def info(msg, *args):
+    _logger.info(msg, *args)
+
+
+def debug(msg):
+    _logger.debug(msg)
+
+
+def trace(msg):
+    if _traceEnabled:
+        _logger.debug(msg)
+
+
+def isEnabledForError():
+    return _logger.isEnabledFor(logging.ERROR)
+
+
+def isEnabledForDebug():
+    return _logger.isEnabledFor(logging.DEBUG)

Contents/Libraries/Shared/websocket/_socket.py

@@ -0,0 +1,125 @@
+"""
+websocket - WebSocket client library for Python
+
+Copyright (C) 2010 Hiroki Ohtani(liris)
+
+    This library is free software; you can redistribute it and/or
+    modify it under the terms of the GNU Lesser General Public
+    License as published by the Free Software Foundation; either
+    version 2.1 of the License, or (at your option) any later version.
+
+    This library is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+    Lesser General Public License for more details.
+
+    You should have received a copy of the GNU Lesser General Public
+    License along with this library; if not, write to the Free Software
+    Foundation, Inc., 51 Franklin Street, Fifth Floor,
+    Boston, MA 02110-1335  USA
+
+"""
+import socket
+
+import six
+
+from ._exceptions import *
+from ._ssl_compat import *
+from ._utils import *
+
+DEFAULT_SOCKET_OPTION = [(socket.SOL_TCP, socket.TCP_NODELAY, 1)]
+if hasattr(socket, "SO_KEEPALIVE"):
+    DEFAULT_SOCKET_OPTION.append((socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1))
+if hasattr(socket, "TCP_KEEPIDLE"):
+    DEFAULT_SOCKET_OPTION.append((socket.SOL_TCP, socket.TCP_KEEPIDLE, 30))
+if hasattr(socket, "TCP_KEEPINTVL"):
+    DEFAULT_SOCKET_OPTION.append((socket.SOL_TCP, socket.TCP_KEEPINTVL, 10))
+if hasattr(socket, "TCP_KEEPCNT"):
+    DEFAULT_SOCKET_OPTION.append((socket.SOL_TCP, socket.TCP_KEEPCNT, 3))
+
+_default_timeout = None
+
+__all__ = ["DEFAULT_SOCKET_OPTION", "sock_opt", "setdefaulttimeout", "getdefaulttimeout",
+           "recv", "recv_line", "send"]
+
+
+class sock_opt(object):
+
+    def __init__(self, sockopt, sslopt):
+        if sockopt is None:
+            sockopt = []
+        if sslopt is None:
+            sslopt = {}
+        self.sockopt = sockopt
+        self.sslopt = sslopt
+        self.timeout = None
+
+
+def setdefaulttimeout(timeout):
+    """
+    Set the global timeout setting to connect.
+
+    timeout: default socket timeout time. This value is second.
+    """
+    global _default_timeout
+    _default_timeout = timeout
+
+
+def getdefaulttimeout():
+    """
+    Return the global timeout setting(second) to connect.
+    """
+    return _default_timeout
+
+
+def recv(sock, bufsize):
+    if not sock:
+        raise WebSocketConnectionClosedException("socket is already closed.")
+
+    try:
+        bytes_ = sock.recv(bufsize)
+    except socket.timeout as e:
+        message = extract_err_message(e)
+        raise WebSocketTimeoutException(message)
+    except SSLError as e:
+        message = extract_err_message(e)
+        if message == "The read operation timed out":
+            raise WebSocketTimeoutException(message)
+        else:
+            raise
+
+    if not bytes_:
+        raise WebSocketConnectionClosedException(
+            "Connection is already closed.")
+
+    return bytes_
+
+
+def recv_line(sock):
+    line = []
+    while True:
+        c = recv(sock, 1)
+        line.append(c)
+        if c == six.b("\n"):
+            break
+    return six.b("").join(line)
+
+
+def send(sock, data):
+    if isinstance(data, six.text_type):
+        data = data.encode('utf-8')
+
+    if not sock:
+        raise WebSocketConnectionClosedException("socket is already closed.")
+
+    try:
+        return sock.send(data)
+    except socket.timeout as e:
+        message = extract_err_message(e)
+        raise WebSocketTimeoutException(message)
+    except Exception as e:
+        message = extract_err_message(e)
+        if isinstance(message, str) and "timed out" in message:
+            raise WebSocketTimeoutException(message)
+        else:
+            raise

Contents/Libraries/Shared/websocket/_ssl_compat.py

@@ -0,0 +1,44 @@
+"""
+websocket - WebSocket client library for Python
+
+Copyright (C) 2010 Hiroki Ohtani(liris)
+
+    This library is free software; you can redistribute it and/or
+    modify it under the terms of the GNU Lesser General Public
+    License as published by the Free Software Foundation; either
+    version 2.1 of the License, or (at your option) any later version.
+
+    This library is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+    Lesser General Public License for more details.
+
+    You should have received a copy of the GNU Lesser General Public
+    License along with this library; if not, write to the Free Software
+    Foundation, Inc., 51 Franklin Street, Fifth Floor,
+    Boston, MA 02110-1335  USA
+
+"""
+__all__ = ["HAVE_SSL", "ssl", "SSLError"]
+
+try:
+    import ssl
+    from ssl import SSLError
+    if hasattr(ssl, 'SSLContext') and hasattr(ssl.SSLContext, 'check_hostname'):
+        HAVE_CONTEXT_CHECK_HOSTNAME = True
+    else:
+        HAVE_CONTEXT_CHECK_HOSTNAME = False
+        if hasattr(ssl, "match_hostname"):
+            from ssl import match_hostname
+        else:
+            from backports.ssl_match_hostname import match_hostname
+        __all__.append("match_hostname")
+    __all__.append("HAVE_CONTEXT_CHECK_HOSTNAME")
+
+    HAVE_SSL = True
+except ImportError:
+    # dummy class of SSLError for ssl none-support environment.
+    class SSLError(Exception):
+        pass
+
+    HAVE_SSL = False

Contents/Libraries/Shared/websocket/_url.py

@@ -0,0 +1,166 @@
+"""
+websocket - WebSocket client library for Python
+
+Copyright (C) 2010 Hiroki Ohtani(liris)
+
+    This library is free software; you can redistribute it and/or
+    modify it under the terms of the GNU Lesser General Public
+    License as published by the Free Software Foundation; either
+    version 2.1 of the License, or (at your option) any later version.
+
+    This library is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+    Lesser General Public License for more details.
+
+    You should have received a copy of the GNU Lesser General Public
+    License along with this library; if not, write to the Free Software
+    Foundation, Inc., 51 Franklin Street, Fifth Floor,
+    Boston, MA  02110-1335  USA
+
+"""
+
+import os
+import socket
+import struct
+
+from six.moves.urllib.parse import urlparse
+
+
+__all__ = ["parse_url", "get_proxy_info"]
+
+
+def parse_url(url):
+    """
+    parse url and the result is tuple of
+    (hostname, port, resource path and the flag of secure mode)
+
+    url: url string.
+    """
+    if ":" not in url:
+        raise ValueError("url is invalid")
+
+    scheme, url = url.split(":", 1)
+
+    parsed = urlparse(url, scheme="ws")
+    if parsed.hostname:
+        hostname = parsed.hostname
+    else:
+        raise ValueError("hostname is invalid")
+    port = 0
+    if parsed.port:
+        port = parsed.port
+
+    is_secure = False
+    if scheme == "ws":
+        if not port:
+            port = 80
+    elif scheme == "wss":
+        is_secure = True
+        if not port:
+            port = 443
+    else:
+        raise ValueError("scheme %s is invalid" % scheme)
+
+    if parsed.path:
+        resource = parsed.path
+    else:
+        resource = "/"
+
+    if parsed.query:
+        resource += "?" + parsed.query
+
+    return hostname, port, resource, is_secure
+
+
+DEFAULT_NO_PROXY_HOST = ["localhost", "127.0.0.1"]
+
+
+def _is_ip_address(addr):
+    try:
+        socket.inet_aton(addr)
+    except socket.error:
+        return False
+    else:
+        return True
+
+
+def _is_subnet_address(hostname):
+    try:
+        addr, netmask = hostname.split("/")
+        return _is_ip_address(addr) and 0 <= int(netmask) < 32
+    except ValueError:
+        return False
+
+
+def _is_address_in_network(ip, net):
+    ipaddr = struct.unpack('I', socket.inet_aton(ip))[0]
+    netaddr, bits = net.split('/')
+    netmask = struct.unpack('I', socket.inet_aton(netaddr))[0] & ((2 << int(bits) - 1) - 1)
+    return ipaddr & netmask == netmask
+
+
+def _is_no_proxy_host(hostname, no_proxy):
+    # Retrieve "no_proxy" variable from environment
+    if not no_proxy:
+        value = os.environ.get("no_proxy", "").replace(" ", "")
+
+        # Split environment variable into hostname values (and ignore empty values)
+        no_proxy = [v for v in value.split(",") if v]
+
+    # Use default value (if none provided)
+    if not no_proxy:
+        no_proxy = DEFAULT_NO_PROXY_HOST
+
+    # Check if `hostname` should ignore the proxy
+    if hostname in no_proxy:
+        return True
+    elif _is_ip_address(hostname):
+        return any([_is_address_in_network(hostname, subnet) for subnet in no_proxy if _is_subnet_address(subnet)])
+
+    return False
+
+
+def get_proxy_info(
+        hostname, is_secure, proxy_host=None, proxy_port=0, proxy_auth=None,
+        no_proxy=None):
+    """
+    try to retrieve proxy host and port from environment
+    if not provided in options.
+    result is (proxy_host, proxy_port, proxy_auth).
+    proxy_auth is tuple of username and password
+     of proxy authentication information.
+
+    hostname: websocket server name.
+
+    is_secure:  is the connection secure? (wss)
+                looks for "https_proxy" in env
+                before falling back to "http_proxy"
+
+    options:    "http_proxy_host" - http proxy host name.
+                "http_proxy_port" - http proxy port.
+                "http_no_proxy"   - host names, which doesn't use proxy.
+                "http_proxy_auth" - http proxy auth information.
+                                    tuple of username and password.
+                                    default is None
+    """
+    if _is_no_proxy_host(hostname, no_proxy):
+        return None, 0, None
+
+    if proxy_host:
+        port = proxy_port
+        auth = proxy_auth
+        return proxy_host, port, auth
+
+    env_keys = ["http_proxy"]
+    if is_secure:
+        env_keys.insert(0, "https_proxy")
+
+    for key in env_keys:
+        value = os.environ.get(key, None)
+        if value:
+            proxy = urlparse(value)
+            auth = (proxy.username, proxy.password) if proxy.username else None
+            return proxy.hostname, proxy.port, auth
+
+    return None, 0, None

Contents/Libraries/Shared/websocket/_utils.py

@@ -0,0 +1,105 @@
+"""
+websocket - WebSocket client library for Python
+
+Copyright (C) 2010 Hiroki Ohtani(liris)
+
+    This library is free software; you can redistribute it and/or
+    modify it under the terms of the GNU Lesser General Public
+    License as published by the Free Software Foundation; either
+    version 2.1 of the License, or (at your option) any later version.
+
+    This library is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+    Lesser General Public License for more details.
+
+    You should have received a copy of the GNU Lesser General Public
+    License along with this library; if not, write to the Free Software
+    Foundation, Inc., 51 Franklin Street, Fifth Floor,
+    Boston, MA 02110-1335  USA
+
+"""
+import six
+
+__all__ = ["NoLock", "validate_utf8", "extract_err_message"]
+
+
+class NoLock(object):
+
+    def __enter__(self):
+        pass
+
+    def __exit__(self, exc_type, exc_value, traceback):
+        pass
+
+try:
+    # If wsaccel is available we use compiled routines to validate UTF-8
+    # strings.
+    from wsaccel.utf8validator import Utf8Validator
+
+    def _validate_utf8(utfbytes):
+        return Utf8Validator().validate(utfbytes)[0]
+
+except ImportError:
+    # UTF-8 validator
+    # python implementation of http://bjoern.hoehrmann.de/utf-8/decoder/dfa/
+
+    _UTF8_ACCEPT = 0
+    _UTF8_REJECT = 12
+
+    _UTF8D = [
+        # The first part of the table maps bytes to character classes that
+        # to reduce the size of the transition table and create bitmasks.
+        0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,  0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
+        0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,  0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
+        0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,  0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
+        0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,  0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
+        1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,  9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,
+        7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,  7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,
+        8,8,2,2,2,2,2,2,2,2,2,2,2,2,2,2,  2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,
+        10,3,3,3,3,3,3,3,3,3,3,3,3,4,3,3, 11,6,6,6,5,8,8,8,8,8,8,8,8,8,8,8,
+
+        # The second part is a transition table that maps a combination
+        # of a state of the automaton and a character class to a state.
+        0,12,24,36,60,96,84,12,12,12,48,72, 12,12,12,12,12,12,12,12,12,12,12,12,
+        12, 0,12,12,12,12,12, 0,12, 0,12,12, 12,24,12,12,12,12,12,24,12,24,12,12,
+        12,12,12,12,12,12,12,24,12,12,12,12, 12,24,12,12,12,12,12,12,12,24,12,12,
+        12,12,12,12,12,12,12,36,12,36,12,12, 12,36,12,12,12,12,12,36,12,36,12,12,
+        12,36,12,12,12,12,12,12,12,12,12,12, ]
+
+    def _decode(state, codep, ch):
+        tp = _UTF8D[ch]
+
+        codep = (ch & 0x3f) | (codep << 6) if (
+            state != _UTF8_ACCEPT) else (0xff >> tp) & ch
+        state = _UTF8D[256 + state + tp]
+
+        return state, codep
+
+    def _validate_utf8(utfbytes):
+        state = _UTF8_ACCEPT
+        codep = 0
+        for i in utfbytes:
+            if six.PY2:
+                i = ord(i)
+            state, codep = _decode(state, codep, i)
+            if state == _UTF8_REJECT:
+                return False
+
+        return True
+
+
+def validate_utf8(utfbytes):
+    """
+    validate utf8 byte string.
+    utfbytes: utf byte string to check.
+    return value: if valid utf8 string, return true. Otherwise, return false.
+    """
+    return _validate_utf8(utfbytes)
+
+
+def extract_err_message(exception):
+    if exception.args:
+        return exception.args[0]
+    else:
+        return None

Contents/Libraries/Shared/wraptor/__init__.py

@@ -0,0 +1,7 @@
+""" Wraptor
+
+Provides a set of useful decorators and other wrap-like python utility functions
+
+"""
+
+__version__ = "0.6.0"

Contents/Libraries/Shared/wraptor/context/__init__.py

@@ -0,0 +1,5 @@
+from wraptor.context.maybe import maybe
+from wraptor.context.throttle import throttle
+from wraptor.context.timer import timer
+
+__all__ = ['maybe', 'throttle', 'timer']

Contents/Libraries/Shared/wraptor/context/maybe.py

@@ -0,0 +1,27 @@
+import sys
+import inspect
+
+class _SkippedBlock(Exception):
+    pass
+
+class maybe(object):
+    def __init__(self, predicate):
+        self.predicate = predicate
+
+    def __empty_fn(self, *args, **kwargs):
+        return None
+
+    def __enter__(self):
+        if not self.predicate():
+            sys.settrace(self.__empty_fn)
+            frame = inspect.currentframe(1)
+            frame.f_trace = self.__trace
+
+    def __trace(self, *args, **kwargs):
+        raise _SkippedBlock()
+
+    def __exit__(self, type, value, traceback):
+        if isinstance(value, _SkippedBlock):
+            sys.settrace(None)
+            return True
+        return False

Contents/Libraries/Shared/wraptor/context/test/test_maybe.py

@@ -0,0 +1,26 @@
+from threading import Thread
+
+from wraptor.context import maybe
+
+def test_basic():
+    with maybe(lambda: False):
+        assert False
+
+    check = False
+    with maybe(lambda: True):
+        check = True
+    assert check
+
+def test_threads():
+    def worker(arr, index):
+        for i in range(5):
+            with maybe(lambda: i == 3):
+                arr[index] = True
+
+    workers = 100
+    arr = [False for i in range(workers)]
+    threads = [Thread(target=worker, args=(arr, i)) for i in range(workers)]
+    [t.start() for t in threads]
+    [t.join() for t in threads]
+
+    assert all(arr)

Contents/Libraries/Shared/wraptor/context/test/test_throttle.py

@@ -0,0 +1,17 @@
+import time
+
+from wraptor.context import throttle
+
+def test_basic():
+    arr = []
+    t = throttle(.1)
+
+    with t:
+        arr.append(1)
+    with t:
+        arr.append(1)
+    time.sleep(.2)
+    with t:
+        arr.append(1)
+
+    assert arr == [1, 1]

Contents/Libraries/Shared/wraptor/context/test/test_timer.py

@@ -0,0 +1,15 @@
+from wraptor.context import timer
+import time
+
+def test_basic():
+    with timer() as t:
+        time.sleep(0.1000000)     # sleep 100 ms
+
+    assert t.interval >= 100
+
+def test_params():
+    with timer('test') as t:
+        pass
+
+    assert t.name == 'test'
+    assert str(t).startswith('test')

Contents/Libraries/Shared/wraptor/context/throttle.py

@@ -0,0 +1,16 @@
+import time
+from wraptor.context import maybe
+
+class throttle(maybe):
+    def __init__(self, seconds=1):
+        self.seconds = seconds
+        self.last_run = 0
+
+        def predicate():
+            now = time.time()
+            if now > self.last_run + self.seconds:
+                self.last_run = now
+                return True
+            return False
+
+        maybe.__init__(self, predicate)

Contents/Libraries/Shared/wraptor/context/timer.py

@@ -0,0 +1,18 @@
+import time
+
+class timer(object):
+    __slots__ = ('name', 'interval', 'start', 'end')
+
+    def __init__(self, name=None):
+        self.name = name
+
+    def __enter__(self):
+        self.start = time.time() * 1e3
+        return self
+
+    def __exit__(self, *args):
+        self.end = time.time() * 1e3
+        self.interval = self.end - self.start
+
+    def __str__(self):
+        return "%s took %.03f ms" % (self.name, self.interval)

Contents/Libraries/Shared/wraptor/decorators/__init__.py

@@ -0,0 +1,6 @@
+from wraptor.decorators.memoize import memoize
+from wraptor.decorators.throttle import throttle
+from wraptor.decorators.timeout import timeout, TimeoutException
+from wraptor.decorators.exception_catcher import exception_catcher
+
+__all__ = ['memoize', 'throttle', 'timeout', 'TimeoutException', 'exception_catcher']

Contents/Libraries/Shared/wraptor/decorators/exception_catcher.py

@@ -0,0 +1,29 @@
+from functools import wraps
+import sys
+import Queue
+
+def exception_catcher(fn):
+    """ Catch exceptions raised by the decorated function.
+        Call check() to raise any caught exceptions.
+    """
+    exceptions = Queue.Queue()
+
+    @wraps(fn)
+    def wrapped(*args, **kwargs):
+        try:
+            ret = fn(*args, **kwargs)
+        except Exception:
+            exceptions.put(sys.exc_info())
+            raise
+        return ret
+
+    def check():
+        try:
+            item = exceptions.get(block=False)
+            klass, value, tb = item
+            raise klass, value, tb
+        except Queue.Empty:
+            pass
+
+    setattr(wrapped, 'check', check)
+    return wrapped

Contents/Libraries/Shared/wraptor/decorators/memoize.py

@@ -0,0 +1,70 @@
+from functools import wraps
+import time
+from hashlib import md5
+import threading
+
+class memoize(object):
+    """ Memoize the results of a function.  Supports an optional timeout
+        for automatic cache expiration.
+
+        If the optional manual_flush argument is True, a function called
+        "flush_cache" will be added to the wrapped function.  When
+        called, it will remove all the timed out values from the cache.
+
+        If you use this decorator as a class method, you must specify
+        instance_method=True otherwise you will have a single shared
+        cache for every instance of your class.
+
+        This decorator is thread safe.
+    """
+    def __init__(self, timeout=None, manual_flush=False, instance_method=False):
+        self.timeout = timeout
+        self.manual_flush = manual_flush
+        self.instance_method = instance_method
+        self.cache = {}
+        self.cache_lock = threading.RLock()
+
+    def __call__(self, fn):
+        if self.instance_method:
+            @wraps(fn)
+            def rewrite_instance_method(instance, *args, **kwargs):
+                # the first time we are called we overwrite the method
+                # on the class instance with a new memoize instance
+                if hasattr(instance, fn.__name__):
+                    bound_fn = fn.__get__(instance, instance.__class__)
+                    new_memoizer = memoize(self.timeout, self.manual_flush)(bound_fn)
+                    setattr(instance, fn.__name__, new_memoizer)
+                    return getattr(instance, fn.__name__)(*args, **kwargs)
+
+            return rewrite_instance_method
+
+        def flush_cache():
+            with self.cache_lock:
+                for key in self.cache.keys():
+                    if (time.time() - self.cache[key][1]) > self.timeout:
+                        del(self.cache[key])
+
+        @wraps(fn)
+        def wrapped(*args, **kwargs):
+            kw = kwargs.items()
+            kw.sort()
+            key_str = repr((args, kw))
+            key = md5(key_str).hexdigest()
+
+            with self.cache_lock:
+                try:
+                    result, cache_time = self.cache[key]
+                    if self.timeout is not None and (time.time() - cache_time) > self.timeout:
+                        raise KeyError
+                except KeyError:
+                    result, _ = self.cache[key] = (fn(*args, **kwargs), time.time())
+
+            if not self.manual_flush and self.timeout is not None:
+                flush_cache()
+
+            return result
+
+        if self.manual_flush:
+            wrapped.flush_cache = flush_cache
+
+        return wrapped

Contents/Libraries/Shared/wraptor/decorators/test/test_called_with_args.py

@@ -0,0 +1,18 @@
+from wraptor.decorators import timeout, throttle, memoize
+import pytest
+
+with_decorators = pytest.mark.parametrize("decorator", [
+    timeout, throttle, memoize
+])
+
+@with_decorators
+def test_called_with_args(decorator):
+    test_args = [1, 2, [1, 2, 3], { 'asdf': 5 }]
+    test_kwargs = { 'a': 1, 'b': [1, 2, 3] }
+
+    @decorator()
+    def fn(*args, **kwargs):
+        assert tuple(test_args) == args
+        assert test_kwargs == kwargs
+
+    fn(*test_args, **test_kwargs)

Contents/Libraries/Shared/wraptor/decorators/test/test_exception_catcher.py

@@ -0,0 +1,16 @@
+from wraptor.decorators import exception_catcher
+import threading
+import pytest
+
+def test_basic():
+
+    @exception_catcher
+    def work():
+        raise Exception()
+
+    t = threading.Thread(target=work)
+    t.start()
+    t.join()
+
+    with pytest.raises(Exception):
+        work.check()