release 1.4.27.965

Update Readme to fix broken link

.gitignore

@@ -13,7 +13,6 @@ build/
 develop-eggs/
 dist/
 eggs/
-lib/
 lib64/
 parts/
 sdist/
@@ -53,3 +52,5 @@ coverage.xml
 # Sphinx documentation
 docs/_build/
 
+# pycharm
+.idea

CHANGELOG.md

@@ -0,0 +1,398 @@
+1.4.23.920
+- core: handle undecodable paths better #255
+- core: don't fail on unrecoverable data #257
+- core: increase default scores from 110 (series) and 23 (movies) to 116 and 33
+- core: fix global subtitle folder handling #234
+- core: better invoking of configured executable after subtitle addition #247
+
+
+1.4.22.908
+- core: hotfix for more robust migrations
+
+
+1.4.22.898
+- core: migrate history and subtitle storage to a better implementation, making it far more stable. subtitle storage now also stores the downloaded subtitle data for future usage, so it will be possible to switch between them
+- core/menu: manual subtitle download and the FindBetterSubtitles-task now also work with metadata storage (hi @ shield users)
+- core: optimize FindBetterSubtitles-task
+
+
+1.4.19.882
+- core: fix tasks for new users
+- core: double check pin correctness/existance when pin is enabled
+
+
+1.4.19.878
+- core/menu: fix a task's last runtime display
+- core: task optimizations
+- core: fix leftover subtitles cleanup handling in case of a custom subtitle folder #234
+- core: run the scheduler even if permissions for libraries are wrong ("fixes" #236)
+- core: store subtitle history data in a different data format; reduce used storage size drastically (#233)
+
+
+1.4.19.866
+- core: fix wrong usage of LogKit
+
+
+1.4.19.857
+
+- core: add option to enable/disable channel and/or agent modes (fixes #220)
+- core: skip inexistent internal streams when scanning for internal subtitles (fixes #222)
+- core: fix filename encoding (fixes #223)
+- core: storage optimizations
+- menu: add pin-based channel menu locking (the whole channel or only the advanced menu)
+
+
+1.4.17.836
+- core: support for any media file that PMS supports (internal subtitles on mp4 for example)
+- core: fix broken ignore folders containing "subzero.ignore/.subzero.ignore/.nosz"
+- core: fix duplicate subtitles (lowercase/default case)
+- core: fix broken tasks queue due to oversight
+
+
+1.4.16.822
+- menu: add per-section recently added menu
+- menu: fix accidentally double-triggering a just triggered force-refresh
+- core: reorder settings in a more logical, grouped way
+- core: add simple automatic filesystem/external leftover subtitle cleaning (#133, #152)
+- core: fix force-refresh for big seasons/series
+- core: add setting to look for forced/foreign-only subtitles only (only works for opensubtitles and podnapisi)
+- core: fix custom subtitle folder was being ignored (#211)
+- core: only trust PMS for its movie name, not the series title (fixes #210)
+- core: full support (in filesystem/external mode) for forced/default/normal subtitle tags
+- core: ignore "non-standard" external subtitle files when scanning by default (everything but .srt, .ass, .ssa, fixes #192)
+- core: lower default max_recent_items_per_library to 500
+- core: skip forced/foreign-only subtitles if not specifically wanted
+- core: modify the task queue, hopefully helping #206
+- core: update anonymous usage collection
+
+
+1.4.11.781
+- core: cleanup, logging
+- core/menu: fix addic7ed display in manual subtitle list
+- core: use HTTP for OpenSubtitles instead of HTTPS because of current certificate errors
+- core: find better subtitles should now run smoothly even with replaced files (newer parts)
+
+
+1.4.10.769
+- core: hotfix for legacy intent storage regression
+
+1.4.10.768
+- core: automatically find better subtitles (configurable)
+- menu: display how the subtitle was downloaded (auto, manual, auto-better), in history menu
+- menu/core: correctly handle subtitle list for multiple languages
+- core: lower minimum series score to list subtitles for to 66
+- core: better matching of garbage filenames; we trust Plex now for the series name/movie title fully
+- core: add setting to specifically set the file permissions (chmod)
+
+
+1.4.5.742
+- core: fix force-refresh in certain situations
+- menu: add history
+- menu: add manual subtitle selection
+- menu: run Items with missing subtitles in separate thread for big libraries
+- settings: add history list size option (default: 100)
+- settings: add new default scores (TV: 110); use input instead of dropdown
+- settings: increase default missing subtitles amount per library to 2000
+- core: generic rewrites and optimizations
+- core: better hash verification
+- core: add anonymous usage data (opt-out in settings)
+- core: fix pt-BR display (IETF) again
+- wiki: update (thanks @dane22!) - quick URL: http://v.ht/szwiki
+- wiki: add score explanation - quick URL: http://v.ht/szscores
+- core: add persian/farsi encoding support
+
+
+1.3.49.636
+- core/menu: fix force refreshing (again)
+- core/menu: fix redundant route calls
+
+
+1.3.49.630 (backported some changes of the develop-1.4 branch to 1.3)
+- core/menu: make addic7ed boost configurable; lower the default boost value massively (to 10)
+- core: fix force refreshing (hopefully)
+- core: add (thai) tis-620 subtitle encoding support
+- menu: lower letter based menu browsing from 200 to 80 items
+- core: support greek encodings (windows-1253, cp1253, cp737, iso8859_7, cp875, cp869, iso2022_jp_2, mac_greek); hopefully fixes badly saved greek subs
+- menu: add generic back-to-home button to the top of every container view
+- menu: warn the user when SZ isn't enabled for any sections/libraries
+- menu: always re-check permissions status and enabled sections when opening the main menu; no server restart necessary anymore
+
+
+1.3.46.606
+- core: hotfix for new users (who've never downloaded a subtitle with SZ before); fixes #169
+
+
+1.3.46.605
+
+- add wiki (thanks @ukdtom / @dane22)
+- core: remove necessity of Plex credentials; fixes #148
+- core: fix non-SRT subtitle support; fixes #138
+- core: generic source overhaul in preparation for release 1.4
+- core: better filesystem encoding detection; may fix #159
+- core: add encoding handling for windows-1250 and windows-1251 encoding (eastern europe); fixes #162
+- core: overhaul ignore handling; fixes #164
+- core: implement ignore by path setting; fixes #134
+- core: add setting for optional fallback to metadata storage, if filesystem storage failed; fixes #100
+- core: add setting for notifying an executable after a subtitle has been downloaded (see Wiki); fixes #65
+- core: only handle sections for which Sub-Zero is enabled (in PMS agent settings); fixes #167
+- menu: add series/season force-refresh
+- menu: show item thumbnail/art where applicable
+- menu: mitigate PlexWeb behaviour of calling our handlers twice; fixes #168
+
+
+1.3.33.522
+
+- core: fix library permission detection on windows; fixes #151
+- core: "Restrict to one language" now behaves like it should (one found subtitle of any language is treated as sufficient); fixes #149
+- core: add support for other subtitle formats such as ssa/ass/microdvd, convert to srt; fixes #138
+- core: hopefully more consistent force-refresh handling (intent); fixes #118
+
+
+1.3.31.513
+
+- core: add option to only download one language again (and skip the addition of .lang to the subtitle filename) (default: off); fixes #126 
+- core: add option to always encode saved subtitles to UTF-8 (default: on); fixes #128
+- core: add fallback encoding detection using bs4.UnicodeDammit; hopefully fixes #101
+- core: update libraries: chardet, beautifulsoup, six
+- menu/core: check Plex libraries for permission problems on plugin start and report them in the channel menu (option, default: on); fixes #143
+- menu: while a manual refresh takes place, add a refresh button to the top of the SZ menu for convenience
+- menu: move the "add/remove X to ignore list" menu item to the bottom of the list on item detail 
+
+
+1.3.27.491
+
+- menu/core: make Sub-Zero channel menu optional (setting: "Enable Sub-Zero channel (disabling doesn't affect the subtitle features)?")
+- OpenSubtitles: detect and match video/subtitle FPS (framerate) to reduce out of sync subtitle matches
+- core: internal fixes; add _markerlib library (rare)
+- core: don't score tvshow episode title matches, should improve episode subtitle matches quite a bit (and reduce out of sync subtitles)
+- OpenSubtitles: make tag/exact filename matches optional (setting: "I keep the exact (release-) filename of my media files")
+- menu: unicode video title errors fixed
+- TVSubtitles: correctly match certain show IDs (such as "Series Name (US)")
+- core: don't break subtitle evaluation on crashed guessing
+
+
+1.3.23.459
+
+- core: slight code cleanup and fixes
+- core: add physical (filesystem) ignore mode (create files named `subzero.ignore`, `.subzero.ignore`, `.nosz` to ignore specific files/seasons/series/libraries)
+- core: fix guessit hinting of tv series with rare folder layout (e.g. series_name/a/S01E01.mkv)
+- core: remove "format" necessity from (opensubtitles) hash-validation
+- OpenSubtitles: dramatically improve matching: add tag (exact filename) matching and treat it just like hash matches
+- core: ignore embedded forced subtitles (fixes #106)
+- docs: update
+- settings: clarify
+
+
+1.3.20.422  
+- tvsubtitles: show matching was partially broken
+- addic7ed: better show matching
+- core: correctly skip subtitles stored in filesystem if metadata storage was selected (Local Media Assets agent may still pick them up)  
+- core: fix local API access (switch from HTTPS to HTTP)
+- core: fix handling of library names and media paths with non-ascii chars in it  
+- core: fix bundle version to correctly display current bundle version
+- core: skip downloading multi-CD subtitle
+- settings: clarify
+
+
+1.3.20.403
+- core: handle & and - ("and" and dash) in names
+- core: fixed handling of internal metadata subtitles
+- re-upped the minimum tv score to 85 (may be even higher in the future)
+- opensubtitles: possibly significantly better movie matching (now also query for movie title, instead of only querying for video hash)
+
+
+1.3.20.396
+- core: fix logging handlers (when saving log_level settings loggers got duplicated)
+- core: better movie matching by only hinting the filename and the last subdirectory to guessit (instead of the full path)
+- core: don't fail on wrong detection/scanning of media file
+- lower minimum tv series score from 85 to 67 (removed title; composed of: series=44 + season=11 + episode=11 + hearing_impaired=1)
+
+
+1.3.19.379
+- core: new recent items implementation (used in "Items with missing subtitles"), now really picking up everything instead of using Plex's recently_added API endpoint
+- core: be more strict about title matching - a matched title doesn't automatically mean season and episode are correct, too
+- core: rewrote the hash matching algorithm to not blindly trust hash matches anymore, but instead episodes have to match the series name, season number, episode number and format (BluRay, HDTV...); movie have to at least match the title, format and codec for the hash to be considered
+- core: remove TheSubDB support for now, as it only supports hash-based matching
+- scheduler: more robust item-fail-handling (fixes #81)
+- config: "Scan: include embedded subtitles" now by default is off, as embedded subs have proven to be pretty unreliable
+- config: add configuration option for how many items per library are to be considered recent (default: 200)
+- config: make logging verbosity configurable, default: WARNING - log files should be considerably smaller now
+- config: make console logging optional, default: off - good for development/debugging
+- config: removed the ignore lists
+- menu: added "Browse all items", where you can browse all your libraries and manage your ignore list (add/remove sections/series/items)
+- menu: added "Display ignore list", where you can manage your ignored sections, series and items
+- menu: the submenu titles are now dynamically composed of a breadcrumb-style tree so you see where you are
+- menu: show the current and past state of the important menu actions such as (force)-refresh an item or refreshing the menu, on the Refresh-button's description
+- plugin now isn't in the dev mode by default and has logging to the console off (in certain configurations this resulted in huge syslogs)
+
+
+1.3.6.316
+- scheduler: missing subtitles task now able to handle huge libraries (thanks @chopeta, @comrade)
+- scheduler: detect item-stalling, add wait and retry logic to make missing subtitles task more robust
+- scheduler: report failed items to logs after task run completion
+- hint series name and episode title, or movie title to guessit to make detection way better (e.g. for Mr. Robot)
+
+1.3.6.304
+- scheduler: correct the recent-determination of the search for missing subtitles in recently_added task
+- scheduler: rewrote search for missing subtitles task; it now requests refreshes one by one and not in bulk anymore (hopefully fixes stalling)
+- handle rare cases of weird file system encodings (ANSI_X3.4-1968 for example)
+- fix simplejson warning on startup
+
+1.3.6.297
+- rename Sub-Zero to Sub-Zero.bundle (requirement for adding Sub-Zero to the Plex channel directory)
+- channel: add logging actions for the internal storage to the advanced menu
+- channel: handle item titles with foreign characters in them correctly
+- (hopefully) fix handling file names with foreign characters in them when scanning for local media
+- reformat the whole project, mostly honoring pep8
+- scheduler: fixed some serious bugs; broken tasks (stalled) and some errors many of you have seen should be gone now
+- scheduler: partly rewritten to be more robust, again
+- settings: move Plex.tv credentials to the top
+
+1.3.5.281
+- fix tasks broken for 1.2 -> 1.3.5 upgraders
+
+1.3.5.273 (same build as Beta Release 1.3.0.273) - changes from previous stable 1.2.11.180
+- add a channel menu, making this plugin a hybrid (Agent+Channel)
+- add a generic background task scheduler
+- add a task to search for subtitles for items with missing subtitles (manually triggered and automatic)
+- add artwork
+- add Plex.tv credentials/token-generation support (needed for Plex Home users for the API to work)
+- addic7ed: improve show name matching again
+- channel: able to browse current on-deck and recently-added items, and refresh or force-refresh (search for new subtitles) single items
+- add library/series/video blacklist for items which should be skipped in "Search for missing subtitles"-task
+- add donation links
+- change the license to The Unlicense (while keeping the original MIT license from subliminal.bundle intact)
+- store subtitle information in internal plugin storage (for later usage)
+- many internal code improvements
+- update documentation
+
+1.3.0.273
+- more robust update functionality
+- menu: add refresh button to menu (to see the task state updating)
+- scheduler: actually skip a task if it's already running
+- scheduler: better behaviour when a task is running and a single item is refreshed at the same time
+- menu: enforce ascii on item titles
+
+1.3.0.261
+- removed localization again
+
+1.3.0.259
+- forgot locale-data
+
+1.3.0.256
+- fix force-refresh single items to actually force-refresh
+- re-add babel library
+
+1.3.0.253
+- rewrote background tasks subsystem
+- keep track of the status of a task and its runtime
+- add task state in channel menu to "Search for missing subtitles"
+- add date/time localization to channel menu
+- hide plex token from logs, when requesting
+- fix addic7ed show id parsing for shows with year set
+- test PMS API connectivity and fail miserably if needed (channel disabled, scheduler disabled)
+- feature-freeze for 1.3.0 final
+
+1.3.0.245
+- add the option to buy me a beer
+- clarify menu items
+- more robust scheduler handling (should fix the issues of scheduler runs in the past)
+- internal cleanups
+- add date_added to stored subtitle info (all of the 1.3.0 testers: please delete your internal subtitle storage using the channel->advanced menu)
+
+1.3.0.232
+- integrate plex.tv authentication for plex home users (test phase)
+- menu cleanup
+- more info in the menu (scheduler last and next run for example)
+- hopefully fixed intent handling (should throw less errors now)
+- fix version display in agent names
+
+1.3.0.222
+- bugfix for search missing subtitles
+- schedduler: honor "never"
+
+1.3.0.216
+- add channel menu
+- add generic task scheduler
+- add functionality to search for missing subtitles (via recently added items)
+- add artwork
+- change license to The Unlicense
+- ...
+
+1.2.11.180
+- fix #49 (metadata storage didn't work)
+- add better detection for existing subtitles stored in metadata
+
+1.2.11.177
+- updated naming scheme to reflect rewrite.major.minor.build (this release is the same as 1.1.0.5)
+
+1.1.0.5
+- addic7ed: fixed error in show id search
+- addic7ed: even better show matching
+- adjusted default scores: TV: 85, movies: 23
+- add support for com.plexapp.agents.xbmcnfo/xbmcnfotv (proposed to the author [here](https://github.com/gboudreau/XBMCnfoMoviesImporter.bundle/pull/63) and [here](https://github.com/gboudreau/XBMCnfoTVImporter.bundle/pull/70))
+
+1.1.0.3
+- addic7ed/tvsubtitles: be way smarter about punctuation in series names (*A.G.E.N.T.S. ...*)
+- ditch LocalMediaExtended and incorporate the functionality in Sub-Zero (**RC-users: delete LocalMediaExtended.bundle and re-enable LocalMedia!**)
+- remove (unused) setting "Restrict to one language"
+- add "Treat IETF language tags as ISO 639-1 (e.g. pt-BR = pt)" setting (default: true)
+- change default external storage to "current folder" instead of "/subs"
+- adjust default scores
+
+RC-5.2
+- revert back to /plexinc-agents/LocalMedia.bundle/tree/dist instead of /plexinc-agents/LocalMedia.bundle/tree/master, as the current public PMS version is too old for that
+
+RC-5.1
+- make hearing_impaired option more configurable and clear (see #configuration-)
+
+RC-5
+- fix wrong video type matching by hinting video type to guessit
+- update to newest LocalMediaExtended.bundle (incorporated plex-inc's changes)
+- show page links for subtitles in log file instead of subtitle ID
+- add custom language setting in addition to the three hardcoded ones
+- if a subtitle doesn't match our hearing_impaired setting, ignore it
+- add an optional boost for addic7ed subtitles, if their series, season, episode, year, and format (e.g. WEB-DL) matches
+
+RC-4
+- rename project to Sub-Zero
+- incorporate LocalMediaExtended.bundle
+- making this a multi-bundle plugin
+- update default scores
+- add icon
+
+RC-3
+- addic7ed/tvsubtitles: punctuation fixes (correctly get show ids for series like "Mr. Poopster" now)
+- podnapisi: fix logging
+- opensubtitles: add login credentials (for VIPs)
+- add retry functionality to retry failed subtitle downloads, including configurable amount of retries until discarding of provider
+- move possibly not needed setting "Restrict to one language" to the bottom
+- more detailed logging
+- some cleanup
+
+RC-2
+- fix empty custom subtitle folder creation
+- fix detection of existing embedded subtitles (switch to https://github.com/tonswieb/enzyme)
+- better logging
+- set default TV score to 15; movie score to 30
+
+RC-1
+- fix subliminal's logging error on min_score not met (fixes #15)
+- separated tv and movies subtitle scores settings (fixes #16)
+- add option to save only one subtitle per video (skipping the ".lang." naming scheme plex supports) (fixes #3)
+
+beta5
+- fix storing subtitles besides the actual video file, not subfolder (fixes #14)
+- "custom folder" setting now always used if given (properly overrides "subtitle folder" setting)
+- also scan (custom) given subtitle folders for existing subtitles instead of redownloading them on every refresh (fixes #9, #2)
+
+beta4
+- ~~increased score of addic7ed subtitles a bit~~ (not existing currently)
+- **support for newest Subliminal ([1.0.1](27a6e51cd36ffb2910cd9a7add6d797a2c6469b7)) and guessit ([0.11.0](2814f57e8999dcc31575619f076c0c1a63ce78f2))**
+- **plugin now also [works with com.plexapp.agents.thetvdbdvdorder](924470d2c0db3a71529278bce4b7247eaf2f85b8)**
+- providers fixed for subliminal 1.0.1 ([at least addic7ed](131504e7eed8b3400c457fbe49beea3b115bc916))
+- providers [don't simply fail and get excluded on non-detected language](1a779020792e0201ad689eefbf5a126155e89c97)
+- support for addic7ed languages: [French (Canadian)](b11a051c233fd72033f0c3b5a8c1965260e7e19f)
+- support for additional languages: [pt-br (Portuguese (Brasil)), fa (Persian (Farsi))](131504e7eed8b3400c457fbe49beea3b115bc916)
+- support for [three (two optional) subtitle languages](e543c927cf49c264eaece36640c99d67a99c7da2)
+- optionally use [random user agent for addic7ed provider](83ace14faf75fbd75313f0ceda9b78161895fbcf) (should not be needed)

Contents/Code/__init__.py

@@ -1,172 +1,256 @@
-# hdbits.org
+# coding=utf-8
+import datetime
+import sys
+import traceback
+
+from subzero.sandbox import restore_builtins
+
+module = sys.modules['__main__']
+restore_builtins(module, {})
+
+globals = getattr(module, "__builtins__")["globals"]
+for key, value in getattr(module, "__builtins__").iteritems():
+    if key != "globals":
+        globals()[key] = value
 
-import string, os, urllib, zipfile, re, copy
-from babelfish import Language
-from datetime import timedelta
-import subliminal
-import subliminal_patch
 import logger
 
-OS_PLEX_USERAGENT = 'plexapp.com v9.0'
+sys.modules["logger"] = logger
+
+import subliminal
+import support
+
+import interface
+sys.modules["interface"] = interface
+
+from subzero.constants import OS_PLEX_USERAGENT, PERSONAL_MEDIA_IDENTIFIER
+from interface.menu import *
+from support.plex_media import media_to_videos, get_media_item_ids, scan_videos
+from support.subtitlehelpers import get_subtitles_from_metadata
+from support.storage import whack_missing_parts, save_subtitles
+from support.items import is_ignored
+from support.config import config
+from support.lib import get_intent
+from support.helpers import track_usage, get_title_for_video_metadata, get_identifier, cast_bool
+from support.history import get_history
+from support.data import dispatch_migrate
+from support.activities import activity
 
-DEPENDENCY_MODULE_NAMES = ['subliminal', 'subliminal_patch', 'enzyme', 'guessit', 'requests']
 
 def Start():
     HTTP.CacheTime = 0
     HTTP.Headers['User-agent'] = OS_PLEX_USERAGENT
-    Log.Debug("START CALLED")
-    logger.registerLoggingHander(DEPENDENCY_MODULE_NAMES)
+
     # configured cache to be in memory as per https://github.com/Diaoul/subliminal/issues/303
     subliminal.region.configure('dogpile.cache.memory')
 
-    
+    # clear expired intents
+    intent = get_intent()
+    intent.cleanup()
 
-def ValidatePrefs():
-    Log.Debug("Validate Prefs called.")
-    return 
+    # clear expired menu history items
+    now = datetime.datetime.now()
+    if "menu_history" in Dict:
+        for key, timeout in Dict["menu_history"].items():
+            if now > timeout:
+                del Dict["menu_history"][key]
 
-# Prepare a list of languages we want subs for
-def getLangList():
-    langList = {Language.fromietf(Prefs["langPref1"])}
-    if(Prefs["langPref2"] != "None"):
-        langList.update({Language.fromietf(Prefs["langPref2"])})
-    if(Prefs["langPref3"] != "None"):
-        langList.update({Language.fromietf(Prefs["langPref3"])})
-        
-    return langList
+    # run migrations
+    if "subs" in Dict or "history" in Dict:
+        Thread.Create(dispatch_migrate)
 
-def getSubtitleDestinationFolder():
-    if not Prefs["subtitles.save.filesystem"]:
-	return
+    # clear old task data
+    scheduler.clear_task_data()
 
-    fld_custom = Prefs["subtitles.save.subFolder.Custom"].strip() if bool(Prefs["subtitles.save.subFolder.Custom"]) else None
-    return fld_custom or (Prefs["subtitles.save.subFolder"] if Prefs["subtitles.save.subFolder"] != "current folder" else None)
+    # init defaults; perhaps not the best idea to use ValidatePrefs here, but we'll see
+    ValidatePrefs()
+    Log.Debug(config.full_version)
 
-def initSubliminalPatches():
-    # configure custom subtitle destination folders for scanning pre-existing subs
-    dest_folder = getSubtitleDestinationFolder()
-    subliminal_patch.patch_video.CUSTOM_PATHS = [dest_folder] if dest_folder else []
+    if not config.permissions_ok:
+        Log.Error("Insufficient permissions on library folders:")
+        for title, path in config.missing_permissions:
+            Log.Error("Insufficient permissions on library %s, folder: %s" % (title, path))
 
-def getProviders():
-    providers = {'opensubtitles' : Prefs['provider.opensubtitles.enabled'],
-                 'thesubdb' : Prefs['provider.thesubdb.enabled'],
-                 'podnapisi' : Prefs['provider.podnapisi.enabled'],
-                 'addic7ed' : Prefs['provider.addic7ed.enabled'],
-                 'tvsubtitles' : Prefs['provider.tvsubtitles.enabled']
-                 }
-    return filter(lambda prov: providers[prov], providers)
+    # run task scheduler
+    scheduler.run()
 
-def getProviderSettings():
-    provider_settings = {'addic7ed': {'username': Prefs['provider.addic7ed.username'], 
-                                      'password': Prefs['provider.addic7ed.password'],
-				      'use_random_agents': Prefs['provider.addic7ed.use_random_agents'],
-                                      },
-                         }
+    # bind activities
+    Thread.Create(activity.start)
 
-    return provider_settings
+    if "anon_id" not in Dict:
+        Dict["anon_id"] = get_identifier()
 
-def scanTvMedia(media):
-    videos = {}
-    for season in media.seasons:
-        for episode in media.seasons[season].episodes:
-            for item in media.seasons[season].episodes[episode].items:
-                for part in item.parts:
-                    scannedVideo = scanVideo(part)
-                    videos[scannedVideo] = part
-    return videos
+    # track usage
+    if cast_bool(Prefs["track_usage"]):
+        if "first_use" not in Dict:
+            Dict["first_use"] = datetime.datetime.utcnow()
+            Dict.Save()
+            track_usage("General", "plugin", "first_start", config.version)
+        track_usage("General", "plugin", "start", config.version)
 
-def scanMovieMedia(media):
-    videos = {}
-    for item in media.items:
-        for part in item.parts:
-            scannedVideo = scanVideo(part)
-            videos[scannedVideo] = part 
-    return videos
 
-def scanVideo(part):
-    embedded_subtitles = Prefs['subtitles.scan.embedded']
-    external_subtitles = Prefs['subtitles.scan.external']
-    
-    Log.Debug("Scanning video: %s, subtitles=%s, embedded_subtitles=%s" % (part.file, external_subtitles, embedded_subtitles))
-    try:
-        return subliminal.video.scan_video(part.file, subtitles=external_subtitles, embedded_subtitles=embedded_subtitles)
-    except ValueError:
-        Log.Warn("File could not be guessed by subliminal")
-
-def downloadBestSubtitles(videos):
-    min_score = int(Prefs['subtitles.search.minimumScore'])
+def download_best_subtitles(video_part_map, min_score=0):
     hearing_impaired = Prefs['subtitles.search.hearingImpaired']
-    Log.Debug("Download best subtitles using settings: min_score: %s, hearing_impaired: %s" %(min_score, hearing_impaired))
-    
-    return subliminal.api.download_best_subtitles(videos, getLangList(), min_score, hearing_impaired, providers=getProviders(), provider_configs=getProviderSettings())
+    languages = config.lang_list
+    if not languages:
+        return
 
-def saveSubtitles(videos, subtitles):
-    if Prefs['subtitles.save.filesystem']:
-        Log.Debug("Saving subtitles to filesystem")
-        saveSubtitlesToFile(subtitles)
-    else:
-        Log.Debug("Saving subtitles as metadata")
-        saveSubtitlesToMetadata(videos, subtitles)
+    missing_languages = False
+    for video, part in video_part_map.iteritems():
+        if not Prefs['subtitles.save.filesystem']:
+            # scan for existing metadata subtitles
+            meta_subs = get_subtitles_from_metadata(part)
+            for language, subList in meta_subs.iteritems():
+                if subList:
+                    video.subtitle_languages.add(language)
+                    Log.Debug("Found metadata subtitle %s for %s", language, video)
 
+        missing_subs = (languages - video.subtitle_languages)
 
-
-def saveSubtitlesToFile(subtitles):
-    fld_custom = Prefs["subtitles.save.subFolder.Custom"].strip() if bool(Prefs["subtitles.save.subFolder.Custom"]) else None
-    
-    for video, video_subtitles in subtitles.items():
-	fld = None
-	if fld_custom or Prefs["subtitles.save.subFolder"] != "current folder":
-    	    # specific subFolder requested, create it if it doesn't exist
-            fld_base = os.path.split(video.name)[0]
-            if fld_custom:
-                if fld_custom.startswith("/"):
-                    # absolute folder
-                    fld = fld_custom
-                else:
-                    fld = os.path.join(fld_base, fld_custom)
+        # all languages are found if we either really have subs for all languages or we only want to have exactly one language
+        # and we've only found one (the case for a selected language, Prefs['subtitles.only_one'] (one found sub matches any language))
+        found_one_which_is_enough = len(video.subtitle_languages) >= 1 and Prefs['subtitles.only_one']
+        if not missing_subs or found_one_which_is_enough:
+            if found_one_which_is_enough:
+                Log.Debug('Only one language was requested, and we\'ve got a subtitle for %s', video)
             else:
-                fld = os.path.join(fld_base, Prefs["subtitles.save.subFolder"])
-            if not os.path.exists(fld):
-                os.makedirs(fld)
-        subliminal.api.save_subtitles(video, video_subtitles, directory=fld)
+                Log.Debug('All languages %r exist for %s', languages, video)
+            continue
+        missing_languages = True
+        break
 
-def saveSubtitlesToMetadata(videos, subtitles):
-    for video, video_subtitles in subtitles.items():
-        mediaPart = videos[video]
-        for subtitle in video_subtitles: 
-            mediaPart.subtitles[Locale.Language.Match(subtitle.language.alpha2)][subtitle.page_link] = Proxy.Media(subtitle.content, ext="srt")
+    if missing_languages:
+        Log.Debug("Download best subtitles using settings: min_score: %s, hearing_impaired: %s" % (min_score, hearing_impaired))
 
-class SubliminalSubtitlesAgentMovies(Agent.Movies):
-    name = 'Subliminal Movie Subtitles'
+        return subliminal.api.download_best_subtitles(video_part_map.keys(), languages, min_score, hearing_impaired, providers=config.providers,
+                                                      provider_configs=config.provider_settings)
+    Log.Debug("All languages for all requested videos exist. Doing nothing.")
+
+
+def update_local_media(metadata, media, media_type="movies"):
+    # Look for subtitles
+    if media_type == "movies":
+        for item in media.items:
+            for part in item.parts:
+                support.localmedia.find_subtitles(part)
+        return
+
+    # Look for subtitles for each episode.
+    for s in media.seasons:
+        # If we've got a date based season, ignore it for now, otherwise it'll collide with S/E folders/XML and PMS
+        # prefers date-based (why?)
+        if int(s) < 1900 or metadata.guid.startswith(PERSONAL_MEDIA_IDENTIFIER):
+            for e in media.seasons[s].episodes:
+                for i in media.seasons[s].episodes[e].items:
+
+                    # Look for subtitles.
+                    for part in i.parts:
+                        support.localmedia.find_subtitles(part)
+        else:
+            pass
+
+
+class SubZeroAgent(object):
+    agent_type = None
+    agent_type_verbose = None
     languages = [Locale.Language.English]
     primary_provider = False
-    contributes_to = ['com.plexapp.agents.imdb']
+    score_prefs_key = None
+
+    def __init__(self, *args, **kwargs):
+        super(SubZeroAgent, self).__init__(*args, **kwargs)
+        self.agent_type = "movies" if isinstance(self, Agent.Movies) else "series"
+        self.name = "Sub-Zero Subtitles (%s, %s)" % (self.agent_type_verbose, config.get_version())
 
     def search(self, results, media, lang):
-        Log.Debug("MOVIE SEARCH CALLED")
+        Log.Debug("Sub-Zero %s, %s search" % (config.version, self.agent_type))
         results.Append(MetadataSearchResult(id='null', score=100))
 
     def update(self, metadata, media, lang):
-        Log.Debug("MOVIE UPDATE CALLED")
-	initSubliminalPatches()
-        videos = scanMovieMedia(media)
-        subtitles = downloadBestSubtitles(videos.keys())
-        saveSubtitles(videos, subtitles)
+        if not config.enable_agent:
+            Log.Debug("Skipping Sub-Zero agent(s)")
+            return
 
-class SubliminalSubtitlesAgentTvShows(Agent.TV_Shows):
-    
-    name = 'Subliminal TV Subtitles'
-    languages = [Locale.Language.English]
-    primary_provider = False
-    contributes_to = ['com.plexapp.agents.thetvdb', 'com.plexapp.agents.thetvdbdvdorder']
+        Log.Debug("Sub-Zero %s, %s update called" % (config.version, self.agent_type))
+        intent = get_intent()
 
-    def search(self, results, media, lang):
-        Log.Debug("TV SEARCH CALLED")
-        results.Append(MetadataSearchResult(id='null', score=100))
+        if not media:
+            Log.Error("Called with empty media, something is really wrong with your setup!")
+            return
 
-    def update(self, metadata, media, lang):
-        Log.Debug("TvUpdate. Lang %s" % lang)
-	initSubliminalPatches()
-        videos = scanTvMedia(media)
-        subtitles = downloadBestSubtitles(videos.keys())
-        saveSubtitles(videos, subtitles)
+        item_ids = []
+        try:
+            config.init_subliminal_patches()
+            videos = media_to_videos(media, kind=self.agent_type)
+
+            # media ignored?
+            use_any_parts = False
+            for video in videos:
+                if is_ignored(video["id"]):
+                    Log.Debug(u"Ignoring %s" % video)
+                    continue
+                use_any_parts = True
+
+            if not use_any_parts:
+                Log.Debug(u"Nothing to do.")
+                return
+
+            try:
+                use_score = int(Prefs[self.score_prefs_key].strip())
+            except ValueError:
+                Log.Error("Please only put numbers into the scores setting. Exiting")
+                return
+
+            set_refresh_menu_state(media, media_type=self.agent_type)
+
+            # find local media
+            update_local_media(metadata, media, media_type=self.agent_type)
+
+            # scanned_video_part_map = {subliminal.Video: plex_part, ...}
+            scanned_video_part_map = scan_videos(videos, kind=self.agent_type)
+
+            # downloaded_subtitles = {subliminal.Video: [subtitle, subtitle, ...]}
+            downloaded_subtitles = download_best_subtitles(scanned_video_part_map, min_score=use_score)
+            item_ids = get_media_item_ids(media, kind=self.agent_type)
+
+            whack_missing_parts(scanned_video_part_map)
+
+            if downloaded_subtitles:
+                save_subtitles(scanned_video_part_map, downloaded_subtitles)
+                track_usage("Subtitle", "refreshed", "download", 1)
+
+                for video, video_subtitles in downloaded_subtitles.items():
+                    # store item(s) in history
+                    for subtitle in video_subtitles:
+                        item_title = get_title_for_video_metadata(video.plexapi_metadata, add_section_title=False)
+                        history = get_history()
+                        history.add(item_title, video.id, section_title=video.plexapi_metadata["section"],
+                                    subtitle=subtitle)
+
+            update_local_media(metadata, media, media_type=self.agent_type)
+
+        finally:
+            # update the menu state
+            set_refresh_menu_state(None)
+
+            # notify any running tasks about our finished update
+            for item_id in item_ids:
+                scheduler.signal("updated_metadata", item_id)
+
+                # resolve existing intent for that id
+                intent.resolve("force", item_id)
+
+            Dict.Save()
+
+
+class SubZeroSubtitlesAgentMovies(SubZeroAgent, Agent.Movies):
+    contributes_to = ['com.plexapp.agents.imdb', 'com.plexapp.agents.xbmcnfo', 'com.plexapp.agents.themoviedb', 'com.plexapp.agents.hama']
+    score_prefs_key = "subtitles.search.minimumMovieScore1"
+    agent_type_verbose = "Movies"
+
+
+class SubZeroSubtitlesAgentTvShows(SubZeroAgent, Agent.TV_Shows):
+    contributes_to = ['com.plexapp.agents.thetvdb', 'com.plexapp.agents.themoviedb',
+                      'com.plexapp.agents.thetvdbdvdorder', 'com.plexapp.agents.xbmcnfotv', 'com.plexapp.agents.hama']
+    score_prefs_key = "subtitles.search.minimumTVScore1"
+    agent_type_verbose = "TV"

Contents/Code/interface/__init__.py

@@ -0,0 +1,7 @@
+import sys
+
+import menu
+sys.modules["interface.menu"] = menu
+
+import menu_helpers
+sys.modules["interface.menu_helpers"] = menu_helpers

Contents/Code/interface/menu.py

@@ -0,0 +1,887 @@
+# coding=utf-8
+import logging
+
+import datetime
+
+import logger
+import os
+
+from menu_helpers import add_ignore_options, dig_tree, set_refresh_menu_state, \
+    should_display_ignore, enable_channel_wrapper, default_thumb, debounce, ObjectContainer, SubFolderObjectContainer
+from subzero.constants import TITLE, ART, ICON, PREFIX, PLUGIN_IDENTIFIER, DEPENDENCY_MODULE_NAMES
+from subzero.history_storage import mode_map
+from support.background import scheduler
+from support.config import config
+from support.helpers import pad_title, timestamp, get_language, df, cast_bool
+from support.ignore import ignore_list
+from support.items import get_item, get_on_deck_items, refresh_item, get_all_items, get_items_info, \
+    get_item_thumb, get_item_kind_from_rating_key
+from support.lib import Plex
+from support.missing_subtitles import items_get_all_missing_subs
+from support.plex_media import get_plex_metadata, scan_videos
+from support.storage import reset_storage, log_storage, get_subtitle_storage
+
+# init GUI
+ObjectContainer.art = R(ART)
+ObjectContainer.no_cache = True
+
+# default thumb for DirectoryObjects
+DirectoryObject.thumb = default_thumb
+
+
+# noinspection PyUnboundLocalVariable
+route = enable_channel_wrapper(route)
+# noinspection PyUnboundLocalVariable
+handler = enable_channel_wrapper(handler)
+
+
+@handler(PREFIX, TITLE, art=ART, thumb=ICON)
+@route(PREFIX)
+def fatality(randomize=None, force_title=None, header=None, message=None, only_refresh=False, no_history=False, replace_parent=False):
+    """
+    subzero main menu
+    """
+    title = config.full_version#force_title if force_title is not None else config.full_version
+    oc = ObjectContainer(title1=title, title2=title, header=unicode(header) if header else title, message=message, no_history=no_history,
+                         replace_parent=replace_parent, no_cache=True)
+
+    # always re-check permissions
+    config.refresh_permissions_status()
+
+    # always re-check enabled sections
+    config.refresh_enabled_sections()
+
+    if config.lock_menu and not config.pin_correct:
+        oc.add(DirectoryObject(
+            key=Callback(PinMenu, randomize=timestamp()),
+            title=pad_title("Enter PIN"),
+            summary="The owner has restricted the access to this menu. Please enter the correct pin",
+        ))
+        return oc
+
+    if not config.permissions_ok and config.missing_permissions:
+        for title, path in config.missing_permissions:
+            oc.add(DirectoryObject(
+                key=Callback(fatality, randomize=timestamp()),
+                title=pad_title("Insufficient permissions"),
+                summary="Insufficient permissions on library %s, folder: %s" % (title, path),
+            ))
+        return oc
+
+    if not config.enabled_sections:
+        oc.add(DirectoryObject(
+            key=Callback(fatality, randomize=timestamp()),
+            title=pad_title("I'm not enabled!"),
+            summary="Please enable me for some of your libraries in your server settings; currently I do nothing",
+        ))
+        return oc
+
+    if not only_refresh:
+        if Dict["current_refresh_state"]:
+            oc.add(DirectoryObject(
+                key=Callback(fatality, force_title=" ", randomize=timestamp()),
+                title=pad_title("Working ... refresh here"),
+                summary="Current state: %s; Last state: %s" % (
+                    (Dict["current_refresh_state"] or "Idle") if "current_refresh_state" in Dict else "Idle",
+                    (Dict["last_refresh_state"] or "None") if "last_refresh_state" in Dict else "None"
+                )
+            ))
+
+        oc.add(DirectoryObject(
+            key=Callback(OnDeckMenu),
+            title="On Deck items",
+            summary="Shows the current on deck items and allows you to individually (force-) refresh their metadata/"
+                    "subtitles."
+        ))
+        oc.add(DirectoryObject(
+            key=Callback(RecentlyAddedMenu),
+            title="Recently Added items",
+            summary="Shows the recently added items per section."
+        ))
+        oc.add(DirectoryObject(
+            key=Callback(RecentMissingSubtitlesMenu, randomize=timestamp()),
+            title="Items with missing subtitles",
+            summary="Shows the items honoring the configured 'Item age to be considered recent'-setting (%s)"
+                    " and allowing you to individually (force-) refresh their metadata/subtitles. " %
+                    Prefs["scheduler.item_is_recent_age"]
+        ))
+        oc.add(DirectoryObject(
+            key=Callback(SectionsMenu),
+            title="Browse all items",
+            summary="Go through your whole library and manage your ignore list. You can also "
+                    "(force-) refresh the metadata/subtitles of individual items."
+        ))
+
+        task_name = "SearchAllRecentlyAddedMissing"
+        task = scheduler.task(task_name)
+
+        if task.ready_for_display:
+            task_state = "Running: %s/%s (%s%%)" % (len(task.items_done), len(task.items_searching), task.percentage)
+        else:
+            task_state = "Last scheduler run: %s; Next scheduled run: %s; Last runtime: %s" % (df(scheduler.last_run(task_name)) or "never",
+                                                                                               df(scheduler.next_run(task_name)) or "never",
+                                                                                               str(task.last_run_time).split(".")[0])
+
+        oc.add(DirectoryObject(
+            key=Callback(RefreshMissing, randomize=timestamp()),
+            title="Search for missing subtitles (in recently-added items, max-age: %s)" % Prefs["scheduler.item_is_recent_age"],
+            summary="Automatically run periodically by the scheduler, if configured. %s" % task_state
+        ))
+
+        oc.add(DirectoryObject(
+            key=Callback(IgnoreListMenu),
+            title="Display ignore list (%d)" % len(ignore_list),
+            summary="Show the current ignore list (mainly used for the automatic tasks)"
+        ))
+
+        oc.add(DirectoryObject(
+            key=Callback(HistoryMenu),
+            title="History",
+            summary="Show the last %i downloaded subtitles" % int(Prefs["history_size"])
+        ))
+
+    oc.add(DirectoryObject(
+        key=Callback(fatality, force_title=" ", randomize=timestamp()),
+        title=pad_title("Refresh"),
+        summary="Current state: %s; Last state: %s" % (
+            (Dict["current_refresh_state"] or "Idle") if "current_refresh_state" in Dict else "Idle",
+            (Dict["last_refresh_state"] or "None") if "last_refresh_state" in Dict else "None"
+        )
+    ))
+
+    # add re-lock after pin unlock
+    if config.pin:
+        oc.add(DirectoryObject(
+            key=Callback(ClearPin, randomize=timestamp()),
+            title=pad_title("Re-lock menu(s)"),
+            summary="Enabled the PIN again for menu(s)"
+        ))
+
+    if not only_refresh:
+        oc.add(DirectoryObject(
+            key=Callback(AdvancedMenu),
+            title=pad_title("Advanced functions"),
+            summary="Use at your own risk"
+        ))
+
+    return oc
+
+
+@route(PREFIX + '/pin')
+def PinMenu(pin="", randomize=None, success_go_to="channel"):
+    oc = ObjectContainer(title2="Enter PIN number %s" % (len(pin) + 1), no_cache=True, no_history=True,
+                         skip_pin_lock=True)
+
+    if pin == config.pin:
+        Dict["pin_correct_time"] = datetime.datetime.now()
+        config.locked = False
+        if success_go_to == "channel":
+            return fatality(force_title="PIN correct", header="PIN correct", no_history=True)
+        elif success_go_to == "advanced":
+            return AdvancedMenu(randomize=timestamp())
+
+    for i in range(10):
+        oc.add(DirectoryObject(
+            key=Callback(PinMenu, randomize=timestamp(), pin=pin + str(i),success_go_to=success_go_to),
+            title=pad_title(str(i)),
+        ))
+    oc.add(DirectoryObject(
+        key=Callback(PinMenu, randomize=timestamp(),success_go_to=success_go_to),
+        title=pad_title("Reset"),
+    ))
+    return oc
+
+
+@route(PREFIX + '/pin_lock')
+def ClearPin(randomize=None):
+    Dict["pin_correct_time"] = None
+    config.locked = True
+    return fatality(force_title="Menu locked", header=" ", no_history=True)
+
+
+@route(PREFIX + '/on_deck')
+def OnDeckMenu(message=None):
+    """
+    displays the items on deck
+    :param message:
+    :return:
+    """
+    return mergedItemsMenu(title="Items On Deck", base_title="Items On Deck", itemGetter=get_on_deck_items)
+
+
+@route(PREFIX + '/recently_added')
+def RecentlyAddedMenu(message=None):
+    """
+    displays the items recently added per section
+    :param message:
+    :return:
+    """
+    return SectionsMenu(base_title="Recently added", section_items_key="recently_added", ignore_options=False)
+
+
+@route(PREFIX + '/recent', force=bool)
+@debounce
+def RecentMissingSubtitlesMenu(force=False, randomize=None):
+    title="Items with missing subtitles"
+    oc = SubFolderObjectContainer(title2=title, no_cache=True, no_history=True)
+
+    running = scheduler.is_task_running("MissingSubtitles")
+    task_data = scheduler.get_task_data("MissingSubtitles")
+    missing_items = task_data["missing_subtitles"] if task_data else None
+
+    if ((missing_items is None) or force) and not running:
+        scheduler.dispatch_task("MissingSubtitles")
+        running = True
+
+    if not running:
+        oc.add(DirectoryObject(
+            key=Callback(RecentMissingSubtitlesMenu, force=True, randomize=timestamp()),
+            title=u"Get items with missing subtitles",
+            thumb=default_thumb
+        ))
+    else:
+        oc.add(DirectoryObject(
+            key=Callback(RecentMissingSubtitlesMenu, force=False, randomize=timestamp()),
+            title=u"Updating, refresh here ...",
+            thumb=default_thumb
+        ))
+
+    if missing_items is not None:
+        for added_at, item_id, item_title, item, missing_languages in missing_items:
+            oc.add(DirectoryObject(
+                key=Callback(ItemDetailsMenu, title=title + " > " + item_title, item_title=item_title, rating_key=item_id),
+                title=item_title,
+                summary="Missing: %s" % ", ".join(l.name for l in missing_languages),
+                thumb=get_item_thumb(item) or default_thumb
+            ))
+
+        scheduler.clear_task_data("MissingSubtitles")
+
+    return oc
+
+
+def mergedItemsMenu(title, itemGetter, itemGetterKwArgs=None, base_title=None, *args, **kwargs):
+    """
+    displays an item list of dynamic kinds of items
+    :param title:
+    :param itemGetter:
+    :param itemGetterKwArgs:
+    :param base_title:
+    :param args:
+    :param kwargs:
+    :return:
+    """
+    oc = SubFolderObjectContainer(title2=title, no_cache=True, no_history=True)
+    items = itemGetter(*args, **kwargs)
+
+    for kind, title, item_id, deeper, item in items:
+        oc.add(DirectoryObject(
+            title=title,
+            key=Callback(ItemDetailsMenu, title=base_title + " > " + title, item_title=title, rating_key=item_id),
+            thumb=get_item_thumb(item) or default_thumb
+        ))
+
+    return oc
+
+
+def determine_section_display(kind, item, pass_kwargs=None):
+    """
+    returns the menu function for a section based on the size of it (amount of items)
+    :param kind:
+    :param item:
+    :return:
+    """
+    if pass_kwargs and pass_kwargs.get("section_items_key", "all") != "all":
+        return SectionMenu
+    if item.size > 80:
+        return SectionFirstLetterMenu
+    return SectionMenu
+
+
+@route(PREFIX + '/ignore/set/{kind}/{rating_key}/{todo}/sure={sure}', kind=str, rating_key=str, todo=str, sure=bool)
+def IgnoreMenu(kind, rating_key, title=None, sure=False, todo="not_set"):
+    """
+    displays the ignore options for a menu
+    :param kind:
+    :param rating_key:
+    :param title:
+    :param sure:
+    :param todo:
+    :return:
+    """
+    is_ignored = rating_key in ignore_list[kind]
+    if not sure:
+        oc = SubFolderObjectContainer(no_history=True, replace_parent=True, title1="%s %s %s %s the ignore list" % (
+            "Add" if not is_ignored else "Remove", ignore_list.verbose(kind), title, "to" if not is_ignored else "from"), title2="Are you sure?")
+        oc.add(DirectoryObject(
+            key=Callback(IgnoreMenu, kind=kind, rating_key=rating_key, title=title, sure=True, todo="add" if not is_ignored else "remove"),
+            title=pad_title("Are you sure?"),
+        ))
+        return oc
+
+    rel = ignore_list[kind]
+    dont_change = False
+    if todo == "remove":
+        if not is_ignored:
+            dont_change = True
+        else:
+            rel.remove(rating_key)
+            Log.Info("Removed %s (%s) from the ignore list", title, rating_key)
+            ignore_list.remove_title(kind, rating_key)
+            ignore_list.save()
+            state = "removed from"
+    elif todo == "add":
+        if is_ignored:
+            dont_change = True
+        else:
+            rel.append(rating_key)
+            Log.Info("Added %s (%s) to the ignore list", title, rating_key)
+            ignore_list.add_title(kind, rating_key, title)
+            ignore_list.save()
+            state = "added to"
+    else:
+        dont_change = True
+
+    if dont_change:
+        return fatality(force_title=" ", header="Didn't change the ignore list", no_history=True)
+
+    return fatality(force_title=" ", header="%s %s the ignore list" % (title, state), no_history=True)
+
+
+@route(PREFIX + '/sections')
+def SectionsMenu(base_title="Sections", section_items_key="all", ignore_options=True):
+    """
+    displays the menu for all sections
+    :return:
+    """
+    items = get_all_items("sections")
+
+    return dig_tree(SubFolderObjectContainer(title2="Sections", no_cache=True, no_history=True), items, None,
+                    menu_determination_callback=determine_section_display, pass_kwargs={"base_title": base_title,
+                                                                                        "section_items_key": section_items_key,
+                                                                                        "ignore_options": ignore_options},
+                    fill_args={"title": "section_title"})
+
+
+@route(PREFIX + '/section', ignore_options=bool)
+def SectionMenu(rating_key, title=None, base_title=None, section_title=None, ignore_options=True,
+                section_items_key="all"):
+    """
+    displays the contents of a section
+    :param section_items_key:
+    :param rating_key:
+    :param title:
+    :param base_title:
+    :param section_title:
+    :param ignore_options:
+    :return:
+    """
+    items = get_all_items(key=section_items_key, value=rating_key, base="library/sections")
+
+    kind, deeper = get_items_info(items)
+    title = unicode(title)
+
+    section_title = title
+    title = base_title + " > " + title
+    oc = SubFolderObjectContainer(title2=title, no_cache=True, no_history=True)
+    if ignore_options:
+        add_ignore_options(oc, "sections", title=section_title, rating_key=rating_key, callback_menu=IgnoreMenu)
+
+    return dig_tree(oc, items, MetadataMenu,
+                    pass_kwargs={"base_title": title, "display_items": deeper, "previous_item_type": "section",
+                                 "previous_rating_key": rating_key})
+
+
+@route(PREFIX + '/section/firstLetter', deeper=bool)
+def SectionFirstLetterMenu(rating_key, title=None, base_title=None, section_title=None, ignore_options=True,
+                           section_items_key="all"):
+    """
+    displays the contents of a section indexed by its first char (A-Z, 0-9...)
+    :param ignore_options: ignored
+    :param section_items_key: ignored
+    :param rating_key:
+    :param title:
+    :param base_title:
+    :param section_title:
+    :return:
+    """
+    items = get_all_items(key="first_character", value=rating_key, base="library/sections")
+
+    kind, deeper = get_items_info(items)
+
+    title = unicode(title)
+    oc = SubFolderObjectContainer(title2=section_title, no_cache=True, no_history=True)
+    title = base_title + " > " + title
+    add_ignore_options(oc, "sections", title=section_title, rating_key=rating_key, callback_menu=IgnoreMenu)
+
+    oc.add(DirectoryObject(
+        key=Callback(SectionMenu, title="All", base_title=title, rating_key=rating_key, ignore_options=False),
+        title="All"
+    )
+    )
+    return dig_tree(oc, items, FirstLetterMetadataMenu, force_rating_key=rating_key, fill_args={"key": "key"},
+                    pass_kwargs={"base_title": title, "display_items": deeper, "previous_rating_key": rating_key})
+
+
+@route(PREFIX + '/section/firstLetter/key', deeper=bool)
+def FirstLetterMetadataMenu(rating_key, key, title=None, base_title=None, display_items=False, previous_item_type=None,
+                            previous_rating_key=None):
+    """
+    displays the contents of a section filtered by the first letter
+    :param rating_key: actually is the section's key
+    :param key: the firstLetter wanted
+    :param title: the first letter, or #
+    :param deeper:
+    :return:
+    """
+    title = base_title + " > " + unicode(title)
+    oc = SubFolderObjectContainer(title2=title, no_cache=True, no_history=True)
+
+    items = get_all_items(key="first_character", value=[rating_key, key], base="library/sections", flat=False)
+    kind, deeper = get_items_info(items)
+    dig_tree(oc, items, MetadataMenu,
+             pass_kwargs={"base_title": title, "display_items": deeper, "previous_item_type": kind, "previous_rating_key": rating_key})
+    return oc
+
+
+@route(PREFIX + '/section/contents', display_items=bool)
+def MetadataMenu(rating_key, title=None, base_title=None, display_items=False, previous_item_type=None,
+                 previous_rating_key=None):
+    """
+    displays the contents of a section based on whether it has a deeper tree or not (movies->movie (item) list; series->series list)
+    :param rating_key:
+    :param title:
+    :param base_title:
+    :param display_items:
+    :param previous_item_type:
+    :param previous_rating_key:
+    :return:
+    """
+    title = unicode(title)
+    item_title = title
+    title = base_title + " > " + title
+    oc = SubFolderObjectContainer(title2=title, no_cache=True, no_history=True)
+
+    current_kind = get_item_kind_from_rating_key(rating_key)
+
+    if display_items:
+        items = get_all_items(key="children", value=rating_key, base="library/metadata")
+        kind, deeper = get_items_info(items)
+        dig_tree(oc, items, MetadataMenu,
+                 pass_kwargs={"base_title": title, "display_items": deeper, "previous_item_type": kind, "previous_rating_key": rating_key})
+        # we don't know exactly where we are here, only add ignore option to series
+        if should_display_ignore(items, previous=previous_item_type):
+            add_ignore_options(oc, "series", title=item_title, rating_key=rating_key, callback_menu=IgnoreMenu)
+
+        timeout = 30
+        if current_kind == "season":
+            timeout = 360
+        elif current_kind == "series":
+            timeout = 1800
+
+        # add refresh
+        oc.add(DirectoryObject(
+            key=Callback(RefreshItem, rating_key=rating_key, item_title=title, refresh_kind=current_kind,
+                         previous_rating_key=previous_rating_key, timeout=timeout*1000, randomize=timestamp()),
+            title=u"Refresh: %s" % item_title,
+            summary="Refreshes the %s, possibly searching for missing and picking up new subtitles on disk" % current_kind
+        ))
+        oc.add(DirectoryObject(
+            key=Callback(RefreshItem, rating_key=rating_key, item_title=title, force=True,
+                         refresh_kind=current_kind, previous_rating_key=previous_rating_key, timeout=timeout*1000,
+                         randomize=timestamp()),
+            title=u"Auto-Find subtitles: %s" % item_title,
+            summary="Issues a forced refresh, ignoring known subtitles and searching for new ones"
+        ))
+    else:
+        return ItemDetailsMenu(rating_key=rating_key, title=title, item_title=item_title)
+
+    return oc
+
+
+@route(PREFIX + '/ignore_list')
+def IgnoreListMenu():
+    oc = SubFolderObjectContainer(title2="Ignore list", replace_parent=True)
+    for key in ignore_list.key_order:
+        values = ignore_list[key]
+        for value in values:
+            add_ignore_options(oc, key, title=ignore_list.get_title(key, value), rating_key=value, callback_menu=IgnoreMenu)
+    return oc
+
+
+@route(PREFIX + '/history')
+def HistoryMenu():
+    from support.history import get_history
+    history = get_history()
+    oc = SubFolderObjectContainer(title2="History", replace_parent=True)
+
+    for item in history.history_items:
+        oc.add(DirectoryObject(
+            key=Callback(ItemDetailsMenu, title=item.title, item_title=item.item_title,
+                         rating_key=item.rating_key),
+            title=u"%s (%s)" % (item.item_title, item.mode_verbose),
+            summary=u"%s in %s (%s, score: %s), %s" % (item.lang_name, item.section_title,
+                                                       item.provider_name, item.score, df(item.time))
+        ))
+
+    return oc
+
+
+@route(PREFIX + '/item/{rating_key}/actions')
+@debounce
+def ItemDetailsMenu(rating_key, title=None, base_title=None, item_title=None, randomize=None):
+    """
+    displays the item details menu of an item that doesn't contain any deeper tree, such as a movie or an episode
+    :param rating_key:
+    :param title:
+    :param base_title:
+    :param item_title:
+    :param randomize:
+    :return:
+    """
+    title = unicode(base_title) + " > " + unicode(title) if base_title else unicode(title)
+    item = get_item(rating_key)
+    current_kind = get_item_kind_from_rating_key(rating_key)
+
+    timeout = 30
+
+    oc = SubFolderObjectContainer(title2=title, replace_parent=True)
+    oc.add(DirectoryObject(
+        key=Callback(RefreshItem, rating_key=rating_key, item_title=item_title, randomize=timestamp(),
+                     timeout=timeout*1000),
+        title=u"Refresh: %s" % item_title,
+        summary="Refreshes the %s, possibly searching for missing and picking up new subtitles on disk" % current_kind,
+        thumb=item.thumb or default_thumb
+    ))
+    oc.add(DirectoryObject(
+        key=Callback(RefreshItem, rating_key=rating_key, item_title=item_title, force=True, randomize=timestamp(),
+                     timeout=timeout*1000),
+        title=u"Auto-search: %s" % item_title,
+        summary="Issues a forced refresh, ignoring known subtitles and searching for new ones",
+        thumb=item.thumb or default_thumb
+    ))
+
+    # get stored subtitle info for item id
+    subtitle_storage = get_subtitle_storage()
+    stored_subs = subtitle_storage.load_or_new(item)
+
+    # get the plex item
+    plex_item = list(Plex["library"].metadata(rating_key))[0]
+
+    # get current media info for that item
+    media = plex_item.media
+
+    # look for subtitles for all available media parts and all of their languages
+    for part in media.parts:
+        filename = os.path.basename(part.file)
+        part_id = str(part.id)
+
+        # iterate through all configured languages
+        for lang in config.lang_list:
+            lang_a2 = lang.alpha2
+            # ietf lang?
+            if cast_bool(Prefs["subtitles.language.ietf"]) and "-" in lang_a2:
+                lang_a2 = lang_a2.split("-")[0]
+
+            # get corresponding stored subtitle data for that media part (physical media item), for language
+            current_sub = stored_subs.get_any(part_id, lang_a2)
+            current_sub_id = None
+            current_sub_provider_name = None
+
+            summary = u"No current subtitle in storage"
+            current_score = None
+            if current_sub:
+                current_sub_id = current_sub.id
+                current_sub_provider_name = current_sub.provider_name
+                current_score = current_sub.score
+
+                summary = u"Current subtitle: %s (added: %s, %s), Language: %s, Score: %i, Storage: %s" % \
+                          (current_sub.provider_name, df(current_sub.date_added), current_sub.mode_verbose, lang,
+                           current_sub.score, current_sub.storage_type)
+
+            oc.add(DirectoryObject(
+                key=Callback(ListAvailableSubsForItemMenu, rating_key=rating_key, part_id=part_id, title=title,
+                             item_title=item_title, language=lang, current_id=current_sub_id,
+                             item_type=plex_item.type, filename=filename, current_data=summary,
+                             randomize=timestamp(), current_provider=current_sub_provider_name,
+                             current_score=current_score),
+                title=u"List %s subtitles" % lang.name,
+                summary=summary
+            ))
+
+    add_ignore_options(oc, "videos", title=item_title, rating_key=rating_key, callback_menu=IgnoreMenu)
+
+    return oc
+
+
+def get_item_task_data(task_name, rating_key, language):
+    task_data = scheduler.get_task_data(task_name)
+    search_results = task_data.get(rating_key, {}) if task_data else {}
+    return search_results.get(language)
+
+
+@route(PREFIX + '/item/search/{rating_key}/{part_id}', force=bool)
+@debounce
+def ListAvailableSubsForItemMenu(rating_key=None, part_id=None, title=None, item_title=None, filename=None,
+                                 item_type="episode", language=None, force=False, current_id=None, current_data=None,
+                                 current_provider=None, current_score=None, randomize=None):
+    assert rating_key, part_id
+
+    running = scheduler.is_task_running("AvailableSubsForItem")
+    search_results = get_item_task_data("AvailableSubsForItem", rating_key, language)
+
+    if (search_results is None or force) and not running:
+        scheduler.dispatch_task("AvailableSubsForItem", rating_key=rating_key, item_type=item_type, part_id=part_id,
+                                language=language)
+        running = True
+
+    oc = SubFolderObjectContainer(title2=unicode(title), replace_parent=True)
+    oc.add(DirectoryObject(
+        key=Callback(ItemDetailsMenu, rating_key=rating_key, item_title=item_title, title=title, randomize=timestamp()),
+        title=u"Back to: %s" % title,
+        summary=current_data,
+        thumb=default_thumb
+    ))
+
+    metadata = get_plex_metadata(rating_key, part_id, item_type)
+    scanned_parts = scan_videos([metadata], kind="series" if item_type == "episode" else "movie", ignore_all=True)
+
+    if not scanned_parts:
+        Log.Error("Couldn't list available subtitles for %s", rating_key)
+        return oc
+
+    video, plex_part = scanned_parts.items()[0]
+
+    video_display_data = [video.format] if video.format else []
+    if video.release_group:
+        video_display_data.append(u"by %s" % video.release_group)
+    video_display_data = " ".join(video_display_data)
+
+    current_display = (u"Current: %s (%s) " % (current_provider, current_score) if current_provider else "")
+    if not running:
+        oc.add(DirectoryObject(
+            key=Callback(ListAvailableSubsForItemMenu, rating_key=rating_key, item_title=item_title, language=language,
+                         filename=filename, part_id=part_id, title=title, current_id=current_id, force=True,
+                         current_provider=current_provider, current_score=current_score,
+                         current_data=current_data, item_type=item_type, randomize=timestamp()),
+            title=u"Search for %s subs (%s)" % (get_language(language).name, video_display_data),
+            summary=u"%sFilename: %s" % (current_display, filename),
+            thumb=default_thumb
+        ))
+    else:
+        oc.add(DirectoryObject(
+            key=Callback(ListAvailableSubsForItemMenu, rating_key=rating_key, item_title=item_title,
+                         language=language, filename=filename, current_data=current_data,
+                         part_id=part_id, title=title, current_id=current_id, item_type=item_type,
+                         current_provider=current_provider, current_score=current_score,
+                         randomize=timestamp()),
+            title=u"Searching for %s subs (%s), refresh here ..." % (get_language(language).name, video_display_data),
+            summary=u"%sFilename: %s" % (current_display, filename),
+            thumb=default_thumb
+        ))
+
+    if not search_results:
+        return oc
+
+    for subtitle in search_results:
+        oc.add(DirectoryObject(
+            key=Callback(TriggerDownloadSubtitle, rating_key=rating_key, randomize=timestamp(), item_title=item_title,
+                         subtitle_id=str(subtitle.id), language=language),
+            title=u"%s: %s, score: %s" % ("Available" if current_id != subtitle.id else "Current",
+                                    subtitle.provider_name, subtitle.score),
+            summary=u"Release: %s, Matches: %s" % (subtitle.release_info, ", ".join(subtitle.matches)),
+            thumb=default_thumb
+        ))
+
+    return oc
+
+
+@route(PREFIX + '/download_subtitle/{rating_key}')
+@debounce
+def TriggerDownloadSubtitle(rating_key=None, subtitle_id=None, item_title=None, language=None, randomize=None):
+    set_refresh_menu_state("Downloading subtitle for %s" % item_title or rating_key)
+    search_results = get_item_task_data("AvailableSubsForItem", rating_key, language)
+
+    download_subtitle = None
+    for subtitle in search_results:
+        if str(subtitle.id) == subtitle_id:
+            download_subtitle = subtitle
+            break
+    if not download_subtitle:
+        Log.Error(u"Something went horribly wrong")
+
+    else:
+        scheduler.dispatch_task("DownloadSubtitleForItem", rating_key=rating_key, subtitle=download_subtitle)
+
+    return fatality(randomize=timestamp(), header=" ", replace_parent=True)
+
+
+@route(PREFIX + '/item/{rating_key}')
+@debounce
+def RefreshItem(rating_key=None, came_from="/recent", item_title=None, force=False, refresh_kind=None,
+                previous_rating_key=None, timeout=8000, randomize=None, trigger=True):
+    assert rating_key
+    header = " "
+    if trigger:
+        set_refresh_menu_state(u"Triggering %sRefresh for %s" % ("Force-" if force else "", item_title))
+        Thread.Create(refresh_item, rating_key=rating_key, force=force, refresh_kind=refresh_kind,
+                      parent_rating_key=previous_rating_key, timeout=int(timeout))
+
+        header = u"%s of item %s triggered" % ("Refresh" if not force else "Forced-refresh", rating_key)
+    return fatality(randomize=timestamp(), header=header, replace_parent=True)
+
+
+@route(PREFIX + '/missing/refresh')
+@debounce
+def RefreshMissing(randomize=None):
+    scheduler.dispatch_task("SearchAllRecentlyAddedMissing")
+    header = "Refresh of recently added items with missing subtitles triggered"
+    return fatality(header=header, replace_parent=True)
+
+
+@route(PREFIX + '/advanced')
+def AdvancedMenu(randomize=None, header=None, message=None):
+    oc = SubFolderObjectContainer(header=header or "Internal stuff, pay attention!", message=message, no_cache=True, no_history=True,
+                                  replace_parent=False, title2="Advanced")
+
+    if config.lock_advanced_menu and not config.pin_correct:
+        oc.add(DirectoryObject(
+            key=Callback(PinMenu, randomize=timestamp(), success_go_to="advanced"),
+            title=pad_title("Enter PIN"),
+            summary="The owner has restricted the access to this menu. Please enter the correct pin",
+        ))
+        return oc
+
+    oc.add(DirectoryObject(
+        key=Callback(TriggerRestart, randomize=timestamp()),
+        title=pad_title("Restart the plugin"),
+    ))
+    oc.add(DirectoryObject(
+        key=Callback(TriggerBetterSubtitles, randomize=timestamp()),
+        title=pad_title("Trigger find better subtitles"),
+    ))
+    oc.add(DirectoryObject(
+        key=Callback(LogStorage, key="tasks", randomize=timestamp()),
+        title=pad_title("Log the plugin's scheduled tasks state storage"),
+    ))
+    oc.add(DirectoryObject(
+        key=Callback(LogStorage, key="ignore", randomize=timestamp()),
+        title=pad_title("Log the plugin's internal ignorelist storage"),
+    ))
+    oc.add(DirectoryObject(
+        key=Callback(ResetStorage, key="tasks", randomize=timestamp()),
+        title=pad_title("Reset the plugin's scheduled tasks state storage"),
+    ))
+    oc.add(DirectoryObject(
+        key=Callback(ResetStorage, key="ignore", randomize=timestamp()),
+        title=pad_title("Reset the plugin's internal ignorelist storage"),
+    ))
+    return oc
+
+
+@route(PREFIX + '/ValidatePrefs', enforce_route=True)
+def ValidatePrefs():
+    Core.log.setLevel(logging.DEBUG)
+    Log.Debug("Validate Prefs called.")
+
+    # cache the channel state
+    update_dict = False
+    restart = False
+
+    # reset pin
+    Dict["pin_correct_time"] = None
+
+    config.initialize()
+    if "channel_enabled" not in Dict:
+        update_dict = True
+
+    elif Dict["channel_enabled"] != config.enable_channel:
+        Log.Debug("Channel features %s, restarting plugin", "enabled" if config.enable_channel else "disabled")
+        update_dict = True
+        restart = True
+
+    if update_dict:
+        Dict["channel_enabled"] = config.enable_channel
+        Dict.Save()
+
+    if restart:
+        DispatchRestart()
+
+    scheduler.setup_tasks()
+    set_refresh_menu_state(None)
+
+    if Prefs["log_console"]:
+        Core.log.addHandler(logger.console_handler)
+        Log.Debug("Logging to console from now on")
+    else:
+        Core.log.removeHandler(logger.console_handler)
+        Log.Debug("Stop logging to console")
+
+    Log.Debug("Setting log-level to %s", Prefs["log_level"])
+    logger.register_logging_handler(DEPENDENCY_MODULE_NAMES, level=Prefs["log_level"])
+    Core.log.setLevel(logging.getLevelName(Prefs["log_level"]))
+
+    return
+
+
+def DispatchRestart():
+    Thread.CreateTimer(1.0, Restart)
+
+
+@route(PREFIX + '/advanced/restart/trigger')
+@debounce
+def TriggerRestart(randomize=None):
+    set_refresh_menu_state("Restarting the plugin")
+    DispatchRestart()
+    return fatality(header="Restart triggered, please wait about 5 seconds", force_title=" ", only_refresh=True, replace_parent=True,
+                    no_history=True, randomize=timestamp())
+
+
+@route(PREFIX + '/advanced/restart/execute')
+def Restart():
+    Plex[":/plugins"].restart(PLUGIN_IDENTIFIER)
+
+
+@route(PREFIX + '/storage/reset', sure=bool)
+def ResetStorage(key, randomize=None, sure=False):
+    if not sure:
+        oc = SubFolderObjectContainer(no_history=True, title1="Reset subtitle storage", title2="Are you sure?")
+        oc.add(DirectoryObject(
+            key=Callback(ResetStorage, key=key, sure=True, randomize=timestamp()),
+            title=pad_title("Are you really sure?"),
+
+        ))
+        return oc
+
+    reset_storage(key)
+
+    if key == "tasks":
+        # reinitialize the scheduler
+        scheduler.init_storage()
+        scheduler.setup_tasks()
+
+    return AdvancedMenu(
+        randomize=timestamp(),
+        header='Success',
+        message='Information Storage (%s) reset' % key
+    )
+
+
+@route(PREFIX + '/storage/log')
+def LogStorage(key, randomize=None):
+    log_storage(key)
+    return AdvancedMenu(
+        randomize=timestamp(),
+        header='Success',
+        message='Information Storage (%s) logged' % key
+    )
+
+
+@route(PREFIX + '/triggerbetter')
+def TriggerBetterSubtitles(randomize=None):
+    scheduler.dispatch_task("FindBetterSubtitles")
+    return AdvancedMenu(
+        randomize=timestamp(),
+        header='Success',
+        message='FindBetterSubtitles triggered'
+    )

Contents/Code/interface/menu_helpers.py

@@ -0,0 +1,194 @@
+# coding=utf-8
+import types
+import datetime
+
+from support.items import get_kind, get_item_thumb
+from support.helpers import get_video_display_title
+from support.ignore import ignore_list
+from support.lib import get_intent
+from support.config import config
+from subzero.constants import ICON
+
+default_thumb = R(ICON)
+
+
+def should_display_ignore(items, previous=None):
+    kind = get_kind(items)
+    return items and (
+        (kind in ("show", "season")) or
+        (kind == "episode" and previous != "season")
+    )
+
+
+def add_ignore_options(oc, kind, callback_menu=None, title=None, rating_key=None, add_kind=True):
+    """
+
+    :param oc: oc to add our options to
+    :param kind: movie, show, episode ... - gets translated to the ignore key (sections, series, items)
+    :param callback_menu: menu to inject
+    :param title:
+    :param rating_key:
+    :return:
+    """
+    # try to translate kind to the ignore key
+    use_kind = kind
+    if kind not in ignore_list:
+        use_kind = ignore_list.translate_key(kind)
+    if not use_kind or use_kind not in ignore_list:
+        return
+
+    in_list = rating_key in ignore_list[use_kind]
+
+    oc.add(DirectoryObject(
+        key=Callback(callback_menu, kind=use_kind, rating_key=rating_key, title=title),
+        title=u"%s %s \"%s\" %s the ignore list" % (
+            "Remove" if in_list else "Add", ignore_list.verbose(kind) if add_kind else "", unicode(title), "from" if in_list else "to")
+    )
+    )
+
+
+def dig_tree(oc, items, menu_callback, menu_determination_callback=None, force_rating_key=None, fill_args=None,
+             pass_kwargs=None, thumb=default_thumb):
+    for kind, title, key, dig_deeper, item in items:
+        thumb = get_item_thumb(item) or thumb
+
+        add_kwargs = {}
+        if fill_args:
+            add_kwargs = dict((name, getattr(item, k)) for k, name in fill_args.iteritems() if item and hasattr(item, k))
+        if pass_kwargs:
+            add_kwargs.update(pass_kwargs)
+
+        # force details view for show/season
+        summary = " " if kind in ("show", "season") else None
+
+        oc.add(DirectoryObject(
+            key=Callback(menu_callback or menu_determination_callback(kind, item, pass_kwargs=pass_kwargs), title=title,
+                         rating_key=force_rating_key or key, **add_kwargs),
+            title=title, thumb=thumb, summary=summary
+        ))
+    return oc
+
+
+def set_refresh_menu_state(state_or_media, media_type="movies"):
+    """
+
+    :param state_or_media: string, None, or Media argument from Agent.update()
+    :param media_type: movies or series
+    :return:
+    """
+    if not state_or_media:
+        # store it in last state and remove the current
+        Dict["last_refresh_state"] = Dict["current_refresh_state"]
+        Dict["current_refresh_state"] = None
+        return
+
+    if isinstance(state_or_media, types.StringTypes):
+        Dict["current_refresh_state"] = state_or_media
+        return
+
+    media = state_or_media
+    media_id = media.id
+    title = None
+    if media_type == "series":
+        for season in media.seasons:
+            for episode in media.seasons[season].episodes:
+                ep = media.seasons[season].episodes[episode]
+                media_id = ep.id
+                title = get_video_display_title("show", ep.title, parent_title=media.title, season=int(season), episode=int(episode))
+    else:
+        title = get_video_display_title("movie", media.title)
+
+    intent = get_intent()
+    force_refresh = intent.get("force", media_id)
+
+    Dict["current_refresh_state"] = u"%sRefreshing %s" % ("Force-" if force_refresh else "", unicode(title))
+
+
+def enable_channel_wrapper(func):
+    """
+    returns the original wrapper :func: (route or handler) if applicable, else the plain to-be-wrapped function
+    :param func: original wrapper
+    :return: original wrapper or wrapped function
+    """
+    def noop(*args, **kwargs):
+        def inner(*a, **k):
+            """
+            :param a: args
+            :param k: kwargs
+            :return: originally to-be-wrapped function
+            """
+            return a[0]
+
+        return inner
+
+    def wrap(*args, **kwargs):
+        enforce_route = kwargs.pop("enforce_route", None)
+        return (func if config.enable_channel or enforce_route else noop)(*args, **kwargs)
+
+    return wrap
+
+
+def debounce(func):
+    """
+    prevent func from being called twice with the same arguments
+    :param func:
+    :return:
+    """
+    def get_lookup_key(args, kwargs):
+        func_name = list(args).pop(0).__name__
+        return tuple([func_name] + [(key, value) for key, value in kwargs.iteritems()])
+
+    def wrap(*args, **kwargs):
+        if "randomize" in kwargs:
+            if not "menu_history" in Dict:
+                Dict["menu_history"] = {}
+
+            key = get_lookup_key([func] + list(args), kwargs)
+            if key in Dict["menu_history"]:
+                Log.Debug("not triggering %s twice with %s, %s" % (func, args, kwargs))
+                return ObjectContainer()
+            else:
+                Dict["menu_history"][key] = datetime.datetime.now() + datetime.timedelta(days=1)
+                Dict.Save()
+        return func(*args, **kwargs)
+
+    return wrap
+
+
+class SZObjectContainer(ObjectContainer):
+    def __init__(self, *args, **kwargs):
+        skip_pin_lock = kwargs.pop("skip_pin_lock", False)
+
+        super(SZObjectContainer, self).__init__(*args, **kwargs)
+
+        if (config.lock_menu or config.lock_advanced_menu) and not config.pin_correct and not skip_pin_lock:
+            config.locked = True
+
+    def add(self, *args, **kwargs):
+        # disable self.add if we're in lockdown
+        container = args[0]
+        current_menu_target = container.key.split("?")[0]
+        is_pin_menu = current_menu_target.endswith("/pin")
+
+        if config.locked and config.lock_menu and not is_pin_menu:
+            return
+        return super(SZObjectContainer, self).add(*args, **kwargs)
+
+
+OriginalObjectContainer = ObjectContainer
+ObjectContainer = SZObjectContainer
+
+
+class SubFolderObjectContainer(ObjectContainer):
+    def __init__(self, *args, **kwargs):
+        super(SubFolderObjectContainer, self).__init__(*args, **kwargs)
+        from interface.menu import fatality
+        from support.helpers import pad_title, timestamp
+        self.add(DirectoryObject(
+            key=Callback(fatality, force_title=" ", randomize=timestamp()),
+            title=pad_title("<< Back to home"),
+            summary="Current state: %s; Last state: %s" % (
+                (Dict["current_refresh_state"] or "Idle") if "current_refresh_state" in Dict else "Idle",
+                (Dict["last_refresh_state"] or "None") if "last_refresh_state" in Dict else "None"
+            )
+        ))

Contents/Code/logger.py

@@ -1,15 +1,22 @@
 import logging
 
-def registerLoggingHander(dependencies):
-    plexHandler = PlexLoggerHandler()
-    for dependency in dependencies:     
-        Log.Debug("Registering LoggerHandler for dependency: %s" % dependency)   
+
+def register_logging_handler(dependencies, level="ERROR"):
+    plex_handler = PlexLoggerHandler()
+    for dependency in dependencies:
+        Log.Debug("Registering LoggerHandler for dependency: %s" % dependency)
         log = logging.getLogger(dependency)
-        log.setLevel('DEBUG')
-        log.addHandler(plexHandler)
+        # remove previous plex logging handlers
+        # fixme: this is not the most elegant solution...
+        for handler in log.handlers:
+            if isinstance(handler, PlexLoggerHandler):
+                log.removeHandler(handler)
+
+        log.setLevel(level)
+        log.addHandler(plex_handler)
+
 
 class PlexLoggerHandler(logging.StreamHandler):
-    
     def __init__(self, level=0):
         super(PlexLoggerHandler, self).__init__(level)
 
@@ -30,4 +37,9 @@ class PlexLoggerHandler(logging.StreamHandler):
         elif record.levelno == logging.FATAL:
             Log.Exception(self.getFormattedString(record))
         else:
-            Log.Error("UNKNOWN LEVEL: %s", record.getMessage())
+            Log.Error("UNKNOWN LEVEL: %s", record.getMessage())
+
+
+console_handler = logging.StreamHandler()
+console_formatter = Framework.core.LogFormatter('%(asctime)-15s - %(name)-32s (%(thread)x) :  %(levelname)s (%(module)s:%(lineno)d) - %(message)s')
+console_handler.setFormatter(console_formatter)

Contents/Code/support/LICENSE

@@ -0,0 +1,6 @@
+License for parts taken out of plexinc-agents/LocalMedia.bundle
+
+License
+-------
+
+If the software submitted to this repository accesses or calls any software provided by Plex (“Interfacing Software”), then as a condition for receiving services from Plex in response to such accesses or calls, you agree to grant and do hereby grant to Plex and its affiliates worldwide a worldwide, nonexclusive, and royalty-free right and license to use (including testing, hosting and linking to), copy, publicly perform, publicly display, reproduce in copies for distribution, and distribute the copies of any Interfacing Software made by you or with your assistance; provided, however, that you may notify Plex at legal@plex.tv if you do not wish for Plex to use, distribute, copy, publicly perform, publicly display, reproduce in copies for distribution, or distribute copies of an Interfacing Software that was created by you, and Plex will reasonable efforts to comply with such a request within a reasonable time.

Contents/Code/support/__init__.py

@@ -0,0 +1,60 @@
+import sys
+# thanks, https://github.com/trakt/Plex-Trakt-Scrobbler/blob/master/Trakttv.bundle/Contents/Code/core/__init__.py
+
+import config
+
+sys.modules["support.config"] = config
+
+import helpers
+
+sys.modules["support.helpers"] = helpers
+
+import lib
+
+sys.modules["support.lib"] = lib
+
+import plex_media
+sys.modules["support.plex_media"] = plex_media
+
+import localmedia
+
+sys.modules["subzero.localmedia"] = localmedia
+
+import subtitlehelpers
+
+sys.modules["support.subtitlehelpers"] = subtitlehelpers
+
+import items
+
+sys.modules["support.items"] = items
+
+import missing_subtitles
+
+sys.modules["support.missing_subtitles"] = missing_subtitles
+
+import background
+
+sys.modules["support.background"] = background
+
+import tasks
+
+sys.modules["support.tasks"] = tasks
+
+import storage
+
+sys.modules["support.storage"] = storage
+
+import ignore
+
+sys.modules["support.ignore"] = ignore
+
+import history
+
+sys.modules["support.history"] = history
+
+import data
+
+sys.modules["support.data"] = data
+
+import activities
+sys.modules["support.activities"] = activities

Contents/Code/support/activities.py

@@ -0,0 +1,105 @@
+# coding=utf-8
+from wraptor.decorators import throttle
+from config import config
+from items import get_item, get_item_kind_from_item, refresh_item
+
+from plex_activity import Activity
+from plex_activity.sources.s_logging.main import Logging as Activity_Logging
+
+
+class PlexActivityManager(object):
+    def start(self):
+        activity_sources_enabled = None
+
+        if config.universal_plex_token:
+            from plex import Plex
+            Plex.configuration.defaults.authentication(config.universal_plex_token)
+            activity_sources_enabled = ["websocket"]
+            Activity.on('websocket.playing', self.on_playing)
+
+        elif config.server_log_path:
+            Activity_Logging.add_hint(config.server_log_path, None)
+            activity_sources_enabled = ["logging"]
+            Activity.on('logging.playing', self.on_playing)
+
+        if activity_sources_enabled:
+            Activity.start(activity_sources_enabled)
+
+    @throttle(5, instance_method=True)
+    def on_playing(self, info):
+        if not config.use_activities:
+            return
+
+        # don't trigger on the first hit ever
+        if "last_played_item" not in Dict:
+            Dict["last_played_item"] = None
+            Dict.Save()
+            return
+
+        rating_key = info["ratingKey"]
+        if Dict["last_played_item"] != rating_key:
+            # new playing
+            Dict["last_played_item"] = rating_key
+            Dict.Save()
+
+            debug_msg = "Started playing %s. Refreshing it." % rating_key
+
+            key_to_refresh = None
+            if config.activity_mode in ["refresh", "next_episode", "hybrid"]:
+                # next episode or next episode and current movie
+                if config.activity_mode in ["next_episode", "hybrid"]:
+                    plex_item = get_item(rating_key)
+                    if not plex_item:
+                        Log.Warn("Can't determine media type of %s, skipping" % rating_key)
+                        return
+
+                    if get_item_kind_from_item(plex_item) == "episode":
+                        next_ep = self.get_next_episode(rating_key)
+                        if next_ep:
+                            key_to_refresh = next_ep.rating_key
+                            debug_msg = "Started playing %s. Refreshing next episode (%s, S%02iE%02i)." % \
+                                        (rating_key, next_ep.rating_key, int(next_ep.season.index), int(next_ep.index))
+
+                    else:
+                        if config.activity_mode == "hybrid":
+                            key_to_refresh = rating_key
+                elif config.activity_mode == "refresh":
+                    key_to_refresh = rating_key
+
+                if key_to_refresh:
+                    Log.Debug(debug_msg)
+                    refresh_item(key_to_refresh)
+
+    def get_next_episode(self, rating_key):
+        plex_item = get_item(rating_key)
+        if not plex_item:
+            return
+
+        if get_item_kind_from_item(plex_item) == "episode":
+            # get season
+            season = get_item(plex_item.season.rating_key)
+            if not season:
+                return
+
+            # determine next episode
+            # next episode is in the same season
+            if plex_item.index < season.episode_count:
+                # get next ep
+                for ep in season.children():
+                    if ep.index == plex_item.index + 1:
+                        return ep
+
+            # it's not, try getting the first episode of the next season
+            else:
+                # get show
+                show = get_item(plex_item.show.rating_key)
+                # is there a next season?
+                if season.index < show.season_count:
+                    for other_season in show.children():
+                        if other_season.index == season.index + 1:
+                            next_season = other_season
+                            for ep in next_season.children():
+                                if ep.index == 1:
+                                    return ep
+
+activity = PlexActivityManager()

Contents/Code/support/auth.py

@@ -0,0 +1,42 @@
+# coding=utf-8
+
+
+def refresh_plex_token():
+    username = Prefs["plex_username"]
+    password = Prefs["plex_password"]
+
+    if not username or not password:
+        if "token" in Dict:
+            del Dict["token"]
+            Dict.Save()
+        return
+
+    if "uuid" not in Dict:
+        Dict["uuid"] = String.UUID()
+        Dict.Save()
+
+    current_uuid = Dict["uuid"]
+
+    headers = {
+        'X-Plex-Device-Name': 'Sub-Zero',
+        'X-Plex-Product': 'Sub-Zero',
+        'X-Plex-Version': '1.3.0',
+        'X-Plex-Client-Identifier': "%s" % current_uuid,
+    }
+
+    request = HTTP.Request("https://plex.tv/users/sign_in.json", headers=headers,
+                           values={'user[login]': Prefs["plex_username"], 'user[password]': Prefs["plex_password"]}, immediate=True)
+    token = None
+    if request:
+        try:
+            data = JSON.ObjectFromString(request.content)
+            token = data["user"]["authentication_token"]
+            log_data = data.copy()
+            log_data["user"]["authentication_token"] = "xxxxxxxxxxxxxxxxxx"
+            Log.Debug("Data returned from plex.tv: %s", log_data)
+        except:
+            pass
+        if token:
+            Dict["token"] = token
+            Dict.Save()
+            return True

Contents/Code/support/background.py

@@ -0,0 +1,189 @@
+# coding=utf-8
+
+import datetime
+import logging
+import traceback
+
+
+def parse_frequency(s):
+    if s == "never" or s == None:
+        return None, None
+    kind, num, unit = s.split()
+    return int(num), unit
+
+
+class DefaultScheduler(object):
+    thread = None
+    running = False
+    registry = None
+
+    def __init__(self):
+        self.thread = None
+        self.running = False
+        self.registry = []
+
+        self.tasks = {}
+        self.init_storage()
+
+    def init_storage(self):
+        if "tasks" not in Dict:
+            Dict["tasks"] = {"queue": []}
+            Dict.Save()
+
+        if "queue" not in Dict["tasks"]:
+            Dict["tasks"]["queue"] = []
+
+    def get_task_data(self, name):
+        if name not in Dict["tasks"]:
+            raise NotImplementedError("Task missing! %s" % name)
+
+        if "data" in Dict["tasks"][name]:
+            return Dict["tasks"][name]["data"]
+
+    def clear_task_data(self, name=None):
+        if name is None:
+            # full clean
+            Log.Debug("Clearing previous task data")
+            if Dict["tasks"]:
+                for task_name in Dict["tasks"].keys():
+                    if task_name == "queue":
+                        continue
+
+                    Dict["tasks"][task_name]["data"] = {}
+                    Dict["tasks"][task_name]["running"] = False
+                Dict.Save()
+            return
+
+        if name not in Dict["tasks"]:
+            raise NotImplementedError("Task missing! %s" % name)
+
+        Dict["tasks"][name]["data"] = {}
+        Dict.Save()
+        Log.Debug("Task data cleared: %s", name)
+
+    def register(self, task):
+        self.registry.append(task)
+
+    def setup_tasks(self):
+        # discover tasks;
+        self.tasks = {}
+        for cls in self.registry:
+            task = cls(self)
+            try:
+                task_frequency = Prefs["scheduler.tasks.%s.frequency" % task.name]
+            except KeyError:
+                task_frequency = None
+
+            self.tasks[task.name] = {"task": task, "frequency": parse_frequency(task_frequency)}
+
+    def run(self):
+        self.running = True
+        self.thread = Thread.Create(self.worker)
+
+    def stop(self):
+        self.running = False
+
+    def task(self, name):
+        if name not in self.tasks:
+            return None
+        return self.tasks[name]["task"]
+
+    def is_task_running(self, name):
+        task = self.task(name)
+        if task:
+            return task.running
+
+    def last_run(self, task):
+        if task not in self.tasks:
+            return None
+        return self.tasks[task]["task"].last_run
+
+    def next_run(self, task):
+        if task not in self.tasks or not self.tasks[task]["task"].periodic:
+            return None
+        frequency_num, frequency_key = self.tasks[task]["frequency"]
+        if not frequency_num:
+            return None
+        last = self.tasks[task]["task"].last_run
+        use_date = last
+        now = datetime.datetime.now()
+        if not use_date:
+            use_date = now
+        return max(use_date + datetime.timedelta(**{frequency_key: frequency_num}), now)
+
+    def run_task(self, name, *args, **kwargs):
+        task = self.tasks[name]["task"]
+        if task.running:
+            Log.Debug("Scheduler: Not running %s, as it's currently running.", name)
+            return False
+
+        Log.Debug("Scheduler: Running task %s", name)
+        try:
+            task.prepare(*args, **kwargs)
+            task.run()
+        except Exception, e:
+            Log.Error("Scheduler: Something went wrong when running %s: %s", name, traceback.format_exc())
+        finally:
+            task.post_run(Dict["tasks"][name]["data"])
+            Dict.Save()
+
+    def dispatch_task(self, *args, **kwargs):
+        if "queue" not in Dict["tasks"]:
+            Dict["tasks"]["queue"] = []
+
+        Dict["tasks"]["queue"].append((args, kwargs))
+
+    def signal(self, name, *args, **kwargs):
+        for task_name, info in self.tasks.iteritems():
+            task = info["task"]
+            if not task.periodic:
+                continue
+
+            if task.running:
+                Log.Debug("Scheduler: Sending signal %s to task %s (%s, %s)", name, task_name, args, kwargs)
+                status = task.signal(name, *args, **kwargs)
+                if status:
+                    Log.Debug("Scheduler: Signal accepted by %s", task_name)
+                else:
+                    Log.Debug("Scheduler: Signal not accepted by %s", task_name)
+                continue
+            Log.Debug("Scheduler: Not sending signal %s to task %s, because: not running", name, task_name)
+
+    def worker(self):
+        Thread.Sleep(10.0)
+        while 1:
+            if not self.running:
+                break
+
+            # single dispatch requested?
+            if Dict["tasks"]["queue"]:
+                # work queue off
+                queue = Dict["tasks"]["queue"][:]
+                Dict["tasks"]["queue"] = []
+                Dict.Save()
+                for args, kwargs in queue:
+                    Log.Debug("Dispatching single task: %s, %s", args, kwargs)
+                    Thread.Create(self.run_task, True, *args, **kwargs)
+
+            # scheduled tasks
+            for name, info in self.tasks.iteritems():
+                now = datetime.datetime.now()
+                task = info["task"]
+
+                if name not in Dict["tasks"] or not task.periodic:
+                    continue
+
+                if task.running:
+                    continue
+
+                frequency_num, frequency_key = info["frequency"]
+                if not frequency_num:
+                    continue
+
+                if not task.last_run or (task.last_run + datetime.timedelta(**{frequency_key: frequency_num}) <= now):
+                    self.run_task(name)
+
+            Thread.Sleep(5.0)
+
+
+scheduler = DefaultScheduler()

Contents/Code/support/config.py

@@ -0,0 +1,392 @@
+# coding=utf-8
+
+import os
+import re
+import inspect
+
+import datetime
+
+import subliminal
+import subliminal_patch
+from babelfish import Language
+from subzero.lib.io import FileIO, get_viable_encoding
+from subzero.constants import PLUGIN_NAME, PLUGIN_IDENTIFIER, MOVIE, SHOW
+from lib import Plex
+from helpers import check_write_permissions, cast_bool
+
+SUBTITLE_EXTS = ['utf', 'utf8', 'utf-8', 'srt', 'smi', 'rt', 'ssa', 'aqt', 'jss', 'ass', 'idx', 'sub', 'txt', 'psb']
+VIDEO_EXTS = ['3g2', '3gp', 'asf', 'asx', 'avc', 'avi', 'avs', 'bivx', 'bup', 'divx', 'dv', 'dvr-ms', 'evo', 'fli',
+              'flv',
+              'm2t', 'm2ts', 'm2v', 'm4v', 'mkv', 'mov', 'mp4', 'mpeg', 'mpg', 'mts', 'nsv', 'nuv', 'ogm', 'ogv', 'tp',
+              'pva', 'qt', 'rm', 'rmvb', 'sdp', 'svq3', 'strm', 'ts', 'ty', 'vdr', 'viv', 'vob', 'vp3', 'wmv', 'wpl',
+              'wtv', 'xsp', 'xvid',
+              'webm']
+
+IGNORE_FN = ("subzero.ignore", ".subzero.ignore", ".nosz")
+
+VERSION_RE = re.compile(ur'CFBundleVersion.+?<string>([0-9\.]+)</string>', re.DOTALL)
+
+
+def int_or_default(s, default):
+    try:
+        return int(s)
+    except ValueError:
+        return default
+
+
+class Config(object):
+    version = None
+    full_version = None
+    server_log_path = None
+    app_support_path = None
+    universal_plex_token = None
+
+    enable_channel = True
+    enable_agent = True
+    pin = None
+    lock_menu = False
+    lock_advanced_menu = False
+    locked = False
+    pin_valid_minutes = 10
+    lang_list = None
+    subtitle_destination_folder = None
+    providers = None
+    provider_settings = None
+    max_recent_items_per_library = 200
+    permissions_ok = False
+    missing_permissions = None
+    ignore_sz_files = False
+    ignore_paths = None
+    fs_encoding = None
+    notify_executable = None
+    sections = None
+    enabled_sections = None
+    enforce_encoding = False
+    chmod = None
+    forced_only = False
+    treat_und_as_first = False
+    ext_match_strictness = False
+    use_activities = False
+    activity_mode = None
+
+    initialized = False
+
+    def initialize(self):
+        self.fs_encoding = get_viable_encoding()
+        self.version = self.get_version()
+        self.full_version = u"%s %s" % (PLUGIN_NAME, self.version)
+        self.server_log_path = self.get_server_log_path()
+        self.app_support_path = Core.app_support_path
+        self.universal_plex_token = self.get_universal_plex_token()
+
+        self.set_plugin_mode()
+        self.set_plugin_lock()
+        self.set_activity_modes()
+
+        self.lang_list = self.get_lang_list()
+        self.subtitle_destination_folder = self.get_subtitle_destination_folder()
+        self.providers = self.get_providers()
+        self.provider_settings = self.get_provider_settings()
+        self.max_recent_items_per_library = int_or_default(Prefs["scheduler.max_recent_items_per_library"], 2000)
+        self.sections = list(Plex["library"].sections())
+        self.missing_permissions = []
+        self.ignore_sz_files = cast_bool(Prefs["subtitles.ignore_fs"])
+        self.ignore_paths = self.parse_ignore_paths()
+        self.enabled_sections = self.check_enabled_sections()
+        self.permissions_ok = self.check_permissions()
+        self.notify_executable = self.check_notify_executable()
+        self.enforce_encoding = cast_bool(Prefs['subtitles.enforce_encoding'])
+        self.chmod = self.check_chmod()
+        self.forced_only = cast_bool(Prefs["subtitles.only_foreign"])
+        self.treat_und_as_first = cast_bool(Prefs["subtitles.language.treat_und_as_first"])
+        self.ext_match_strictness = self.determine_ext_sub_strictness()
+        self.initialized = True
+
+    def get_server_log_path(self):
+        # find log handler
+        for handler in Core.log.handlers:
+            if getattr(getattr(handler, "__class__"), "__name__") in (
+                    'FileHandler', 'RotatingFileHandler', 'TimedRotatingFileHandler'):
+                plugin_log_file = handler.baseFilename
+
+                if plugin_log_file:
+                    server_log_file = os.path.realpath(os.path.join(plugin_log_file, "../../Plex Media Server.log"))
+                    if os.path.isfile(server_log_file):
+                        return server_log_file
+
+    def get_universal_plex_token(self):
+        # thanks to: https://forums.plex.tv/discussion/247136/read-current-x-plex-token-in-an-agent-ensure-that-a-http-request-gets-executed-exactly-once#latest
+        pref_path = os.path.join(self.app_support_path, "Preferences.xml")
+        if os.path.exists(pref_path):
+            try:
+                global_prefs = Core.storage.load(pref_path)
+                return XML.ElementFromString(global_prefs).xpath('//Preferences/@PlexOnlineToken')[0]
+            except:
+                Log.Warn("Couldn't determine Plex Token")
+        else:
+            Log("Did NOT find Preferences file - please check logfile and hierarchy. Aborting!")
+
+    def set_plugin_mode(self):
+        if Prefs["plugin_mode"] == "only agent":
+            self.enable_channel = False
+        elif Prefs["plugin_mode"] == "only channel":
+            self.enable_agent = False
+
+    def set_plugin_lock(self):
+        if Prefs["plugin_pin_mode"] in ("channel menu", "advanced menu"):
+            # check pin
+            pin = Prefs["plugin_pin"]
+            if not pin or not len(pin):
+                Log.Warn("PIN enabled but not set, disabling PIN!")
+                return
+
+            pin = pin.strip()
+            try:
+                int(pin)
+            except ValueError:
+                Log.Warn("PIN has to be an integer (0-9)")
+            self.pin = pin
+            self.lock_advanced_menu = Prefs["plugin_pin_mode"] == "advanced menu"
+            self.lock_menu = Prefs["plugin_pin_mode"] == "channel menu"
+
+            try:
+                self.pin_valid_minutes = int(Prefs["plugin_pin_valid_for"].strip())
+            except ValueError:
+                pass
+
+    @property
+    def pin_correct(self):
+        if isinstance(Dict["pin_correct_time"], datetime.datetime) \
+                and Dict["pin_correct_time"] + datetime.timedelta(
+                    minutes=self.pin_valid_minutes) > datetime.datetime.now():
+            return True
+
+    def refresh_permissions_status(self):
+        self.permissions_ok = self.check_permissions()
+
+    def check_permissions(self):
+        if not Prefs["subtitles.save.filesystem"] or not Prefs["check_permissions"]:
+            return True
+
+        self.missing_permissions = []
+        use_ignore_fs = Prefs["subtitles.ignore_fs"]
+        all_permissions_ok = True
+        for section in self.sections:
+            if section.key not in self.enabled_sections:
+                continue
+
+            title = section.title
+            for location in section:
+                path_str = location.path
+                if isinstance(path_str, unicode):
+                    path_str = path_str.encode(self.fs_encoding)
+
+                if use_ignore_fs:
+                    # check whether we've got an ignore file inside the section path
+                    if self.is_physically_ignored(path_str):
+                        continue
+
+                if self.is_path_ignored(path_str):
+                    # is the path in our ignored paths setting?
+                    continue
+
+                # section not ignored, check for write permissions
+                if not check_write_permissions(path_str):
+                    # not enough permissions
+                    self.missing_permissions.append((title, location.path))
+                    all_permissions_ok = False
+
+        return all_permissions_ok
+
+    def get_version(self):
+        curDir = os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
+        info_file_path = os.path.abspath(os.path.join(curDir, "..", "..", "Info.plist"))
+        data = FileIO.read(info_file_path)
+        result = VERSION_RE.search(data)
+        if result:
+            return result.group(1)
+
+    def parse_ignore_paths(self):
+        paths = Prefs["subtitles.ignore_paths"]
+        if paths:
+            try:
+                return [path.strip() for path in paths.split(",")]
+            except:
+                Log.Error("Couldn't parse your ignore paths settings: %s" % paths)
+        return []
+
+    def is_physically_ignored(self, folder):
+        # check whether we've got an ignore file inside the path
+        for ifn in IGNORE_FN:
+            if os.path.isfile(os.path.join(folder, ifn)):
+                Log.Info(u'Ignoring "%s" because "%s" exists', folder, ifn)
+                return True
+
+        return False
+
+    def is_path_ignored(self, fn):
+        for path in self.ignore_paths:
+            if fn.startswith(path):
+                return True
+        return False
+
+    def check_notify_executable(self):
+        fn = Prefs["notify_executable"]
+        if not fn:
+            return
+
+        splitted_fn = fn.split()
+        exe_fn = splitted_fn[0]
+        arguments = [arg.strip() for arg in splitted_fn[1:]]
+
+        if os.path.isfile(exe_fn) and os.access(exe_fn, os.X_OK):
+            return exe_fn, arguments
+        Log.Error("Notify executable not existing or not executable: %s" % exe_fn)
+
+    def refresh_enabled_sections(self):
+        self.enabled_sections = self.check_enabled_sections()
+
+    def check_enabled_sections(self):
+        enabled_for_primary_agents = []
+        enabled_sections = {}
+
+        # find which agents we're enabled for
+        for agent in Plex.agents():
+            if not agent.primary:
+                continue
+
+            for t in list(agent.media_types):
+                if t.media_type in (MOVIE, SHOW):
+                    related_agents = Plex.primary_agent(agent.identifier, t.media_type)
+                    for a in related_agents:
+                        if a.identifier == PLUGIN_IDENTIFIER and a.enabled:
+                            enabled_for_primary_agents.append(agent.identifier)
+
+        # find the libraries that use them
+        for library in self.sections:
+            if library.agent in enabled_for_primary_agents:
+                enabled_sections[library.key] = library
+
+        Log.Debug(u"I'm enabled for: %s" % [lib.title for key, lib in enabled_sections.iteritems()])
+        return enabled_sections
+
+    # Prepare a list of languages we want subs for
+    def get_lang_list(self):
+        l = {Language.fromietf(Prefs["langPref1"])}
+        lang_custom = Prefs["langPrefCustom"].strip()
+
+        if Prefs['subtitles.only_one']:
+            return l
+
+        if Prefs["langPref2"] != "None":
+            l.update({Language.fromietf(Prefs["langPref2"])})
+
+        if Prefs["langPref3"] != "None":
+            l.update({Language.fromietf(Prefs["langPref3"])})
+
+        if len(lang_custom) and lang_custom != "None":
+            for lang in lang_custom.split(u","):
+                lang = lang.strip()
+                try:
+                    real_lang = Language.fromietf(lang)
+                except:
+                    try:
+                        real_lang = Language.fromname(lang)
+                    except:
+                        continue
+                l.update({real_lang})
+
+        return l
+
+    def get_subtitle_destination_folder(self):
+        if not Prefs["subtitles.save.filesystem"]:
+            return
+
+        fld_custom = Prefs["subtitles.save.subFolder.Custom"].strip() if cast_bool(
+            Prefs["subtitles.save.subFolder.Custom"]) else None
+        return fld_custom or (
+        Prefs["subtitles.save.subFolder"] if Prefs["subtitles.save.subFolder"] != "current folder" else None)
+
+    def get_providers(self):
+        providers = {'opensubtitles': cast_bool(Prefs['provider.opensubtitles.enabled']),
+                     # 'thesubdb': Prefs['provider.thesubdb.enabled'],
+                     'podnapisi': cast_bool(Prefs['provider.podnapisi.enabled']),
+                     'addic7ed': cast_bool(Prefs['provider.addic7ed.enabled']),
+                     'tvsubtitles': cast_bool(Prefs['provider.tvsubtitles.enabled'])
+                     }
+
+        # ditch non-forced-subtitles-reporting providers
+        if cast_bool(Prefs['subtitles.only_foreign']):
+            providers["addic7ed"] = False
+            providers["tvsubtitles"] = False
+
+        return filter(lambda prov: providers[prov], providers)
+
+    def get_provider_settings(self):
+        provider_settings = {'addic7ed': {'username': Prefs['provider.addic7ed.username'],
+                                          'password': Prefs['provider.addic7ed.password'],
+                                          'use_random_agents': cast_bool(Prefs['provider.addic7ed.use_random_agents']),
+                                          },
+                             'opensubtitles': {'username': Prefs['provider.opensubtitles.username'],
+                                               'password': Prefs['provider.opensubtitles.password'],
+                                               'use_tag_search': cast_bool(Prefs['provider.opensubtitles.use_tags']),
+                                               'only_foreign': cast_bool(Prefs['subtitles.only_foreign'])
+                                               },
+                             'podnapisi': {
+                                 'only_foreign': cast_bool(Prefs['subtitles.only_foreign'])
+                             },
+                             }
+
+        return provider_settings
+
+    def check_chmod(self):
+        val = Prefs["subtitles.save.chmod"]
+        if not val or not len(val):
+            return
+
+        wrong_chmod = False
+        if len(val) != 4:
+            wrong_chmod = True
+
+        try:
+            return int(val, 8)
+        except ValueError:
+            wrong_chmod = True
+
+        if wrong_chmod:
+            Log.Warn("Chmod setting ignored, please use only 4-digit integers with leading 0 (e.g.: 775)")
+
+    def determine_ext_sub_strictness(self):
+        val = Prefs["subtitles.scan.filename_strictness"]
+        if val == "any":
+            return "any"
+        elif val.startswith("loose"):
+            return "loose"
+        return "strict"
+
+    def set_activity_modes(self):
+        val = Prefs["activity.on_playback"]
+        if val == "never":
+            self.use_activities = False
+            return
+
+        self.use_activities = True
+        if val == "current media item":
+            self.activity_mode = "refresh"
+        elif val == "hybrid: current item or next episode":
+            self.activity_mode = "hybrid"
+        else:
+            self.activity_mode = "next_episode"
+
+    def init_subliminal_patches(self):
+        # configure custom subtitle destination folders for scanning pre-existing subs
+        Log.Debug("Patching subliminal ...")
+        dest_folder = self.subtitle_destination_folder
+        subliminal_patch.patch_video.CUSTOM_PATHS = [dest_folder] if dest_folder else []
+        subliminal_patch.patch_video.INCLUDE_EXOTIC_SUBS = cast_bool(Prefs["subtitles.scan.exotic_ext"])
+        subliminal_patch.patch_provider_pool.DOWNLOAD_TRIES = int(Prefs['subtitles.try_downloads'])
+        subliminal.video.Episode.scores["addic7ed_boost"] = int(Prefs['provider.addic7ed.boost_by'])
+
+
+config = Config()
+config.initialize()

Contents/Code/support/data.py

@@ -0,0 +1,84 @@
+# coding=utf-8
+
+
+def dispatch_migrate():
+    try:
+        migrate()
+    except:
+        Log.Error("Migration failed: %s" % traceback.format_exc())
+
+
+def migrate():
+    """
+    some Dict/Data migrations here, no need for a more in-depth migration path for now
+    :return:
+    """
+
+    # migrate subtitle history from Dict to Data
+    if "history" in Dict and Dict["history"].get("history_items"):
+        Log.Debug("Running migration for history data")
+        from support.history import get_history
+        history = get_history()
+
+        for item in reversed(Dict["history"]["history_items"]):
+            history.add(item.item_title, item.rating_key, item.section_title, subtitle=item.subtitle, mode=item.mode,
+                        time=item.time)
+
+        del Dict["history"]
+        Dict.Save()
+
+    # migrate subtitle storage from Dict to Data
+    if "subs" in Dict:
+        from support.storage import get_subtitle_storage
+        from subzero.subtitle_storage import StoredSubtitle
+        from support.plex_media import get_item
+
+        subtitle_storage = get_subtitle_storage()
+
+        for video_id, parts in Dict["subs"].iteritems():
+            try:
+                item = get_item(video_id)
+            except:
+                continue
+
+            if not item:
+                continue
+
+            stored_subs = subtitle_storage.load_or_new(item)
+            stored_subs.version = 1
+
+            Log.Debug(u"Migrating %s" % video_id)
+
+            stored_any = False
+            for part_id, lang_dict in parts.iteritems():
+                part_id = str(part_id)
+                Log.Debug(u"Migrating %s, %s" % (video_id, part_id))
+
+                for lang, subs in lang_dict.iteritems():
+                    lang = str(lang)
+                    if "current" in subs:
+                        current_key = subs["current"]
+                        provider_name, subtitle_id = current_key
+                        sub = subs.get(current_key)
+                        if sub and sub.get("title") and sub.get("mode"):  # ditch legacy data without sufficient info
+                            stored_subs.title = sub["title"]
+                            new_sub = StoredSubtitle(sub["score"], sub["storage"], sub["hash"], provider_name,
+                                                     subtitle_id, date_added=sub["date_added"], mode=sub["mode"])
+
+                            if part_id not in stored_subs.parts:
+                                stored_subs.parts[part_id] = {}
+
+                            if lang not in stored_subs.parts[part_id]:
+                                stored_subs.parts[part_id][lang] = {}
+
+                            Log.Debug(u"Migrating %s, %s, %s" % (video_id, part_id, current_key))
+
+                            stored_subs.parts[part_id][lang][current_key] = new_sub
+                            stored_subs.parts[part_id][lang]["current"] = current_key
+                            stored_any = True
+
+            if stored_any:
+                subtitle_storage.save(stored_subs)
+
+        del Dict["subs"]
+        Dict.Save()

Contents/Code/support/helpers.py

@@ -0,0 +1,294 @@
+# coding=utf-8
+import os
+import traceback
+import types
+import unicodedata
+import datetime
+import urllib
+import time
+import re
+import platform
+import subprocess
+
+from bs4 import UnicodeDammit
+
+import chardet
+
+from babelfish import Language
+
+from subzero.analytics import track_event
+
+# Unicode control characters can appear in ID3v2 tags but are not legal in XML.
+RE_UNICODE_CONTROL = u'([\u0000-\u0008\u000b-\u000c\u000e-\u001f\ufffe-\uffff])' + \
+                     u'|' + \
+                     u'([%s-%s][^%s-%s])|([^%s-%s][%s-%s])|([%s-%s]$)|(^[%s-%s])' % \
+                     (
+                         unichr(0xd800), unichr(0xdbff), unichr(0xdc00), unichr(0xdfff),
+                         unichr(0xd800), unichr(0xdbff), unichr(0xdc00), unichr(0xdfff),
+                         unichr(0xd800), unichr(0xdbff), unichr(0xdc00), unichr(0xdfff)
+                     )
+
+
+def cast_bool(value):
+    return str(value) in ("true", "True")
+
+
+# A platform independent way to split paths which might come in with different separators.
+def split_path(str):
+    if str.find('\\') != -1:
+        return str.split('\\')
+    else:
+        return str.split('/')
+
+
+def unicodize(s):
+    filename = s
+    try:
+        filename = unicodedata.normalize('NFC', unicode(s.decode('utf-8')))
+    except:
+        Log('Failed to unicodize: ' + repr(filename))
+    try:
+        filename = re.sub(RE_UNICODE_CONTROL, '', filename)
+    except:
+        Log('Couldn\'t strip control characters: ' + repr(filename))
+    return filename
+
+
+def force_unicode(s):
+    if not isinstance(s, types.UnicodeType):
+        try:
+            s = s.decode("utf-8")
+        except UnicodeDecodeError:
+            t = chardet.detect(s)
+            try:
+                s = s.decode(t["encoding"])
+            except UnicodeDecodeError:
+                s = UnicodeDammit(s).unicode_markup
+    return s
+
+
+def clean_filename(filename):
+    # this will remove any whitespace and punctuation chars and replace them with spaces, strip and return as lowercase
+    return string.translate(filename.encode('utf-8'), string.maketrans(string.punctuation + string.whitespace,
+                                                                       ' ' * len(
+                                                                           string.punctuation + string.whitespace))).strip().lower()
+
+
+def is_recent(t):
+    now = datetime.datetime.now()
+    when = datetime.datetime.fromtimestamp(t)
+    value, key = Prefs["scheduler.item_is_recent_age"].split()
+    if now - datetime.timedelta(**{key: int(value)}) < when:
+        return True
+    return False
+
+
+# thanks, Plex-Trakt-Scrobbler
+def str_pad(s, length, align='left', pad_char=' ', trim=False):
+    if not s:
+        return s
+
+    if not isinstance(s, (str, unicode)):
+        s = str(s)
+
+    if len(s) == length:
+        return s
+    elif len(s) > length and not trim:
+        return s
+
+    if align == 'left':
+        if len(s) > length:
+            return s[:length]
+        else:
+            return s + (pad_char * (length - len(s)))
+    elif align == 'right':
+        if len(s) > length:
+            return s[len(s) - length:]
+        else:
+            return (pad_char * (length - len(s))) + s
+    else:
+        raise ValueError("Unknown align type, expected either 'left' or 'right'")
+
+
+def pad_title(value):
+    """Pad a title to 30 characters to force the 'details' view."""
+    return str_pad(value, 30, pad_char=' ')
+
+
+def get_plex_item_display_title(item, kind, parent=None, parent_title=None, section_title=None,
+                                add_section_title=False):
+    """
+    :param item: plex item
+    :param kind: show or movie
+    :param parent: season or None
+    :param parent_title: parentTitle or None
+    :return:
+    """
+    return get_video_display_title(kind, item.title,
+                                   section_title=(
+                                       section_title or (parent.section.title if parent and getattr(parent, "section")
+                                                         else None)),
+                                   parent_title=(parent_title or (parent.show.title if parent else None)),
+                                   season=parent.index if parent else None,
+                                   episode=item.index if kind == "show" else None,
+                                   add_section_title=add_section_title)
+
+
+def get_video_display_title(kind, title, section_title=None, parent_title=None, season=None, episode=None,
+                            add_section_title=False):
+    section_add = ""
+    if add_section_title:
+        section_add = ("%s: " % section_title) if section_title else ""
+
+    if kind == "show" and parent_title:
+        if season and episode:
+            return '%s%s S%02dE%02d%s' % (section_add, parent_title, season or 0, episode or 0,
+                                          (", %s" % title if title else ""))
+        return '%s%s%s' % (section_add, parent_title, (", %s" % title if title else ""))
+    return "%s%s" % (section_add, title)
+
+
+def get_title_for_video_metadata(metadata, add_section_title=True, add_episode_title=False):
+    """
+
+    :param metadata:
+    :param add_section_title:
+    :param add_episode_title: add the episode's title if its an episode else always add title
+    :return:
+    """
+    # compute item title
+    add_title = (add_episode_title and metadata["series_id"]) or not metadata["series_id"]
+    return get_video_display_title(
+        "show" if metadata["series_id"] else "movie",
+        metadata["title"] if add_title else "",
+        parent_title=metadata.get("series", None),
+        season=metadata.get("season", None),
+        episode=metadata.get("episode", None),
+        section_title=metadata.get("section", None),
+        add_section_title=add_section_title
+    )
+
+
+def get_identifier():
+    identifier = None
+    try:
+        identifier = Platform.MachineIdentifier
+    except:
+        pass
+
+    if not identifier:
+        identifier = String.UUID()
+
+    return Hash.SHA1(identifier + "SUBZEROOOOOOOOOO")
+
+
+def encode_message(base, s):
+    return "%s?message=%s" % (base, urllib.quote_plus(s))
+
+
+def decode_message(s):
+    return urllib.unquote_plus(s)
+
+
+def timestamp():
+    return int(time.time())
+
+
+def df(d):
+    return d.strftime("%Y-%m-%d %H:%M:%S") if d else "legacy data"
+
+
+def query_plex(url, args):
+    """
+    simple http query to the plex API without parsing anything too complicated
+    :param url:
+    :param args:
+    :return:
+    """
+    use_args = args.copy()
+
+    computed_args = "&".join(["%s=%s" % (key, String.Quote(value)) for key, value in use_args.iteritems()])
+
+    return HTTP.Request(url + ("?%s" % computed_args) if computed_args else "", immediate=True)
+
+
+def check_write_permissions(path):
+    if platform.system() == "Windows":
+        # physical access check
+        check_path = os.path.join(os.path.realpath(path), ".sz_perm_chk")
+        try:
+            if os.path.exists(check_path):
+                os.rmdir(check_path)
+            os.mkdir(check_path)
+            os.rmdir(check_path)
+            return True
+        except OSError:
+            pass
+
+    else:
+        # os.access check
+        return os.access(path, os.W_OK | os.X_OK)
+    return False
+
+
+def get_item_hints(title, kind, series=None):
+    hints = {"expected_title": [title]}
+    hints.update({"type": "episode", "expected_series": [series]} if kind == "series" else {"type": "movie"})
+    return hints
+
+
+def notify_executable(exe_info, videos, subtitles, storage):
+    variables = (
+        "subtitle_language", "subtitle_path", "subtitle_filename", "provider", "score", "storage", "series_id",
+        "series", "title", "section", "filename", "path", "folder", "season_id", "type", "id", "season"
+    )
+    exe, arguments = exe_info
+    for video, video_subtitles in subtitles.items():
+        for subtitle in video_subtitles:
+            lang = Locale.Language.Match(subtitle.language.alpha2)
+            data = video.plexapi_metadata.copy()
+            data.update({
+                "subtitle_language": lang,
+                "provider": subtitle.provider_name,
+                "score": subtitle.score,
+                "storage": storage,
+                "subtitle_path": subtitle.storage_path,
+                "subtitle_filename": os.path.basename(subtitle.storage_path)
+            })
+
+            # fill missing data with None
+            prepared_data = dict((v, data.get(v)) for v in variables)
+
+            prepared_arguments = [arg % prepared_data for arg in arguments]
+
+            Log.Debug(u"Calling %s with arguments: %s" % (exe, prepared_arguments))
+            try:
+                output = subprocess.check_output(subprocess.list2cmdline([exe] + prepared_arguments),
+                                                 stderr=subprocess.STDOUT, shell=True)
+            except subprocess.CalledProcessError:
+                Log.Error(u"Calling %s failed: %s" % (exe, traceback.format_exc()))
+            else:
+                Log.Debug(u"Process output: %s" % output)
+
+
+def track_usage(category=None, action=None, label=None, value=None):
+    if not cast_bool(Prefs["track_usage"]):
+        return
+
+    Thread.Create(dispatch_track_usage, category, action, label, value,
+                  identifier=Dict["anon_id"], first_use=Dict["first_use"],
+                  add=Network.PublicAddress)
+
+
+def dispatch_track_usage(*args, **kwargs):
+    identifier = kwargs.pop("identifier")
+    first_use = kwargs.pop("first_use")
+    add = kwargs.pop("add")
+    try:
+        track_event(identifier=identifier, first_use=first_use, add=add, *[str(a) for a in args])
+    except:
+        Log.Debug("Something went wrong when reporting anonymous user statistics: %s", traceback.format_exc())
+
+
+def get_language(lang_short):
+    return Language.fromietf(lang_short)

Contents/Code/support/history.py

@@ -0,0 +1,4 @@
+# coding=utf-8
+from subzero.history_storage import SubtitleHistory
+
+get_history = lambda: SubtitleHistory(Data, int(Prefs["history_size"]))

Contents/Code/support/ignore.py

@@ -0,0 +1,62 @@
+# coding=utf-8
+
+from subzero.lib.dict import DictProxy
+
+
+class IgnoreDict(DictProxy):
+    store = "ignore"
+
+    # single item keys returned by helpers.items.getItems mapped to their parents
+    translate_keys = {
+        "section": "sections",
+        "show": "series",
+        "movie": "videos",
+        "episode": "videos"
+    }
+
+    # getItems types mapped to their verbose names
+    keys_verbose = {
+        "sections": "Section",
+        "series": "Series",
+        "videos": "Item",
+    }
+
+    key_order = ("sections", "series", "videos")
+
+    def __len__(self):
+        try:
+            return sum(len(self.Dict[self.store][key]) for key in self.key_order)
+        except KeyError:
+            # old version
+            self.Dict[self.store] = self.setup_defaults()
+        return 0
+
+    def translate_key(self, name):
+        return self.translate_keys.get(name)
+
+    def verbose(self, name):
+        return self.keys_verbose.get(name)
+
+    def get_title_key(self, kind, key):
+        return "%s_%s" % (kind, key)
+
+    def add_title(self, kind, key, title):
+        self["titles"][self.get_title_key(kind, key)] = title
+
+    def remove_title(self, kind, key):
+        title_key = self.get_title_key(kind, key)
+        if title_key in self.titles:
+            del self.titles[title_key]
+
+    def get_title(self, kind, key):
+        title_key = self.get_title_key(kind, key)
+        if title_key in self.titles:
+            return self.titles[title_key]
+
+    def save(self):
+        Dict.Save()
+
+    def setup_defaults(self):
+        return {"sections": [], "series": [], "videos": [], "titles": {}}
+
+ignore_list = IgnoreDict(Dict)

Contents/Code/support/items.py

@@ -0,0 +1,285 @@
+# coding=utf-8
+
+import logging
+import re
+import types
+import os
+from ignore import ignore_list
+from helpers import is_recent, get_plex_item_display_title, query_plex
+from lib import Plex, get_intent
+from config import config, IGNORE_FN
+
+logger = logging.getLogger(__name__)
+
+MI_KIND, MI_TITLE, MI_KEY, MI_DEEPER, MI_ITEM = 0, 1, 2, 3, 4
+
+container_size_re = re.compile(ur'totalSize="(\d+)"')
+
+
+def get_item(key):
+    item_id = int(key)
+    item_container = Plex["library"].metadata(item_id)
+
+    try:
+        return list(item_container)[0]
+    except IndexError:
+        pass
+
+
+def get_item_kind(item):
+    return type(item).__name__
+
+
+PLEX_API_TYPE_MAP = {
+    "Show": "series",
+    "Season": "season",
+    "Episode": "episode",
+    "Movie": "movie",
+}
+
+
+def get_item_kind_from_rating_key(key):
+    item = get_item(key)
+    return PLEX_API_TYPE_MAP[get_item_kind(item)]
+
+
+def get_item_kind_from_item(item):
+    return PLEX_API_TYPE_MAP[get_item_kind(item)]
+
+
+def get_item_thumb(item):
+    kind = get_item_kind(item)
+    if kind == "Episode":
+        return item.show.thumb
+    elif kind == "Section":
+        return item.art
+    return item.thumb
+
+
+def get_items_info(items):
+    return items[0][MI_KIND], items[0][MI_DEEPER]
+
+
+def get_kind(items):
+    return items[0][MI_KIND]
+
+
+def get_section_size(key):
+    """
+    quick query to determine the section size
+    :param key:
+    :return:
+    """
+    size = None
+    url = "http://127.0.0.1:32400/library/sections/%s/all" % int(key)
+    use_args = {
+        "X-Plex-Container-Size": "0",
+        "X-Plex-Container-Start": "0"
+    }
+    response = query_plex(url, use_args)
+    matches = container_size_re.findall(response.content)
+    if matches:
+        size = int(matches[0])
+
+    return size
+
+
+def get_items(key="recently_added", base="library", value=None, flat=False, add_section_title=False):
+    """
+    try to handle all return types plex throws at us and return a generalized item tuple
+    """
+    items = []
+    apply_value = None
+    if value:
+        if isinstance(value, types.ListType):
+            apply_value = value
+        else:
+            apply_value = [value]
+    result = getattr(Plex[base], key)(*(apply_value or []))
+
+    for item in result:
+        cls = getattr(getattr(item, "__class__"), "__name__")
+        if hasattr(item, "scanner"):
+            kind = "section"
+        elif cls == "Directory":
+            kind = "directory"
+        else:
+            kind = item.type
+
+        # only return items for our enabled sections
+        section_key = None
+        if kind == "section":
+            section_key = item.key
+        else:
+            if hasattr(item, "section_key"):
+                section_key = getattr(item, "section_key")
+
+        if section_key and section_key not in config.enabled_sections:
+            continue
+
+        if kind == "season":
+            # fixme: i think this case is unused now
+            if flat:
+                # return episodes
+                for child in item.children():
+                    items.append(("episode", get_plex_item_display_title(child, "show", parent=item, add_section_title=add_section_title), int(item.rating_key),
+                                  False, child))
+            else:
+                # return seasons
+                items.append(("season", item.title, int(item.rating_key), True, item))
+
+        elif kind == "directory":
+            items.append(("directory", item.title, item.key, True, item))
+
+        elif kind == "section":
+            if item.type in ['movie', 'show']:
+                item.size = get_section_size(item.key)
+                items.append(("section", item.title, int(item.key), True, item))
+
+        elif kind == "episode":
+            items.append(
+                (kind, get_plex_item_display_title(item, "show", parent=item.season, parent_title=item.show.title, section_title=item.section.title,
+                                                   add_section_title=add_section_title), int(item.rating_key), False, item))
+
+        elif kind in ("movie", "artist", "photo"):
+            items.append((kind, get_plex_item_display_title(item, kind, section_title=item.section.title, add_section_title=add_section_title),
+                          int(item.rating_key), False, item))
+
+        elif kind == "show":
+            items.append((
+                kind, get_plex_item_display_title(item, kind, section_title=item.section.title, add_section_title=add_section_title), int(item.rating_key), True,
+                item))
+
+    return items
+
+
+def get_recent_items():
+    """
+    actually get the recent items, not limited like /library/recentlyAdded
+    :return:
+    """
+    args = {
+        "sort": "addedAt:desc",
+        "X-Plex-Container-Start": "0",
+        "X-Plex-Container-Size": "%s" % config.max_recent_items_per_library
+    }
+
+    episode_re = re.compile(ur'ratingKey="(?P<key>\d+)"'
+                            ur'.+?grandparentRatingKey="(?P<parent_key>\d+)"'
+                            ur'.+?title="(?P<title>.*?)"'
+                            ur'.+?grandparentTitle="(?P<parent_title>.*?)"'
+                            ur'.+?index="(?P<episode>\d+?)"'
+                            ur'.+?parentIndex="(?P<season>\d+?)".+?addedAt="(?P<added>\d+)"')
+    movie_re = re.compile(ur'ratingKey="(?P<key>\d+)".+?title="(?P<title>.*?)".+?addedAt="(?P<added>\d+)"')
+    available_keys = ("key", "title", "parent_key", "parent_title", "season", "episode", "added")
+    recent = []
+
+    for section in Plex["library"].sections():
+        if section.type not in ("movie", "show") \
+                or section.key not in config.enabled_sections \
+                or section.key in ignore_list.sections:
+            Log.Debug(u"Skipping section: %s" % section.title)
+            continue
+
+        use_args = args.copy()
+        if section.type == "show":
+            use_args["type"] = "4"
+
+        url = "http://127.0.0.1:32400/library/sections/%s/all" % int(section.key)
+        response = query_plex(url, use_args)
+
+        matcher = episode_re if section.type == "show" else movie_re
+        matches = [m.groupdict() for m in matcher.finditer(response.content)]
+        for match in matches:
+            data = dict((key, match[key] if key in match else None) for key in available_keys)
+            if section.type == "show" and data["parent_key"] in ignore_list.series:
+                Log.Debug(u"Skipping series: %s" % data["parent_title"])
+                continue
+            if data["key"] in ignore_list.videos:
+                Log.Debug(u"Skipping item: %s" % data["title"])
+                continue
+            if is_recent(int(data["added"])):
+                recent.append((int(data["added"]), section.type, section.title, data["key"]))
+
+    return recent
+
+
+def get_on_deck_items():
+    return get_items(key="on_deck", add_section_title=True)
+
+
+def get_recently_added_items():
+    return get_items(key="recently_added", add_section_title=True, flat=False)
+
+
+def get_all_items(key, base="library", value=None, flat=False):
+    return get_items(key, base=base, value=value, flat=flat)
+
+
+def is_ignored(rating_key, item=None):
+    """
+    check whether an item, its show/season/section is in the soft or the hard ignore list
+    :param rating_key:
+    :param item:
+    :return:
+    """
+    # item in soft ignore list
+    if rating_key in ignore_list["videos"]:
+        Log.Debug("Item %s is in the soft ignore list" % rating_key)
+        return True
+
+    item = item or get_item(rating_key)
+    kind = get_item_kind(item)
+
+    # show in soft ignore list
+    if kind == "Episode" and item.show.rating_key in ignore_list["series"]:
+        Log.Debug("Item %s's show is in the soft ignore list" % rating_key)
+        return True
+
+    # section in soft ignore list
+    if item.section.key in ignore_list["sections"]:
+        Log.Debug("Item %s's section is in the soft ignore list" % rating_key)
+        return True
+
+    # physical/path ignore
+    if config.ignore_sz_files or config.ignore_paths:
+        # normally check current item folder and the library
+        check_ignore_paths = [".", "../"]
+        if kind == "Episode":
+            # series/episode, we've got a season folder here, also
+            check_ignore_paths.append("../../")
+
+        for part in item.media.parts:
+            if config.ignore_paths and config.is_path_ignored(part.file):
+                Log.Debug("Item %s's path is manually ignored" % rating_key)
+                return True
+
+            if config.ignore_sz_files:
+                for sub_path in check_ignore_paths:
+                    if config.is_physically_ignored(os.path.abspath(os.path.join(os.path.dirname(part.file), sub_path))):
+                        Log.Debug("An ignore file exists in either the items or its parent folders")
+                        return True
+
+    return False
+
+
+def refresh_item(rating_key, force=False, timeout=8000, refresh_kind=None, parent_rating_key=None):
+    intent = get_intent()
+
+    # timeout actually is the time for which the intent will be valid
+    if force:
+        Log.Debug("Setting intent for force-refresh of %s to timeout: %s", rating_key, timeout)
+        intent.set("force", rating_key, timeout=timeout)
+
+        # force Dict.Save()
+        intent.store.save()
+
+    refresh = [rating_key]
+
+    if refresh_kind == "season":
+        # season refresh, needs explicit per-episode refresh
+        refresh = [item.rating_key for item in list(Plex["library/metadata"].children(int(rating_key)))]
+
+    for key in refresh:
+        Log.Info("%s item %s", "Refreshing" if not force else "Forced-refreshing", key)
+        Plex["library/metadata"].refresh(key)

Contents/Code/support/lib.py

@@ -0,0 +1,56 @@
+# coding=utf-8
+
+import plex
+from subzero.intent import TempIntent
+from subzero.lib.dict import DictProxy
+from subzero.lib.httpfake import PlexPyNativeResponseProxy
+from subzero.constants import DEFAULT_TIMEOUT
+
+
+class PlexPyNativeRequestProxy(object):
+    """
+    A really dumb object that tries to mimic requests.Request in an incomplete way, so that plex.Plex
+    uses native plex HTTPRequests instead of the better requests.Request class.
+
+    This allows us to operate freely on 127.0.0.1's PMS.
+
+    To be used in conjunction with subzero.lib.httpfake.PlexPyNativeResponseProxy
+    """
+    url = None
+    data = None
+    headers = None
+    method = None
+
+    def prepare(self):
+        return self
+
+    def send(self):
+        # fixme: add self.data to HTTP.Request
+        data = None
+        status_code = 200
+        try:
+            data = HTTP.Request(self.url, headers=self.headers, immediate=True, method=self.method,
+                                timeout=DEFAULT_TIMEOUT)
+        except Ex.HTTPError as e:
+            status_code = e.code
+        return PlexPyNativeResponseProxy(data, status_code, self)
+
+
+plex.request.Request = PlexPyNativeRequestProxy
+
+Plex = plex.Plex
+
+
+class IntentDictStorage(DictProxy):
+    store = "intent"
+
+    def setup_defaults(self):
+        return {"force": {}}
+
+
+def get_intent():
+    """
+    use this to get an intent from inside a separate thread
+    :return:
+    """
+    return TempIntent(store=IntentDictStorage(Dict))

Contents/Code/support/localmedia.py

@@ -0,0 +1,193 @@
+# coding=utf-8
+
+import os
+
+import config
+import helpers
+import subtitlehelpers
+
+from config import config as sz_config
+
+
+def find_subtitles(part):
+    lang_sub_map = {}
+    part_filename = helpers.unicodize(part.file)
+    part_basename = os.path.splitext(os.path.basename(part_filename))[0]
+    use_filesystem = helpers.cast_bool(Prefs["subtitles.save.filesystem"])
+    paths = [os.path.dirname(part_filename)] if use_filesystem else []
+
+    global_subtitle_folder = None
+
+    global_folders = []
+
+    if use_filesystem:
+        # Check for local subtitles subdirectory
+        sub_dir_base = paths[0]
+
+        sub_dir_list = []
+
+        if Prefs["subtitles.save.subFolder"] != "current folder":
+            # got selected subfolder
+            sub_dir_list.append(os.path.join(sub_dir_base, Prefs["subtitles.save.subFolder"]))
+
+        sub_dir_custom = Prefs["subtitles.save.subFolder.Custom"].strip() \
+            if Prefs["subtitles.save.subFolder.Custom"] else None
+
+        if sub_dir_custom:
+            # got custom subfolder
+            sub_dir_custom = os.path.normpath(sub_dir_custom)
+            if os.path.isdir(sub_dir_custom) and os.path.isabs(sub_dir_custom):
+                # absolute folder
+                sub_dir_list.append(sub_dir_custom)
+                global_folders.append(sub_dir_custom)
+            else:
+                # relative folder
+                fld = os.path.join(sub_dir_base, sub_dir_custom)
+                sub_dir_list.append(fld)
+
+        for sub_dir in sub_dir_list:
+            if os.path.isdir(sub_dir):
+                paths.append(sub_dir)
+
+        # Check for a global subtitle location
+        global_subtitle_folder = os.path.join(Core.app_support_path, 'Subtitles')
+        if os.path.exists(global_subtitle_folder):
+            paths.append(global_subtitle_folder)
+            global_folders.append(global_subtitle_folder)
+
+    # normalize all paths
+    paths = [os.path.normpath(os.path.realpath(helpers.unicodize(path))) for path in paths]
+
+    # We start by building a dictionary of files to their absolute paths. We also need to know
+    # the number of media files that are actually present, in case the found local media asset
+    # is limited to a single instance per media file.
+    #
+    file_paths = {}
+    total_media_files = 0
+    media_files = []
+    for path in paths:
+        for file_path_listing in os.listdir(path.encode(sz_config.fs_encoding)):
+            # When using os.listdir with a unicode path, it will always return a string using the
+            # NFD form. However, we internally are using the form NFC and therefore need to convert
+            # it to allow correct regex / comparisons to be performed.
+            #
+            file_path_listing = helpers.unicodize(file_path_listing)
+            if os.path.isfile(os.path.join(path, file_path_listing).encode(sz_config.fs_encoding)):
+                file_paths[file_path_listing.lower()] = os.path.join(path, file_path_listing)
+
+            # If we've found an actual media file, we should record it.
+            (root, ext) = os.path.splitext(file_path_listing)
+            if ext.lower()[1:] in config.VIDEO_EXTS:
+                total_media_files += 1
+
+                # collect found media files
+                media_files.append(root)
+
+    # cleanup any leftover subtitle if no associated media file was found
+    if helpers.cast_bool(Prefs["subtitles.autoclean"]):
+        for path in paths:
+            # we can't housekeep the global subtitle folders as we don't know about *all* media files
+            # in a library; skip them
+            skip_path = False
+            for fld in global_folders:
+                if path.startswith(fld):
+                    Log.Info("Skipping housekeeping of folder: %s", path)
+                    skip_path = True
+                    break
+
+            if skip_path:
+                continue
+
+            for file_path_listing in os.listdir(path.encode(sz_config.fs_encoding)):
+                file_path_listing = helpers.unicodize(file_path_listing)
+                enc_fn = os.path.join(path, file_path_listing).encode(sz_config.fs_encoding)
+
+                if os.path.isfile(enc_fn):
+                    (root, ext) = os.path.splitext(file_path_listing)
+                    # it's a subtitle file
+                    if ext.lower()[1:] in config.SUBTITLE_EXTS:
+                        # get fn without forced/default/normal tag
+                        split_tag = root.rsplit(".", 1)
+                        if len(split_tag) > 1 and split_tag[1].lower() in ['forced', 'normal', 'default']:
+                            root = split_tag[0]
+
+                        # get associated media file name without language
+                        sub_fn = subtitlehelpers.ENDSWITH_LANGUAGECODE_RE.sub("", root)
+
+                        # subtitle basename and basename without possible language tag  not found in collected
+                        # media files? kill.
+                        if root not in media_files and sub_fn not in media_files:
+                            Log.Info("Removing leftover subtitle: %s", os.path.join(path, file_path_listing))
+                            try:
+                                os.remove(enc_fn)
+                            except (OSError, IOError):
+                                Log.Error("Removing failed")
+
+    Log('Looking for subtitle media in %d paths with %d media files.', len(paths), total_media_files)
+    Log('Paths: %s', ", ".join([helpers.unicodize(p) for p in paths]))
+
+    for file_path in file_paths.values():
+        local_filename = os.path.basename(file_path)
+        bn, ext = os.path.splitext(local_filename)
+        local_basename = helpers.unicodize(bn)
+
+        # get fn without forced/default/normal tag
+        split_tag = local_basename.rsplit(".", 1)
+        if len(split_tag) > 1 and split_tag[1].lower() in ['forced', 'normal', 'default']:
+            local_basename = split_tag[0]
+
+        # split off possible language tag
+        local_basename2 = local_basename.rsplit('.', 1)[0]
+        filename_matches_part = local_basename == part_basename or local_basename2 == part_basename
+        filename_contains_part = part_basename in local_basename
+
+        if not ext.lower()[1:] in config.SUBTITLE_EXTS:
+            continue
+
+        # if the file is located within the global subtitle folders and its name doesn't match exactly, ignore it
+        if global_folders and not filename_matches_part:
+            skip_path = False
+            for fld in global_folders:
+                if file_path.startswith(fld):
+                    skip_path = True
+                    break
+
+            if skip_path:
+                continue
+
+        # determine whether to pick up the subtitle based on our match strictness
+        elif not filename_matches_part:
+            if sz_config.ext_match_strictness == "strict" or (
+                    sz_config.ext_match_strictness == "loose" and not filename_contains_part):
+
+                Log.Debug("%s doesn't match %s, skipping" % (helpers.unicodize(local_filename),
+                                                             helpers.unicodize(part_basename)))
+                continue
+
+        subtitle_helper = subtitlehelpers.subtitle_helpers(file_path)
+        if subtitle_helper is not None:
+            local_lang_map = subtitle_helper.process_subtitles(part)
+            for new_language, subtitles in local_lang_map.items():
+
+                # Add the possible new language along with the located subtitles so that we can validate them
+                # at the end...
+                #
+                if not lang_sub_map.has_key(new_language):
+                    lang_sub_map[new_language] = []
+                lang_sub_map[new_language] = lang_sub_map[new_language] + subtitles
+
+    # add known metadata subs to our sub list
+    if not use_filesystem:
+        for language, sub_list in subtitlehelpers.get_subtitles_from_metadata(part).iteritems():
+            if sub_list:
+                if language not in lang_sub_map:
+                    lang_sub_map[language] = []
+                lang_sub_map[language] = lang_sub_map[language] + sub_list
+
+    # Now whack subtitles that don't exist anymore.
+    for language in lang_sub_map.keys():
+        part.subtitles[language].validate_keys(lang_sub_map[language])
+
+    # Now whack the languages that don't exist anymore.
+    for language in list(set(part.subtitles.keys()) - set(lang_sub_map.keys())):
+        part.subtitles[language].validate_keys({})

Contents/Code/support/missing_subtitles.py

@@ -0,0 +1,77 @@
+# coding=utf-8
+import traceback
+
+from support.config import config
+from support.helpers import get_plex_item_display_title, cast_bool
+from support.items import get_item
+from support.lib import Plex
+
+
+def item_discover_missing_subs(rating_key, kind="show", added_at=None, section_title=None, internal=False, external=True, languages=()):
+    existing_subs = {"internal": [], "external": [], "count": 0}
+
+    item_id = int(rating_key)
+    item = get_item(rating_key)
+
+    if kind == "show":
+        item_title = get_plex_item_display_title(item, kind, parent=item.season, section_title=section_title, parent_title=item.show.title)
+    else:
+        item_title = get_plex_item_display_title(item, kind, section_title=section_title)
+
+    video = item.media
+
+    for part in video.parts:
+        for stream in part.streams:
+            if stream.stream_type == 3:
+                if stream.index:
+                    key = "internal"
+                else:
+                    key = "external"
+
+                existing_subs[key].append(Locale.Language.Match(stream.language_code or ""))
+                existing_subs["count"] = existing_subs["count"] + 1
+
+    missing = languages
+    if existing_subs["count"]:
+        existing_flat = (existing_subs["internal"] if internal else []) + (existing_subs["external"] if external else [])
+        languages_set = set(languages)
+        if languages_set.issubset(existing_flat) or (len(existing_flat) >= 1 and Prefs['subtitles.only_one']):
+            # all subs found
+            Log.Info(u"All subtitles exist for '%s'", item_title)
+            return
+
+        missing = languages_set - set(existing_flat)
+        Log.Info(u"Subs still missing for '%s': %s", item_title, missing)
+
+    if missing:
+        return added_at, item_id, item_title, item, missing
+
+
+def items_get_all_missing_subs(items):
+    missing = []
+    for added_at, kind, section_title, key in items:
+        try:
+            state = item_discover_missing_subs(
+                key,
+                kind=kind,
+                added_at=added_at,
+                section_title=section_title,
+                languages=config.lang_list,
+                internal=cast_bool(Prefs["subtitles.scan.embedded"]),
+                external=cast_bool(Prefs["subtitles.scan.external"])
+            )
+            if state:
+                # (added_at, item_id, title, item, missing_languages)
+                missing.append(state)
+        except:
+            Log.Error("Something went wrong when getting the state of item %s: %s", key, traceback.format_exc())
+    return missing
+
+
+def refresh_item(item):
+    Plex["library/metadata"].refresh(item)
+
+
+def refresh_items(items):
+    for item, title in items:
+        refresh_item(item)

Contents/Code/support/plex_media.py

@@ -0,0 +1,244 @@
+# coding=utf-8
+
+import os
+
+import subliminal
+import helpers
+
+from items import get_item
+from lib import get_intent, Plex
+from config import config
+
+
+def get_metadata_dict(item, part, add):
+    data = {
+        "item": item,
+        "section": item.section.title,
+        "path": part.file,
+        "folder": os.path.dirname(part.file),
+        "filename": os.path.basename(part.file)
+    }
+    data.update(add)
+    return data
+
+
+def media_to_videos(media, kind="series"):
+    """
+    iterates through media and returns the associated parts (videos)
+    :param media:
+    :param kind:
+    :return:
+    """
+    videos = []
+
+    if kind == "series":
+        for season in media.seasons:
+            season_object = media.seasons[season]
+            for episode in media.seasons[season].episodes:
+                ep = media.seasons[season].episodes[episode]
+
+                # get plex item via API for additional metadata
+                plex_episode = get_item(ep.id)
+
+                for item in media.seasons[season].episodes[episode].items:
+                    for part in item.parts:
+                        videos.append(
+                            get_metadata_dict(plex_episode, part,
+                                              {"plex_part": part, "type": "episode", "title": ep.title,
+                                               "series": media.title, "id": ep.id,
+                                               "series_id": media.id, "season_id": season_object.id,
+                                               "episode": plex_episode.index, "season": plex_episode.season.index,
+                                               "section": plex_episode.section.title
+                                               })
+                        )
+    else:
+        plex_item = get_item(media.id)
+        for item in media.items:
+            for part in item.parts:
+                videos.append(
+                    get_metadata_dict(plex_item, part, {"plex_part": part, "type": "movie",
+                                                        "title": media.title, "id": media.id,
+                                                        "series_id": None,
+                                                        "season_id": None,
+                                                        "section": plex_item.section.title})
+                )
+    return videos
+
+
+IGNORE_FN = ("subzero.ignore", ".subzero.ignore", ".nosz")
+
+
+def get_stream_fps(streams):
+    """
+    accepts a list of plex streams or a list of the plex api streams
+    """
+    for stream in streams:
+        # video
+        stream_type = getattr(stream, "type", getattr(stream, "stream_type", None))
+        if stream_type == 1:
+            return getattr(stream, "frameRate", getattr(stream, "frame_rate", "25.000"))
+    return "25.000"
+
+
+def get_media_item_ids(media, kind="series"):
+    ids = []
+    if kind == "movies":
+        ids.append(media.id)
+    else:
+        for season in media.seasons:
+            for episode in media.seasons[season].episodes:
+                ids.append(media.seasons[season].episodes[episode].id)
+
+    return ids
+
+
+def scan_video(plex_part, ignore_all=False, hints=None, rating_key=None):
+    embedded_subtitles = not ignore_all and Prefs['subtitles.scan.embedded']
+    external_subtitles = not ignore_all and Prefs['subtitles.scan.external']
+
+    if ignore_all:
+        Log.Debug("Force refresh intended.")
+
+    Log.Debug("Scanning video: %s, subtitles=%s, embedded_subtitles=%s" % (plex_part.file, external_subtitles, embedded_subtitles))
+
+    known_embedded = []
+    parts = list(Plex["library"].metadata(rating_key))[0].media.parts
+    plexpy_part = None
+    for part in parts:
+        if int(part.id) == int(plex_part.id):
+            plexpy_part = part
+
+    # embedded subtitles
+    if plexpy_part:
+        for stream in plexpy_part.streams:
+            if stream.stream_type == 3:
+                if (config.forced_only and getattr(stream, "forced")) or \
+                        (not config.forced_only and not getattr(stream, "forced")):
+                    if not stream.stream_key and stream.codec in ("srt", "ass", "ssa"):
+                        lang_code = stream.language_code
+
+                        # treat unknown language as lang1?
+                        if not lang_code and config.treat_und_as_first:
+                            lang_code = list(config.lang_list)[0].alpha3
+                        known_embedded.append(lang_code)
+    else:
+        Log.Warn("Part %s missing of %s, not able to scan internal streams", plex_part.id, rating_key)
+
+    try:
+        return subliminal.video.scan_video(plex_part.file, subtitles=external_subtitles,
+                                           embedded_subtitles=embedded_subtitles, hints=hints or {},
+                                           video_fps=plex_part.fps, forced_tag=config.forced_only,
+                                           known_embedded_subtitle_streams=known_embedded)
+
+    except ValueError:
+        Log.Warn("File could not be guessed by subliminal")
+
+
+def scan_videos(videos, kind="series", ignore_all=False):
+    """
+    receives a list of videos containing dictionaries returned by media_to_videos
+    :param videos:
+    :param kind: series or movies
+    :return: dictionary of subliminal.video.scan_video, key=subliminal scanned video, value=plex file part
+    """
+    ret = {}
+    for video in videos:
+        intent = get_intent()
+        force_refresh = intent.get("force", video["id"], video["series_id"], video["season_id"])
+        Log.Debug("Determining force-refresh (video: %s, series: %s, season: %s), result: %s"
+                  % (video["id"], video["series_id"], video["season_id"], force_refresh))
+
+        hints = helpers.get_item_hints(video["title"], kind, series=video["series"] if kind == "series" else None)
+        video["plex_part"].fps = get_stream_fps(video["plex_part"].streams)
+        scanned_video = scan_video(video["plex_part"], ignore_all=force_refresh or ignore_all, hints=hints,
+                                   rating_key=video["id"])
+
+        if not scanned_video:
+            continue
+
+        scanned_video.id = video["id"]
+        part_metadata = video.copy()
+        del part_metadata["plex_part"]
+        scanned_video.plexapi_metadata = part_metadata
+        ret[scanned_video] = video["plex_part"]
+    return ret
+
+
+class PartUnknownException(Exception):
+    pass
+
+
+def get_plex_metadata(rating_key, part_id, item_type):
+    """
+    uses the Plex 3rd party API accessor to get metadata information
+
+    :param rating_key:
+    :param part_id:
+    :param item_type:
+    :return:
+    """
+
+    plex_item = list(Plex["library"].metadata(rating_key))[0]
+
+    # find current part
+    current_part = None
+    for part in plex_item.media.parts:
+        if str(part.id) == part_id:
+            current_part = part
+
+    if not current_part:
+        raise PartUnknownException("Part unknown")
+
+    # get normalized metadata
+    if item_type == "episode":
+        metadata = get_metadata_dict(plex_item, current_part,
+                                     {"plex_part": current_part, "type": "episode", "title": plex_item.title,
+                                      "series": plex_item.show.title, "id": plex_item.rating_key,
+                                      "series_id": plex_item.show.rating_key,
+                                      "season_id": plex_item.season.rating_key,
+                                      "season": plex_item.season.index,
+                                      "episode": plex_item.index
+                                      })
+    else:
+        metadata = get_metadata_dict(plex_item, current_part, {"plex_part": current_part, "type": "movie",
+                                                               "title": plex_item.title, "id": plex_item.rating_key,
+                                                               "series_id": None,
+                                                               "season_id": None,
+                                                               "season": None,
+                                                               "episode": None,
+                                                               "section": plex_item.section.title})
+    return metadata
+
+
+class PMSMediaProxy(object):
+    """
+    Proxy object for getting data from a mediatree items "internally" via the PMS
+
+    note: this could be useful later on: Media.TV_Show(getattr(Metadata, "_access_point"), id=XXXXXX)
+    """
+
+    def __init__(self, media_id):
+        self.mediatree = Media.TreeForDatabaseID(media_id)
+
+    def get_part(self, part_id=None):
+        """
+        walk the mediatree until the given part was found; if no part was given, return the first one
+        :param part_id:
+        :return:
+        """
+        m = self.mediatree
+        while 1:
+            if m.items:
+                media_item = m.items[0]
+                if not part_id:
+                    return media_item.parts[0] if media_item.parts else None
+
+                for part in media_item.parts:
+                    if str(part.id) == str(part_id):
+                        return part
+                break
+
+            if not m.children:
+                break
+
+            m = m.children[0]

Contents/Code/support/storage.py

@@ -0,0 +1,186 @@
+# coding=utf-8
+
+import datetime
+import os
+import pprint
+import copy
+
+import subliminal
+from items import get_item
+from subzero.subtitle_storage import StoredSubtitlesManager
+
+from subtitlehelpers import force_utf8
+from config import config
+from helpers import notify_executable, get_title_for_video_metadata, cast_bool, force_unicode
+from plex_media import PMSMediaProxy
+
+
+get_subtitle_storage = lambda: StoredSubtitlesManager(Data, get_item)
+
+
+def whack_missing_parts(scanned_video_part_map, existing_parts=None):
+    """
+    cleans out our internal storage's video parts (parts may get updated/deleted/whatever)
+    :param existing_parts: optional list of part ids known
+    :param scanned_video_part_map: videos to check for
+    :return:
+    """
+    # shortcut
+
+    if "subs" not in Dict:
+        return
+
+    if not existing_parts:
+        existing_parts = []
+        for part in scanned_video_part_map.viewvalues():
+            existing_parts.append(str(part.id))
+
+    whacked_parts = False
+    for video in scanned_video_part_map.keys():
+        video_id = str(video.id)
+        if video_id not in Dict["subs"]:
+            continue
+
+        parts = Dict["subs"][video_id].keys()
+
+        for part_id in parts:
+            part_id = str(part_id)
+            if part_id not in existing_parts:
+                Log.Info("Whacking part %s in internal storage of video %s (%s, %s)", part_id, video_id,
+                         repr(existing_parts), repr(parts))
+                del Dict["subs"][video_id][part_id]
+                whacked_parts = True
+
+    if whacked_parts:
+        Dict.Save()
+
+
+def store_subtitle_info(scanned_video_part_map, downloaded_subtitles, storage_type, mode="a"):
+    """
+    stores information about downloaded subtitles in plex's Dict()
+    """
+    existing_parts = []
+    for video, video_subtitles in downloaded_subtitles.items():
+        part = scanned_video_part_map[video]
+        part_id = str(part.id)
+        video_id = str(video.id)
+        plex_item = get_item(video_id)
+        metadata = video.plexapi_metadata
+        title = get_title_for_video_metadata(metadata)
+
+        subtitle_storage = get_subtitle_storage()
+        stored_subs = subtitle_storage.load_or_new(plex_item)
+
+        existing_parts.append(part_id)
+
+        stored_any = False
+        for subtitle in video_subtitles:
+            lang = Locale.Language.Match(subtitle.language.alpha2)
+            Log.Debug(u"Adding subtitle to storage: %s, %s, %s" % (video_id, part_id, title))
+            ret_val = stored_subs.add(part_id, lang, subtitle, storage_type, mode=mode)
+
+            if ret_val:
+                Log.Debug("Subtitle stored")
+                stored_any = True
+
+            else:
+                Log.Debug("Subtitle already existing in storage")
+
+        if stored_any:
+            Log.Debug("Saving subtitle storage for %s" % video_id)
+            subtitle_storage.save(stored_subs)
+
+    #if existing_parts:
+    #    whack_missing_parts(scanned_video_part_map, existing_parts=existing_parts)
+
+
+def reset_storage(key):
+    """
+    resets the Dict[key] storage, thanks to https://docs.google.com/document/d/1hhLjV1pI-TA5y91TiJq64BdgKwdLnFt4hWgeOqpz1NA/edit#
+    We can't use the nice Plex interface for this, as it calls get multiple times before set
+    #Plex[":/plugins/*/prefs"].set("com.plexapp.agents.subzero", "reset_storage", False)
+    """
+
+    Log.Debug("resetting storage")
+    Dict[key] = {}
+    Dict.Save()
+
+
+def log_storage(key):
+    if key in Dict:
+        Log.Debug(pprint.pformat(Dict[key]))
+
+
+def save_subtitles_to_file(subtitles):
+    fld_custom = Prefs["subtitles.save.subFolder.Custom"].strip() \
+        if Prefs["subtitles.save.subFolder.Custom"] else None
+
+    for video, video_subtitles in subtitles.items():
+        if not video_subtitles:
+            continue
+
+        fld = None
+        if fld_custom or Prefs["subtitles.save.subFolder"] != "current folder":
+            # specific subFolder requested, create it if it doesn't exist
+            fld_base = os.path.split(video.name)[0]
+            if fld_custom:
+                if fld_custom.startswith("/"):
+                    # absolute folder
+                    fld = fld_custom
+                else:
+                    fld = os.path.join(fld_base, fld_custom)
+            else:
+                fld = os.path.join(fld_base, Prefs["subtitles.save.subFolder"])
+            fld = force_unicode(fld)
+            if not os.path.exists(fld):
+                os.makedirs(fld)
+        subliminal.api.save_subtitles(video, video_subtitles, directory=fld, single=cast_bool(Prefs['subtitles.only_one']),
+                                      encode_with=force_utf8 if config.enforce_encoding else None,
+                                      chmod=config.chmod, forced_tag=config.forced_only, path_decoder=force_unicode)
+    return True
+
+
+def save_subtitles_to_metadata(videos, subtitles):
+    for video, video_subtitles in subtitles.items():
+        mediaPart = videos[video]
+        for subtitle in video_subtitles:
+            content = force_utf8(subtitle.text) if config.enforce_encoding else subtitle.content
+
+            if not isinstance(mediaPart, Framework.api.agentkit.MediaPart):
+                # we're being handed a Plex.py model instance here, not an internal PMS MediaPart object.
+                # get the correct one
+                mp = PMSMediaProxy(video.id).get_part(mediaPart.id)
+            else:
+                mp = mediaPart
+            mp.subtitles[Locale.Language.Match(subtitle.language.alpha2)][subtitle.id] = Proxy.Media(content, ext="srt")
+    return True
+
+
+def save_subtitles(scanned_video_part_map, downloaded_subtitles, mode="a"):
+    meta_fallback = False
+    save_successful = False
+    storage = "metadata"
+    if Prefs['subtitles.save.filesystem']:
+        storage = "filesystem"
+        try:
+            Log.Debug("Using filesystem as subtitle storage")
+            save_subtitles_to_file(downloaded_subtitles)
+        except OSError:
+            if Prefs["subtitles.save.metadata_fallback"]:
+                meta_fallback = True
+            else:
+                raise
+        else:
+            save_successful = True
+
+    if not Prefs['subtitles.save.filesystem'] or meta_fallback:
+        if meta_fallback:
+            Log.Debug("Using metadata as subtitle storage, because filesystem storage failed")
+        else:
+            Log.Debug("Using metadata as subtitle storage")
+        save_successful = save_subtitles_to_metadata(scanned_video_part_map, downloaded_subtitles)
+
+    if save_successful and config.notify_executable:
+        notify_executable(config.notify_executable, scanned_video_part_map, downloaded_subtitles, storage)
+
+    store_subtitle_info(scanned_video_part_map, downloaded_subtitles, storage, mode=mode)

Contents/Code/support/subtitlehelpers.py

@@ -0,0 +1,203 @@
+# coding=utf-8
+
+import re, os
+import config
+import helpers
+
+from bs4 import UnicodeDammit
+
+
+class SubtitleHelper(object):
+    def __init__(self, filename):
+        self.filename = filename
+
+
+def subtitle_helpers(filename):
+    filename = helpers.unicodize(filename)
+    helper_classes = [DefaultSubtitleHelper]
+
+    if helpers.cast_bool(Prefs["subtitles.scan.exotic_ext"]):
+        helper_classes.insert(0, VobSubSubtitleHelper)
+
+    for cls in helper_classes:
+        if cls.is_helper_for(filename):
+            return cls(filename)
+    return None
+
+
+#####################################################################################################################
+
+class VobSubSubtitleHelper(SubtitleHelper):
+    @classmethod
+    def is_helper_for(cls, filename):
+        (file, file_extension) = os.path.splitext(filename)
+
+        # We only support idx (and maybe sub)
+        if not file_extension.lower() in ['.idx', '.sub']:
+            return False
+
+        # If we've been given a sub, we only support it if there exists a matching idx file
+        return os.path.exists(file + '.idx')
+
+    def process_subtitles(self, part):
+
+        lang_sub_map = {}
+
+        # We don't directly process the sub file, only the idx. Therefore if we are passed on of these files, we simply
+        # ignore it.
+        (file, ext) = os.path.splitext(self.filename)
+        if ext == '.sub':
+            return lang_sub_map
+
+        # If we have an idx file, we need to confirm there is an identically names sub file before we can proceed.
+        sub_filename = file + ".sub"
+        if not os.path.exists(sub_filename):
+            return lang_sub_map
+
+        Log('Attempting to parse VobSub file: ' + self.filename)
+        idx = Core.storage.load(os.path.join(self.filename))
+        if idx.count('VobSub index file') == 0:
+            Log('The idx file does not appear to be a VobSub, skipping...')
+            return lang_sub_map
+
+        languages = {}
+        language_index = 0
+        basename = os.path.basename(self.filename)
+        for language in re.findall('\nid: ([A-Za-z]{2})', idx):
+
+            if not languages.has_key(language):
+                languages[language] = []
+
+            Log('Found .idx subtitle file: ' + self.filename + ' language: ' + language + ' stream index: ' + str(language_index))
+            languages[language].append(Proxy.LocalFile(self.filename, index=str(language_index), format="vobsub"))
+            language_index += 1
+
+            if not lang_sub_map.has_key(language):
+                lang_sub_map[language] = []
+            lang_sub_map[language].append(basename)
+
+        for language, subs in languages.items():
+            part.subtitles[language][basename] = subs
+
+        return lang_sub_map
+
+
+#####################################################################################################################
+
+
+IETF_MATCH = ".+\.([^-.]+)(?:-[A-Za-z]+)?$"
+ENDSWITH_LANGUAGECODE_RE = re.compile("\.([^-.]{2,3})(?:-[A-Za-z]{2})?$")
+
+
+def match_ietf_language(s):
+    language_match = re.match(".+\.([^\.]+)$" if not helpers.cast_bool(Prefs["subtitles.language.ietf"])
+                              else IETF_MATCH, s)
+    if language_match and len(language_match.groups()) == 1:
+        language = language_match.groups()[0]
+        return language
+    return s
+
+
+class DefaultSubtitleHelper(SubtitleHelper):
+    @classmethod
+    def is_helper_for(cls, filename):
+        (file, file_extension) = os.path.splitext(filename)
+        return file_extension.lower()[1:] in config.SUBTITLE_EXTS
+
+    def process_subtitles(self, part):
+
+        lang_sub_map = {}
+
+        if not os.path.exists(self.filename):
+            return lang_sub_map
+
+        basename = os.path.basename(self.filename)
+        (file, ext) = os.path.splitext(self.filename)
+
+        # Remove the initial '.' from the extension
+        ext = ext[1:]
+
+        forced = ''
+        default = ''
+        split_tag = file.rsplit('.', 1)
+        if len(split_tag) > 1 and split_tag[1].lower() in ['forced', 'normal', 'default']:
+            file = split_tag[0]
+            # don't do anything with 'normal', we don't need it
+            if 'forced' == split_tag[1].lower():
+                forced = '1'
+            if 'default' == split_tag[1].lower():
+                default = '1'
+
+        # Attempt to extract the language from the filename (e.g. Avatar (2009).eng)
+        language = ""
+
+        # IETF support thanks to https://github.com/hpsbranco/LocalMedia.bundle/commit/4fad9aefedece78a1fa96401304351347f644369
+        language = Locale.Language.Match(match_ietf_language(file))
+
+        # skip non-SRT if wanted
+        if not helpers.cast_bool(Prefs["subtitles.scan.exotic_ext"]) and ext not in ["srt", "ass", "ssa"]:
+            return lang_sub_map
+
+        codec = None
+        format = None
+        if ext in ['txt', 'sub']:
+            try:
+
+                file_contents = Core.storage.load(self.filename)
+                lines = [line.strip() for line in file_contents.splitlines(True)]
+                if re.match('^\{[0-9]+\}\{[0-9]*\}', lines[1]):
+                    format = 'microdvd'
+                elif re.match('^[0-9]{1,2}:[0-9]{2}:[0-9]{2}[:=,]', lines[1]):
+                    format = 'txt'
+                elif '[SUBTITLE]' in lines[1]:
+                    format = 'subviewer'
+                else:
+                    Log("The subtitle file does not have a known format, skipping... : " + self.filename)
+                    return lang_sub_map
+            except:
+                Log("An error occurred while attempting to parse the subtitle file, skipping... : " + self.filename)
+                return lang_sub_map
+
+        if codec is None and ext in ['ass', 'ssa', 'smi', 'srt', 'psb']:
+            codec = ext.replace('ass', 'ssa')
+
+        if format is None:
+            format = codec
+
+        Log('Found subtitle file: ' + self.filename + ' language: ' + language + ' codec: ' + str(
+            codec) + ' format: ' + str(format) + ' default: ' + default + ' forced: ' + forced)
+        part.subtitles[language][basename] = Proxy.LocalFile(self.filename, codec=codec, format=format, default=default,
+                                                             forced=forced)
+
+        lang_sub_map[language] = [basename]
+        return lang_sub_map
+
+
+def get_subtitles_from_metadata(part):
+    subs = {}
+    for language in part.subtitles:
+        subs[language] = []
+        for key, proxy in getattr(part.subtitles[language], "_proxies").iteritems():
+            if not proxy or not len(proxy) >= 5:
+                Log.Debug("Can't parse metadata: %s" % repr(proxy))
+                continue
+
+            p_type = proxy[0]
+
+            if p_type == "Media":
+                # metadata subtitle
+                Log.Debug(u"Found metadata subtitle: %s, %s" % (language, repr(proxy)))
+                subs[language].append(key)
+    return subs
+
+
+def force_utf8(content):
+    a = UnicodeDammit(content)
+
+    Log.Debug("detected encoding: %s (None: most likely already successfully decoded)" % a.original_encoding)
+
+    # easy way out - already utf-8
+    if a.original_encoding and a.original_encoding == "utf-8":
+        return content
+
+    return (a.unicode_markup if a.unicode_markup else content.decode('ascii', 'replace')).encode("utf-8")

Contents/Code/support/tasks.py

@@ -0,0 +1,454 @@
+# coding=utf-8
+
+import datetime
+import time
+
+import operator
+import traceback
+
+import subliminal
+import subliminal_patch
+
+from subliminal_patch.patch_api import list_all_subtitles, download_subtitles
+from babelfish import Language
+from subliminal_patch.patch_subtitle import compute_score
+from missing_subtitles import items_get_all_missing_subs, refresh_item
+from background import scheduler
+from storage import save_subtitles, whack_missing_parts, get_subtitle_storage
+from support.config import config
+from support.items import get_recent_items, is_ignored, get_item
+from support.lib import Plex
+from support.helpers import track_usage, get_title_for_video_metadata, cast_bool
+from support.plex_media import scan_videos, get_plex_metadata, PartUnknownException
+
+
+class Task(object):
+    name = None
+    scheduler = None
+    periodic = False
+    running = False
+    time_start = None
+    data = None
+
+    stored_attributes = ("last_run", "last_run_time", "running")
+    default_data = {"last_run": None, "last_run_time": None, "running": False, "data": {}}
+
+    # task ready for being status-displayed?
+    ready_for_display = False
+
+    def __init__(self, scheduler):
+        self.name = self.get_class_name()
+        self.ready_for_display = False
+        self.time_start = None
+        self.scheduler = scheduler
+        self.setup_defaults()
+
+        self.running = False
+
+    def get_class_name(self):
+        return getattr(getattr(self, "__class__"), "__name__")
+
+    def __getattribute__(self, name):
+        if name in object.__getattribute__(self, "stored_attributes"):
+            return Dict["tasks"].get(self.name, {}).get(name, None)
+
+        return object.__getattribute__(self, name)
+
+    def __setattr__(self, name, value):
+        if name in object.__getattribute__(self, "stored_attributes"):
+            Dict["tasks"][self.name][name] = value
+            Dict.Save()
+            return
+
+        object.__setattr__(self, name, value)
+
+    def setup_defaults(self):
+        if self.name not in Dict["tasks"]:
+            Dict["tasks"][self.name] = self.default_data.copy()
+            return
+
+        sd = Dict["tasks"][self.name]
+
+        # forward-migration
+        for key, def_value in self.default_data.iteritems():
+            hasval = key in sd
+            if not hasval:
+                sd[key] = def_value
+
+    def signal(self, *args, **kwargs):
+        raise NotImplementedError
+
+    def prepare(self, *args, **kwargs):
+        return
+
+    def run(self):
+        self.time_start = datetime.datetime.now()
+
+    def post_run(self, data_holder):
+        self.running = False
+        self.last_run = datetime.datetime.now()
+        if self.time_start:
+            self.last_run_time = self.last_run - self.time_start
+        self.time_start = None
+
+
+class SearchAllRecentlyAddedMissing(Task):
+    periodic = True
+    items_done = None
+    items_searching = None
+    items_searching_ids = None
+    items_failed = None
+    percentage = 0
+
+    stall_time = 30
+
+    def __init__(self, scheduler):
+        super(SearchAllRecentlyAddedMissing, self).__init__(scheduler)
+        self.items_done = None
+        self.items_searching = None
+        self.items_searching_ids = None
+        self.items_failed = None
+        self.percentage = 0
+
+    def signal(self, signal_name, *args, **kwargs):
+        handler = getattr(self, "signal_%s" % signal_name)
+        return handler(*args, **kwargs) if handler else None
+
+    def signal_updated_metadata(self, *args, **kwargs):
+        item_id = int(args[0])
+
+        if item_id in self.items_searching_ids:
+            self.items_done.append(item_id)
+            return True
+
+    def prepare(self, *args, **kwargs):
+        self.items_done = []
+        recent_items = get_recent_items()
+        missing = items_get_all_missing_subs(recent_items)
+        ids = set([id for added_at, id, title, item, missing_languages in missing if not is_ignored(id, item=item)])
+        self.items_searching = missing
+        self.items_searching_ids = ids
+        self.items_failed = []
+        self.percentage = 0
+        self.ready_for_display = True
+
+    def run(self):
+        super(SearchAllRecentlyAddedMissing, self).run()
+        self.running = True
+        missing_count = len(self.items_searching)
+        items_done_count = 0
+
+        for added_at, item_id, title, item, missing_languages in self.items_searching:
+            Log.Debug(u"Task: %s, triggering refresh for %s (%s)", self.name, title, item_id)
+            refresh_item(item_id)
+            search_started = datetime.datetime.now()
+            tries = 1
+            while 1:
+                if item_id in self.items_done:
+                    items_done_count += 1
+                    Log.Debug(u"Task: %s, item %s done", self.name, item_id)
+                    self.percentage = int(items_done_count * 100 / missing_count)
+                    break
+
+                # item considered stalled after self.stall_time seconds passed after last refresh
+                if (datetime.datetime.now() - search_started).total_seconds() > self.stall_time:
+                    if tries > 3:
+                        self.items_failed.append(item_id)
+                        Log.Debug(u"Task: %s, item stalled for %s times: %s, skipping", self.name, tries, item_id)
+                        break
+
+                    Log.Debug(u"Task: %s, item stalled for %s seconds: %s, retrying", self.name, self.stall_time,
+                              item_id)
+                    tries += 1
+                    refresh_item(item_id)
+                    search_started = datetime.datetime.now()
+                    time.sleep(1)
+                time.sleep(0.1)
+            # we can't hammer the PMS, otherwise requests will be stalled
+            time.sleep(1)
+
+        Log.Debug("Task: %s, done. Failed items: %s", self.name, self.items_failed)
+        self.running = False
+
+    def post_run(self, task_data):
+        super(SearchAllRecentlyAddedMissing, self).post_run(task_data)
+        self.ready_for_display = False
+        self.percentage = 0
+        self.items_done = None
+        self.items_failed = None
+        self.items_searching = None
+        self.items_searching_ids = None
+
+
+class SubtitleListingMixin(object):
+    def list_subtitles(self, rating_key, item_type, part_id, language):
+        metadata = get_plex_metadata(rating_key, part_id, item_type)
+
+        if item_type == "episode":
+            min_score = 66
+        else:
+            min_score = 23
+
+        scanned_parts = scan_videos([metadata], kind="series" if item_type == "episode" else "movie", ignore_all=True)
+        if not scanned_parts:
+            Log.Error("Couldn't list available subtitles for %s", rating_key)
+            return
+
+        video, plex_part = scanned_parts.items()[0]
+        config.init_subliminal_patches()
+
+        available_subs = list_all_subtitles(scanned_parts, {Language.fromietf(language)},
+                                            providers=config.providers,
+                                            provider_configs=config.provider_settings)
+
+        use_hearing_impaired = Prefs['subtitles.search.hearingImpaired'] in ("prefer", "force HI")
+
+        # sort subtitles by score
+        unsorted_subtitles = []
+        for s in available_subs[video]:
+            Log.Debug("Starting score computation for %s", s)
+            try:
+                matches = s.get_matches(video, hearing_impaired=use_hearing_impaired)
+            except AttributeError:
+                Log.Error("Match computation failed for %s: %s", s, traceback.format_exc())
+                continue
+
+            unsorted_subtitles.append((s, compute_score(matches, video), matches))
+        scored_subtitles = sorted(unsorted_subtitles, key=operator.itemgetter(1), reverse=True)
+
+        subtitles = []
+        for subtitle, score, matches in scored_subtitles:
+            # check score
+            if score < min_score:
+                Log.Info('Score %d is below min_score (%d)', score, min_score)
+                continue
+            subtitle.score = score
+            subtitle.matches = matches
+            subtitle.part_id = part_id
+            subtitle.item_type = item_type
+            subtitles.append(subtitle)
+        return subtitles
+
+
+class DownloadSubtitleMixin(object):
+    def download_subtitle(self, subtitle, rating_key, mode="m"):
+        from interface.menu_helpers import set_refresh_menu_state
+
+        item_type = subtitle.item_type
+        part_id = subtitle.part_id
+        metadata = get_plex_metadata(rating_key, part_id, item_type)
+        scanned_parts = scan_videos([metadata], kind="series" if item_type == "episode" else "movie", ignore_all=True)
+        video, plex_part = scanned_parts.items()[0]
+
+        # downloaded_subtitles = {subliminal.Video: [subtitle, subtitle, ...]}
+        download_subtitles([subtitle], providers=config.providers, provider_configs=config.provider_settings)
+        download_successful = False
+
+        if subtitle.content:
+            try:
+                whack_missing_parts(scanned_parts)
+                save_subtitles(scanned_parts, {video: [subtitle]}, mode=mode)
+                Log.Debug("Manually downloaded subtitle for: %s", rating_key)
+                download_successful = True
+                refresh_item(rating_key)
+                track_usage("Subtitle", "manual", "download", 1)
+            except:
+                Log.Error("Something went wrong when downloading specific subtitle: %s", traceback.format_exc())
+            finally:
+                set_refresh_menu_state(None)
+
+                if download_successful:
+                    # store item in history
+                    from support.history import get_history
+                    item_title = get_title_for_video_metadata(metadata, add_section_title=False)
+                    history = get_history()
+                    history.add(item_title, video.id, section_title=video.plexapi_metadata["section"], subtitle=subtitle,
+                                mode=mode)
+        return download_successful
+
+
+class AvailableSubsForItem(SubtitleListingMixin, Task):
+    item_type = None
+    part_id = None
+    language = None
+    rating_key = None
+
+    def prepare(self, *args, **kwargs):
+        self.item_type = kwargs.get("item_type")
+        self.part_id = kwargs.get("part_id")
+        self.language = kwargs.get("language")
+        self.rating_key = kwargs.get("rating_key")
+
+    def setup_defaults(self):
+        super(AvailableSubsForItem, self).setup_defaults()
+
+        # reset any previous data
+        Dict["tasks"][self.name]["data"] = {}
+
+    def run(self):
+        super(AvailableSubsForItem, self).run()
+        self.running = True
+        track_usage("Subtitle", "manual", "list", 1)
+        self.data = self.list_subtitles(self.rating_key, self.item_type, self.part_id, self.language)
+
+    def post_run(self, task_data):
+        super(AvailableSubsForItem, self).post_run(task_data)
+        if self.rating_key not in task_data:
+            task_data[self.rating_key] = {}
+
+        task_data[self.rating_key][self.language] = self.data
+
+
+class DownloadSubtitleForItem(DownloadSubtitleMixin, Task):
+    subtitle = None
+    rating_key = None
+
+    def prepare(self, *args, **kwargs):
+        self.subtitle = kwargs["subtitle"]
+        self.rating_key = kwargs["rating_key"]
+
+    def run(self):
+        super(DownloadSubtitleForItem, self).run()
+        self.running = True
+        self.download_subtitle(self.subtitle, self.rating_key)
+        self.running = False
+
+
+class MissingSubtitles(Task):
+    rating_key = None
+    item_type = None
+    part_id = None
+    language = None
+
+    def run(self):
+        super(MissingSubtitles, self).run()
+        self.running = True
+        self.data = []
+        recent_items = get_recent_items()
+        if recent_items:
+            self.data = items_get_all_missing_subs(recent_items)
+
+    def post_run(self, task_data):
+        super(MissingSubtitles, self).post_run(task_data)
+        task_data["missing_subtitles"] = self.data
+
+
+class FindBetterSubtitles(DownloadSubtitleMixin, SubtitleListingMixin, Task):
+    periodic = True
+
+    # TV: episode, format, series, year, season, video_codec, release_group, hearing_impaired
+    series_cutoff = 132
+
+    # movies: format, title, release_group, year, video_codec, resolution, hearing_impaired
+    movies_cutoff = 61
+
+    def signal_updated_metadata(self, *args, **kwargs):
+        return True
+
+    def run(self):
+        super(FindBetterSubtitles, self).run()
+        self.running = True
+        better_found = 0
+        try:
+            max_search_days = int(Prefs["scheduler.tasks.FindBetterSubtitles.max_days_after_added"].strip())
+        except ValueError:
+            Log.Error("Please only put numbers into the FindBetterSubtitles.max_days_after_added setting. Exiting")
+            return
+        else:
+            if max_search_days > 30:
+                Log.Error("FindBetterSubtitles.max_days_after_added is too big. Max is 30 days.")
+                return
+
+        now = datetime.datetime.now()
+
+        subtitle_storage = get_subtitle_storage()
+        recent_subs = subtitle_storage.load_recent_files(age_days=max_search_days)
+
+        for fn, stored_subs in recent_subs.iteritems():
+            video_id = stored_subs.video_id
+            cutoff = self.series_cutoff if stored_subs.item_type == "episode" else self.movies_cutoff
+
+            # don't search for better subtitles until at least 30 minutes have passed
+            if stored_subs.added_at + datetime.timedelta(minutes=30) > now:
+                Log.Debug("Item %s too new, skipping", video_id)
+                continue
+
+            # added_date <= max_search_days?
+            if stored_subs.added_at + datetime.timedelta(days=max_search_days) <= now:
+                continue
+
+            ditch_parts = []
+
+            # look through all stored subtitle data
+            for part_id, languages in stored_subs.parts.iteritems():
+                part_id = str(part_id)
+
+                # all languages
+                for language, current_subs in languages.iteritems():
+                    current_key = current_subs.get("current")
+                    current = current_subs.get(current_key)
+
+                    # currently got subtitle?
+                    if not current:
+                        continue
+                    current_score = current.score
+                    current_mode = current.mode
+
+                    # late cutoff met? skip
+                    if current_score >= cutoff:
+                        Log.Debug(u"Skipping finding better subs, cutoff met (current: %s, cutoff: %s): %s",
+                                  current_score, cutoff, stored_subs.title)
+                        continue
+
+                    # got manual subtitle but don't want to touch those?
+                    if current_mode == "m" and \
+                            not cast_bool(Prefs["scheduler.tasks.FindBetterSubtitles.overwrite_manually_selected"]):
+                        Log.Debug(u"Skipping finding better subs, had manual: %s", stored_subs.title)
+                        continue
+
+                    try:
+                        subs = self.list_subtitles(video_id, stored_subs.item_type, part_id, language)
+                    except PartUnknownException:
+                        Log.Info("Part %s unknown/gone; ditching subtitle info", part_id)
+                        ditch_parts.append(part_id)
+                        continue
+
+                    if subs:
+                        # subs are already sorted by score
+                        better_downloaded = False
+                        better_tried_download = 0
+                        for sub in subs:
+                            if sub.score > current_score:
+                                Log.Debug("Better subtitle found for %s, downloading", video_id)
+                                better_tried_download += 1
+                                ret = self.download_subtitle(sub, video_id, mode="b")
+                                if ret:
+                                    better_found += 1
+                                    better_downloaded = True
+                                    break
+                                else:
+                                    Log.Debug("Couldn't download/save subtitle. Continuing to the next one")
+                        if better_tried_download and not better_downloaded:
+                            Log.Debug("Tried downloading better subtitle for %s, but every try failed.", video_id)
+
+                        elif better_downloaded:
+                            Log.Debug("Better subtitle downloaded for %s", video_id)
+
+            if ditch_parts:
+                for part_id in ditch_parts:
+                    try:
+                        del stored_subs.parts[part_id]
+                    except KeyError:
+                        pass
+                subtitle_storage.save(stored_subs)
+
+        if better_found:
+            Log.Debug("Task: %s, done. Better subtitles found for %s items", self.name, better_found)
+        self.running = False
+
+
+scheduler.register(SearchAllRecentlyAddedMissing)
+scheduler.register(AvailableSubsForItem)
+scheduler.register(DownloadSubtitleForItem)
+scheduler.register(MissingSubtitles)
+scheduler.register(FindBetterSubtitles)

Contents/DefaultPrefs.json

@@ -1,117 +1,573 @@
 [
-		{
-                "id": "provider.addic7ed.username",
-                "label": "Addic7ed Username",
-                "type": "text",
-                "default": "Username"
-        },
-        {
-                "id": "provider.addic7ed.password",
-                "label": "Addic7ed Password",
-                "type": "text",
-                "option": "hidden",
-                "default": "",
-                "secure": "true" 
-        },
-	{
-                "id": "provider.addic7ed.use_random_agents",
-                "label": "Addic7ed: Use random user agents (should not be necessary)",
-                "type": "bool",
-                "default": "false"
-        },
-        {
-                "id": "langPref1",
-                "label": "Subtitle Language (1)",
-                "type": "enum",
-                "values": ["sq","ar","be","bs","bg","ca","zh","cs","da","nl","en","et","fi","fr","de","el","he","hi","hu","is","id","it","ja","ko","lv","lt","mk","ms","no","fa","pl","pt","pt-br","ro","ru","sr","sk","sl","es","sv","th","tr","uk","vi","hr"],
-                "default": "en"
-        },
-        {
-                "id": "langPref2",
-                "label": "Subtitle Language (2)",
-                "type": "enum",
-                "values": ["None", "sq","ar","be","bs","bg","ca","zh","cs","da","nl","en","et","fi","fr","de","el","he","hi","hu","is","id","it","ja","ko","lv","lt","mk","ms","no","fa","pl","pt","pt-br","ro","ru","sr","sk","sl","es","sv","th","tr","uk","vi","hr"],
-                "default": "None"
-        },
-	{
-                "id": "langPref3",
-                "label": "Subtitle Language (3)",
-                "type": "enum",
-                "values": ["None", "sq","ar","be","bs","bg","ca","zh","cs","da","nl","en","et","fi","fr","de","el","he","hi","hu","is","id","it","ja","ko","lv","lt","mk","ms","no","fa","pl","pt","pt-br","ro","ru","sr","sk","sl","es","sv","th","tr","uk","vi","hr"],
-                "default": "None"
-        },
-	{
-                "id": "provider.opensubtitles.enabled",
-                "label": "Provider: Enable OpenSubtitles",
-                "type": "bool",
-                "default": "true"
-        },
-		{
-                "id": "provider.thesubdb.enabled",
-                "label": "Provider: Enable TheSubDB",
-                "type": "bool",
-                "default": "true"
-        },
-		{
-                "id": "provider.podnapisi.enabled",
-                "label": "Provider: Enable Podnapisi.NET",
-                "type": "bool",
-                "default": "true"
-        },
-		{
-                "id": "provider.addic7ed.enabled",
-                "label": "Provider: Enable Addic7ed",
-                "type": "bool",
-                "default": "true"
-        },
-		{
-                "id": "provider.tvsubtitles.enabled",
-                "label": "Provider: Enable TVsubtitles.net",
-                "type": "bool",
-                "default": "true"
-        },
-        {
-                "id": "subtitles.scan.embedded",
-                "label": "Scan: include embedded subtitles (skip if existing)",
-                "type": "bool",
-                "default": "false"
-        },
-        {
-                "id": "subtitles.scan.external",
-                "label": "Scan: include external subtitles (skip if existing)",
-                "type": "bool",
-                "default": "false"
-        },
-        {
-                "id": "subtitles.search.minimumScore",
-                "label": "Minimum score for subtitles to download",
-                "type": "enum",
-                "values": ["100","95","90","85","80","75","70","65","60","55","50","45","40","35","30","25","20","15","10","5","0"],
-                "default": "0"
-        },
-        {
-                "id": "subtitles.search.hearingImpaired",
-                "label": "Download hearing impaired subtitles.",
-                "type": "bool",
-                "default": "false"
-        },
-        {
-                "id": "subtitles.save.filesystem",
-                "label": "Store subtitles next to media files (instead of metadata)",
-                "type": "bool",
-                "default": "false"
-        },
-    	{
-                "id": "subtitles.save.subFolder",
-                "label": "Subtitle Folder (\"current folder\" is the folder the current media file lives in)",
-                "type": "enum",
-                "values": ["current folder", "sub", "subs", "subtitle", "subtitles"],
-                "default": "current folder"
-        },
-		{
-                "id": "subtitles.save.subFolder.Custom",
-                "label": "Custom Subtitle folder (overrides \"Subtitle Folder\"; computes to real paths; use for example \"bla\" as a subfolder of the current media file folder or an absolute path)",
-                "type": "text",
-                "default": ""
-        },
+  {
+    "id": "langPref1",
+    "label": "Subtitle Language (1)",
+    "type": "enum",
+    "values": [
+      "sq",
+      "ar",
+      "be",
+      "bs",
+      "bg",
+      "ca",
+      "zh",
+      "cs",
+      "da",
+      "nl",
+      "en",
+      "et",
+      "fi",
+      "fr",
+      "de",
+      "el",
+      "he",
+      "hi",
+      "hu",
+      "is",
+      "id",
+      "it",
+      "ja",
+      "ko",
+      "lv",
+      "lt",
+      "mk",
+      "ms",
+      "no",
+      "fa",
+      "pl",
+      "pt",
+      "pt-br",
+      "ro",
+      "ru",
+      "sr",
+      "sk",
+      "sl",
+      "es",
+      "sv",
+      "th",
+      "tr",
+      "uk",
+      "vi",
+      "hr"
+    ],
+    "default": "en"
+  },
+  {
+    "id": "langPref2",
+    "label": "Subtitle Language (2)",
+    "type": "enum",
+    "values": [
+      "None",
+      "sq",
+      "ar",
+      "be",
+      "bs",
+      "bg",
+      "ca",
+      "zh",
+      "cs",
+      "da",
+      "nl",
+      "en",
+      "et",
+      "fi",
+      "fr",
+      "de",
+      "el",
+      "he",
+      "hi",
+      "hu",
+      "is",
+      "id",
+      "it",
+      "ja",
+      "ko",
+      "lv",
+      "lt",
+      "mk",
+      "ms",
+      "no",
+      "fa",
+      "pl",
+      "pt",
+      "pt-br",
+      "ro",
+      "ru",
+      "sr",
+      "sk",
+      "sl",
+      "es",
+      "sv",
+      "th",
+      "tr",
+      "uk",
+      "vi",
+      "hr"
+    ],
+    "default": "None"
+  },
+  {
+    "id": "langPref3",
+    "label": "Subtitle Language (3)",
+    "type": "enum",
+    "values": [
+      "None",
+      "sq",
+      "ar",
+      "be",
+      "bs",
+      "bg",
+      "ca",
+      "zh",
+      "cs",
+      "da",
+      "nl",
+      "en",
+      "et",
+      "fi",
+      "fr",
+      "de",
+      "el",
+      "he",
+      "hi",
+      "hu",
+      "is",
+      "id",
+      "it",
+      "ja",
+      "ko",
+      "lv",
+      "lt",
+      "mk",
+      "ms",
+      "no",
+      "fa",
+      "pl",
+      "pt",
+      "pt-br",
+      "ro",
+      "ru",
+      "sr",
+      "sk",
+      "sl",
+      "es",
+      "sv",
+      "th",
+      "tr",
+      "uk",
+      "vi",
+      "hr"
+    ],
+    "default": "None"
+  },
+  {
+    "id": "langPrefCustom",
+    "label": "Additional Subtitle Languages (use ISO-639-1 codes; comma-separated)",
+    "type": "text",
+    "default": "None"
+  },
+  {
+    "id": "subtitles.only_foreign",
+    "label": "Only download foreign/forced subtitles",
+    "type": "bool",
+    "default": "false"
+  },
+  {
+    "id": "subtitles.language.ietf",
+    "label": "Treat IETF language tags as ISO 639-1 (e.g. pt-BR = pt)",
+    "type": "bool",
+    "default": "true"
+  },
+  {
+    "id": "subtitles.only_one",
+    "label": "Restrict to one language (skips adding \".lang.\" to the subtitle filename; only uses \"Subtitle Language (1)\")",
+    "type": "bool",
+    "default": "false"
+  },
+  {
+    "id": "subtitles.language.treat_und_as_first",
+    "label": "Embedded subtitles: Treat \"Undefined\" (und) as language 1",
+    "type": "bool",
+    "default": "true"
+  },
+  {
+    "id": "provider.opensubtitles.enabled",
+    "label": "Provider: Enable OpenSubtitles",
+    "type": "bool",
+    "default": "true"
+  },
+  {
+    "id": "provider.opensubtitles.username",
+    "label": "Opensubtitles Username (VIP)",
+    "type": "text",
+    "default": ""
+  },
+  {
+    "id": "provider.opensubtitles.password",
+    "label": "Opensubtitles Password",
+    "type": "text",
+    "option": "hidden",
+    "default": "",
+    "secure": "true"
+  },
+  {
+    "id": "provider.podnapisi.enabled",
+    "label": "Provider: Enable Podnapisi.NET",
+    "type": "bool",
+    "default": "true"
+  },
+  {
+    "id": "provider.addic7ed.enabled",
+    "label": "Provider: Enable Addic7ed",
+    "type": "bool",
+    "default": "true"
+  },
+  {
+    "id": "provider.addic7ed.username",
+    "label": "Addic7ed Username",
+    "type": "text",
+    "default": ""
+  },
+  {
+    "id": "provider.addic7ed.password",
+    "label": "Addic7ed Password",
+    "type": "text",
+    "option": "hidden",
+    "default": "",
+    "secure": "true"
+  },
+  {
+    "id": "provider.addic7ed.boost_by",
+    "label": "Addic7ed: boost score (if requirements met)",
+    "type": "enum",
+    "values": [
+      "100",
+      "95",
+      "90",
+      "85",
+      "80",
+      "75",
+      "70",
+      "67",
+      "65",
+      "60",
+      "55",
+      "50",
+      "45",
+      "40",
+      "35",
+      "30",
+      "25",
+      "20",
+      "15",
+      "10",
+      "5",
+      "0"
+    ],
+    "default": "10"
+  },
+  {
+    "id": "provider.addic7ed.use_random_agents",
+    "label": "Addic7ed: Use random user agents",
+    "type": "bool",
+    "default": "false"
+  },
+  {
+    "id": "provider.tvsubtitles.enabled",
+    "label": "Provider: Enable TVsubtitles.net",
+    "type": "bool",
+    "default": "true"
+  },
+  {
+    "id": "provider.opensubtitles.use_tags",
+    "label": "I keep the exact (release-) filename of my media files",
+    "type": "bool",
+    "default": "true"
+  },
+  {
+    "id": "subtitles.scan.embedded",
+    "label": "Scan: include embedded subtitles (in the media file (MKV/MP4), don't download if existing)",
+    "type": "bool",
+    "default": "false"
+  },
+  {
+    "id": "subtitles.scan.external",
+    "label": "Scan: include external subtitles (metadata/filesystem, don't download if existing)",
+    "type": "bool",
+    "default": "true"
+  },
+  {
+    "id": "subtitles.scan.exotic_ext",
+    "label": "Scan: include \"exotic\" external subtitle formats (anything else than .srt/.ssa/.ass)",
+    "type": "bool",
+    "default": "false"
+  },
+  {
+    "id": "subtitles.scan.filename_strictness",
+    "label": "Scan: which external subtitles should be picked up?",
+    "type": "enum",
+    "values": [
+      "exact: media filename match",
+      "loose: filename contains media filename",
+      "any"
+    ],
+    "default": "loose: filename contains media filename"
+  },
+  {
+    "id": "subtitles.search.minimumTVScore1",
+    "label": "Minimum score for TV (min: 77, sane: 110; min-ideal: 116; see http://v.ht/szscores)",
+    "type": "text",
+    "default": "116"
+  },
+  {
+    "id": "subtitles.search.minimumMovieScore1",
+    "label": "Minimum score for movies (min: 23, def/sane/min-ideal: 33; see http://v.ht/szscores)",
+    "type": "text",
+    "default": "33"
+  },
+  {
+    "id": "subtitles.search.hearingImpaired",
+    "label": "Download hearing impaired subtitles.",
+    "type": "enum",
+    "values": [
+      "prefer",
+      "don't prefer",
+      "force HI",
+      "force non-HI"
+    ],
+    "default": "don't prefer"
+  },
+  {
+    "id": "subtitles.enforce_encoding",
+    "label": "Normalize subtitle encoding to UTF-8",
+    "type": "bool",
+    "default": "true"
+  },
+  {
+    "id": "subtitles.save.filesystem",
+    "label": "Store subtitles next to media files (instead of metadata)",
+    "type": "bool",
+    "default": "true"
+  },
+  {
+    "id": "subtitles.save.subFolder",
+    "label": "Subtitle Folder (\"current folder\" is the folder the current media file lives in)",
+    "type": "enum",
+    "values": [
+      "current folder",
+      "sub",
+      "subs",
+      "subtitle",
+      "subtitles"
+    ],
+    "default": "current folder"
+  },
+  {
+    "id": "subtitles.save.subFolder.Custom",
+    "label": "Custom Subtitle folder (overrides \"Subtitle Folder\"; computes to real paths)",
+    "type": "text",
+    "default": ""
+  },
+  {
+    "id": "subtitles.save.metadata_fallback",
+    "label": "Fall back to metadata storage if filesystem storage failed",
+    "type": "bool",
+    "default": "false"
+  },
+  {
+    "id": "subtitles.save.chmod",
+      "label": "Set subtitle file permissions to (integer, e.g.: 0775)",
+      "type": "text",
+      "default": ""
+  },
+  {
+    "id": "subtitles.autoclean",
+    "label": "Automatically delete leftover/unused (externally saved) subtitles",
+    "type": "bool",
+    "default": "true"
+  },
+  {
+    "id": "activity.on_playback",
+    "label": "On media playback: search for missing subtitles (refresh item)",
+    "type": "enum",
+    "values": [
+      "never",
+      "current media item",
+      "next episode (series)",
+      "hybrid: current item or next episode"
+    ],
+    "default": "never"
+  },
+  {
+    "id": "scheduler.tasks.SearchAllRecentlyAddedMissing.frequency",
+    "label": "Scheduler: Periodically search for recent items with missing subtitles",
+    "type": "enum",
+    "values": [
+      "never",
+      "every 1 hours",
+      "every 3 hours",
+      "every 6 hours",
+      "every 12 hours",
+      "every 24 hours"
+    ],
+    "default": "every 6 hours"
+  },
+  {
+    "id": "scheduler.item_is_recent_age",
+    "label": "Scheduler: Item age to be considered recent",
+    "type": "enum",
+    "values": [
+      "1 days",
+      "2 days",
+      "3 days",
+      "4 days",
+      "1 weeks",
+      "2 weeks",
+      "3 weeks",
+      "4 weeks",
+      "5 weeks",
+      "6 weeks"
+    ],
+    "default": "2 weeks"
+  },
+  {
+    "id": "scheduler.max_recent_items_per_library",
+    "label": "Scheduler: Recent items to consider per library",
+    "type": "text",
+    "default": "500"
+  },
+  {
+    "id": "scheduler.tasks.FindBetterSubtitles.frequency",
+    "label": "Scheduler: Periodically search for better subtitles",
+    "type": "enum",
+    "values": [
+      "never",
+      "every 6 hours",
+      "every 12 hours",
+      "every 24 hours"
+    ],
+    "default": "every 12 hours"
+  },
+  {
+    "id": "scheduler.tasks.FindBetterSubtitles.max_days_after_added",
+    "label": "Scheduler: Days to search for better subtitles (max: 30 days)",
+    "type": "text",
+    "default": "7"
+  },
+  {
+    "id": "scheduler.tasks.FindBetterSubtitles.overwrite_manually_selected",
+    "label": "Scheduler: Overwrite manually selected subtitles when better found",
+    "type": "bool",
+    "default": "true"
+  },
+  {
+    "id": "history_size",
+    "label": "History: amount of items to store historical data for",
+    "type": "enum",
+    "values": [
+      "50",
+      "100",
+      "150",
+      "250",
+      "500"
+    ],
+    "default": "100"
+  },
+  {
+    "id": "subtitles.try_downloads",
+    "label": "How many download tries per subtitle (on timeout or error)",
+    "type": "enum",
+    "values": [
+      "1",
+      "2",
+      "3",
+      "4"
+    ],
+    "default": "2"
+  },
+  {
+    "id": "subtitles.ignore_fs",
+    "label": "Ignore folders (with \"subzero.ignore/.subzero.ignore/.nosz\" files in them)",
+    "type": "bool",
+    "default": "false"
+  },
+  {
+    "id": "subtitles.ignore_paths",
+    "label": "Ignore anything in the following paths (comma-separated)",
+    "type": "text",
+    "default": ""
+  },
+  {
+    "id": "plugin_mode",
+    "label": "Sub-Zero mode",
+    "type": "enum",
+    "values": [
+      "agent + channel",
+      "only agent",
+      "only channel"
+    ],
+    "default": "agent + channel"
+  },
+  {
+    "id": "plugin_pin",
+    "label": "Access PIN (any amount of numbers, 0-9)",
+    "type": "text",
+    "option": "hidden",
+    "default": "",
+    "secure": "true"
+  },
+  {
+    "id": "plugin_pin_valid_for",
+    "label": "Access PIN valid for minutes",
+    "type": "text",
+    "default": "10"
+  },
+  {
+    "id": "plugin_pin_mode",
+    "label": "Use PIN to restrict access to (needs plugin or PMS restart)",
+    "type": "enum",
+    "values": [
+      "disabled",
+      "channel menu",
+      "advanced menu"
+    ],
+    "default": "disabled"
+  },
+  {
+    "id": "notify_executable",
+    "label": "Call this executable upon successful subtitle download",
+    "type": "text",
+    "default": ""
+  },
+  {
+    "id": "check_permissions",
+    "label": "Check for correct folder permissions of every library on plugin start",
+    "type": "bool",
+    "default": "true"
+  },
+  {
+    "id": "log_level",
+    "label": "How verbose should the logging be?",
+    "type": "enum",
+    "values": [
+      "CRITICAL",
+      "ERROR",
+      "WARNING",
+      "INFO",
+      "DEBUG"
+    ],
+    "default": "WARNING"
+  },
+  {
+    "id": "log_console",
+    "label": "Log to console (for development/debugging)",
+    "type": "bool",
+    "default": "false"
+  },
+  {
+    "id": "track_usage",
+    "label": "Collect anonymous usage statistics",
+    "type": "bool",
+    "default": "true"
+  }
 ]

Contents/Info.plist

@@ -4,30 +4,48 @@
     <dict>
         <key>CFBundleDevelopmentRegion</key>
         <string>English</string>
-        <key>CFBundleExecutable</key>
-        <string>Test Plug-in</string>
         <key>CFBundleIdentifier</key>
-        <string>com.plexapp.agents.subliminal</string>
+        <string>com.plexapp.agents.subzero</string>
         <key>CFBundleInfoDictionaryVersion</key>
         <string>6.0</string>
         <key>CFBundleShortVersionString</key>
-        <string>1.0</string>
+        <string>1.4.27</string>
         <key>CFBundleSignature</key>
         <string>????</string>
         <key>CFBundleVersion</key>
-        <string>1.0</string>
+        <string>1.4.27.965</string>
         <key>PlexFrameworkVersion</key>
         <string>2</string>
         <key>PlexPluginClass</key>
         <string>Agent</string>
         <key>PlexPluginMode</key>
-        <string>AlwaysOn</string>
+        <string>Daemon</string>
         <key>PlexPluginConsoleLogging</key>
-        <string>1</string>
+        <string>0</string>
         <key>PlexPluginDevMode</key>
-        <string>1</string>
+        <string>0</string>
          <key>PlexPluginCodePolicy</key>
-            <!-- this allows channels to access some python methods which are otherwise blocked, as well as import external code libraries, and interact with the PMS HTTP API -->     
+            <!-- this allows channels to access some python methods which are otherwise blocked, as well as import external code libraries, and interact with the PMS HTTP API -->
             <string>Elevated</string>
+	<key>PlexAgentAttributionText</key>
+	<string>&lt;div style=&quot;white-space: pre;&quot;&gt;&lt;img src=&quot;https://raw.githubusercontent.com/pannal/Sub-Zero.bundle/master/Contents/Resources/subzero.gif&quot; /&gt;
+
+&lt;h1&gt;Sub-Zero for Plex&lt;/h1&gt;&lt;i&gt;Subtitles done right&lt;/i&gt;
+
+Version 1.4.27.965
+
+Originally based on @bramwalet's awesome &lt;a href=&quot;https://github.com/bramwalet/Subliminal.bundle&quot;&gt;Subliminal.bundle&lt;/a&gt;
+
+If you like this, buy me a beer: &lt;a href=&quot;https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&amp;hosted_button_id=G9VKR2B8PMNKG&quot; target=&quot;_blank&quot; title=&quot;donate&quot;&gt;&lt;img src=&quot;https://www.paypalobjects.com/en_US/i/btn/btn_donate_LG.gif&quot; alt=&quot;donate&quot; title=&quot;donate&quot; /&gt;&lt;/a&gt;
+
+&lt;strong&gt;Need help?&lt;/strong&gt;
+Wiki: &lt;a href=&quot;http://v.ht/szwiki&quot;&gt;http://v.ht/szwiki&lt;/a&gt;
+Score info: &lt;a href=&quot;http://v.ht/szscores&quot;&gt;http://v.ht/szscores&lt;/a&gt;
+Plex thread: &lt;a href=&quot;https://forums.plex.tv/discussion/186575&quot;>https://forums.plex.tv/discussion/186575&lt;/a&gt;
+Github: &lt;a href=&quot;https://github.com/pannal/Sub-Zero.bundle&quot;&gt;https://github.com/pannal/Sub-Zero&lt;/a&gt;
+
+panni, 2016
+&lt;/div&gt;
+	</string>
     </dict>
 </plist>

Contents/Libraries/Shared/_markerlib/__init__.py

@@ -0,0 +1,16 @@
+try:
+    import ast
+    from _markerlib.markers import default_environment, compile, interpret
+except ImportError:
+    if 'ast' in globals():
+        raise
+    def default_environment():
+        return {}
+    def compile(marker):
+        def marker_fn(environment=None, override=None):
+            # 'empty markers are True' heuristic won't install extra deps.
+            return not marker.strip()
+        marker_fn.__doc__ = marker
+        return marker_fn
+    def interpret(marker, environment=None, override=None):
+        return compile(marker)()

Contents/Libraries/Shared/_markerlib/markers.py

@@ -0,0 +1,119 @@
+# -*- coding: utf-8 -*-
+"""Interpret PEP 345 environment markers.
+
+EXPR [in|==|!=|not in] EXPR [or|and] ...
+
+where EXPR belongs to any of those:
+
+    python_version = '%s.%s' % (sys.version_info[0], sys.version_info[1])
+    python_full_version = sys.version.split()[0]
+    os.name = os.name
+    sys.platform = sys.platform
+    platform.version = platform.version()
+    platform.machine = platform.machine()
+    platform.python_implementation = platform.python_implementation()
+    a free string, like '2.6', or 'win32'
+"""
+
+__all__ = ['default_environment', 'compile', 'interpret']
+
+import ast
+import os
+import platform
+import sys
+import weakref
+
+_builtin_compile = compile
+
+try:
+    from platform import python_implementation
+except ImportError:
+    if os.name == "java":
+        # Jython 2.5 has ast module, but not platform.python_implementation() function.
+        def python_implementation():
+            return "Jython"
+    else:
+        raise
+
+
+# restricted set of variables
+_VARS = {'sys.platform': sys.platform,
+         'python_version': '%s.%s' % sys.version_info[:2],
+         # FIXME parsing sys.platform is not reliable, but there is no other
+         # way to get e.g. 2.7.2+, and the PEP is defined with sys.version
+         'python_full_version': sys.version.split(' ', 1)[0],
+         'os.name': os.name,
+         'platform.version': platform.version(),
+         'platform.machine': platform.machine(),
+         'platform.python_implementation': python_implementation(),
+         'extra': None # wheel extension
+        }
+
+for var in list(_VARS.keys()):
+    if '.' in var:
+        _VARS[var.replace('.', '_')] = _VARS[var]
+
+def default_environment():
+    """Return copy of default PEP 385 globals dictionary."""
+    return dict(_VARS)
+
+class ASTWhitelist(ast.NodeTransformer):
+    def __init__(self, statement):
+        self.statement = statement # for error messages
+
+    ALLOWED = (ast.Compare, ast.BoolOp, ast.Attribute, ast.Name, ast.Load, ast.Str)
+    # Bool operations
+    ALLOWED += (ast.And, ast.Or)
+    # Comparison operations
+    ALLOWED += (ast.Eq, ast.Gt, ast.GtE, ast.In, ast.Is, ast.IsNot, ast.Lt, ast.LtE, ast.NotEq, ast.NotIn)
+
+    def visit(self, node):
+        """Ensure statement only contains allowed nodes."""
+        if not isinstance(node, self.ALLOWED):
+            raise SyntaxError('Not allowed in environment markers.\n%s\n%s' %
+                               (self.statement,
+                               (' ' * node.col_offset) + '^'))
+        return ast.NodeTransformer.visit(self, node)
+
+    def visit_Attribute(self, node):
+        """Flatten one level of attribute access."""
+        new_node = ast.Name("%s.%s" % (node.value.id, node.attr), node.ctx)
+        return ast.copy_location(new_node, node)
+
+def parse_marker(marker):
+    tree = ast.parse(marker, mode='eval')
+    new_tree = ASTWhitelist(marker).generic_visit(tree)
+    return new_tree
+
+def compile_marker(parsed_marker):
+    return _builtin_compile(parsed_marker, '<environment marker>', 'eval',
+                   dont_inherit=True)
+
+_cache = weakref.WeakValueDictionary()
+
+def compile(marker):
+    """Return compiled marker as a function accepting an environment dict."""
+    try:
+        return _cache[marker]
+    except KeyError:
+        pass
+    if not marker.strip():
+        def marker_fn(environment=None, override=None):
+            """"""
+            return True
+    else:
+        compiled_marker = compile_marker(parse_marker(marker))
+        def marker_fn(environment=None, override=None):
+            """override updates environment"""
+            if override is None:
+                override = {}
+            if environment is None:
+                environment = default_environment()
+            environment.update(override)
+            return eval(compiled_marker, environment)
+    marker_fn.__doc__ = marker
+    _cache[marker] = marker_fn
+    return _cache[marker]
+
+def interpret(marker, environment=None):
+    return compile(marker)(environment)

Contents/Libraries/Shared/asio/__init__.py

@@ -0,0 +1,61 @@
+# Copyright 2013 Dean Gardiner <gardiner91@gmail.com>
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from asio.file import SEEK_ORIGIN_CURRENT
+from asio.file_opener import FileOpener
+from asio.open_parameters import OpenParameters
+from asio.interfaces.posix import PosixInterface
+from asio.interfaces.windows import WindowsInterface
+
+import os
+
+
+class ASIO(object):
+    platform_handler = None
+
+    @classmethod
+    def get_handler(cls):
+        if cls.platform_handler:
+            return cls.platform_handler
+
+        if os.name == 'nt':
+            cls.platform_handler = WindowsInterface
+        elif os.name == 'posix':
+            cls.platform_handler = PosixInterface
+        else:
+            raise NotImplementedError()
+
+        return cls.platform_handler
+
+    @classmethod
+    def open(cls, file_path, opener=True, parameters=None):
+        """Open file
+
+        :type file_path: str
+
+        :param opener: Use FileOpener, for use with the 'with' statement
+        :type opener: bool
+
+        :rtype: asio.file.File
+        """
+        if not parameters:
+            parameters = OpenParameters()
+
+        if opener:
+            return FileOpener(file_path, parameters)
+
+        return ASIO.get_handler().open(
+            file_path,
+            parameters=parameters.handlers.get(ASIO.get_handler())
+        )

Contents/Libraries/Shared/asio/file.py

@@ -0,0 +1,92 @@
+# Copyright 2013 Dean Gardiner <gardiner91@gmail.com>
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from io import RawIOBase
+import time
+
+DEFAULT_BUFFER_SIZE = 4096
+
+SEEK_ORIGIN_BEGIN = 0
+SEEK_ORIGIN_CURRENT = 1
+SEEK_ORIGIN_END = 2
+
+
+class ReadTimeoutError(Exception):
+    pass
+
+
+class File(RawIOBase):
+    platform_handler = None
+
+    def __init__(self, *args, **kwargs):
+        super(File, self).__init__(*args, **kwargs)
+
+    def get_handler(self):
+        """
+        :rtype: asio.interfaces.base.Interface
+        """
+        if not self.platform_handler:
+            raise ValueError()
+
+        return self.platform_handler
+
+    def get_size(self):
+        """Get the current file size
+
+        :rtype: int
+        """
+        return self.get_handler().get_size(self)
+
+    def get_path(self):
+        """Get the path of this file
+
+        :rtype: str
+        """
+        return self.get_handler().get_path(self)
+
+    def seek(self, offset, origin):
+        """Sets a reference point of a file to the given value.
+
+        :param offset: The point relative to origin to move
+        :type offset: int
+
+        :param origin: Reference point to seek (SEEK_ORIGIN_BEGIN, SEEK_ORIGIN_CURRENT, SEEK_ORIGIN_END)
+        :type origin: int
+        """
+        return self.get_handler().seek(self, offset, origin)
+
+    def read(self, n=-1):
+        """Read up to n bytes from the object and return them.
+
+        :type n: int
+        :rtype: str
+        """
+        return self.get_handler().read(self, n)
+
+    def readinto(self, b):
+        """Read up to len(b) bytes into bytearray b and return the number of bytes read."""
+        data = self.read(len(b))
+
+        if data is None:
+            return None
+
+        b[:len(data)] = data
+        return len(data)
+
+    def close(self):
+        """Close the file handle"""
+        return self.get_handler().close(self)
+
+    def readable(self, *args, **kwargs):
+        return True

Contents/Libraries/Shared/asio/file_opener.py

@@ -0,0 +1,21 @@
+class FileOpener(object):
+    def __init__(self, file_path, parameters=None):
+        self.file_path = file_path
+        self.parameters = parameters
+
+        self.file = None
+
+    def __enter__(self):
+        self.file = ASIO.get_handler().open(
+            self.file_path,
+            self.parameters.handlers.get(ASIO.get_handler())
+        )
+
+        return self.file
+
+    def __exit__(self, exc_type, exc_val, exc_tb):
+        if not self.file:
+            return
+
+        self.file.close()
+        self.file = None

Contents/Libraries/Shared/asio/interfaces/base.py

@@ -0,0 +1,41 @@
+# Copyright 2013 Dean Gardiner <gardiner91@gmail.com>
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from asio.file import DEFAULT_BUFFER_SIZE
+
+
+class Interface(object):
+    @classmethod
+    def open(cls, file_path, parameters=None):
+        raise NotImplementedError()
+
+    @classmethod
+    def get_size(cls, fp):
+        raise NotImplementedError()
+
+    @classmethod
+    def get_path(cls, fp):
+        raise NotImplementedError()
+
+    @classmethod
+    def seek(cls, fp, pointer, distance):
+        raise NotImplementedError()
+
+    @classmethod
+    def read(cls, fp, n=DEFAULT_BUFFER_SIZE):
+        raise NotImplementedError()
+
+    @classmethod
+    def close(cls, fp):
+        raise NotImplementedError()

Contents/Libraries/Shared/asio/interfaces/posix.py

@@ -0,0 +1,123 @@
+# Copyright 2013 Dean Gardiner <gardiner91@gmail.com>
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from asio.file import File, DEFAULT_BUFFER_SIZE
+from asio.interfaces.base import Interface
+
+import sys
+import os
+
+if os.name == 'posix':
+    import select
+
+    # fcntl is only required on darwin
+    if sys.platform == 'darwin':
+        import fcntl
+
+F_GETPATH = 50
+
+
+class PosixInterface(Interface):
+    @classmethod
+    def open(cls, file_path, parameters=None):
+        """
+        :type file_path: str
+        :rtype: asio.interfaces.posix.PosixFile
+        """
+        if not parameters:
+            parameters = {}
+
+        if not parameters.get('mode'):
+            parameters.pop('mode')
+
+        if not parameters.get('buffering'):
+            parameters.pop('buffering')
+
+        fd = os.open(file_path, os.O_RDONLY | os.O_NONBLOCK)
+
+        return PosixFile(fd)
+
+    @classmethod
+    def get_size(cls, fp):
+        """
+        :type fp: asio.interfaces.posix.PosixFile
+        :rtype: int
+        """
+        return os.fstat(fp.fd).st_size
+
+    @classmethod
+    def get_path(cls, fp):
+        """
+        :type fp: asio.interfaces.posix.PosixFile
+        :rtype: int
+        """
+
+        # readlink /dev/fd fails on darwin, so instead use fcntl F_GETPATH
+        if sys.platform == 'darwin':
+            return fcntl.fcntl(fp.fd, F_GETPATH, '\0' * 1024).rstrip('\0')
+
+        # Use /proc/self/fd if available
+        if os.path.lexists("/proc/self/fd/"):
+            return os.readlink("/proc/self/fd/%s" % fp.fd)
+
+        # Fallback to /dev/fd
+        if os.path.lexists("/dev/fd/"):
+            return os.readlink("/dev/fd/%s" % fp.fd)
+
+        raise NotImplementedError('Environment not supported (fdescfs not mounted?)')
+
+    @classmethod
+    def seek(cls, fp, offset, origin):
+        """
+        :type fp: asio.interfaces.posix.PosixFile
+        :type offset: int
+        :type origin: int
+        """
+        os.lseek(fp.fd, offset, origin)
+
+    @classmethod
+    def read(cls, fp, n=DEFAULT_BUFFER_SIZE):
+        """
+        :type fp: asio.interfaces.posix.PosixFile
+        :type n: int
+        :rtype: str
+        """
+        r, w, x = select.select([fp.fd], [], [], 5)
+
+        if r:
+            return os.read(fp.fd, n)
+
+        return None
+
+    @classmethod
+    def close(cls, fp):
+        """
+        :type fp: asio.interfaces.posix.PosixFile
+        """
+        os.close(fp.fd)
+
+
+class PosixFile(File):
+    platform_handler = PosixInterface
+
+    def __init__(self, fd, *args, **kwargs):
+        """
+        :type fd: asio.file.File
+        """
+        super(PosixFile, self).__init__(*args, **kwargs)
+
+        self.fd = fd
+
+    def __str__(self):
+        return "<asio_posix.PosixFile file: %s>" % self.fd

Contents/Libraries/Shared/asio/interfaces/windows/__init__.py

@@ -0,0 +1,201 @@
+# Copyright 2013 Dean Gardiner <gardiner91@gmail.com>
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from asio.file import File, DEFAULT_BUFFER_SIZE
+from asio.interfaces.base import Interface
+
+import os
+
+
+NULL = 0
+
+if os.name == 'nt':
+    from asio.interfaces.windows.interop import WindowsInterop
+
+
+class WindowsInterface(Interface):
+    @classmethod
+    def open(cls, file_path, parameters=None):
+        """
+        :type file_path: str
+        :rtype: asio.interfaces.windows.WindowsFile
+        """
+        if not parameters:
+            parameters = {}
+
+        return WindowsFile(WindowsInterop.create_file(
+            file_path,
+            parameters.get('desired_access', WindowsInterface.GenericAccess.READ),
+            parameters.get('share_mode', WindowsInterface.ShareMode.ALL),
+            parameters.get('creation_disposition', WindowsInterface.CreationDisposition.OPEN_EXISTING),
+            parameters.get('flags_and_attributes', NULL)
+        ))
+
+    @classmethod
+    def get_size(cls, fp):
+        """
+        :type fp: asio.interfaces.windows.WindowsFile
+        :rtype: int
+        """
+        return WindowsInterop.get_file_size(fp.handle)
+
+    @classmethod
+    def get_path(cls, fp):
+        """
+        :type fp: asio.interfaces.windows.WindowsFile
+        :rtype: str
+        """
+
+        if not fp.file_map:
+            fp.file_map = WindowsInterop.create_file_mapping(fp.handle, WindowsInterface.Protection.READONLY)
+
+        if not fp.map_view:
+            fp.map_view = WindowsInterop.map_view_of_file(fp.file_map, WindowsInterface.FileMapAccess.READ, 1)
+
+        file_name = WindowsInterop.get_mapped_file_name(fp.map_view)
+
+        return file_name
+
+    @classmethod
+    def seek(cls, fp, offset, origin):
+        """
+        :type fp: asio.interfaces.windows.WindowsFile
+        :type offset: int
+        :type origin: int
+        :rtype: int
+        """
+
+        return WindowsInterop.set_file_pointer(
+            fp.handle,
+            offset,
+            origin
+        )
+
+    @classmethod
+    def read(cls, fp, n=DEFAULT_BUFFER_SIZE):
+        """
+        :type fp: asio.interfaces.windows.WindowsFile
+        :type n: int
+        :rtype: str
+        """
+        return WindowsInterop.read(fp.handle, n)
+
+    @classmethod
+    def read_into(cls, fp, b):
+        """
+        :type fp: asio.interfaces.windows.WindowsFile
+        :type b: str
+        :rtype: int
+        """
+        return WindowsInterop.read_into(fp.handle, b)
+
+    @classmethod
+    def close(cls, fp):
+        """
+        :type fp: asio.interfaces.windows.WindowsFile
+        :rtype: bool
+        """
+        if fp.map_view:
+            WindowsInterop.unmap_view_of_file(fp.map_view)
+
+        if fp.file_map:
+            WindowsInterop.close_handle(fp.file_map)
+
+        return bool(WindowsInterop.close_handle(fp.handle))
+
+    class GenericAccess(object):
+        READ = 0x80000000
+        WRITE = 0x40000000
+        EXECUTE = 0x20000000
+        ALL = 0x10000000
+
+    class ShareMode(object):
+        READ = 0x00000001
+        WRITE = 0x00000002
+        DELETE = 0x00000004
+        ALL = READ | WRITE | DELETE
+
+    class CreationDisposition(object):
+        CREATE_NEW = 1
+        CREATE_ALWAYS = 2
+        OPEN_EXISTING = 3
+        OPEN_ALWAYS = 4
+        TRUNCATE_EXISTING = 5
+
+    class Attribute(object):
+        READONLY = 0x00000001
+        HIDDEN = 0x00000002
+        SYSTEM = 0x00000004
+        DIRECTORY = 0x00000010
+        ARCHIVE = 0x00000020
+        DEVICE = 0x00000040
+        NORMAL = 0x00000080
+        TEMPORARY = 0x00000100
+        SPARSE_FILE = 0x00000200
+        REPARSE_POINT = 0x00000400
+        COMPRESSED = 0x00000800
+        OFFLINE = 0x00001000
+        NOT_CONTENT_INDEXED = 0x00002000
+        ENCRYPTED = 0x00004000
+
+    class Flag(object):
+        WRITE_THROUGH = 0x80000000
+        OVERLAPPED = 0x40000000
+        NO_BUFFERING = 0x20000000
+        RANDOM_ACCESS = 0x10000000
+        SEQUENTIAL_SCAN = 0x08000000
+        DELETE_ON_CLOSE = 0x04000000
+        BACKUP_SEMANTICS = 0x02000000
+        POSIX_SEMANTICS = 0x01000000
+        OPEN_REPARSE_POINT = 0x00200000
+        OPEN_NO_RECALL = 0x00100000
+        FIRST_PIPE_INSTANCE = 0x00080000
+
+    class Protection(object):
+        NOACCESS = 0x01
+        READONLY = 0x02
+        READWRITE = 0x04
+        WRITECOPY = 0x08
+        EXECUTE = 0x10
+        EXECUTE_READ = 0x20,
+        EXECUTE_READWRITE = 0x40
+        EXECUTE_WRITECOPY = 0x80
+        GUARD = 0x100
+        NOCACHE = 0x200
+        WRITECOMBINE = 0x400
+
+    class FileMapAccess(object):
+        COPY = 0x0001
+        WRITE = 0x0002
+        READ = 0x0004
+        ALL_ACCESS = 0x001f
+        EXECUTE = 0x0020
+
+
+class WindowsFile(File):
+    platform_handler = WindowsInterface
+
+    def __init__(self, handle, *args, **kwargs):
+        super(WindowsFile, self).__init__(*args, **kwargs)
+
+        self.handle = handle
+
+        self.file_map = None
+        self.map_view = None
+
+    def readinto(self, b):
+        return self.get_handler().read_into(self, b)
+
+    def __str__(self):
+        return "<asio_windows.WindowsFile file: %s>" % self.handle

Contents/Libraries/Shared/asio/interfaces/windows/interop.py

@@ -0,0 +1,230 @@
+# Copyright 2013 Dean Gardiner <gardiner91@gmail.com>
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from ctypes.wintypes import *
+from ctypes import *
+import logging
+
+log = logging.getLogger(__name__)
+
+
+CreateFileW = windll.kernel32.CreateFileW
+CreateFileW.argtypes = (LPCWSTR, DWORD, DWORD, c_void_p, DWORD, DWORD, HANDLE)
+CreateFileW.restype = HANDLE
+
+ReadFile = windll.kernel32.ReadFile
+ReadFile.argtypes = (HANDLE, c_void_p, DWORD, POINTER(DWORD), HANDLE)
+ReadFile.restype = BOOL
+
+
+NULL = 0
+MAX_PATH = 260
+DEFAULT_BUFFER_SIZE = 4096
+LPSECURITY_ATTRIBUTES = c_void_p
+
+
+class WindowsInterop(object):
+    ri_buffer = None
+
+    @classmethod
+    def create_file(cls, path, desired_access, share_mode, creation_disposition, flags_and_attributes):
+        h = CreateFileW(
+            path,
+            desired_access,
+            share_mode,
+            NULL,
+            creation_disposition,
+            flags_and_attributes,
+            NULL
+        )
+
+        error = GetLastError()
+        if error != 0:
+            raise Exception('[WindowsASIO.open] "%s"' % FormatError(error))
+
+        return h
+
+    @classmethod
+    def read(cls, handle, buf_size=DEFAULT_BUFFER_SIZE):
+        buf = create_string_buffer(buf_size)
+        bytes_read = c_ulong(0)
+
+        success = ReadFile(handle, buf, buf_size, byref(bytes_read), NULL)
+
+        error = GetLastError()
+        if error:
+            log.debug('read_file - error: (%s) "%s"', error, FormatError(error))
+
+        if not success and error:
+            raise Exception('[WindowsInterop.read_file] (%s) "%s"' % (error, FormatError(error)))
+
+        # Return if we have a valid buffer
+        if success and bytes_read.value:
+            return buf.value
+
+        return None
+
+    @classmethod
+    def read_into(cls, handle, b):
+        if cls.ri_buffer is None or len(cls.ri_buffer) < len(b):
+            cls.ri_buffer = create_string_buffer(len(b))
+
+        bytes_read = c_ulong(0)
+
+        success = ReadFile(handle, cls.ri_buffer, len(b), byref(bytes_read), NULL)
+        bytes_read = int(bytes_read.value)
+
+        b[:bytes_read] = cls.ri_buffer[:bytes_read]
+
+        error = GetLastError()
+        
+        if not success and error:
+            raise Exception('[WindowsInterop.read_file] (%s) "%s"' % (error, FormatError(error)))
+
+        # Return if we have a valid buffer
+        if success and bytes_read:
+            return bytes_read
+
+        return None
+
+    @classmethod
+    def set_file_pointer(cls, handle, distance, method):
+        pos_high = DWORD(NULL)
+
+        result = windll.kernel32.SetFilePointer(
+            handle,
+            c_ulong(distance),
+            byref(pos_high),
+            DWORD(method)
+        )
+
+        if result == -1:
+            raise Exception('[WindowsASIO.seek] INVALID_SET_FILE_POINTER: "%s"' % FormatError(GetLastError()))
+
+        return result
+
+    @classmethod
+    def get_file_size(cls, handle):
+        return windll.kernel32.GetFileSize(
+            handle,
+            DWORD(NULL)
+        )
+
+    @classmethod
+    def close_handle(cls, handle):
+        return windll.kernel32.CloseHandle(handle)
+
+    @classmethod
+    def create_file_mapping(cls, handle, protect, maximum_size_high=0, maximum_size_low=1):
+        return HANDLE(windll.kernel32.CreateFileMappingW(
+            handle,
+            LPSECURITY_ATTRIBUTES(NULL),
+            DWORD(protect),
+            DWORD(maximum_size_high),
+            DWORD(maximum_size_low),
+            LPCSTR(NULL)
+        ))
+
+    @classmethod
+    def map_view_of_file(cls, map_handle, desired_access, num_bytes, file_offset_high=0, file_offset_low=0):
+        return HANDLE(windll.kernel32.MapViewOfFile(
+            map_handle,
+            DWORD(desired_access),
+            DWORD(file_offset_high),
+            DWORD(file_offset_low),
+            num_bytes
+        ))
+
+    @classmethod
+    def unmap_view_of_file(cls, view_handle):
+        return windll.kernel32.UnmapViewOfFile(view_handle)
+
+    @classmethod
+    def get_mapped_file_name(cls, view_handle, translate_device_name=True):
+        buf = create_string_buffer(MAX_PATH + 1)
+
+        result = windll.psapi.GetMappedFileNameW(
+            cls.get_current_process(),
+            view_handle,
+            buf,
+            MAX_PATH
+        )
+
+        # Raise exception on error
+        error = GetLastError()
+        if result == 0:
+            raise Exception(FormatError(error))
+
+        # Retrieve a clean file name (skipping over NUL bytes)
+        file_name = cls.clean_buffer_value(buf)
+
+        # If we are not translating the device name return here
+        if not translate_device_name:
+            return file_name
+
+        drives = cls.get_logical_drive_strings()
+
+        # Find the drive matching the file_name device name
+        translated = False
+        for drive in drives:
+            device_name = cls.query_dos_device(drive)
+
+            if file_name.startswith(device_name):
+                file_name = drive + file_name[len(device_name):]
+                translated = True
+                break
+
+        if not translated:
+            raise Exception('Unable to translate device name')
+
+        return file_name
+
+    @classmethod
+    def get_logical_drive_strings(cls, buf_size=512):
+        buf = create_string_buffer(buf_size)
+
+        result = windll.kernel32.GetLogicalDriveStringsW(buf_size, buf)
+
+        error = GetLastError()
+        if result == 0:
+            raise Exception(FormatError(error))
+
+        drive_strings = cls.clean_buffer_value(buf)
+        return [dr for dr in drive_strings.split('\\') if dr != '']
+
+    @classmethod
+    def query_dos_device(cls, drive, buf_size=MAX_PATH):
+        buf = create_string_buffer(buf_size)
+
+        result = windll.kernel32.QueryDosDeviceA(
+            drive,
+            buf,
+            buf_size
+        )
+
+        return cls.clean_buffer_value(buf)
+
+    @classmethod
+    def get_current_process(cls):
+        return HANDLE(windll.kernel32.GetCurrentProcess())
+
+    @classmethod
+    def clean_buffer_value(cls, buf):
+        value = ""
+
+        for ch in buf.raw:
+            if ord(ch) != 0:
+                value += ch
+
+        return value

Contents/Libraries/Shared/asio/open_parameters.py

@@ -0,0 +1,47 @@
+from asio.interfaces.posix import PosixInterface
+from asio.interfaces.windows import WindowsInterface
+
+
+class OpenParameters(object):
+    def __init__(self):
+        self.handlers = {}
+
+        # Update handler_parameters with defaults
+        self.posix()
+        self.windows()
+
+    def posix(self, mode=None, buffering=None):
+        """
+        :type mode: str
+        :type buffering: int
+        """
+        self.handlers.update({PosixInterface: {
+            'mode': mode,
+            'buffering': buffering
+        }})
+
+    def windows(self, desired_access=WindowsInterface.GenericAccess.READ,
+                share_mode=WindowsInterface.ShareMode.ALL,
+                creation_disposition=WindowsInterface.CreationDisposition.OPEN_EXISTING,
+                flags_and_attributes=0):
+
+        """
+        :param desired_access: WindowsInterface.DesiredAccess
+        :type desired_access: int
+
+        :param share_mode: WindowsInterface.ShareMode
+        :type share_mode: int
+
+        :param creation_disposition: WindowsInterface.CreationDisposition
+        :type creation_disposition: int
+
+        :param flags_and_attributes: WindowsInterface.Attribute, WindowsInterface.Flag
+        :type flags_and_attributes: int
+        """
+
+        self.handlers.update({WindowsInterface: {
+            'desired_access': desired_access,
+            'share_mode': share_mode,
+            'creation_disposition': creation_disposition,
+            'flags_and_attributes': flags_and_attributes
+        }})

Contents/Libraries/Shared/bs4/COPYING.txt

@@ -1,6 +1,6 @@
 Beautiful Soup is made available under the MIT license:
 
- Copyright (c) 2004-2012 Leonard Richardson
+ Copyright (c) 2004-2015 Leonard Richardson
 
  Permission is hereby granted, free of charge, to any person obtaining
  a copy of this software and associated documentation files (the
@@ -20,7 +20,8 @@ Beautiful Soup is made available under the MIT license:
  BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
  ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
  CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- SOFTWARE, DAMMIT.
+ SOFTWARE.
 
 Beautiful Soup incorporates code from the html5lib library, which is
-also made available under the MIT license.
+also made available under the MIT license. Copyright (c) 2006-2013
+James Graham and other contributors

Contents/Libraries/Shared/bs4/NEWS.txt

@@ -1,3 +1,127 @@
+= 4.4.1 (20150928) =
+
+* Fixed a bug that deranged the tree when part of it was
+  removed. Thanks to Eric Weiser for the patch and John Wiseman for a
+  test. [bug=1481520]
+
+* Fixed a parse bug with the html5lib tree-builder. Thanks to Roel
+  Kramer for the patch. [bug=1483781]
+
+* Improved the implementation of CSS selector grouping. Thanks to
+  Orangain for the patch. [bug=1484543]
+
+* Fixed the test_detect_utf8 test so that it works when chardet is
+  installed. [bug=1471359]
+
+* Corrected the output of Declaration objects. [bug=1477847]
+
+
+= 4.4.0 (20150703) =
+
+Especially important changes:
+
+* Added a warning when you instantiate a BeautifulSoup object without
+  explicitly naming a parser. [bug=1398866]
+
+* __repr__ now returns an ASCII bytestring in Python 2, and a Unicode
+  string in Python 3, instead of a UTF8-encoded bytestring in both
+  versions. In Python 3, __str__ now returns a Unicode string instead
+  of a bytestring. [bug=1420131]
+
+* The `text` argument to the find_* methods is now called `string`,
+  which is more accurate. `text` still works, but `string` is the
+  argument described in the documentation. `text` may eventually
+  change its meaning, but not for a very long time. [bug=1366856]
+
+* Changed the way soup objects work under copy.copy(). Copying a
+  NavigableString or a Tag will give you a new NavigableString that's
+  equal to the old one but not connected to the parse tree. Patch by
+  Martijn Peters. [bug=1307490]
+
+* Started using a standard MIT license. [bug=1294662]
+
+* Added a Chinese translation of the documentation by Delong .w.
+
+New features:
+
+* Introduced the select_one() method, which uses a CSS selector but
+  only returns the first match, instead of a list of
+  matches. [bug=1349367]
+
+* You can now create a Tag object without specifying a
+  TreeBuilder. Patch by Martijn Pieters. [bug=1307471]
+
+* You can now create a NavigableString or a subclass just by invoking
+  the constructor. [bug=1294315]
+
+* Added an `exclude_encodings` argument to UnicodeDammit and to the
+  Beautiful Soup constructor, which lets you prohibit the detection of
+  an encoding that you know is wrong. [bug=1469408]
+
+* The select() method now supports selector grouping. Patch by
+  Francisco Canas [bug=1191917]
+
+Bug fixes:
+
+* Fixed yet another problem that caused the html5lib tree builder to
+  create a disconnected parse tree. [bug=1237763]
+
+* Force object_was_parsed() to keep the tree intact even when an element
+  from later in the document is moved into place. [bug=1430633]
+
+* Fixed yet another bug that caused a disconnected tree when html5lib
+  copied an element from one part of the tree to another. [bug=1270611]
+
+* Fixed a bug where Element.extract() could create an infinite loop in
+  the remaining tree.
+
+* The select() method can now find tags whose names contain
+  dashes. Patch by Francisco Canas. [bug=1276211]
+
+* The select() method can now find tags with attributes whose names
+  contain dashes. Patch by Marek Kapolka. [bug=1304007]
+
+* Improved the lxml tree builder's handling of processing
+  instructions. [bug=1294645]
+
+* Restored the helpful syntax error that happens when you try to
+  import the Python 2 edition of Beautiful Soup under Python
+  3. [bug=1213387]
+
+* In Python 3.4 and above, set the new convert_charrefs argument to
+  the html.parser constructor to avoid a warning and future
+  failures. Patch by Stefano Revera. [bug=1375721]
+
+* The warning when you pass in a filename or URL as markup will now be
+  displayed correctly even if the filename or URL is a Unicode
+  string. [bug=1268888]
+
+* If the initial <html> tag contains a CDATA list attribute such as
+  'class', the html5lib tree builder will now turn its value into a
+  list, as it would with any other tag. [bug=1296481]
+
+* Fixed an import error in Python 3.5 caused by the removal of the
+  HTMLParseError class. [bug=1420063]
+
+* Improved docstring for encode_contents() and
+  decode_contents(). [bug=1441543]
+
+* Fixed a crash in Unicode, Dammit's encoding detector when the name
+  of the encoding itself contained invalid bytes. [bug=1360913]
+
+* Improved the exception raised when you call .unwrap() or
+  .replace_with() on an element that's not attached to a tree.
+
+* Raise a NotImplementedError whenever an unsupported CSS pseudoclass
+  is used in select(). Previously some cases did not result in a
+  NotImplementedError.
+
+* It's now possible to pickle a BeautifulSoup object no matter which
+  tree builder was used to create it. However, the only tree builder
+  that survives the pickling process is the HTMLParserTreeBuilder
+  ('html.parser'). If you unpickle a BeautifulSoup object created with
+  some other tree builder, soup.builder will be None. [bug=1231545]
+
 = 4.3.2 (20131002) =
 
 * Fixed a bug in which short Unicode input was improperly encoded to

Contents/Libraries/Shared/bs4/TODO.txt

@@ -0,0 +1,31 @@
+Additions
+---------
+
+More of the jQuery API: nextUntil?
+
+Optimizations
+-------------
+
+The html5lib tree builder doesn't use the standard tree-building API,
+which worries me and has resulted in a number of bugs.
+
+markup_attr_map can be optimized since it's always a map now.
+
+Upon encountering UTF-16LE data or some other uncommon serialization
+of Unicode, UnicodeDammit will convert the data to Unicode, then
+encode it at UTF-8. This is wasteful because it will just get decoded
+back to Unicode.
+
+CDATA
+-----
+
+The elementtree XMLParser has a strip_cdata argument that, when set to
+False, should allow Beautiful Soup to preserve CDATA sections instead
+of treating them as text. Except it doesn't. (This argument is also
+present for HTMLParser, and also does nothing there.)
+
+Currently, htm5lib converts CDATA sections into comments. An
+as-yet-unreleased version of html5lib changes the parser's handling of
+CDATA sections to allow CDATA sections in tags like <svg> and
+<math>. The HTML5TreeBuilder will need to be updated to create CData
+objects instead of Comment objects in this situation.

Contents/Libraries/Shared/bs4/__init__.py

@@ -17,8 +17,8 @@ http://www.crummy.com/software/BeautifulSoup/bs4/doc/
 """
 
 __author__ = "Leonard Richardson (leonardr@segfault.org)"
-__version__ = "4.3.2"
-__copyright__ = "Copyright (c) 2004-2013 Leonard Richardson"
+__version__ = "4.4.1"
+__copyright__ = "Copyright (c) 2004-2015 Leonard Richardson"
 __license__ = "MIT"
 
 __all__ = ['BeautifulSoup']
@@ -45,7 +45,7 @@ from .element import (
 
 # The very first thing we do is give a useful error if someone is
 # running this code under Python 3 without converting it.
-syntax_error = u'You are trying to run the Python 2 version of Beautiful Soup under Python 3. This will not work. You need to convert the code, either by installing it (`python setup.py install`) or by running 2to3 (`2to3 -w bs4`).'
+'You are trying to run the Python 2 version of Beautiful Soup under Python 3. This will not work.'<>'You need to convert the code, either by installing it (`python setup.py install`) or by running 2to3 (`2to3 -w bs4`).'
 
 class BeautifulSoup(Tag):
     """
@@ -77,8 +77,11 @@ class BeautifulSoup(Tag):
 
     ASCII_SPACES = '\x20\x0a\x09\x0c\x0d'
 
+    NO_PARSER_SPECIFIED_WARNING = "No parser was explicitly specified, so I'm using the best available %(markup_type)s parser for this system (\"%(parser)s\"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.\n\nTo get rid of this warning, change this:\n\n BeautifulSoup([your markup])\n\nto this:\n\n BeautifulSoup([your markup], \"%(parser)s\")\n"
+
     def __init__(self, markup="", features=None, builder=None,
-                 parse_only=None, from_encoding=None, **kwargs):
+                 parse_only=None, from_encoding=None, exclude_encodings=None,
+                 **kwargs):
         """The Soup object is initialized as the 'root tag', and the
         provided markup (which can be a string or a file-like object)
         is fed into the underlying parser."""
@@ -114,9 +117,9 @@ class BeautifulSoup(Tag):
             del kwargs['isHTML']
             warnings.warn(
                 "BS4 does not respect the isHTML argument to the "
-                "BeautifulSoup constructor. You can pass in features='html' "
-                "or features='xml' to get a builder capable of handling "
-                "one or the other.")
+                "BeautifulSoup constructor. Suggest you use "
+                "features='lxml' for HTML and features='lxml-xml' for "
+                "XML.")
 
         def deprecated_argument(old_name, new_name):
             if old_name in kwargs:
@@ -140,6 +143,7 @@ class BeautifulSoup(Tag):
                 "__init__() got an unexpected keyword argument '%s'" % arg)
 
         if builder is None:
+            original_features = features
             if isinstance(features, basestring):
                 features = [features]
             if features is None or len(features) == 0:
@@ -151,6 +155,16 @@ class BeautifulSoup(Tag):
                     "requested: %s. Do you need to install a parser library?"
                     % ",".join(features))
             builder = builder_class()
+            if not (original_features == builder.NAME or
+                    original_features in builder.ALTERNATE_NAMES):
+                if builder.is_xml:
+                    markup_type = "XML"
+                else:
+                    markup_type = "HTML"
+                warnings.warn(self.NO_PARSER_SPECIFIED_WARNING % dict(
+                    parser=builder.NAME,
+                    markup_type=markup_type))
+
         self.builder = builder
         self.is_xml = builder.is_xml
         self.builder.soup = self
@@ -178,6 +192,8 @@ class BeautifulSoup(Tag):
                 # system. Just let it go.
                 pass
             if is_file:
+                if isinstance(markup, unicode):
+                    markup = markup.encode("utf8")
                 warnings.warn(
                     '"%s" looks like a filename, not markup. You should probably open this file and pass the filehandle into Beautiful Soup.' % markup)
             if markup[:5] == "http:" or markup[:6] == "https:":
@@ -185,12 +201,15 @@ class BeautifulSoup(Tag):
                 # Python 3 otherwise.
                 if ((isinstance(markup, bytes) and not b' ' in markup)
                     or (isinstance(markup, unicode) and not u' ' in markup)):
+                    if isinstance(markup, unicode):
+                        markup = markup.encode("utf8")
                     warnings.warn(
                         '"%s" looks like a URL. Beautiful Soup is not an HTTP client. You should probably use an HTTP client to get the document behind the URL, and feed that document to Beautiful Soup.' % markup)
 
         for (self.markup, self.original_encoding, self.declared_html_encoding,
          self.contains_replacement_characters) in (
-            self.builder.prepare_markup(markup, from_encoding)):
+             self.builder.prepare_markup(
+                 markup, from_encoding, exclude_encodings=exclude_encodings)):
             self.reset()
             try:
                 self._feed()
@@ -203,6 +222,16 @@ class BeautifulSoup(Tag):
         self.markup = None
         self.builder.soup = None
 
+    def __copy__(self):
+        return type(self)(self.encode(), builder=self.builder)
+
+    def __getstate__(self):
+        # Frequently a tree builder can't be pickled.
+        d = dict(self.__dict__)
+        if 'builder' in d and not self.builder.picklable:
+            del d['builder']
+        return d
+
     def _feed(self):
         # Convert the document to Unicode.
         self.builder.reset()
@@ -229,9 +258,7 @@ class BeautifulSoup(Tag):
 
     def new_string(self, s, subclass=NavigableString):
         """Create a new NavigableString associated with this soup."""
-        navigable = subclass(s)
-        navigable.setup()
-        return navigable
+        return subclass(s)
 
     def insert_before(self, successor):
         raise NotImplementedError("BeautifulSoup objects don't support insert_before().")
@@ -290,14 +317,49 @@ class BeautifulSoup(Tag):
     def object_was_parsed(self, o, parent=None, most_recent_element=None):
         """Add an object to the parse tree."""
         parent = parent or self.currentTag
-        most_recent_element = most_recent_element or self._most_recent_element
-        o.setup(parent, most_recent_element)
+        previous_element = most_recent_element or self._most_recent_element
+
+        next_element = previous_sibling = next_sibling = None
+        if isinstance(o, Tag):
+            next_element = o.next_element
+            next_sibling = o.next_sibling
+            previous_sibling = o.previous_sibling
+            if not previous_element:
+                previous_element = o.previous_element
+
+        o.setup(parent, previous_element, next_element, previous_sibling, next_sibling)
 
-        if most_recent_element is not None:
-            most_recent_element.next_element = o
         self._most_recent_element = o
         parent.contents.append(o)
 
+        if parent.next_sibling:
+            # This node is being inserted into an element that has
+            # already been parsed. Deal with any dangling references.
+            index = parent.contents.index(o)
+            if index == 0:
+                previous_element = parent
+                previous_sibling = None
+            else:
+                previous_element = previous_sibling = parent.contents[index-1]
+            if index == len(parent.contents)-1:
+                next_element = parent.next_sibling
+                next_sibling = None
+            else:
+                next_element = next_sibling = parent.contents[index+1]
+
+            o.previous_element = previous_element
+            if previous_element:
+                previous_element.next_element = o
+            o.next_element = next_element
+            if next_element:
+                next_element.previous_element = o
+            o.next_sibling = next_sibling
+            if next_sibling:
+                next_sibling.previous_sibling = o
+            o.previous_sibling = previous_sibling
+            if previous_sibling:
+                previous_sibling.next_sibling = o
+
     def _popToTag(self, name, nsprefix=None, inclusivePop=True):
         """Pops the tag stack up to and including the most recent
         instance of the given tag. If inclusivePop is false, pops the tag

Contents/Libraries/Shared/bs4/builder/__init__.py

@@ -80,9 +80,12 @@ builder_registry = TreeBuilderRegistry()
 class TreeBuilder(object):
     """Turn a document into a Beautiful Soup object tree."""
 
+    NAME = "[Unknown tree builder]"
+    ALTERNATE_NAMES = []
     features = []
 
     is_xml = False
+    picklable = False
     preserve_whitespace_tags = set()
     empty_element_tags = None # A tag will be considered an empty-element
                               # tag when and only when it has no contents.

Contents/Libraries/Shared/bs4/builder/_html5lib.py

@@ -2,6 +2,7 @@ __all__ = [
     'HTML5TreeBuilder',
     ]
 
+from pdb import set_trace
 import warnings
 from bs4.builder import (
     PERMISSIVE,
@@ -9,7 +10,10 @@ from bs4.builder import (
     HTML_5,
     HTMLTreeBuilder,
     )
-from bs4.element import NamespacedAttribute
+from bs4.element import (
+    NamespacedAttribute,
+    whitespace_re,
+)
 import html5lib
 from html5lib.constants import namespaces
 from bs4.element import (
@@ -22,11 +26,20 @@ from bs4.element import (
 class HTML5TreeBuilder(HTMLTreeBuilder):
     """Use html5lib to build a tree."""
 
-    features = ['html5lib', PERMISSIVE, HTML_5, HTML]
+    NAME = "html5lib"
 
-    def prepare_markup(self, markup, user_specified_encoding):
+    features = [NAME, PERMISSIVE, HTML_5, HTML]
+
+    def prepare_markup(self, markup, user_specified_encoding,
+                       document_declared_encoding=None, exclude_encodings=None):
         # Store the user-specified encoding for use later on.
         self.user_specified_encoding = user_specified_encoding
+
+        # document_declared_encoding and exclude_encodings aren't used
+        # ATM because the html5lib TreeBuilder doesn't use
+        # UnicodeDammit.
+        if exclude_encodings:
+            warnings.warn("You provided a value for exclude_encoding, but the html5lib tree builder doesn't support exclude_encoding.")
         yield (markup, None, None, False)
 
     # These methods are defined by Beautiful Soup.
@@ -101,7 +114,16 @@ class AttrList(object):
     def __iter__(self):
         return list(self.attrs.items()).__iter__()
     def __setitem__(self, name, value):
-        "set attr", name, value
+        # If this attribute is a multi-valued attribute for this element,
+        # turn its value into a list.
+        list_attr = HTML5TreeBuilder.cdata_list_attributes
+        if (name in list_attr['*']
+            or (self.element.name in list_attr
+                and name in list_attr[self.element.name])):
+            # A node that is being cloned may have already undergone
+            # this procedure.
+            if not isinstance(value, list):
+                value = whitespace_re.split(value)
         self.element[name] = value
     def items(self):
         return list(self.attrs.items())
@@ -161,6 +183,12 @@ class Element(html5lib.treebuilders._base.Node):
             # immediately after the parent, if it has no children.)
             if self.element.contents:
                 most_recent_element = self.element._last_descendant(False)
+            elif self.element.next_element is not None:
+                # Something from further ahead in the parse tree is
+                # being inserted into this earlier element. This is
+                # very annoying because it means an expensive search
+                # for the last element in the tree.
+                most_recent_element = self.soup._last_descendant()
             else:
                 most_recent_element = self.element
 
@@ -172,6 +200,7 @@ class Element(html5lib.treebuilders._base.Node):
         return AttrList(self.element)
 
     def setAttributes(self, attributes):
+
         if attributes is not None and len(attributes) > 0:
 
             converted_attributes = []
@@ -218,6 +247,9 @@ class Element(html5lib.treebuilders._base.Node):
 
     def reparentChildren(self, new_parent):
         """Move all of this tag's children into another tag."""
+        # print "MOVE", self.element.contents
+        # print "FROM", self.element
+        # print "TO", new_parent.element
         element = self.element
         new_parent_element = new_parent.element
         # Determine what this tag's next_element will be once all the children
@@ -236,17 +268,28 @@ class Element(html5lib.treebuilders._base.Node):
             new_parents_last_descendant_next_element = new_parent_element.next_element
 
         to_append = element.contents
-        append_after = new_parent.element.contents
+        append_after = new_parent_element.contents
         if len(to_append) > 0:
             # Set the first child's previous_element and previous_sibling
             # to elements within the new parent
             first_child = to_append[0]
-            first_child.previous_element = new_parents_last_descendant
+            if new_parents_last_descendant:
+                first_child.previous_element = new_parents_last_descendant
+            else:
+                first_child.previous_element = new_parent_element
             first_child.previous_sibling = new_parents_last_child
+            if new_parents_last_descendant:
+                new_parents_last_descendant.next_element = first_child
+            else:
+                new_parent_element.next_element = first_child
+            if new_parents_last_child:
+                new_parents_last_child.next_sibling = first_child
 
             # Fix the last child's next_element and next_sibling
             last_child = to_append[-1]
             last_child.next_element = new_parents_last_descendant_next_element
+            if new_parents_last_descendant_next_element:
+                new_parents_last_descendant_next_element.previous_element = last_child
             last_child.next_sibling = None
 
         for child in to_append:
@@ -257,6 +300,10 @@ class Element(html5lib.treebuilders._base.Node):
         element.contents = []
         element.next_element = final_next_element
 
+        # print "DONE WITH MOVE"
+        # print "FROM", self.element
+        # print "TO", new_parent_element
+
     def cloneNode(self):
         tag = self.soup.new_tag(self.element.name, self.namespace)
         node = Element(tag, self.soup, self.namespace)

Contents/Libraries/Shared/bs4/builder/_htmlparser.py

@@ -4,10 +4,16 @@ __all__ = [
     'HTMLParserTreeBuilder',
     ]
 
-from HTMLParser import (
-    HTMLParser,
-    HTMLParseError,
-    )
+from HTMLParser import HTMLParser
+
+try:
+    from HTMLParser import HTMLParseError
+except ImportError, e:
+    # HTMLParseError is removed in Python 3.5. Since it can never be
+    # thrown in 3.5, we can just define our own class as a placeholder.
+    class HTMLParseError(Exception):
+        pass
+
 import sys
 import warnings
 
@@ -19,10 +25,10 @@ import warnings
 # At the end of this file, we monkeypatch HTMLParser so that
 # strict=True works well on Python 3.2.2.
 major, minor, release = sys.version_info[:3]
-CONSTRUCTOR_TAKES_STRICT = (
-    major > 3
-    or (major == 3 and minor > 2)
-    or (major == 3 and minor == 2 and release >= 3))
+CONSTRUCTOR_TAKES_STRICT = major == 3 and minor == 2 and release >= 3
+CONSTRUCTOR_STRICT_IS_DEPRECATED = major == 3 and minor == 3
+CONSTRUCTOR_TAKES_CONVERT_CHARREFS = major == 3 and minor >= 4
+
 
 from bs4.element import (
     CData,
@@ -63,7 +69,8 @@ class BeautifulSoupHTMLParser(HTMLParser):
 
     def handle_charref(self, name):
         # XXX workaround for a bug in HTMLParser. Remove this once
-        # it's fixed.
+        # it's fixed in all supported versions.
+        # http://bugs.python.org/issue13633
         if name.startswith('x'):
             real_name = int(name.lstrip('x'), 16)
         elif name.startswith('X'):
@@ -113,14 +120,6 @@ class BeautifulSoupHTMLParser(HTMLParser):
 
     def handle_pi(self, data):
         self.soup.endData()
-        if data.endswith("?") and data.lower().startswith("xml"):
-            # "An XHTML processing instruction using the trailing '?'
-            # will cause the '?' to be included in data." - HTMLParser
-            # docs.
-            #
-            # Strip the question mark so we don't end up with two
-            # question marks.
-            data = data[:-1]
         self.soup.handle_data(data)
         self.soup.endData(ProcessingInstruction)
 
@@ -128,15 +127,19 @@ class BeautifulSoupHTMLParser(HTMLParser):
 class HTMLParserTreeBuilder(HTMLTreeBuilder):
 
     is_xml = False
-    features = [HTML, STRICT, HTMLPARSER]
+    picklable = True
+    NAME = HTMLPARSER
+    features = [NAME, HTML, STRICT]
 
     def __init__(self, *args, **kwargs):
-        if CONSTRUCTOR_TAKES_STRICT:
+        if CONSTRUCTOR_TAKES_STRICT and not CONSTRUCTOR_STRICT_IS_DEPRECATED:
             kwargs['strict'] = False
+        if CONSTRUCTOR_TAKES_CONVERT_CHARREFS:
+            kwargs['convert_charrefs'] = False
         self.parser_args = (args, kwargs)
 
     def prepare_markup(self, markup, user_specified_encoding=None,
-                       document_declared_encoding=None):
+                       document_declared_encoding=None, exclude_encodings=None):
         """
         :return: A 4-tuple (markup, original encoding, encoding
         declared within markup, whether any characters had to be
@@ -147,7 +150,8 @@ class HTMLParserTreeBuilder(HTMLTreeBuilder):
             return
 
         try_encodings = [user_specified_encoding, document_declared_encoding]
-        dammit = UnicodeDammit(markup, try_encodings, is_html=True)
+        dammit = UnicodeDammit(markup, try_encodings, is_html=True,
+                               exclude_encodings=exclude_encodings)
         yield (dammit.markup, dammit.original_encoding,
                dammit.declared_html_encoding,
                dammit.contains_replacement_characters)

Contents/Libraries/Shared/bs4/builder/_lxml.py

@@ -7,7 +7,12 @@ from io import BytesIO
 from StringIO import StringIO
 import collections
 from lxml import etree
-from bs4.element import Comment, Doctype, NamespacedAttribute
+from bs4.element import (
+    Comment,
+    Doctype,
+    NamespacedAttribute,
+    ProcessingInstruction,
+)
 from bs4.builder import (
     FAST,
     HTML,
@@ -25,8 +30,11 @@ class LXMLTreeBuilderForXML(TreeBuilder):
 
     is_xml = True
 
+    NAME = "lxml-xml"
+    ALTERNATE_NAMES = ["xml"]
+
     # Well, it's permissive by XML parser standards.
-    features = [LXML, XML, FAST, PERMISSIVE]
+    features = [NAME, LXML, XML, FAST, PERMISSIVE]
 
     CHUNK_SIZE = 512
 
@@ -70,6 +78,7 @@ class LXMLTreeBuilderForXML(TreeBuilder):
             return (None, tag)
 
     def prepare_markup(self, markup, user_specified_encoding=None,
+                       exclude_encodings=None,
                        document_declared_encoding=None):
         """
         :yield: A series of 4-tuples.
@@ -95,7 +104,8 @@ class LXMLTreeBuilderForXML(TreeBuilder):
         # the document as each one in turn.
         is_html = not self.is_xml
         try_encodings = [user_specified_encoding, document_declared_encoding]
-        detector = EncodingDetector(markup, try_encodings, is_html)
+        detector = EncodingDetector(
+            markup, try_encodings, is_html, exclude_encodings)
         for encoding in detector.encodings:
             yield (detector.markup, encoding, document_declared_encoding, False)
 
@@ -189,7 +199,9 @@ class LXMLTreeBuilderForXML(TreeBuilder):
             self.nsmaps.pop()
 
     def pi(self, target, data):
-        pass
+        self.soup.endData()
+        self.soup.handle_data(target + ' ' + data)
+        self.soup.endData(ProcessingInstruction)
 
     def data(self, content):
         self.soup.handle_data(content)
@@ -212,7 +224,10 @@ class LXMLTreeBuilderForXML(TreeBuilder):
 
 class LXMLTreeBuilder(HTMLTreeBuilder, LXMLTreeBuilderForXML):
 
-    features = [LXML, HTML, FAST, PERMISSIVE]
+    NAME = LXML
+    ALTERNATE_NAMES = ["lxml-html"]
+
+    features = ALTERNATE_NAMES + [NAME, HTML, FAST, PERMISSIVE]
     is_xml = False
 
     def default_parser(self, encoding):

Contents/Libraries/Shared/bs4/dammit.py

@@ -3,10 +3,12 @@
 
 This library converts a bytestream to Unicode through any means
 necessary. It is heavily based on code from Mark Pilgrim's Universal
-Feed Parser. It works best on XML and XML, but it does not rewrite the
+Feed Parser. It works best on XML and HTML, but it does not rewrite the
 XML or HTML to reflect a new encoding; that's the tree builder's job.
 """
+__license__ = "MIT"
 
+from pdb import set_trace
 import codecs
 from htmlentitydefs import codepoint2name
 import re
@@ -212,8 +214,11 @@ class EncodingDetector:
 
     5. Windows-1252.
     """
-    def __init__(self, markup, override_encodings=None, is_html=False):
+    def __init__(self, markup, override_encodings=None, is_html=False,
+                 exclude_encodings=None):
         self.override_encodings = override_encodings or []
+        exclude_encodings = exclude_encodings or []
+        self.exclude_encodings = set([x.lower() for x in exclude_encodings])
         self.chardet_encoding = None
         self.is_html = is_html
         self.declared_encoding = None
@@ -224,6 +229,8 @@ class EncodingDetector:
     def _usable(self, encoding, tried):
         if encoding is not None:
             encoding = encoding.lower()
+            if encoding in self.exclude_encodings:
+                return False
             if encoding not in tried:
                 tried.add(encoding)
                 return True
@@ -266,6 +273,9 @@ class EncodingDetector:
     def strip_byte_order_mark(cls, data):
         """If a byte-order mark is present, strip it and return the encoding it implies."""
         encoding = None
+        if isinstance(data, unicode):
+            # Unicode data cannot have a byte-order mark.
+            return data, encoding
         if (len(data) >= 4) and (data[:2] == b'\xfe\xff') \
                and (data[2:4] != '\x00\x00'):
             encoding = 'utf-16be'
@@ -306,7 +316,7 @@ class EncodingDetector:
             declared_encoding_match = html_meta_re.search(markup, endpos=html_endpos)
         if declared_encoding_match is not None:
             declared_encoding = declared_encoding_match.groups()[0].decode(
-                'ascii')
+                'ascii', 'replace')
         if declared_encoding:
             return declared_encoding.lower()
         return None
@@ -331,13 +341,14 @@ class UnicodeDammit:
         ]
 
     def __init__(self, markup, override_encodings=[],
-                 smart_quotes_to=None, is_html=False):
+                 smart_quotes_to=None, is_html=False, exclude_encodings=[]):
         self.smart_quotes_to = smart_quotes_to
         self.tried_encodings = []
         self.contains_replacement_characters = False
         self.is_html = is_html
 
-        self.detector = EncodingDetector(markup, override_encodings, is_html)
+        self.detector = EncodingDetector(
+            markup, override_encodings, is_html, exclude_encodings)
 
         # Short-circuit if the data is in Unicode to begin with.
         if isinstance(markup, unicode) or markup == '':

Contents/Libraries/Shared/bs4/diagnose.py

@@ -1,4 +1,7 @@
 """Diagnostic functions, mainly for use when doing tech support."""
+
+__license__ = "MIT"
+
 import cProfile
 from StringIO import StringIO
 from HTMLParser import HTMLParser
@@ -33,12 +36,21 @@ def diagnose(data):
 
     if 'lxml' in basic_parsers:
         basic_parsers.append(["lxml", "xml"])
-        from lxml import etree
-        print "Found lxml version %s" % ".".join(map(str,etree.LXML_VERSION))
+        try:
+            from lxml import etree
+            print "Found lxml version %s" % ".".join(map(str,etree.LXML_VERSION))
+        except ImportError, e:
+            print (
+                "lxml is not installed or couldn't be imported.")
+
 
     if 'html5lib' in basic_parsers:
-        import html5lib
-        print "Found html5lib version %s" % html5lib.__version__
+        try:
+            import html5lib
+            print "Found html5lib version %s" % html5lib.__version__
+        except ImportError, e:
+            print (
+                "html5lib is not installed or couldn't be imported.")
 
     if hasattr(data, 'read'):
         data = data.read()

Contents/Libraries/Shared/bs4/element.py

@@ -1,3 +1,6 @@
+__license__ = "MIT"
+
+from pdb import set_trace
 import collections
 import re
 import sys
@@ -185,24 +188,40 @@ class PageElement(object):
             return self.HTML_FORMATTERS.get(
                 name, HTMLAwareEntitySubstitution.substitute_xml)
 
-    def setup(self, parent=None, previous_element=None):
+    def setup(self, parent=None, previous_element=None, next_element=None,
+              previous_sibling=None, next_sibling=None):
         """Sets up the initial relations between this element and
         other elements."""
         self.parent = parent
+
         self.previous_element = previous_element
         if previous_element is not None:
             self.previous_element.next_element = self
-        self.next_element = None
-        self.previous_sibling = None
-        self.next_sibling = None
-        if self.parent is not None and self.parent.contents:
-            self.previous_sibling = self.parent.contents[-1]
+
+        self.next_element = next_element
+        if self.next_element:
+            self.next_element.previous_element = self
+
+        self.next_sibling = next_sibling
+        if self.next_sibling:
+            self.next_sibling.previous_sibling = self
+
+        if (not previous_sibling
+            and self.parent is not None and self.parent.contents):
+            previous_sibling = self.parent.contents[-1]
+
+        self.previous_sibling = previous_sibling
+        if previous_sibling:
             self.previous_sibling.next_sibling = self
 
     nextSibling = _alias("next_sibling")  # BS3
     previousSibling = _alias("previous_sibling")  # BS3
 
     def replace_with(self, replace_with):
+        if not self.parent:
+            raise ValueError(
+                "Cannot replace one element with another when the"
+                "element to be replaced is not part of a tree.")
         if replace_with is self:
             return
         if replace_with is self.parent:
@@ -216,6 +235,10 @@ class PageElement(object):
 
     def unwrap(self):
         my_parent = self.parent
+        if not self.parent:
+            raise ValueError(
+                "Cannot replace an element with its contents when that"
+                "element is not part of a tree.")
         my_index = self.parent.index(self)
         self.extract()
         for child in reversed(self.contents[:]):
@@ -240,17 +263,20 @@ class PageElement(object):
         last_child = self._last_descendant()
         next_element = last_child.next_element
 
-        if self.previous_element is not None:
+        if (self.previous_element is not None and
+            self.previous_element is not next_element):
             self.previous_element.next_element = next_element
-        if next_element is not None:
+        if next_element is not None and next_element is not self.previous_element:
             next_element.previous_element = self.previous_element
         self.previous_element = None
         last_child.next_element = None
 
         self.parent = None
-        if self.previous_sibling is not None:
+        if (self.previous_sibling is not None
+            and self.previous_sibling is not self.next_sibling):
             self.previous_sibling.next_sibling = self.next_sibling
-        if self.next_sibling is not None:
+        if (self.next_sibling is not None
+            and self.next_sibling is not self.previous_sibling):
             self.next_sibling.previous_sibling = self.previous_sibling
         self.previous_sibling = self.next_sibling = None
         return self
@@ -263,13 +289,15 @@ class PageElement(object):
             last_child = self
             while isinstance(last_child, Tag) and last_child.contents:
                 last_child = last_child.contents[-1]
-        if not accept_self and last_child == self:
+        if not accept_self and last_child is self:
             last_child = None
         return last_child
     # BS3: Not part of the API!
     _lastRecursiveChild = _last_descendant
 
     def insert(self, position, new_child):
+        if new_child is None:
+            raise ValueError("Cannot insert None into a tag.")
         if new_child is self:
             raise ValueError("Cannot insert a tag into itself.")
         if (isinstance(new_child, basestring)
@@ -478,6 +506,10 @@ class PageElement(object):
     def _find_all(self, name, attrs, text, limit, generator, **kwargs):
         "Iterates over a generator looking for things that match."
 
+        if text is None and 'string' in kwargs:
+            text = kwargs['string']
+            del kwargs['string']
+
         if isinstance(name, SoupStrainer):
             strainer = name
         else:
@@ -548,17 +580,17 @@ class PageElement(object):
 
     # Methods for supporting CSS selectors.
 
-    tag_name_re = re.compile('^[a-z0-9]+$')
+    tag_name_re = re.compile('^[a-zA-Z0-9][-.a-zA-Z0-9:_]*$')
 
-    # /^(\w+)\[(\w+)([=~\|\^\$\*]?)=?"?([^\]"]*)"?\]$/
-    #   \---/  \---/\-------------/    \-------/
-    #     |      |         |               |
-    #     |      |         |           The value
-    #     |      |    ~,|,^,$,* or =
-    #     |   Attribute
+    # /^([a-zA-Z0-9][-.a-zA-Z0-9:_]*)\[(\w+)([=~\|\^\$\*]?)=?"?([^\]"]*)"?\]$/
+    #   \---------------------------/  \---/\-------------/    \-------/
+    #     |                              |         |               |
+    #     |                              |         |           The value
+    #     |                              |    ~,|,^,$,* or =
+    #     |                           Attribute
     #    Tag
     attribselect_re = re.compile(
-        r'^(?P<tag>\w+)?\[(?P<attribute>\w+)(?P<operator>[=~\|\^\$\*]?)' +
+        r'^(?P<tag>[a-zA-Z0-9][-.a-zA-Z0-9:_]*)?\[(?P<attribute>[\w-]+)(?P<operator>[=~\|\^\$\*]?)' +
         r'=?"?(?P<value>[^\]"]*)"?\]$'
         )
 
@@ -654,11 +686,17 @@ class NavigableString(unicode, PageElement):
         how to handle non-ASCII characters.
         """
         if isinstance(value, unicode):
-            return unicode.__new__(cls, value)
-        return unicode.__new__(cls, value, DEFAULT_OUTPUT_ENCODING)
+            u = unicode.__new__(cls, value)
+        else:
+            u = unicode.__new__(cls, value, DEFAULT_OUTPUT_ENCODING)
+        u.setup()
+        return u
 
     def __copy__(self):
-        return self
+        """A copy of a NavigableString has the same contents and class
+        as the original, but it is not connected to the parse tree.
+        """
+        return type(self)(self)
 
     def __getnewargs__(self):
         return (unicode(self),)
@@ -707,7 +745,7 @@ class CData(PreformattedString):
 class ProcessingInstruction(PreformattedString):
 
     PREFIX = u'<?'
-    SUFFIX = u'?>'
+    SUFFIX = u'>'
 
 class Comment(PreformattedString):
 
@@ -716,8 +754,8 @@ class Comment(PreformattedString):
 
 
 class Declaration(PreformattedString):
-    PREFIX = u'<!'
-    SUFFIX = u'!>'
+    PREFIX = u'<?'
+    SUFFIX = u'?>'
 
 
 class Doctype(PreformattedString):
@@ -759,9 +797,12 @@ class Tag(PageElement):
         self.prefix = prefix
         if attrs is None:
             attrs = {}
-        elif attrs and builder.cdata_list_attributes:
-            attrs = builder._replace_cdata_list_attribute_values(
-                self.name, attrs)
+        elif attrs:
+            if builder is not None and builder.cdata_list_attributes:
+                attrs = builder._replace_cdata_list_attribute_values(
+                    self.name, attrs)
+            else:
+                attrs = dict(attrs)
         else:
             attrs = dict(attrs)
         self.attrs = attrs
@@ -778,6 +819,18 @@ class Tag(PageElement):
 
     parserClass = _alias("parser_class")  # BS3
 
+    def __copy__(self):
+        """A copy of a Tag is a new Tag, unconnected to the parse tree.
+        Its contents are a copy of the old Tag's contents.
+        """
+        clone = type(self)(None, self.builder, self.name, self.namespace,
+                           self.nsprefix, self.attrs)
+        for attr in ('can_be_empty_element', 'hidden'):
+            setattr(clone, attr, getattr(self, attr))
+        for child in self.contents:
+            clone.append(child.__copy__())
+        return clone
+
     @property
     def is_empty_element(self):
         """Is this tag an empty-element tag? (aka a self-closing tag)
@@ -971,15 +1024,25 @@ class Tag(PageElement):
         as defined in __eq__."""
         return not self == other
 
-    def __repr__(self, encoding=DEFAULT_OUTPUT_ENCODING):
+    def __repr__(self, encoding="unicode-escape"):
         """Renders this tag as a string."""
-        return self.encode(encoding)
+        if PY3K:
+            # "The return value must be a string object", i.e. Unicode
+            return self.decode()
+        else:
+            # "The return value must be a string object", i.e. a bytestring.
+            # By convention, the return value of __repr__ should also be
+            # an ASCII string.
+            return self.encode(encoding)
 
     def __unicode__(self):
         return self.decode()
 
     def __str__(self):
-        return self.encode()
+        if PY3K:
+            return self.decode()
+        else:
+            return self.encode()
 
     if PY3K:
         __str__ = __repr__ = __unicode__
@@ -1103,12 +1166,18 @@ class Tag(PageElement):
                        formatter="minimal"):
         """Renders the contents of this tag as a Unicode string.
 
+        :param indent_level: Each line of the rendering will be
+           indented this many spaces.
+
         :param eventual_encoding: The tag is destined to be
            encoded into this encoding. This method is _not_
            responsible for performing that encoding. This information
            is passed in so that it can be substituted in if the
            document contains a <META> tag that mentions the document's
            encoding.
+
+        :param formatter: The output formatter responsible for converting
+           entities to Unicode characters.
         """
         # First off, turn a string formatter into a function. This
         # will stop the lookup from happening over and over again.
@@ -1137,7 +1206,17 @@ class Tag(PageElement):
     def encode_contents(
         self, indent_level=None, encoding=DEFAULT_OUTPUT_ENCODING,
         formatter="minimal"):
-        """Renders the contents of this tag as a bytestring."""
+        """Renders the contents of this tag as a bytestring.
+
+        :param indent_level: Each line of the rendering will be
+           indented this many spaces.
+
+        :param eventual_encoding: The bytestring will be in this encoding.
+
+        :param formatter: The output formatter responsible for converting
+           entities to Unicode characters.
+        """
+
         contents = self.decode_contents(indent_level, encoding, formatter)
         return contents.encode(encoding)
 
@@ -1201,26 +1280,57 @@ class Tag(PageElement):
 
     _selector_combinators = ['>', '+', '~']
     _select_debug = False
-    def select(self, selector, _candidate_generator=None):
+    def select_one(self, selector):
         """Perform a CSS selection operation on the current element."""
+        value = self.select(selector, limit=1)
+        if value:
+            return value[0]
+        return None
+
+    def select(self, selector, _candidate_generator=None, limit=None):
+        """Perform a CSS selection operation on the current element."""
+
+        # Handle grouping selectors if ',' exists, ie: p,a
+        if ',' in selector:
+            context = []
+            for partial_selector in selector.split(','):
+                partial_selector = partial_selector.strip()
+                if partial_selector == '':
+                    raise ValueError('Invalid group selection syntax: %s' % selector)
+                candidates = self.select(partial_selector, limit=limit)
+                for candidate in candidates:
+                    if candidate not in context:
+                        context.append(candidate)
+
+                if limit and len(context) >= limit:
+                    break
+            return context
+
         tokens = selector.split()
         current_context = [self]
 
         if tokens[-1] in self._selector_combinators:
             raise ValueError(
                 'Final combinator "%s" is missing an argument.' % tokens[-1])
+
         if self._select_debug:
             print 'Running CSS selector "%s"' % selector
+
         for index, token in enumerate(tokens):
-            if self._select_debug:
-                print ' Considering token "%s"' % token
-            recursive_candidate_generator = None
-            tag_name = None
+            new_context = []
+            new_context_ids = set([])
+
             if tokens[index-1] in self._selector_combinators:
                 # This token was consumed by the previous combinator. Skip it.
                 if self._select_debug:
                     print '  Token was consumed by the previous combinator.'
                 continue
+
+            if self._select_debug:
+                print ' Considering token "%s"' % token
+            recursive_candidate_generator = None
+            tag_name = None
+
             # Each operation corresponds to a checker function, a rule
             # for determining whether a candidate matches the
             # selector. Candidates are generated by the active
@@ -1256,35 +1366,38 @@ class Tag(PageElement):
                         "A pseudo-class must be prefixed with a tag name.")
                 pseudo_attributes = re.match('([a-zA-Z\d-]+)\(([a-zA-Z\d]+)\)', pseudo)
                 found = []
-                if pseudo_attributes is not None:
+                if pseudo_attributes is None:
+                    pseudo_type = pseudo
+                    pseudo_value = None
+                else:
                     pseudo_type, pseudo_value = pseudo_attributes.groups()
-                    if pseudo_type == 'nth-of-type':
-                        try:
-                            pseudo_value = int(pseudo_value)
-                        except:
-                            raise NotImplementedError(
-                                'Only numeric values are currently supported for the nth-of-type pseudo-class.')
-                        if pseudo_value < 1:
-                            raise ValueError(
-                                'nth-of-type pseudo-class value must be at least 1.')
-                        class Counter(object):
-                            def __init__(self, destination):
-                                self.count = 0
-                                self.destination = destination
-
-                            def nth_child_of_type(self, tag):
-                                self.count += 1
-                                if self.count == self.destination:
-                                    return True
-                                if self.count > self.destination:
-                                    # Stop the generator that's sending us
-                                    # these things.
-                                    raise StopIteration()
-                                return False
-                        checker = Counter(pseudo_value).nth_child_of_type
-                    else:
+                if pseudo_type == 'nth-of-type':
+                    try:
+                        pseudo_value = int(pseudo_value)
+                    except:
                         raise NotImplementedError(
-                            'Only the following pseudo-classes are implemented: nth-of-type.')
+                            'Only numeric values are currently supported for the nth-of-type pseudo-class.')
+                    if pseudo_value < 1:
+                        raise ValueError(
+                            'nth-of-type pseudo-class value must be at least 1.')
+                    class Counter(object):
+                        def __init__(self, destination):
+                            self.count = 0
+                            self.destination = destination
+
+                        def nth_child_of_type(self, tag):
+                            self.count += 1
+                            if self.count == self.destination:
+                                return True
+                            if self.count > self.destination:
+                                # Stop the generator that's sending us
+                                # these things.
+                                raise StopIteration()
+                            return False
+                    checker = Counter(pseudo_value).nth_child_of_type
+                else:
+                    raise NotImplementedError(
+                        'Only the following pseudo-classes are implemented: nth-of-type.')
 
             elif token == '*':
                 # Star selector -- matches everything
@@ -1311,7 +1424,6 @@ class Tag(PageElement):
             else:
                 raise ValueError(
                     'Unsupported or invalid CSS selector: "%s"' % token)
-
             if recursive_candidate_generator:
                 # This happens when the selector looks like  "> foo".
                 #
@@ -1361,8 +1473,7 @@ class Tag(PageElement):
             else:
                 _use_candidate_generator = _candidate_generator
 
-            new_context = []
-            new_context_ids = set([])
+            count = 0
             for tag in current_context:
                 if self._select_debug:
                     print "    Running candidate generator on %s %s" % (
@@ -1387,9 +1498,12 @@ class Tag(PageElement):
                             # don't include it in the context more than once.
                             new_context.append(candidate)
                             new_context_ids.add(id(candidate))
+                            if limit and len(new_context) >= limit:
+                                break
                     elif self._select_debug:
                         print "     FAILURE %s %s" % (candidate.name, repr(candidate.attrs))
 
+
             current_context = new_context
 
         if self._select_debug:

Contents/Libraries/Shared/bs4/testing.py

@@ -1,5 +1,8 @@
 """Helper classes for tests."""
 
+__license__ = "MIT"
+
+import pickle
 import copy
 import functools
 import unittest
@@ -43,6 +46,16 @@ class SoupTest(unittest.TestCase):
 
         self.assertEqual(obj.decode(), self.document_for(compare_parsed_to))
 
+    def assertConnectedness(self, element):
+        """Ensure that next_element and previous_element are properly
+        set for all descendants of the given element.
+        """
+        earlier = None
+        for e in element.descendants:
+            if earlier:
+                self.assertEqual(e, earlier.next_element)
+                self.assertEqual(earlier, e.previous_element)
+            earlier = e
 
 class HTMLTreeBuilderSmokeTest(object):
 
@@ -54,6 +67,15 @@ class HTMLTreeBuilderSmokeTest(object):
     markup in these tests, there's not much room for interpretation.
     """
 
+    def test_pickle_and_unpickle_identity(self):
+        # Pickling a tree, then unpickling it, yields a tree identical
+        # to the original.
+        tree = self.soup("<a><b>foo</a>")
+        dumped = pickle.dumps(tree, 2)
+        loaded = pickle.loads(dumped)
+        self.assertEqual(loaded.__class__, BeautifulSoup)
+        self.assertEqual(loaded.decode(), tree.decode())
+
     def assertDoctypeHandled(self, doctype_fragment):
         """Assert that a given doctype string is handled correctly."""
         doctype_str, soup = self._document_with_doctype(doctype_fragment)
@@ -114,6 +136,11 @@ class HTMLTreeBuilderSmokeTest(object):
             soup.encode("utf-8").replace(b"\n", b""),
             markup.replace(b"\n", b""))
 
+    def test_processing_instruction(self):
+        markup = b"""<?PITarget PIContent?>"""
+        soup = self.soup(markup)
+        self.assertEqual(markup, soup.encode("utf8"))
+
     def test_deepcopy(self):
         """Make sure you can copy the tree builder.
 
@@ -155,6 +182,23 @@ class HTMLTreeBuilderSmokeTest(object):
     def test_nested_formatting_elements(self):
         self.assertSoupEquals("<em><em></em></em>")
 
+    def test_double_head(self):
+        html = '''<!DOCTYPE html>
+<html>
+<head>
+<title>Ordinary HEAD element test</title>
+</head>
+<script type="text/javascript">
+alert("Help!");
+</script>
+<body>
+Hello, world!
+</body>
+</html>
+'''
+        soup = self.soup(html)
+        self.assertEqual("text/javascript", soup.find('script')['type'])
+
     def test_comment(self):
         # Comments are represented as Comment objects.
         markup = "<p>foo<!--foobar-->baz</p>"
@@ -221,6 +265,14 @@ class HTMLTreeBuilderSmokeTest(object):
         soup = self.soup(markup)
         self.assertEqual(["css"], soup.div.div['class'])
 
+    def test_multivalued_attribute_on_html(self):
+        # html5lib uses a different API to set the attributes ot the
+        # <html> tag. This has caused problems with multivalued
+        # attributes.
+        markup = '<html class="a b"></html>'
+        soup = self.soup(markup)
+        self.assertEqual(["a", "b"], soup.html['class'])
+
     def test_angle_brackets_in_attribute_values_are_escaped(self):
         self.assertSoupEquals('<a b="<a>"></a>', '<a b="&lt;a&gt;"></a>')
 
@@ -253,6 +305,35 @@ class HTMLTreeBuilderSmokeTest(object):
         soup = self.soup("<html><h2>\nfoo</h2><p></p></html>")
         self.assertEqual("p", soup.h2.string.next_element.name)
         self.assertEqual("p", soup.p.name)
+        self.assertConnectedness(soup)
+
+    def test_head_tag_between_head_and_body(self):
+        "Prevent recurrence of a bug in the html5lib treebuilder."
+        content = """<html><head></head>
+  <link></link>
+  <body>foo</body>
+</html>
+"""
+        soup = self.soup(content)
+        self.assertNotEqual(None, soup.html.body)
+        self.assertConnectedness(soup)
+
+    def test_multiple_copies_of_a_tag(self):
+        "Prevent recurrence of a bug in the html5lib treebuilder."
+        content = """<!DOCTYPE html>
+<html>
+ <body>
+   <article id="a" >
+   <div><a href="1"></div>
+   <footer>
+     <a href="2"></a>
+   </footer>
+  </article>
+  </body>
+</html>
+"""
+        soup = self.soup(content)
+        self.assertConnectedness(soup.article)
 
     def test_basic_namespaces(self):
         """Parsers don't need to *understand* namespaces, but at the
@@ -463,11 +544,25 @@ class HTMLTreeBuilderSmokeTest(object):
 
 class XMLTreeBuilderSmokeTest(object):
 
+    def test_pickle_and_unpickle_identity(self):
+        # Pickling a tree, then unpickling it, yields a tree identical
+        # to the original.
+        tree = self.soup("<a><b>foo</a>")
+        dumped = pickle.dumps(tree, 2)
+        loaded = pickle.loads(dumped)
+        self.assertEqual(loaded.__class__, BeautifulSoup)
+        self.assertEqual(loaded.decode(), tree.decode())
+
     def test_docstring_generated(self):
         soup = self.soup("<root/>")
         self.assertEqual(
             soup.encode(), b'<?xml version="1.0" encoding="utf-8"?>\n<root/>')
 
+    def test_xml_declaration(self):
+        markup = b"""<?xml version="1.0" encoding="utf8"?>\n<foo/>"""
+        soup = self.soup(markup)
+        self.assertEqual(markup, soup.encode("utf8"))
+
     def test_real_xhtml_document(self):
         """A real XHTML document should come out *exactly* the same as it went in."""
         markup = b"""<?xml version="1.0" encoding="utf-8"?>
@@ -485,7 +580,7 @@ class XMLTreeBuilderSmokeTest(object):
   <script type="text/javascript">
   </script>
 """
-        soup = BeautifulSoup(doc, "xml")
+        soup = BeautifulSoup(doc, "lxml-xml")
         # lxml would have stripped this while parsing, but we can add
         # it later.
         soup.script.string = 'console.log("< < hey > > ");'

Contents/Libraries/Shared/bs4/tests/test_builder_registry.py

@@ -1,6 +1,7 @@
 """Tests of the builder registry."""
 
 import unittest
+import warnings
 
 from bs4 import BeautifulSoup
 from bs4.builder import (
@@ -67,10 +68,15 @@ class BuiltInRegistryTest(unittest.TestCase):
                           HTMLParserTreeBuilder)
 
     def test_beautifulsoup_constructor_does_lookup(self):
-        # You can pass in a string.
-        BeautifulSoup("", features="html")
-        # Or a list of strings.
-        BeautifulSoup("", features=["html", "fast"])
+
+        with warnings.catch_warnings(record=True) as w:
+            # This will create a warning about not explicitly
+            # specifying a parser, but we'll ignore it.
+
+            # You can pass in a string.
+            BeautifulSoup("", features="html")
+            # Or a list of strings.
+            BeautifulSoup("", features=["html", "fast"])
 
         # You'll get an exception if BS can't find an appropriate
         # builder.

Contents/Libraries/Shared/bs4/tests/test_html5lib.py

@@ -83,3 +83,16 @@ class HTML5LibBuilderSmokeTest(SoupTest, HTML5TreeBuilderSmokeTest):
         soup = self.soup(markup)
         self.assertEqual(u"<body><p><em>foo</em></p><em>\n</em><p><em>bar<a></a></em></p>\n</body>", soup.body.decode())
         self.assertEqual(2, len(soup.find_all('p')))
+
+    def test_processing_instruction(self):
+        """Processing instructions become comments."""
+        markup = b"""<?PITarget PIContent?>"""
+        soup = self.soup(markup)
+        assert str(soup).startswith("<!--?PITarget PIContent?-->")
+
+    def test_cloned_multivalue_node(self):
+        markup = b"""<a class="my_class"><p></a>"""
+        soup = self.soup(markup)
+        a1, a2 = soup.find_all('a')
+        self.assertEqual(a1, a2)
+        assert a1 is not a2

Contents/Libraries/Shared/bs4/tests/test_htmlparser.py

@@ -1,6 +1,8 @@
 """Tests to ensure that the html.parser tree builder generates good
 trees."""
 
+from pdb import set_trace
+import pickle
 from bs4.testing import SoupTest, HTMLTreeBuilderSmokeTest
 from bs4.builder import HTMLParserTreeBuilder
 
@@ -17,3 +19,14 @@ class HTMLParserTreeBuilderSmokeTest(SoupTest, HTMLTreeBuilderSmokeTest):
     def test_namespaced_public_doctype(self):
         # html.parser can't handle namespaced doctypes, so skip this one.
         pass
+
+    def test_builder_is_pickled(self):
+        """Unlike most tree builders, HTMLParserTreeBuilder and will
+        be restored after pickling.
+        """
+        tree = self.soup("<a><b>foo</a>")
+        dumped = pickle.dumps(tree, 2)
+        loaded = pickle.loads(dumped)
+        self.assertTrue(isinstance(loaded.builder, type(tree.builder)))
+
+

Contents/Libraries/Shared/bs4/tests/test_lxml.py

@@ -65,21 +65,6 @@ class LXMLTreeBuilderSmokeTest(SoupTest, HTMLTreeBuilderSmokeTest):
         self.assertEqual(u"<b/>", unicode(soup.b))
         self.assertTrue("BeautifulStoneSoup class is deprecated" in str(w[0].message))
 
-    def test_real_xhtml_document(self):
-        """lxml strips the XML definition from an XHTML doc, which is fine."""
-        markup = b"""<?xml version="1.0" encoding="utf-8"?>
-<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN">
-<html xmlns="http://www.w3.org/1999/xhtml">
-<head><title>Hello.</title></head>
-<body>Goodbye.</body>
-</html>"""
-        soup = self.soup(markup)
-        self.assertEqual(
-            soup.encode("utf-8").replace(b"\n", b''),
-            markup.replace(b'\n', b'').replace(
-                b'<?xml version="1.0" encoding="utf-8"?>', b''))
-
-
 @skipIf(
     not LXML_PRESENT,
     "lxml seems not to be present, not testing its XML tree builder.")

Contents/Libraries/Shared/bs4/tests/test_soup.py

@@ -1,6 +1,7 @@
 # -*- coding: utf-8 -*-
 """Tests of Beautiful Soup as a whole."""
 
+from pdb import set_trace
 import logging
 import unittest
 import sys
@@ -20,6 +21,7 @@ import bs4.dammit
 from bs4.dammit import (
     EntitySubstitution,
     UnicodeDammit,
+    EncodingDetector,
 )
 from bs4.testing import (
     SoupTest,
@@ -48,8 +50,34 @@ class TestConstructor(SoupTest):
         soup = self.soup(data)
         self.assertEqual(u"foo\0bar", soup.h1.string)
 
+    def test_exclude_encodings(self):
+        utf8_data = u"Räksmörgås".encode("utf-8")
+        soup = self.soup(utf8_data, exclude_encodings=["utf-8"])
+        self.assertEqual("windows-1252", soup.original_encoding)
 
-class TestDeprecatedConstructorArguments(SoupTest):
+
+class TestWarnings(SoupTest):
+
+    def _no_parser_specified(self, s, is_there=True):
+        v = s.startswith(BeautifulSoup.NO_PARSER_SPECIFIED_WARNING[:80])
+        self.assertTrue(v)
+
+    def test_warning_if_no_parser_specified(self):
+        with warnings.catch_warnings(record=True) as w:
+            soup = self.soup("<a><b></b></a>")
+        msg = str(w[0].message)
+        self._assert_no_parser_specified(msg)
+
+    def test_warning_if_parser_specified_too_vague(self):
+        with warnings.catch_warnings(record=True) as w:
+            soup = self.soup("<a><b></b></a>", "html")
+        msg = str(w[0].message)
+        self._assert_no_parser_specified(msg)
+
+    def test_no_warning_if_explicit_parser_specified(self):
+        with warnings.catch_warnings(record=True) as w:
+            soup = self.soup("<a><b></b></a>", "html.parser")
+        self.assertEquals([], w)
 
     def test_parseOnlyThese_renamed_to_parse_only(self):
         with warnings.catch_warnings(record=True) as w:
@@ -271,10 +299,11 @@ class TestUnicodeDammit(unittest.TestCase):
             dammit.unicode_markup, """<foo>''""</foo>""")
 
     def test_detect_utf8(self):
-        utf8 = b"\xc3\xa9"
+        utf8 = b"Sacr\xc3\xa9 bleu! \xe2\x98\x83"
         dammit = UnicodeDammit(utf8)
-        self.assertEqual(dammit.unicode_markup, u'\xe9')
         self.assertEqual(dammit.original_encoding.lower(), 'utf-8')
+        self.assertEqual(dammit.unicode_markup, u'Sacr\xe9 bleu! \N{SNOWMAN}')
+
 
     def test_convert_hebrew(self):
         hebrew = b"\xed\xe5\xec\xf9"
@@ -299,6 +328,26 @@ class TestUnicodeDammit(unittest.TestCase):
             dammit = UnicodeDammit(utf8_data, [bad_encoding])
             self.assertEqual(dammit.original_encoding.lower(), 'utf-8')
 
+    def test_exclude_encodings(self):
+        # This is UTF-8.
+        utf8_data = u"Räksmörgås".encode("utf-8")
+
+        # But if we exclude UTF-8 from consideration, the guess is
+        # Windows-1252.
+        dammit = UnicodeDammit(utf8_data, exclude_encodings=["utf-8"])
+        self.assertEqual(dammit.original_encoding.lower(), 'windows-1252')
+
+        # And if we exclude that, there is no valid guess at all.
+        dammit = UnicodeDammit(
+            utf8_data, exclude_encodings=["utf-8", "windows-1252"])
+        self.assertEqual(dammit.original_encoding, None)
+
+    def test_encoding_detector_replaces_junk_in_encoding_name_with_replacement_character(self):
+        detected = EncodingDetector(
+            b'<?xml version="1.0" encoding="UTF-\xdb" ?>')
+        encodings = list(detected.encodings)
+        assert u'utf-\N{REPLACEMENT CHARACTER}' in encodings
+
     def test_detect_html5_style_meta_tag(self):
 
         for data in (

Contents/Libraries/Shared/bs4/tests/test_tree.py

@@ -9,6 +9,7 @@ same markup, but all Beautiful Soup trees can be traversed with the
 methods tested here.
 """
 
+from pdb import set_trace
 import copy
 import pickle
 import re
@@ -19,8 +20,10 @@ from bs4.builder import (
     HTMLParserTreeBuilder,
 )
 from bs4.element import (
+    PY3K,
     CData,
     Comment,
+    Declaration,
     Doctype,
     NavigableString,
     SoupStrainer,
@@ -68,7 +71,13 @@ class TestFind(TreeTest):
 
     def test_unicode_text_find(self):
         soup = self.soup(u'<h1>Räksmörgås</h1>')
-        self.assertEqual(soup.find(text=u'Räksmörgås'), u'Räksmörgås')
+        self.assertEqual(soup.find(string=u'Räksmörgås'), u'Räksmörgås')
+
+    def test_unicode_attribute_find(self):
+        soup = self.soup(u'<h1 id="Räksmörgås">here it is</h1>')
+        str(soup)
+        self.assertEqual("here it is", soup.find(id=u'Räksmörgås').text)
+
 
     def test_find_everything(self):
         """Test an optimization that finds all tags."""
@@ -87,6 +96,7 @@ class TestFindAll(TreeTest):
         """You can search the tree for text nodes."""
         soup = self.soup("<html>Foo<b>bar</b>\xbb</html>")
         # Exact match.
+        self.assertEqual(soup.find_all(string="bar"), [u"bar"])
         self.assertEqual(soup.find_all(text="bar"), [u"bar"])
         # Match any of a number of strings.
         self.assertEqual(
@@ -688,7 +698,7 @@ class TestTagCreation(SoupTest):
 
     def test_tag_inherits_self_closing_rules_from_builder(self):
         if XML_BUILDER_PRESENT:
-            xml_soup = BeautifulSoup("", "xml")
+            xml_soup = BeautifulSoup("", "lxml-xml")
             xml_br = xml_soup.new_tag("br")
             xml_p = xml_soup.new_tag("p")
 
@@ -697,7 +707,7 @@ class TestTagCreation(SoupTest):
             self.assertEqual(b"<br/>", xml_br.encode())
             self.assertEqual(b"<p/>", xml_p.encode())
 
-        html_soup = BeautifulSoup("", "html")
+        html_soup = BeautifulSoup("", "html.parser")
         html_br = html_soup.new_tag("br")
         html_p = html_soup.new_tag("p")
 
@@ -773,6 +783,14 @@ class TestTreeModification(SoupTest):
         new_a = a.unwrap()
         self.assertEqual(a, new_a)
 
+    def test_replace_with_and_unwrap_give_useful_exception_when_tag_has_no_parent(self):
+        soup = self.soup("<a><b>Foo</b></a><c>Bar</c>")
+        a = soup.a
+        a.extract()
+        self.assertEqual(None, a.parent)
+        self.assertRaises(ValueError, a.unwrap)
+        self.assertRaises(ValueError, a.replace_with, soup.c)
+
     def test_replace_tag_with_itself(self):
         text = "<a><b></b><c>Foo<d></d></c></a><a><e></e></a>"
         soup = self.soup(text)
@@ -1067,6 +1085,31 @@ class TestTreeModification(SoupTest):
         self.assertEqual(foo_2, soup.a.string)
         self.assertEqual(bar_2, soup.b.string)
 
+    def test_extract_multiples_of_same_tag(self):
+        soup = self.soup("""
+<html>
+<head>
+<script>foo</script>
+</head>
+<body>
+ <script>bar</script>
+ <a></a>
+</body>
+<script>baz</script>
+</html>""")
+        [soup.script.extract() for i in soup.find_all("script")]
+        self.assertEqual("<body>\n\n<a></a>\n</body>", unicode(soup.body))
+
+
+    def test_extract_works_when_element_is_surrounded_by_identical_strings(self):
+        soup = self.soup(
+ '<html>\n'
+ '<body>hi</body>\n'
+ '</html>')
+        soup.find('body').extract()
+        self.assertEqual(None, soup.find('body'))
+
+
     def test_clear(self):
         """Tag.clear()"""
         soup = self.soup("<p><a>String <em>Italicized</em></a> and another</p>")
@@ -1293,6 +1336,51 @@ class TestPersistence(SoupTest):
         loaded = pickle.loads(dumped)
         self.assertEqual(loaded.decode(), soup.decode())
 
+    def test_copy_navigablestring_is_not_attached_to_tree(self):
+        html = u"<b>Foo<a></a></b><b>Bar</b>"
+        soup = self.soup(html)
+        s1 = soup.find(string="Foo")
+        s2 = copy.copy(s1)
+        self.assertEqual(s1, s2)
+        self.assertEqual(None, s2.parent)
+        self.assertEqual(None, s2.next_element)
+        self.assertNotEqual(None, s1.next_sibling)
+        self.assertEqual(None, s2.next_sibling)
+        self.assertEqual(None, s2.previous_element)
+
+    def test_copy_navigablestring_subclass_has_same_type(self):
+        html = u"<b><!--Foo--></b>"
+        soup = self.soup(html)
+        s1 = soup.string
+        s2 = copy.copy(s1)
+        self.assertEqual(s1, s2)
+        self.assertTrue(isinstance(s2, Comment))
+
+    def test_copy_entire_soup(self):
+        html = u"<div><b>Foo<a></a></b><b>Bar</b></div>end"
+        soup = self.soup(html)
+        soup_copy = copy.copy(soup)
+        self.assertEqual(soup, soup_copy)
+
+    def test_copy_tag_copies_contents(self):
+        html = u"<div><b>Foo<a></a></b><b>Bar</b></div>end"
+        soup = self.soup(html)
+        div = soup.div
+        div_copy = copy.copy(div)
+
+        # The two tags look the same, and evaluate to equal.
+        self.assertEqual(unicode(div), unicode(div_copy))
+        self.assertEqual(div, div_copy)
+
+        # But they're not the same object.
+        self.assertFalse(div is div_copy)
+
+        # And they don't have the same relation to the parse tree. The
+        # copy is not associated with a parse tree at all.
+        self.assertEqual(None, div_copy.parent)
+        self.assertEqual(None, div_copy.previous_element)
+        self.assertEqual(None, div_copy.find(string='Bar').next_element)
+        self.assertNotEqual(None, div.find(string='Bar').next_element)
 
 class TestSubstitutions(SoupTest):
 
@@ -1366,7 +1454,7 @@ class TestSubstitutions(SoupTest):
    console.log("< < hey > > ");
   </script>
 """
-        encoded = BeautifulSoup(doc).encode()
+        encoded = BeautifulSoup(doc, 'html.parser').encode()
         self.assertTrue(b"< < hey > >" in encoded)
 
     def test_formatter_skips_style_tag_for_html_documents(self):
@@ -1375,7 +1463,7 @@ class TestSubstitutions(SoupTest):
    console.log("< < hey > > ");
   </style>
 """
-        encoded = BeautifulSoup(doc).encode()
+        encoded = BeautifulSoup(doc, 'html.parser').encode()
         self.assertTrue(b"< < hey > >" in encoded)
 
     def test_prettify_leaves_preformatted_text_alone(self):
@@ -1387,7 +1475,7 @@ class TestSubstitutions(SoupTest):
             soup.div.prettify())
 
     def test_prettify_accepts_formatter(self):
-        soup = BeautifulSoup("<html><body>foo</body></html>")
+        soup = BeautifulSoup("<html><body>foo</body></html>", 'html.parser')
         pretty = soup.prettify(formatter = lambda x: x.upper())
         self.assertTrue("FOO" in pretty)
 
@@ -1484,6 +1572,14 @@ class TestEncoding(SoupTest):
         self.assertEqual(
             u"\N{SNOWMAN}".encode("utf8"), soup.b.renderContents())
 
+    def test_repr(self):
+        html = u"<b>\N{SNOWMAN}</b>"
+        soup = self.soup(html)
+        if PY3K:
+            self.assertEqual(html, repr(soup))
+        else:
+            self.assertEqual(b'<b>\\u2603</b>', repr(soup))
+
 class TestNavigableStringSubclasses(SoupTest):
 
     def test_cdata(self):
@@ -1522,6 +1618,9 @@ class TestNavigableStringSubclasses(SoupTest):
         soup.insert(1, doctype)
         self.assertEqual(soup.encode(), b"<!DOCTYPE foo>\n")
 
+    def test_declaration(self):
+        d = Declaration("foo")
+        self.assertEqual("<?foo?>", d.output_ready())
 
 class TestSoupSelector(TreeTest):
 
@@ -1534,7 +1633,7 @@ class TestSoupSelector(TreeTest):
 <link rel="stylesheet" href="blah.css" type="text/css" id="l1">
 </head>
 <body>
-
+<custom-dashed-tag class="dashed" id="dash1">Hello there.</custom-dashed-tag>
 <div id="main" class="fancy">
 <div id="inner">
 <h1 id="header1">An H1</h1>
@@ -1552,8 +1651,18 @@ class TestSoupSelector(TreeTest):
 <a href="#" id="s2a1">span2a1</a>
 </span>
 <span class="span3"></span>
+<custom-dashed-tag class="dashed" id="dash2"/>
+<div data-tag="dashedvalue" id="data1"/>
 </span>
 </div>
+<x id="xid">
+<z id="zida"/>
+<z id="zidab"/>
+<z id="zidac"/>
+</x>
+<y id="yid">
+<z id="zidb"/>
+</y>
 <p lang="en" id="lang-en">English</p>
 <p lang="en-gb" id="lang-en-gb">English UK</p>
 <p lang="en-us" id="lang-en-us">English US</p>
@@ -1565,7 +1674,7 @@ class TestSoupSelector(TreeTest):
 """
 
     def setUp(self):
-        self.soup = BeautifulSoup(self.HTML)
+        self.soup = BeautifulSoup(self.HTML, 'html.parser')
 
     def assertSelects(self, selector, expected_ids):
         el_ids = [el['id'] for el in self.soup.select(selector)]
@@ -1591,17 +1700,25 @@ class TestSoupSelector(TreeTest):
 
     def test_one_tag_many(self):
         els = self.soup.select('div')
-        self.assertEqual(len(els), 3)
+        self.assertEqual(len(els), 4)
         for div in els:
             self.assertEqual(div.name, 'div')
 
+        el = self.soup.select_one('div')
+        self.assertEqual('main', el['id'])
+
+    def test_select_one_returns_none_if_no_match(self):
+        match = self.soup.select_one('nonexistenttag')
+        self.assertEqual(None, match)
+
+
     def test_tag_in_tag_one(self):
         els = self.soup.select('div div')
-        self.assertSelects('div div', ['inner'])
+        self.assertSelects('div div', ['inner', 'data1'])
 
     def test_tag_in_tag_many(self):
         for selector in ('html div', 'html body div', 'body div'):
-            self.assertSelects(selector, ['main', 'inner', 'footer'])
+            self.assertSelects(selector, ['data1', 'main', 'inner', 'footer'])
 
     def test_tag_no_match(self):
         self.assertEqual(len(self.soup.select('del')), 0)
@@ -1609,6 +1726,20 @@ class TestSoupSelector(TreeTest):
     def test_invalid_tag(self):
         self.assertRaises(ValueError, self.soup.select, 'tag%t')
 
+    def test_select_dashed_tag_ids(self):
+        self.assertSelects('custom-dashed-tag', ['dash1', 'dash2'])
+
+    def test_select_dashed_by_id(self):
+        dashed = self.soup.select('custom-dashed-tag[id=\"dash2\"]')
+        self.assertEqual(dashed[0].name, 'custom-dashed-tag')
+        self.assertEqual(dashed[0]['id'], 'dash2')
+
+    def test_dashed_tag_text(self):
+        self.assertEqual(self.soup.select('body > custom-dashed-tag')[0].text, u'Hello there.')
+
+    def test_select_dashed_matches_find_all(self):
+        self.assertEqual(self.soup.select('custom-dashed-tag'), self.soup.find_all('custom-dashed-tag'))
+
     def test_header_tags(self):
         self.assertSelectMultiple(
             ('h1', ['header1']),
@@ -1709,6 +1840,7 @@ class TestSoupSelector(TreeTest):
             ('[id^="m"]', ['me', 'main']),
             ('div[id^="m"]', ['main']),
             ('a[id^="m"]', ['me']),
+            ('div[data-tag^="dashed"]', ['data1'])
         )
 
     def test_attribute_endswith(self):
@@ -1716,8 +1848,8 @@ class TestSoupSelector(TreeTest):
             ('[href$=".css"]', ['l1']),
             ('link[href$=".css"]', ['l1']),
             ('link[id$="1"]', ['l1']),
-            ('[id$="1"]', ['l1', 'p1', 'header1', 's1a1', 's2a1', 's1a2s1']),
-            ('div[id$="1"]', []),
+            ('[id$="1"]', ['data1', 'l1', 'p1', 'header1', 's1a1', 's2a1', 's1a2s1', 'dash1']),
+            ('div[id$="1"]', ['data1']),
             ('[id$="noending"]', []),
         )
 
@@ -1730,7 +1862,6 @@ class TestSoupSelector(TreeTest):
             ('[rel*="notstyle"]', []),
             ('link[rel*="notstyle"]', []),
             ('link[href*="bla"]', ['l1']),
-            ('a[href*="http://"]', ['bob', 'me']),
             ('[href*="http://"]', ['bob', 'me']),
             ('[id*="p"]', ['pmulti', 'p1']),
             ('div[id*="m"]', ['main']),
@@ -1739,8 +1870,8 @@ class TestSoupSelector(TreeTest):
             ('[href*=".css"]', ['l1']),
             ('link[href*=".css"]', ['l1']),
             ('link[id*="1"]', ['l1']),
-            ('[id*="1"]', ['l1', 'p1', 'header1', 's1a1', 's1a2', 's2a1', 's1a2s1']),
-            ('div[id*="1"]', []),
+            ('[id*="1"]', ['data1', 'l1', 'p1', 'header1', 's1a1', 's1a2', 's2a1', 's1a2s1', 'dash1']),
+            ('div[id*="1"]', ['data1']),
             ('[id*="noending"]', []),
             # New for this test
             ('[href*="."]', ['bob', 'me', 'l1']),
@@ -1748,6 +1879,7 @@ class TestSoupSelector(TreeTest):
             ('link[href*="."]', ['l1']),
             ('div[id*="n"]', ['main', 'inner']),
             ('div[id*="nn"]', ['inner']),
+            ('div[data-tag*="edval"]', ['data1'])
         )
 
     def test_attribute_exact_or_hypen(self):
@@ -1767,8 +1899,17 @@ class TestSoupSelector(TreeTest):
             ('p[class]', ['p1', 'pmulti']),
             ('[blah]', []),
             ('p[blah]', []),
+            ('div[data-tag]', ['data1'])
         )
 
+    def test_unsupported_pseudoclass(self):
+        self.assertRaises(
+            NotImplementedError, self.soup.select, "a:no-such-pseudoclass")
+
+        self.assertRaises(
+            NotImplementedError, self.soup.select, "a:nth-of-type(a)")
+
+
     def test_nth_of_type(self):
         # Try to select first paragraph
         els = self.soup.select('div#inner p:nth-of-type(1)')
@@ -1803,7 +1944,7 @@ class TestSoupSelector(TreeTest):
         selected = inner.select("div")
         # The <div id="inner"> tag was selected. The <div id="footer">
         # tag was not.
-        self.assertSelectsIDs(selected, ['inner'])
+        self.assertSelectsIDs(selected, ['inner', 'data1'])
 
     def test_overspecified_child_id(self):
         self.assertSelects(".fancy #inner", ['inner'])
@@ -1827,3 +1968,44 @@ class TestSoupSelector(TreeTest):
 
     def test_sibling_combinator_wont_select_same_tag_twice(self):
         self.assertSelects('p[lang] ~ p', ['lang-en-gb', 'lang-en-us', 'lang-fr'])
+
+    # Test the selector grouping operator (the comma)
+    def test_multiple_select(self):
+        self.assertSelects('x, y', ['xid', 'yid'])
+
+    def test_multiple_select_with_no_space(self):
+        self.assertSelects('x,y', ['xid', 'yid'])
+
+    def test_multiple_select_with_more_space(self):
+        self.assertSelects('x,    y', ['xid', 'yid'])
+
+    def test_multiple_select_duplicated(self):
+        self.assertSelects('x, x', ['xid'])
+
+    def test_multiple_select_sibling(self):
+        self.assertSelects('x, y ~ p[lang=fr]', ['xid', 'lang-fr'])
+
+    def test_multiple_select_tag_and_direct_descendant(self):
+        self.assertSelects('x, y > z', ['xid', 'zidb'])
+
+    def test_multiple_select_direct_descendant_and_tags(self):
+        self.assertSelects('div > x, y, z', ['xid', 'yid', 'zida', 'zidb', 'zidab', 'zidac'])
+
+    def test_multiple_select_indirect_descendant(self):
+        self.assertSelects('div x,y,  z', ['xid', 'yid', 'zida', 'zidb', 'zidab', 'zidac'])
+
+    def test_invalid_multiple_select(self):
+        self.assertRaises(ValueError, self.soup.select, ',x, y')
+        self.assertRaises(ValueError, self.soup.select, 'x,,y')
+
+    def test_multiple_select_attrs(self):
+        self.assertSelects('p[lang=en], p[lang=en-gb]', ['lang-en', 'lang-en-gb'])
+
+    def test_multiple_select_ids(self):
+        self.assertSelects('x, y > z[id=zida], z[id=zidab], z[id=zidb]', ['xid', 'zidb', 'zidab'])
+
+    def test_multiple_select_nested(self):
+        self.assertSelects('body > div > x, y > z', ['xid', 'zidb'])
+
+
+

Contents/Libraries/Shared/chardet/README.rst

@@ -4,7 +4,7 @@ Chardet: The Universal Character Encoding Detector
 Detects
  - ASCII, UTF-8, UTF-16 (2 variants), UTF-32 (4 variants)
  - Big5, GB2312, EUC-TW, HZ-GB-2312, ISO-2022-CN (Traditional and Simplified Chinese)
- - EUC-JP, SHIFT_JIS, ISO-2022-JP (Japanese)
+ - EUC-JP, SHIFT_JIS, CP932, ISO-2022-JP (Japanese)
  - EUC-KR, ISO-2022-KR (Korean)
  - KOI8-R, MacCyrillic, IBM855, IBM866, ISO-8859-5, windows-1251 (Cyrillic)
  - ISO-8859-2, windows-1250 (Hungarian)
@@ -16,6 +16,14 @@ Detects
 
 Requires Python 2.6 or later
 
+Installation
+------------
+
+Install from `PyPI <https://pypi.python.org/pypi/chardet>`_::
+
+    pip install chardet
+
+
 Command-line Tool
 -----------------
 
@@ -31,7 +39,7 @@ About
 
 This is a continuation of Mark Pilgrim's excellent chardet. Previously, two
 versions needed to be maintained: one that supported python 2.x and one that
-supported python 3.x.  We've recently merged with `Ian Corduscano <https://github.com/sigmavirus24>`_'s
+supported python 3.x.  We've recently merged with `Ian Cordasco <https://github.com/sigmavirus24>`_'s
 `charade <https://github.com/sigmavirus24/charade>`_ fork, so now we have one
 coherent version that works for Python 2.6+.
 

Contents/Libraries/Shared/chardet/__init__.py

@@ -15,7 +15,7 @@
 # 02110-1301  USA
 ######################### END LICENSE BLOCK #########################
 
-__version__ = "2.2.1"
+__version__ = "2.3.0"
 from sys import version_info
 
 

Contents/Libraries/Shared/chardet/chardetect.py

@@ -12,34 +12,68 @@ Example::
 If no paths are provided, it takes its input from stdin.
 
 """
-from io import open
-from sys import argv, stdin
 
+from __future__ import absolute_import, print_function, unicode_literals
+
+import argparse
+import sys
+from io import open
+
+from chardet import __version__
 from chardet.universaldetector import UniversalDetector
 
 
-def description_of(file, name='stdin'):
-    """Return a string describing the probable encoding of a file."""
+def description_of(lines, name='stdin'):
+    """
+    Return a string describing the probable encoding of a file or
+    list of strings.
+
+    :param lines: The lines to get the encoding of.
+    :type lines: Iterable of bytes
+    :param name: Name of file or collection of lines
+    :type name: str
+    """
     u = UniversalDetector()
-    for line in file:
+    for line in lines:
         u.feed(line)
     u.close()
     result = u.result
     if result['encoding']:
-        return '%s: %s with confidence %s' % (name,
-                                              result['encoding'],
-                                              result['confidence'])
+        return '{0}: {1} with confidence {2}'.format(name, result['encoding'],
+                                                     result['confidence'])
     else:
-        return '%s: no result' % name
+        return '{0}: no result'.format(name)
 
 
-def main():
-    if len(argv) <= 1:
-        print(description_of(stdin))
-    else:
-        for path in argv[1:]:
-            with open(path, 'rb') as f:
-                print(description_of(f, path))
+def main(argv=None):
+    '''
+    Handles command line arguments and gets things started.
+
+    :param argv: List of arguments, as if specified on the command-line.
+                 If None, ``sys.argv[1:]`` is used instead.
+    :type argv: list of str
+    '''
+    # Get command line arguments
+    parser = argparse.ArgumentParser(
+        description="Takes one or more file paths and reports their detected \
+                     encodings",
+        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
+        conflict_handler='resolve')
+    parser.add_argument('input',
+                        help='File whose encoding we would like to determine.',
+                        type=argparse.FileType('rb'), nargs='*',
+                        default=[sys.stdin])
+    parser.add_argument('--version', action='version',
+                        version='%(prog)s {0}'.format(__version__))
+    args = parser.parse_args(argv)
+
+    for f in args.input:
+        if f.isatty():
+            print("You are running chardetect interactively. Press " +
+                  "CTRL-D twice at the start of a blank line to signal the " +
+                  "end of your input. If you want help, run chardetect " +
+                  "--help\n", file=sys.stderr)
+        print(description_of(f, f.name))
 
 
 if __name__ == '__main__':

Contents/Libraries/Shared/chardet/jpcntx.py

@@ -177,6 +177,12 @@ class JapaneseContextAnalysis:
         return -1, 1
 
 class SJISContextAnalysis(JapaneseContextAnalysis):
+    def __init__(self):
+        self.charset_name = "SHIFT_JIS"
+
+    def get_charset_name(self):
+        return self.charset_name
+
     def get_order(self, aBuf):
         if not aBuf:
             return -1, 1
@@ -184,6 +190,8 @@ class SJISContextAnalysis(JapaneseContextAnalysis):
         first_char = wrap_ord(aBuf[0])
         if ((0x81 <= first_char <= 0x9F) or (0xE0 <= first_char <= 0xFC)):
             charLen = 2
+            if (first_char == 0x87) or (0xFA <= first_char <= 0xFC):
+                self.charset_name = "CP932"
         else:
             charLen = 1
 

Contents/Libraries/Shared/chardet/latin1prober.py

@@ -129,11 +129,11 @@ class Latin1Prober(CharSetProber):
         if total < 0.01:
             confidence = 0.0
         else:
-            confidence = ((self._mFreqCounter[3] / total)
-                          - (self._mFreqCounter[1] * 20.0 / total))
+            confidence = ((self._mFreqCounter[3] - self._mFreqCounter[1] * 20.0)
+                          / total)
         if confidence < 0.0:
             confidence = 0.0
         # lower the confidence of latin1 so that other more accurate
         # detector can take priority.
-        confidence = confidence * 0.5
+        confidence = confidence * 0.73
         return confidence

Contents/Libraries/Shared/chardet/mbcssm.py

@@ -353,7 +353,7 @@ SJIS_cls = (
     2,2,2,2,2,2,2,2,  # 68 - 6f
     2,2,2,2,2,2,2,2,  # 70 - 77
     2,2,2,2,2,2,2,1,  # 78 - 7f
-    3,3,3,3,3,3,3,3,  # 80 - 87
+    3,3,3,3,3,2,2,3,  # 80 - 87
     3,3,3,3,3,3,3,3,  # 88 - 8f
     3,3,3,3,3,3,3,3,  # 90 - 97
     3,3,3,3,3,3,3,3,  # 98 - 9f
@@ -369,9 +369,8 @@ SJIS_cls = (
     2,2,2,2,2,2,2,2,  # d8 - df
     3,3,3,3,3,3,3,3,  # e0 - e7
     3,3,3,3,3,4,4,4,  # e8 - ef
-    4,4,4,4,4,4,4,4,  # f0 - f7
-    4,4,4,4,4,0,0,0   # f8 - ff
-)
+    3,3,3,3,3,3,3,3,  # f0 - f7
+    3,3,3,3,3,0,0,0)  # f8 - ff
 
 
 SJIS_st = (
@@ -571,5 +570,3 @@ UTF8SMModel = {'classTable': UTF8_cls,
                'stateTable': UTF8_st,
                'charLenTable': UTF8CharLenTable,
                'name': 'UTF-8'}
-
-# flake8: noqa

Contents/Libraries/Shared/chardet/sjisprober.py

@@ -47,7 +47,7 @@ class SJISProber(MultiByteCharSetProber):
         self._mContextAnalyzer.reset()
 
     def get_charset_name(self):
-        return "SHIFT_JIS"
+        return self._mContextAnalyzer.get_charset_name()
 
     def feed(self, aBuf):
         aLen = len(aBuf)

Contents/Libraries/Shared/chardet/universaldetector.py

@@ -71,9 +71,9 @@ class UniversalDetector:
 
         if not self._mGotData:
             # If the data starts with BOM, we know it is UTF
-            if aBuf[:3] == codecs.BOM:
+            if aBuf[:3] == codecs.BOM_UTF8:
                 # EF BB BF  UTF-8 with BOM
-                self.result = {'encoding': "UTF-8", 'confidence': 1.0}
+                self.result = {'encoding': "UTF-8-SIG", 'confidence': 1.0}
             elif aBuf[:4] == codecs.BOM_UTF32_LE:
                 # FF FE 00 00  UTF-32, little-endian BOM
                 self.result = {'encoding': "UTF-32LE", 'confidence': 1.0}

Contents/Libraries/Shared/enzyme/__init__.py

@@ -8,6 +8,7 @@ __copyright__ = 'Copyright 2013 Antoine Bertin'
 import logging
 from .exceptions import *
 from .mkv import *
+from .subtitle import *
 
 
 logging.getLogger(__name__).addHandler(logging.NullHandler())

Contents/Libraries/Shared/enzyme/mkv.py

@@ -65,30 +65,53 @@ class MKV(object):
                 continue
             if element_name == 'Info':
                 logger.info('Processing element %s from SeekHead at position %d', element_name, element_position)
-                stream.seek(element_position)
-                self.info = Info.fromelement(ebml.parse_element(stream, specs, True, ignore_element_names=['Void', 'CRC-32']))
+                element = self._load_element(stream, specs, element_position)
+                self.info = Info.fromelement(element)
             elif element_name == 'Tracks':
                 logger.info('Processing element %s from SeekHead at position %d', element_name, element_position)
-                stream.seek(element_position)
-                tracks = ebml.parse_element(stream, specs, True, ignore_element_names=['Void', 'CRC-32'])
+                tracks = self._load_element(stream, specs, element_position)
                 self.video_tracks.extend([VideoTrack.fromelement(t) for t in tracks if t['TrackType'].data == VIDEO_TRACK])
                 self.audio_tracks.extend([AudioTrack.fromelement(t) for t in tracks if t['TrackType'].data == AUDIO_TRACK])
                 self.subtitle_tracks.extend([SubtitleTrack.fromelement(t) for t in tracks if t['TrackType'].data == SUBTITLE_TRACK])
             elif element_name == 'Chapters':
                 logger.info('Processing element %s from SeekHead at position %d', element_name, element_position)
-                stream.seek(element_position)
-                self.chapters.extend([Chapter.fromelement(c) for c in ebml.parse_element(stream, specs, True, ignore_element_names=['Void', 'CRC-32'])[0] if c.name == 'ChapterAtom'])
+                element = self._load_element(stream, specs, element_position)
+                self.chapters.extend([Chapter.fromelement(c) for c in element[0] if c.name == 'ChapterAtom'])
             elif element_name == 'Tags':
                 logger.info('Processing element %s from SeekHead at position %d', element_name, element_position)
-                stream.seek(element_position)
-                self.tags.extend([Tag.fromelement(t) for t in ebml.parse_element(stream, specs, True, ignore_element_names=['Void', 'CRC-32'])])
+                element = self._load_element(stream, specs, element_position)
+                self.tags.extend([Tag.fromelement(t) for t in element])
             elif element_name == 'SeekHead' and self.recurse_seek_head:
                 logger.info('Processing element %s from SeekHead at position %d', element_name, element_position)
-                stream.seek(element_position)
-                self._parse_seekhead(ebml.parse_element(stream, specs, True, ignore_element_names=['Void', 'CRC-32']), segment, stream, specs)
+                element = self._load_element(stream, specs, element_position)
+                self._parse_seekhead(element, segment, stream, specs)
             else:
                 logger.debug('Element %s ignored', element_name)
             self._parsed_positions.add(element_position)
+            
+    def _load_element(self,stream, specs, position):
+        stream.seek(position)
+        element = ebml.parse_element(stream,specs)
+        element.load(stream, specs, ignore_element_names=['Void', 'CRC-32'])
+        return element
+    
+    def get_srt_subtitles_track_by_language(self):
+        """get a dictionary of the SRT subtitles track id's indexed by language"""
+        
+        subtitles = dict()
+        for track in self.subtitle_tracks:
+
+            logger.info("Found subtitle language %s, with codec %s and lacing %s",
+                         track.language,track.codec_id,track.lacing)
+            
+            if not track.is_srt():
+                logger.debug("Ignoring subtitle language %s with codec %s",track.language,track.codec_id)
+            elif track.lacing:
+                logger.info("Ignoring subtitle language %s with lacing %s",track.language,track.lacing)
+            else:
+                subtitles[track.language] = track
+
+        return subtitles        
 
     def to_dict(self):
         return {'info': self.info.__dict__, 'video_tracks': [t.__dict__ for t in self.video_tracks],
@@ -103,6 +126,7 @@ class Info(object):
     """Object for the Info EBML element"""
     def __init__(self, title=None, duration=None, date_utc=None, timecode_scale=None, muxing_app=None, writing_app=None):
         self.title = title
+        self.timecode_scale = timecode_scale
         self.duration = timedelta(microseconds=duration * (timecode_scale or 1000000) // 1000) if duration else None
         self.date_utc = date_utc
         self.muxing_app = muxing_app
@@ -119,7 +143,7 @@ class Info(object):
         title = element.get('Title')
         duration = element.get('Duration')
         date_utc = element.get('DateUTC')
-        timecode_scale = element.get('TimecodeScale')
+        timecode_scale = element.get('TimecodeScale',1000000)
         muxing_app = element.get('MuxingApp')
         writing_app = element.get('WritingApp')
         return cls(title, duration, date_utc, timecode_scale, muxing_app, writing_app)
@@ -133,7 +157,7 @@ class Info(object):
 
 class Track(object):
     """Base object for the Tracks EBML element"""
-    def __init__(self, type=None, number=None, name=None, language=None, enabled=None, default=None, forced=None, lacing=None,  # @ReservedAssignment
+    def __init__(self, type=None, number=None, name=None, language=None, enabled=None, default=None, forced=None, lacing=None,
                  codec_id=None, codec_name=None):
         self.type = type
         self.number = number
@@ -154,10 +178,10 @@ class Track(object):
         :type element: :class:`~enzyme.parsers.ebml.Element`
 
         """
-        type = element.get('TrackType')  # @ReservedAssignment
+        type = element.get('TrackType')
         number = element.get('TrackNumber', 0)
         name = element.get('Name')
-        language = element.get('Language')
+        language = element.get('Language','eng')
         enabled = bool(element.get('FlagEnabled', 1))
         default = bool(element.get('FlagDefault', 1))
         forced = bool(element.get('FlagForced', 0))
@@ -256,8 +280,9 @@ class AudioTrack(Track):
 
 class SubtitleTrack(Track):
     """Object for the Tracks EBML element with :data:`SUBTITLE_TRACK` TrackType"""
-    pass
-
+    
+    def is_srt(self):
+        return self.codec_id == 'S_TEXT/UTF8'
 
 class Tag(object):
     """Object for the Tag EBML element"""
@@ -344,8 +369,7 @@ class Chapter(object):
         if chapterdisplays:
             string = chapterdisplays[0].get('ChapString')
             language = chapterdisplays[0].get('ChapLanguage')
-            return cls(start, hidden, enabled, end, string, language)
-        return cls(start, hidden, enabled, end)
+        return cls(start, hidden, enabled, end, string, language)
 
     def __repr__(self):
         return '<%s [%s, enabled=%s]>' % (self.__class__.__name__, self.start, self.enabled)

Contents/Libraries/Shared/enzyme/parsers/ebml/core.py

@@ -38,8 +38,15 @@ READERS = {
     BINARY: read_element_binary
 }
 
+class BaseElement(object):
 
-class Element(object):
+    def __init__(self, id=None, position=None, size=None, data=None):
+        self.id = id
+        self.position = position
+        self.size = size
+        self.data = data
+
+class Element(BaseElement):
     """Base object of EBML
 
     :param int id: id of the element, best represented as hexadecimal (0x18538067 for Matroska Segment element)
@@ -52,14 +59,11 @@ class Element(object):
     :param data: data as read by the corresponding :data:`READERS`
 
     """
-    def __init__(self, id=None, type=None, name=None, level=None, position=None, size=None, data=None):  # @ReservedAssignment
-        self.id = id
+    def __init__(self, id=None, type=None, name=None, level=None, position=None, size=None, data=None):
+        super(Element, self).__init__(id, position, size, data)
         self.type = type
         self.name = name
         self.level = level
-        self.position = position
-        self.size = size
-        self.data = data
 
     def __repr__(self):
         return '<%s [%s, %r]>' % (self.__class__.__name__, self.name, self.data)
@@ -89,7 +93,7 @@ class MasterElement(Element):
         Element(DocType, u'matroska')
 
     """
-    def __init__(self, id=None, name=None, level=None, position=None, size=None, data=None):  # @ReservedAssignment
+    def __init__(self, id=None, name=None, level=None, position=None, size=None, data=None):
         super(MasterElement, self).__init__(id, MASTER, name, level, position, size, data)
 
     def load(self, stream, specs, ignore_element_types=None, ignore_element_names=None, max_level=None):
@@ -137,8 +141,7 @@ class MasterElement(Element):
     def __iter__(self):
         return iter(self.data)
 
-
-def parse(stream, specs, size=None, ignore_element_types=None, ignore_element_names=None, max_level=None):
+def parse(stream, specs, size=None, ignore_element_types=None, ignore_element_names=None, max_level=None, include_element_names=None):
     """Parse a stream for `size` bytes according to the `specs`
 
     :param stream: file-like object from which to read
@@ -148,6 +151,7 @@ def parse(stream, specs, size=None, ignore_element_types=None, ignore_element_na
     :param list ignore_element_types: list of element types to ignore
     :param list ignore_element_names: list of element names to ignore
     :param int max_level: maximum level of elements
+    :param list include_element_names: list of element names to include exclusively, so ignoring all other element names
     :return: parsed data as a tree of :class:`~enzyme.parsers.ebml.core.Element`
     :rtype: list
 
@@ -158,26 +162,32 @@ def parse(stream, specs, size=None, ignore_element_types=None, ignore_element_na
     """
     ignore_element_types = ignore_element_types if ignore_element_types is not None else []
     ignore_element_names = ignore_element_names if ignore_element_names is not None else []
+    include_element_names = include_element_names if include_element_names is not None else []
     start = stream.tell()
     elements = []
     while size is None or stream.tell() - start < size:
         try:
             element = parse_element(stream, specs)
-            if element is None:
+            if element.type is None:
+                logger.error('Element with id 0x%x is not in the specs' % element_id)
+                stream.seek(element_size, 1)
                 continue
-            logger.debug('%s %s parsed', element.__class__.__name__, element.name)
-            if element.type in ignore_element_types or element.name in ignore_element_names:
-                logger.info('%s %s ignored', element.__class__.__name__, element.name)
-                if element.type == MASTER:
-                    stream.seek(element.size, 1)
+            elif element.type in ignore_element_types or element.name in ignore_element_names:
+                logger.info('%s %s %s ignored', element.__class__.__name__, element.name, element.type)
+                stream.seek(element.size, 1)
                 continue
-            if element.type == MASTER:
+            elif len(include_element_names) > 0 and element.name not in include_element_names:
+                stream.seek(element.size, 1)
+                continue
+            elif element.type == MASTER:
                 if max_level is not None and element.level >= max_level:
                     logger.info('Maximum level %d reached for children of %s %s', max_level, element.__class__.__name__, element.name)
                     stream.seek(element.size, 1)
                 else:
                     logger.debug('Loading child elements for %s %s with size %d', element.__class__.__name__, element.name, element.size)
-                    element.data = parse(stream, specs, element.size, ignore_element_types, ignore_element_names, max_level)
+                    element.data = parse(stream, specs, element.size, ignore_element_types, ignore_element_names, max_level,include_element_names)
+            else:
+                element.data = READERS[element.type](stream, element.size)
             elements.append(element)
         except ReadError:
             if size is not None:
@@ -186,21 +196,15 @@ def parse(stream, specs, size=None, ignore_element_types=None, ignore_element_na
     return elements
 
 
-def parse_element(stream, specs, load_children=False, ignore_element_types=None, ignore_element_names=None, max_level=None):
+def parse_element(stream, specs):
     """Extract a single :class:`Element` from the `stream` according to the `specs`
 
     :param stream: file-like object from which to read
     :param dict specs: see :ref:`specs`
-    :param bool load_children: load children elements if the parsed element is a :class:`MasterElement`
-    :param list ignore_element_types: list of element types to ignore
-    :param list ignore_element_names: list of element names to ignore
-    :param int max_level: maximum level for children elements
     :return: the parsed element
     :rtype: :class:`Element`
 
     """
-    ignore_element_types = ignore_element_types if ignore_element_types is not None else []
-    ignore_element_names = ignore_element_names if ignore_element_names is not None else []
     element_id = read_element_id(stream)
     if element_id is None:
         raise ReadError('Cannot read element id')
@@ -208,20 +212,14 @@ def parse_element(stream, specs, load_children=False, ignore_element_types=None,
     if element_size is None:
         raise ReadError('Cannot read element size')
     if element_id not in specs:
-        logger.error('Element with id 0x%x is not in the specs' % element_id)
-        stream.seek(element_size, 1)
-        return None
+        return BaseElement(element_id,stream.tell(),element_size)
     element_type, element_name, element_level = specs[element_id]
     if element_type == MASTER:
         element = MasterElement(element_id, element_name, element_level, stream.tell(), element_size)
-        if load_children:
-            element.data = parse(stream, specs, element.size, ignore_element_types, ignore_element_names, max_level)
     else:
         element = Element(element_id, element_type, element_name, element_level, stream.tell(), element_size)
-        element.data = READERS[element_type](stream, element_size)
     return element
 
-
 def get_matroska_specs(webm_only=False):
     """Get the Matroska specs
 

Contents/Libraries/Shared/enzyme/subtitle.py

@@ -0,0 +1,185 @@
+# -*- coding: utf-8 -*-
+from .exceptions import ReadError
+from .parsers import ebml
+from .mkv import MKV
+from .parsers import ebml
+import logging
+import codecs
+import os
+import io
+
+__all__ = ['Subtitle']
+logger = logging.getLogger(__name__)
+
+class Subtitle(object):
+    """Subtitle extractor for Matroska Video File.
+    
+    Currently only SRT subtitles stored without lacing are supported
+    """
+
+    def __init__(self, stream):
+        """Read the available subtitles from a MKV file-like object"""
+        self._stream = stream
+        #Use the MKV class to parse the META information
+        mkv = MKV(stream)
+        self._timecode_scale = mkv.info.timecode_scale
+        self._subtitles = mkv.get_srt_subtitles_track_by_language()
+
+    def has_subtitle(self, language):
+        return language in self._subtitles
+
+    def write_subtitle_to_stream(self, language):
+        """Write a single subtitle to stream or return None if language not available"""
+        if language in self._subtitles:
+            subtitle = self._subtitles[language]            
+            return _write_track_to_srt_stream(self._stream,subtitle.number,self._timecode_scale)
+            logger.info("Writing subtitle for language %s to stream",language)            
+        else:
+            logger.info("Subtitle for language %s not found",language)
+
+    def write_subtitles_to_stream(self):
+        """Write all available subtitles as streams to a dictionary with language as the key"""        
+        subtitles = dict()
+        for language in self._subtitles:
+            subtitles[language] = self.write_subtitle_to_stream(language)
+        return subtitles
+            
+def _write_track_to_srt_stream(mkv_stream, track, timecode_scale):
+    
+    srt_stream = io.StringIO()
+    index = 0
+    for cluster in _parse_segment(mkv_stream,track):
+        for blockgroup in cluster.blockgroups:
+            index = index + 1
+            timeRange = _print_time_range(timecode_scale,cluster.timecode,blockgroup.block.timecode,blockgroup.duration)
+            srt_stream.write(str(index) + '\n')
+            srt_stream.write(timeRange + '\n')
+            srt_stream.write(codecs.decode(blockgroup.block.data.read(),'utf-8') + '\n')
+            srt_stream.write('\n')
+    return srt_stream
+            
+def _parse_segment(stream,track):
+    
+    stream.seek(0)
+    specs = ebml.get_matroska_specs()
+
+    # Find all level 1 Cluster elements and its subelements. Speed up this process by excluding all other currently known level 1 elements
+    try:
+        segments = ebml.parse(stream, specs,include_element_names=['Segment','Cluster','BlockGroup','Timecode','Block','BlockDuration',],max_level=3)
+    except ReadError:
+        pass
+    
+    clusters = []
+    for cluster in segments[0].data:
+        _parse_cluster(track, clusters, cluster)
+    return clusters
+
+def _parse_cluster(track, clusters, cluster):
+
+    blockgroups = []
+    timecode = None
+    for child in cluster.data:
+        if child.name == 'BlockGroup':
+            _parse_blockgroup(track, blockgroups, child)
+        elif child.name == 'Timecode':
+            timecode = child.data
+    
+    if len(blockgroups) > 0 and timecode != None:
+        clusters.append(Cluster(timecode, blockgroups))
+
+def _parse_blockgroup(track, blockgroups, blockgroup):
+    
+    block = None
+    duration = None
+    for child in blockgroup.data:
+        if child.name == 'Block':
+            block = Block.fromelement(child)
+            if block.track != track:
+                block = None
+        elif child.name == 'BlockDuration':
+            duration = child.data
+    
+    if duration != None and block != None:
+        blockgroups.append(BlockGroup(block, duration))
+
+def _print_time_range(timecode_scale,clusterTimecode,blockTimecode,duration):
+
+    timecode_scale_ms = timecode_scale / 1000000 #Timecode
+    rawTimecode = clusterTimecode + blockTimecode        
+    startTimeMilleSeconds = (rawTimecode) * timecode_scale_ms
+    endTimeMilleSeconds = (rawTimecode + duration) * timecode_scale_ms
+    
+    return _print_time(startTimeMilleSeconds) + " --> " + _print_time(endTimeMilleSeconds)
+
+def _print_time(timeInMilleSeconds):
+
+    timeInSeconds, milleSeconds = divmod(timeInMilleSeconds, 1000)
+    timeInMinutes, seconds = divmod(timeInSeconds, 60)
+    hours, minutes = divmod(timeInMinutes, 60)
+    
+    return '%d:%02d:%02d,%d' % (hours,minutes,seconds,milleSeconds)
+
+class Cluster(object):
+    
+    def __init__(self,timecode=None, blockgroups=[]):
+        self.timecode = timecode
+        self.blockgroups = blockgroups
+
+class BlockGroup(object):
+    
+    def __init__(self,block=None,duration=None):
+        self.block = block
+        self.duration = duration
+
+class Block(object):
+    
+    def __init__(self, track=None, timecode=None, invisible=False, lacing=None, flags=None, data=None):
+        self.track = track
+        self.timecode = timecode
+        self.invisible = invisible
+        self.lacing = lacing
+        self.flags = flags
+        self.data = data
+    
+    @classmethod
+    def fromelement(cls,element):
+        stream = element.data
+        track = ebml.read_element_size(stream)
+        timecode = ebml.read_element_integer(stream,2)
+        flags = ord(stream.read(1))
+        
+        invisible = bool(flags & 0x8)
+        
+        if (flags & 0x6):
+            lacing = 'EBML'
+        elif (flags & 0x4):
+            lacing = 'fixed-size'
+        elif (flags & 0x2):
+            lacing = 'Xiph'
+        else:
+            lacing = None
+    
+        if lacing:
+            raise ReadError('Laced blocks are not implemented yet')
+        
+        data = ebml.read_element_binary(stream, element.size - stream.tell())    
+        return cls(track,timecode,invisible,lacing,flags,data)    
+
+    def __repr__(self):
+        return '<%s track=%d, timecode=%d, invisible=%d, lacing=%s>' % (self.__class__.__name__, self.track,self.timecode,self.invisible,self.lacing)
+
+class SimpleBlock(Block):
+    
+    def __init__(self, track=None, timecode=None, keyframe=False, invisible=False, lacing=None, flags=None, data=None, discardable=False):
+        super(SimpleBlock,self).__init__(track,timecode,invisible,lacing,flags,data)
+        self.keyframe = keyframe
+        self.discardable = discardable
+        
+    def fromelement(cls,element):
+        simpleblock = super(SimpleBlock, cls).fromelement(element)
+        simpleblock.keyframe = bool(simpleblock.flags & 0x80)
+        simpleblock.discardable = bool(simpleblock.flags & 0x1)
+        return simpleblock
+
+    def __repr__(self):
+        return '<%s track=%d, timecode=%d, keyframe=%d, invisible=%d, lacing=%s, discardable=%d>' % (self.__class__.__name__, self.track,self.timecode,self.keyframe,self.invisible,self.lacing,self.discardable)

Contents/Libraries/Shared/enzyme/tests/__init__.py

@@ -1,9 +1,11 @@
 # -*- coding: utf-8 -*-
-from . import test_mkv, test_parsers
+from . import test_mkv, test_parsers, test_subtitle
 import unittest
 
 
-suite = unittest.TestSuite([test_mkv.suite(), test_parsers.suite()])
+suite = unittest.TestSuite([test_mkv.suite(), test_parsers.suite(), test_subtitle.suite()])
+
+
 
 
 if __name__ == '__main__':

Contents/Libraries/Shared/enzyme/tests/test_mkv.py

@@ -193,7 +193,7 @@ class MKVTestCase(unittest.TestCase):
         self.assertTrue(mkv.audio_tracks[0].type == AUDIO_TRACK)
         self.assertTrue(mkv.audio_tracks[0].number == 2)
         self.assertTrue(mkv.audio_tracks[0].name is None)
-        self.assertTrue(mkv.audio_tracks[0].language is None)
+        self.assertTrue(mkv.audio_tracks[0].language == 'eng')
         self.assertTrue(mkv.audio_tracks[0].enabled == True)
         self.assertTrue(mkv.audio_tracks[0].default == True)
         self.assertTrue(mkv.audio_tracks[0].forced == False)
@@ -276,7 +276,7 @@ class MKVTestCase(unittest.TestCase):
         self.assertTrue(mkv.audio_tracks[1].type == AUDIO_TRACK)
         self.assertTrue(mkv.audio_tracks[1].number == 10)
         self.assertTrue(mkv.audio_tracks[1].name == 'Commentary')
-        self.assertTrue(mkv.audio_tracks[1].language is None)
+        self.assertTrue(mkv.audio_tracks[1].language == 'eng')
         self.assertTrue(mkv.audio_tracks[1].enabled == True)
         self.assertTrue(mkv.audio_tracks[1].default == False)
         self.assertTrue(mkv.audio_tracks[1].forced == False)
@@ -292,7 +292,7 @@ class MKVTestCase(unittest.TestCase):
         self.assertTrue(mkv.subtitle_tracks[0].type == SUBTITLE_TRACK)
         self.assertTrue(mkv.subtitle_tracks[0].number == 3)
         self.assertTrue(mkv.subtitle_tracks[0].name is None)
-        self.assertTrue(mkv.subtitle_tracks[0].language is None)
+        self.assertTrue(mkv.subtitle_tracks[0].language == 'eng')
         self.assertTrue(mkv.subtitle_tracks[0].enabled == True)
         self.assertTrue(mkv.subtitle_tracks[0].default == True)
         self.assertTrue(mkv.subtitle_tracks[0].forced == False)

Contents/Libraries/Shared/enzyme/tests/test_parsers.py

@@ -33,7 +33,7 @@ class EBMLTestCase(unittest.TestCase):
         self.stream.close()
 
     def check_element(self, element_id, element_type, element_name, element_level, element_position, element_size, element_data, element,
-                      ignore_element_types=None, ignore_element_names=None, max_level=None):
+                      ignore_element_types=None, ignore_element_names=None, max_level=None, include_element_names=None):
         """Recursively check an element"""
         # base
         self.assertTrue(element.id == element_id)
@@ -53,6 +53,8 @@ class EBMLTestCase(unittest.TestCase):
             element_data = [e for e in element_data if e[1] not in ignore_element_types]
         if ignore_element_names is not None:  # filter validation on element names
             element_data = [e for e in element_data if e[2] not in ignore_element_names]
+        if include_element_names is not None:  # filter validation on element names
+            element_data = [e for e in element_data if e[2] in include_element_names]
         if element.level == max_level:  # special check when maximum level is reached
             self.assertTrue(element.data is None)
             return
@@ -60,7 +62,7 @@ class EBMLTestCase(unittest.TestCase):
         for i in range(len(element.data)):
             self.check_element(element_data[i][0], element_data[i][1], element_data[i][2], element_data[i][3],
                                element_data[i][4], element_data[i][5], element_data[i][6], element.data[i], ignore_element_types,
-                               ignore_element_names, max_level)
+                               ignore_element_names, max_level,include_element_names)
 
     def test_parse_full(self):
         result = ebml.parse(self.stream, self.specs)
@@ -87,6 +89,15 @@ class EBMLTestCase(unittest.TestCase):
             self.check_element(self.validation[i][0], self.validation[i][1], self.validation[i][2], self.validation[i][3],
                                self.validation[i][4], self.validation[i][5], self.validation[i][6], result[i], ignore_element_names=ignore_element_names)
 
+    def test_parse_include_element_names(self):
+        include_element_names = ['Segment','Cluster']
+        result = ebml.parse(self.stream, self.specs, include_element_names=include_element_names)
+        self.validation = [e for e in self.validation if e[2] in include_element_names]
+        self.assertTrue(len(result) == len(self.validation))
+        for i in range(len(self.validation)):
+            self.check_element(self.validation[i][0], self.validation[i][1], self.validation[i][2], self.validation[i][3],
+                               self.validation[i][4], self.validation[i][5], self.validation[i][6], result[i], include_element_names=include_element_names)
+            
     def test_parse_max_level(self):
         max_level = 3
         result = ebml.parse(self.stream, self.specs, max_level=max_level)

Contents/Libraries/Shared/enzyme/tests/test_subtitle.py

@@ -0,0 +1,86 @@
+# -*- coding: utf-8 -*-
+from enzyme.subtitle import Subtitle, _print_time_range, _print_time
+import unittest
+import os
+import io
+import requests
+import zipfile
+import glob
+
+# Test directory
+TEST_DIR = os.path.join(os.path.dirname(__file__), os.path.splitext(__file__)[0])
+
+def setUpModule():
+    if not os.path.exists(TEST_DIR):
+        r = requests.get('http://downloads.sourceforge.net/project/matroska/test_files/matroska_test_w1_1.zip')
+        with zipfile.ZipFile(io.BytesIO(r.content), 'r') as f:
+            f.extractall(TEST_DIR)
+
+class SubtitleTestCase(unittest.TestCase):
+    
+    @classmethod
+    def setUpClass(cls):
+
+        file = 'test5.mkv'
+        stream = io.open(os.path.join(TEST_DIR, file), 'rb')
+        cls.subtitle = Subtitle(stream)
+    
+    def test_subtitles_found(self):
+        
+        subtitles = self.subtitle._subtitles
+        self.assertTrue('eng' in subtitles)
+        self.assertTrue('hun' in subtitles)
+        self.assertTrue('ger' in subtitles)
+        self.assertTrue('fre' in subtitles)
+        self.assertTrue('spa' in subtitles)
+        self.assertTrue('ita' in subtitles)
+        self.assertTrue('jpn' in subtitles)
+        self.assertTrue('und' in subtitles)
+
+    def test_write_subtitle_to_stream(self):
+
+        subtitle_stream = self.subtitle.write_subtitle_to_stream("eng")
+        self.assertIsInstance(subtitle_stream,io.StringIO,"Expecting a StringIO stream")
+
+    def test_write_subtitle_to_stream(self):
+        
+        subtitle_streams = self.subtitle.write_subtitles_to_stream()
+        
+        self.assertIn("eng", subtitle_streams, "Expecting a subtitle stream for language eng")
+        self.assertIsInstance(subtitle_streams["eng"],io.StringIO,"Expecting a StringIO stream")
+        self.assertIn("hun", subtitle_streams, "Expecting a subtitle stream for language hun")
+        self.assertIsInstance(subtitle_streams["hun"],io.StringIO,"Expecting a StringIO stream")
+        self.assertIn("ger", subtitle_streams, "Expecting a subtitle stream for language ger")
+        self.assertIsInstance(subtitle_streams["ger"],io.StringIO,"Expecting a StringIO stream")
+        self.assertIn("fre", subtitle_streams, "Expecting a subtitle stream for language fre")
+        self.assertIsInstance(subtitle_streams["fre"],io.StringIO,"Expecting a StringIO stream")
+        self.assertIn("spa", subtitle_streams, "Expecting a subtitle stream for language spa")
+        self.assertIsInstance(subtitle_streams["spa"],io.StringIO,"Expecting a StringIO stream")
+        self.assertIn("ita", subtitle_streams, "Expecting a subtitle stream for language ita")
+        self.assertIsInstance(subtitle_streams["ita"],io.StringIO,"Expecting a StringIO stream")
+        self.assertIn("jpn", subtitle_streams, "Expecting a subtitle stream for language jpn")
+        self.assertIsInstance(subtitle_streams["jpn"],io.StringIO,"Expecting a StringIO stream")
+        
+    def test_print_time(self):
+        
+        self.assertEqual('0:00:00,0',_print_time(0))
+        self.assertEqual('0:00:00,1',_print_time(1))
+        self.assertEqual('0:00:00,999',_print_time(999))
+        self.assertEqual('0:00:01,0',_print_time(1000))
+        self.assertEqual('0:00:59,999',_print_time(1000*60-1))
+        self.assertEqual('0:01:00,0',_print_time(1000*60))
+        self.assertEqual('0:59:59,999',_print_time(1000*60*60-1))
+        self.assertEqual('1:00:00,0',_print_time(1000*60*60))
+
+    def test_print_time_range(self):
+        
+        self.assertEqual('0:00:00,0 --> 0:00:00,0',_print_time_range(1000000,0,0,0))
+        self.assertEqual('0:01:00,0 --> 0:01:01,0',_print_time_range(1000000,0,60000,1000))
+
+def suite():
+    suite = unittest.TestSuite()
+    suite.addTest(unittest.TestLoader().loadTestsFromTestCase(SubtitleTestCase))
+    return suite
+
+if __name__ == '__main__':
+    unittest.TextTestRunner().run(suite())

Contents/Libraries/Shared/guessit/LICENSE

@@ -0,0 +1,165 @@
+                   GNU LESSER GENERAL PUBLIC LICENSE
+                       Version 3, 29 June 2007
+
+ Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+
+  This version of the GNU Lesser General Public License incorporates
+the terms and conditions of version 3 of the GNU General Public
+License, supplemented by the additional permissions listed below.
+
+  0. Additional Definitions.
+
+  As used herein, "this License" refers to version 3 of the GNU Lesser
+General Public License, and the "GNU GPL" refers to version 3 of the GNU
+General Public License.
+
+  "The Library" refers to a covered work governed by this License,
+other than an Application or a Combined Work as defined below.
+
+  An "Application" is any work that makes use of an interface provided
+by the Library, but which is not otherwise based on the Library.
+Defining a subclass of a class defined by the Library is deemed a mode
+of using an interface provided by the Library.
+
+  A "Combined Work" is a work produced by combining or linking an
+Application with the Library.  The particular version of the Library
+with which the Combined Work was made is also called the "Linked
+Version".
+
+  The "Minimal Corresponding Source" for a Combined Work means the
+Corresponding Source for the Combined Work, excluding any source code
+for portions of the Combined Work that, considered in isolation, are
+based on the Application, and not on the Linked Version.
+
+  The "Corresponding Application Code" for a Combined Work means the
+object code and/or source code for the Application, including any data
+and utility programs needed for reproducing the Combined Work from the
+Application, but excluding the System Libraries of the Combined Work.
+
+  1. Exception to Section 3 of the GNU GPL.
+
+  You may convey a covered work under sections 3 and 4 of this License
+without being bound by section 3 of the GNU GPL.
+
+  2. Conveying Modified Versions.
+
+  If you modify a copy of the Library, and, in your modifications, a
+facility refers to a function or data to be supplied by an Application
+that uses the facility (other than as an argument passed when the
+facility is invoked), then you may convey a copy of the modified
+version:
+
+   a) under this License, provided that you make a good faith effort to
+   ensure that, in the event an Application does not supply the
+   function or data, the facility still operates, and performs
+   whatever part of its purpose remains meaningful, or
+
+   b) under the GNU GPL, with none of the additional permissions of
+   this License applicable to that copy.
+
+  3. Object Code Incorporating Material from Library Header Files.
+
+  The object code form of an Application may incorporate material from
+a header file that is part of the Library.  You may convey such object
+code under terms of your choice, provided that, if the incorporated
+material is not limited to numerical parameters, data structure
+layouts and accessors, or small macros, inline functions and templates
+(ten or fewer lines in length), you do both of the following:
+
+   a) Give prominent notice with each copy of the object code that the
+   Library is used in it and that the Library and its use are
+   covered by this License.
+
+   b) Accompany the object code with a copy of the GNU GPL and this license
+   document.
+
+  4. Combined Works.
+
+  You may convey a Combined Work under terms of your choice that,
+taken together, effectively do not restrict modification of the
+portions of the Library contained in the Combined Work and reverse
+engineering for debugging such modifications, if you also do each of
+the following:
+
+   a) Give prominent notice with each copy of the Combined Work that
+   the Library is used in it and that the Library and its use are
+   covered by this License.
+
+   b) Accompany the Combined Work with a copy of the GNU GPL and this license
+   document.
+
+   c) For a Combined Work that displays copyright notices during
+   execution, include the copyright notice for the Library among
+   these notices, as well as a reference directing the user to the
+   copies of the GNU GPL and this license document.
+
+   d) Do one of the following:
+
+       0) Convey the Minimal Corresponding Source under the terms of this
+       License, and the Corresponding Application Code in a form
+       suitable for, and under terms that permit, the user to
+       recombine or relink the Application with a modified version of
+       the Linked Version to produce a modified Combined Work, in the
+       manner specified by section 6 of the GNU GPL for conveying
+       Corresponding Source.
+
+       1) Use a suitable shared library mechanism for linking with the
+       Library.  A suitable mechanism is one that (a) uses at run time
+       a copy of the Library already present on the user's computer
+       system, and (b) will operate properly with a modified version
+       of the Library that is interface-compatible with the Linked
+       Version.
+
+   e) Provide Installation Information, but only if you would otherwise
+   be required to provide such information under section 6 of the
+   GNU GPL, and only to the extent that such information is
+   necessary to install and execute a modified version of the
+   Combined Work produced by recombining or relinking the
+   Application with a modified version of the Linked Version. (If
+   you use option 4d0, the Installation Information must accompany
+   the Minimal Corresponding Source and Corresponding Application
+   Code. If you use option 4d1, you must provide the Installation
+   Information in the manner specified by section 6 of the GNU GPL
+   for conveying Corresponding Source.)
+
+  5. Combined Libraries.
+
+  You may place library facilities that are a work based on the
+Library side by side in a single library together with other library
+facilities that are not Applications and are not covered by this
+License, and convey such a combined library under terms of your
+choice, if you do both of the following:
+
+   a) Accompany the combined library with a copy of the same work based
+   on the Library, uncombined with any other library facilities,
+   conveyed under the terms of this License.
+
+   b) Give prominent notice with the combined library that part of it
+   is a work based on the Library, and explaining where to find the
+   accompanying uncombined form of the same work.
+
+  6. Revised Versions of the GNU Lesser General Public License.
+
+  The Free Software Foundation may publish revised and/or new versions
+of the GNU Lesser General Public License from time to time. Such new
+versions will be similar in spirit to the present version, but may
+differ in detail to address new problems or concerns.
+
+  Each version is given a distinguishing version number. If the
+Library as you received it specifies that a certain numbered version
+of the GNU Lesser General Public License "or any later version"
+applies to it, you have the option of following the terms and
+conditions either of that published version or of any later version
+published by the Free Software Foundation. If the Library as you
+received it does not specify a version number of the GNU Lesser
+General Public License, you may choose any version of the GNU Lesser
+General Public License ever published by the Free Software Foundation.
+
+  If the Library as you received it specifies that a proxy can decide
+whether future versions of the GNU Lesser General Public License shall
+apply, that proxy's public statement of acceptance of any version is
+permanent authorization for you to choose that version for the
+Library.

Contents/Libraries/Shared/guessit/README.rst

@@ -0,0 +1,227 @@
+GuessIt
+=======
+
+.. image:: http://img.shields.io/pypi/v/guessit.svg
+    :target: https://pypi.python.org/pypi/guessit
+    :alt: Latest Version
+
+.. image:: http://img.shields.io/badge/license-LGPLv3-blue.svg
+    :target: https://pypi.python.org/pypi/guessit
+    :alt: License
+
+.. image:: http://img.shields.io/travis/wackou/guessit.svg?branch=master
+    :target: http://travis-ci.org/wackou/guessit
+    :alt: Build Status
+
+.. image:: http://img.shields.io/coveralls/wackou/guessit.svg?branch=master
+    :target: https://coveralls.io/r/wackou/guessit
+    :alt: Coveralls
+
+`HuBoard <https://huboard.com/wackou/guessit>`_
+
+
+GuessIt is a python library that extracts as much information as
+possible from a video file.
+
+It has a very powerful filename matcher that allows to guess a lot of
+metadata from a video using its filename only. This matcher works with
+both movies and tv shows episodes.
+
+For example, GuessIt can do the following::
+
+    $ guessit "Treme.1x03.Right.Place,.Wrong.Time.HDTV.XviD-NoTV.avi"
+    For: Treme.1x03.Right.Place,.Wrong.Time.HDTV.XviD-NoTV.avi
+    GuessIt found: {
+        [1.00] "mimetype": "video/x-msvideo",
+        [0.80] "episodeNumber": 3,
+        [0.80] "videoCodec": "XviD",
+        [1.00] "container": "avi",
+        [1.00] "format": "HDTV",
+        [0.70] "series": "Treme",
+        [0.50] "title": "Right Place, Wrong Time",
+        [0.80] "releaseGroup": "NoTV",
+        [0.80] "season": 1,
+        [1.00] "type": "episode"
+    }
+
+
+Install
+-------
+
+Installing GuessIt is simple with `pip <http://www.pip-installer.org/>`_::
+
+    $ pip install guessit
+
+or, with `easy_install <http://pypi.python.org/pypi/setuptools>`_::
+
+    $ easy_install guessit
+
+But, you really `shouldn't do that <http://stackoverflow.com/questions/3220404/why-use-pip-over-easy-install>`_.
+
+You can now launch a demo::
+
+    $ guessit -d
+
+and guess your own filename::
+
+    $ guessit "Breaking.Bad.S05E08.720p.MP4.BDRip.[KoTuWa].mkv"
+    For: Breaking.Bad.S05E08.720p.MP4.BDRip.[KoTuWa].mkv
+    GuessIt found: {
+        [1.00] "mimetype": "video/x-matroska",
+        [1.00] "episodeNumber": 8,
+        [0.30] "container": "mkv",
+        [1.00] "format": "BluRay",
+        [0.70] "series": "Breaking Bad",
+        [1.00] "releaseGroup": "KoTuWa",
+        [1.00] "screenSize": "720p",
+        [1.00] "season": 5,
+        [1.00] "type": "episode"
+    }
+
+
+
+Filename matcher
+----------------
+
+The filename matcher is based on pattern matching and is able to recognize many properties from the filename,
+like ``title``, ``year``, ``series``, ``episodeNumber``, ``seasonNumber``,
+``videoCodec``, ``screenSize``, ``language``. Guessed values are cleaned up and given in a readable format
+which may not match exactly the raw filename.
+
+The full list of available properties can be seen in the
+`main documentation <http://guessit.readthedocs.org/en/latest/user/properties.html>`_.
+
+
+Other features
+--------------
+
+GuessIt also allows you to compute a whole lot of hashes from a file,
+namely all the ones you can find in the hashlib python module (md5,
+sha1, ...), but also the Media Player Classic hash that is used (amongst
+others) by OpenSubtitles and SMPlayer, as well as the ed2k hash.
+
+If you have the 'guess-language' python package installed, GuessIt can also
+analyze a subtitle file's contents and detect which language it is written in.
+
+If you have the 'enzyme' python package installed, GuessIt can also detect the
+properties from the actual video file metadata.
+
+
+Usage
+-----
+
+guessit can be use from command line::
+
+    $ guessit
+    usage: guessit [-h] [-t TYPE] [-n] [-c] [-X DISABLED_TRANSFORMERS] [-v]
+                   [-P SHOW_PROPERTY] [-u] [-a] [-y] [-f INPUT_FILE] [-d] [-p]
+                   [-V] [-s] [--version] [-b] [-i INFO] [-S EXPECTED_SERIES]
+                   [-T EXPECTED_TITLE] [-Y] [-D] [-L ALLOWED_LANGUAGES] [-E]
+                   [-C ALLOWED_COUNTRIES] [-G EXPECTED_GROUP]
+                   [filename [filename ...]]
+
+    positional arguments:
+      filename              Filename or release name to guess
+
+    optional arguments:
+      -h, --help            show this help message and exit
+
+    Naming:
+      -t TYPE, --type TYPE  The suggested file type: movie, episode. If undefined,
+                            type will be guessed.
+      -n, --name-only       Parse files as name only. Disable folder parsing,
+                            extension parsing, and file content analysis.
+      -c, --split-camel     Split camel case part of filename.
+      -X DISABLED_TRANSFORMERS, --disabled-transformer DISABLED_TRANSFORMERS
+                            Transformer to disable (can be used multiple time)
+      -S EXPECTED_SERIES, --expected-series EXPECTED_SERIES
+                            Expected series to parse (can be used multiple times)
+      -T EXPECTED_TITLE, --expected-title EXPECTED_TITLE
+                            Expected title (can be used multiple times)
+      -Y, --date-year-first
+                            If short date is found, consider the first digits as
+                            the year.
+      -D, --date-day-first  If short date is found, consider the second digits as
+                            the day.
+      -L ALLOWED_LANGUAGES, --allowed-languages ALLOWED_LANGUAGES
+                            Allowed language (can be used multiple times)
+      -E, --episode-prefer-number
+                            Guess "serie.213.avi" as the episodeNumber 213.
+                            Without this option, it will be guessed as season 2,
+                            episodeNumber 13
+      -C ALLOWED_COUNTRIES, --allowed-country ALLOWED_COUNTRIES
+                            Allowed country (can be used multiple times)
+      -G EXPECTED_GROUP, --expected-group EXPECTED_GROUP
+                            Expected release group (can be used multiple times)
+
+    Output:
+      -v, --verbose         Display debug output
+      -P SHOW_PROPERTY, --show-property SHOW_PROPERTY
+                            Display the value of a single property (title, series,
+                            videoCodec, year, type ...)
+      -u, --unidentified    Display the unidentified parts.
+      -a, --advanced        Display advanced information for filename guesses, as
+                            json output
+      -y, --yaml            Display information for filename guesses as yaml
+                            output (like unit-test)
+      -f INPUT_FILE, --input-file INPUT_FILE
+                            Read filenames from an input file.
+      -d, --demo            Run a few builtin tests instead of analyzing a file
+
+    Information:
+      -p, --properties      Display properties that can be guessed.
+      -V, --values          Display property values that can be guessed.
+      -s, --transformers    Display transformers that can be used.
+      --version             Display the guessit version.
+
+    guessit.io:
+      -b, --bug             Submit a wrong detection to the guessit.io service
+
+    Other features:
+      -i INFO, --info INFO  The desired information type: filename, video,
+                            hash_mpc or a hash from python's hashlib module, such
+                            as hash_md5, hash_sha1, ...; or a list of any of them,
+                            comma-separated
+
+
+It can also be used as a python module::
+
+    >>> from guessit import guess_file_info
+    >>> guess_file_info('Treme.1x03.Right.Place,.Wrong.Time.HDTV.XviD-NoTV.avi')
+    {u'mimetype': 'video/x-msvideo', u'episodeNumber': 3, u'videoCodec': u'XviD', u'container': u'avi', u'format':     u'HDTV', u'series': u'Treme', u'title': u'Right Place, Wrong Time', u'releaseGroup': u'NoTV', u'season': 1, u'type': u'episode'}
+
+
+Support
+-------
+
+The project website for GuessIt is hosted at `ReadTheDocs <http://guessit.readthedocs.org/>`_.
+There you will also find the User guide and Developer documentation.
+
+This project is hosted on GitHub: `<https://github.com/wackou/guessit>`_
+
+Please report issues and/or feature requests via the `bug tracker <https://github.com/wackou/guessit/issues>`_.
+
+You can also report issues using the command-line tool::
+
+    $ guessit --bug "filename.that.fails.avi"
+
+
+Contribute
+----------
+
+GuessIt is under active development, and contributions are more than welcome!
+
+#. Check for open issues or open a fresh issue to start a discussion around a feature idea or a bug.
+   There is a Contributor Friendly tag for issues that should be ideal for people who are not very
+   familiar with the codebase yet.
+#. Fork `the repository`_ on Github to start making your changes to the **master**
+   branch (or branch off of it).
+#. Write a test which shows that the bug was fixed or that the feature works as expected.
+#. Send a pull request and bug the maintainer until it gets merged and published. :)
+
+.. _the repository: https://github.com/wackou/guessit
+
+License
+-------
+
+GuessIt is licensed under the `LGPLv3 license <http://www.gnu.org/licenses/lgpl.html>`_.

Contents/Libraries/Shared/guessit/__version__.py

@@ -17,4 +17,4 @@
 # You should have received a copy of the Lesser GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
-__version__ = '0.11.0'
+__version__ = '0.11.1.dev0'

Contents/Libraries/Shared/plex/__init__.py

@@ -0,0 +1,11 @@
+import logging
+
+log = logging.getLogger(__name__)
+
+__version__ = '0.7.0'
+
+
+try:
+    from plex.client import Plex
+except Exception as ex:
+    log.warn('Unable to import submodules - %s', ex, exc_info=True)

Contents/Libraries/Shared/plex/client.py

@@ -0,0 +1,116 @@
+from plex.core.configuration import ConfigurationManager
+from plex.core.http import HttpClient
+from plex.helpers import has_attribute
+from plex.interfaces import construct_map
+from plex.interfaces.core.base import InterfaceProxy
+from plex.lib.six import add_metaclass
+from plex.objects.core.manager import ObjectManager
+
+import logging
+import socket
+
+log = logging.getLogger(__name__)
+
+
+class PlexClient(object):
+    __interfaces = None
+
+    def __init__(self):
+        # Construct interfaces
+        self.http = HttpClient(self)
+        self.configuration = ConfigurationManager()
+
+        self.__interfaces = construct_map(self)
+
+        # Discover modules
+        ObjectManager.construct()
+
+    @property
+    def base_url(self):
+        host = self.configuration.get('server.host', '127.0.0.1')
+        port = self.configuration.get('server.port', 32400)
+
+        return 'http://%s:%s' % (host, port)
+
+    def __getitem__(self, path):
+        parts = path.strip('/').split('/')
+
+        cur = self.__interfaces
+        parameters = []
+
+        while parts and type(cur) is dict:
+            key = parts.pop(0)
+
+            if key == '*':
+                key = None
+            elif key not in cur:
+                if None in cur:
+                    parameters.append(key)
+
+                    cur = cur[None]
+                    continue
+
+                return None
+
+            cur = cur[key]
+
+        while type(cur) is dict:
+            cur = cur.get(None)
+
+        if parts:
+            parameters.extend(parts)
+
+        if parameters:
+            return InterfaceProxy(cur, parameters)
+
+        return cur
+
+    def __getattr__(self, name):
+        interface = self.__interfaces.get(None)
+
+        if not interface:
+            raise Exception("Root interface not found")
+
+        return getattr(interface, name)
+
+
+class PlexMeta(type):
+    @property
+    def client(cls):
+        if cls._client is None:
+            cls.construct()
+
+        return cls._client
+
+    def __getattr__(self, name):
+        if has_attribute(self, name):
+            return super(PlexMeta, self).__getattribute__(name)
+
+        if self.client is None:
+            self.construct()
+
+        return getattr(self.client, name)
+
+    def __setattr__(self, name, value):
+        if has_attribute(self, name):
+            return super(PlexMeta, self).__setattr__(name, value)
+
+        if self.client is None:
+            self.construct()
+
+        setattr(self.client, name, value)
+
+    def __getitem__(self, key):
+        if self.client is None:
+            self.construct()
+
+        return self.client[key]
+
+
+@add_metaclass(PlexMeta)
+class Plex(object):
+    _client = None
+
+    @classmethod
+    def construct(cls):
+        cls._client = PlexClient()

Contents/Libraries/Shared/plex/core/configuration.py

@@ -0,0 +1,115 @@
+class ConfigurationManager(object):
+    def __init__(self):
+        self.stack = [
+            Configuration(self)
+        ]
+
+    @property
+    def current(self):
+        return self.stack[-1]
+
+    @property
+    def defaults(self):
+        return self.stack[0]
+
+    def authentication(self, token):
+        return Configuration(self).authentication(token)
+
+    def cache(self, **definitions):
+        return Configuration(self).cache(**definitions)
+
+    def client(self, identifier, product, version):
+        return Configuration(self).client(identifier, product, version)
+
+    def device(self, name, system):
+        return Configuration(self).device(name, system)
+
+    def headers(self, headers):
+        return Configuration(self).headers(headers)
+
+    def platform(self, name, version):
+        return Configuration(self).platform(name, version)
+
+    def server(self, host='127.0.0.1', port=32400):
+        return Configuration(self).server(host, port)
+
+    def get(self, key, default=None):
+        for x in range(len(self.stack) - 1, -1, -1):
+            value = self.stack[x].get(key)
+
+            if value is not None:
+                return value
+
+        return default
+
+    def __getitem__(self, key):
+        return self.get(key)
+
+    def __setitem__(self, key, value):
+        self.current[key] = value
+
+
+class Configuration(object):
+    def __init__(self, manager):
+        self.manager = manager
+
+        self.data = {}
+
+    def authentication(self, token):
+        self.data['authentication.token'] = token
+
+        return self
+
+    def cache(self, **definitions):
+        for key, value in definitions.items():
+            self.data['cache.%s' % key] = value
+
+        return self
+
+    def client(self, identifier, product, version):
+        self.data['client.identifier'] = identifier
+
+        self.data['client.product'] = product
+        self.data['client.version'] = version
+
+        return self
+
+    def device(self, name, system):
+        self.data['device.name'] = name
+        self.data['device.system'] = system
+
+        return self
+
+    def headers(self, headers):
+        self.data['headers'] = headers
+
+        return self
+
+    def platform(self, name, version):
+        self.data['platform.name'] = name
+        self.data['platform.version'] = version
+
+        return self
+
+    def server(self, host='127.0.0.1', port=32400):
+        self.data['server.host'] = host
+        self.data['server.port'] = port
+
+        return self
+
+    def get(self, key, default=None):
+        return self.data.get(key, default)
+
+    def __enter__(self):
+        self.manager.stack.append(self)
+
+    def __exit__(self, exc_type, exc_val, exc_tb):
+        item = self.manager.stack.pop()
+
+        assert item == self
+
+    def __getitem__(self, key):
+        return self.data[key]
+
+    def __setitem__(self, key, value):
+        self.data[key] = value

Contents/Libraries/Shared/plex/core/context.py

@@ -0,0 +1,26 @@
+from threading import Lock
+
+
+class Context(object):
+    def __init__(self, **kwargs):
+        self.kwargs = kwargs
+
+    def __getattr__(self, key):
+        return self.kwargs.get(key)
+
+
+class ContextStack(object):
+    def __init__(self):
+        self._list = []
+        self._lock = Lock()
+
+    def pop(self):
+        context = self._list.pop()
+
+        self._lock.release()
+        return context
+
+    def push(self, **kwargs):
+        self._lock.acquire()
+
+        return self._list.append(Context(**kwargs))

Contents/Libraries/Shared/plex/core/extension.py

@@ -0,0 +1,105 @@
+# ExtensionImporter (```flask.exthook```)
+# ----------------------------------
+# :copyright: (c) 2014 by Armin Ronacher.
+# :license: BSD, see LICENSE for more details.
+
+from plex.lib.six import reraise
+
+import os
+import sys
+
+
+class ExtensionImporter(object):
+    """This importer redirects imports from this submodule to other locations.
+    This makes it possible to transition from the old flaskext.name to the
+    newer flask_name without people having a hard time.
+    """
+
+    def __init__(self, module_choices, wrapper_module):
+        self.module_choices = module_choices
+        self.wrapper_module = wrapper_module
+        self.prefix = wrapper_module + '.'
+        self.prefix_cutoff = wrapper_module.count('.') + 1
+
+    def __eq__(self, other):
+        return self.__class__.__module__ == other.__class__.__module__ and \
+               self.__class__.__name__ == other.__class__.__name__ and \
+               self.wrapper_module == other.wrapper_module and \
+               self.module_choices == other.module_choices
+
+    def __ne__(self, other):
+        return not self.__eq__(other)
+
+    def install(self):
+        sys.meta_path[:] = [x for x in sys.meta_path if self != x] + [self]
+
+    def find_module(self, fullname, path=None):
+        if fullname.startswith(self.prefix):
+            return self
+
+    def load_module(self, fullname):
+        if fullname in sys.modules:
+            return sys.modules[fullname]
+        modname = fullname.split('.', self.prefix_cutoff)[self.prefix_cutoff]
+        for path in self.module_choices:
+            realname = path % modname
+            try:
+                __import__(realname)
+            except ImportError:
+                exc_type, exc_value, tb = sys.exc_info()
+                # since we only establish the entry in sys.modules at the
+                # very this seems to be redundant, but if recursive imports
+                # happen we will call into the move import a second time.
+                # On the second invocation we still don't have an entry for
+                # fullname in sys.modules, but we will end up with the same
+                # fake module name and that import will succeed since this
+                # one already has a temporary entry in the modules dict.
+                # Since this one "succeeded" temporarily that second
+                # invocation now will have created a fullname entry in
+                # sys.modules which we have to kill.
+                sys.modules.pop(fullname, None)
+
+                # If it's an important traceback we reraise it, otherwise
+                # we swallow it and try the next choice.  The skipped frame
+                # is the one from __import__ above which we don't care about
+                if self.is_important_traceback(realname, tb):
+                    reraise(exc_type, exc_value, tb.tb_next)
+                continue
+            module = sys.modules[fullname] = sys.modules[realname]
+            if '.' not in modname:
+                setattr(sys.modules[self.wrapper_module], modname, module)
+            return module
+        raise ImportError('No module named %s' % fullname)
+
+    def is_important_traceback(self, important_module, tb):
+        """Walks a traceback's frames and checks if any of the frames
+        originated in the given important module.  If that is the case then we
+        were able to import the module itself but apparently something went
+        wrong when the module was imported.  (Eg: import of an import failed).
+        """
+        while tb is not None:
+            if self.is_important_frame(important_module, tb):
+                return True
+            tb = tb.tb_next
+        return False
+
+    def is_important_frame(self, important_module, tb):
+        """Checks a single frame if it's important."""
+        g = tb.tb_frame.f_globals
+        if '__name__' not in g:
+            return False
+
+        module_name = g['__name__']
+
+        # Python 2.7 Behavior.  Modules are cleaned up late so the
+        # name shows up properly here.  Success!
+        if module_name == important_module:
+            return True
+
+        # Some python versions will will clean up modules so early that the
+        # module name at that point is no longer set.  Try guessing from
+        # the filename then.
+        filename = os.path.abspath(tb.tb_frame.f_code.co_filename)
+        test_string = os.path.sep + important_module.replace('.', os.path.sep)
+        return test_string + '.py' in filename or \
+               test_string + os.path.sep + '__init__.py' in filename

Contents/Libraries/Shared/plex/core/helpers.py

@@ -0,0 +1,59 @@
+from plex.lib import six
+
+import re
+import unicodedata
+
+def flatten(text):
+    if text is None:
+        return None
+
+    # Normalize `text` to ascii
+    text = normalize(text)
+
+    # Remove special characters
+    text = re.sub('[^A-Za-z0-9\s]+', '', text)
+
+    # Merge duplicate spaces
+    text = ' '.join(text.split())
+
+    # Convert to lower-case
+    return text.lower()
+
+def normalize(text):
+    if text is None:
+        return None
+
+    # Normalize unicode characters
+    if type(text) is six.text_type:
+        text = unicodedata.normalize('NFKD', text)
+
+    # Ensure text is ASCII, ignore unknown characters
+    text = text.encode('ascii', 'ignore')
+
+    # Return decoded `text`
+    return text.decode('ascii')
+
+def to_iterable(value):
+    if value is None:
+        return None
+
+    if isinstance(value, (list, tuple)):
+        return value
+
+    return [value]
+
+
+def synchronized(func):
+    def wrapper(self, *__args, **__kw):
+        self._lock.acquire()
+
+        try:
+            return func(self, *__args, **__kw)
+        finally:
+            self._lock.release()
+
+    wrapper.__name__ = func.__name__
+    wrapper.__dict__ = func.__dict__
+    wrapper.__doc__ = func.__doc__
+
+    return wrapper

Contents/Libraries/Shared/plex/core/http.py

@@ -0,0 +1,151 @@
+from plex.core.context import ContextStack
+from plex.core.helpers import synchronized
+from plex.request import PlexRequest
+
+from threading import Condition
+import hashlib
+import logging
+import requests
+import socket
+
+log = logging.getLogger(__name__)
+
+
+class HttpClient(object):
+    def __init__(self, client):
+        self.client = client
+
+        self.configuration = ContextStack()
+
+        self.session = None
+
+        # Private
+        self._lock = Condition()
+
+        # Build requests session
+        self._build()
+
+    @property
+    def cache(self):
+        return self.client.configuration.get('cache.http')
+
+    def configure(self, path=None):
+        self.configuration.push(base_path=path)
+        return self
+
+    def request(self, method, path=None, params=None, query=None, data=None, credentials=None, **kwargs):
+        # retrieve configuration
+        ctx = self.configuration.pop()
+
+        if path is not None and type(path) is not str:
+            # Convert `path` to string (excluding NoneType)
+            path = str(path)
+
+        if ctx.base_path and path:
+            # Prepend `base_path` to relative `path`s
+            if not path.startswith('/'):
+                path = ctx.base_path + '/' + path
+
+        elif ctx.base_path:
+            path = ctx.base_path
+        elif not path:
+            path = ''
+
+        request = PlexRequest(
+            self.client,
+            method=method,
+            path=path,
+
+            params=params,
+            query=query,
+            data=data,
+
+            credentials=credentials,
+            **kwargs
+        )
+
+        prepared = request.prepare()
+
+        # Try retrieve cached response
+        response = self._cache_lookup(prepared)
+
+        if response:
+            return response
+
+        # TODO retrying requests on 502, 503 errors?
+        # try:
+        #     response = self.session.send(prepared)
+        # except socket.gaierror as e:
+        #     code, _ = e
+#
+        #     if code != 8:
+        #         raise e
+#
+        #     log.warn('Encountered socket.gaierror (code: 8)')
+#
+        #     response = self._build().send(prepared)
+        response = request.request.send()
+
+        # Store response in cache
+        self._cache_store(prepared, response)
+
+        return response
+
+    def get(self, path=None, params=None, query=None, data=None, **kwargs):
+        return self.request('GET', path, params, query, data, **kwargs)
+
+    def put(self, path=None, params=None, query=None, data=None, **kwargs):
+        return self.request('PUT', path, params, query, data, **kwargs)
+
+    def post(self, path=None, params=None, query=None, data=None, **kwargs):
+        return self.request('POST', path, params, query, data, **kwargs)
+
+    def delete(self, path=None, params=None, query=None, data=None, **kwargs):
+        return self.request('DELETE', path, params, query, data, **kwargs)
+
+    def _build(self):
+        if self.session:
+            log.info('Rebuilding session and connection pools...')
+
+        # Rebuild the connection pool (old pool has stale connections)
+        self.session = requests.Session()
+
+        return self.session
+
+    @synchronized
+    def _cache_lookup(self, request):
+        if self.cache is None:
+            return None
+
+        if request.method not in ['GET']:
+            return None
+
+        # Retrieve from cache
+        return self.cache.get(self._cache_key(request))
+
+    @synchronized
+    def _cache_store(self, request, response):
+        if self.cache is None:
+            return None
+
+        if request.method not in ['GET']:
+            return None
+
+        # Store in cache
+        self.cache[self._cache_key(request)] = response
+
+    @staticmethod
+    def _cache_key(request):
+        raw = ','.join([request.method, request.url])
+
+        # Generate MD5 hash of key
+        m = hashlib.md5()
+        m.update(raw.encode('utf-8'))
+
+        return m.hexdigest()
+
+    def __enter__(self):
+        return self
+
+    def __exit__(self, exc_type, exc_val, exc_tb):
+        pass

Contents/Libraries/Shared/plex/core/idict.py

@@ -0,0 +1,54 @@
+from plex.lib.six import string_types
+
+class idict(dict):
+    def __init__(self, initial=None):
+        if initial:
+            self.update(initial)
+
+    def get(self, k, d=None):
+        if isinstance(k, string_types):
+            k = k.lower()
+
+        if super(idict, self).__contains__(k):
+            return self[k]
+
+        return d
+
+    def update(self, E=None, **F):
+        if E:
+            if hasattr(E, 'keys'):
+                # Update with `E` dictionary
+                for k in E:
+                    self[k] = E[k]
+            else:
+                # Update with `E` items
+                for (k, v) in E:
+                    self[k] = v
+
+        # Update with `F` dictionary
+        for k in F:
+            self[k] = F[k]
+
+    def __contains__(self, k):
+        if isinstance(k, string_types):
+            k = k.lower()
+
+        return super(idict, self).__contains__(k)
+
+    def __delitem__(self, k):
+        if isinstance(k, string_types):
+            k = k.lower()
+
+        super(idict, self).__delitem__(k)
+
+    def __getitem__(self, k):
+        if isinstance(k, string_types):
+            k = k.lower()
+
+        return super(idict, self).__getitem__(k)
+
+    def __setitem__(self, k, value):
+        if isinstance(k, string_types):
+            k = k.lower()
+
+        super(idict, self).__setitem__(k, value)

Contents/Libraries/Shared/plex/ext/__init__.py

@@ -0,0 +1,4 @@
+from plex.core.extension import ExtensionImporter
+
+importer = ExtensionImporter(['plex_%s'], __name__)
+importer.install()

Contents/Libraries/Shared/plex/helpers.py

@@ -0,0 +1,6 @@
+def has_attribute(obj, name):
+    try:
+        object.__getattribute__(obj, name)
+        return True
+    except AttributeError:
+        return False

Contents/Libraries/Shared/plex/interfaces/__init__.py

@@ -0,0 +1,81 @@
+from plex.interfaces.channel import ChannelInterface
+from plex.interfaces.library import LibraryInterface
+from plex.interfaces.library.metadata import LibraryMetadataInterface
+from plex.interfaces.plugin import PluginInterface
+from plex.interfaces.plugin.preferences import PluginPreferencesInterface
+from plex.interfaces.preferences import PreferencesInterface
+from plex.interfaces.root import RootInterface
+from plex.interfaces.section import SectionInterface
+from plex.interfaces.status import StatusInterface
+from plex.interfaces.timeline import TimelineInterface
+
+
+# TODO automatic interface discovery
+
+INTERFACES = [
+    RootInterface,
+
+    # /
+    ChannelInterface,
+    StatusInterface,
+
+    # /library
+    LibraryInterface,
+    LibraryMetadataInterface,
+    SectionInterface,
+
+    # /:
+    PreferencesInterface,
+    TimelineInterface,
+
+    # /:/plugins
+    PluginInterface,
+    PluginPreferencesInterface
+]
+
+
+def get_interfaces():
+    for interface in INTERFACES:
+        if interface.path:
+            path = interface.path.strip('/')
+        else:
+            path = ''
+
+        if path:
+            path = path.split('/')
+        else:
+            path = []
+
+        yield path, interface
+
+
+def construct_map(client, d=None, interfaces=None):
+    if d is None:
+        d = {}
+
+    if interfaces is None:
+        interfaces = get_interfaces()
+
+    for path, interface in interfaces:
+        if len(path) > 0:
+            key = path.pop(0)
+        else:
+            key = None
+
+        if key == '*':
+            key = None
+
+        if len(path) == 0:
+            d[key] = interface(client)
+            continue
+
+        value = d.get(key, {})
+
+        if type(value) is not dict:
+            value = {None: value}
+
+        construct_map(client, value, [(path, interface)])
+
+        d[key] = value
+
+    return d

Contents/Libraries/Shared/plex/interfaces/channel.py

@@ -0,0 +1,8 @@
+from plex.interfaces.core.base import Interface
+
+
+class ChannelInterface(Interface):
+    path = 'channels'
+
+    def all(self):
+        raise NotImplementedError()

Contents/Libraries/Shared/plex/interfaces/core/base.py

@@ -0,0 +1,216 @@
+from plex.lib.six import string_types, StringIO
+from plex.lib.six.moves.urllib_parse import urlparse
+
+from functools import wraps
+import logging
+
+# Import available parser
+PARSER = None
+
+try:
+    from lxml import etree
+    PARSER = 'etree.HTMLParser'
+except ImportError:
+    from xml.etree import ElementTree as etree
+    PARSER = 'etree.XMLParser'
+
+log = logging.getLogger(__name__)
+
+
+class Helpers(object):
+    @staticmethod
+    def get(node, attr):
+        if PARSER == 'etree.HTMLParser':
+            return node.get(attr.lower())
+
+        return node.get(attr)
+
+    @staticmethod
+    def find(node, tag):
+        if PARSER == 'etree.HTMLParser':
+            return node.find(tag.lower())
+
+        return node.find(tag)
+
+    @staticmethod
+    def findall(node, tag):
+        if PARSER == 'etree.HTMLParser':
+            return node.findall(tag.lower())
+
+        return node.findall(tag)
+
+
+class Interface(object):
+    helpers = Helpers
+
+    path = None
+    object_map = {}
+
+    def __init__(self, client):
+        self.client = client
+
+    def __getitem__(self, name):
+        if hasattr(self, name):
+            return getattr(self, name)
+
+        raise ValueError('Unknown action "%s" on %s', name, self)
+
+    @property
+    def http(self):
+        if not self.client:
+            return None
+
+        return self.client.http.configure(self.path)
+
+    def parse(self, response, schema):
+        if response.status_code < 200 or response.status_code >= 300:
+            return None
+
+        try:
+            root = self.__parse_xml(response.content)
+        except SyntaxError as ex:
+            log.error('Unable to parse XML response: %s', ex, exc_info=True, extra={
+                'data': {
+                    'snippet': self.__error_snippet(response, ex)
+                }
+            })
+
+            return None
+        except Exception as ex:
+            log.error('Unable to parse XML response: %s', ex, exc_info=True)
+
+            return None
+
+        url = urlparse(response.url)
+        path = url.path
+
+        return self.__construct(self.client, path, root, schema)
+
+    @staticmethod
+    def __parse_xml(content):
+        if PARSER == 'etree.HTMLParser':
+            html = etree.fromstring(content, parser=etree.HTMLParser())
+            assert html.tag == 'html'
+
+            bodies = [e for e in html if e.tag == 'body']
+            assert len(bodies) == 1
+
+            body = bodies[0]
+            assert len(body) == 1
+
+            return body[0]
+
+        return etree.fromstring(content)
+
+    @staticmethod
+    def __error_snippet(response, ex):
+        # Retrieve the error line
+        position = getattr(ex, 'position', None)
+
+        if not position or len(position) != 2:
+            return None
+
+        n_line, n_column = position
+        snippet = None
+
+        # Create StringIO stream
+        stream = StringIO(response.text)
+
+        # Iterate over `content` to find `n_line`
+        for x, l in enumerate(stream):
+            if x < n_line - 1:
+                continue
+
+            # Line found
+            snippet = l
+            break
+
+        # Close the stream
+        stream.close()
+
+        if not snippet:
+            # Couldn't find the line
+            return None
+
+        # Find an attribute value containing `n_column`
+        start = snippet.find('"', n_column)
+        end = snippet.find('"', start + 1)
+
+        # Trim `snippet` (if attribute value was found)
+        if start >= 0 and end >= 0:
+            return snippet[start:end + 1]
+
+        return snippet
+
+    @classmethod
+    def __construct(cls, client, path, node, schema):
+        if not schema:
+            return None
+
+        # Try retrieve schema for `tag`
+        item = schema.get(node.tag)
+
+        if item is None:
+            raise ValueError('Unknown node with tag "%s"' % node.tag)
+
+        if type(item) is dict:
+            value = cls.helpers.get(node, item.get('_', 'type'))
+
+            if value is None:
+                return None
+
+            item = item.get(value)
+
+            if item is None:
+                raise ValueError('Unknown node type "%s"' % value)
+
+        descriptor = None
+        child_schema = None
+
+        if type(item) is tuple and len(item) == 2:
+            descriptor, child_schema = item
+        else:
+            descriptor = item
+
+        if isinstance(descriptor, string_types):
+            if descriptor not in cls.object_map:
+                raise Exception('Unable to find descriptor by name "%s"' % descriptor)
+
+            descriptor = cls.object_map.get(descriptor)
+
+        if descriptor is None:
+            raise Exception('Unable to find descriptor')
+
+        keys_used, obj = descriptor.construct(client, node, path=path)
+
+        # Lazy-construct children
+        def iter_children():
+            for child_node in node:
+                item = cls.__construct(client, path, child_node, child_schema)
+
+                if item:
+                    yield item
+
+        obj._children = iter_children()
+
+        return obj
+
+
+class InterfaceProxy(object):
+    def __init__(self, interface, args):
+        self.interface = interface
+        self.args = list(args)
+
+    def __getattr__(self, name):
+        value = getattr(self.interface, name)
+
+        if not hasattr(value, '__call__'):
+            return value
+
+        @wraps(value)
+        def wrap(*args, **kwargs):
+            args = self.args + list(args)
+
+            return value(*args, **kwargs)
+
+        return wrap

Contents/Libraries/Shared/plex/interfaces/library/__init__.py

@@ -0,0 +1,104 @@
+from plex.core.idict import idict
+from plex.interfaces.core.base import Interface
+
+
+class LibraryInterface(Interface):
+    path = 'library'
+
+    def metadata(self, rating_key):
+        response = self.http.get('metadata', rating_key)
+
+        return self.parse(response, idict({
+            'MediaContainer': ('MediaContainer', idict({
+                'Directory': {
+                    'album':    'Album',
+                    'artist':   'Artist',
+
+                    'season':   'Season',
+                    'show':     'Show'
+                },
+                'Video': {
+                    'episode':  'Episode',
+                    'clip':     'Clip',
+                    'movie':    'Movie'
+                },
+
+                'Track': 'Track'
+            }))
+        }))
+
+    def on_deck(self):
+        response = self.http.get('onDeck')
+
+        return self.parse(response, idict({
+            'MediaContainer': ('MediaContainer', idict({
+                'Video': {
+                    'movie':    'Movie',
+                    'episode':  'Episode'
+                }
+            }))
+        }))
+
+    def recently_added(self):
+        response = self.http.get('recentlyAdded')
+
+        return self.parse(response, idict({
+            'MediaContainer': ('MediaContainer', idict({
+                'Directory': {
+                    'album':    'Album',
+                    'season':   'Season'
+                },
+                'Video': {
+                    'movie':    'Movie'
+                }
+            }))
+        }))
+
+    def sections(self):
+        response = self.http.get('sections')
+
+        return self.parse(response, idict({
+            'MediaContainer': ('SectionContainer', idict({
+                'Directory': ('Section', idict({
+                    'Location': 'Location'
+                }))
+            }))
+        }))
+
+    #
+    # Item actions
+    #
+
+    def rate(self, key, rating):
+        response = self.http.get(
+            '/:/rate',
+            query={
+                'identifier': 'com.plexapp.plugins.library',
+                'key': key,
+                'rating': int(round(rating, 0))
+            }
+        )
+
+        return response.status_code == 200
+
+    def scrobble(self, key):
+        response = self.http.get(
+            '/:/scrobble',
+            query={
+                'identifier': 'com.plexapp.plugins.library',
+                'key': key
+            }
+        )
+
+        return response.status_code == 200
+
+    def unscrobble(self, key):
+        response = self.http.get(
+            '/:/unscrobble',
+            query={
+                'identifier': 'com.plexapp.plugins.library',
+                'key': key
+            }
+        )
+
+        return response.status_code == 200

Contents/Libraries/Shared/plex/interfaces/library/metadata.py

@@ -0,0 +1,65 @@
+from plex.core.idict import idict
+from plex.interfaces.core.base import Interface
+
+
+class LibraryMetadataInterface(Interface):
+    path = 'library/metadata'
+
+    def refresh(self, key):
+	response = self.http.put(str(key) + "/refresh")
+
+    def all_leaves(self, key):
+        response = self.http.get(key, 'allLeaves')
+
+        return self.parse(response, idict({
+            'MediaContainer': {
+                '_': 'viewGroup',
+
+                'episode': ('ShowLeavesContainer', idict({
+                    'Video': {
+                        'episode': 'Episode'
+                    }
+                })),
+
+                'track': ('ArtistLeavesContainer', idict({
+                    'Track': 'Track'
+                }))
+            }
+        }))
+
+    def children(self, key):
+        response = self.http.get(key, 'children')
+
+        return self.parse(response, idict({
+            'MediaContainer': {
+                '_': 'viewGroup',
+
+                # ---------------------------------------
+                # Music
+                # ---------------------------------------
+                'album': ('ArtistChildrenContainer', idict({
+                    'Directory': {
+                        'album': 'Album'
+                    }
+                })),
+
+                'track': ('AlbumChildrenContainer', idict({
+                    'Track': 'Track'
+                })),
+
+                # ---------------------------------------
+                # TV
+                # ---------------------------------------
+                'season': ('ShowChildrenContainer', idict({
+                    'Directory': {
+                        'season': 'Season'
+                    }
+                })),
+
+                'episode': ('SeasonChildrenContainer', idict({
+                    'Video': {
+                        'episode': 'Episode'
+                    }
+                }))
+            }
+        }))

Contents/Libraries/Shared/plex/interfaces/plugin/__init__.py

@@ -0,0 +1,13 @@
+from plex.interfaces.core.base import Interface
+
+
+class PluginInterface(Interface):
+    path = ':/plugins'
+
+    def reload_services(self, plugin_id):
+        response = self.http.get(plugin_id, 'services/reload')
+        return response.status_code == 200
+
+    def restart(self, plugin_id):
+        response = self.http.get(plugin_id, 'restart')
+        return response.status_code == 200

Contents/Libraries/Shared/plex/interfaces/plugin/preferences.py

@@ -0,0 +1,40 @@
+from plex.core.idict import idict
+from plex.interfaces.core.base import Interface
+
+
+class PluginPreferencesInterface(Interface):
+    path = ':/plugins/*/prefs'
+
+    def get(self, plugin_id, id=None):
+        response = self.http.get('/:/plugins/%s/prefs' % plugin_id)
+
+        container = self.parse(response, idict({
+            'MediaContainer': ('MediaContainer', idict({
+                'Setting': 'Setting'
+            }))
+        }))
+
+        if container is None or id is None:
+            return container
+
+        for setting in container:
+            if setting.id == id:
+                return setting
+
+        return None
+
+    def set(self, plugin_id, id, value):
+        response = self.http.get('/:/plugins/%s/prefs/set' % plugin_id, query={
+            id: self.to_setting_value(value, type(value))
+        })
+
+        return response.status_code == 200
+
+    def to_setting_value(self, value, value_type=None):
+        if value is None:
+            return None
+
+        if value_type is bool:
+            return str(value).lower()
+
+        return str(value)

Contents/Libraries/Shared/plex/interfaces/preferences.py

@@ -0,0 +1,40 @@
+from plex.core.idict import idict
+from plex.interfaces.core.base import Interface
+
+
+class PreferencesInterface(Interface):
+    path = ':/prefs'
+
+    def get(self, id=None):
+        response = self.http.get()
+
+        container = self.parse(response, idict({
+            'MediaContainer': ('MediaContainer', idict({
+                'Setting': 'Setting'
+            }))
+        }))
+
+        if container is None or id is None:
+            return container
+
+        for setting in container:
+            if setting.id == id:
+                return setting
+
+        return None
+
+    def set(self, id, value):
+        response = self.http.put(query={
+            id: self.to_setting_value(value, type(value))
+        })
+
+        return response.status_code == 200
+
+    def to_setting_value(self, value, value_type=None):
+        if value is None:
+            return None
+
+        if value_type is bool:
+            return str(value).lower()
+
+        return str(value)

Contents/Libraries/Shared/plex/interfaces/root.py

@@ -0,0 +1,59 @@
+from plex.core.idict import idict
+from plex.interfaces.core.base import Interface
+
+
+class RootInterface(Interface):
+    def detail(self):
+        response = self.http.get()
+
+        return self.parse(response, idict({
+            'MediaContainer': ('Detail', idict({
+                'Directory': 'Directory'
+            }))
+        }))
+
+    def version(self):
+        detail = self.detail()
+
+        if not detail:
+            return None
+
+        return detail.version
+
+    def clients(self):
+        response = self.http.get('clients')
+
+        return self.parse(response, idict({
+            'MediaContainer': ('ClientContainer', idict({
+                'Server': 'Client'
+            }))
+        }))
+
+    def players(self):
+        pass
+
+    def servers(self):
+        response = self.http.get('servers')
+
+        return self.parse(response, idict({
+            'MediaContainer': ('Container', idict({
+                'Server': 'Server'
+            }))
+        }))
+
+    def agents(self):
+        response = self.http.get('system/agents')
+
+        return self.parse(response, idict({
+            'MediaContainer': ('Container', idict({
+                'Agent': 'Agent'
+            }))
+        }))
+
+    def primary_agent(self, guid, media_type):
+        response = self.http.get('/system/agents/%s/config/%s' % (guid, media_type))
+        return self.parse(response, idict({
+            'MediaContainer': ('Container', idict({
+                'Agent': 'Agent'
+            }))
+        }))

Contents/Libraries/Shared/plex/interfaces/section.py

@@ -0,0 +1,69 @@
+from plex.core.idict import idict
+from plex.interfaces.core.base import Interface
+
+
+class SectionInterface(Interface):
+    path = 'library/sections'
+
+    def all(self, key):
+        response = self.http.get(key, 'all')
+
+        return self.parse(response, idict({
+            'MediaContainer': ('MediaContainer', idict({
+                'Directory': {
+                    'artist':   'Artist',
+                    'show':     'Show'
+                },
+                'Video': {
+                    'movie':    'Movie'
+                }
+            }))
+        }))
+
+    def recently_added(self, key):
+        response = self.http.get(key, 'recentlyAdded')
+
+        return self.parse(response, idict({
+            'MediaContainer': ('MediaContainer', idict({
+                'Directory': {
+                    'artist':   'Artist',
+                    'show':     'Show'
+                },
+                'Video': {
+                    'movie':    'Movie',
+                    'episode':  'Episode',
+                    'clip':     'Clip',
+                }
+            }))
+        }))
+
+    def first_character(self, key, character=None):
+        if character:
+            response = self.http.get(key, ['firstCharacter', character])
+
+            # somehow plex wrongly returns items of other libraries when character is #
+            return self.parse(response, idict({
+                'MediaContainer': ('MediaContainer', idict({
+                    'Directory': {
+                        'album':    'Album',
+                        'artist':   'Artist',
+
+                        'season':   'Season',
+                        'show':     'Show'
+                    },
+                    'Video': {
+                        'episode':  'Episode',
+                        'clip':     'Clip',
+                        'movie':    'Movie'
+                    },
+                    'Track': 'Track'
+                }))
+            }))
+
+        response = self.http.get(key, 'firstCharacter')
+
+        return self.parse(response, idict({
+            'MediaContainer': ('MediaContainer', idict({
+                'Directory': 'Directory'
+            }))
+        }))

Contents/Libraries/Shared/plex/interfaces/status.py

@@ -0,0 +1,21 @@
+from plex.core.idict import idict
+from plex.interfaces.core.base import Interface
+
+
+class StatusInterface(Interface):
+    path = 'status'
+
+    def sessions(self):
+        response = self.http.get('sessions')
+
+        return self.parse(response, idict({
+            'MediaContainer': ('SessionContainer', idict({
+                'Track': 'Track',
+
+                'Video': {
+                    'episode':  'Episode',
+                    'clip':     'Clip',
+                    'movie':    'Movie'
+                }
+            }))
+        }))

Contents/Libraries/Shared/plex/interfaces/timeline.py

@@ -0,0 +1,36 @@
+from plex.interfaces.core.base import Interface
+
+TIMELINE_STATES = [
+    'buffering',
+    'paused',
+    'playing',
+    'stopped'
+]
+
+
+class TimelineInterface(Interface):
+    path = ':/timeline'
+
+    def update(self, rating_key, state, time, duration, key=None, play_queue_item_id=None):
+        if not rating_key:
+            raise ValueError('Invalid "rating_key" parameter')
+
+        if time is None or duration is None:
+            raise ValueError('"time" and "duration" parameters are required')
+
+        if state not in TIMELINE_STATES:
+            raise ValueError('Unknown "state"')
+
+        response = self.http.get(query=[
+            ('ratingKey', rating_key),
+            ('state', state),
+
+            ('time', time),
+            ('duration', duration),
+
+            # Optional parameters
+            ('key', key),
+            ('playQueueItemID', play_queue_item_id)
+        ])
+
+        return response and response.status_code == 200