Merge branch 'develop-2.5'

# Conflicts: # Contents/Info.plist # README.md
add Jose to beta team
2018-03-31 16:49:28 +02:00 · 2018-03-31 16:47:35 +02:00 · 2018-03-31 16:47:11 +02:00 · 2018-03-31 16:46:59 +02:00 · 2018-03-31 03:43:32 +02:00 · 2018-03-31 03:34:01 +02:00
705 changed files with 66805 additions and 95127 deletions
@@ -0,0 +1,3 @@
+.gitattributes export-ignore
+/Wiki export-ignore
+.gitignore export-ignore
@@ -55,4 +55,5 @@ docs/_build/
 # pycharm
 .idea

-icon.psd
+icon.psd
+main-icon.psd
@@ -1,3 +1,472 @@
+
+
+2.5.3.2422
+
+- core: don't fail on embedded subtitle streams without language code set, fixes #473
+- providers: catch ResponseNotReady in list_subtitles_provider as well (partly fixes OpenSubtitles)
+- providers: don't use retry logic in case of ResponseNotReady
+- providers: addic7ed: use new search endpoint
+
+
+2.5.3.2414
+
+- core: expand user agent list
+- core: update subliminal to 4ad5d31
+- core: treat 23.976, 23.98, 24.0 fps as equal
+- core: correctly skip blacklist entries when iterating through currently known subs
+- core: fix unpacking of packs without asked-for-release-group
+- core: fix embedded subtitle language detection; add debug log
+- core: treat embedded subtitle containing "forced" in its title as forced
+- core: improve embedded subtitles detection
+- core: store extracted embedded forced subtitles with the "forced" suffix (e.g.: video.en.forced.srt)
+- core: don't bother trying to extract embedded subtitle if transcoder wasn't found
+- core: fix automatic extraction of unknown embedded subtitle streams
+- core: skip immediately searching for new subtitle after successfully extracting embedded
+- core: extract embedded ASS: don't transcode to SRT using ffmpeg (Plex Transcoder), do the transcoding later using pysubs2; fixes offset issues
+- core: extract embedded: let ffmpeg auto convert mov_text/tx3g to srt
+- core: fix transcoder detection; add fallback #460
+- core: remove LD_LIBRARY_PATH from environment before calling notification executable
+- core: auto extract embedded subtitles in a separate thread
+- core: reduce encoding change log spam
+- core: only allow one automatic extraction at a time; add optional advanced settings "auto_extract_multithread"
+- core: add minimum score a subtitle has to have when considered by the find better subtitles task, when the current subtitle is an extracted embedded one; add advanced_settings entries
+- core/config: automatic extraction: add config setting to indicate whether there should be an immediate search for available subtitles after extraction or not (default: off)
+- core/menu/submod: add reverse_rtl modification for Hebrew; fixes #409
+- core: scoring: assume title match on tvdb_id match
+- tasks: search all recently added missing: fix attribute access on missing stored subtitle info
+- providers: add hosszupuska (hungarian, thanks morpheus133 for the basic implementation)
+- providers: add argenteam (spanish, thanks mmiraglia for the basic implementation)
+- providers: addic7ed: use random user agent by default (enforce for existing configs)
+- providers: enable subscene by default
+- providers: opensubtitles: add fallback for dict based query response in contrast to list/array based
+- advanced settings: make text-based-subtitle-formats configurable
+- menu: submod: inverse-reverse subtitle timing time-choices for better accessibility
+- submod: reduce log spam in case of debug logs enabled
+- submod: style tags could result in no output at all
+- submod: fix empty content if only non-line-mods were used, no line-mods; fixes #449
+- submod: HI: correctly handle style tags when checking for brackets
+- submod: HI: don't remove anything that's surrounded by quotes
+- submod: HI: double or triple dash is em dash
+- submod: HI: HI_before_colon_noncaps, don't assume single quotes are sentence enders
+- submod: common: don't uppercase after abbreviations
+- submod: common: don't break phone numbers (more than one spaced number pair found)
+- submod: common: also count lines only consisting of dots as removable
+- submod: common: replace more than 3 consecutive dots with 3 dots
+- submod: OCR: "H i." = "Hi."
+
+
+2.5.0.2287
+
+- core: reduce main icon size
+- core: fix usage on NVIDIA SHIELD (hopefully, please report back), #441
+- core: add scandir fallback to listdir in case of badly configured locale in environment, #441, #440
+- core: get subtitles from archive: don't assume an episode match
+- core: get subtitles from archive: don't assume any attributes in guess
+- core: improve release group detection for drone/filebot/file_info refiners
+- core: fix language detection for embedded subtitle streams
+- core: support extraction of embedded mov_text subtitles in mp4 video files
+- refiners: drone: add http:// to url if not given
+- providers: opensubtitles: retry/reinitialize request when encountering ResponseNotReady
+- config: clarify subscene being only enabled for TV series by default
+- menu: when encountering permission errors when scanning media files, warn in the menu about them
+- submod: common: don't break -- addic7ed --
+- submod: common: remove lines that consist only of dash, underscore
+- submod: OCR: fix Ls = Is
+- submod: OCR: fix bad HI colons (ANNOUNCER; instead of ANNOUNCER:)
+- submod: common: fix lines consisting only of bad music symbols (*#¶ = ♪)
+- submod: HI: remove music-symbol-only-lines
+- submod: HI: be less aggressive about lines ending with a colon; please re-apply all your mods via advanced menu
+- submod: OCR: fix it'sjust, isn'tjust, Iam, Ican
+
+
+2.5.0.2247
+- fix ignoring by-hash-matched episodes
+
+
+2.5.0.2241
+
+- fix issue when removing crap from filenames to not accidentally remove release group #436
+- fix initialization of soft ignore list after upgrade fron 2.0
+
+
+2.5.0.2221
+
+- refiners: add support for retrieving original filename from
+ 	- drone derivates: sonarr, radarr
+ 	- filebot
+ 	- symlinks
+    - file_info meta file lists (see wiki)
+
+- providers: add subscene (disabled by default to not flood subscene on release)
+    - normal search
+    - season pack search if season has concluded
+
+- core: add provider subtitle-archive/pack cache for retrieving single subtitles from previously downloaded (season-) packs (subscene)
+- core/agent: massive performance improvements over 2.0
+- core/agent/background-tasks: reduce memory usage to a fraction of 2.0
+- core/providers: add dynamic provider throttling when certain events occur (ServiceUnavailable, too many downloads, ...), to lighten the provider-load
+- core/agent/config: automatically extract embedded subtitles (and use them if no current subtitle)
+- core: fix internal subtitle info storage issues
+- core: always store internal subtitle information even if no subtitle was downloaded (fixes SearchAllRecentlyAddedMissing)
+- core: fix internal subtitle info storage on windows (gzip handling is broken there)
+- core: don't fail on missing logfile paths
+- core: fix default encoding order for non-script-serbian
+- core: improve logging
+- core: add AsRequested to cleanup garbage names
+- core: treat SDTV and HDTV the same when searching for subtitles
+- core: parse_video: trust PMS season and episode numbers
+- core: parse_video: add series year information from PMS if none found
+- core: upgrade dependencies
+- core: update subliminal to 62cdb3c
+- core: add new file based cache mechanism, rendering DBM/memory backends obsolete
+- core: treat 23.980 fps as 23.976 and vice-versa
+- core: add HTTP proxy support for querying the providers (supports credentials)
+- core: only compute file hashes for enabled providers
+- core: massive speedup; refine only when needed, exit early otherwise
+- core: store last modified timestamp in subtitle info storage
+- core: only write to subtitle info storage if we haven't had one or any subtitle was downloaded
+- core: only clean up the sub-folder if a subtitle-sub-folder has been selected, and not the parent one also
+- core: support for CP437 encoded filenames in ZIP-Archives
+- core: use scandir library instead of os.listdir if possible, reducing performance-impact
+- core: archives: support multi-episode subtitles (partly)
+- core: subtitle cleanup: add support for hi, cc, sdh secondary filename tags; don't autoclean .txt
+- core: increase request timeout by three times in case a proxy is being used
+- core: fix language=Unknown in Plex when "Restrict to one language"-setting is set
+- core: refining: re-add old detected title as alternative title after re-refining with plex metadata's title; fixes #428
+- core: implement advanced_settings.json (see advanced_settings.json.template for reference, copy to "Plug-in Support/Data/com.plexapp.agents.subzero" to use it)
+- core/tasks: fix search all recently added missing (the total number of items will change in the menu while running), reduces memory usage
+- core/menu: add support for extracting embedded subtitles using the builtin plex transcoder
+- core/menu: skip wrong season or episode in returned subtitle results
+- core/config: fix language handling if treat undefined as first language is set
+- providers: remove shooter.cn
+- providers: add support for zip/rar archives containing more than one subtitle file
+- submod: common: remove redundant interpunction ("Hello !!!" -> "Hello!")
+- submod: skip provider hashing when applying mods
+- submod: correctly drop empty line (fixing broken display)
+- submod: OCR: fix F'xxxxx -> Fxxxxx
+- submod: HI: improve bracket matching
+- submod: OCR: fix l/L instead of I more aggressively
+- submod: common: fix uppercase I's in lowercase words more aggressively
+- submod: HI: improve HI_before_colon
+- submod: common: be more aggressive when fixing numbers; correctly space out spaced ellipses; don't break spaced ellipses; handle multiple spaces in numbers
+- menu: add support for extracting embedded subtitles for a whole season
+- menu: add reapply mods to current subtitle
+- menu: pad titles for more submenus, resulting in detail view in PlexWeb
+- menu: add subtitle selection submenu (if multiple subtitles are inside the subtitle info storage; e.g. previously downloaded ones or extracted embedded)
+- menu: advanced: add skip findbettersubtitles menu item, which sets the last_run to now (for debugging purposes)
+- menu: ignore: add more natural title for seasons and episodes (kills your old ignore lists!)
+- config: skip provider hashing on low impact mode
+- config: add limit by air date setting to consider for FindBetterSubtitles task (default: 1 year)
+- advanced settings: define enabled-for media types per provider
+- advanced settings: define enabled-for languages per provider
+- advanced settings: add deep-clean option (clean up the subtitle-sub-folder and the parent one)
+
+
+
+2.0.33.1871
+- core: normalize line endings in subtitles to LF (\n)
+- core: add subtitle storage lock to avoid race condition
+- core: be more verbose about subtitle storage addition
+- core: fix MPL2 newline parsing, which resulted in broken subtitles
+- core: encoding change: reduce log spam
+- submod: common: fix CM_starting_spacedots
+- opensubtitles: fix request/response handling
+
+
+
+2.0.33.1849
+- opensubtitles: add VIP server handling + preference; VIP benefits: 10€/year, ad-free subs, 1000 subs/day, no-cache VIP server, help SZ and subscribe via http://v.ht/osvip
+- opensubtitles: try to reuse previous token instead of logging in every time
+- core: add throttling between searches (10 seconds)
+- core: fix IETF handling for good
+- core: fix no subtitles being searched in certain situations (when an external subtitle without special tag exists)
+- core: add subtitle blacklist
+- core: fixes
+- core: fix detection of certain PMS media stream language tags ("FR" for example)
+- core: missing subtitles: correctly skip unwanted subtitle extensions
+- core: missing subtitles: honor "treat undefined as first language" option correctly
+- api: add blacklisting endpoints for quickly searching for new subtitls via bookmarklet
+- submod: colors: apply color mods at the end of processing modifications; fix color mods
+- submod: new remove_tags modification to remove all styling tags from subtitles
+- submod: HI: be more aggressive at handling brackets
+- submod: OCR: update en and hrv
+- submod: common: remove "torrent downloaded from ..." lines
+- submod: OCR: fix WholeWord handling, improving modification
+- submod: apply OCR fixes before HI
+- submod: OCR: fix broken HI tag colons (ANNOUNCER'. instead of ANNOUNCER:)
+- menu: advanced: speed up batch modifications
+- menu: add subtitle blacklist
+- menu: recently played: show only TV episodes and movies (music tracks were listed here as well)
+
+
+2.0.29.1767
+- core: fix internal subtitle storage issues
+- core: handle "embedded-forced" tag (futureproofing)
+- core: remove more garbage tags from release groups (nzbgeek, chamele0n, buymore, xpost, postbot)
+- submod: OCR fix: fix music icon = paragraph
+
+
+2.0.29.1756
+- core: don't fail on uppercase file extensions
+- core: don't re-download a subtitle if we already downloaded one, it still physically exists and external subtitles are configured to be ignored
+- core: fix VTT subtitle duplication
+- core: if forced subtitles not explicitly wanted, ignore existing forced subtitles when searching
+- core: add full IETF language support for `Treat languages with country attribute as ISO 639-1 (e.g. don't download pt-BR if pt subtitle exists)`-setting for embedded subtitles
+- menu: remove buggy dynamic permission-based channel icon introduced in 1715
+- menu: improve `Items with missing subtitles` menu usage and item display
+- menu: `Advanced -> Get my logs` handle custom domains without port
+- menu: correctly show country/script part of languages with such attributes (e.g. pt-BR)
+- config: rename `Scan:` settings; make them better understandable and translatable
+- config: rephrase IETF options as "languages with country attribute" (e.g. pt-BR)
+- config: separate IETF options into how to display languages with country attribute and how they should be handled when searching/scanning (e.g. pt-BR)
+- config: `Scheduler: Item age to be considered recent` now can go up to 12 weeks
+- config: `Scheduler: Periodically search for recent items with missing subtitles` added `every 2 hours`
+- submod: swe: add Ĺ to Å
+
+
+2.0.26.1715
+- core: submod: OCR fixes: swe: replace ĺ with å inside words
+- core: fix handling of non-existant PMS audio_codec info
+- core: filename matching ignored the strictness setting in certain global directory configurations (thanks @raduc)
+- core: don't fail on migration errors
+- provider titlovi: handle multiple subtitles per archive
+- provider addic7ed: reset default boost to 19 (was 21)
+- menu: add warning icon on missing permissions
+- menu: manual subtitle list sometimes listed duplicates (thanks @andreashoyer)
+- menu: don't request PMS metadata in item details menu twice
+- menu: don't fail badly on non existant PMS metadata in item details menu
+
+
+2.0.26.1695
+## ATTENTION: THIS RELEASE RESETS YOUR CONFIGURED LANGUAGES TO DEFAULT!
+- core: fix bug that caused SZ not to work for Windows users with special characters in their username
+- core: fix issues when logging failed manual download actions
+- core: update guessit to 2.1.4
+- core: fix issue causing the background task scheduler to stop after changing preferences
+- core: fix polish encoding (try windows-1250 first, then iso 8859-2)
+- core: remove subscenter provider as it now uses captchas
+- core: add titlovi as default provider (thanks viking!)
+- core: increase default PMS API request timeout to 15 (old: 10, max: 45); add preference for that
+- core: re-add separate legacy FindMissingSubtitles task and run it on the first run to prime SZ's internal subtitle storage
+- core: add "low impact mode" for people with remote filesystems (currently enabled for List LANGUAGE subtitles in detail menu); alleviates certain plexweb timeout issues
+- menu: change naming of find missing subtitles menu item
+- legendastv: fix multi value guessit issues
+- submod: OCR: update eng and hrv OCR replace dictionaries; fix ". L am huge"
+
+
+2.0.25.1635
+- core: update memory handling, possibly reduce memory problems of 2.0
+- core: support for MPL2 subtitle format
+- core: update task handling
+- core: re-enable NVIDIA SHIELD support by fixing rarfile behaviour
+- core: add SZ_UNRAR_TOOL environment variable for custom unrar location
+- core: disable SZ when no providers are enabled
+- core: only start activity monitor if channel or agent are enabled
+- core: improve custom provider integration
+- core: update eastern european encoding detection (especially Romanian)
+- tasks: reduce provider stress by introducing wait times between searches/downloads
+- windows: correctly ship UnRAR.exe
+- windows: skip DBM checks
+- addic7ed: fix Nip/Tuck
+- subscenter: use new domain
+
+
+2.0.24.1581
+- legendastv: ship unrar.exe for Windows users (fixes unrar issues)
+- addic7ed: fix TooManyRequests error
+- submod: OCR fixes NL: add custom dictionary data for malformed characters
+- submod: OCR fixes: update hrv/NL dictionaries
+- submod: common: remove spaces before punctuation
+- podnapisi: now returns more subtitles again
+ATTENTION: Sub-Zero is still broken on PMS for SHIELD. Help needed!
+
+
+2.0.24.1565
+- core: fix searchallrecentlymissing task erroring if item not found
+- core: fix non-plex-items appearing in and crashing the recently played list
+- core: add hybrid-plus activity setting (current media file and next episode)
+- podnapisi: fix by using correct guessit parameters
+
+
+2.0.24.1558
+- core: fix handling of broken RAR files from legendas
+
+
+2.0.24.1555
+- core: fix rare microdvd issue from OpenSubtitles by generally providing FPS info when encountering a microdvd subtitle
+
+
+2.0.24.1549
+Changes from 1.4
+- wiki: new wiki! (thanks @dane22!)
+- core: update subliminal to version 2
+- core: update all dependencies
+- core: add new providers: legendastv (pt-BR), napiprojekt (pl), shooter (cn), subscenter (heb)
+- core: rewritten all subliminal patches for version 2
+- core: use SSL again for opensubtitles
+- core: improved matching due to subliminal 2 (and SZ custom) tvdb/omdb refiners
+- core: improved matching by relying on existing metadata provided by the PMS
+- core: improved performance due to multithreaded provider-querying
+- core: improved performance due to less physical media file access (no more MKV metadata scanning)
+- core: VTT subtitle format output supported (for Chromecast)
+- core: rewrote and streamlined internal subtitle data storage format
+- core: support Cyrillic and Latin variants of Serbian language
+- core: simplified (custom) provider registration; add own provider registry
+- core: rewrote recently added missing task
+- core: automatically fix badly (re-) encoded unicode entities in subtitles
+- core: always store subtitles in proper UTF-8 encoding
+- core: add periodic internal subtitle data storage cleanup task
+- core: on non-windows systems, utilize a file-based cache database for provider media lists and subliminal refiner results
+- core: add manual and automatic subtitle modification framework (fix common OCR issues, remove hearing impaired etc.)
+- core: relieve some stress on providers by providing better fine-grained retry handling
+- menu: add icons for menu items; update main channel icon
+- menu: add subtitle modifications (subtitle content fixes, offset-based shifting, framerate conversion)
+- menu: add recently played menu
+- menu: add "Get my logs" function to the advanced menu, which zips up all necessary logs suitable for posting in the forums
+- menu: add generic "back to season" and "back to series" entries to item detail views to make navigation easier
+- config: all scores changed (defaults updated)
+- config: remove "Force UTF-8 when storing subtitles" (it's now always implied)
+- improve almost everything Sub-Zero did in 1.4 :)
+
+
+2.0.23.1464 RC10.1
+ - core: huge bugfix; please check `Library/Application Support/Plex Media\ Server/Plug-in Support/Data/com.plexapp.agents.subzero/DataItems`
+         for any `subs_XXXXX.json.gz` file bigger than 500kb and delete them
+
+
+2.0.23.1456 RC10
+- core: findBetterSubtitles: increase series cutoff by 2 (resolution match)
+- core: add VTT format
+- core: fix crashes regarding DBM/cache management
+- core: update rarfile.py
+- core: add missing encodings
+- core: full support for Serbian subtitles (Cyrillic and Latin)
+- podnapisi: fix pt-BR, srp-cyrl and srp-latn
+- core: implement own provider registry and ditch the subliminal one
+- core: use ftfy library to fix re-encoding errors inside subtitles introduced by the subtitle author
+- core: always store and save subtitles normalized to UTF-8
+- core: replace spaced dashes in movie/series names before re-refining with plex metadata info
+- submod: remove_HI: handle multiline brackets correctly
+
+
+2.0.20.1364 RC9
+- core: performance improvements
+- core: if info couldn't be guessed from the filename, fill missing info from PMS #270
+- submod: OCR: add more to the eng dictionary
+- submod: HI: fixed some issues with font style tags
+- core: don't ignore subtitles from providers that don't have hearing impaired info, when hearing impaired mode is set to "force non-HI"
+- legendastv/menu: fix manual subtitle selection issues in menu
+- core: improve specials matching on OpenSubtitles
+- core: update guessit
+
+
+2.0.19.1337 RC8
+- napiprojekt: fixed: couldn't convert microdvd to SRT in certain occasions
+- core: when normalize to UTF-8 is enabled, also store the subtitle in UTF-8 encoding in the internal storage
+- core: add more encodings for western/eastern/northern europe
+- submod: OCR: update dictionaries from SubtitleEdit
+- submod: common: be smarter about uppercase i's in words that should have lowercase L's
+- submod: fix unopened/unclosed font style tags after modification
+- core: re-enable OMDB support
+- core: update guessit for better matching
+- core: fix SearchAllRecentlyMissing (was broken since RC3)
+
+
+2.0.19.1299 RC7
+- submod: offset mods now get merged internally when applied multiple times (to avoid errors and increase performance)
+- submod: improve performance
+- submod: core mods (OCR, common, remove_HI) now are always applied in a fixed order internally, regardless of the order they were added in
+- submod: CM_spaces_in_numbers: don't break up ellipses (30... 29... 28...)
+- submod: CM_spaces_in_numbers: don't fix countdown numbers (30, 29, 28)
+- submod: remove_HI: make bracket removal more aggressive
+- submod: remove_HI: be less aggressive when removing text-before-colon
+- submod: remove_HI: remove all-uppercase-before-sentence (THIS IS ALL UPPERCASE And here starts a sentence -> And here starts a sentence)
+- submod: fix all character ranges to include non-ASCII characters
+- add new README for 2.0
+
+
+2.0.19.1267 RC6
+- core: add new SZ subtitle storage format
+  - smaller data files and less cumbersome
+  - it will auto migrate when old data is accessed - to speed this up, use "Trigger subtitle storage migration (expensive)" in advanced menu)
+- core: performance optimizations
+- addic7ed: when release group matches, assume the format matches, too (leftover change from RC5)
+- submod: fix patterns for beginlines/endlines
+- submod: add our own dictionaries to OCR fixes (english)
+- submod: hearing impaired: also remove full-caps with punctuation inside
+- submod: correctly handle partiallines
+- submod: in numbers with spaces (incorrect), also allow for some punctuation (,.:')
+
+
+2.0.18.1245 RC5
+- core: add more debug info
+- core: fix subtitle modifications (was broken in RC4, created non-usable subtitles)
+- submod: add ANSI colors
+- menu/submod: add color mod menu
+- submod: exclusive mods now are mutually exclusive and get cleaned on duplicate
+- menu/core: naming
+
+For everyone who runs RC4: your subtitles are broken. Go to the advanced menu and trigger `Re-Apply mods of all stored subtitles` to fix them.
+
+
+2.0.17.1234 RC4
+- core: backport provider-download-retry implementation
+- core: implement custom user agent (for OpenSubtitles)
+- core/menu: correct handling of media with multiple files
+- core: fix SearchAllRecentlyMissing; also wait 5 seconds between searches
+- core: SearchAllRecentlyMissing: honor physical ignores
+- submod: pattern fixes
+- submod: better unicode handling
+- submod: add color mod (only automatic by now)
+
+
+2.0.15.1216 RC3
+- core: fixes
+- scheduler: revert some of the aggressive changes in RC2
+- submod: be smarter about WholeLine matches
+
+
+2.0.15.1209 RC2
+- core: fixes
+- core: submod-common: fix multiple dots at start of line
+- core/menu: add subtitle modification debug setting
+- core/menu: when manually listing available subtitles in menu, display those with wrong FPS also (opensubtitles), because you can fix them later
+- core/menu: advanced-menu: add apply-all-default-mods menu item; add re-apply all mods menu item
+- core: always look for currently (not-) existing subtitles when called; hopefully fixes #276
+- scheduler/menu: be faster; also launch scheduled tasks in threads, not just manually launched ones
+- core: don't delete subtitles with .custom or .embedded in their filenames when running auto cleanup, if the correct media file exists
+- menu: add back-to-previous menu items
+
+
+2.0.12.1180 RC1
+- core: update subliminal to version 2
+- core: update all dependencies
+- core: add new providers: legendastv (pt-BR), napiprojekt (pl), shooter (cn), subscenter (heb)
+- core: rewritten all subliminal patches for version 2
+- menu: add icons for menu items; update main channel icon
+- core: use SSL again for opensubtitles
+- core: improved matching due to subliminal 2 (and SZ custom) tvdb/omdb refiners
+- menu: add "Get my logs" function to the advanced menu, which zips up all necessary logs suitable for posting in the forums
+- core: on non-windows systems, utilize a file-based cache database for provider media lists and subliminal refiner results
+- core: add manual and automatic subtitle modification framework (fix common OCR issues, remove hearing impaired etc.)
+- menu: add subtitle modifications (subtitle content fixes, offset-based shifting, framerate conversion)
+- menu: add recently played menu
+- improve almost everything Sub-Zero did in 1.4 :)
+
+
+1.4.27.973
+- core: ignore "obfuscated" and "scrambled" tags in filenames when searching for subtitles
+- core: exotic embedded subtitles are now also considered when searching (and when the option is enabled); fixes #264
+
+
+1.4.27.967
+- core: remember the last 10 played items; only consider on_playback for "playing" state within the first 60 seconds of an item
+
+
+1.4.27.965
+- core: on_playback activity bugfixes
+
+
 1.4.27.957
 - core: correctly fall back to the next best subtitle if the current one couldn't be downloaded; hopefully fixes #231
 - core: add "Scan: which external subtitles should be picked up?"-setting
@@ -1,11 +1,11 @@
 # coding=utf-8
 import sys
+import datetime

-from subliminal_patch import compute_score
-from subzero.sandbox import restore_builtins
+from subzero.sandbox import fix_environment_stuff

 module = sys.modules['__main__']
-restore_builtins(module, {})
+fix_environment_stuff(module, {})

 globals = getattr(module, "__builtins__")["globals"]
 for key, value in getattr(module, "__builtins__").iteritems():
@@ -16,7 +16,6 @@ import logger

 sys.modules["logger"] = logger

-import subliminal
 import support

 import interface
@@ -24,9 +23,9 @@ sys.modules["interface"] = interface

 from subzero.constants import OS_PLEX_USERAGENT, PERSONAL_MEDIA_IDENTIFIER
 from interface.menu import *
-from support.plex_media import media_to_videos, get_media_item_ids, scan_videos
-from support.subtitlehelpers import get_subtitles_from_metadata
-from support.storage import whack_missing_parts, save_subtitles
+from support.plex_media import media_to_videos, get_media_item_ids
+from support.scanning import scan_videos
+from support.storage import save_subtitles, store_subtitle_info, get_subtitle_storage
 from support.items import is_ignored
 from support.config import config
 from support.lib import get_intent
@@ -34,14 +33,14 @@ from support.helpers import track_usage, get_title_for_video_metadata, get_ident
 from support.history import get_history
 from support.data import dispatch_migrate
 from support.activities import activity
+from support.download import download_best_subtitles


 def Start():
    HTTP.CacheTime = 0
    HTTP.Headers['User-agent'] = OS_PLEX_USERAGENT

-    # configured cache to be in memory as per https://github.com/Diaoul/subliminal/issues/303
-    subliminal.region.configure('dogpile.cache.memory')
+    config.init_cache()

    # clear expired intents
    intent = get_intent()
@@ -50,9 +49,12 @@ def Start():
    # clear expired menu history items
    now = datetime.datetime.now()
    if "menu_history" in Dict:
-        for key, timeout in Dict["menu_history"].items():
+        for key, timeout in Dict["menu_history"].copy().items():
            if now > timeout:
-                del Dict["menu_history"][key]
+                try:
+                    del Dict["menu_history"][key]
+                except:
+                    pass

    # run migrations
    if "subs" in Dict or "history" in Dict:
@@ -74,7 +76,8 @@ def Start():
    scheduler.run()

    # bind activities
-    Thread.Create(activity.start)
+    if config.enable_channel:
+        Thread.Create(activity.start)

    if "anon_id" not in Dict:
        Dict["anon_id"] = get_identifier()
@@ -88,45 +91,6 @@ def Start():
        track_usage("General", "plugin", "start", config.version)


-def download_best_subtitles(video_part_map, min_score=0):
-    hearing_impaired = Prefs['subtitles.search.hearingImpaired']
-    languages = config.lang_list
-    if not languages:
-        return
-
-    missing_languages = False
-    for video, part in video_part_map.iteritems():
-        if not Prefs['subtitles.save.filesystem']:
-            # scan for existing metadata subtitles
-            meta_subs = get_subtitles_from_metadata(part)
-            for language, subList in meta_subs.iteritems():
-                if subList:
-                    video.subtitle_languages.add(language)
-                    Log.Debug("Found metadata subtitle %s for %s", language, video)
-
-        missing_subs = (languages - video.subtitle_languages)
-
-        # all languages are found if we either really have subs for all languages or we only want to have exactly one language
-        # and we've only found one (the case for a selected language, Prefs['subtitles.only_one'] (one found sub matches any language))
-        found_one_which_is_enough = len(video.subtitle_languages) >= 1 and Prefs['subtitles.only_one']
-        if not missing_subs or found_one_which_is_enough:
-            if found_one_which_is_enough:
-                Log.Debug('Only one language was requested, and we\'ve got a subtitle for %s', video)
-            else:
-                Log.Debug('All languages %r exist for %s', languages, video)
-            continue
-        missing_languages = True
-        break
-
-    if missing_languages:
-        Log.Debug("Download best subtitles using settings: min_score: %s, hearing_impaired: %s" % (min_score, hearing_impaired))
-
-        return subliminal.download_best_subtitles(video_part_map.keys(), languages, min_score, hearing_impaired, providers=config.providers,
-                                                  provider_configs=config.provider_settings, pool_class=config.provider_pool,
-                                                  compute_score=compute_score)
-    Log.Debug("All languages for all requested videos exist. Doing nothing.")
-
-
 def update_local_media(metadata, media, media_type="movies"):
    # Look for subtitles
    if media_type == "movies":
@@ -150,12 +114,52 @@ def update_local_media(metadata, media, media_type="movies"):
            pass


+def agent_extract_embedded(video_part_map):
+    try:
+        subtitle_storage = get_subtitle_storage()
+
+        to_extract = []
+        item_count = 0
+
+        for scanned_video, part_info in video_part_map.iteritems():
+            plexapi_item = scanned_video.plexapi_metadata["item"]
+            stored_subs = subtitle_storage.load_or_new(plexapi_item)
+
+            for plexapi_part in get_all_parts(plexapi_item):
+                item_count = item_count + 1
+                for requested_language in config.lang_list:
+                    embedded_subs = stored_subs.get_by_provider(plexapi_part.id, requested_language, "embedded")
+                    current = stored_subs.get_any(plexapi_part.id, requested_language)
+                    if not embedded_subs:
+                        stream_data = get_embedded_subtitle_streams(plexapi_part, requested_language=requested_language,
+                                                                    get_forced=config.forced_only)
+
+                        if stream_data:
+                            stream = stream_data[0]["stream"]
+
+                            to_extract.append(({scanned_video: part_info}, plexapi_part, str(stream.index),
+                                               str(requested_language), not current))
+
+                            if not cast_bool(Prefs["subtitles.search_after_autoextract"]):
+                                scanned_video.subtitle_languages.update({requested_language})
+                    else:
+                        Log.Debug("Skipping embedded subtitle extraction for %s, already got %r from %s",
+                                  plexapi_item.rating_key, requested_language, embedded_subs[0].id)
+        if to_extract:
+            Log.Info("Triggering extraction of %d embedded subtitles of %d items", len(to_extract), item_count)
+            Thread.Create(multi_extract_embedded, stream_list=to_extract, refresh=True, with_mods=True,
+                          single_thread=not config.advanced.auto_extract_multithread)
+    except:
+        Log.Error("Something went wrong when auto-extracting subtitles, continuing: %s", traceback.format_exc())
+
+
 class SubZeroAgent(object):
    agent_type = None
    agent_type_verbose = None
    languages = [Locale.Language.English]
    primary_provider = False
    score_prefs_key = None
+    debounce = 10

    def __init__(self, *args, **kwargs):
        super(SubZeroAgent, self).__init__(*args, **kwargs)
@@ -166,6 +170,9 @@ class SubZeroAgent(object):
        Log.Debug("Sub-Zero %s, %s search" % (config.version, self.agent_type))
        results.Append(MetadataSearchResult(id='null', score=100))

+    def store_blank_subtitle_metadata(self, video_part_map):
+        store_subtitle_info(video_part_map, dict((k, []) for k in video_part_map.keys()), None, mode="a")
+
    def update(self, metadata, media, lang):
        if not config.enable_agent:
            Log.Debug("Skipping Sub-Zero agent(s)")
@@ -183,6 +190,9 @@ class SubZeroAgent(object):
            config.init_subliminal_patches()
            videos = media_to_videos(media, kind=self.agent_type)

+            # find local media
+            update_local_media(metadata, media, media_type=self.agent_type)
+
            # media ignored?
            use_any_parts = False
            for video in videos:
@@ -203,29 +213,82 @@ class SubZeroAgent(object):

            set_refresh_menu_state(media, media_type=self.agent_type)

-            # find local media
-            update_local_media(metadata, media, media_type=self.agent_type)
-
            # scanned_video_part_map = {subliminal.Video: plex_part, ...}
-            scanned_video_part_map = scan_videos(videos, kind=self.agent_type)
+            providers = config.get_providers(media_type=self.agent_type)
+            try:
+                scanned_video_part_map = scan_videos(videos, providers=providers)
+            except IOError, e:
+                Log.Exception("Permission error, please check your folder/file permissions. Exiting.")
+                if cast_bool(Prefs["check_permissions"]):
+                    config.permissions_ok = False
+                    config.missing_permissions = e.message
+                return
+
+            # auto extract embedded
+            if config.embedded_auto_extract:
+                if config.plex_transcoder:
+                    agent_extract_embedded(scanned_video_part_map)
+                else:
+                    Log.Warning("Plex Transcoder not found, can't auto extract")
+
+            # clear missing subtitles menu data
+            if not scheduler.is_task_running("MissingSubtitles"):
+                scheduler.clear_task_data("MissingSubtitles")
+
+            downloaded_subtitles = None
+
+            # debounce for self.debounce seconds
+            now = datetime.datetime.now()
+            if "last_call" in Dict:
+                last_call = Dict["last_call"]
+                if last_call + datetime.timedelta(seconds=self.debounce) > now:
+                    wait = self.debounce - (now - last_call).seconds
+                    if wait >= 1:
+                        Log.Debug("Waiting %s seconds until continuing", wait)
+                        Thread.Sleep(wait)

            # downloaded_subtitles = {subliminal.Video: [subtitle, subtitle, ...]}
-            downloaded_subtitles = download_best_subtitles(scanned_video_part_map, min_score=use_score)
+            try:
+                downloaded_subtitles = download_best_subtitles(scanned_video_part_map, min_score=use_score,
+                                                               throttle_time=self.debounce, providers=providers)
+            except:
+                Log.Exception("Something went wrong when downloading subtitles")
+
+            if downloaded_subtitles is not None:
+                Dict["last_call"] = datetime.datetime.now()
+
            item_ids = get_media_item_ids(media, kind=self.agent_type)

-            whack_missing_parts(scanned_video_part_map)
-
+            downloaded_any = False
            if downloaded_subtitles:
-                save_subtitles(scanned_video_part_map, downloaded_subtitles)
+                downloaded_any = any(downloaded_subtitles.values())
+
+            if downloaded_any:
+                save_successful = False
+                try:
+                    save_successful = save_subtitles(scanned_video_part_map, downloaded_subtitles,
+                                                     mods=config.default_mods)
+                except:
+                    Log.Exception("Something went wrong when saving subtitles")
+
                track_usage("Subtitle", "refreshed", "download", 1)

-                for video, video_subtitles in downloaded_subtitles.items():
-                    # store item(s) in history
-                    for subtitle in video_subtitles:
-                        item_title = get_title_for_video_metadata(video.plexapi_metadata, add_section_title=False)
-                        history = get_history()
-                        history.add(item_title, video.id, section_title=video.plexapi_metadata["section"],
-                                    subtitle=subtitle)
+                # store SZ meta info even if download wasn't successful
+                if not save_successful:
+                    self.store_blank_subtitle_metadata(scanned_video_part_map)
+
+                else:
+                    for video, video_subtitles in downloaded_subtitles.items():
+                        # store item(s) in history
+                        for subtitle in video_subtitles:
+                            item_title = get_title_for_video_metadata(video.plexapi_metadata, add_section_title=False)
+                            history = get_history()
+                            history.add(item_title, video.id, section_title=video.plexapi_metadata["section"],
+                                        subtitle=subtitle)
+                            history.destroy()
+            else:
+                # store SZ meta info even if we've downloaded none
+                self.store_blank_subtitle_metadata(scanned_video_part_map)

            update_local_media(metadata, media, media_type=self.agent_type)

@@ -235,13 +298,17 @@ class SubZeroAgent(object):

            # notify any running tasks about our finished update
            for item_id in item_ids:
-                scheduler.signal("updated_metadata", item_id)
+                #scheduler.signal("updated_metadata", item_id)

                # resolve existing intent for that id
                intent.resolve("force", item_id)

            Dict.Save()

+            # fsync cache
+            if config.new_style_cache:
+                config.sync_cache()
+

 class SubZeroSubtitlesAgentMovies(SubZeroAgent, Agent.Movies):
    contributes_to = ['com.plexapp.agents.imdb', 'com.plexapp.agents.xbmcnfo', 'com.plexapp.agents.themoviedb', 'com.plexapp.agents.hama']
@@ -2,6 +2,22 @@ import sys

 import menu
 sys.modules["interface.menu"] = menu
+sys.modules["menu"] = menu

 import menu_helpers
-sys.modules["interface.menu_helpers"] = menu_helpers
+sys.modules["interface.menu_helpers"] = menu_helpers
+
+import advanced
+sys.modules["interface.advanced"] = advanced
+
+import main
+sys.modules["interface.main"] = main
+
+import refresh_item
+sys.modules["interface.refresh_item"] = refresh_item
+
+import item_details
+sys.modules["interface.item_details"] = item_details
+
+import sub_mod
+sys.modules["interface.modification"] = sub_mod
@@ -0,0 +1,391 @@
+# coding=utf-8
+import datetime
+import StringIO
+import glob
+import os
+import traceback
+import urlparse
+
+from zipfile import ZipFile, ZIP_DEFLATED
+
+from subzero.language import Language
+
+from subzero.lib.io import FileIO
+from subzero.constants import PREFIX, PLUGIN_IDENTIFIER
+from menu_helpers import SubFolderObjectContainer, debounce, set_refresh_menu_state, ZipObject, ObjectContainer, route
+from main import fatality
+from support.helpers import timestamp, pad_title
+from support.config import config
+from support.lib import Plex
+from support.storage import reset_storage, log_storage, get_subtitle_storage
+from support.scheduler import scheduler
+from support.items import set_mods_for_part, get_item_kind_from_rating_key
+
+
+@route(PREFIX + '/advanced')
+def AdvancedMenu(randomize=None, header=None, message=None):
+    oc = SubFolderObjectContainer(header=header or "Internal stuff, pay attention!", message=message, no_cache=True,
+                                  no_history=True,
+                                  replace_parent=False, title2="Advanced")
+
+    if config.lock_advanced_menu and not config.pin_correct:
+        oc.add(DirectoryObject(
+            key=Callback(PinMenu, randomize=timestamp(), success_go_to="advanced"),
+            title=pad_title("Enter PIN"),
+            summary="The owner has restricted the access to this menu. Please enter the correct pin",
+        ))
+        return oc
+
+    oc.add(DirectoryObject(
+        key=Callback(TriggerRestart, randomize=timestamp()),
+        title=pad_title("Restart the plugin"),
+    ))
+    oc.add(DirectoryObject(
+        key=Callback(GetLogsLink),
+        title="Get my logs (copy the appearing link and open it in your browser, please)",
+        summary="Copy the appearing link and open it in your browser, please",
+    ))
+    oc.add(DirectoryObject(
+        key=Callback(TriggerBetterSubtitles, randomize=timestamp()),
+        title=pad_title("Trigger find better subtitles"),
+    ))
+    oc.add(DirectoryObject(
+        key=Callback(SkipFindBetterSubtitles, randomize=timestamp()),
+        title=pad_title("Skip next find better subtitles (sets last run to now)"),
+    ))
+    oc.add(DirectoryObject(
+        key=Callback(TriggerStorageMaintenance, randomize=timestamp()),
+        title=pad_title("Trigger subtitle storage maintenance"),
+    ))
+    oc.add(DirectoryObject(
+        key=Callback(TriggerStorageMigration, randomize=timestamp()),
+        title=pad_title("Trigger subtitle storage migration (expensive)"),
+    ))
+    oc.add(DirectoryObject(
+        key=Callback(TriggerCacheMaintenance, randomize=timestamp()),
+        title=pad_title("Trigger cache maintenance (refiners, providers and packs/archives)"),
+    ))
+    oc.add(DirectoryObject(
+        key=Callback(ApplyDefaultMods, randomize=timestamp()),
+        title=pad_title("Apply configured default subtitle mods to all (active) stored subtitles"),
+    ))
+    oc.add(DirectoryObject(
+        key=Callback(ReApplyMods, randomize=timestamp()),
+        title=pad_title("Re-Apply mods of all stored subtitles"),
+    ))
+    oc.add(DirectoryObject(
+        key=Callback(LogStorage, key="tasks", randomize=timestamp()),
+        title=pad_title("Log the plugin's scheduled tasks state storage"),
+    ))
+    oc.add(DirectoryObject(
+        key=Callback(LogStorage, key="ignore", randomize=timestamp()),
+        title=pad_title("Log the plugin's internal ignorelist storage"),
+    ))
+    oc.add(DirectoryObject(
+        key=Callback(LogStorage, key=None, randomize=timestamp()),
+        title=pad_title("Log the plugin's complete state storage"),
+    ))
+    oc.add(DirectoryObject(
+        key=Callback(ResetStorage, key="tasks", randomize=timestamp()),
+        title=pad_title("Reset the plugin's scheduled tasks state storage"),
+    ))
+    oc.add(DirectoryObject(
+        key=Callback(ResetStorage, key="ignore", randomize=timestamp()),
+        title=pad_title("Reset the plugin's internal ignorelist storage"),
+    ))
+    oc.add(DirectoryObject(
+        key=Callback(InvalidateCache, randomize=timestamp()),
+        title=pad_title("Invalidate Sub-Zero metadata caches (subliminal)"),
+    ))
+    oc.add(DirectoryObject(
+        key=Callback(ResetProviderThrottle, randomize=timestamp()),
+        title=pad_title("Reset provider throttle states"),
+    ))
+    return oc
+
+
+def DispatchRestart():
+    Thread.CreateTimer(1.0, Restart)
+
+
+@route(PREFIX + '/advanced/restart/trigger')
+@debounce
+def TriggerRestart(randomize=None):
+    set_refresh_menu_state("Restarting the plugin")
+    DispatchRestart()
+    return fatality(header="Restart triggered, please wait about 5 seconds", force_title=" ", only_refresh=True,
+                    replace_parent=True,
+                    no_history=True, randomize=timestamp())
+
+
+@route(PREFIX + '/advanced/restart/execute')
+def Restart():
+    Plex[":/plugins"].restart(PLUGIN_IDENTIFIER)
+
+
+@route(PREFIX + '/storage/reset', sure=bool)
+@debounce
+def ResetStorage(key, randomize=None, sure=False):
+    if not sure:
+        oc = SubFolderObjectContainer(no_history=True, title1="Reset subtitle storage", title2="Are you sure?")
+        oc.add(DirectoryObject(
+            key=Callback(ResetStorage, key=key, sure=True, randomize=timestamp()),
+            title=pad_title("Are you really sure?"),
+
+        ))
+        return oc
+
+    reset_storage(key)
+
+    if key == "tasks":
+        # reinitialize the scheduler
+        scheduler.init_storage()
+        scheduler.setup_tasks()
+
+    return AdvancedMenu(
+        randomize=timestamp(),
+        header='Success',
+        message='Information Storage (%s) reset' % key
+    )
+
+
+@route(PREFIX + '/storage/log')
+def LogStorage(key, randomize=None):
+    log_storage(key)
+    return AdvancedMenu(
+        randomize=timestamp(),
+        header='Success',
+        message='Information Storage (%s) logged' % key
+    )
+
+
+@route(PREFIX + '/triggerbetter')
+@debounce
+def TriggerBetterSubtitles(randomize=None):
+    scheduler.dispatch_task("FindBetterSubtitles")
+    return AdvancedMenu(
+        randomize=timestamp(),
+        header='Success',
+        message='FindBetterSubtitles triggered'
+    )
+
+
+
+@route(PREFIX + '/skipbetter')
+@debounce
+def SkipFindBetterSubtitles(randomize=None):
+    task = scheduler.task("FindBetterSubtitles")
+    task.last_run = datetime.datetime.now()
+
+    return AdvancedMenu(
+        randomize=timestamp(),
+        header='Success',
+        message='FindBetterSubtitles skipped'
+    )
+
+
+@route(PREFIX + '/triggermaintenance')
+@debounce
+def TriggerStorageMaintenance(randomize=None):
+    scheduler.dispatch_task("SubtitleStorageMaintenance")
+    return AdvancedMenu(
+        randomize=timestamp(),
+        header='Success',
+        message='SubtitleStorageMaintenance triggered'
+    )
+
+
+@route(PREFIX + '/triggerstoragemigration')
+@debounce
+def TriggerStorageMigration(randomize=None):
+    scheduler.dispatch_task("MigrateSubtitleStorage")
+    return AdvancedMenu(
+        randomize=timestamp(),
+        header='Success',
+        message='MigrateSubtitleStorage triggered'
+    )
+
+
+@route(PREFIX + '/triggercachemaintenance')
+@debounce
+def TriggerCacheMaintenance(randomize=None):
+    scheduler.dispatch_task("CacheMaintenance")
+    return AdvancedMenu(
+        randomize=timestamp(),
+        header='Success',
+        message='TriggerCacheMaintenance triggered'
+    )
+
+
+def apply_default_mods(reapply_current=False):
+    storage = get_subtitle_storage()
+    subs_applied = 0
+    for fn in storage.get_all_files():
+        data = storage.load(None, filename=fn)
+        if data:
+            video_id = data.video_id
+            item_type = get_item_kind_from_rating_key(video_id)
+            if not item_type:
+                continue
+
+            for part_id, part in data.parts.iteritems():
+                for lang, subs in part.iteritems():
+                    current_sub = subs.get("current")
+                    if not current_sub:
+                        continue
+                    sub = subs[current_sub]
+
+                    if not sub.content:
+                        continue
+
+                    current_mods = sub.mods or []
+                    if not reapply_current:
+                        add_mods = list(set(config.default_mods).difference(set(current_mods)))
+                        if not add_mods:
+                            continue
+                    else:
+                        if not current_mods:
+                            continue
+                        add_mods = []
+
+                    try:
+                        set_mods_for_part(video_id, part_id, Language.fromietf(lang), item_type, add_mods, mode="add")
+                    except:
+                        Log.Error("Couldn't set mods for %s:%s: %s", video_id, part_id, traceback.format_exc())
+                        continue
+
+                    subs_applied += 1
+    storage.destroy()
+    Log.Debug("Applied mods to %i items" % subs_applied)
+
+
+@route(PREFIX + '/applydefaultmods')
+@debounce
+def ApplyDefaultMods(randomize=None):
+    Thread.CreateTimer(1.0, apply_default_mods)
+    return AdvancedMenu(
+        randomize=timestamp(),
+        header='Success',
+        message='This may take some time ...'
+    )
+
+
+@route(PREFIX + '/reapplyallmods')
+@debounce
+def ReApplyMods(randomize=None):
+    Thread.CreateTimer(1.0, apply_default_mods, reapply_current=True)
+    return AdvancedMenu(
+        randomize=timestamp(),
+        header='Success',
+        message='This may take some time ...'
+    )
+
+
+@route(PREFIX + '/get_logs_link')
+def GetLogsLink():
+    if not config.plex_token:
+        oc = ObjectContainer(title2="Download Logs", no_cache=True, no_history=True,
+                             header="Sorry, feature unavailable",
+                             message="Universal Plex token not available")
+        return oc
+
+    # try getting the link base via the request in context, first, otherwise use the public ip
+    req_headers = Core.sandbox.context.request.headers
+    get_external_ip = True
+    link_base = ""
+
+    if "Origin" in req_headers:
+        link_base = req_headers["Origin"]
+        Log.Debug("Using origin-based link_base")
+        get_external_ip = False
+
+    elif "Referer" in req_headers:
+        parsed = urlparse.urlparse(req_headers["Referer"])
+        link_base = "%s://%s%s" % (parsed.scheme, parsed.hostname, (":%s" % parsed.port) if parsed.port else "")
+        Log.Debug("Using referer-based link_base")
+        get_external_ip = False
+
+    if get_external_ip or "plex.tv" in link_base:
+        ip = Core.networking.http_request("http://www.plexapp.com/ip.php", cacheTime=7200).content.strip()
+        link_base = "https://%s:32400" % ip
+        Log.Debug("Using ip-based fallback link_base")
+
+    logs_link = "%s%s?X-Plex-Token=%s" % (link_base, PREFIX + '/logs', config.plex_token)
+    oc = ObjectContainer(title2=logs_link, no_cache=True, no_history=True,
+                         header="Copy this link and open this in your browser, please",
+                         message=logs_link)
+    return oc
+
+
+@route(PREFIX + '/logs')
+def DownloadLogs():
+    buff = StringIO.StringIO()
+    zip_archive = ZipFile(buff, mode='w', compression=ZIP_DEFLATED)
+
+    logs = sorted(glob.glob(config.plugin_log_path + '*')) + [config.server_log_path]
+    for path in logs:
+        data = StringIO.StringIO()
+        data.write(FileIO.read(path))
+        zip_archive.writestr(os.path.basename(path), data.getvalue())
+
+    zip_archive.close()
+
+    return ZipObject(buff.getvalue())
+
+
+@route(PREFIX + '/invalidatecache')
+@debounce
+def InvalidateCache(randomize=None):
+    from subliminal.cache import region
+    if config.new_style_cache:
+        region.backend.clear()
+    else:
+        region.invalidate()
+    return AdvancedMenu(
+        randomize=timestamp(),
+        header='Success',
+        message='Cache invalidated'
+    )
+
+
+@route(PREFIX + '/pin')
+def PinMenu(pin="", randomize=None, success_go_to="channel"):
+    oc = ObjectContainer(title2="Enter PIN number %s" % (len(pin) + 1), no_cache=True, no_history=True,
+                         skip_pin_lock=True)
+
+    if pin == config.pin:
+        Dict["pin_correct_time"] = datetime.datetime.now()
+        config.locked = False
+        if success_go_to == "channel":
+            return fatality(force_title="PIN correct", header="PIN correct", no_history=True)
+        elif success_go_to == "advanced":
+            return AdvancedMenu(randomize=timestamp())
+
+    for i in range(10):
+        oc.add(DirectoryObject(
+            key=Callback(PinMenu, randomize=timestamp(), pin=pin + str(i), success_go_to=success_go_to),
+            title=pad_title(str(i)),
+        ))
+    oc.add(DirectoryObject(
+        key=Callback(PinMenu, randomize=timestamp(), success_go_to=success_go_to),
+        title=pad_title("Reset"),
+    ))
+    return oc
+
+
+@route(PREFIX + '/pin_lock')
+def ClearPin(randomize=None):
+    Dict["pin_correct_time"] = None
+    config.locked = True
+    return fatality(force_title="Menu locked", header=" ", no_history=True)
+
+
+@route(PREFIX + '/reset_throttle')
+def ResetProviderThrottle(randomize=None):
+    Dict["provider_throttle"] = {}
+    Dict.Save()
+    return AdvancedMenu(
+        randomize=timestamp(),
+        header='Success',
+        message='Provider throttles reset'
+    )
@@ -0,0 +1,27 @@
+# coding=utf-8
+
+from support.config import config
+
+
+def enable_channel_wrapper(func):
+    """
+    returns the original wrapper :func: (route or handler) if applicable, else the plain to-be-wrapped function
+    :param func: original wrapper
+    :return: original wrapper or wrapped function
+    """
+    def noop(*args, **kwargs):
+        def inner(*a, **k):
+            """
+            :param a: args
+            :param k: kwargs
+            :return: originally to-be-wrapped function
+            """
+            return a[0]
+
+        return inner
+
+    def wrap(*args, **kwargs):
+        enforce_route = kwargs.pop("enforce_route", None)
+        return (func if (config.enable_channel or enforce_route) else noop)(*args, **kwargs)
+
+    return wrap
@@ -0,0 +1,641 @@
+# coding=utf-8
+import os
+
+from subzero.language import Language
+
+from sub_mod import SubtitleModificationsMenu
+from menu_helpers import debounce, SubFolderObjectContainer, default_thumb, add_ignore_options, get_item_task_data, \
+    set_refresh_menu_state, route, extract_embedded_sub
+
+from refresh_item import RefreshItem
+from subzero.constants import PREFIX
+from support.config import config, TEXT_SUBTITLE_EXTS
+from support.helpers import timestamp, df, get_language, display_language, get_language_from_stream
+from support.items import get_item_kind_from_rating_key, get_item, get_current_sub, get_item_title, save_stored_sub
+from support.plex_media import get_plex_metadata, get_part, get_embedded_subtitle_streams
+from support.scanning import scan_videos
+from support.scheduler import scheduler
+from support.storage import get_subtitle_storage
+
+
+# fixme: needs kwargs cleanup
+
+@route(PREFIX + '/item/{rating_key}/actions')
+@debounce
+def ItemDetailsMenu(rating_key, title=None, base_title=None, item_title=None, randomize=None, header=None,
+                    message=None):
+    """
+    displays the item details menu of an item that doesn't contain any deeper tree, such as a movie or an episode
+    :param rating_key:
+    :param title:
+    :param base_title:
+    :param item_title:
+    :param randomize:
+    :return:
+    """
+    from interface.main import IgnoreMenu
+
+    title = unicode(base_title) + " > " + unicode(title) if base_title else unicode(title)
+    item = plex_item = get_item(rating_key)
+    current_kind = get_item_kind_from_rating_key(rating_key)
+
+    timeout = 30
+
+    oc = SubFolderObjectContainer(title2=title, replace_parent=True, header=header, message=message)
+
+    if not item:
+        oc.add(DirectoryObject(
+            key=Callback(ItemDetailsMenu, rating_key=rating_key, title=title, base_title=base_title,
+                         item_title=item_title, randomize=timestamp()),
+            title=u"Item not found: %s!" % item_title,
+            summary="Plex didn't return any information about the item, please refresh it and come back later",
+            thumb=default_thumb
+        ))
+        return oc
+
+    # add back to season for episode
+    if current_kind == "episode":
+        from interface.menu import MetadataMenu
+        show = get_item(item.show.rating_key)
+        season = get_item(item.season.rating_key)
+
+        oc.add(DirectoryObject(
+            key=Callback(MetadataMenu, rating_key=season.rating_key, title=season.title, base_title=show.title,
+                         previous_item_type="show", previous_rating_key=show.rating_key,
+                         display_items=True, randomize=timestamp()),
+            title=u"< Back to %s" % season.title,
+            summary="Back to %s > %s" % (show.title, season.title),
+            thumb=season.thumb or default_thumb
+        ))
+
+    oc.add(DirectoryObject(
+        key=Callback(RefreshItem, rating_key=rating_key, item_title=item_title, randomize=timestamp(),
+                     timeout=timeout * 1000),
+        title=u"Refresh: %s" % item_title,
+        summary="Refreshes the %s, possibly searching for missing and picking up new subtitles on disk" % current_kind,
+        thumb=item.thumb or default_thumb
+    ))
+    oc.add(DirectoryObject(
+        key=Callback(RefreshItem, rating_key=rating_key, item_title=item_title, force=True, randomize=timestamp(),
+                     timeout=timeout * 1000),
+        title=u"Force-find subtitles: %s" % item_title,
+        summary="Issues a forced refresh, ignoring known subtitles and searching for new ones",
+        thumb=item.thumb or default_thumb
+    ))
+
+    # get stored subtitle info for item id
+    subtitle_storage = get_subtitle_storage()
+    stored_subs = subtitle_storage.load_or_new(item)
+
+    # look for subtitles for all available media parts and all of their languages
+    has_multiple_parts = len(plex_item.media) > 1
+    part_index = 0
+    for media in plex_item.media:
+        for part in media.parts:
+            filename = os.path.basename(part.file)
+            if not os.path.exists(part.file):
+                continue
+
+            part_id = str(part.id)
+            part_index += 1
+
+            part_index_addon = ""
+            part_summary_addon = ""
+            if has_multiple_parts:
+                part_index_addon = u"File %s: " % part_index
+                part_summary_addon = "%s " % filename
+
+            # iterate through all configured languages
+            for lang in config.lang_list:
+                # get corresponding stored subtitle data for that media part (physical media item), for language
+                current_sub = stored_subs.get_any(part_id, lang)
+                current_sub_id = None
+                current_sub_provider_name = None
+
+                summary = u"%sNo current subtitle in storage" % part_summary_addon
+                current_score = None
+                if current_sub:
+                    current_sub_id = current_sub.id
+                    current_sub_provider_name = current_sub.provider_name
+                    current_score = current_sub.score
+
+                    summary = u"%sCurrent subtitle: %s (added: %s, %s), Language: %s, Score: %i, Storage: %s" % \
+                              (part_summary_addon, current_sub.provider_name, df(current_sub.date_added),
+                               current_sub.mode_verbose, display_language(lang), current_sub.score,
+                               current_sub.storage_type)
+
+                    oc.add(DirectoryObject(
+                        key=Callback(SubtitleOptionsMenu, rating_key=rating_key, part_id=part_id, title=title,
+                                     item_title=item_title, language=lang, language_name=display_language(lang),
+                                     current_id=current_sub_id,
+                                     item_type=plex_item.type, filename=filename, current_data=summary,
+                                     randomize=timestamp(), current_provider=current_sub_provider_name,
+                                     current_score=current_score),
+                        title=u"%sManage %s subtitle" % (part_index_addon, display_language(lang)),
+                        summary=summary
+                    ))
+                else:
+                    oc.add(DirectoryObject(
+                        key=Callback(ListAvailableSubsForItemMenu, rating_key=rating_key, part_id=part_id, title=title,
+                                     item_title=item_title, language=lang, language_name=display_language(lang),
+                                     current_id=current_sub_id,
+                                     item_type=plex_item.type, filename=filename, current_data=summary,
+                                     randomize=timestamp(), current_provider=current_sub_provider_name,
+                                     current_score=current_score),
+                        title=u"%sList %s subtitles" % (part_index_addon, display_language(lang)),
+                        summary=summary
+                    ))
+
+            if config.plex_transcoder:
+                # embedded subtitles
+                embedded_count = 0
+                embedded_langs = []
+                for stream in part.streams:
+                    # subtitle stream
+                    if stream.stream_type == 3 and not stream.stream_key and stream.codec in TEXT_SUBTITLE_EXTS:
+                        lang = get_language_from_stream(stream.language_code)
+
+                        if not lang and config.treat_und_as_first:
+                            lang = list(config.lang_list)[0]
+
+                        if lang:
+                            embedded_langs.append(lang)
+                            embedded_count += 1
+
+                if embedded_count:
+                    oc.add(DirectoryObject(
+                        key=Callback(ListEmbeddedSubsForItemMenu, rating_key=rating_key, part_id=part_id, title=title,
+                                     item_type=plex_item.type, item_title=item_title, base_title=base_title,
+                                     randomize=timestamp()),
+                        title=u"%sEmbedded subtitles (%s)" % (part_index_addon, ", ".join(display_language(l) for l in
+                                                                                          set(embedded_langs))),
+                        summary=u"Extract and activate embedded subtitle streams"
+                    ))
+
+    ignore_title = item_title
+    if current_kind == "episode":
+        ignore_title = get_item_title(item)
+    add_ignore_options(oc, "videos", title=ignore_title, rating_key=rating_key, callback_menu=IgnoreMenu)
+    subtitle_storage.destroy()
+
+    return oc
+
+
+@route(PREFIX + '/item/current_sub/{rating_key}/{part_id}')
+def SubtitleOptionsMenu(**kwargs):
+    oc = SubFolderObjectContainer(title2=unicode(kwargs["title"]), replace_parent=True, header=kwargs.get("header"),
+                                  message=kwargs.get("message"))
+    rating_key = kwargs["rating_key"]
+    part_id = kwargs["part_id"]
+    language = kwargs["language"]
+    current_data = kwargs["current_data"]
+
+    current_sub, stored_subs, storage = get_current_sub(rating_key, part_id, language)
+    subs_count = stored_subs.count(part_id, language)
+    kwargs.pop("randomize")
+
+    oc.add(DirectoryObject(
+        key=Callback(ItemDetailsMenu, rating_key=kwargs["rating_key"], item_title=kwargs["item_title"],
+                     title=kwargs["title"], randomize=timestamp()),
+        title=u"< Back to %s" % kwargs["title"],
+        summary=kwargs["current_data"],
+        thumb=default_thumb
+    ))
+    if subs_count:
+        oc.add(DirectoryObject(
+            key=Callback(ListStoredSubsForItemMenu, randomize=timestamp(), **kwargs),
+            title=u"Select active %s subtitle" % kwargs["language_name"],
+            summary=u"%d subtitles in storage" % subs_count
+        ))
+
+    oc.add(DirectoryObject(
+        key=Callback(ListAvailableSubsForItemMenu, randomize=timestamp(), **kwargs),
+        title=u"List available %s subtitles" % kwargs["language_name"],
+        summary=kwargs["current_data"]
+    ))
+    if current_sub:
+        oc.add(DirectoryObject(
+            key=Callback(SubtitleModificationsMenu, randomize=timestamp(), **kwargs),
+            title=u"Modify current %s subtitle" % kwargs["language_name"],
+            summary=u"Currently applied mods: %s" % (", ".join(current_sub.mods) if current_sub.mods else "none")
+        ))
+
+        if current_sub.provider_name != "embedded":
+            oc.add(DirectoryObject(
+                key=Callback(BlacklistSubtitleMenu, randomize=timestamp(), **kwargs),
+                title=u"Blacklist current %s subtitle and search for a new one" % kwargs["language_name"],
+                summary=current_data
+            ))
+
+        current_bl, subs = stored_subs.get_blacklist(part_id, language)
+        if current_bl:
+            oc.add(DirectoryObject(
+                key=Callback(ManageBlacklistMenu, randomize=timestamp(), **kwargs),
+                title=u"Manage blacklist (%s contained)" % len(current_bl),
+                summary=u"Inspect currently blacklisted subtitles"
+            ))
+
+    storage.destroy()
+    return oc
+
+
+@route(PREFIX + '/item/list_stored_subs/{rating_key}/{part_id}')
+def ListStoredSubsForItemMenu(**kwargs):
+    oc = SubFolderObjectContainer(title2=unicode(kwargs["title"]), replace_parent=True)
+    rating_key = kwargs["rating_key"]
+    part_id = kwargs["part_id"]
+    language = Language.fromietf(kwargs["language"])
+
+    current_sub, stored_subs, storage = get_current_sub(rating_key, part_id, language)
+    all_subs = stored_subs.get_all(part_id, language)
+    kwargs.pop("randomize")
+
+    for key, subtitle in sorted(filter(lambda x: x[0] not in ("current", "blacklist"), all_subs.items()),
+                                key=lambda x: x[1].date_added, reverse=True):
+        is_current = key == all_subs["current"]
+
+        summary = u"added: %s, %s, Language: %s, Score: %i, Storage: %s" % \
+                  (df(subtitle.date_added),
+                   subtitle.mode_verbose, display_language(language), subtitle.score,
+                   subtitle.storage_type)
+
+        sub_name = subtitle.provider_name
+        if sub_name == "embedded":
+            sub_name += " (%s)" % subtitle.id
+
+        oc.add(DirectoryObject(
+            key=Callback(SelectStoredSubForItemMenu, randomize=timestamp(), sub_key="__".join(key), **kwargs),
+            title=u"%s%s, Score: %s" % ("Current: " if is_current else "Stored: ", sub_name,
+                                        subtitle.score),
+            summary=summary
+        ))
+
+    return oc
+
+
+@route(PREFIX + '/item/set_current_sub/{rating_key}/{part_id}')
+@debounce
+def SelectStoredSubForItemMenu(**kwargs):
+    rating_key = kwargs["rating_key"]
+    part_id = kwargs["part_id"]
+    language = Language.fromietf(kwargs["language"])
+    item_type = kwargs["item_type"]
+    sub_key = tuple(kwargs.pop("sub_key").split("__"))
+
+    plex_item = get_item(rating_key)
+    storage = get_subtitle_storage()
+    stored_subs = storage.load(plex_item.rating_key)
+
+    subtitles = stored_subs.get_all(part_id, language)
+    subtitle = subtitles[sub_key]
+
+    subtitles["current"] = sub_key
+
+    save_stored_sub(subtitle, rating_key, part_id, language, item_type, plex_item=plex_item, storage=storage,
+                    stored_subs=stored_subs)
+
+    storage.save(stored_subs)
+    storage.destroy()
+
+    kwargs.pop("randomize")
+
+    kwargs["header"] = 'Success'
+    kwargs["message"] = 'Subtitle saved to disk'
+
+    return SubtitleOptionsMenu(randomize=timestamp(), **kwargs)
+
+
+@route(PREFIX + '/item/blacklist_recent/{language}')
+@route(PREFIX + '/item/blacklist_recent')
+def BlacklistRecentSubtitleMenu(**kwargs):
+    if "last_played_items" not in Dict or not Dict["last_played_items"]:
+        return
+
+    rating_key = Dict["last_played_items"][0]
+    kwargs["rating_key"] = rating_key
+    return BlacklistAllPartsSubtitleMenu(**kwargs)
+
+
+@route(PREFIX + '/item/blacklist_all/{rating_key}/{language}')
+@route(PREFIX + '/item/blacklist_all/{rating_key}')
+def BlacklistAllPartsSubtitleMenu(**kwargs):
+    rating_key = kwargs.get("rating_key")
+    language = kwargs.get("language")
+    if language:
+        language = Language.fromietf(language)
+
+    item = get_item(rating_key)
+
+    if not item:
+        return
+
+    item_title = get_item_title(item)
+
+    subtitle_storage = get_subtitle_storage()
+    stored_subs = subtitle_storage.load_or_new(item)
+    for part_id, languages in stored_subs.parts.iteritems():
+        sub_dict = languages
+        if language:
+            key = str(language)
+            if key not in sub_dict:
+                continue
+
+            sub_dict = {key: sub_dict[key]}
+
+        for language, subs in sub_dict.iteritems():
+            if "current" in subs:
+                stored_subs.blacklist(part_id, language, subs["current"])
+                Log.Info("Added %s to blacklist", subs["current"])
+
+    subtitle_storage.save(stored_subs)
+    subtitle_storage.destroy()
+
+    return RefreshItem(rating_key=rating_key, item_title=item_title, force=True, randomize=timestamp(), timeout=30000)
+
+
+def blacklist(rating_key, part_id, language):
+    current_sub, stored_subs, storage = get_current_sub(rating_key, part_id, language)
+    if not current_sub:
+        return
+
+    stored_subs.blacklist(part_id, language, current_sub.key)
+    storage.save(stored_subs)
+    storage.destroy()
+
+    Log.Info("Added %s to blacklist", current_sub.key)
+
+    return True
+
+
+@route(PREFIX + '/item/blacklist/{rating_key}/{part_id}')
+@debounce
+def BlacklistSubtitleMenu(**kwargs):
+    rating_key = kwargs["rating_key"]
+    part_id = kwargs["part_id"]
+    language = kwargs["language"]
+    item_title = kwargs["item_title"]
+
+    blacklist(rating_key, part_id, language)
+    kwargs.pop("randomize")
+
+    return RefreshItem(rating_key=rating_key, item_title=item_title, force=True, randomize=timestamp(), timeout=30000)
+
+
+@route(PREFIX + '/item/manage_blacklist/{rating_key}/{part_id}', force=bool)
+@debounce
+def ManageBlacklistMenu(**kwargs):
+    oc = SubFolderObjectContainer(title2=unicode(kwargs["title"]), replace_parent=True)
+    rating_key = kwargs["rating_key"]
+    part_id = kwargs["part_id"]
+    language = kwargs["language"]
+    remove_sub_key = kwargs.pop("remove_sub_key", None)
+
+    current_sub, stored_subs, storage = get_current_sub(rating_key, part_id, language)
+    current_bl, subs = stored_subs.get_blacklist(part_id, language)
+
+    if remove_sub_key:
+        remove_sub_key = tuple(remove_sub_key.split("__"))
+        stored_subs.blacklist(part_id, language, remove_sub_key, add=False)
+        storage.save(stored_subs)
+        Log.Info("Removed %s from blacklist", remove_sub_key)
+
+    kwargs.pop("randomize")
+
+    oc.add(DirectoryObject(
+        key=Callback(ItemDetailsMenu, rating_key=kwargs["rating_key"], item_title=kwargs["item_title"],
+                     title=kwargs["title"], randomize=timestamp()),
+        title=u"< Back to %s" % kwargs["title"],
+        summary=kwargs["current_data"],
+        thumb=default_thumb
+    ))
+
+    def sorter(pair):
+        # thanks RestrictedModule parser for messing with lambda (x, y)
+        return pair[1]["date_added"]
+
+    for sub_key, data in sorted(current_bl.iteritems(), key=sorter, reverse=True):
+        provider_name, subtitle_id = sub_key
+        title = u"%s, %s (added: %s, %s), Language: " \
+                u"%s, Score: %i, Storage: %s" % (provider_name, subtitle_id, df(data["date_added"]),
+                                                 current_sub.get_mode_verbose(data["mode"]),
+                                                 display_language(Language.fromietf(language)), data["score"],
+                                                 data["storage_type"])
+        oc.add(DirectoryObject(
+            key=Callback(ManageBlacklistMenu, remove_sub_key="__".join(sub_key), randomize=timestamp(), **kwargs),
+            title=title,
+            summary=u"Remove subtitle from blacklist"
+        ))
+
+    storage.destroy()
+
+    return oc
+
+
+@route(PREFIX + '/item/search/{rating_key}/{part_id}', force=bool)
+@debounce
+def ListAvailableSubsForItemMenu(rating_key=None, part_id=None, title=None, item_title=None, filename=None,
+                                 item_type="episode", language=None, language_name=None, force=False, current_id=None,
+                                 current_data=None,
+                                 current_provider=None, current_score=None, randomize=None):
+    assert rating_key, part_id
+
+    running = scheduler.is_task_running("AvailableSubsForItem")
+    search_results = get_item_task_data("AvailableSubsForItem", rating_key, language)
+
+    if (search_results is None or force) and not running:
+        scheduler.dispatch_task("AvailableSubsForItem", rating_key=rating_key, item_type=item_type, part_id=part_id,
+                                language=language)
+        running = True
+
+    oc = SubFolderObjectContainer(title2=unicode(title), replace_parent=True)
+    oc.add(DirectoryObject(
+        key=Callback(ItemDetailsMenu, rating_key=rating_key, item_title=item_title, title=title, randomize=timestamp()),
+        title=u"< Back to %s" % title,
+        summary=current_data,
+        thumb=default_thumb
+    ))
+
+    metadata = get_plex_metadata(rating_key, part_id, item_type)
+    plex_part = None
+    if not config.low_impact_mode:
+        scanned_parts = scan_videos([metadata], ignore_all=True)
+
+        if not scanned_parts:
+            Log.Error("Couldn't list available subtitles for %s", rating_key)
+            return oc
+
+        video, plex_part = scanned_parts.items()[0]
+
+        video_display_data = [video.format] if video.format else []
+        if video.release_group:
+            video_display_data.append(u"by %s" % video.release_group)
+        video_display_data = " ".join(video_display_data)
+    else:
+        video_display_data = metadata["filename"]
+
+    current_display = (u"Current: %s (%s) " % (current_provider, current_score) if current_provider else "")
+    if not running:
+        oc.add(DirectoryObject(
+            key=Callback(ListAvailableSubsForItemMenu, rating_key=rating_key, item_title=item_title, language=language,
+                         filename=filename, part_id=part_id, title=title, current_id=current_id, force=True,
+                         current_provider=current_provider, current_score=current_score,
+                         current_data=current_data, item_type=item_type, randomize=timestamp()),
+            title=u"Search for %s subs (%s)" % (get_language(language).name, video_display_data),
+            summary=u"%sFilename: %s" % (current_display, filename),
+            thumb=default_thumb
+        ))
+
+        if search_results == "found_none":
+            oc.add(DirectoryObject(
+                key=Callback(ListAvailableSubsForItemMenu, rating_key=rating_key, item_title=item_title,
+                             language=language, filename=filename, current_data=current_data, force=True,
+                             part_id=part_id, title=title, current_id=current_id, item_type=item_type,
+                             current_provider=current_provider, current_score=current_score,
+                             randomize=timestamp()),
+                title=u"No subtitles found",
+                summary=u"%sFilename: %s" % (current_display, filename),
+                thumb=default_thumb
+            ))
+    else:
+        oc.add(DirectoryObject(
+            key=Callback(ListAvailableSubsForItemMenu, rating_key=rating_key, item_title=item_title,
+                         language=language, filename=filename, current_data=current_data,
+                         part_id=part_id, title=title, current_id=current_id, item_type=item_type,
+                         current_provider=current_provider, current_score=current_score,
+                         randomize=timestamp()),
+            title=u"Searching for %s subs (%s), refresh here ..." % (display_language(get_language(language)),
+                                                                     video_display_data),
+            summary=u"%sFilename: %s" % (current_display, filename),
+            thumb=default_thumb
+        ))
+
+    if not search_results or search_results == "found_none":
+        return oc
+
+    current_sub, stored_subs, storage = get_current_sub(rating_key, part_id, language)
+    current_bl, subs = stored_subs.get_blacklist(part_id, language)
+
+    seen = []
+    for subtitle in search_results:
+        if subtitle.id in seen:
+            continue
+
+        bl_addon = ""
+        if (str(subtitle.provider_name), str(subtitle.id)) in current_bl:
+            bl_addon = "Blacklisted "
+
+        wrong_fps_addon = ""
+        if subtitle.wrong_fps:
+            if plex_part:
+                wrong_fps_addon = " (wrong FPS, sub: %s, media: %s)" % (subtitle.fps, plex_part.fps)
+            else:
+                wrong_fps_addon = " (wrong FPS, sub: %s, media: unknown, low impact mode)" % subtitle.fps
+
+        oc.add(DirectoryObject(
+            key=Callback(TriggerDownloadSubtitle, rating_key=rating_key, randomize=timestamp(), item_title=item_title,
+                         subtitle_id=str(subtitle.id), language=language),
+            title=u"%s%s: %s, score: %s%s" % (bl_addon, "Available" if current_id != subtitle.id else "Current",
+                                              subtitle.provider_name, subtitle.score, wrong_fps_addon),
+            summary=u"Release: %s, Matches: %s" % (subtitle.release_info, ", ".join(subtitle.matches)),
+            thumb=default_thumb
+        ))
+
+        seen.append(subtitle.id)
+
+    return oc
+
+
+@route(PREFIX + '/download_subtitle/{rating_key}')
+@debounce
+def TriggerDownloadSubtitle(rating_key=None, subtitle_id=None, item_title=None, language=None, randomize=None):
+    from interface.main import fatality
+
+    set_refresh_menu_state("Downloading subtitle for %s" % item_title or rating_key)
+    search_results = get_item_task_data("AvailableSubsForItem", rating_key, language)
+
+    download_subtitle = None
+    for subtitle in search_results:
+        if str(subtitle.id) == subtitle_id:
+            download_subtitle = subtitle
+            break
+    if not download_subtitle:
+        Log.Error(u"Something went horribly wrong")
+
+    else:
+        scheduler.dispatch_task("DownloadSubtitleForItem", rating_key=rating_key, subtitle=download_subtitle)
+
+    scheduler.clear_task_data("AvailableSubsForItem")
+
+    return fatality(randomize=timestamp(), header=" ", replace_parent=True)
+
+
+@route(PREFIX + '/item/embedded/{rating_key}/{part_id}')
+def ListEmbeddedSubsForItemMenu(**kwargs):
+    rating_key = kwargs["rating_key"]
+    part_id = kwargs["part_id"]
+    title = kwargs["title"]
+    kwargs.pop("randomize")
+
+    oc = SubFolderObjectContainer(title2=title, replace_parent=True)
+
+    oc.add(DirectoryObject(
+        key=Callback(ItemDetailsMenu, rating_key=kwargs["rating_key"], item_title=kwargs["item_title"],
+                     base_title=kwargs["base_title"], title=kwargs["item_title"], randomize=timestamp()),
+        title=u"< Back to %s" % kwargs["title"],
+        thumb=default_thumb
+    ))
+
+    plex_item = get_item(rating_key)
+    part = get_part(plex_item, part_id)
+
+    if part:
+        for stream_data in get_embedded_subtitle_streams(part, skip_duplicate_unknown=False):
+            language = stream_data["language"]
+            is_unknown = stream_data["is_unknown"]
+            stream = stream_data["stream"]
+            is_forced = stream_data["is_forced"]
+
+            if language:
+                oc.add(DirectoryObject(
+                    key=Callback(TriggerExtractEmbeddedSubForItemMenu, randomize=timestamp(),
+                                 stream_index=str(stream.index), language=language, with_mods=True, **kwargs),
+                    title=u"Extract stream %s, "
+                          u"%s%s%s%s with default mods" % (stream.index, display_language(language),
+                                                           " (unknown)" if is_unknown else "",
+                                                           " (forced)" if is_forced else "",
+                                                           " (\"%s\")" % stream.title if stream.title else ""),
+                ))
+                oc.add(DirectoryObject(
+                    key=Callback(TriggerExtractEmbeddedSubForItemMenu, randomize=timestamp(),
+                                 stream_index=str(stream.index), language=language, **kwargs),
+                    title=u"Extract stream %s, %s%s%s%s" % (stream.index, display_language(language),
+                                                            " (unknown)" if is_unknown else "",
+                                                            " (forced)" if is_forced else "",
+                                                            " (\"%s\")" % stream.title if stream.title else ""),
+                ))
+    return oc
+
+
+@route(PREFIX + '/item/extract_embedded/{rating_key}/{part_id}/{stream_index}')
+@debounce
+def TriggerExtractEmbeddedSubForItemMenu(**kwargs):
+    rating_key = kwargs["rating_key"]
+    part_id = kwargs.get("part_id")
+    stream_index = kwargs.get("stream_index")
+
+    Thread.Create(extract_embedded_sub, **kwargs)
+    header = u"Extracting of embedded subtitle %s of part %s:%s triggered" % (stream_index, rating_key, part_id)
+
+    kwargs.pop("randomize")
+    kwargs.pop("item_type")
+    kwargs.pop("stream_index")
+    kwargs.pop("part_id")
+    kwargs.pop("with_mods", False)
+    kwargs.pop("language")
+    kwargs["title"] = kwargs["item_title"]
+    kwargs["header"] = header
+    kwargs["message"] = header
+
+    return ItemDetailsMenu(randomize=timestamp(), **kwargs)
+
+
@@ -0,0 +1,438 @@
+# coding=utf-8
+
+from subzero.constants import PREFIX, TITLE, ART
+from support.config import config
+from support.helpers import pad_title, timestamp, df, display_language
+from support.scheduler import scheduler
+from support.ignore import ignore_list
+from support.items import get_item_thumb, get_on_deck_items, get_all_items, get_items_info, get_item, get_item_title
+from menu_helpers import main_icon, debounce, SubFolderObjectContainer, default_thumb, dig_tree, add_ignore_options, \
+    ObjectContainer, route, handler
+from item_details import ItemDetailsMenu
+
+
+@handler(PREFIX, TITLE if not config.is_development else TITLE + " DEV", art=ART, thumb=main_icon)
+@route(PREFIX)
+def fatality(randomize=None, force_title=None, header=None, message=None, only_refresh=False, no_history=False,
+             replace_parent=False):
+    """
+    subzero main menu
+    """
+    from interface.advanced import PinMenu, ClearPin, AdvancedMenu
+    from interface.menu import RefreshMissing, IgnoreListMenu, HistoryMenu
+
+    title = config.full_version  # force_title if force_title is not None else config.full_version
+    oc = ObjectContainer(title1=title, title2=title, header=unicode(header) if header else title, message=message,
+                         no_history=no_history,
+                         replace_parent=replace_parent, no_cache=True)
+
+    # always re-check permissions
+    config.refresh_permissions_status()
+
+    # always re-check enabled sections
+    config.refresh_enabled_sections()
+
+    if config.lock_menu and not config.pin_correct:
+        oc.add(DirectoryObject(
+            key=Callback(PinMenu, randomize=timestamp()),
+            title=pad_title("Enter PIN"),
+            summary="The owner has restricted the access to this menu. Please enter the correct pin",
+        ))
+        return oc
+
+    if not config.permissions_ok and config.missing_permissions:
+        if not isinstance(config.missing_permissions, list):
+            oc.add(DirectoryObject(
+                key=Callback(fatality, randomize=timestamp()),
+                title=pad_title("Insufficient permissions"),
+                summary=config.missing_permissions,
+            ))
+        else:
+            for title, path in config.missing_permissions:
+                oc.add(DirectoryObject(
+                    key=Callback(fatality, randomize=timestamp()),
+                    title=pad_title("Insufficient permissions"),
+                    summary="Insufficient permissions on library %s, folder: %s" % (title, path),
+                ))
+        return oc
+
+    if not config.enabled_sections:
+        oc.add(DirectoryObject(
+            key=Callback(fatality, randomize=timestamp()),
+            title=pad_title("I'm not enabled!"),
+            summary="Please enable me for some of your libraries in your server settings; currently I do nothing",
+        ))
+        return oc
+
+    if not only_refresh:
+        if Dict["current_refresh_state"]:
+            oc.add(DirectoryObject(
+                key=Callback(fatality, force_title=" ", randomize=timestamp()),
+                title=pad_title("Working ... refresh here"),
+                summary="Current state: %s; Last state: %s" % (
+                    (Dict["current_refresh_state"] or "Idle") if "current_refresh_state" in Dict else "Idle",
+                    (Dict["last_refresh_state"] or "None") if "last_refresh_state" in Dict else "None"
+                )
+            ))
+
+        oc.add(DirectoryObject(
+            key=Callback(OnDeckMenu),
+            title="On-deck items",
+            summary="Shows the current on deck items and allows you to individually (force-) refresh their metadata/"
+                    "subtitles.",
+            thumb=R("icon-ondeck.jpg")
+        ))
+        if "last_played_items" in Dict and Dict["last_played_items"]:
+            oc.add(DirectoryObject(
+                key=Callback(RecentlyPlayedMenu),
+                title=pad_title("Recently played items"),
+                summary="Shows the %i recently played items and allows you to individually (force-) refresh their "
+                        "metadata/subtitles." % config.store_recently_played_amount,
+                thumb=R("icon-played.jpg")
+            ))
+        oc.add(DirectoryObject(
+            key=Callback(RecentlyAddedMenu),
+            title="Recently-added items",
+            summary="Shows the recently added items per section.",
+            thumb=R("icon-added.jpg")
+        ))
+        oc.add(DirectoryObject(
+            key=Callback(RecentMissingSubtitlesMenu, randomize=timestamp()),
+            title="Show recently added items with missing subtitles",
+            summary="Lists items with missing subtitles. Click on \"Find recent items with missing subs\" "
+                    "to update list",
+            thumb=R("icon-missing.jpg")
+        ))
+        oc.add(DirectoryObject(
+            key=Callback(SectionsMenu),
+            title="Browse all items",
+            summary="Go through your whole library and manage your ignore list. You can also "
+                    "(force-) refresh the metadata/subtitles of individual items.",
+            thumb=R("icon-browse.jpg")
+        ))
+
+        task_name = "SearchAllRecentlyAddedMissing"
+        task = scheduler.task(task_name)
+
+        if task.ready_for_display:
+            task_state = "Running: %s/%s (%s%%)" % (task.items_done, task.items_searching, task.percentage)
+        else:
+            lr = scheduler.last_run(task_name)
+            nr = scheduler.next_run(task_name)
+            task_state = "Last run: %s; Next scheduled run: %s; Last runtime: %s" % (
+                df(scheduler.last_run(task_name)) if lr else "never",
+                df(scheduler.next_run(task_name)) if nr else "never",
+                str(task.last_run_time).split(".")[0])
+
+        oc.add(DirectoryObject(
+            key=Callback(RefreshMissing, randomize=timestamp()),
+            title="Search for missing subtitles (in recently-added items, max-age: %s)" % Prefs[
+                "scheduler.item_is_recent_age"],
+            summary="Automatically run periodically by the scheduler, if configured. %s" % task_state,
+            thumb=R("icon-search.jpg")
+        ))
+
+        oc.add(DirectoryObject(
+            key=Callback(IgnoreListMenu),
+            title="Display ignore list (%d)" % len(ignore_list),
+            summary="Show the current ignore list (mainly used for the automatic tasks)",
+            thumb=R("icon-ignore.jpg")
+        ))
+
+        oc.add(DirectoryObject(
+            key=Callback(HistoryMenu),
+            title="History",
+            summary="Show the last %i downloaded subtitles" % int(Prefs["history_size"]),
+            thumb=R("icon-history.jpg")
+        ))
+
+    oc.add(DirectoryObject(
+        key=Callback(fatality, force_title=" ", randomize=timestamp()),
+        title=pad_title("Refresh"),
+        summary="Current state: %s; Last state: %s" % (
+            (Dict["current_refresh_state"] or "Idle") if "current_refresh_state" in Dict else "Idle",
+            (Dict["last_refresh_state"] or "None") if "last_refresh_state" in Dict else "None"
+        ),
+        thumb=R("icon-refresh.jpg")
+    ))
+
+    # add re-lock after pin unlock
+    if config.pin:
+        oc.add(DirectoryObject(
+            key=Callback(ClearPin, randomize=timestamp()),
+            title=pad_title("Re-lock menu(s)"),
+            summary="Enabled the PIN again for menu(s)"
+        ))
+
+    if not only_refresh:
+        if "provider_throttle" in Dict and Dict["provider_throttle"].keys():
+            summary_data = []
+            for provider, data in Dict["provider_throttle"].iteritems():
+                reason, until, desc = data
+                summary_data.append("%s until %s (%s)" % (provider, until.strftime("%y/%m/%d %H:%M"), reason))
+
+            oc.add(DirectoryObject(
+                key=Callback(fatality, force_title=" ", randomize=timestamp()),
+                title=pad_title("Throttled providers: %s" % ", ".join(Dict["provider_throttle"].keys())),
+                summary=", ".join(summary_data),
+                thumb=R("icon-throttled.jpg")
+            ))
+
+        oc.add(DirectoryObject(
+            key=Callback(AdvancedMenu),
+            title=pad_title("Advanced functions"),
+            summary="Use at your own risk",
+            thumb=R("icon-advanced.jpg")
+        ))
+
+    return oc
+
+
+@route(PREFIX + '/on_deck')
+def OnDeckMenu(message=None):
+    """
+    displays the items on deck
+    :param message:
+    :return:
+    """
+    return mergedItemsMenu(title="Items On Deck", base_title="Items On Deck", itemGetter=get_on_deck_items)
+
+
+@route(PREFIX + '/recently_played')
+def RecentlyPlayedMenu():
+    base_title = "Recently Played"
+    oc = SubFolderObjectContainer(title2=base_title, replace_parent=True)
+
+    for item in [get_item(rating_key) for rating_key in Dict["last_played_items"]]:
+        if not item:
+            continue
+
+        if getattr(getattr(item, "__class__"), "__name__") not in ("Episode", "Movie"):
+            continue
+
+        item_title = get_item_title(item)
+
+        oc.add(DirectoryObject(
+            title=item_title,
+            key=Callback(ItemDetailsMenu, title=base_title + " > " + item.title, item_title=item.title,
+                         rating_key=item.rating_key)
+        ))
+
+    return oc
+
+
+@route(PREFIX + '/recently_added')
+def RecentlyAddedMenu(message=None):
+    """
+    displays the items recently added per section
+    :param message:
+    :return:
+    """
+    return SectionsMenu(base_title="Recently added", section_items_key="recently_added", ignore_options=False)
+
+
+@route(PREFIX + '/recent', force=bool)
+@debounce
+def RecentMissingSubtitlesMenu(force=False, randomize=None):
+    title = "Items with missing subtitles"
+    oc = SubFolderObjectContainer(title2=title, no_cache=True, no_history=True)
+
+    running = scheduler.is_task_running("MissingSubtitles")
+    task_data = scheduler.get_task_data("MissingSubtitles")
+    missing_items = task_data["missing_subtitles"] if task_data else None
+
+    if ((missing_items is None) or force) and not running:
+        scheduler.dispatch_task("MissingSubtitles")
+        running = True
+
+    if not running:
+        oc.add(DirectoryObject(
+            key=Callback(RecentMissingSubtitlesMenu, force=True, randomize=timestamp()),
+            title=u"Find recent items with missing subtitles",
+            thumb=default_thumb
+        ))
+    else:
+        oc.add(DirectoryObject(
+            key=Callback(RecentMissingSubtitlesMenu, force=False, randomize=timestamp()),
+            title=u"Updating, refresh here ...",
+            thumb=default_thumb
+        ))
+
+    if missing_items is not None:
+        for added_at, item_id, item_title, item, missing_languages in missing_items:
+            oc.add(DirectoryObject(
+                key=Callback(ItemDetailsMenu, title=title + " > " + item_title, item_title=item_title,
+                             rating_key=item_id),
+                title=item_title,
+                summary="Missing: %s" % ", ".join(display_language(l) for l in missing_languages),
+                thumb=get_item_thumb(item) or default_thumb
+            ))
+
+    return oc
+
+
+def mergedItemsMenu(title, itemGetter, itemGetterKwArgs=None, base_title=None, *args, **kwargs):
+    """
+    displays an item list of dynamic kinds of items
+    :param title:
+    :param itemGetter:
+    :param itemGetterKwArgs:
+    :param base_title:
+    :param args:
+    :param kwargs:
+    :return:
+    """
+    oc = SubFolderObjectContainer(title2=title, no_cache=True, no_history=True)
+    items = itemGetter(*args, **kwargs)
+
+    for kind, title, item_id, deeper, item in items:
+        oc.add(DirectoryObject(
+            title=title,
+            key=Callback(ItemDetailsMenu, title=base_title + " > " + title, item_title=title, rating_key=item_id),
+            thumb=get_item_thumb(item) or default_thumb
+        ))
+
+    return oc
+
+
+def determine_section_display(kind, item, pass_kwargs=None):
+    """
+    returns the menu function for a section based on the size of it (amount of items)
+    :param kind:
+    :param item:
+    :return:
+    """
+    if pass_kwargs and pass_kwargs.get("section_items_key", "all") != "all":
+        return SectionMenu
+    if item.size > 80:
+        return SectionFirstLetterMenu
+    return SectionMenu
+
+
+@route(PREFIX + '/ignore/set/{kind}/{rating_key}/{todo}/sure={sure}', kind=str, rating_key=str, todo=str, sure=bool)
+def IgnoreMenu(kind, rating_key, title=None, sure=False, todo="not_set"):
+    """
+    displays the ignore options for a menu
+    :param kind:
+    :param rating_key:
+    :param title:
+    :param sure:
+    :param todo:
+    :return:
+    """
+    is_ignored = rating_key in ignore_list[kind]
+    if not sure:
+        oc = SubFolderObjectContainer(no_history=True, replace_parent=True, title1="%s %s %s %s the ignore list" % (
+            "Add" if not is_ignored else "Remove", ignore_list.verbose(kind), title,
+            "to" if not is_ignored else "from"), title2="Are you sure?")
+        oc.add(DirectoryObject(
+            key=Callback(IgnoreMenu, kind=kind, rating_key=rating_key, title=title, sure=True,
+                         todo="add" if not is_ignored else "remove"),
+            title=pad_title("Are you sure?"),
+        ))
+        return oc
+
+    rel = ignore_list[kind]
+    dont_change = False
+    if todo == "remove":
+        if not is_ignored:
+            dont_change = True
+        else:
+            rel.remove(rating_key)
+            Log.Info("Removed %s (%s) from the ignore list", title, rating_key)
+            ignore_list.remove_title(kind, rating_key)
+            ignore_list.save()
+            state = "removed from"
+    elif todo == "add":
+        if is_ignored:
+            dont_change = True
+        else:
+            rel.append(rating_key)
+            Log.Info("Added %s (%s) to the ignore list", title, rating_key)
+            ignore_list.add_title(kind, rating_key, title)
+            ignore_list.save()
+            state = "added to"
+    else:
+        dont_change = True
+
+    if dont_change:
+        return fatality(force_title=" ", header="Didn't change the ignore list", no_history=True)
+
+    return fatality(force_title=" ", header="%s %s the ignore list" % (title, state), no_history=True)
+
+
+@route(PREFIX + '/sections')
+def SectionsMenu(base_title="Sections", section_items_key="all", ignore_options=True):
+    """
+    displays the menu for all sections
+    :return:
+    """
+    items = get_all_items("sections")
+
+    return dig_tree(SubFolderObjectContainer(title2="Sections", no_cache=True, no_history=True), items, None,
+                    menu_determination_callback=determine_section_display, pass_kwargs={"base_title": base_title,
+                                                                                        "section_items_key": section_items_key,
+                                                                                        "ignore_options": ignore_options},
+                    fill_args={"title": "section_title"})
+
+
+@route(PREFIX + '/section', ignore_options=bool)
+def SectionMenu(rating_key, title=None, base_title=None, section_title=None, ignore_options=True,
+                section_items_key="all"):
+    """
+    displays the contents of a section
+    :param section_items_key:
+    :param rating_key:
+    :param title:
+    :param base_title:
+    :param section_title:
+    :param ignore_options:
+    :return:
+    """
+    from menu import MetadataMenu
+    items = get_all_items(key=section_items_key, value=rating_key, base="library/sections")
+
+    kind, deeper = get_items_info(items)
+    title = unicode(title)
+
+    section_title = title
+    title = base_title + " > " + title
+    oc = SubFolderObjectContainer(title2=title, no_cache=True, no_history=True)
+    if ignore_options:
+        add_ignore_options(oc, "sections", title=section_title, rating_key=rating_key, callback_menu=IgnoreMenu)
+
+    return dig_tree(oc, items, MetadataMenu,
+                    pass_kwargs={"base_title": title, "display_items": deeper, "previous_item_type": "section",
+                                 "previous_rating_key": rating_key})
+
+
+@route(PREFIX + '/section/firstLetter', deeper=bool)
+def SectionFirstLetterMenu(rating_key, title=None, base_title=None, section_title=None, ignore_options=True,
+                           section_items_key="all"):
+    """
+    displays the contents of a section indexed by its first char (A-Z, 0-9...)
+    :param ignore_options: ignored
+    :param section_items_key: ignored
+    :param rating_key:
+    :param title:
+    :param base_title:
+    :param section_title:
+    :return:
+    """
+    from menu import FirstLetterMetadataMenu
+    items = get_all_items(key="first_character", value=rating_key, base="library/sections")
+
+    kind, deeper = get_items_info(items)
+
+    title = unicode(title)
+    oc = SubFolderObjectContainer(title2=section_title, no_cache=True, no_history=True)
+    title = base_title + " > " + title
+    add_ignore_options(oc, "sections", title=section_title, rating_key=rating_key, callback_menu=IgnoreMenu)
+
+    oc.add(DirectoryObject(
+        key=Callback(SectionMenu, title="All", base_title=title, rating_key=rating_key, ignore_options=False),
+        title="All"
+    )
+    )
+    return dig_tree(oc, items, FirstLetterMetadataMenu, force_rating_key=rating_key, fill_args={"key": "key"},
+                    pass_kwargs={"base_title": title, "display_items": deeper, "previous_rating_key": rating_key})
@@ -1,23 +1,32 @@
 # coding=utf-8
+import traceback
 import types
 import datetime
+import subprocess
+import os

-from support.items import get_kind, get_item_thumb
-from support.helpers import get_video_display_title
+from func import enable_channel_wrapper
+from subzero.language import Language
+from support.items import get_kind, get_item_thumb, get_item, get_item_kind_from_item, refresh_item
+from support.helpers import get_video_display_title, pad_title, display_language, quote_args, is_stream_forced
 from support.ignore import ignore_list
 from support.lib import get_intent
 from support.config import config
-from subzero.constants import ICON_SUB
+from subzero.constants import ICON_SUB, ICON
+from support.plex_media import get_part, get_plex_metadata
+from support.scheduler import scheduler
+from support.scanning import scan_videos
+from support.storage import save_subtitles
+
+from subliminal_patch.subtitle import ModifiedSubtitle

 default_thumb = R(ICON_SUB)
+main_icon = ICON if not config.is_development else "icon-dev.jpg"

-
-def should_display_ignore(items, previous=None):
-    kind = get_kind(items)
-    return items and (
-        (kind in ("show", "season")) or
-        (kind == "episode" and previous != "season")
-    )
+# noinspection PyUnboundLocalVariable
+route = enable_channel_wrapper(route)
+# noinspection PyUnboundLocalVariable
+handler = enable_channel_wrapper(handler)


 def add_ignore_options(oc, kind, callback_menu=None, title=None, rating_key=None, add_kind=True):
@@ -41,8 +50,8 @@ def add_ignore_options(oc, kind, callback_menu=None, title=None, rating_key=None

    oc.add(DirectoryObject(
        key=Callback(callback_menu, kind=use_kind, rating_key=rating_key, title=title),
-        title=u"%s %s \"%s\" %s the ignore list" % (
-            "Remove" if in_list else "Add", ignore_list.verbose(kind) if add_kind else "", unicode(title), "from" if in_list else "to")
+        title=u"%s %s \"%s\"" % (
+            "Un-Ignore" if in_list else "Ignore", ignore_list.verbose(kind) if add_kind else "", unicode(title))
    )
    )

@@ -64,7 +73,7 @@ def dig_tree(oc, items, menu_callback, menu_determination_callback=None, force_r
        oc.add(DirectoryObject(
            key=Callback(menu_callback or menu_determination_callback(kind, item, pass_kwargs=pass_kwargs), title=title,
                         rating_key=force_rating_key or key, **add_kwargs),
-            title=title, thumb=thumb, summary=summary
+            title=pad_title(title) if kind in ("show", "season") else title, thumb=thumb, summary=summary
        ))
    return oc

@@ -104,28 +113,10 @@ def set_refresh_menu_state(state_or_media, media_type="movies"):
    Dict["current_refresh_state"] = u"%sRefreshing %s" % ("Force-" if force_refresh else "", unicode(title))


-def enable_channel_wrapper(func):
-    """
-    returns the original wrapper :func: (route or handler) if applicable, else the plain to-be-wrapped function
-    :param func: original wrapper
-    :return: original wrapper or wrapped function
-    """
-    def noop(*args, **kwargs):
-        def inner(*a, **k):
-            """
-            :param a: args
-            :param k: kwargs
-            :return: originally to-be-wrapped function
-            """
-            return a[0]
-
-        return inner
-
-    def wrap(*args, **kwargs):
-        enforce_route = kwargs.pop("enforce_route", None)
-        return (func if config.enable_channel or enforce_route else noop)(*args, **kwargs)
-
-    return wrap
+def get_item_task_data(task_name, rating_key, language):
+    task_data = scheduler.get_task_data(task_name)
+    search_results = task_data.get(rating_key, {}) if task_data else {}
+    return search_results.get(language)


 def debounce(func):
@@ -140,7 +131,7 @@ def debounce(func):

    def wrap(*args, **kwargs):
        if "randomize" in kwargs:
-            if not "menu_history" in Dict:
+            if "menu_history" not in Dict:
                Dict["menu_history"] = {}

            key = get_lookup_key([func] + list(args), kwargs)
@@ -148,13 +139,80 @@ def debounce(func):
                Log.Debug("not triggering %s twice with %s, %s" % (func, args, kwargs))
                return ObjectContainer()
            else:
-                Dict["menu_history"][key] = datetime.datetime.now() + datetime.timedelta(days=1)
-                Dict.Save()
+                Dict["menu_history"][key] = datetime.datetime.now() + datetime.timedelta(hours=6)
+                try:
+                    Dict.Save()
+                except TypeError:
+                    Log.Error("Can't save menu history for: %r", key)
+                    del Dict["menu_history"][key]
+
        return func(*args, **kwargs)

    return wrap


+def extract_embedded_sub(**kwargs):
+    rating_key = kwargs["rating_key"]
+    part_id = kwargs.pop("part_id")
+    stream_index = kwargs.pop("stream_index")
+    with_mods = kwargs.pop("with_mods", False)
+    language = Language.fromietf(kwargs.pop("language"))
+    refresh = kwargs.pop("refresh", True)
+    set_current = kwargs.pop("set_current", True)
+
+    plex_item = kwargs.pop("plex_item", get_item(rating_key))
+    item_type = get_item_kind_from_item(plex_item)
+    part = kwargs.pop("part", get_part(plex_item, part_id))
+    scanned_videos = kwargs.pop("scanned_videos", None)
+
+    any_successful = False
+
+    if part:
+        if not scanned_videos:
+            metadata = get_plex_metadata(rating_key, part_id, item_type, plex_item=plex_item)
+            scanned_videos = scan_videos([metadata], ignore_all=True, skip_hashing=True)
+
+        for stream in part.streams:
+            # subtitle stream
+            if str(stream.index) == stream_index:
+                is_forced = is_stream_forced(stream)
+                bn = os.path.basename(part.file)
+
+                set_refresh_menu_state(u"Extracting subtitle %s of %s" % (stream_index, bn))
+                Log.Info(u"Extracting stream %s (%s) of %s", stream_index, display_language(language), bn)
+
+                out_codec = stream.codec if stream.codec != "mov_text" else "srt"
+
+                args = [
+                    config.plex_transcoder, "-i", part.file, "-map", "0:%s" % stream_index, "-f", out_codec, "-"
+                ]
+                output = None
+                try:
+                    output = subprocess.check_output(quote_args(args), stderr=subprocess.PIPE, shell=True)
+                except:
+                    Log.Error("Extraction failed: %s", traceback.format_exc())
+
+                if output:
+                    subtitle = ModifiedSubtitle(language, mods=config.default_mods if with_mods else None)
+                    subtitle.content = output
+                    subtitle.provider_name = "embedded"
+                    subtitle.id = "stream_%s" % stream_index
+                    subtitle.score = 0
+                    subtitle.set_encoding("utf-8")
+
+                    # fixme: speedup video; only video.name is needed
+                    save_successful = save_subtitles(scanned_videos, {scanned_videos.keys()[0]: [subtitle]}, mode="m",
+                                                     set_current=set_current, is_forced=is_forced)
+                    set_refresh_menu_state(None)
+
+                    if save_successful and refresh:
+                        refresh_item(rating_key)
+
+                    any_successful = True
+
+    return any_successful
+
+
 class SZObjectContainer(ObjectContainer):
    def __init__(self, *args, **kwargs):
        skip_pin_lock = kwargs.pop("skip_pin_lock", False)
@@ -0,0 +1,23 @@
+# coding=utf-8
+
+from subzero.constants import PREFIX
+from menu_helpers import debounce, set_refresh_menu_state, route
+from support.items import refresh_item
+from support.helpers import timestamp
+
+
+@route(PREFIX + '/item/refresh/{rating_key}/force', force=True)
+@route(PREFIX + '/item/refresh/{rating_key}')
+@debounce
+def RefreshItem(rating_key=None, came_from="/recent", item_title=None, force=False, refresh_kind=None,
+                previous_rating_key=None, timeout=8000, randomize=None, trigger=True):
+    assert rating_key
+    from interface.main import fatality
+    header = " "
+    if trigger:
+        set_refresh_menu_state(u"Triggering %sRefresh for %s" % ("Force-" if force else "", item_title))
+        Thread.Create(refresh_item, rating_key=rating_key, force=force, refresh_kind=refresh_kind,
+                      parent_rating_key=previous_rating_key, timeout=int(timeout))
+
+        header = u"%s of item %s triggered" % ("Refresh" if not force else "Forced-refresh", rating_key)
+    return fatality(randomize=timestamp(), header=header, replace_parent=True)
@@ -0,0 +1,281 @@
+# coding=utf-8
+
+import traceback
+import types
+
+from subzero.language import Language
+
+from menu_helpers import debounce, SubFolderObjectContainer, default_thumb, route
+from subzero.modification import registry as mod_registry, SubtitleModifications
+from subzero.constants import PREFIX
+from support.plex_media import get_plex_metadata
+from support.scanning import scan_videos
+from support.helpers import timestamp, pad_title
+from support.items import get_current_sub, set_mods_for_part
+
+
+@route(PREFIX + '/item/sub_mods/{rating_key}/{part_id}', force=bool)
+@debounce
+def SubtitleModificationsMenu(**kwargs):
+    rating_key = kwargs["rating_key"]
+    part_id = kwargs["part_id"]
+    language = kwargs["language"]
+    lang_instance = Language.fromietf(language)
+    current_sub, stored_subs, storage = get_current_sub(rating_key, part_id, language)
+    kwargs.pop("randomize")
+
+    current_mods = current_sub.mods or []
+
+    oc = SubFolderObjectContainer(title2=kwargs["title"], replace_parent=True)
+
+    from interface.item_details import SubtitleOptionsMenu
+    oc.add(DirectoryObject(
+        key=Callback(SubtitleOptionsMenu, randomize=timestamp(), **kwargs),
+        title=u"< Back to subtitle options for: %s" % kwargs["title"],
+        summary=kwargs["current_data"],
+        thumb=default_thumb
+    ))
+
+    for identifier, mod in mod_registry.mods.iteritems():
+        if mod.advanced:
+            continue
+
+        if mod.exclusive and identifier in current_mods:
+            continue
+
+        if mod.languages and lang_instance not in mod.languages:
+            continue
+
+        oc.add(DirectoryObject(
+            key=Callback(SubtitleSetMods, mods=identifier, mode="add", randomize=timestamp(), **kwargs),
+            title=pad_title(mod.description), summary=mod.long_description or ""
+        ))
+
+    fps_mod = SubtitleModifications.get_mod_class("change_FPS")
+    oc.add(DirectoryObject(
+        key=Callback(SubtitleFPSModMenu, randomize=timestamp(), **kwargs),
+        title=pad_title(fps_mod.description), summary=fps_mod.long_description or ""
+    ))
+
+    shift_mod = SubtitleModifications.get_mod_class("shift_offset")
+    oc.add(DirectoryObject(
+        key=Callback(SubtitleShiftModUnitMenu, randomize=timestamp(), **kwargs),
+        title=pad_title(shift_mod.description), summary=shift_mod.long_description or ""
+    ))
+
+    color_mod = SubtitleModifications.get_mod_class("color")
+    oc.add(DirectoryObject(
+        key=Callback(SubtitleColorModMenu, randomize=timestamp(), **kwargs),
+        title=pad_title(color_mod.description), summary=color_mod.long_description or ""
+    ))
+
+    if current_mods:
+        oc.add(DirectoryObject(
+            key=Callback(SubtitleSetMods, mods=None, mode="remove_last", randomize=timestamp(), **kwargs),
+            title=pad_title("Remove last applied mod (%s)" % current_mods[-1]),
+            summary=u"Currently applied mods: %s" % (", ".join(current_mods) if current_mods else "none")
+        ))
+        oc.add(DirectoryObject(
+            key=Callback(SubtitleListMods, randomize=timestamp(), **kwargs),
+            title=pad_title("Manage applied mods"),
+            summary=u"Currently applied mods: %s" % (", ".join(current_mods))
+        ))
+        oc.add(DirectoryObject(
+            key=Callback(SubtitleReapplyMods, randomize=timestamp(), **kwargs),
+            title=pad_title("Reapply applied mods"),
+            summary=u"Currently applied mods: %s" % (", ".join(current_mods) if current_mods else "none")
+        ))
+
+    oc.add(DirectoryObject(
+        key=Callback(SubtitleSetMods, mods=None, mode="clear", randomize=timestamp(), **kwargs),
+        title=pad_title("Restore original version"),
+        summary=u"Currently applied mods: %s" % (", ".join(current_mods) if current_mods else "none")
+    ))
+
+    storage.destroy()
+
+    return oc
+
+
+@route(PREFIX + '/item/sub_mod_fps/{rating_key}/{part_id}', force=bool)
+def SubtitleFPSModMenu(**kwargs):
+    rating_key = kwargs["rating_key"]
+    part_id = kwargs["part_id"]
+    item_type = kwargs["item_type"]
+
+    kwargs.pop("randomize")
+
+    oc = SubFolderObjectContainer(title2=kwargs["title"], replace_parent=True)
+
+    oc.add(DirectoryObject(
+        key=Callback(SubtitleModificationsMenu, randomize=timestamp(), **kwargs),
+        title="< Back to subtitle modification menu"
+    ))
+
+    metadata = get_plex_metadata(rating_key, part_id, item_type)
+    scanned_parts = scan_videos([metadata], ignore_all=True, skip_hashing=True)
+    video, plex_part = scanned_parts.items()[0]
+
+    target_fps = plex_part.fps
+
+    for fps in ["23.980", "23.976", "24.000", "25.000", "29.970", "30.000", "50.000", "59.940", "60.000"]:
+        if float(fps) == float(target_fps):
+            continue
+
+        if float(fps) > float(target_fps):
+            indicator = "subs constantly getting faster"
+        else:
+            indicator = "subs constantly getting slower"
+
+        mod_ident = SubtitleModifications.get_mod_signature("change_FPS", **{"from": fps, "to": target_fps})
+        oc.add(DirectoryObject(
+            key=Callback(SubtitleSetMods, mods=mod_ident, mode="add", randomize=timestamp(), **kwargs),
+            title="%s fps -> %s fps (%s)" % (fps, target_fps, indicator)
+        ))
+
+    return oc
+
+
+POSSIBLE_UNITS = (("ms", "milliseconds"), ("s", "seconds"), ("m", "minutes"), ("h", "hours"))
+POSSIBLE_UNITS_D = dict(POSSIBLE_UNITS)
+
+
+@route(PREFIX + '/item/sub_mod_shift_unit/{rating_key}/{part_id}', force=bool)
+def SubtitleShiftModUnitMenu(**kwargs):
+    oc = SubFolderObjectContainer(title2=kwargs["title"], replace_parent=True)
+
+    kwargs.pop("randomize")
+
+    oc.add(DirectoryObject(
+        key=Callback(SubtitleModificationsMenu, randomize=timestamp(), **kwargs),
+        title="< Back to subtitle modifications"
+    ))
+
+    for unit, title in POSSIBLE_UNITS:
+        oc.add(DirectoryObject(
+            key=Callback(SubtitleShiftModMenu, unit=unit, randomize=timestamp(), **kwargs),
+            title="Adjust by %s" % title
+        ))
+
+    return oc
+
+
+@route(PREFIX + '/item/sub_mod_shift/{rating_key}/{part_id}/{unit}', force=bool)
+def SubtitleShiftModMenu(unit=None, **kwargs):
+    if unit not in POSSIBLE_UNITS_D:
+        raise NotImplementedError
+
+    kwargs.pop("randomize")
+
+    oc = SubFolderObjectContainer(title2=kwargs["title"], replace_parent=True)
+
+    oc.add(DirectoryObject(
+        key=Callback(SubtitleShiftModUnitMenu, randomize=timestamp(), **kwargs),
+        title="< Back to unit selection"
+    ))
+
+    rng = []
+    if unit == "h":
+        rng = list(reversed(range(-10, 0))) + list(reversed(range(1, 11)))
+    elif unit in ("m", "s"):
+        rng = list(reversed(range(-15, 0))) + list(reversed(range(1, 16)))
+    elif unit == "ms":
+        rng = list(reversed(range(-900, 0, 100))) + list(reversed(range(100, 1000, 100)))
+
+    for i in rng:
+        if i == 0:
+            continue
+
+        mod_ident = SubtitleModifications.get_mod_signature("shift_offset", **{unit: i})
+        oc.add(DirectoryObject(
+            key=Callback(SubtitleSetMods, mods=mod_ident, mode="add", randomize=timestamp(), **kwargs),
+            title="%s %s" % (("%s" if i < 0 else "+%s") % i, unit)
+        ))
+
+    return oc
+
+
+@route(PREFIX + '/item/sub_mod_colors/{rating_key}/{part_id}', force=bool)
+def SubtitleColorModMenu(**kwargs):
+    kwargs.pop("randomize")
+
+    color_mod = SubtitleModifications.get_mod_class("color")
+
+    oc = SubFolderObjectContainer(title2=kwargs["title"], replace_parent=True)
+
+    oc.add(DirectoryObject(
+        key=Callback(SubtitleModificationsMenu, randomize=timestamp(), **kwargs),
+        title="< Back to subtitle modification menu"
+    ))
+
+    for color, code in color_mod.colors.iteritems():
+        mod_ident = SubtitleModifications.get_mod_signature("color", **{"name": color})
+        oc.add(DirectoryObject(
+            key=Callback(SubtitleSetMods, mods=mod_ident, mode="add", randomize=timestamp(), **kwargs),
+            title="%s (%s)" % (color, code)
+        ))
+
+    return oc
+
+
+@route(PREFIX + '/item/sub_set_mods/{rating_key}/{part_id}/{mods}/{mode}', force=bool)
+@debounce
+def SubtitleSetMods(mods=None, mode=None, **kwargs):
+    if not isinstance(mods, types.ListType) and mods:
+        mods = [mods]
+
+    rating_key = kwargs["rating_key"]
+    part_id = kwargs["part_id"]
+    lang_a2 = kwargs["language"]
+    item_type = kwargs["item_type"]
+
+    language = Language.fromietf(lang_a2)
+
+    set_mods_for_part(rating_key, part_id, language, item_type, mods, mode=mode)
+
+    kwargs.pop("randomize")
+    return SubtitleModificationsMenu(randomize=timestamp(), **kwargs)
+
+
+@route(PREFIX + '/item/sub_reapply_mods/{rating_key}/{part_id}', force=bool)
+@debounce
+def SubtitleReapplyMods(**kwargs):
+    rating_key = kwargs["rating_key"]
+    part_id = kwargs["part_id"]
+    lang_a2 = kwargs["language"]
+    item_type = kwargs["item_type"]
+
+    language = Language.fromietf(lang_a2)
+
+    set_mods_for_part(rating_key, part_id, language, item_type, [], mode="add")
+
+    kwargs.pop("randomize")
+    return SubtitleModificationsMenu(randomize=timestamp(), **kwargs)
+
+
+@route(PREFIX + '/item/sub_list_mods/{rating_key}/{part_id}', force=bool)
+@debounce
+def SubtitleListMods(**kwargs):
+    rating_key = kwargs["rating_key"]
+    part_id = kwargs["part_id"]
+    language = kwargs["language"]
+    current_sub, stored_subs, storage = get_current_sub(rating_key, part_id, language)
+
+    kwargs.pop("randomize")
+
+    oc = SubFolderObjectContainer(title2=kwargs["title"], replace_parent=True)
+
+    oc.add(DirectoryObject(
+        key=Callback(SubtitleModificationsMenu, randomize=timestamp(), **kwargs),
+        title="< Back to subtitle modifications"
+    ))
+
+    for identifier in current_sub.mods:
+        oc.add(DirectoryObject(
+            key=Callback(SubtitleSetMods, mods=identifier, mode="remove", randomize=timestamp(), **kwargs),
+            title="Remove: %s" % identifier
+        ))
+
+    storage.destroy()
+
+    return oc
@@ -18,7 +18,7 @@ sys.modules["support.plex_media"] = plex_media

 import localmedia

-sys.modules["subzero.localmedia"] = localmedia
+sys.modules["support.localmedia"] = localmedia

 import subtitlehelpers

@@ -28,22 +28,25 @@ import items

 sys.modules["support.items"] = items

-import missing_subtitles
+import scheduler

-sys.modules["support.missing_subtitles"] = missing_subtitles
-
-import background
-
-sys.modules["support.background"] = background
-
-import tasks
-
-sys.modules["support.tasks"] = tasks
+sys.modules["support.scheduler"] = scheduler

 import storage

 sys.modules["support.storage"] = storage

+import scanning
+sys.modules["support.scanning"] = scanning
+
+import missing_subtitles
+
+sys.modules["support.missing_subtitles"] = missing_subtitles
+
+import tasks
+
+sys.modules["support.tasks"] = tasks
+
 import ignore

 sys.modules["support.ignore"] = ignore
@@ -58,3 +61,6 @@ sys.modules["support.data"] = data

 import activities
 sys.modules["support.activities"] = activities
+
+import download
+sys.modules["support.download"] = download
@@ -3,33 +3,31 @@ from wraptor.decorators import throttle
 from config import config
 from items import get_item, get_item_kind_from_item, refresh_item

-from plex_activity import Activity
-from plex_activity.sources.s_logging.main import Logging as Activity_Logging
+Activity = None
+try:
+    from plex_activity import Activity
+except ImportError:
+    pass


 class PlexActivityManager(object):
    def start(self):
        activity_sources_enabled = None

-        if config.universal_plex_token:
+        if not Activity:
+            return
+
+        if config.plex_token:
            from plex import Plex
-            Plex.configuration.defaults.authentication(config.universal_plex_token)
+            Plex.configuration.defaults.authentication(config.plex_token)
            activity_sources_enabled = ["websocket"]
            Activity.on('websocket.playing', self.on_playing)

-        elif config.server_log_path:
-            Activity_Logging.add_hint(config.server_log_path, None)
-            activity_sources_enabled = ["logging"]
-            Activity.on('logging.playing', self.on_playing)
-
        if activity_sources_enabled:
            Activity.start(activity_sources_enabled)

    @throttle(5, instance_method=True)
    def on_playing(self, info):
-        if not config.use_activities:
-            return
-
        # ignore non-playing states and anything too far in
        if info["state"] != "playing" or info["viewOffset"] > 60000:
            return
@@ -41,19 +39,37 @@ class PlexActivityManager(object):
            return

        rating_key = info["ratingKey"]
-        if rating_key not in Dict["last_played_items"]:
-            # new playing; store last 10 recently played items
+
+        # only use integer based rating keys
+        try:
+            int(rating_key)
+        except ValueError:
+            return
+
+        if rating_key in Dict["last_played_items"] and rating_key != Dict["last_played_items"][0]:
+            # shift last played
+            Dict["last_played_items"].insert(0,
+                                             Dict["last_played_items"].pop(Dict["last_played_items"].index(rating_key)))
+            Dict.Save()
+
+        elif rating_key not in Dict["last_played_items"]:
+            # new playing; store last X recently played items
            Dict["last_played_items"].insert(0, rating_key)
-            Dict["last_played_items"] = Dict["last_played_items"][:10]
+            Dict["last_played_items"] = Dict["last_played_items"][:config.store_recently_played_amount]

            Dict.Save()

+            if not config.react_to_activities:
+                return
+
            debug_msg = "Started playing %s. Refreshing it." % rating_key

-            key_to_refresh = None
-            if config.activity_mode in ["refresh", "next_episode", "hybrid"]:
+            # todo: cleanup debug messages for hybrid-plus
+
+            keys_to_refresh = []
+            if config.activity_mode in ["refresh", "next_episode", "hybrid", "hybrid-plus"]:
                # next episode or next episode and current movie
-                if config.activity_mode in ["next_episode", "hybrid"]:
+                if config.activity_mode in ["next_episode", "hybrid", "hybrid-plus"]:
                    plex_item = get_item(rating_key)
                    if not plex_item:
                        Log.Warn("Can't determine media type of %s, skipping" % rating_key)
@@ -61,20 +77,24 @@ class PlexActivityManager(object):

                    if get_item_kind_from_item(plex_item) == "episode":
                        next_ep = self.get_next_episode(rating_key)
+                        if config.activity_mode == "hybrid-plus":
+                            keys_to_refresh.append(rating_key)
                        if next_ep:
-                            key_to_refresh = next_ep.rating_key
+                            keys_to_refresh.append(next_ep.rating_key)
                            debug_msg = "Started playing %s. Refreshing next episode (%s, S%02iE%02i)." % \
                                        (rating_key, next_ep.rating_key, int(next_ep.season.index), int(next_ep.index))

                    else:
                        if config.activity_mode == "hybrid":
-                            key_to_refresh = rating_key
+                            keys_to_refresh.append(rating_key)
                elif config.activity_mode == "refresh":
-                    key_to_refresh = rating_key
+                    keys_to_refresh.append(rating_key)

-                if key_to_refresh:
+                if keys_to_refresh:
                    Log.Debug(debug_msg)
-                    refresh_item(key_to_refresh)
+                    Log.Debug("Refreshing %s", keys_to_refresh)
+                    for key in keys_to_refresh:
+                        refresh_item(key)

    def get_next_episode(self, rating_key):
        plex_item = get_item(rating_key)
@@ -108,4 +128,5 @@ class PlexActivityManager(object):
                                if ep.index == 1:
                                    return ep

+
 activity = PlexActivityManager()
@@ -1,20 +1,41 @@
 # coding=utf-8
-
+import copy
 import os
 import re
 import inspect
-
+import sys
+import rarfile
+import jstyleson
 import datetime

 import subliminal
 import subliminal_patch
-from babelfish import Language
-from subzero.lib.io import FileIO, get_viable_encoding
-from subzero.constants import PLUGIN_NAME, PLUGIN_IDENTIFIER, MOVIE, SHOW
-from lib import Plex
-from helpers import check_write_permissions, cast_bool
+import subzero.constants
+import lib
+from subliminal.exceptions import ServiceUnavailable, DownloadLimitExceeded

-SUBTITLE_EXTS = ['utf', 'utf8', 'utf-8', 'srt', 'smi', 'rt', 'ssa', 'aqt', 'jss', 'ass', 'idx', 'sub', 'txt', 'psb']
+from subliminal_patch.core import is_windows_special_path
+from whichdb import whichdb
+
+from subliminal_patch.exceptions import TooManyRequests
+from subzero.language import Language
+from subliminal.cli import MutexLock
+from subzero.lib.io import FileIO, get_viable_encoding
+from subzero.lib.dict import Dicked
+from subzero.util import get_root_path
+from subzero.constants import PLUGIN_NAME, PLUGIN_IDENTIFIER, MOVIE, SHOW, MEDIA_TYPE_TO_STRING
+from dogpile.cache.region import register_backend as register_cache_backend
+from lib import Plex
+from helpers import check_write_permissions, cast_bool, cast_int, mswindows
+
+register_cache_backend(
+    "subzero.cache.file", "subzero.cache_backends.file", "SZFileBackend")
+
+SUBTITLE_EXTS_BASE = ['utf', 'utf8', 'utf-8', 'srt', 'smi', 'rt', 'ssa', 'aqt', 'jss', 'ass', 'idx', 'sub', 'psb',
+                      'vtt']
+SUBTITLE_EXTS = SUBTITLE_EXTS_BASE + ["txt"]
+
+TEXT_SUBTITLE_EXTS = ("srt", "ass", "ssa", "vtt", "mov_text")
 VIDEO_EXTS = ['3g2', '3gp', 'asf', 'asx', 'avc', 'avi', 'avs', 'bivx', 'bup', 'divx', 'dv', 'dvr-ms', 'evo', 'fli',
              'flv',
              'm2t', 'm2ts', 'm2v', 'm4v', 'mkv', 'mov', 'mp4', 'mpeg', 'mpg', 'mts', 'nsv', 'nuv', 'ogm', 'ogv', 'tp',
@@ -35,15 +56,43 @@ def int_or_default(s, default):
        return default


+VALID_THROTTLE_EXCEPTIONS = (TooManyRequests, DownloadLimitExceeded, ServiceUnavailable)
+
+PROVIDER_THROTTLE_MAP = {
+    "default": {
+        TooManyRequests: (datetime.timedelta(hours=1), "1 hour"),
+        DownloadLimitExceeded: (datetime.timedelta(hours=3), "3 hours"),
+        ServiceUnavailable: (datetime.timedelta(minutes=20), "20 minutes"),
+    },
+    "opensubtitles": {
+        TooManyRequests: (datetime.timedelta(hours=3), "3 hours"),
+        DownloadLimitExceeded: (datetime.timedelta(hours=6), "6 hours"),
+    },
+    "addic7ed": {
+        DownloadLimitExceeded: (datetime.timedelta(hours=24), "24 hours"),
+    }
+}
+
+
 class Config(object):
+    libraries_root = None
    plugin_info = ""
    version = None
    full_version = None
    plugin_log_path = None
    server_log_path = None
    app_support_path = None
+    data_path = None
+    data_items_path = None
    universal_plex_token = None
+    plex_token = None
    is_development = False
+    dbm_supported = False
+    pms_request_timeout = 15
+    low_impact_mode = False
+    new_style_cache = False
+    pack_cache_dir = None
+    advanced = None

    enable_channel = True
    enable_agent = True
@@ -52,10 +101,8 @@ class Config(object):
    lock_advanced_menu = False
    locked = False
    pin_valid_minutes = 10
-    lang_list = None
    subtitle_destination_folder = None
-    providers = None
-    provider_settings = None
+    subtitle_formats = None
    max_recent_items_per_library = 200
    permissions_ok = False
    missing_permissions = None
@@ -65,18 +112,42 @@ class Config(object):
    notify_executable = None
    sections = None
    enabled_sections = None
-    enforce_encoding = False
+    remove_hi = False
+    remove_tags = False
+    fix_ocr = False
+    fix_common = False
+    reverse_rtl = False
+    colors = ""
    chmod = None
    forced_only = False
    exotic_ext = False
    treat_und_as_first = False
+    subtitle_sub_dir = None, None
    ext_match_strictness = False
-    use_activities = False
+    default_mods = None
+    debug_mods = False
+    react_to_activities = False
    activity_mode = None
+    no_refresh = False
+    plex_transcoder = None
+    refiner_settings = None
+    exact_filenames = False
+    only_one = False
+    embedded_auto_extract = False
+    ietf_as_alpha3 = False
+
+    store_recently_played_amount = 40

    initialized = False

    def initialize(self):
+        self.libraries_root = os.path.abspath(os.path.join(get_root_path(), ".."))
+        self.init_libraries()
+
+        if is_windows_special_path:
+            Log.Warn("The Plex metadata folder is residing inside a folder with special characters. "
+                     "Multithreading and playback activities will be disabled.")
+
        self.fs_encoding = get_viable_encoding()
        self.plugin_info = self.get_plugin_info()
        self.is_development = self.get_dev_mode()
@@ -84,16 +155,28 @@ class Config(object):
        self.full_version = u"%s %s" % (PLUGIN_NAME, self.version)
        self.set_log_paths()
        self.app_support_path = Core.app_support_path
+        self.data_path = getattr(Data, "_core").storage.data_path
+        self.data_items_path = os.path.join(self.data_path, "DataItems")
        self.universal_plex_token = self.get_universal_plex_token()
+        self.plex_token = os.environ.get("PLEXTOKEN", self.universal_plex_token)
+        subzero.constants.DEFAULT_TIMEOUT = lib.DEFAULT_TIMEOUT = self.pms_request_timeout = \
+            min(cast_int(Prefs['pms_request_timeout'], 15), 45)
+        self.low_impact_mode = cast_bool(Prefs['low_impact_mode'])
+        self.new_style_cache = cast_bool(Prefs['new_style_cache'])
+        self.pack_cache_dir = self.get_pack_cache_dir()
+        self.advanced = self.get_advanced_config()

+        os.environ["SZ_USER_AGENT"] = self.get_user_agent()
+
+        self.setup_proxies()
        self.set_plugin_mode()
        self.set_plugin_lock()
        self.set_activity_modes()
+        self.parse_rename_mode()

-        self.lang_list = self.get_lang_list()
        self.subtitle_destination_folder = self.get_subtitle_destination_folder()
-        self.providers = self.get_providers()
-        self.provider_settings = self.get_provider_settings()
+        self.subtitle_formats = self.get_subtitle_formats()
+        self.forced_only = cast_bool(Prefs["subtitles.only_foreign"])
        self.max_recent_items_per_library = int_or_default(Prefs["scheduler.max_recent_items_per_library"], 2000)
        self.sections = list(Plex["library"].sections())
        self.missing_permissions = []
@@ -102,20 +185,122 @@ class Config(object):
        self.enabled_sections = self.check_enabled_sections()
        self.permissions_ok = self.check_permissions()
        self.notify_executable = self.check_notify_executable()
-        self.enforce_encoding = cast_bool(Prefs['subtitles.enforce_encoding'])
+        self.remove_hi = cast_bool(Prefs['subtitles.remove_hi'])
+        self.remove_tags = cast_bool(Prefs['subtitles.remove_tags'])
+        self.fix_ocr = cast_bool(Prefs['subtitles.fix_ocr'])
+        self.fix_common = cast_bool(Prefs['subtitles.fix_common'])
+        self.reverse_rtl = cast_bool(Prefs['subtitles.reverse_rtl'])
+        self.colors = Prefs['subtitles.colors'] if Prefs['subtitles.colors'] != "don't change" else None
        self.chmod = self.check_chmod()
-        self.forced_only = cast_bool(Prefs["subtitles.only_foreign"])
        self.exotic_ext = cast_bool(Prefs["subtitles.scan.exotic_ext"])
        self.treat_und_as_first = cast_bool(Prefs["subtitles.language.treat_und_as_first"])
+        self.subtitle_sub_dir = self.get_subtitle_sub_dir()
        self.ext_match_strictness = self.determine_ext_sub_strictness()
+        self.default_mods = self.get_default_mods()
+        self.debug_mods = cast_bool(Prefs['log_debug_mods'])
+        self.no_refresh = os.environ.get("SZ_NO_REFRESH", False)
+        self.plex_transcoder = self.get_plex_transcoder()
+        self.only_one = cast_bool(Prefs['subtitles.only_one'])
+        self.embedded_auto_extract = cast_bool(Prefs["subtitles.embedded.autoextract"])
+        self.ietf_as_alpha3 = cast_bool(Prefs["subtitles.language.ietf_normalize"])
        self.initialized = True

+    def init_libraries(self):
+        if Core.runtime.os == "Windows":
+            unrar_exe = os.path.abspath(os.path.join(self.libraries_root, "Windows", "i386", "UnRAR", "UnRAR.exe"))
+            if os.path.isfile(unrar_exe):
+                rarfile.UNRAR_TOOL = unrar_exe
+                Log.Info("Using UnRAR from: %s", unrar_exe)
+
+        custom_unrar = os.environ.get("SZ_UNRAR_TOOL")
+        if custom_unrar and os.path.isfile(custom_unrar):
+            rarfile.UNRAR_TOOL = custom_unrar
+            Log.Info("Using UnRAR from: %s", custom_unrar)
+
+    def init_cache(self):
+        if self.new_style_cache:
+            subliminal.region.configure('subzero.cache.file', expiration_time=datetime.timedelta(days=30),
+                                        arguments={'appname': "sz_cache",
+                                                   'app_cache_dir': self.data_path})
+            Log.Info("Using new style file based cache!")
+            return
+
+        names = ['dbhash', 'gdbm', 'dbm']
+        dbfn = None
+        self.dbm_supported = False
+
+        # try importing dbm modules
+        if Core.runtime.os != "Windows":
+            impawrt = None
+            try:
+                impawrt = getattr(sys.modules['__main__'], "__builtins__").get("__import__")
+            except:
+                pass
+
+            if impawrt:
+                for name in names:
+                    try:
+                        impawrt(name)
+                    except:
+                        continue
+                    if not self.dbm_supported:
+                        self.dbm_supported = name
+                        break
+
+                if self.dbm_supported:
+                    # anydbm checks; try guessing the format and importing the correct module
+                    dbfn = os.path.join(config.data_items_path, 'subzero.dbm')
+                    db_which = whichdb(dbfn)
+                    if db_which is not None and db_which != "":
+                        try:
+                            impawrt(db_which)
+                        except ImportError:
+                            self.dbm_supported = False
+
+        if self.dbm_supported:
+            try:
+                subliminal.region.configure('dogpile.cache.dbm', expiration_time=datetime.timedelta(days=30),
+                                            arguments={'filename': dbfn,
+                                                       'lock_factory': MutexLock})
+                Log.Info("Using file based cache!")
+                return
+            except:
+                self.dbm_supported = False
+
+        Log.Warn("Not using file based cache!")
+        subliminal.region.configure('dogpile.cache.memory')
+
+    def sync_cache(self):
+        if not self.new_style_cache:
+            return
+        Log.Debug("Syncing cache")
+        subliminal.region.backend.sync()
+
+    def get_pack_cache_dir(self):
+        pack_cache_dir = os.path.join(config.data_path, "pack_cache")
+        if not os.path.isdir(pack_cache_dir):
+            os.makedirs(pack_cache_dir)
+
+        return pack_cache_dir
+
+    def get_advanced_config(self):
+        path = os.path.join(config.data_path, "advanced_settings.json")
+        if os.path.isfile(path):
+            data = FileIO.read(path, "r")
+
+            return Dicked(**jstyleson.loads(data))
+
+        return Dicked()
+
    def set_log_paths(self):
        # find log handler
        for handler in Core.log.handlers:
-            if getattr(getattr(handler, "__class__"), "__name__") in (
-                    'FileHandler', 'RotatingFileHandler', 'TimedRotatingFileHandler'):
+            cls_name = getattr(getattr(handler, "__class__"), "__name__")
+            if cls_name in ('FileHandler', 'RotatingFileHandler', 'TimedRotatingFileHandler'):
                plugin_log_file = handler.baseFilename
+                if cls_name in ("RotatingFileHandler", "TimedRotatingFileHandler"):
+                    handler.backupCount = int_or_default(Prefs['log_rotate_keep'], 5)
+
                if os.path.isfile(os.path.realpath(plugin_log_file)):
                    self.plugin_log_path = plugin_log_file

@@ -134,9 +319,21 @@ class Config(object):
            except:
                Log.Warn("Couldn't determine Plex Token")
        else:
-            Log("Did NOT find Preferences file - please check logfile and hierarchy. Aborting!")
+            Log.Warn("Did NOT find Preferences file - most likely Windows OS. Otherwise please check logfile and hierarchy.")
+
+        # fixme: windows

    def set_plugin_mode(self):
+        self.enable_agent = True
+        self.enable_channel = True
+
+        # any provider enabled?
+        if not self.providers:
+            self.enable_agent = False
+            self.enable_channel = False
+            Log.Warn("No providers enabled, disabling agent and channel!")
+            return
+
        if Prefs["plugin_mode"] == "only agent":
            self.enable_channel = False
        elif Prefs["plugin_mode"] == "only channel":
@@ -175,7 +372,7 @@ class Config(object):
        self.permissions_ok = self.check_permissions()

    def check_permissions(self):
-        if not Prefs["subtitles.save.filesystem"] or not Prefs["check_permissions"]:
+        if not cast_bool(Prefs["subtitles.save.filesystem"]) or not cast_bool(Prefs["check_permissions"]):
            return True

        self.missing_permissions = []
@@ -191,6 +388,9 @@ class Config(object):
                if isinstance(path_str, unicode):
                    path_str = path_str.encode(self.fs_encoding)

+                if not os.path.exists(path_str):
+                    continue
+
                if use_ignore_fs:
                    # check whether we've got an ignore file inside the section path
                    if self.is_physically_ignored(path_str):
@@ -209,11 +409,17 @@ class Config(object):
        return all_permissions_ok

    def get_version(self):
+        return self.get_bare_version() + ("" if not self.is_development else " DEV")
+
+    def get_bare_version(self):
        result = VERSION_RE.search(self.plugin_info)
-        add = "" if not self.is_development else " DEV"

        if result:
-            return result.group(1) + add
+            return result.group(1)
+        return "2.x.x.x"
+
+    def get_user_agent(self):
+        return "Sub-Zero/%s" % (self.get_bare_version() + ("" if not self.is_development else "-dev"))

    def get_dev_mode(self):
        dev = DEV_RE.search(self.plugin_info)
@@ -266,7 +472,7 @@ class Config(object):
        self.enabled_sections = self.check_enabled_sections()

    def check_enabled_sections(self):
-        enabled_for_primary_agents = []
+        enabled_for_primary_agents = {"movie": [], "show": []}
        enabled_sections = {}

        # find which agents we're enabled for
@@ -279,29 +485,55 @@ class Config(object):
                    related_agents = Plex.primary_agent(agent.identifier, t.media_type)
                    for a in related_agents:
                        if a.identifier == PLUGIN_IDENTIFIER and a.enabled:
-                            enabled_for_primary_agents.append(agent.identifier)
+                            enabled_for_primary_agents[MEDIA_TYPE_TO_STRING[t.media_type]].append(agent.identifier)

        # find the libraries that use them
        for library in self.sections:
-            if library.agent in enabled_for_primary_agents:
+            if library.agent in enabled_for_primary_agents.get(library.type, []):
                enabled_sections[library.key] = library

        Log.Debug(u"I'm enabled for: %s" % [lib.title for key, lib in enabled_sections.iteritems()])
        return enabled_sections

    # Prepare a list of languages we want subs for
-    def get_lang_list(self):
-        l = {Language.fromietf(Prefs["langPref1"])}
+    def get_lang_list(self, provider=None):
+        # advanced settings
+        if provider and self.advanced.providers and provider in self.advanced.providers:
+            adv_languages = self.advanced.providers[provider].get("languages", None)
+            if adv_languages:
+                adv_out = set()
+                for adv_lang in adv_languages:
+                    adv_lang = adv_lang.strip()
+                    try:
+                        real_lang = Language.fromietf(adv_lang)
+                    except:
+                        try:
+                            real_lang = Language.fromname(adv_lang)
+                        except:
+                            continue
+                    adv_out.update({real_lang})
+
+                # fallback to default languages if no valid language was found in advanced settings
+                if adv_out:
+                    return adv_out
+
+        l = {Language.fromietf(Prefs["langPref1a"])}
        lang_custom = Prefs["langPrefCustom"].strip()

        if Prefs['subtitles.only_one']:
            return l

-        if Prefs["langPref2"] != "None":
-            l.update({Language.fromietf(Prefs["langPref2"])})
+        if Prefs["langPref2a"] != "None":
+            try:
+                l.update({Language.fromietf(Prefs["langPref2a"])})
+            except:
+                pass

-        if Prefs["langPref3"] != "None":
-            l.update({Language.fromietf(Prefs["langPref3"])})
+        if Prefs["langPref3a"] != "None":
+            try:
+                l.update({Language.fromietf(Prefs["langPref3a"])})
+            except:
+                pass

        if len(lang_custom) and lang_custom != "None":
            for lang in lang_custom.split(u","):
@@ -317,6 +549,8 @@ class Config(object):

        return l

+    lang_list = property(get_lang_list)
+
    def get_subtitle_destination_folder(self):
        if not Prefs["subtitles.save.filesystem"]:
            return
@@ -326,49 +560,139 @@ class Config(object):
        return fld_custom or (
            Prefs["subtitles.save.subFolder"] if Prefs["subtitles.save.subFolder"] != "current folder" else None)

-    def get_providers(self):
+    def get_subtitle_formats(self):
+        formats = Prefs["subtitles.save.formats"]
+        out = []
+        if "SRT" in formats:
+            out.append("srt")
+        if "VTT" in formats:
+            out.append("vtt")
+        return out
+
+    def get_providers(self, media_type="series"):
        providers = {'opensubtitles': cast_bool(Prefs['provider.opensubtitles.enabled']),
                     # 'thesubdb': Prefs['provider.thesubdb.enabled'],
                     'podnapisi': cast_bool(Prefs['provider.podnapisi.enabled']),
+                     'titlovi': cast_bool(Prefs['provider.titlovi.enabled']),
                     'addic7ed': cast_bool(Prefs['provider.addic7ed.enabled']),
                     'tvsubtitles': cast_bool(Prefs['provider.tvsubtitles.enabled']),
                     'legendastv': cast_bool(Prefs['provider.legendastv.enabled']),
                     'napiprojekt': cast_bool(Prefs['provider.napiprojekt.enabled']),
-                     'shooter': cast_bool(Prefs['provider.shooter.enabled']),
-                     'subscenter': cast_bool(Prefs['provider.subscenter.enabled']),
+                     'hosszupuska': cast_bool(Prefs['provider.hosszupuska.enabled']),
+                     'shooter': False,
+                     'subscene': cast_bool(Prefs['provider.subscene.enabled']),
+                     'argenteam': cast_bool(Prefs['provider.argenteam.enabled']),
+                     'subscenter': False,
                     }

+        providers_by_prefs = copy.deepcopy(providers)
+
+        # disable subscene for movies by default
+        if media_type == "movies":
+            providers["subscene"] = False
+
        # ditch non-forced-subtitles-reporting providers
-        if cast_bool(Prefs['subtitles.only_foreign']):
+        if self.forced_only:
            providers["addic7ed"] = False
            providers["tvsubtitles"] = False
            providers["legendastv"] = False
+            providers["napiprojekt"] = False
+            providers["shooter"] = False
+            providers["hosszupuska"] = False
+            providers["titlovi"] = False
+            providers["argenteam"] = False
+
+        # advanced settings
+        if media_type and self.advanced.providers:
+            for provider, data in self.advanced.providers.iteritems():
+                if provider not in providers or not providers_by_prefs[provider]:
+                    continue
+
+                if data["enabled_for"] is not None:
+                    providers[provider] = media_type in data["enabled_for"]
+
+        if "provider_throttle" not in Dict:
+            Dict["provider_throttle"] = {}
+
+        changed = False
+        for provider, enabled in dict(providers).iteritems():
+            reason, until, throttle_desc = Dict["provider_throttle"].get(provider, (None, None, None))
+            if reason:
+                now = datetime.datetime.now()
+                if now < until:
+                    Log.Info("Not using %s until %s, because of: %s", provider,
+                             until.strftime("%y/%m/%d %H:%M"), reason)
+                    providers[provider] = False
+                else:
+                    Log.Info("Using %s again after %s, (disabled because: %s)", provider, throttle_desc, reason)
+                    del Dict["provider_throttle"][provider]
+                    changed = True
+
+        if changed:
+            Dict.Save()

        return filter(lambda prov: providers[prov], providers)

+    providers = property(get_providers)
+
    def get_provider_settings(self):
+        os_use_https = self.advanced.providers.opensubtitles.use_https \
+            if self.advanced.providers.opensubtitles.use_https != None else True
        provider_settings = {'addic7ed': {'username': Prefs['provider.addic7ed.username'],
                                          'password': Prefs['provider.addic7ed.password'],
-                                          'use_random_agents': cast_bool(Prefs['provider.addic7ed.use_random_agents']),
+                                          'use_random_agents': cast_bool(Prefs['provider.addic7ed.use_random_agents1']),
                                          },
                             'opensubtitles': {'username': Prefs['provider.opensubtitles.username'],
                                               'password': Prefs['provider.opensubtitles.password'],
-                                               'use_tag_search': cast_bool(Prefs['provider.opensubtitles.use_tags']),
-                                               'only_foreign': cast_bool(Prefs['subtitles.only_foreign'])
+                                               'use_tag_search': self.exact_filenames,
+                                               'only_foreign': self.forced_only,
+                                               'is_vip': cast_bool(Prefs['provider.opensubtitles.is_vip']),
+                                               'use_ssl': os_use_https,
                                               },
                             'podnapisi': {
-                                 'only_foreign': cast_bool(Prefs['subtitles.only_foreign'])
+                                 'only_foreign': self.forced_only,
                             },
                             'legendastv': {'username': Prefs['provider.legendastv.username'],
                                            'password': Prefs['provider.legendastv.password'],
-                                            },
-                             'subscenter': {'username': Prefs['provider.subscenter.username'],
-                                            'password': Prefs['provider.subscenter.password'],
-                                            },
+                                            }
                             }

        return provider_settings

+    provider_settings = property(get_provider_settings)
+
+    def provider_throttle(self, name, exception):
+        """
+        throttle a provider :name: for X hours based on the :exception: type
+        :param name:
+        :param exception:
+        :return:
+        """
+        cls = getattr(exception, "__class__")
+        cls_name = getattr(cls, "__name__")
+        if cls not in VALID_THROTTLE_EXCEPTIONS:
+            for valid_cls in VALID_THROTTLE_EXCEPTIONS:
+                if isinstance(cls, valid_cls):
+                    cls = valid_cls
+
+        throttle_data = PROVIDER_THROTTLE_MAP.get(name, PROVIDER_THROTTLE_MAP["default"]).get(cls, None) or \
+            PROVIDER_THROTTLE_MAP["default"].get(cls, None)
+
+        if not throttle_data:
+            return
+
+        throttle_delta, throttle_description = throttle_data
+
+        if "provider_throttle" not in Dict:
+            Dict["provider_throttle"] = {}
+
+        throttle_until = datetime.datetime.now() + throttle_delta
+        Dict["provider_throttle"][name] = (cls_name, throttle_until, throttle_description)
+
+        Log.Info("Throttling %s for %s, until %s, because of: %s", name, throttle_description,
+                 throttle_until.strftime("%y/%m/%d %H:%M"), cls_name)
+        Dict.Save()
+
    @property
    def provider_pool(self):
        if cast_bool(Prefs['providers.multithreading']):
@@ -392,6 +716,22 @@ class Config(object):
        if wrong_chmod:
            Log.Warn("Chmod setting ignored, please use only 4-digit integers with leading 0 (e.g.: 775)")

+    def get_subtitle_sub_dir(self):
+        """
+
+        :return: folder, is_absolute
+        """
+        if not cast_bool(Prefs['subtitles.save.filesystem']):
+            return None, None
+
+        if Prefs["subtitles.save.subFolder.Custom"]:
+            return Prefs["subtitles.save.subFolder.Custom"], os.path.isabs(Prefs["subtitles.save.subFolder.Custom"])
+
+        if Prefs["subtitles.save.subFolder"] == "current folder":
+            return ".", False
+
+        return Prefs["subtitles.save.subFolder"], False
+
    def determine_ext_sub_strictness(self):
        val = Prefs["subtitles.scan.filename_strictness"]
        if val == "any":
@@ -400,20 +740,113 @@ class Config(object):
            return "loose"
        return "strict"

+    def get_default_mods(self):
+        mods = []
+        if self.remove_hi:
+            mods.append("remove_HI")
+        if self.remove_tags:
+            mods.append("remove_tags")
+        if self.fix_ocr:
+            mods.append("OCR_fixes")
+        if self.fix_common:
+            mods.append("common")
+        if self.colors:
+            mods.append("color(name=%s)" % self.colors)
+        if self.reverse_rtl:
+            mods.append("reverse_rtl")
+
+        return mods
+
+    def setup_proxies(self):
+        proxy = Prefs["proxy"]
+        if proxy:
+            os.environ["SZ_HTTP_PROXY"] = proxy.strip()
+            Log.Debug("Using HTTP Proxy: %s", proxy)
+
    def set_activity_modes(self):
        val = Prefs["activity.on_playback"]
        if val == "never":
-            self.use_activities = False
+            self.react_to_activities = False
            return

-        self.use_activities = True
+        self.react_to_activities = True
        if val == "current media item":
            self.activity_mode = "refresh"
        elif val == "hybrid: current item or next episode":
            self.activity_mode = "hybrid"
+        elif val == "hybrid-plus: current item and next episode":
+            self.activity_mode = "hybrid-plus"
        else:
            self.activity_mode = "next_episode"

+    def get_plex_transcoder(self):
+        base_path = os.environ.get("PLEX_MEDIA_SERVER_HOME", None)
+        if not base_path:
+            # fall back to bundled plugins path
+            bundle_path = os.environ.get("PLEXBUNDLEDPLUGINSPATH", None)
+            if bundle_path:
+                base_path = os.path.normpath(os.path.join(bundle_path, "..", ".."))
+
+        if sys.platform == "darwin":
+            fn = os.path.join(base_path, "MacOS", "Plex Transcoder")
+        elif mswindows:
+            fn = os.path.join(base_path, "plextranscoder.exe")
+        else:
+            fn = os.path.join(base_path, "Plex Transcoder")
+
+        if os.path.isfile(fn):
+            return fn
+
+        # look inside Resources folder as fallback, as well
+        fn = os.path.join(base_path, "Resources", "Plex Transcoder")
+        if os.path.isfile(fn):
+            return fn
+
+    def parse_rename_mode(self):
+        # fixme: exact_filenames should be determined via callback combined with info about the current video
+        # (original_name)
+
+        mode = str(Prefs["media_rename1"])
+        self.refiner_settings = {}
+
+        if cast_bool(Prefs['use_file_info_file']):
+            self.refiner_settings["file_info_file"] = True
+            self.exact_filenames = True
+
+        if mode == "none of the above":
+            return
+
+        elif mode == "Symlink to original file":
+            self.refiner_settings["symlinks"] = True
+            self.exact_filenames = True
+            return
+
+        elif mode == "I keep the original filenames":
+            self.exact_filenames = True
+            return
+
+        if mode in ("Filebot", "Sonarr/Radarr/Filebot"):
+            self.refiner_settings["filebot"] = True
+
+        if mode in ("Sonarr/Radarr (fill api info below)", "Sonarr/Radarr/Filebot"):
+            if Prefs["drone_api.sonarr.url"] and Prefs["drone_api.sonarr.api_key"]:
+                self.refiner_settings["sonarr"] = {
+                    "base_url": Prefs["drone_api.sonarr.url"],
+                    "api_key": Prefs["drone_api.sonarr.api_key"]
+                }
+                self.exact_filenames = True
+
+            if Prefs["drone_api.radarr.url"] and Prefs["drone_api.radarr.api_key"]:
+                self.refiner_settings["radarr"] = {
+                    "base_url": Prefs["drone_api.radarr.url"],
+                    "api_key": Prefs["drone_api.radarr.api_key"]
+                }
+                self.exact_filenames = True
+
+    @property
+    def text_based_formats(self):
+        return self.advanced.text_subtitle_formats or TEXT_SUBTITLE_EXTS
+
    def init_subliminal_patches(self):
        # configure custom subtitle destination folders for scanning pre-existing subs
        Log.Debug("Patching subliminal ...")
@@ -422,7 +855,7 @@ class Config(object):
        subliminal_patch.core.INCLUDE_EXOTIC_SUBS = self.exotic_ext

        subliminal_patch.core.DOWNLOAD_TRIES = int(Prefs['subtitles.try_downloads'])
-        subliminal.score.episode_scores["addic7ed_boost"] = int(Prefs['provider.addic7ed.boost_by1'])
+        subliminal.score.episode_scores["addic7ed_boost"] = int(Prefs['provider.addic7ed.boost_by2'])


 config = Config()
@@ -1,4 +1,5 @@
 # coding=utf-8
+import traceback


 def dispatch_migrate():
@@ -6,6 +7,8 @@ def dispatch_migrate():
        migrate()
    except:
        Log.Error("Migration failed: %s" % traceback.format_exc())
+        del Dict["subs"]
+        Dict.Save()


 def migrate():
@@ -25,6 +28,7 @@ def migrate():
                        time=item.time)

        del Dict["history"]
+        history.destroy()
        Dict.Save()

    # migrate subtitle storage from Dict to Data
@@ -80,5 +84,6 @@ def migrate():
            if stored_any:
                subtitle_storage.save(stored_subs)

+        subtitle_storage.destroy()
        del Dict["subs"]
        Dict.Save()
@@ -0,0 +1,120 @@
+# coding=utf-8
+import os
+
+from subzero.language import Language
+
+import subliminal_patch as subliminal
+
+from support.config import config
+from support.helpers import cast_bool
+from subtitlehelpers import get_subtitles_from_metadata
+from subliminal_patch import compute_score
+from support.plex_media import get_blacklist_from_part_map
+from subzero.video import refine_video
+from support.storage import get_pack_data, store_pack_data
+
+
+def get_missing_languages(video, part):
+    languages = set([Language.fromietf(str(l)) for l in config.lang_list])
+
+    # should we treat IETF as alpha3? (ditch the country part)
+    alpha3_map = {}
+    if config.ietf_as_alpha3:
+        for language in languages:
+            if language.country:
+                alpha3_map[language.alpha3] = language.country
+                language.country = None
+
+    if not Prefs['subtitles.save.filesystem']:
+        # scan for existing metadata subtitles
+        meta_subs = get_subtitles_from_metadata(part)
+        for language, subList in meta_subs.iteritems():
+            if subList:
+                video.subtitle_languages.add(language)
+                Log.Debug("Found metadata subtitle %s for %s", language, video)
+
+    have_languages = video.subtitle_languages.copy()
+    if config.ietf_as_alpha3:
+        for language in have_languages:
+            if language.country:
+                alpha3_map[language.alpha3] = language.country
+                language.country = None
+
+    missing_languages = (set(str(l) for l in languages) - set(str(l) for l in have_languages))
+
+    # all languages are found if we either really have subs for all languages or we only want to have exactly one language
+    # and we've only found one (the case for a selected language, Prefs['subtitles.only_one'] (one found sub matches any language))
+    found_one_which_is_enough = len(video.subtitle_languages) >= 1 and Prefs['subtitles.only_one']
+    if not missing_languages or found_one_which_is_enough:
+        if found_one_which_is_enough:
+            Log.Debug('Only one language was requested, and we\'ve got a subtitle for %s', video)
+        else:
+            Log.Debug('All languages %r exist for %s', languages, video)
+        return False
+
+    # re-add country codes to the missing languages, in case we've removed them above
+    if config.ietf_as_alpha3:
+        for language in languages:
+            language.country = alpha3_map.get(language.alpha3, None)
+
+    return missing_languages
+
+
+def pre_download_hook(subtitle):
+    if subtitle.is_pack:
+        # try retrieving the subtitle from a cached pack archive
+        pack_data = get_pack_data(subtitle)
+        if pack_data:
+            subtitle.pack_data = pack_data
+
+
+def post_download_hook(subtitle):
+    # if a new pack was downloaded, store it in the cache; providers' download method is responsible for
+    # setting subtitle.pack_data to None in case the cached pack data we provided was successfully used
+    if subtitle.is_pack and subtitle.pack_data:
+        # store pack data in cache
+        store_pack_data(subtitle, subtitle.pack_data)
+
+    # may be redundant
+    subtitle.pack_data = None
+
+
+def language_hook(provider):
+    return config.get_lang_list(provider=provider)
+
+
+def download_best_subtitles(video_part_map, min_score=0, throttle_time=None, providers=None):
+    hearing_impaired = Prefs['subtitles.search.hearingImpaired']
+    languages = set([Language.fromietf(str(l)) for l in config.lang_list])
+    if not languages:
+        return
+
+    use_videos = []
+    for video, part in video_part_map.iteritems():
+        if not video.ignore_all:
+            missing_languages = get_missing_languages(video, part)
+        else:
+            missing_languages = languages
+
+        if missing_languages:
+            Log.Info(u"%s has missing languages: %s", os.path.basename(video.name), missing_languages)
+            refine_video(video, refiner_settings=config.refiner_settings)
+            use_videos.append(video)
+
+    # prepare blacklist
+    blacklist = get_blacklist_from_part_map(video_part_map, languages)
+
+    if use_videos:
+        Log.Debug("Download best subtitles using settings: min_score: %s, hearing_impaired: %s, languages: %s" %
+                  (min_score, hearing_impaired, languages))
+
+        return subliminal.download_best_subtitles(set(use_videos), languages, min_score, hearing_impaired,
+                                                  providers=providers or config.providers,
+                                                  provider_configs=config.provider_settings,
+                                                  pool_class=config.provider_pool,
+                                                  compute_score=compute_score, throttle_time=throttle_time,
+                                                  blacklist=blacklist, throttle_callback=config.provider_throttle,
+                                                  pre_download_hook=pre_download_hook,
+                                                  post_download_hook=post_download_hook,
+                                                  language_hook=language_hook)
+    Log.Debug("All languages for all requested videos exist. Doing nothing.")
@@ -9,15 +9,26 @@ import time
 import re
 import platform
 import subprocess
-
-from bs4 import UnicodeDammit
+import sys
+from collections import OrderedDict

 import chardet

-from babelfish import Language
-
+from bs4 import UnicodeDammit
+from subzero.language import Language
 from subzero.analytics import track_event

+mswindows = (sys.platform == "win32")
+if mswindows:
+    from subprocess import list2cmdline
+    quote_args = list2cmdline
+else:
+    # POSIX
+    from pipes import quote
+
+    def quote_args(seq):
+        return ' '.join(quote(arg) for arg in seq)
+
 # Unicode control characters can appear in ID3v2 tags but are not legal in XML.
 RE_UNICODE_CONTROL = u'([\u0000-\u0008\u000b-\u000c\u000e-\u001f\ufffe-\uffff])' + \
                     u'|' + \
@@ -30,7 +41,14 @@ RE_UNICODE_CONTROL = u'([\u0000-\u0008\u000b-\u000c\u000e-\u001f\ufffe-\uffff])'


 def cast_bool(value):
-    return str(value) in ("true", "True")
+    return str(value).strip() in ("true", "True")
+
+
+def cast_int(value, default=None):
+    try:
+        return int(value)
+    except ValueError:
+        return default


 # A platform independent way to split paths which might come in with different separators.
@@ -110,9 +128,9 @@ def str_pad(s, length, align='left', pad_char=' ', trim=False):
        raise ValueError("Unknown align type, expected either 'left' or 'right'")


-def pad_title(value):
+def pad_title(value, width=49):
    """Pad a title to 30 characters to force the 'details' view."""
-    return str_pad(value, 49, pad_char=' ')
+    return str_pad(value, width, pad_char=' ')


 def get_plex_item_display_title(item, kind, parent=None, parent_title=None, section_title=None,
@@ -140,10 +158,11 @@ def get_video_display_title(kind, title, section_title=None, parent_title=None,
    if add_section_title:
        section_add = ("%s: " % section_title) if section_title else ""

-    if kind == "show" and parent_title:
+    if kind in ("season", "show") and parent_title:
        if season and episode:
            return '%s%s S%02dE%02d%s' % (section_add, parent_title, season or 0, episode or 0,
                                          (", %s" % title if title else ""))
+
        return '%s%s%s' % (section_add, parent_title, (", %s" % title if title else ""))
    return "%s%s" % (section_add, title)

@@ -191,7 +210,7 @@ def decode_message(s):


 def timestamp():
-    return int(time.time())
+    return int(time.time()*1000)


 def df(d):
@@ -236,13 +255,13 @@ def get_item_hints(data):
    :param data: video item dict of media_to_videos 
    :return: 
    """
-    hints = {"title": data["title"], "type": "movie"}
+    hints = {"title": data["original_title"] or data["title"], "type": "movie"}
    if data["type"] == "episode":
        hints.update(
            {
                "type": "episode",
                "episode_title": data["title"],
-                "title": data["series"],
+                "title": data["original_title"] or data["series"],
            }
        )
    return hints
@@ -256,7 +275,7 @@ def notify_executable(exe_info, videos, subtitles, storage):
    exe, arguments = exe_info
    for video, video_subtitles in subtitles.items():
        for subtitle in video_subtitles:
-            lang = Locale.Language.Match(subtitle.language.alpha2)
+            lang = str(subtitle.language)
            data = video.plexapi_metadata.copy()
            data.update({
                "subtitle_language": lang,
@@ -273,9 +292,23 @@ def notify_executable(exe_info, videos, subtitles, storage):
            prepared_arguments = [arg % prepared_data for arg in arguments]

            Log.Debug(u"Calling %s with arguments: %s" % (exe, prepared_arguments))
+            env = dict(os.environ)
+            if not mswindows:
+                env_path = {"PATH": os.pathsep.join(
+                                        [
+                                            "/usr/local/bin",
+                                            "/usr/bin",
+                                            os.environ.get("PATH", "")
+                                        ]
+                                    )
+                            }
+                env = dict(os.environ, **env_path)
+
+            env.pop("LD_LIBRARY_PATH", None)
+
            try:
-                output = subprocess.check_output(subprocess.list2cmdline([exe] + prepared_arguments),
-                                                 stderr=subprocess.STDOUT, shell=True)
+                output = subprocess.check_output(quote_args([exe] + prepared_arguments),
+                                                 stderr=subprocess.STDOUT, shell=True, env=env)
            except subprocess.CalledProcessError:
                Log.Error(u"Calling %s failed: %s" % (exe, traceback.format_exc()))
            else:
@@ -286,9 +319,32 @@ def track_usage(category=None, action=None, label=None, value=None):
    if not cast_bool(Prefs["track_usage"]):
        return

-    Thread.Create(dispatch_track_usage, category, action, label, value,
-                  identifier=Dict["anon_id"], first_use=Dict["first_use"],
-                  add=Network.PublicAddress)
+    if "last_tracked" not in Dict:
+        Dict["last_tracked"] = OrderedDict()
+        Dict.Save()
+
+    event_key = (category, action, label, value)
+    now = datetime.datetime.now()
+    if event_key in Dict["last_tracked"] and (Dict["last_tracked"][event_key] + datetime.timedelta(minutes=30)) < now:
+        return
+
+    Dict["last_tracked"][event_key] = now
+
+    # maintenance
+    for key, value in Dict["last_tracked"].copy().iteritems():
+        # kill day old values
+        if value < now - datetime.timedelta(days=1):
+            try:
+                del Dict["last_tracked"][key]
+            except:
+                pass
+
+    try:
+        Thread.Create(dispatch_track_usage, category, action, label, value,
+                      identifier=Dict["anon_id"], first_use=Dict["first_use"],
+                      add=Network.PublicAddress)
+    except:
+        Log.Debug("Something went wrong when reporting anonymous user statistics: %s", traceback.format_exc())


 def dispatch_track_usage(*args, **kwargs):
@@ -301,5 +357,36 @@ def dispatch_track_usage(*args, **kwargs):
        Log.Debug("Something went wrong when reporting anonymous user statistics: %s", traceback.format_exc())


+def get_language_from_stream(lang_code):
+    if lang_code:
+        lang = Locale.Language.Match(lang_code)
+        if lang and lang != "xx":
+            # Log.Debug("Found language: %r", lang)
+            return Language.fromietf(lang)
+
+
 def get_language(lang_short):
    return Language.fromietf(lang_short)
+
+
+def display_language(l):
+    addons = []
+    if l.country:
+        addons.append(l.country.alpha2)
+    if l.script:
+        addons.append(l.script.code)
+
+    return l.name if not addons else "%s (%s)" % (l.name, ", ".join(addons))
+
+
+def is_stream_forced(stream):
+    stream_title = getattr(stream, "title", "") or ""
+    forced = getattr(stream, "forced", False)
+    if not forced and stream_title and "forced" in stream_title.strip().lower():
+        forced = True
+
+    return forced
+
+
+class PartUnknownException(Exception):
+    pass
@@ -1,4 +1,4 @@
 # coding=utf-8
 from subzero.history_storage import SubtitleHistory

-get_history = lambda: SubtitleHistory(Data, int(Prefs["history_size"]))
+get_history = lambda: SubtitleHistory(Data, Thread, int(Prefs["history_size"]))
@@ -11,7 +11,8 @@ class IgnoreDict(DictProxy):
        "section": "sections",
        "show": "series",
        "movie": "videos",
-        "episode": "videos"
+        "episode": "videos",
+        "season": "seasons",
    }

    # getItems types mapped to their verbose names
@@ -19,9 +20,10 @@ class IgnoreDict(DictProxy):
        "sections": "Section",
        "series": "Series",
        "videos": "Item",
+        "seasons": "Season",
    }

-    key_order = ("sections", "series", "videos")
+    key_order = ("sections", "series", "videos", "seasons")

    def __len__(self):
        try:
@@ -35,7 +37,7 @@ class IgnoreDict(DictProxy):
        return self.translate_keys.get(name)

    def verbose(self, name):
-        return self.keys_verbose.get(name)
+        return self.keys_verbose.get(self.translate_key(name) or name)

    def get_title_key(self, kind, key):
        return "%s_%s" % (kind, key)
@@ -57,6 +59,7 @@ class IgnoreDict(DictProxy):
        Dict.Save()

    def setup_defaults(self):
-        return {"sections": [], "series": [], "videos": [], "titles": {}}
+        return {"sections": [], "series": [], "videos": [], "titles": {}, "seasons": []}
+

 ignore_list = IgnoreDict(Dict)
@@ -2,12 +2,20 @@

 import logging
 import re
+import traceback
 import types
 import os
+
+import time
+
+import datetime
+
 from ignore import ignore_list
-from helpers import is_recent, get_plex_item_display_title, query_plex
+from helpers import is_recent, get_plex_item_display_title, query_plex, PartUnknownException
 from lib import Plex, get_intent
 from config import config, IGNORE_FN
+from subliminal_patch.subtitle import ModifiedSubtitle
+from subzero.modification import registry as mod_registry, SubtitleModifications

 logger = logging.getLogger(__name__)

@@ -17,12 +25,16 @@ container_size_re = re.compile(ur'totalSize="(\d+)"')


 def get_item(key):
-    item_id = int(key)
+    try:
+        item_id = int(key)
+    except ValueError:
+        return
+
    item_container = Plex["library"].metadata(item_id)

    try:
        return list(item_container)[0]
-    except IndexError:
+    except:
        pass


@@ -40,11 +52,26 @@ PLEX_API_TYPE_MAP = {

 def get_item_kind_from_rating_key(key):
    item = get_item(key)
-    return PLEX_API_TYPE_MAP[get_item_kind(item)]
+    return PLEX_API_TYPE_MAP.get(get_item_kind(item))


 def get_item_kind_from_item(item):
-    return PLEX_API_TYPE_MAP[get_item_kind(item)]
+    return PLEX_API_TYPE_MAP.get(get_item_kind(item))
+
+
+def get_item_title(item):
+    kind = get_item_kind_from_item(item)
+    if kind not in ("episode", "movie", "season", "series"):
+        return
+
+    if kind == "episode":
+        return get_plex_item_display_title(item, "show", parent=item.season, section_title=None,
+                                                 parent_title=item.show.title)
+    elif kind == "season":
+        return get_plex_item_display_title(item, "season", parent=item.show, section_title="Season",
+                                           parent_title=item.show.title)
+    else:
+        return get_plex_item_display_title(item, kind, section_title=None)


 def get_item_thumb(item):
@@ -164,14 +191,17 @@ def get_recent_items():
        "X-Plex-Container-Size": "%s" % config.max_recent_items_per_library
    }

-    episode_re = re.compile(ur'ratingKey="(?P<key>\d+)"'
+    episode_re = re.compile(ur'(?su)ratingKey="(?P<key>\d+)"'
                            ur'.+?grandparentRatingKey="(?P<parent_key>\d+)"'
                            ur'.+?title="(?P<title>.*?)"'
                            ur'.+?grandparentTitle="(?P<parent_title>.*?)"'
                            ur'.+?index="(?P<episode>\d+?)"'
-                            ur'.+?parentIndex="(?P<season>\d+?)".+?addedAt="(?P<added>\d+)"')
-    movie_re = re.compile(ur'ratingKey="(?P<key>\d+)".+?title="(?P<title>.*?)".+?addedAt="(?P<added>\d+)"')
-    available_keys = ("key", "title", "parent_key", "parent_title", "season", "episode", "added")
+                            ur'.+?parentIndex="(?P<season>\d+?)".+?addedAt="(?P<added>\d+)"'
+                            ur'.+?<Part.+? file="(?P<filename>[^"]+?)"')
+    movie_re = re.compile(ur'(?su)ratingKey="(?P<key>\d+)".+?title="(?P<title>.*?)'
+                          ur'".+?addedAt="(?P<added>\d+)"'
+                          ur'.+?<Part.+? file="(?P<filename>[^"]+?)"')
+    available_keys = ("key", "title", "parent_key", "parent_title", "season", "episode", "added", "filename")
    recent = []

    for section in Plex["library"].sections():
@@ -182,8 +212,10 @@ def get_recent_items():
            continue

        use_args = args.copy()
+        plex_item_type = "Movie"
        if section.type == "show":
            use_args["type"] = "4"
+            plex_item_type = "Episode"

        url = "http://127.0.0.1:32400/library/sections/%s/all" % int(section.key)
        response = query_plex(url, use_args)
@@ -198,6 +230,10 @@ def get_recent_items():
            if data["key"] in ignore_list.videos:
                Log.Debug(u"Skipping item: %s" % data["title"])
                continue
+            if is_physically_ignored(data["filename"], plex_item_type):
+                Log.Debug(u"Skipping item: %s" % data["title"])
+                continue
+
            if is_recent(int(data["added"])):
                recent.append((int(data["added"]), section.type, section.title, data["key"]))

@@ -224,7 +260,7 @@ def is_ignored(rating_key, item=None):
    :return:
    """
    # item in soft ignore list
-    if rating_key in ignore_list["videos"]:
+    if ignore_list["videos"] and rating_key in ignore_list["videos"]:
        Log.Debug("Item %s is in the soft ignore list" % rating_key)
        return True

@@ -232,16 +268,31 @@ def is_ignored(rating_key, item=None):
    kind = get_item_kind(item)

    # show in soft ignore list
-    if kind == "Episode" and item.show.rating_key in ignore_list["series"]:
+    if kind == "Episode" and ignore_list["series"] and item.show.rating_key in ignore_list["series"]:
        Log.Debug("Item %s's show is in the soft ignore list" % rating_key)
        return True

+    # season in soft ignore list
+    if kind == "Episode" and ignore_list["seasons"] and item.season.rating_key in ignore_list["seasons"]:
+        Log.Debug("Item %s's season is in the soft ignore list" % rating_key)
+        return True
+
    # section in soft ignore list
-    if item.section.key in ignore_list["sections"]:
+    if ignore_list["sections"] and item.section.key in ignore_list["sections"]:
        Log.Debug("Item %s's section is in the soft ignore list" % rating_key)
        return True

    # physical/path ignore
+    if config.ignore_sz_files or config.ignore_paths:
+        for media in item.media:
+            for part in media.parts:
+                if is_physically_ignored(part.file, kind):
+                    return True
+
+    return False
+
+
+def is_physically_ignored(fn, kind):
    if config.ignore_sz_files or config.ignore_paths:
        # normally check current item folder and the library
        check_ignore_paths = [".", "../"]
@@ -249,18 +300,15 @@ def is_ignored(rating_key, item=None):
            # series/episode, we've got a season folder here, also
            check_ignore_paths.append("../../")

-        for part in item.media.parts:
-            if config.ignore_paths and config.is_path_ignored(part.file):
-                Log.Debug("Item %s's path is manually ignored" % rating_key)
-                return True
+        if config.ignore_paths and config.is_path_ignored(fn):
+            Log.Debug("Item %s's path is manually ignored" % fn)
+            return True

-            if config.ignore_sz_files:
-                for sub_path in check_ignore_paths:
-                    if config.is_physically_ignored(os.path.abspath(os.path.join(os.path.dirname(part.file), sub_path))):
-                        Log.Debug("An ignore file exists in either the items or its parent folders")
-                        return True
-
-    return False
+        if config.ignore_sz_files:
+            for sub_path in check_ignore_paths:
+                if config.is_physically_ignored(os.path.normpath(os.path.join(os.path.dirname(fn), sub_path))):
+                    Log.Debug("An ignore file exists in either the items or its parent folders")
+                    return True


 def refresh_item(rating_key, force=False, timeout=8000, refresh_kind=None, parent_rating_key=None):
@@ -280,6 +328,131 @@ def refresh_item(rating_key, force=False, timeout=8000, refresh_kind=None, paren
        # season refresh, needs explicit per-episode refresh
        refresh = [item.rating_key for item in list(Plex["library/metadata"].children(int(rating_key)))]

+    multiple = len(refresh) > 1
    for key in refresh:
        Log.Info("%s item %s", "Refreshing" if not force else "Forced-refreshing", key)
        Plex["library/metadata"].refresh(key)
+        if multiple:
+            Thread.Sleep(10.0)
+
+
+def get_current_sub(rating_key, part_id, language, plex_item=None):
+    from support.storage import get_subtitle_storage
+
+    item = plex_item or get_item(rating_key)
+    subtitle_storage = get_subtitle_storage()
+    stored_subs = subtitle_storage.load_or_new(item)
+    current_sub = stored_subs.get_any(part_id, language)
+    return current_sub, stored_subs, subtitle_storage
+
+
+def save_stored_sub(stored_subtitle, rating_key, part_id, language, item_type, plex_item=None, storage=None,
+                    stored_subs=None):
+    """
+    in order for this to work, if the calling supplies stored_subs and storage, it has to trigger its saving and
+    destruction explicitly
+    :param stored_subtitle:
+    :param rating_key:
+    :param part_id:
+    :param language:
+    :param item_type:
+    :param plex_item:
+    :param storage:
+    :param stored_subs:
+    :return:
+    """
+    from support.plex_media import get_plex_metadata
+    from support.scanning import scan_videos
+    from support.storage import save_subtitles, get_subtitle_storage
+
+    plex_item = plex_item or get_item(rating_key)
+
+    stored_subs_was_provided = True
+    if not stored_subs or not storage:
+        storage = get_subtitle_storage()
+        stored_subs = storage.load(plex_item.rating_key)
+        stored_subs_was_provided = False
+
+    if not all([plex_item, stored_subs]):
+        return
+
+    try:
+        metadata = get_plex_metadata(rating_key, part_id, item_type, plex_item=plex_item)
+    except PartUnknownException:
+        return
+
+    scanned_parts = scan_videos([metadata], ignore_all=True, skip_hashing=True)
+    video, plex_part = scanned_parts.items()[0]
+
+    subtitle = ModifiedSubtitle(language, mods=stored_subtitle.mods)
+    subtitle.content = stored_subtitle.content
+    if stored_subtitle.encoding:
+        # thanks plex
+        setattr(subtitle, "_guessed_encoding", stored_subtitle.encoding)
+
+        if stored_subtitle.encoding != "utf-8":
+            subtitle.normalize()
+            stored_subtitle.content = subtitle.content
+            stored_subtitle.encoding = "utf-8"
+            storage.save(stored_subs)
+
+    subtitle.plex_media_fps = plex_part.fps
+    subtitle.page_link = stored_subtitle.id
+    subtitle.language = language
+    subtitle.id = stored_subtitle.id
+
+    try:
+        save_subtitles(scanned_parts, {video: [subtitle]}, mode="m", bare_save=True)
+        Log.Debug("Modified %s subtitle for: %s:%s with: %s", language.name, rating_key, part_id,
+                  ", ".join(stored_subtitle.mods) if stored_subtitle.mods else "none")
+    except:
+        Log.Error("Something went wrong when modifying subtitle: %s", traceback.format_exc())
+
+    if subtitle.storage_path:
+        stored_subtitle.last_mod = datetime.datetime.fromtimestamp(os.path.getmtime(subtitle.storage_path))
+
+    if not stored_subs_was_provided:
+        storage.save(stored_subs)
+        storage.destroy()
+
+
+def set_mods_for_part(rating_key, part_id, language, item_type, mods, mode="add"):
+    plex_item = get_item(rating_key)
+
+    if not plex_item:
+        return
+
+    current_sub, stored_subs, storage = get_current_sub(rating_key, part_id, language, plex_item=plex_item)
+    if mode == "add":
+        for mod in mods:
+            identifier, args = SubtitleModifications.parse_identifier(mod)
+            mod_class = SubtitleModifications.get_mod_class(identifier)
+
+            if identifier not in mod_registry.mods_available:
+                raise NotImplementedError("Mod unknown or not registered")
+
+            # clean exclusive mods
+            if mod_class.exclusive and current_sub.mods:
+                for current_mod in current_sub.mods[:]:
+                    if current_mod.startswith(identifier):
+                        current_sub.mods.remove(current_mod)
+                        Log.Info("Removing superseded mod %s" % current_mod)
+
+            current_sub.add_mod(mod)
+    elif mode == "clear":
+        current_sub.add_mod(None)
+    elif mode == "remove":
+        for mod in mods:
+            current_sub.mods.remove(mod)
+
+    elif mode == "remove_last":
+        if current_sub.mods:
+            current_sub.mods.pop()
+    else:
+        raise NotImplementedError("Wrong mode given")
+
+    save_stored_sub(current_sub, rating_key, part_id, language, item_type, plex_item=plex_item, storage=storage,
+                    stored_subs=stored_subs)
+
+    storage.save(stored_subs)
+    storage.destroy()
@@ -9,29 +9,33 @@ import subtitlehelpers
 from config import config as sz_config


+SECONDARY_TAGS = ['forced', 'normal', 'default', 'embedded', 'embedded-forced', 'custom', 'hi', 'cc', 'sdh']
+
+
 def find_subtitles(part):
    lang_sub_map = {}
    part_filename = helpers.unicodize(part.file)
    part_basename = os.path.splitext(os.path.basename(part_filename))[0]
    use_filesystem = helpers.cast_bool(Prefs["subtitles.save.filesystem"])
-    paths = [os.path.dirname(part_filename)] if use_filesystem else []
+    sub_dir_custom = Prefs["subtitles.save.subFolder.Custom"].strip() \
+        if Prefs["subtitles.save.subFolder.Custom"] else None

-    global_subtitle_folder = None
+    use_sub_subfolder = Prefs["subtitles.save.subFolder"] != "current folder" and not sub_dir_custom
+    sub_subfolder = None
+    paths = [os.path.dirname(part_filename)] if use_filesystem else []

    global_folders = []

    if use_filesystem:
        # Check for local subtitles subdirectory
        sub_dir_base = paths[0]
-
        sub_dir_list = []

-        if Prefs["subtitles.save.subFolder"] != "current folder":
+        if use_sub_subfolder:
            # got selected subfolder
-            sub_dir_list.append(os.path.join(sub_dir_base, Prefs["subtitles.save.subFolder"]))
-
-        sub_dir_custom = Prefs["subtitles.save.subFolder.Custom"].strip() \
-            if Prefs["subtitles.save.subFolder.Custom"] else None
+            sub_subfolder = os.path.join(sub_dir_base, Prefs["subtitles.save.subFolder"])
+            sub_dir_list.append(sub_subfolder)
+            sub_subfolder = os.path.normpath(helpers.unicodize(sub_subfolder))

        if sub_dir_custom:
            # got custom subfolder
@@ -56,7 +60,7 @@ def find_subtitles(part):
            global_folders.append(global_subtitle_folder)

    # normalize all paths
-    paths = [os.path.normpath(os.path.realpath(helpers.unicodize(path))) for path in paths]
+    paths = [os.path.normpath(helpers.unicodize(path)) for path in paths]

    # We start by building a dictionary of files to their absolute paths. We also need to know
    # the number of media files that are actually present, in case the found local media asset
@@ -84,8 +88,12 @@ def find_subtitles(part):
                media_files.append(root)

    # cleanup any leftover subtitle if no associated media file was found
-    if helpers.cast_bool(Prefs["subtitles.autoclean"]):
+    if use_filesystem and helpers.cast_bool(Prefs["subtitles.autoclean"]):
        for path in paths:
+            # only housekeep in sub_subfolder if sub_subfolder is used
+            if use_sub_subfolder and path != sub_subfolder and not sz_config.advanced.thorough_cleaning:
+                continue
+
            # we can't housekeep the global subtitle folders as we don't know about *all* media files
            # in a library; skip them
            skip_path = False
@@ -105,10 +113,10 @@ def find_subtitles(part):
                if os.path.isfile(enc_fn):
                    (root, ext) = os.path.splitext(file_path_listing)
                    # it's a subtitle file
-                    if ext.lower()[1:] in config.SUBTITLE_EXTS:
+                    if ext.lower()[1:] in config.SUBTITLE_EXTS_BASE:
                        # get fn without forced/default/normal tag
                        split_tag = root.rsplit(".", 1)
-                        if len(split_tag) > 1 and split_tag[1].lower() in ['forced', 'normal', 'default']:
+                        if len(split_tag) > 1 and split_tag[1].lower() in SECONDARY_TAGS:
                            root = split_tag[0]

                        # get associated media file name without language
@@ -134,7 +142,7 @@ def find_subtitles(part):
        # get fn without forced/default/normal tag
        split_tag = local_basename.rsplit(".", 1)
        has_additional_tag = False
-        if len(split_tag) > 1 and split_tag[1].lower() in ['forced', 'normal', 'default', 'embedded', 'custom']:
+        if len(split_tag) > 1 and split_tag[1].lower() in SECONDARY_TAGS:
            local_basename = split_tag[0]
            has_additional_tag = True

@@ -158,11 +166,10 @@ def find_subtitles(part):
                continue

        # determine whether to pick up the subtitle based on our match strictness
-        elif not filename_matches_part:
+        if not filename_matches_part:
            if sz_config.ext_match_strictness == "strict" or (
-                    sz_config.ext_match_strictness == "loose" and not filename_contains_part):
-
-                #Log.Debug("%s doesn't match %s, skipping" % (helpers.unicodize(local_filename),
+                            sz_config.ext_match_strictness == "loose" and not filename_contains_part):
+                # Log.Debug("%s doesn't match %s, skipping" % (helpers.unicodize(local_filename),
                #                                             helpers.unicodize(part_basename)))
                continue

@@ -1,15 +1,21 @@
 # coding=utf-8
 import traceback
+import time

-from support.config import config
-from support.helpers import get_plex_item_display_title, cast_bool
+import os
+
+from babelfish import LanguageReverseError
+
+from support.config import config, TEXT_SUBTITLE_EXTS
+from support.helpers import get_plex_item_display_title, cast_bool, get_language_from_stream
 from support.items import get_item
 from support.lib import Plex
+from support.storage import get_subtitle_storage
+from subzero.video import has_external_subtitle
+from subzero.language import Language


 def item_discover_missing_subs(rating_key, kind="show", added_at=None, section_title=None, internal=False, external=True, languages=()):
-    existing_subs = {"internal": [], "external": [], "count": 0}
-
    item_id = int(rating_key)
    item = get_item(rating_key)

@@ -18,36 +24,145 @@ def item_discover_missing_subs(rating_key, kind="show", added_at=None, section_t
    else:
        item_title = get_plex_item_display_title(item, kind, section_title=section_title)

-    video = item.media
+    subtitle_storage = get_subtitle_storage()
+    stored_subs = subtitle_storage.load(rating_key)
+    subtitle_storage.destroy()

-    for part in video.parts:
-        for stream in part.streams:
-            if stream.stream_type == 3:
-                if stream.index:
-                    key = "internal"
-                else:
-                    key = "external"
+    subtitle_target_dir, tdir_is_absolute = config.subtitle_sub_dir

-                existing_subs[key].append(Locale.Language.Match(stream.language_code or ""))
-                existing_subs["count"] = existing_subs["count"] + 1
+    missing = set()
+    languages_set = set([Language.fromietf(str(l)) for l in languages])
+    for media in item.media:
+        existing_subs = {"internal": [], "external": [], "own_external": [], "count": 0}
+        for part in media.parts:

-    missing = languages
-    if existing_subs["count"]:
-        existing_flat = (existing_subs["internal"] if internal else []) + (existing_subs["external"] if external else [])
-        languages_set = set(languages)
-        if languages_set.issubset(existing_flat) or (len(existing_flat) >= 1 and Prefs['subtitles.only_one']):
-            # all subs found
-            Log.Info(u"All subtitles exist for '%s'", item_title)
-            return
+            # did we already download an external subtitle before?
+            if subtitle_target_dir and stored_subs:
+                for language in languages_set:
+                    if has_external_subtitle(part.id, stored_subs, language):
+                        # check the existence of the actual subtitle file

-        missing = languages_set - set(existing_flat)
-        Log.Info(u"Subs still missing for '%s': %s", item_title, missing)
+                        # get media filename without extension
+                        part_basename = os.path.splitext(os.path.basename(part.file))[0]
+
+                        # compute target directory for subtitle
+                        # fixme: move to central location
+                        if tdir_is_absolute:
+                            possible_subtitle_path_base = subtitle_target_dir
+                        else:
+                            possible_subtitle_path_base = os.path.join(os.path.dirname(part.file), subtitle_target_dir)
+
+                        possible_subtitle_path_base = os.path.realpath(possible_subtitle_path_base)
+
+                        # folder actually exists?
+                        if not os.path.isdir(possible_subtitle_path_base):
+                            continue
+
+                        found_any = False
+                        for ext in config.subtitle_formats:
+                            if cast_bool(Prefs['subtitles.only_one']):
+                                possible_subtitle_path = os.path.join(possible_subtitle_path_base,
+                                                                      u"%s.%s" % (part_basename, ext))
+                            else:
+                                possible_subtitle_path = os.path.join(possible_subtitle_path_base,
+                                                                      u"%s.%s.%s" % (part_basename, language, ext))
+
+                            # check for subtitle existence
+                            if os.path.isfile(possible_subtitle_path):
+                                found_any = True
+                                Log.Debug(u"Found: %s", possible_subtitle_path)
+                                break
+
+                        if found_any:
+                            existing_subs["own_external"].append(language)
+                            existing_subs["count"] = existing_subs["count"] + 1
+
+            for stream in part.streams:
+                if stream.stream_type == 3:
+                    if stream.index:
+                        key = "internal"
+                    else:
+                        key = "external"
+
+                    if not config.exotic_ext and stream.codec.lower() not in TEXT_SUBTITLE_EXTS:
+                        continue
+
+                    # treat unknown language as lang1?
+                    if not stream.language_code and config.treat_und_as_first:
+                        lang = Language.fromietf(str(list(config.lang_list)[0]))
+
+                    # we can't parse empty language codes
+                    elif not stream.language_code or not stream.codec:
+                        continue
+
+                    else:
+                        # parse with internal language parser first
+                        try:
+                            lang = get_language_from_stream(stream.language_code)
+                            if not lang:
+                                if config.treat_und_as_first:
+                                    lang = Language.fromietf(str(list(config.lang_list)[0]))
+                                else:
+                                    continue
+
+                        except (ValueError, LanguageReverseError):
+                            continue
+
+                    if lang:
+                        # Log.Debug("Found babelfish language: %r", lang)
+                        existing_subs[key].append(lang)
+                        existing_subs["count"] = existing_subs["count"] + 1
+
+        missing_from_part = set([Language.fromietf(str(l)) for l in languages])
+        if existing_subs["count"]:
+
+            # fixme: this is actually somewhat broken with IETF, as Plex doesn't store the country portion
+            # (pt instead of pt-BR) inside the database. So it might actually download pt-BR if there's a local pt-BR
+            # subtitle but not our own.
+            existing_flat = set((existing_subs["internal"] if internal else [])
+                                + (existing_subs["external"] if external else [])
+                                + existing_subs["own_external"])
+
+            check_languages = set([Language.fromietf(str(l)) for l in languages])
+            alpha3_map = {}
+            if config.ietf_as_alpha3:
+                for language in existing_flat:
+                    if language.country:
+                        alpha3_map[language.alpha3] = language.country
+                        language.country = None
+
+                for language in check_languages:
+                    if language.country:
+                        alpha3_map[language.alpha3] = language.country
+                        language.country = None
+
+            # compare sets of strings, not sets of different Language instances
+            check_languages_str = set(str(l) for l in check_languages)
+            existing_flat_str = set(str(l) for l in existing_flat)
+
+            if check_languages_str.issubset(existing_flat_str) or \
+                    (len(existing_flat) >= 1 and Prefs['subtitles.only_one']):
+                # all subs found
+                #Log.Info(u"All subtitles exist for '%s'", item_title)
+                continue
+
+            missing_from_part = set(Language.fromietf(l) for l in check_languages_str - existing_flat_str)
+            if config.ietf_as_alpha3:
+                for language in missing_from_part:
+                    language.country = alpha3_map.get(language.alpha3, None)
+
+        if missing_from_part:
+            Log.Info(u"Subs still missing for '%s' (%s: %s): %s", item_title, rating_key, media.id,
+                     missing_from_part)
+            missing.update(missing_from_part)

    if missing:
+        # deduplicate
+        missing = set(Language.fromietf(la) for la in set(str(l) for l in missing))
        return added_at, item_id, item_title, item, missing


-def items_get_all_missing_subs(items):
+def items_get_all_missing_subs(items, sleep_after_request=False):
    missing = []
    for added_at, kind, section_title, key in items:
        try:
@@ -56,7 +171,7 @@ def items_get_all_missing_subs(items):
                kind=kind,
                added_at=added_at,
                section_title=section_title,
-                languages=config.lang_list,
+                languages=config.lang_list.copy(),
                internal=cast_bool(Prefs["subtitles.scan.embedded"]),
                external=cast_bool(Prefs["subtitles.scan.external"])
            )
@@ -65,13 +180,13 @@ def items_get_all_missing_subs(items):
                missing.append(state)
        except:
            Log.Error("Something went wrong when getting the state of item %s: %s", key, traceback.format_exc())
+        if sleep_after_request:
+            time.sleep(sleep_after_request)
    return missing


 def refresh_item(item):
-    Plex["library/metadata"].refresh(item)
+    if not config.no_refresh:
+        Plex["library/metadata"].refresh(item)


-def refresh_items(items):
-    for item, title in items:
-        refresh_item(item)
@@ -3,11 +3,9 @@
 import os

 import helpers
-
 from items import get_item
-from lib import get_intent, Plex
-from config import config
-from subzero.video import parse_video
+from lib import Plex
+from support.config import TEXT_SUBTITLE_EXTS, config


 def get_metadata_dict(item, part, add):
@@ -22,6 +20,55 @@ def get_metadata_dict(item, part, add):
    return data


+imdb_guid_identifier = "com.plexapp.agents.imdb://"
+tvdb_guid_identifier = "com.plexapp.agents.thetvdb://"
+
+
+def get_plexapi_stream_info(plex_item, part_id=None):
+    d = {"stream": {}}
+    data = d["stream"]
+
+    # find current part
+    current_part = None
+    current_media = None
+    for media in plex_item.media:
+        for part in media.parts:
+            if not part_id or str(part.id) == part_id:
+                current_part = part
+                current_media = media
+                break
+        if current_part:
+            break
+
+    if not current_part:
+        return d
+
+    data["video_codec"] = current_media.video_codec
+    if current_media.audio_codec:
+        data["audio_codec"] = current_media.audio_codec.upper()
+
+        if data["audio_codec"] == "DCA":
+            data["audio_codec"] = "DTS"
+
+    if current_media.audio_channels == 8:
+        data["audio_channels"] = "7.1"
+
+    elif current_media.audio_channels == 6:
+        data["audio_channels"] = "5.1"
+    else:
+        data["audio_channels"] = "%s.0" % str(current_media.audio_channels)
+
+    # iter streams
+    for stream in current_part.streams:
+        if stream.stream_type == 1:
+            # video stream
+            data["resolution"] = "%s%s" % (current_media.video_resolution,
+                                           "i" if stream.scan_type != "progressive" else "p")
+            break
+
+    return d
+
+
 def media_to_videos(media, kind="series"):
    """
    iterates through media and returns the associated parts (videos)
@@ -31,36 +78,61 @@ def media_to_videos(media, kind="series"):
    """
    videos = []

+    # this is a Show or a Movie object
+    plex_item = get_item(media.id)
+    year = plex_item.year
+    original_title = plex_item.title_original
+
    if kind == "series":
        for season in media.seasons:
            season_object = media.seasons[season]
            for episode in media.seasons[season].episodes:
                ep = media.seasons[season].episodes[episode]

+                tvdb_id = None
+                series_tvdb_id = None
+                if tvdb_guid_identifier in ep.guid:
+                    tvdb_id = ep.guid[len(tvdb_guid_identifier):].split("?")[0]
+                    series_tvdb_id = tvdb_id.split("/")[0]
+
                # get plex item via API for additional metadata
                plex_episode = get_item(ep.id)
+                stream_info = get_plexapi_stream_info(plex_episode)

                for item in media.seasons[season].episodes[episode].items:
                    for part in item.parts:
                        videos.append(
                            get_metadata_dict(plex_episode, part,
-                                              {"plex_part": part, "type": "episode", "title": ep.title,
-                                               "series": media.title, "id": ep.id,
-                                               "series_id": media.id, "season_id": season_object.id,
-                                               "episode": plex_episode.index, "season": plex_episode.season.index,
-                                               "section": plex_episode.section.title
-                                               })
+                                              dict(stream_info, **{"plex_part": part, "type": "episode",
+                                                                    "title": ep.title,
+                                                                    "series": media.title, "id": ep.id, "year": year,
+                                                                    "series_id": media.id,
+                                                                    "season_id": season_object.id,
+                                                                    "imdb_id": None, "series_tvdb_id": series_tvdb_id,
+                                                                    "tvdb_id": tvdb_id,
+                                                                    "original_title": original_title,
+                                                                    "episode": plex_episode.index,
+                                                                    "season": plex_episode.season.index,
+                                                                    "section": plex_episode.section.title
+                                                                    })
+                                              )
                        )
    else:
-        plex_item = get_item(media.id)
+        stream_info = get_plexapi_stream_info(plex_item)
+        imdb_id = None
+        if imdb_guid_identifier in media.guid:
+            imdb_id = media.guid[len(imdb_guid_identifier):].split("?")[0]
        for item in media.items:
            for part in item.parts:
                videos.append(
-                    get_metadata_dict(plex_item, part, {"plex_part": part, "type": "movie",
-                                                        "title": media.title, "id": media.id,
-                                                        "series_id": None,
-                                                        "season_id": None,
-                                                        "section": plex_item.section.title})
+                    get_metadata_dict(plex_item, part, dict(stream_info, **{"plex_part": part, "type": "movie",
+                                                                             "title": media.title, "id": media.id,
+                                                                             "series_id": None, "year": year,
+                                                                             "season_id": None, "imdb_id": imdb_id,
+                                                                             "original_title": original_title,
+                                                                             "series_tvdb_id": None, "tvdb_id": None,
+                                                                             "section": plex_item.section.title})
+                                      )
                )
    return videos

@@ -81,10 +153,9 @@ def get_stream_fps(streams):


 def get_media_item_ids(media, kind="series"):
-    ids = []
-    if kind == "movies":
-        ids.append(media.id)
-    else:
+    # fixme: does this work correctly for full series force-refreshes and its intents?
+    ids = [media.id]
+    if kind == "series":
        for season in media.seasons:
            for episode in media.seasons[season].episodes:
                ids.append(media.seasons[season].episodes[episode].id)
@@ -92,139 +163,145 @@ def get_media_item_ids(media, kind="series"):
    return ids


-def scan_video(plex_part, ignore_all=False, hints=None, rating_key=None):
-    """
-    returnes a subliminal/guessit-refined parsed video
-    :param plex_part: 
-    :param ignore_all: 
-    :param hints: 
-    :param rating_key: 
-    :return: 
-    """
-    embedded_subtitles = not ignore_all and Prefs['subtitles.scan.embedded']
-    external_subtitles = not ignore_all and Prefs['subtitles.scan.external']
+def get_all_parts(plex_item):
+    parts = []
+    for media in plex_item.media:
+        parts += media.parts

-    if ignore_all:
-        Log.Debug("Force refresh intended.")
-
-    Log.Debug("Scanning video: %s, subtitles=%s, embedded_subtitles=%s" % (
-        plex_part.file, external_subtitles, embedded_subtitles))
-
-    known_embedded = []
-    parts = list(Plex["library"].metadata(rating_key))[0].media.parts
-    plexpy_part = None
-    for part in parts:
-        if int(part.id) == int(plex_part.id):
-            plexpy_part = part
-
-    # embedded subtitles
-    if plexpy_part:
-        for stream in plexpy_part.streams:
-            # subtitle stream
-            if stream.stream_type == 3:
-                if (config.forced_only and getattr(stream, "forced")) or \
-                        (not config.forced_only and not getattr(stream, "forced")):
-
-                    # embedded subtitle
-                    if not stream.stream_key:
-                        if config.exotic_ext or stream.codec in ("srt", "ass", "ssa"):
-                            lang_code = stream.language_code
-
-                            # treat unknown language as lang1?
-                            if not lang_code and config.treat_und_as_first:
-                                lang_code = list(config.lang_list)[0].alpha3
-                            known_embedded.append(lang_code)
-    else:
-        Log.Warn("Part %s missing of %s, not able to scan internal streams", plex_part.id, rating_key)
-
-    try:
-        # get basic video info scan (filename)
-        video = parse_video(plex_part.file, hints, external_subtitles=external_subtitles,
-                            embedded_subtitles=embedded_subtitles, known_embedded=known_embedded,
-                            forced_only=config.forced_only, video_fps=plex_part.fps)
-
-        return video
-
-    except ValueError:
-        Log.Warn("File could not be guessed by subliminal: %s" % plex_part.file)
+    return parts


-def scan_videos(videos, kind="series", ignore_all=False):
-    """
-    receives a list of videos containing dictionaries returned by media_to_videos
-    :param videos:
-    :param kind: series or movies
-    :return: dictionary of subliminal.video.scan_video, key=subliminal scanned video, value=plex file part
-    """
-    ret = {}
-    for video in videos:
-        intent = get_intent()
-        force_refresh = intent.get("force", video["id"], video["series_id"], video["season_id"])
-        Log.Debug("Determining force-refresh (video: %s, series: %s, season: %s), result: %s"
-                  % (video["id"], video["series_id"], video["season_id"], force_refresh))
+def get_embedded_subtitle_streams(part, requested_language=None, skip_duplicate_unknown=True, get_forced=None):
+    streams = []
+    has_unknown = False
+    for stream in part.streams:
+        # subtitle stream
+        if stream.stream_type == 3 and not stream.stream_key and stream.codec in TEXT_SUBTITLE_EXTS:
+            language = helpers.get_language_from_stream(stream.language_code)
+            is_unknown = False
+            found_requested_language = requested_language and requested_language == language
+            is_forced = helpers.is_stream_forced(stream)

-        hints = helpers.get_item_hints(video)
-        video["plex_part"].fps = get_stream_fps(video["plex_part"].streams)
-        scanned_video = scan_video(video["plex_part"], ignore_all=force_refresh or ignore_all, hints=hints,
-                                   rating_key=video["id"])
+            if get_forced is not None:
+                if (get_forced and not is_forced) or (not get_forced and is_forced):
+                    continue

-        if not scanned_video:
-            continue
+            if not language and config.treat_und_as_first:
+                # only consider first unknown subtitle stream
+                if has_unknown and skip_duplicate_unknown:
+                    continue

-        scanned_video.id = video["id"]
-        part_metadata = video.copy()
-        del part_metadata["plex_part"]
-        scanned_video.plexapi_metadata = part_metadata
-        ret[scanned_video] = video["plex_part"]
-    return ret
+                language = list(config.lang_list)[0]
+                is_unknown = True
+                has_unknown = True
+
+            if not requested_language or found_requested_language or has_unknown:
+                streams.append({"stream": stream, "is_unknown": is_unknown, "language": language,
+                                "is_forced": is_forced})
+
+                if found_requested_language:
+                    break
+
+    return streams


-class PartUnknownException(Exception):
-    pass
+def get_part(plex_item, part_id):
+    for media in plex_item.media:
+        for part in media.parts:
+            if str(part.id) == str(part_id):
+                return part


-def get_plex_metadata(rating_key, part_id, item_type):
+def get_plex_metadata(rating_key, part_id, item_type, plex_item=None):
    """
    uses the Plex 3rd party API accessor to get metadata information

-    :param rating_key:
+    :param rating_key: movie or episode
    :param part_id:
    :param item_type:
    :return:
    """

-    plex_item = list(Plex["library"].metadata(rating_key))[0]
+    if not plex_item:
+        plex_item = get_item(rating_key)
+
+    if not plex_item:
+        return

    # find current part
-    current_part = None
-    for part in plex_item.media.parts:
-        if str(part.id) == part_id:
-            current_part = part
+    current_part = get_part(plex_item, part_id)

    if not current_part:
-        raise PartUnknownException("Part unknown")
+        raise helpers.PartUnknownException("Part unknown")
+
+    stream_info = get_plexapi_stream_info(plex_item, part_id)

    # get normalized metadata
+    # fixme: duplicated logic of media_to_videos
    if item_type == "episode":
+        show = list(Plex["library"].metadata(plex_item.show.rating_key))[0]
+        year = show.year
+        tvdb_id = None
+        series_tvdb_id = None
+        original_title = show.title_original
+        if tvdb_guid_identifier in plex_item.guid:
+            tvdb_id = plex_item.guid[len(tvdb_guid_identifier):].split("?")[0]
+            series_tvdb_id = tvdb_id.split("/")[0]
        metadata = get_metadata_dict(plex_item, current_part,
-                                     {"plex_part": current_part, "type": "episode", "title": plex_item.title,
-                                      "series": plex_item.show.title, "id": plex_item.rating_key,
-                                      "series_id": plex_item.show.rating_key,
-                                      "season_id": plex_item.season.rating_key,
-                                      "season": plex_item.season.index,
-                                      "episode": plex_item.index
-                                      })
+                                     dict(stream_info,
+                                          **{"plex_part": current_part, "type": "episode", "title": plex_item.title,
+                                             "series": plex_item.show.title, "id": plex_item.rating_key,
+                                             "series_id": plex_item.show.rating_key,
+                                             "season_id": plex_item.season.rating_key,
+                                             "imdb_id": None,
+                                             "year": year,
+                                             "tvdb_id": tvdb_id,
+                                             "series_tvdb_id": series_tvdb_id,
+                                             "original_title": original_title,
+                                             "season": plex_item.season.index,
+                                             "episode": plex_item.index
+                                             })
+                                     )
    else:
-        metadata = get_metadata_dict(plex_item, current_part, {"plex_part": current_part, "type": "movie",
-                                                               "title": plex_item.title, "id": plex_item.rating_key,
-                                                               "series_id": None,
-                                                               "season_id": None,
-                                                               "season": None,
-                                                               "episode": None,
-                                                               "section": plex_item.section.title})
+        imdb_id = None
+        original_title = plex_item.title_original
+        if imdb_guid_identifier in plex_item.guid:
+            imdb_id = plex_item.guid[len(imdb_guid_identifier):].split("?")[0]
+        metadata = get_metadata_dict(plex_item, current_part,
+                                     dict(stream_info, **{"plex_part": current_part, "type": "movie",
+                                                           "title": plex_item.title, "id": plex_item.rating_key,
+                                                           "series_id": None,
+                                                           "season_id": None,
+                                                           "imdb_id": imdb_id,
+                                                           "year": plex_item.year,
+                                                           "tvdb_id": None,
+                                                           "series_tvdb_id": None,
+                                                           "original_title": original_title,
+                                                           "season": None,
+                                                           "episode": None,
+                                                           "section": plex_item.section.title})
+                                     )
    return metadata


+def get_blacklist_from_part_map(video_part_map, languages):
+    from support.storage import get_subtitle_storage
+    subtitle_storage = get_subtitle_storage()
+    blacklist = []
+    for video, part in video_part_map.iteritems():
+        stored_subs = subtitle_storage.load_or_new(video.plexapi_metadata["item"])
+        for language in languages:
+            current_bl, subs = stored_subs.get_blacklist(part.id, language)
+            if not current_bl:
+                continue
+
+            blacklist = blacklist + [(str(a), str(b)) for a, b in current_bl.keys()]
+
+    subtitle_storage.destroy()
+
+    return blacklist
+
+
 class PMSMediaProxy(object):
    """
    Proxy object for getting data from a mediatree items "internally" via the PMS
@@ -257,3 +334,24 @@ class PMSMediaProxy(object):
                break

            m = m.children[0]
+
+    def get_all_parts(self):
+        """
+        walk the mediatree until the given part was found; if no part was given, return the first one
+        :param part_id:
+        :return:
+        """
+        m = self.mediatree
+        parts = []
+        while 1:
+            if m.items:
+                media_item = m.items[0]
+                for part in media_item.parts:
+                    parts.append(part)
+                break
+
+            if not m.children:
+                break
+
+            m = m.children[0]
+        return parts
@@ -0,0 +1,131 @@
+# coding=utf-8
+import traceback
+import helpers
+from babelfish.exceptions import LanguageError
+
+from support.lib import Plex, get_intent
+from support.plex_media import get_stream_fps
+from support.storage import get_subtitle_storage
+from support.config import config, TEXT_SUBTITLE_EXTS
+
+from subzero.video import parse_video, set_existing_languages
+from subzero.language import language_from_stream
+
+
+def scan_video(pms_video_info, ignore_all=False, hints=None, rating_key=None, providers=None, skip_hashing=False):
+    """
+    returnes a subliminal/guessit-refined parsed video
+    :param pms_video_info:
+    :param ignore_all:
+    :param hints:
+    :param rating_key:
+    :return:
+    """
+    embedded_subtitles = not ignore_all and Prefs['subtitles.scan.embedded']
+    external_subtitles = not ignore_all and Prefs['subtitles.scan.external']
+
+    plex_part = pms_video_info["plex_part"]
+
+    if ignore_all:
+        Log.Debug("Force refresh intended.")
+
+    Log.Debug("Scanning video: %s, external_subtitles=%s, embedded_subtitles=%s" % (
+        plex_part.file, external_subtitles, embedded_subtitles))
+
+    known_embedded = []
+    parts = []
+    for media in list(Plex["library"].metadata(rating_key))[0].media:
+        parts += media.parts
+
+    plexpy_part = None
+    for part in parts:
+        if int(part.id) == int(plex_part.id):
+            plexpy_part = part
+
+    # embedded subtitles
+    # fixme: skip the whole scanning process if known_embedded == wanted languages?
+    if plexpy_part:
+        if embedded_subtitles:
+            for stream in plexpy_part.streams:
+                # subtitle stream
+                if stream.stream_type == 3:
+                    is_forced = helpers.is_stream_forced(stream)
+
+                    if (config.forced_only and is_forced) or \
+                            (not config.forced_only and not is_forced):
+
+                        # embedded subtitle
+                        # fixme: tap into external subtitles here instead of scanning for ourselves later?
+                        if stream.codec and getattr(stream, "index", None):
+                            if config.exotic_ext or stream.codec.lower() in config.text_based_formats:
+                                lang = None
+                                try:
+                                    lang = language_from_stream(stream.language_code)
+                                except LanguageError:
+                                    Log.Debug("Couldn't detect embedded subtitle stream language: %s", stream.language_code)
+
+                                # treat unknown language as lang1?
+                                if not lang and config.treat_und_as_first:
+                                    lang = list(config.lang_list)[0]
+
+                                if lang:
+                                    known_embedded.append(lang.alpha3)
+    else:
+        Log.Warn("Part %s missing of %s, not able to scan internal streams", plex_part.id, rating_key)
+
+    Log.Debug("Known embedded: %r", known_embedded)
+
+    subtitle_storage = get_subtitle_storage()
+    stored_subs = subtitle_storage.load(rating_key)
+    subtitle_storage.destroy()
+
+    try:
+        # get basic video info scan (filename)
+        video = parse_video(plex_part.file, hints, skip_hashing=config.low_impact_mode or skip_hashing,
+                            providers=providers)
+
+        if not ignore_all:
+            set_existing_languages(video, pms_video_info, external_subtitles=external_subtitles,
+                                   embedded_subtitles=embedded_subtitles, known_embedded=known_embedded,
+                                   forced_only=config.forced_only, stored_subs=stored_subs, languages=config.lang_list,
+                                   only_one=config.only_one)
+
+        # add video fps info
+        video.fps = plex_part.fps
+        return video
+
+    except ValueError:
+        Log.Warn("File could not be guessed: %s: %s", plex_part.file, traceback.format_exc())
+
+
+def scan_videos(videos, ignore_all=False, providers=None, skip_hashing=False):
+    """
+    receives a list of videos containing dictionaries returned by media_to_videos
+    :param videos:
+    :param kind: series or movies
+    :return: dictionary of subliminal.video.scan_video, key=subliminal scanned video, value=plex file part
+    """
+    ret = {}
+    for video in videos:
+        intent = get_intent()
+        force_refresh = intent.get("force", video["id"], video["series_id"], video["season_id"])
+        Log.Debug("Determining force-refresh (video: %s, series: %s, season: %s), result: %s"
+                  % (video["id"], video["series_id"], video["season_id"], force_refresh))
+
+        hints = helpers.get_item_hints(video)
+        video["plex_part"].fps = get_stream_fps(video["plex_part"].streams)
+        p = providers or config.get_providers(media_type="series" if video["type"] == "episode" else "movies")
+        scanned_video = scan_video(video, ignore_all=force_refresh or ignore_all, hints=hints,
+                                   rating_key=video["id"], providers=p,
+                                   skip_hashing=skip_hashing)
+
+        if not scanned_video:
+            continue
+
+        scanned_video.id = video["id"]
+        part_metadata = video.copy()
+        del part_metadata["plex_part"]
+        scanned_video.plexapi_metadata = part_metadata
+        scanned_video.ignore_all = force_refresh
+        ret[scanned_video] = video["plex_part"]
+    return ret
@@ -4,21 +4,24 @@ import datetime
 import logging
 import traceback

+from config import config

 def parse_frequency(s):
-    if s == "never" or s == None:
+    if s == "never" or s is None:
        return None, None
    kind, num, unit = s.split()
    return int(num), unit


 class DefaultScheduler(object):
-    thread = None
+    queue_thread = None
+    scheduler_thread = None
    running = False
    registry = None

    def __init__(self):
-        self.thread = None
+        self.queue_thread = None
+        self.scheduler_thread = None
        self.running = False
        self.registry = []

@@ -47,6 +50,7 @@ class DefaultScheduler(object):
            if Dict["tasks"]:
                for task_name in Dict["tasks"].keys():
                    if task_name == "queue":
+                        Dict["tasks"][task_name] = []
                        continue

                    Dict["tasks"][task_name]["data"] = {}
@@ -58,6 +62,7 @@ class DefaultScheduler(object):
            raise NotImplementedError("Task missing! %s" % name)

        Dict["tasks"][name]["data"] = {}
+        Dict["tasks"][name]["running"] = False
        Dict.Save()
        Log.Debug("Task data cleared: %s", name)

@@ -68,17 +73,18 @@ class DefaultScheduler(object):
        # discover tasks;
        self.tasks = {}
        for cls in self.registry:
-            task = cls(self)
+            task = cls()
            try:
                task_frequency = Prefs["scheduler.tasks.%s.frequency" % task.name]
            except KeyError:
-                task_frequency = None
+                task_frequency = getattr(task, "frequency", None)

            self.tasks[task.name] = {"task": task, "frequency": parse_frequency(task_frequency)}

    def run(self):
        self.running = True
-        self.thread = Thread.Create(self.worker)
+        self.scheduler_thread = Thread.Create(self.scheduler_worker)
+        self.queue_thread = Thread.Create(self.queue_worker)

    def stop(self):
        self.running = False
@@ -113,6 +119,7 @@ class DefaultScheduler(object):

    def run_task(self, name, *args, **kwargs):
        task = self.tasks[name]["task"]
+
        if task.running:
            Log.Debug("Scheduler: Not running %s, as it's currently running.", name)
            return False
@@ -124,8 +131,12 @@ class DefaultScheduler(object):
        except Exception, e:
            Log.Error("Scheduler: Something went wrong when running %s: %s", name, traceback.format_exc())
        finally:
-            task.post_run(Dict["tasks"][name]["data"])
+            try:
+                task.post_run(Dict["tasks"][name]["data"])
+            except:
+                Log.Error("Scheduler: task.post_run failed for %s: %s", name, traceback.format_exc())
            Dict.Save()
+            config.sync_cache()

    def dispatch_task(self, *args, **kwargs):
        if "queue" not in Dict["tasks"]:
@@ -134,8 +145,12 @@ class DefaultScheduler(object):
        Dict["tasks"]["queue"].append((args, kwargs))

    def signal(self, name, *args, **kwargs):
-        for task_name, info in self.tasks.iteritems():
-            task = info["task"]
+        for task_name in self.tasks.keys():
+            task = self.task(task_name)
+            if not task:
+                Log.Error("Scheduler: Task %s not found (?!)" % task_name)
+                continue
+
            if not task.periodic:
                continue

@@ -153,7 +168,7 @@ class DefaultScheduler(object):
                continue
            Log.Debug("Scheduler: Not sending signal %s to task %s, because: not running", name, task_name)

-    def worker(self):
+    def queue_worker(self):
        Thread.Sleep(10.0)
        while 1:
            if not self.running:
@@ -166,12 +181,25 @@ class DefaultScheduler(object):
                Dict["tasks"]["queue"] = []
                Dict.Save()
                for args, kwargs in queue:
-                    Log.Debug("Dispatching single task: %s, %s", args, kwargs)
+                    Log.Debug("Queue: Dispatching single task: %s, %s", args, kwargs)
                    Thread.Create(self.run_task, True, *args, **kwargs)
+                    Thread.Sleep(5.0)
+
+            Thread.Sleep(1)
+
+    def scheduler_worker(self):
+        Thread.Sleep(10.0)
+        while 1:
+            if not self.running:
+                break

            # scheduled tasks
-            for name, info in self.tasks.iteritems():
+            for name in self.tasks.keys():
                now = datetime.datetime.now()
+                info = self.tasks.get(name)
+                if not info:
+                    Log.Error("Scheduler: Task %s not found (?!)" % name)
+                    continue
                task = info["task"]

                if name not in Dict["tasks"] or not task.periodic:
@@ -184,10 +212,22 @@ class DefaultScheduler(object):
                if not frequency_num:
                    continue

-                if not task.last_run or (task.last_run + datetime.timedelta(**{frequency_key: frequency_num}) <= now):
-                    self.run_task(name)
+                # run legacy SARAM once
+                if name == "SearchAllRecentlyAddedMissing" and ("hasRunLSARAM" not in Dict or not Dict["hasRunLSARAM"]):
+                    task = self.tasks["LegacySearchAllRecentlyAddedMissing"]["task"]
+                    task.last_run = None
+                    name = "LegacySearchAllRecentlyAddedMissing"
+                    Dict["hasRunLSARAM"] = True
+                    Dict.Save()

-            Thread.Sleep(5.0)
+                if not task.last_run or (task.last_run + datetime.timedelta(**{frequency_key: frequency_num}) <= now):
+                    # fixme: scheduled tasks run synchronously. is this the best idea?
+                    Thread.Create(self.run_task, True, name)
+                    #Thread.Sleep(5.0)
+                    #self.run_task(name)
+                    Thread.Sleep(5.0)
+
+            Thread.Sleep(1)


 scheduler = DefaultScheduler()
@@ -4,94 +4,72 @@ import datetime
 import os
 import pprint
 import copy
+import traceback
+import types

-import subliminal
-from items import get_item
+from subliminal_patch.core import save_subtitles as subliminal_save_subtitles
 from subzero.subtitle_storage import StoredSubtitlesManager
+from subzero.lib.io import FileIO

 from subtitlehelpers import force_utf8
 from config import config
 from helpers import notify_executable, get_title_for_video_metadata, cast_bool, force_unicode
 from plex_media import PMSMediaProxy
+from support.items import get_item


-get_subtitle_storage = lambda: StoredSubtitlesManager(Data, get_item)
+def get_subtitle_storage():
+    return StoredSubtitlesManager(Data, Thread, get_item)


-def whack_missing_parts(scanned_video_part_map, existing_parts=None):
-    """
-    cleans out our internal storage's video parts (parts may get updated/deleted/whatever)
-    :param existing_parts: optional list of part ids known
-    :param scanned_video_part_map: videos to check for
-    :return:
-    """
-    # shortcut
-
-    if "subs" not in Dict:
-        return
-
-    if not existing_parts:
-        existing_parts = []
-        for part in scanned_video_part_map.viewvalues():
-            existing_parts.append(str(part.id))
-
-    whacked_parts = False
-    for video in scanned_video_part_map.keys():
-        video_id = str(video.id)
-        if video_id not in Dict["subs"]:
-            continue
-
-        parts = Dict["subs"][video_id].keys()
-
-        for part_id in parts:
-            part_id = str(part_id)
-            if part_id not in existing_parts:
-                Log.Info("Whacking part %s in internal storage of video %s (%s, %s)", part_id, video_id,
-                         repr(existing_parts), repr(parts))
-                del Dict["subs"][video_id][part_id]
-                whacked_parts = True
-
-    if whacked_parts:
-        Dict.Save()
-
-
-def store_subtitle_info(scanned_video_part_map, downloaded_subtitles, storage_type, mode="a"):
+def store_subtitle_info(scanned_video_part_map, downloaded_subtitles, storage_type, mode="a", set_current=True):
    """
    stores information about downloaded subtitles in plex's Dict()
    """
-    existing_parts = []
+    subtitle_storage = get_subtitle_storage()
    for video, video_subtitles in downloaded_subtitles.items():
        part = scanned_video_part_map[video]
        part_id = str(part.id)
        video_id = str(video.id)
        plex_item = get_item(video_id)
+        if not plex_item:
+            Log.Warning("Plex item not found: %s", video_id)
+            continue
+
        metadata = video.plexapi_metadata
        title = get_title_for_video_metadata(metadata)

-        subtitle_storage = get_subtitle_storage()
-        stored_subs = subtitle_storage.load_or_new(plex_item)
+        stored_subs = subtitle_storage.load(video_id)
+        is_new = False
+        if not stored_subs:
+            is_new = True
+            Log.Debug(u"Creating new subtitle storage: %s, %s", video_id, part_id)
+            stored_subs = subtitle_storage.new(plex_item)

-        existing_parts.append(part_id)
-
-        stored_any = False
        for subtitle in video_subtitles:
-            lang = Locale.Language.Match(subtitle.language.alpha2)
-            Log.Debug(u"Adding subtitle to storage: %s, %s, %s" % (video_id, part_id, title))
-            ret_val = stored_subs.add(part_id, lang, subtitle, storage_type, mode=mode)
+            lang = str(subtitle.language)
+            subtitle.normalize()
+            Log.Debug(u"Adding subtitle to storage: %s, %s, %s, %s, %s" % (video_id, part_id, lang, title,
+                                                                           subtitle.guess_encoding()))
+
+            last_mod = None
+            if subtitle.storage_path:
+                last_mod = datetime.datetime.fromtimestamp(os.path.getmtime(subtitle.storage_path))
+
+            ret_val = stored_subs.add(part_id, lang, subtitle, storage_type, mode=mode, last_mod=last_mod,
+                                      set_current=set_current)

            if ret_val:
                Log.Debug("Subtitle stored")
-                stored_any = True

            else:
                Log.Debug("Subtitle already existing in storage")

-        if stored_any:
+        if is_new or video_subtitles:
            Log.Debug("Saving subtitle storage for %s" % video_id)
            subtitle_storage.save(stored_subs)

-    #if existing_parts:
-    #    whack_missing_parts(scanned_video_part_map, existing_parts=existing_parts)
+    subtitle_storage.destroy()


 def reset_storage(key):
@@ -107,44 +85,57 @@ def reset_storage(key):


 def log_storage(key):
+    if not key:
+        Log.Debug(pprint.pformat(getattr(Dict, "_dict")))
    if key in Dict:
        Log.Debug(pprint.pformat(Dict[key]))


-def save_subtitles_to_file(subtitles):
+def get_target_folder(file_path):
+    fld = None
    fld_custom = Prefs["subtitles.save.subFolder.Custom"].strip() \
        if Prefs["subtitles.save.subFolder.Custom"] else None

+    if fld_custom or Prefs["subtitles.save.subFolder"] != "current folder":
+        # specific subFolder requested, create it if it doesn't exist
+        fld_base = os.path.split(file_path)[0]
+        if fld_custom:
+            if fld_custom.startswith("/"):
+                # absolute folder
+                fld = fld_custom
+            else:
+                fld = os.path.join(fld_base, fld_custom)
+        else:
+            fld = os.path.join(fld_base, Prefs["subtitles.save.subFolder"])
+        fld = force_unicode(fld)
+        if not os.path.exists(fld):
+            os.makedirs(fld)
+    return fld
+
+
+def save_subtitles_to_file(subtitles, tags=None, forced_tag=None):
+    forced_tag = forced_tag or config.forced_only
    for video, video_subtitles in subtitles.items():
        if not video_subtitles:
            continue

-        fld = None
-        if fld_custom or Prefs["subtitles.save.subFolder"] != "current folder":
-            # specific subFolder requested, create it if it doesn't exist
-            fld_base = os.path.split(video.name)[0]
-            if fld_custom:
-                if fld_custom.startswith("/"):
-                    # absolute folder
-                    fld = fld_custom
-                else:
-                    fld = os.path.join(fld_base, fld_custom)
-            else:
-                fld = os.path.join(fld_base, Prefs["subtitles.save.subFolder"])
-            fld = force_unicode(fld)
-            if not os.path.exists(fld):
-                os.makedirs(fld)
-        subliminal.save_subtitles(video, video_subtitles, directory=fld, single=cast_bool(Prefs['subtitles.only_one']),
-                                      encode_with=force_utf8 if config.enforce_encoding else None,
-                                      chmod=config.chmod, forced_tag=config.forced_only, path_decoder=force_unicode)
+        if not isinstance(video, types.StringTypes):
+            file_path = video.name
+        else:
+            file_path = video
+
+        fld = get_target_folder(file_path)
+        subliminal_save_subtitles(file_path, video_subtitles, directory=fld, single=cast_bool(Prefs['subtitles.only_one']),
+                                  chmod=config.chmod, forced_tag=forced_tag, path_decoder=force_unicode,
+                                  debug_mods=config.debug_mods, formats=config.subtitle_formats, tags=tags)
    return True


-def save_subtitles_to_metadata(videos, subtitles):
+def save_subtitles_to_metadata(videos, subtitles, is_forced=False):
    for video, video_subtitles in subtitles.items():
        mediaPart = videos[video]
        for subtitle in video_subtitles:
-            content = force_utf8(subtitle.text) if config.enforce_encoding else subtitle.content
+            content = subtitle.get_modified_content(debug=config.debug_mods)

            if not isinstance(mediaPart, Framework.api.agentkit.MediaPart):
                # we're being handed a Plex.py model instance here, not an internal PMS MediaPart object.
@@ -152,35 +143,102 @@ def save_subtitles_to_metadata(videos, subtitles):
                mp = PMSMediaProxy(video.id).get_part(mediaPart.id)
            else:
                mp = mediaPart
-            mp.subtitles[Locale.Language.Match(subtitle.language.alpha2)][subtitle.id] = Proxy.Media(content, ext="srt")
+            pm = Proxy.Media(content, ext="srt", forced="1" if is_forced else None)
+            lang = Locale.Language.Match(subtitle.language.alpha2)
+            mp.subtitles[lang].validate_keys({})
+            mp.subtitles[lang]["subzero"] = pm
    return True


-def save_subtitles(scanned_video_part_map, downloaded_subtitles, mode="a"):
+def save_subtitles(scanned_video_part_map, downloaded_subtitles, mode="a", bare_save=False, mods=None,
+                   set_current=True, is_forced=False):
+    """
+     
+    :param set_current: save the subtitle as the current one
+    :param scanned_video_part_map:
+    :param downloaded_subtitles: 
+    :param mode: 
+    :param bare_save: don't trigger anything; don't store information
+    :param mods: enabled mods
+    :return: 
+    """
    meta_fallback = False
    save_successful = False
+
+    # big fixme: scanned_video_part_map isn't needed to the current extent. rewrite.
+
+    if mods:
+        for video, video_subtitles in downloaded_subtitles.items():
+            if not video_subtitles:
+                continue
+
+            for subtitle in video_subtitles:
+                Log.Info("Applying mods: %s to %s", mods, subtitle)
+                subtitle.mods = mods
+                subtitle.plex_media_fps = video.fps
+
    storage = "metadata"
-    if Prefs['subtitles.save.filesystem']:
+    save_to_fs = cast_bool(Prefs['subtitles.save.filesystem'])
+    if save_to_fs:
        storage = "filesystem"
-        try:
-            Log.Debug("Using filesystem as subtitle storage")
-            save_subtitles_to_file(downloaded_subtitles)
-        except OSError:
-            if Prefs["subtitles.save.metadata_fallback"]:
-                meta_fallback = True
+
+    if set_current:
+        if save_to_fs:
+            try:
+                Log.Debug("Using filesystem as subtitle storage")
+                save_subtitles_to_file(downloaded_subtitles, forced_tag=is_forced)
+            except OSError:
+                if cast_bool(Prefs["subtitles.save.metadata_fallback"]):
+                    meta_fallback = True
+                    storage = "metadata"
+                else:
+                    raise
            else:
-                raise
-        else:
-            save_successful = True
+                save_successful = True

-    if not Prefs['subtitles.save.filesystem'] or meta_fallback:
-        if meta_fallback:
-            Log.Debug("Using metadata as subtitle storage, because filesystem storage failed")
-        else:
-            Log.Debug("Using metadata as subtitle storage")
-        save_successful = save_subtitles_to_metadata(scanned_video_part_map, downloaded_subtitles)
+        if not save_to_fs or meta_fallback:
+            if meta_fallback:
+                Log.Debug("Using metadata as subtitle storage, because filesystem storage failed")
+            else:
+                Log.Debug("Using metadata as subtitle storage")
+            save_successful = save_subtitles_to_metadata(scanned_video_part_map, downloaded_subtitles,
+                                                         is_forced=is_forced)

-    if save_successful and config.notify_executable:
-        notify_executable(config.notify_executable, scanned_video_part_map, downloaded_subtitles, storage)
+        if not bare_save and save_successful and config.notify_executable:
+            notify_executable(config.notify_executable, scanned_video_part_map, downloaded_subtitles, storage)

-    store_subtitle_info(scanned_video_part_map, downloaded_subtitles, storage, mode=mode)
+    if not bare_save and save_successful or not set_current:
+        store_subtitle_info(scanned_video_part_map, downloaded_subtitles, storage, mode=mode, set_current=set_current)
+
+    return save_successful
+
+
+def get_pack_id(subtitle):
+    return "%s_%s" % (subtitle.provider_name, subtitle.numeric_id)
+
+
+def get_pack_data(subtitle):
+    subtitle_id = get_pack_id(subtitle)
+
+    archive = os.path.join(config.pack_cache_dir, subtitle_id + ".archive")
+    if os.path.isfile(archive):
+        Log.Info("Loading archive from pack cache: %s", subtitle_id)
+        try:
+            data = FileIO.read(archive, 'rb')
+
+            return data
+        except:
+            Log.Error("Couldn't load archive from pack cache: %s: %s", subtitle_id, traceback.format_exc())
+
+
+def store_pack_data(subtitle, data):
+    subtitle_id = get_pack_id(subtitle)
+
+    archive = os.path.join(config.pack_cache_dir, subtitle_id + ".archive")
+
+    Log.Info("Storing archive in pack cache: %s", subtitle_id)
+    try:
+        FileIO.write(archive, data, 'wb')
+
+    except:
+        Log.Error("Couldn't store archive in pack cache: %s: %s", subtitle_id, traceback.format_exc())
@@ -1,9 +1,9 @@
 # coding=utf-8

 import re, os
-import config
 import helpers

+from config import config, SUBTITLE_EXTS, TEXT_SUBTITLE_EXTS
 from bs4 import UnicodeDammit


@@ -86,11 +86,11 @@ class VobSubSubtitleHelper(SubtitleHelper):


 IETF_MATCH = ".+\.([^-.]+)(?:-[A-Za-z]+)?$"
-ENDSWITH_LANGUAGECODE_RE = re.compile("\.([^-.]{2,3})(?:-[A-Za-z]{2})?$")
+ENDSWITH_LANGUAGECODE_RE = re.compile("\.([^-.]{2,3})(?:-[A-Za-z]{2,})?$")


 def match_ietf_language(s):
-    language_match = re.match(".+\.([^\.]+)$" if not helpers.cast_bool(Prefs["subtitles.language.ietf"])
+    language_match = re.match(".+\.([^\.]+)$" if not helpers.cast_bool(Prefs["subtitles.language.ietf_display"])
                              else IETF_MATCH, s)
    if language_match and len(language_match.groups()) == 1:
        language = language_match.groups()[0]
@@ -102,7 +102,7 @@ class DefaultSubtitleHelper(SubtitleHelper):
    @classmethod
    def is_helper_for(cls, filename):
        (file, file_extension) = os.path.splitext(filename)
-        return file_extension.lower()[1:] in config.SUBTITLE_EXTS
+        return file_extension.lower()[1:] in SUBTITLE_EXTS

    def process_subtitles(self, part):

@@ -120,22 +120,29 @@ class DefaultSubtitleHelper(SubtitleHelper):
        forced = ''
        default = ''
        split_tag = file.rsplit('.', 1)
-        if len(split_tag) > 1 and split_tag[1].lower() in ['forced', 'normal', 'default', 'embedded', 'custom']:
+        if len(split_tag) > 1 and split_tag[1].lower() in ['forced', 'normal', 'default', 'embedded', 'embedded-forced',
+                                                           'custom']:
            file = split_tag[0]
+            sub_tag = split_tag[1].lower()
            # don't do anything with 'normal', we don't need it
-            if 'forced' == split_tag[1].lower():
+            if 'forced' in sub_tag:
                forced = '1'
-            if 'default' == split_tag[1].lower():
+            elif 'default' == sub_tag:
                default = '1'

        # Attempt to extract the language from the filename (e.g. Avatar (2009).eng)
-        language = ""
-
-        # IETF support thanks to https://github.com/hpsbranco/LocalMedia.bundle/commit/4fad9aefedece78a1fa96401304351347f644369
-        language = Locale.Language.Match(match_ietf_language(file))
+        # IETF support thanks to
+        # https://github.com/hpsbranco/LocalMedia.bundle/commit/4fad9aefedece78a1fa96401304351347f644369
+        lang_part = match_ietf_language(file)
+        if lang_part != file:
+            language = Locale.Language.Match(lang_part)
+        elif config.only_one:
+            language = Locale.Language.Match(list(config.lang_list)[0].alpha2)
+        else:
+            language = Locale.Language.Match("xx")

        # skip non-SRT if wanted
-        if not helpers.cast_bool(Prefs["subtitles.scan.exotic_ext"]) and ext not in ["srt", "ass", "ssa"]:
+        if not config.exotic_ext and ext not in TEXT_SUBTITLE_EXTS:
            return lang_sub_map

        codec = None
@@ -158,6 +165,7 @@ class DefaultSubtitleHelper(SubtitleHelper):
                Log("An error occurred while attempting to parse the subtitle file, skipping... : " + self.filename)
                return lang_sub_map

+        # fixme: re-add vtt once Plex Inc. fixes this line in LocalMedia.bundle
        if codec is None and ext in ['ass', 'ssa', 'smi', 'srt', 'psb']:
            codec = ext.replace('ass', 'ssa')

@@ -175,26 +183,30 @@ class DefaultSubtitleHelper(SubtitleHelper):

 def get_subtitles_from_metadata(part):
    subs = {}
-    for language in part.subtitles:
-        subs[language] = []
-        for key, proxy in getattr(part.subtitles[language], "_proxies").iteritems():
-            if not proxy or not len(proxy) >= 5:
-                Log.Debug("Can't parse metadata: %s" % repr(proxy))
-                continue
+    if hasattr(part, "subtitles") and part.subtitles:
+        for language in part.subtitles:
+            subs[language] = []
+            for key, proxy in getattr(part.subtitles[language], "_proxies").iteritems():
+                if not proxy or not len(proxy) >= 5:
+                    Log.Debug("Can't parse metadata: %s" % repr(proxy))
+                    continue

-            p_type = proxy[0]
+                p_type = proxy[0]

-            if p_type == "Media":
-                # metadata subtitle
-                Log.Debug(u"Found metadata subtitle: %s, %s" % (language, repr(proxy)))
-                subs[language].append(key)
+                if p_type == "Media":
+                    # metadata subtitle
+                    Log.Debug(u"Found metadata subtitle: %s, %s" % (language, repr(proxy)))
+                    subs[language] = [key]
    return subs


 def force_utf8(content):
    a = UnicodeDammit(content)

-    Log.Debug("detected encoding: %s (None: most likely already successfully decoded)" % a.original_encoding)
+    if a.original_encoding:
+        Log.Debug("detected encoding: %s (None: most likely already successfully decoded)" % a.original_encoding)
+    else:
+        Log.Debug("detected encoding: unicode (already decoded)")

    # easy way out - already utf-8
    if a.original_encoding and a.original_encoding == "utf-8":
@@ -1,23 +1,29 @@
 # coding=utf-8
-
+import glob
+import os
 import datetime
-import time
 import operator
 import traceback
+from urllib2 import URLError

 from subliminal_patch.score import compute_score
 from subliminal_patch.core import download_subtitles
-from subliminal import list_subtitles as list_all_subtitles
-from babelfish import Language
+from subliminal import list_subtitles as list_all_subtitles, region as subliminal_cache_region
+from subzero.language import Language
+from subzero.video import refine_video

 from missing_subtitles import items_get_all_missing_subs, refresh_item
-from background import scheduler
-from storage import save_subtitles, whack_missing_parts, get_subtitle_storage
+from scheduler import scheduler
+from storage import save_subtitles, get_subtitle_storage
 from support.config import config
-from support.items import get_recent_items, is_ignored, get_item
-from support.lib import Plex
-from support.helpers import track_usage, get_title_for_video_metadata, cast_bool
-from support.plex_media import scan_videos, get_plex_metadata, PartUnknownException
+from support.items import get_recent_items, get_item, is_ignored, get_item_title
+from support.helpers import track_usage, get_title_for_video_metadata, cast_bool, PartUnknownException
+from support.plex_media import get_plex_metadata
+from support.scanning import scan_videos
+from download import download_best_subtitles, pre_download_hook, post_download_hook, language_hook
+
+PROVIDER_SLACK = 30
+DL_PROVIDER_SLACK = 30


 class Task(object):
@@ -34,11 +40,10 @@ class Task(object):
    # task ready for being status-displayed?
    ready_for_display = False

-    def __init__(self, scheduler):
+    def __init__(self):
        self.name = self.get_class_name()
        self.ready_for_display = False
        self.time_start = None
-        self.scheduler = scheduler
        self.setup_defaults()

        self.running = False
@@ -80,138 +85,88 @@ class Task(object):
        return

    def run(self):
+        Log.Info(u"Task: running: %s", self.name)
        self.time_start = datetime.datetime.now()

    def post_run(self, data_holder):
        self.running = False
        self.last_run = datetime.datetime.now()
-        if self.time_start:
+        if self.time_start and self.last_run:
            self.last_run_time = self.last_run - self.time_start
        self.time_start = None
-
-
-class SearchAllRecentlyAddedMissing(Task):
-    periodic = True
-    items_done = None
-    items_searching = None
-    items_searching_ids = None
-    items_failed = None
-    percentage = 0
-
-    stall_time = 30
-
-    def __init__(self, scheduler):
-        super(SearchAllRecentlyAddedMissing, self).__init__(scheduler)
-        self.items_done = None
-        self.items_searching = None
-        self.items_searching_ids = None
-        self.items_failed = None
-        self.percentage = 0
-
-    def signal(self, signal_name, *args, **kwargs):
-        handler = getattr(self, "signal_%s" % signal_name)
-        return handler(*args, **kwargs) if handler else None
-
-    def signal_updated_metadata(self, *args, **kwargs):
-        item_id = int(args[0])
-
-        if self.items_searching_ids is not None and item_id in self.items_searching_ids:
-            self.items_done.append(item_id)
-            return True
-
-    def prepare(self, *args, **kwargs):
-        self.items_done = []
-        recent_items = get_recent_items()
-        missing = items_get_all_missing_subs(recent_items)
-        ids = set([id for added_at, id, title, item, missing_languages in missing if not is_ignored(id, item=item)])
-        self.items_searching = missing
-        self.items_searching_ids = ids
-        self.items_failed = []
-        self.percentage = 0
-        self.ready_for_display = True
-
-    def run(self):
-        super(SearchAllRecentlyAddedMissing, self).run()
-        self.running = True
-        missing_count = len(self.items_searching)
-        items_done_count = 0
-
-        for added_at, item_id, title, item, missing_languages in self.items_searching:
-            Log.Debug(u"Task: %s, triggering refresh for %s (%s)", self.name, title, item_id)
-            refresh_item(item_id)
-            search_started = datetime.datetime.now()
-            tries = 1
-            while 1:
-                if item_id in self.items_done:
-                    items_done_count += 1
-                    Log.Debug(u"Task: %s, item %s done", self.name, item_id)
-                    self.percentage = int(items_done_count * 100 / missing_count)
-                    break
-
-                # item considered stalled after self.stall_time seconds passed after last refresh
-                if (datetime.datetime.now() - search_started).total_seconds() > self.stall_time:
-                    if tries > 3:
-                        self.items_failed.append(item_id)
-                        Log.Debug(u"Task: %s, item stalled for %s times: %s, skipping", self.name, tries, item_id)
-                        break
-
-                    Log.Debug(u"Task: %s, item stalled for %s seconds: %s, retrying", self.name, self.stall_time,
-                              item_id)
-                    tries += 1
-                    refresh_item(item_id)
-                    search_started = datetime.datetime.now()
-                    time.sleep(1)
-                time.sleep(0.1)
-            # we can't hammer the PMS, otherwise requests will be stalled
-            time.sleep(1)
-
-        Log.Debug("Task: %s, done. Failed items: %s", self.name, self.items_failed)
-        self.running = False
-
-    def post_run(self, task_data):
-        super(SearchAllRecentlyAddedMissing, self).post_run(task_data)
-        self.ready_for_display = False
-        self.percentage = 0
-        self.items_done = None
-        self.items_failed = None
-        self.items_searching = None
-        self.items_searching_ids = None
+        Log.Info(u"Task: ran: %s", self.name)


 class SubtitleListingMixin(object):
-    def list_subtitles(self, rating_key, item_type, part_id, language):
-        metadata = get_plex_metadata(rating_key, part_id, item_type)
+    def list_subtitles(self, rating_key, item_type, part_id, language, skip_wrong_fps=True, metadata=None,
+                       scanned_parts=None, air_date_cutoff=None):
+
+        if not metadata:
+            metadata = get_plex_metadata(rating_key, part_id, item_type)
+
+        if not metadata:
+            return
+
+        providers = config.get_providers(media_type="series" if item_type == "episode" else "movies")
+        if not scanned_parts:
+            scanned_parts = scan_videos([metadata], ignore_all=True, providers=providers)
+            if not scanned_parts:
+                Log.Error(u"%s: Couldn't list available subtitles for %s", self.name, rating_key)
+                return
+
+        video, plex_part = scanned_parts.items()[0]
+        refine_video(video, refiner_settings=config.refiner_settings)
+
+        if air_date_cutoff is not None and metadata["item"].year and \
+            metadata["item"].year + air_date_cutoff < datetime.date.today().year:
+            Log.Debug("Skipping searching for subtitles: %s, it aired over %s year(s) ago.", rating_key,
+                      air_date_cutoff)
+            return
+
+        config.init_subliminal_patches()
+
+        provider_settings = config.provider_settings
+        if not skip_wrong_fps:
+            provider_settings["opensubtitles"]["skip_wrong_fps"] = False

        if item_type == "episode":
            min_score = 240
+            if video.is_special:
+                min_score = 180
        else:
            min_score = 60

-        scanned_parts = scan_videos([metadata], kind="series" if item_type == "episode" else "movie", ignore_all=True)
-        if not scanned_parts:
-            Log.Error("Couldn't list available subtitles for %s", rating_key)
-            return
+        languages = {Language.fromietf(language)}

-        video, plex_part = scanned_parts.items()[0]
-        config.init_subliminal_patches()
-
-        available_subs = list_all_subtitles(scanned_parts, {Language.fromietf(language)},
-                                            providers=config.providers,
-                                            provider_configs=config.provider_settings,
-                                            pool_class=config.provider_pool)
+        available_subs = list_all_subtitles([video], languages,
+                                            providers=providers,
+                                            provider_configs=provider_settings,
+                                            pool_class=config.provider_pool,
+                                            throttle_callback=config.provider_throttle,
+                                            language_hook=language_hook)

        use_hearing_impaired = Prefs['subtitles.search.hearingImpaired'] in ("prefer", "force HI")

        # sort subtitles by score
        unsorted_subtitles = []
        for s in available_subs[video]:
-            Log.Debug("Starting score computation for %s", s)
+            Log.Debug(u"%s: Starting score computation for %s", self.name, s)
            try:
                matches = s.get_matches(video)
            except AttributeError:
-                Log.Error("Match computation failed for %s: %s", s, traceback.format_exc())
+                Log.Error(u"%s: Match computation failed for %s: %s", self.name, s, traceback.format_exc())
                continue

+            # skip wrong season/episodes
+            if item_type == "episode":
+                can_verify_series = True
+                if not s.hash_verifiable and "hash" in matches:
+                    can_verify_series = False
+
+                if can_verify_series and not {"series", "season", "episode"}.issubset(matches):
+                    Log.Debug(u"%s: Skipping %s, because it doesn't match our series/episode", self.name, s)
+                    continue
+
            unsorted_subtitles.append(
                (s, compute_score(matches, s, video, hearing_impaired=use_hearing_impaired), matches))
        scored_subtitles = sorted(unsorted_subtitles, key=operator.itemgetter(1), reverse=True)
@@ -220,7 +175,7 @@ class SubtitleListingMixin(object):
        for subtitle, score, matches in scored_subtitles:
            # check score
            if score < min_score:
-                Log.Info('Score %d is below min_score (%d)', score, min_score)
+                Log.Info(u'%s: Score %d is below min_score (%d)', self.name, score, min_score)
                continue
            subtitle.score = score
            subtitle.matches = matches
@@ -237,24 +192,39 @@ class DownloadSubtitleMixin(object):
        item_type = subtitle.item_type
        part_id = subtitle.part_id
        metadata = get_plex_metadata(rating_key, part_id, item_type)
-        scanned_parts = scan_videos([metadata], kind="series" if item_type == "episode" else "movie", ignore_all=True)
+        providers = config.get_providers(media_type="series" if item_type == "episode" else "movies")
+        scanned_parts = scan_videos([metadata], ignore_all=True, providers=providers)
        video, plex_part = scanned_parts.items()[0]

+        pre_download_hook(subtitle)
+
        # downloaded_subtitles = {subliminal.Video: [subtitle, subtitle, ...]}
-        download_subtitles([subtitle], providers=config.providers, provider_configs=config.provider_settings,
-                           pool_class=config.provider_pool)
+        download_subtitles([subtitle], providers=providers,
+                           provider_configs=config.provider_settings,
+                           pool_class=config.provider_pool, throttle_callback=config.provider_throttle)
+
+        post_download_hook(subtitle)
+
+        # may be redundant
+        subtitle.pack_data = None
+
        download_successful = False

        if subtitle.content:
            try:
-                whack_missing_parts(scanned_parts)
-                save_subtitles(scanned_parts, {video: [subtitle]}, mode=mode)
-                Log.Debug("Manually downloaded subtitle for: %s", rating_key)
+                save_subtitles(scanned_parts, {video: [subtitle]}, mode=mode, mods=config.default_mods)
+                if mode == "m":
+                    Log.Debug(u"%s: Manually downloaded subtitle for: %s", self.name, rating_key)
+                    track_usage("Subtitle", "manual", "download", 1)
+                elif mode == "b":
+                    Log.Debug(u"%s: Downloaded better subtitle for: %s", self.name, rating_key)
+                    track_usage("Subtitle", "better", "download", 1)
                download_successful = True
                refresh_item(rating_key)
-                track_usage("Subtitle", "manual", "download", 1)
+
            except:
-                Log.Error("Something went wrong when downloading specific subtitle: %s", traceback.format_exc())
+                Log.Error(u"%s: Something went wrong when downloading specific subtitle: %s",
+                          self.name, traceback.format_exc())
            finally:
                set_refresh_menu_state(None)

@@ -266,6 +236,13 @@ class DownloadSubtitleMixin(object):
                    history.add(item_title, video.id, section_title=video.plexapi_metadata["section"],
                                subtitle=subtitle,
                                mode=mode)
+                    history.destroy()
+
+                    # clear missing subtitles menu data
+                    if not scheduler.is_task_running("MissingSubtitles"):
+                        scheduler.clear_task_data("MissingSubtitles")
+        else:
+            set_refresh_menu_state(u"%s: Subtitle download failed (%s)" % (self.name, rating_key))
        return download_successful


@@ -290,15 +267,27 @@ class AvailableSubsForItem(SubtitleListingMixin, Task):
    def run(self):
        super(AvailableSubsForItem, self).run()
        self.running = True
-        track_usage("Subtitle", "manual", "list", 1)
-        self.data = self.list_subtitles(self.rating_key, self.item_type, self.part_id, self.language)
+        try:
+            track_usage("Subtitle", "manual", "list", 1)
+        except:
+            Log.Error("Something went wrong with track_usage: %s", traceback.format_exc())
+
+        Log.Debug("Listing available subtitles for: %s", self.rating_key)
+        subs = self.list_subtitles(self.rating_key, self.item_type, self.part_id, self.language, skip_wrong_fps=False)
+        if not subs:
+            self.data = "found_none"
+            return
+
+        # we can't have nasty unpicklable stuff like ZipFile, BytesIO etc in self.data
+        self.data = [s.make_picklable() for s in subs]

    def post_run(self, task_data):
        super(AvailableSubsForItem, self).post_run(task_data)
-        if self.rating_key not in task_data:
-            task_data[self.rating_key] = {}
-
-        task_data[self.rating_key][self.language] = self.data
+        # clean old data
+        for key in task_data.keys():
+            if key != self.rating_key:
+                del task_data[key]
+        task_data.update({self.rating_key: {self.language: self.data}})


 class DownloadSubtitleForItem(DownloadSubtitleMixin, Task):
@@ -335,11 +324,305 @@ class MissingSubtitles(Task):
        task_data["missing_subtitles"] = self.data


+class SearchAllRecentlyAddedMissing(Task):
+    periodic = True
+
+    items_done = None
+    items_searching = None
+    percentage = 0
+
+    def __init__(self):
+        super(SearchAllRecentlyAddedMissing, self).__init__()
+        self.items_done = None
+        self.items_searching = None
+        self.percentage = 0
+
+    def signal_updated_metadata(self, *args, **kwargs):
+        return True
+
+    def prepare(self):
+        self.items_done = 0
+        self.items_searching = 0
+        self.percentage = 0
+        self.ready_for_display = True
+
+    def run(self):
+        super(SearchAllRecentlyAddedMissing, self).run()
+
+        self.running = True
+        self.prepare()
+
+        from support.history import get_history
+        history = get_history()
+
+        now = datetime.datetime.now()
+        min_score_series = int(Prefs["subtitles.search.minimumTVScore2"].strip())
+        min_score_movies = int(Prefs["subtitles.search.minimumMovieScore2"].strip())
+        series_providers = config.get_providers(media_type="series")
+        movie_providers = config.get_providers(media_type="movies")
+
+        is_recent_str = Prefs["scheduler.item_is_recent_age"]
+        num, ident = is_recent_str.split()
+
+        max_search_days = 0
+        if ident == "days":
+            max_search_days = int(num)
+        elif ident == "weeks":
+            max_search_days = int(num) * 7
+
+        subtitle_storage = get_subtitle_storage()
+        recent_files = subtitle_storage.get_recent_files(age_days=max_search_days)
+
+        self.items_searching = len(recent_files)
+
+        download_count = 0
+        videos_with_downloads = 0
+
+        config.init_subliminal_patches()
+
+        Log.Info(u"%s: Searching for subtitles for %s items", self.name, self.items_searching)
+
+        def skip_item():
+            self.items_searching = self.items_searching - 1
+            self.percentage = int(self.items_done * 100 / self.items_searching)
+
+        # search for subtitles in viable items
+        try:
+            for fn in recent_files:
+                stored_subs = subtitle_storage.load(filename=fn)
+                if not stored_subs:
+                    Log.Debug("Skipping item %s because storage is empty", fn)
+                    skip_item()
+                    continue
+
+                video_id = stored_subs.video_id
+
+                # added_date <= max_search_days?
+                if stored_subs.added_at + datetime.timedelta(days=max_search_days) <= now:
+                    Log.Debug("Skipping item %s because it's too old", video_id)
+                    skip_item()
+                    continue
+
+                if stored_subs.item_type == "episode":
+                    min_score = min_score_series
+                    providers = series_providers
+                else:
+                    min_score = min_score_movies
+                    providers = movie_providers
+
+                parts = []
+                plex_item = get_item(video_id)
+
+                if not plex_item:
+                    Log.Info(u"%s: Item %s unknown, skipping", self.name, video_id)
+                    skip_item()
+                    continue
+
+                if is_ignored(video_id, item=plex_item):
+                    skip_item()
+                    continue
+
+                for media in plex_item.media:
+                    parts += media.parts
+
+                downloads_per_video = 0
+                hit_providers = False
+                for part in parts:
+                    part_id = part.id
+
+                    try:
+                        metadata = get_plex_metadata(video_id, part_id, stored_subs.item_type)
+                    except PartUnknownException:
+                        Log.Info(u"%s: Part %s:%s unknown, skipping", self.name, video_id, part_id)
+                        continue
+
+                    if not metadata:
+                        Log.Info(u"%s: Part %s:%s unknown, skipping", self.name, video_id, part_id)
+                        continue
+
+                    Log.Debug(u"%s: Looking for missing subtitles: %s", self.name, get_item_title(plex_item))
+                    scanned_parts = scan_videos([metadata], providers=providers)
+
+                    downloaded_subtitles = download_best_subtitles(scanned_parts, min_score=min_score,
+                                                                   providers=providers)
+                    hit_providers = downloaded_subtitles is not None
+                    download_successful = False
+
+                    if downloaded_subtitles:
+                        downloaded_any = any(downloaded_subtitles.values())
+                        if not downloaded_any:
+                            continue
+
+                        try:
+                            save_subtitles(scanned_parts, downloaded_subtitles, mode="a", mods=config.default_mods)
+                            Log.Debug(u"%s: Downloaded subtitle for item with missing subs: %s", self.name, video_id)
+                            download_successful = True
+                            refresh_item(video_id)
+                            track_usage("Subtitle", "manual", "download", 1)
+                        except:
+                            Log.Error(u"%s: Something went wrong when downloading specific subtitle: %s", self.name,
+                                      traceback.format_exc())
+                        finally:
+                            scanned_parts = None
+                            try:
+                                item_title = get_title_for_video_metadata(metadata, add_section_title=False)
+                                if download_successful:
+                                    # store item in history
+                                    for video, video_subtitles in downloaded_subtitles.items():
+                                        if not video_subtitles:
+                                            continue
+
+                                        for subtitle in video_subtitles:
+                                            downloads_per_video += 1
+                                            history.add(item_title, video.id, section_title=metadata["section"],
+                                                        subtitle=subtitle,
+                                                        mode="a")
+
+                                    downloaded_subtitles = None
+                            except:
+                                Log.Error(u"%s: DEBUG HIT: %s", self.name, traceback.format_exc())
+
+                        Log.Debug(u"%s: Waiting %s seconds before continuing", self.name, PROVIDER_SLACK)
+                        Thread.Sleep(PROVIDER_SLACK)
+
+                download_count += downloads_per_video
+
+                if downloads_per_video:
+                    videos_with_downloads += 1
+
+                self.items_done = self.items_done + 1
+                self.percentage = int(self.items_done * 100 / self.items_searching)
+
+                stored_subs = None
+
+                if downloads_per_video:
+                    Log.Debug(u"%s: Subtitles have been downloaded, "
+                              u"waiting %s seconds before continuing", self.name, DL_PROVIDER_SLACK)
+                    Thread.Sleep(DL_PROVIDER_SLACK)
+                else:
+                    if hit_providers:
+                        Log.Debug(u"%s: Waiting %s seconds before continuing", self.name, PROVIDER_SLACK)
+                        Thread.Sleep(PROVIDER_SLACK)
+        finally:
+            subtitle_storage.destroy()
+            history.destroy()
+
+        if download_count:
+            Log.Debug(u"%s: done. Missing subtitles found for %s/%s items (%s subs downloaded)", self.name,
+                      videos_with_downloads, self.items_searching, download_count)
+        else:
+            Log.Debug(u"%s: done. No subtitles found for %s items", self.name, self.items_searching)
+
+    def post_run(self, task_data):
+        super(SearchAllRecentlyAddedMissing, self).post_run(task_data)
+        self.ready_for_display = False
+        self.percentage = 0
+        self.items_done = None
+        self.items_searching = None
+
+
+class LegacySearchAllRecentlyAddedMissing(Task):
+    periodic = True
+    frequency = "never"
+    items_done = None
+    items_searching = None
+    items_searching_ids = None
+    items_failed = None
+    percentage = 0
+
+    stall_time = 30
+
+    def __init__(self):
+        super(LegacySearchAllRecentlyAddedMissing, self).__init__()
+        self.items_done = None
+        self.items_searching = None
+        self.items_searching_ids = None
+        self.items_failed = None
+        self.percentage = 0
+
+    def signal(self, signal_name, *args, **kwargs):
+        handler = getattr(self, "signal_%s" % signal_name)
+        return handler(*args, **kwargs) if handler else None
+
+    def signal_updated_metadata(self, *args, **kwargs):
+        item_id = int(args[0])
+
+        if self.items_searching_ids is not None and item_id in self.items_searching_ids:
+            self.items_done.append(item_id)
+            return True
+
+    def prepare(self, *args, **kwargs):
+        self.items_done = []
+        recent_items = get_recent_items()
+        missing = items_get_all_missing_subs(recent_items, sleep_after_request=0.2)
+        ids = set([id for added_at, id, title, item, missing_languages in missing if not is_ignored(id, item=item)])
+        self.items_searching = missing
+        self.items_searching_ids = ids
+        self.items_failed = []
+        self.percentage = 0
+        self.ready_for_display = True
+
+    def run(self):
+        super(LegacySearchAllRecentlyAddedMissing, self).run()
+        self.running = True
+        missing_count = len(self.items_searching)
+        items_done_count = 0
+
+        for added_at, item_id, title, item, missing_languages in self.items_searching:
+            Log.Debug(u"Task: %s, triggering refresh for %s (%s)", self.name, title, item_id)
+            try:
+                refresh_item(item_id)
+            except URLError:
+                # timeout
+                pass
+            search_started = datetime.datetime.now()
+            tries = 1
+            while 1:
+                if item_id in self.items_done:
+                    items_done_count += 1
+                    self.percentage = int(items_done_count * 100 / missing_count)
+                    Log.Debug(u"Task: %s, item %s done (%s%%, %s/%s)", self.name, item_id, self.percentage,
+                              items_done_count, missing_count)
+                    break
+
+                # item considered stalled after self.stall_time seconds passed after last refresh
+                if (datetime.datetime.now() - search_started).total_seconds() > self.stall_time:
+                    if tries > 3:
+                        self.items_failed.append(item_id)
+                        Log.Debug(u"Task: %s, item stalled for %s times: %s, skipping", self.name, tries, item_id)
+                        break
+
+                    Log.Debug(u"Task: %s, item stalled for %s seconds: %s, retrying", self.name, self.stall_time,
+                              item_id)
+                    tries += 1
+                    try:
+                        refresh_item(item_id)
+                    except URLError:
+                        pass
+                    search_started = datetime.datetime.now()
+                    Thread.Sleep(1)
+                Thread.Sleep(0.1)
+            # we can't hammer the PMS, otherwise requests will be stalled
+            Thread.Sleep(5)
+
+        Log.Debug("Task: %s, done (%s%%, %s/%s). Failed items: %s", self.name, self.percentage,
+                  items_done_count, missing_count, self.items_failed)
+
+    def post_run(self, task_data):
+        super(LegacySearchAllRecentlyAddedMissing, self).post_run(task_data)
+        self.ready_for_display = False
+        self.percentage = 0
+        self.items_done = None
+        self.items_failed = None
+        self.items_searching = None
+        self.items_searching_ids = None
+
+
 class FindBetterSubtitles(DownloadSubtitleMixin, SubtitleListingMixin, Task):
    periodic = True

-    # TV: episode, format, series, year, season, video_codec, release_group, hearing_impaired
-    series_cutoff = 355
+    # TV: episode, format, series, year, season, video_codec, release_group, hearing_impaired, resolution
+    series_cutoff = 357

    # movies: format, title, release_group, year, video_codec, resolution, hearing_impaired
    movies_cutoff = 117
@@ -354,103 +637,274 @@ class FindBetterSubtitles(DownloadSubtitleMixin, SubtitleListingMixin, Task):
        try:
            max_search_days = int(Prefs["scheduler.tasks.FindBetterSubtitles.max_days_after_added"].strip())
        except ValueError:
-            Log.Error("Please only put numbers into the FindBetterSubtitles.max_days_after_added setting. Exiting")
+            Log.Error(u"Please only put numbers into the FindBetterSubtitles.max_days_after_added setting. Exiting")
            return
        else:
            if max_search_days > 30:
-                Log.Error("FindBetterSubtitles.max_days_after_added is too big. Max is 30 days.")
+                Log.Error(u"%s: FindBetterSubtitles.max_days_after_added is too big. Max is 30 days.", self.name)
                return

        now = datetime.datetime.now()
+        min_score_series = int(Prefs["subtitles.search.minimumTVScore2"].strip())
+        min_score_movies = int(Prefs["subtitles.search.minimumMovieScore2"].strip())
+        min_score_extracted_series = config.advanced.find_better_as_extracted_tv_score or 352
+        min_score_extracted_movies = config.advanced.find_better_as_extracted_movie_score or 82
+        overwrite_manually_modified = cast_bool(
+            Prefs["scheduler.tasks.FindBetterSubtitles.overwrite_manually_modified"])
+        overwrite_manually_selected = cast_bool(
+            Prefs["scheduler.tasks.FindBetterSubtitles.overwrite_manually_selected"])
+
+        air_date_cutoff_pref = Prefs["scheduler.tasks.FindBetterSubtitles.air_date_cutoff"]
+        if air_date_cutoff_pref == "don't limit":
+            air_date_cutoff = None
+        else:
+            air_date_cutoff = int(air_date_cutoff_pref.split()[0])

        subtitle_storage = get_subtitle_storage()
-        recent_subs = subtitle_storage.load_recent_files(age_days=max_search_days)
+        viable_item_count = 0

-        for fn, stored_subs in recent_subs.iteritems():
-            video_id = stored_subs.video_id
-            cutoff = self.series_cutoff if stored_subs.item_type == "episode" else self.movies_cutoff
+        try:
+            for fn in subtitle_storage.get_recent_files(age_days=max_search_days):
+                stored_subs = subtitle_storage.load(filename=fn)
+                if not stored_subs:
+                    continue

-            # don't search for better subtitles until at least 30 minutes have passed
-            if stored_subs.added_at + datetime.timedelta(minutes=30) > now:
-                Log.Debug("Item %s too new, skipping", video_id)
-                continue
+                video_id = stored_subs.video_id

-            # added_date <= max_search_days?
-            if stored_subs.added_at + datetime.timedelta(days=max_search_days) <= now:
-                continue
+                if stored_subs.item_type == "episode":
+                    cutoff = self.series_cutoff
+                    min_score = min_score_series
+                    min_score_extracted = min_score_extracted_series
+                else:
+                    cutoff = self.movies_cutoff
+                    min_score = min_score_movies
+                    min_score_extracted = min_score_extracted_movies

-            ditch_parts = []
+                # don't search for better subtitles until at least 30 minutes have passed
+                if stored_subs.added_at + datetime.timedelta(minutes=30) > now:
+                    Log.Debug(u"%s: Item %s too new, skipping", self.name, video_id)
+                    continue

-            # look through all stored subtitle data
-            for part_id, languages in stored_subs.parts.iteritems():
-                part_id = str(part_id)
+                # added_date <= max_search_days?
+                if stored_subs.added_at + datetime.timedelta(days=max_search_days) <= now:
+                    continue

-                # all languages
-                for language, current_subs in languages.iteritems():
-                    current_key = current_subs.get("current")
-                    current = current_subs.get(current_key)
+                viable_item_count += 1
+                ditch_parts = []

-                    # currently got subtitle?
-                    if not current:
-                        continue
-                    current_score = current.score
-                    current_mode = current.mode
+                # look through all stored subtitle data
+                for part_id, languages in stored_subs.parts.iteritems():
+                    part_id = str(part_id)

-                    # late cutoff met? skip
-                    if current_score >= cutoff:
-                        Log.Debug(u"Skipping finding better subs, cutoff met (current: %s, cutoff: %s): %s",
-                                  current_score, cutoff, stored_subs.title)
-                        continue
+                    # all languages
+                    for language, current_subs in languages.iteritems():
+                        current_key = current_subs.get("current")
+                        current = current_subs.get(current_key)

-                    # got manual subtitle but don't want to touch those?
-                    if current_mode == "m" and \
-                            not cast_bool(Prefs["scheduler.tasks.FindBetterSubtitles.overwrite_manually_selected"]):
-                        Log.Debug(u"Skipping finding better subs, had manual: %s", stored_subs.title)
-                        continue
+                        # currently got subtitle?
+                        # fixme: check for existence
+                        if not current:
+                            continue
+                        current_score = current.score
+                        current_mode = current.mode

-                    try:
-                        subs = self.list_subtitles(video_id, stored_subs.item_type, part_id, language)
-                    except PartUnknownException:
-                        Log.Info("Part %s unknown/gone; ditching subtitle info", part_id)
-                        ditch_parts.append(part_id)
-                        continue
+                        # late cutoff met? skip
+                        if current_score >= cutoff:
+                            Log.Debug(u"%s: Skipping finding better subs, "
+                                      u"cutoff met (current: %s, cutoff: %s): %s (%s)",
+                                      self.name, current_score, cutoff, stored_subs.title, video_id)
+                            continue

-                    if subs:
-                        # subs are already sorted by score
-                        better_downloaded = False
-                        better_tried_download = 0
-                        for sub in subs:
-                            if sub.score > current_score:
-                                Log.Debug("Better subtitle found for %s, downloading", video_id)
-                                better_tried_download += 1
-                                ret = self.download_subtitle(sub, video_id, mode="b")
-                                if ret:
-                                    better_found += 1
-                                    better_downloaded = True
-                                    break
-                                else:
-                                    Log.Debug("Couldn't download/save subtitle. Continuing to the next one")
-                        if better_tried_download and not better_downloaded:
-                            Log.Debug("Tried downloading better subtitle for %s, but every try failed.", video_id)
+                        # got manual subtitle but don't want to touch those?
+                        if current_mode == "m" and not overwrite_manually_selected:
+                            Log.Debug(u"%s: Skipping finding better subs, "
+                                      u"had manual: %s (%s)", self.name, stored_subs.title, video_id)
+                            continue

-                        elif better_downloaded:
-                            Log.Debug("Better subtitle downloaded for %s", video_id)
+                        # subtitle modifications different from default
+                        if not overwrite_manually_modified and current.mods \
+                                and set(current.mods).difference(set(config.default_mods)):
+                            Log.Debug(u"%s: Skipping finding better subs, it has manual modifications: %s (%s)",
+                                      self.name, stored_subs.title, video_id)
+                            continue

-            if ditch_parts:
-                for part_id in ditch_parts:
-                    try:
-                        del stored_subs.parts[part_id]
-                    except KeyError:
-                        pass
-                subtitle_storage.save(stored_subs)
+                        try:
+                            subs = self.list_subtitles(video_id, stored_subs.item_type, part_id, language,
+                                                       air_date_cutoff=air_date_cutoff)
+                        except PartUnknownException:
+                            Log.Info(u"%s: Part %s unknown/gone; ditching subtitle info", self.name, part_id)
+                            ditch_parts.append(part_id)
+                            continue
+
+                        hit_providers = subs is not None
+
+                        if subs:
+                            # subs are already sorted by score
+                            better_downloaded = False
+                            better_tried_download = 0
+                            better_visited = 0
+                            for sub in subs:
+                                if sub.score > current_score and sub.score > min_score:
+                                    if current.provider_name == "embedded" and sub.score < min_score_extracted:
+                                        Log.Debug(u"%s: Not downloading subtitle for %s, we've got an active extracted "
+                                                  u"embedded sub and the min score %s isn't met (%s).",
+                                                  self.name, video_id, min_score_extracted, sub.score)
+                                        better_visited += 1
+                                        break
+
+                                    Log.Debug(u"%s: Better subtitle found for %s, downloading", self.name, video_id)
+                                    better_tried_download += 1
+                                    ret = self.download_subtitle(sub, video_id, mode="b")
+                                    if ret:
+                                        better_found += 1
+                                        better_downloaded = True
+                                        break
+                                    else:
+                                        Log.Debug(u"%s: Couldn't download/save subtitle. "
+                                                  u"Continuing to the next one", self.name)
+                                        Log.Debug(u"%s: Waiting %s seconds before continuing",
+                                                  self.name, DL_PROVIDER_SLACK)
+                                        Thread.Sleep(DL_PROVIDER_SLACK)
+                                better_visited += 1
+
+                            if better_tried_download and not better_downloaded:
+                                Log.Debug(u"%s: Tried downloading better subtitle for %s, "
+                                          u"but every try failed.", self.name, video_id)
+
+                            elif better_downloaded:
+                                Log.Debug(u"%s: Better subtitle downloaded for %s", self.name, video_id)
+
+                            if better_tried_download or better_downloaded:
+                                Log.Debug(u"%s: Waiting %s seconds before continuing", self.name, DL_PROVIDER_SLACK)
+                                Thread.Sleep(DL_PROVIDER_SLACK)
+
+                            elif better_visited:
+                                Log.Debug(u"%s: Waiting %s seconds before continuing", self.name, PROVIDER_SLACK)
+                                Thread.Sleep(PROVIDER_SLACK)
+
+                            subs = None
+
+                        elif hit_providers:
+                            # hit the providers but didn't try downloading? wait.
+                            Log.Debug(u"%s: Waiting %s seconds before continuing", self.name, PROVIDER_SLACK)
+                            Thread.Sleep(PROVIDER_SLACK)
+
+                if ditch_parts:
+                    for part_id in ditch_parts:
+                        try:
+                            del stored_subs.parts[part_id]
+                        except KeyError:
+                            pass
+                    subtitle_storage.save(stored_subs)
+                    ditch_parts = None
+
+                stored_subs = None
+
+                Thread.Sleep(1)
+        finally:
+            subtitle_storage.destroy()

        if better_found:
-            Log.Debug("Task: %s, done. Better subtitles found for %s items", self.name, better_found)
-        self.running = False
+            Log.Debug(u"%s: done. Better subtitles found for %s/%s items", self.name, better_found,
+                      viable_item_count)
+        else:
+            Log.Debug(u"%s: done. No better subtitles found for %s items", self.name, viable_item_count)


+class SubtitleStorageMaintenance(Task):
+    periodic = True
+    frequency = "every 7 days"
+
+    def run(self):
+        super(SubtitleStorageMaintenance, self).run()
+        self.running = True
+        Log.Info(u"%s: Running subtitle storage maintenance", self.name)
+        storage = get_subtitle_storage()
+        deleted_items = storage.delete_missing(wanted_languages=set(str(l) for l in config.lang_list))
+        if deleted_items:
+            Log.Info(u"%s: Subtitle information for %d non-existant videos have been cleaned up",
+                     self.name, len(deleted_items))
+            Log.Debug(u"%s: Videos: %s", self.name, deleted_items)
+        else:
+            Log.Info(u"%s: Nothing to do", self.name)
+
+        storage.destroy()
+
+
+class MenuHistoryMaintenance(Task):
+    periodic = True
+    frequency = "every 7 days"
+
+    def run(self):
+        super(MenuHistoryMaintenance, self).run()
+        self.running = True
+        Log.Info(u"%s: Running menu history maintenance", self.name)
+        now = datetime.datetime.now()
+        if "menu_history" in Dict:
+            for key, timeout in Dict["menu_history"].copy().items():
+                if now > timeout:
+                    try:
+                        del Dict["menu_history"][key]
+                    except:
+                        pass
+
+
+class MigrateSubtitleStorage(Task):
+    periodic = False
+    frequency = None
+
+    def run(self):
+        super(MigrateSubtitleStorage, self).run()
+        self.running = True
+        Log.Info(u"%s: Running subtitle storage migration", self.name)
+        storage = get_subtitle_storage()
+        for fn in storage.get_all_files():
+            if fn.endswith(".json.gz"):
+                continue
+            Log.Debug(u"%s: Migrating %s", self.name, fn)
+            storage.load(None, fn)
+
+        storage.destroy()
+
+
+class CacheMaintenance(Task):
+    periodic = True
+    frequency = "every 1 days"
+
+    main_cache_validity = 14  # days
+    pack_cache_validity = 4  # days
+
+    def run(self):
+        super(CacheMaintenance, self).run()
+        self.running = True
+        Log.Info(u"%s: Running cache maintenance", self.name)
+        now = datetime.datetime.now()
+
+        def remove_expired(path, expiry):
+            mtime = datetime.datetime.fromtimestamp(os.path.getmtime(path))
+            if mtime + datetime.timedelta(days=expiry) < now:
+                try:
+                    os.remove(path)
+                except (IOError, OSError):
+                    Log.Debug("Couldn't remove cache file: %s", os.path.basename(path))
+
+        # main cache
+        if config.new_style_cache:
+            for fn in subliminal_cache_region.backend.all_filenames:
+                remove_expired(fn, self.main_cache_validity)
+
+        # archive cache
+        for fn in glob.iglob(os.path.join(config.pack_cache_dir, "*.archive")):
+            remove_expired(fn, self.pack_cache_validity)
+
+
+scheduler.register(LegacySearchAllRecentlyAddedMissing)
 scheduler.register(SearchAllRecentlyAddedMissing)
 scheduler.register(AvailableSubsForItem)
 scheduler.register(DownloadSubtitleForItem)
 scheduler.register(MissingSubtitles)
 scheduler.register(FindBetterSubtitles)
+scheduler.register(SubtitleStorageMaintenance)
+scheduler.register(MigrateSubtitleStorage)
+scheduler.register(MenuHistoryMaintenance)
+scheduler.register(CacheMaintenance)
@@ -1,6 +1,6 @@
 [
  {
-    "id": "langPref1",
+    "id": "langPref1a",
    "label": "Subtitle Language (1)",
    "type": "enum",
    "values": [
@@ -40,6 +40,8 @@
      "ro",
      "ru",
      "sr",
+      "sr-cyrl",
+      "sr-latn",
      "sk",
      "sl",
      "es",
@@ -53,7 +55,7 @@
    "default": "en"
  },
  {
-    "id": "langPref2",
+    "id": "langPref2a",
    "label": "Subtitle Language (2)",
    "type": "enum",
    "values": [
@@ -94,6 +96,8 @@
      "ro",
      "ru",
      "sr",
+      "sr-cyrl",
+      "sr-latn",
      "sk",
      "sl",
      "es",
@@ -107,7 +111,7 @@
    "default": "None"
  },
  {
-    "id": "langPref3",
+    "id": "langPref3a",
    "label": "Subtitle Language (3)",
    "type": "enum",
    "values": [
@@ -148,6 +152,8 @@
      "ro",
      "ru",
      "sr",
+      "sr-cyrl",
+      "sr-latn",
      "sk",
      "sl",
      "es",
@@ -173,11 +179,17 @@
    "default": "false"
  },
  {
-    "id": "subtitles.language.ietf",
-    "label": "Treat IETF language tags as ISO 639-1 (e.g. pt-BR = pt)",
+    "id": "subtitles.language.ietf_display",
+    "label": "Display languages with country attribute as ISO 639-1 (e.g. pt-BR = pt)",
    "type": "bool",
    "default": "true"
  },
+  {
+    "id": "subtitles.language.ietf_normalize",
+    "label": "Treat languages with country attribute as ISO 639-1 (e.g. don't download pt-BR if pt subtitle exists)",
+    "type": "bool",
+    "default": "false"
+  },
  {
    "id": "subtitles.only_one",
    "label": "Restrict to one language (skips adding \".lang.\" to the subtitle filename; only uses \"Subtitle Language (1)\")",
@@ -190,6 +202,50 @@
    "type": "bool",
    "default": "true"
  },
+  {
+    "id": "media_rename1",
+    "label": "I rename my files using",
+    "type": "enum",
+    "values": [
+      "Sonarr/Radarr (fill api info below)",
+      "Filebot",
+      "Sonarr/Radarr/Filebot",
+      "Symlink to original file",
+      "I keep the original filenames",
+      "none of the above"
+    ],
+    "default": "I keep the original filenames"
+  },
+  {
+    "id": "use_file_info_file",
+    "label": "Retrieve original filename from .file_info/file_info index files (see wiki)",
+    "type": "bool",
+    "default": "false"
+  },
+  {
+    "id": "drone_api.sonarr.url",
+    "label": "Sonarr URL (add URL base if configured)",
+    "type": "text",
+    "default": "http://127.0.0.1:8989"
+  },
+  {
+    "id": "drone_api.sonarr.api_key",
+    "label": "Sonarr API key",
+    "type": "text",
+    "default": ""
+  },
+  {
+    "id": "drone_api.radarr.url",
+    "label": "Radarr URL (add URL base if configured, min. version: 0.2.0.897)",
+    "type": "text",
+    "default": "http://127.0.0.1:7878"
+  },
+  {
+    "id": "drone_api.radarr.api_key",
+    "label": "Radarr API key",
+    "type": "text",
+    "default": ""
+  },
  {
    "id": "provider.opensubtitles.enabled",
    "label": "Provider: Enable OpenSubtitles",
@@ -198,7 +254,7 @@
  },
  {
    "id": "provider.opensubtitles.username",
-    "label": "Opensubtitles Username (VIP)",
+    "label": "Opensubtitles Username",
    "type": "text",
    "default": ""
  },
@@ -210,12 +266,24 @@
    "default": "",
    "secure": "true"
  },
+  {
+    "id": "provider.opensubtitles.is_vip",
+    "label": "OpenSubtitles VIP? (ad-free subs, 1000 subs/day, no-cache VIP server: http://v.ht/osvip)",
+    "type": "bool",
+    "default": "false"
+  },
  {
    "id": "provider.podnapisi.enabled",
    "label": "Provider: Enable Podnapisi.NET",
    "type": "bool",
    "default": "true"
  },
+  {
+    "id": "provider.titlovi.enabled",
+    "label": "Provider: Enable Titlovi.com",
+    "type": "bool",
+    "default": "true"
+  },
  {
    "id": "provider.addic7ed.enabled",
    "label": "Provider: Enable Addic7ed",
@@ -237,7 +305,7 @@
    "secure": "true"
  },
  {
-    "id": "provider.addic7ed.boost_by1",
+    "id": "provider.addic7ed.boost_by2",
    "label": "Addic7ed: boost score (if requirements met)",
    "type": "enum",
    "values": [
@@ -258,23 +326,25 @@
      "35",
      "30",
      "25",
+      "21",
      "20",
+      "19",
      "15",
      "10",
      "5",
      "0"
    ],
-    "default": "25"
+    "default": "19"
  },
  {
-    "id": "provider.addic7ed.use_random_agents",
+    "id": "provider.addic7ed.use_random_agents1",
    "label": "Addic7ed: Use random user agents",
    "type": "bool",
-    "default": "false"
+    "default": "true"
  },
  {
    "id": "provider.legendastv.enabled",
-    "label": "Provider: Enable Legendas TV (mostly pt-BR)",
+    "label": "Provider: Enable Legendas TV (mostly pt-BR; UNRAR NEEDED)",
    "type": "bool",
    "default": "false"
  },
@@ -305,64 +375,56 @@
    "default": "false"
  },
  {
-    "id": "provider.shooter.enabled",
-    "label": "Provider: Enable Shooter.cn (Chinese)",
+    "id": "provider.subscene.enabled",
+    "label": "Provider: Enable SubScene (TV shows)",
+    "type": "bool",
+    "default": "true"
+  },
+  {
+    "id": "provider.hosszupuska.enabled",
+    "label": "Provider: Enable hosszupuskasub.com (Hungarian)",
    "type": "bool",
    "default": "false"
  },
  {
-    "id": "provider.subscenter.enabled",
-    "label": "Provider: Enable SubsCenter (Hebrew)",
+    "id": "provider.argenteam.enabled",
+    "label": "Provider: Enable aRGENTeaM (Spanish)",
    "type": "bool",
    "default": "false"
  },
-  {
-    "id": "provider.subscenter.username",
-    "label": "SubsCenter Username",
-    "type": "text",
-    "default": ""
-  },
-  {
-    "id": "provider.subscenter.password",
-    "label": "SubsCenter Password",
-    "type": "text",
-    "option": "hidden",
-    "default": "",
-    "secure": "true"
-  },
  {
    "id": "providers.multithreading",
-    "label": "Search enabled providers simuntaneously (multithreading)",
+    "label": "Search enabled providers simultaneously (multithreading)",
    "type": "bool",
    "default": "true"
  },
  {
-    "id": "provider.opensubtitles.use_tags",
-    "label": "I keep the exact (release-) filename of my media files",
+    "id": "subtitles.embedded.autoextract",
+    "label": "Automatically extract and use embedded subtitles upon media addition (with configured default mods)",
    "type": "bool",
-    "default": "true"
+    "default": "false"
+  },
+  {
+    "id": "subtitles.search_after_autoextract",
+    "label": "After automatic extraction of embedded subtitles, also immediately search for available subtitles?",
+    "type": "bool",
+    "default": "false"
  },
  {
    "id": "subtitles.scan.embedded",
-    "label": "Scan: include embedded subtitles (in the media file (MKV/MP4), don't download if existing)",
+    "label": "Don't search for subtitles of a language if there are embedded subtitles inside the media file (MKV/MP4)?",
    "type": "bool",
    "default": "false"
  },
  {
    "id": "subtitles.scan.external",
-    "label": "Scan: include external subtitles (metadata/filesystem, don't download if existing)",
+    "label": "Don't search for subtitles of a language if they already exist on the filesystem (metadata/filesystem)?",
    "type": "bool",
    "default": "true"
  },
-  {
-    "id": "subtitles.scan.exotic_ext",
-    "label": "Scan: include \"exotic\" external subtitle formats (anything else than .srt/.ssa/.ass)",
-    "type": "bool",
-    "default": "false"
-  },
  {
    "id": "subtitles.scan.filename_strictness",
-    "label": "Scan: which external subtitles should be picked up?",
+    "label": "How strict should these subtitles existing on the filesystem be detected?",
    "type": "enum",
    "values": [
      "exact: media filename match",
@@ -371,6 +433,12 @@
    ],
    "default": "loose: filename contains media filename"
  },
+  {
+    "id": "subtitles.scan.exotic_ext",
+    "label": "Include non-text subtitle formats (anything else than .srt/.ssa/.ass/.vtt; embedded or external) in the above?",
+    "type": "bool",
+    "default": "false"
+  },
  {
    "id": "subtitles.search.minimumTVScore2",
    "label": "Minimum score for TV (min: 240, def/sane: 337, min-ideal: 352; see http://v.ht/szscores)",
@@ -381,7 +449,7 @@
    "id": "subtitles.search.minimumMovieScore2",
    "label": "Minimum score for movies (min: 60, def/sane: 69, min-ideal: 82; see http://v.ht/szscores)",
    "type": "text",
-    "default": "69"
+    "default": "60"
  },
  {
    "id": "subtitles.search.hearingImpaired",
@@ -396,17 +464,77 @@
    "default": "don't prefer"
  },
  {
-    "id": "subtitles.enforce_encoding",
-    "label": "Normalize subtitle encoding to UTF-8",
+    "id": "subtitles.remove_hi",
+    "label": "Remove Hearing Impaired tags from downloaded subtitles",
+    "type": "bool",
+    "default": "false"
+  },
+  {
+    "id": "subtitles.remove_tags",
+    "label": "Remove style tags from downloaded subtitles (bold, italic, underline, colors, ...)",
+    "type": "bool",
+    "default": "false"
+  },
+  {
+    "id": "subtitles.fix_common",
+    "label": "Fix common issues in subtitles",
    "type": "bool",
    "default": "true"
  },
+  {
+    "id": "subtitles.fix_ocr",
+    "label": "Fix common OCR errors in downloaded subtitles",
+    "type": "bool",
+    "default": "true"
+  },
+  {
+    "id": "subtitles.reverse_rtl",
+    "label": "Reverse punctuation in RTL languages (heb)",
+    "type": "bool",
+    "default": "false"
+  },
+  {
+    "id": "subtitles.colors",
+    "label": "Change colors of subtitles to",
+    "type": "enum",
+    "values": [
+      "don't change",
+      "white",
+      "light-grey",
+      "red",
+      "green",
+      "yellow",
+      "blue",
+      "magenta",
+      "cyan",
+      "black",
+      "dark-red",
+      "dark-green",
+      "dark-yellow",
+      "dark-blue",
+      "dark-magenta",
+      "dark-cyan",
+      "dark-grey"
+    ],
+    "default": "don't change"
+  },
  {
    "id": "subtitles.save.filesystem",
    "label": "Store subtitles next to media files (instead of metadata)",
    "type": "bool",
    "default": "true"
  },
+  {
+    "id": "subtitles.save.formats",
+    "label": "Subtitle formats to save (non-SRT only works if the previous option is enabled)",
+    "type": "enum",
+    "values": [
+      "SRT",
+      "VTT",
+      "SRT+VTT"
+    ],
+    "default": "SRT"
+  },
  {
    "id": "subtitles.save.subFolder",
    "label": "Subtitle Folder (\"current folder\" is the folder the current media file lives in)",
@@ -452,7 +580,8 @@
      "never",
      "current media item",
      "next episode (series)",
-      "hybrid: current item or next episode"
+      "hybrid: current item or next episode",
+      "hybrid-plus: current item and next episode"
    ],
    "default": "never"
  },
@@ -462,8 +591,6 @@
    "type": "enum",
    "values": [
      "never",
-      "every 1 hours",
-      "every 3 hours",
      "every 6 hours",
      "every 12 hours",
      "every 24 hours"
@@ -484,7 +611,8 @@
      "3 weeks",
      "4 weeks",
      "5 weeks",
-      "6 weeks"
+      "6 weeks",
+      "12 weeks"
    ],
    "default": "2 weeks"
  },
@@ -492,7 +620,7 @@
    "id": "scheduler.max_recent_items_per_library",
    "label": "Scheduler: Recent items to consider per library",
    "type": "text",
-    "default": "500"
+    "default": "1000"
  },
  {
    "id": "scheduler.tasks.FindBetterSubtitles.frequency",
@@ -512,12 +640,37 @@
    "type": "text",
    "default": "7"
  },
+  {
+    "id": "scheduler.tasks.FindBetterSubtitles.air_date_cutoff",
+    "label": "Scheduler: Don't search for better subtitles if the item's air date is older than",
+    "type": "enum",
+    "values": [
+      "don't limit",
+      "1 year",
+      "2 years",
+      "3 years",
+      "4 years",
+      "5 years",
+      "6 years",
+      "7 years",
+      "8 years",
+      "9 years",
+      "10 years"
+    ],
+    "default": "1 year"
+  },
  {
    "id": "scheduler.tasks.FindBetterSubtitles.overwrite_manually_selected",
    "label": "Scheduler: Overwrite manually selected subtitles when better found",
    "type": "bool",
    "default": "true"
  },
+  {
+    "id": "scheduler.tasks.FindBetterSubtitles.overwrite_manually_modified",
+    "label": "Scheduler: Overwrite subtitles with non-default subtitle modifications when better found",
+    "type": "bool",
+    "default": "false"
+  },
  {
    "id": "history_size",
    "label": "History: amount of items to store historical data for",
@@ -593,7 +746,7 @@
  },
  {
    "id": "notify_executable",
-    "label": "Call this executable upon successful subtitle download",
+    "label": "Call this executable upon successful subtitle download (see Wiki for details)",
    "type": "text",
    "default": ""
  },
@@ -603,6 +756,30 @@
    "type": "bool",
    "default": "true"
  },
+  {
+    "id": "new_style_cache",
+    "label": "Use new style caching (for subliminal)",
+    "type": "bool",
+    "default": "true"
+  },
+  {
+    "id": "low_impact_mode",
+    "label": "Low impact mode (for remote filesystems)",
+    "type": "bool",
+    "default": "false"
+  },
+  {
+    "id": "pms_request_timeout",
+    "label": "Timeout for API requests sent to the PMS",
+    "type": "text",
+    "default": "15"
+  },
+  {
+    "id": "proxy",
+    "label": "HTTP proxy to use for providers (supports credentials)",
+    "type": "text",
+    "default": ""
+  },
  {
    "id": "log_level",
    "label": "How verbose should the logging be?",
@@ -616,6 +793,18 @@
    ],
    "default": "WARNING"
  },
+  {
+    "id": "log_rotate_keep",
+    "label": "How many log backups to keep?",
+    "type": "text",
+    "default": "5"
+  },
+  {
+    "id": "log_debug_mods",
+    "label": "Log subtitle modification (debug)",
+    "type": "bool",
+    "default": "false"
+  },
  {
    "id": "log_console",
    "label": "Log to console (for development/debugging)",
@@ -9,11 +9,11 @@
        <key>CFBundleInfoDictionaryVersion</key>
        <string>6.0</string>
        <key>CFBundleShortVersionString</key>
-        <string>2.0.0</string>
+        <string>2.5.3</string>
        <key>CFBundleSignature</key>
        <string>????</string>
        <key>CFBundleVersion</key>
-        <string>2.0.0.0</string>
+        <string>2.5.3.2452</string>
        <key>PlexFrameworkVersion</key>
        <string>2</string>
        <key>PlexPluginClass</key>
@@ -23,7 +23,7 @@
        <key>PlexPluginConsoleLogging</key>
        <string>0</string>
        <key>PlexPluginDevMode</key>
-        <string>1</string>
+        <string>0</string>
         <key>PlexPluginCodePolicy</key>
            <!-- this allows channels to access some python methods which are otherwise blocked, as well as import external code libraries, and interact with the PMS HTTP API -->
            <string>Elevated</string>
@@ -32,7 +32,7 @@

 &lt;h1&gt;Sub-Zero for Plex&lt;/h1&gt;&lt;i&gt;Subtitles done right&lt;/i&gt;

-Version 2.0.0.0 DEV #6
+Version 2.5.3.2452

 Originally based on @bramwalet's awesome &lt;a href=&quot;https://github.com/bramwalet/Subliminal.bundle&quot;&gt;Subliminal.bundle&lt;/a&gt;

@@ -44,7 +44,7 @@ Score info: &lt;a href=&quot;http://v.ht/szscores&quot;&gt;http://v.ht/szscores&
 Plex thread: &lt;a href=&quot;https://forums.plex.tv/discussion/186575&quot;>https://forums.plex.tv/discussion/186575&lt;/a&gt;
 Github: &lt;a href=&quot;https://github.com/pannal/Sub-Zero.bundle&quot;&gt;https://github.com/pannal/Sub-Zero&lt;/a&gt;

-panni, 2017
+panni, 2018
 &lt;/div&gt;
 	</string>
    </dict>
@@ -0,0 +1,85 @@
+"""Generic interface to all dbm clones.
+
+Instead of
+
+        import dbm
+        d = dbm.open(file, 'w', 0666)
+
+use
+
+        import anydbm
+        d = anydbm.open(file, 'w')
+
+The returned object is a dbhash, gdbm, dbm or dumbdbm object,
+dependent on the type of database being opened (determined by whichdb
+module) in the case of an existing dbm. If the dbm does not exist and
+the create or new flag ('c' or 'n') was specified, the dbm type will
+be determined by the availability of the modules (tested in the above
+order).
+
+It has the following interface (key and data are strings):
+
+        d[key] = data   # store data at key (may override data at
+                        # existing key)
+        data = d[key]   # retrieve data at key (raise KeyError if no
+                        # such key)
+        del d[key]      # delete data stored at key (raises KeyError
+                        # if no such key)
+        flag = key in d   # true if the key exists
+        list = d.keys() # return a list of all existing keys (slow!)
+
+Future versions may change the order in which implementations are
+tested for existence, and add interfaces to other dbm-like
+implementations.
+"""
+
+class error(Exception):
+    pass
+
+_names = ['dbhash', 'gdbm', 'dbm', 'dumbdbm']
+_errors = [error]
+_defaultmod = None
+
+for _name in _names:
+    try:
+        _mod = __import__(_name)
+    except ImportError:
+        continue
+    if not _defaultmod:
+        _defaultmod = _mod
+    _errors.append(_mod.error)
+
+if not _defaultmod:
+    raise ImportError, "no dbm clone found; tried %s" % _names
+
+error = tuple(_errors)
+
+def open(file, flag='r', mode=0666):
+    """Open or create database at path given by *file*.
+
+    Optional argument *flag* can be 'r' (default) for read-only access, 'w'
+    for read-write access of an existing database, 'c' for read-write access
+    to a new or existing database, and 'n' for read-write access to a new
+    database.
+
+    Note: 'r' and 'w' fail if the database doesn't exist; 'c' creates it
+    only if it doesn't exist; and 'n' always creates a new database.
+    """
+
+    # guess the type of an existing database
+    from whichdb import whichdb
+    result=whichdb(file)
+    if result is None:
+        # db doesn't exist
+        if 'c' in flag or 'n' in flag:
+            # file doesn't exist and the new
+            # flag was used so use default type
+            mod = _defaultmod
+        else:
+            raise error, "need 'c' or 'n' flag to open new db"
+    elif result == "":
+        # db type cannot be determined
+        raise error, "db type could not be determined"
+    else:
+        mod = __import__(result)
+    return mod.open(file, flag, mode)
@@ -1,25 +0,0 @@
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2013 the BabelFish authors. All rights reserved.
-# Use of this source code is governed by the 3-clause BSD license
-# that can be found in the LICENSE file.
-#
-__title__ = 'babelfish'
-__version__ = '0.5.5-dev'
-__author__ = 'Antoine Bertin'
-__license__ = 'BSD'
-__copyright__ = 'Copyright 2015 the BabelFish authors'
-
-import sys
-
-if sys.version_info[0] >= 3:
-    basestr = str
-else:
-    basestr = basestring
-
-from .converters import (LanguageConverter, LanguageReverseConverter, LanguageEquivalenceConverter, CountryConverter,
-    CountryReverseConverter)
-from .country import country_converters, COUNTRIES, COUNTRY_MATRIX, Country
-from .exceptions import Error, LanguageConvertError, LanguageReverseError, CountryConvertError, CountryReverseError
-from .language import language_converters, LANGUAGES, LANGUAGE_MATRIX, Language
-from .script import SCRIPTS, SCRIPT_MATRIX, Script
@@ -1,287 +0,0 @@
-# Copyright (c) 2013 the BabelFish authors. All rights reserved.
-# Use of this source code is governed by the 3-clause BSD license
-# that can be found in the LICENSE file.
-#
-import collections
-from pkg_resources import iter_entry_points, EntryPoint
-from ..exceptions import LanguageConvertError, LanguageReverseError
-
-
-# from https://github.com/kennethreitz/requests/blob/master/requests/structures.py
-class CaseInsensitiveDict(collections.MutableMapping):
-    """A case-insensitive ``dict``-like object.
-
-    Implements all methods and operations of
-    ``collections.MutableMapping`` as well as dict's ``copy``. Also
-    provides ``lower_items``.
-
-    All keys are expected to be strings. The structure remembers the
-    case of the last key to be set, and ``iter(instance)``,
-    ``keys()``, ``items()``, ``iterkeys()``, and ``iteritems()``
-    will contain case-sensitive keys. However, querying and contains
-    testing is case insensitive:
-
-        cid = CaseInsensitiveDict()
-        cid['English'] = 'eng'
-        cid['ENGLISH'] == 'eng'  # True
-        list(cid) == ['English']  # True
-
-    If the constructor, ``.update``, or equality comparison
-    operations are given keys that have equal ``.lower()``s, the
-    behavior is undefined.
-
-    """
-    def __init__(self, data=None, **kwargs):
-        self._store = dict()
-        if data is None:
-            data = {}
-        self.update(data, **kwargs)
-
-    def __setitem__(self, key, value):
-        # Use the lowercased key for lookups, but store the actual
-        # key alongside the value.
-        self._store[key.lower()] = (key, value)
-
-    def __getitem__(self, key):
-        return self._store[key.lower()][1]
-
-    def __delitem__(self, key):
-        del self._store[key.lower()]
-
-    def __iter__(self):
-        return (casedkey for casedkey, mappedvalue in self._store.values())
-
-    def __len__(self):
-        return len(self._store)
-
-    def lower_items(self):
-        """Like iteritems(), but with all lowercase keys."""
-        return (
-            (lowerkey, keyval[1])
-            for (lowerkey, keyval)
-            in self._store.items()
-        )
-
-    def __eq__(self, other):
-        if isinstance(other, collections.Mapping):
-            other = CaseInsensitiveDict(other)
-        else:
-            return NotImplemented
-        # Compare insensitively
-        return dict(self.lower_items()) == dict(other.lower_items())
-
-    # Copy is required
-    def copy(self):
-        return CaseInsensitiveDict(self._store.values())
-
-    def __repr__(self):
-        return '%s(%r)' % (self.__class__.__name__, dict(self.items()))
-
-
-class LanguageConverter(object):
-    """A :class:`LanguageConverter` supports converting an alpha3 language code with an
-    alpha2 country code and a script code into a custom code
-
-    .. attribute:: codes
-
-        Set of possible custom codes
-
-    """
-    def convert(self, alpha3, country=None, script=None):
-        """Convert an alpha3 language code with an alpha2 country code and a script code
-        into a custom code
-
-        :param string alpha3: ISO-639-3 language code
-        :param country: ISO-3166 country code, if any
-        :type country: string or None
-        :param script: ISO-15924 script code, if any
-        :type script: string or None
-        :return: the corresponding custom code
-        :rtype: string
-        :raise: :class:`~babelfish.exceptions.LanguageConvertError`
-
-        """
-        raise NotImplementedError
-
-
-class LanguageReverseConverter(LanguageConverter):
-    """A :class:`LanguageConverter` able to reverse a custom code into a alpha3
-    ISO-639-3 language code, alpha2 ISO-3166-1 country code and ISO-15924 script code
-
-    """
-    def reverse(self, code):
-        """Reverse a custom code into alpha3, country and script code
-
-        :param string code: custom code to reverse
-        :return: the corresponding alpha3 ISO-639-3 language code, alpha2 ISO-3166-1 country code and ISO-15924 script code
-        :rtype: tuple
-        :raise: :class:`~babelfish.exceptions.LanguageReverseError`
-
-        """
-        raise NotImplementedError
-
-
-class LanguageEquivalenceConverter(LanguageReverseConverter):
-    """A :class:`LanguageEquivalenceConverter` is a utility class that allows you to easily define a
-    :class:`LanguageReverseConverter` by only specifying the dict from alpha3 to their corresponding symbols.
-
-    You must specify the dict of equivalence as a class variable named SYMBOLS.
-
-    If you also set the class variable CASE_SENSITIVE to ``True`` then the reverse conversion function will be
-    case-sensitive (it is case-insensitive by default).
-
-    Example::
-
-        class MyCodeConverter(babelfish.LanguageEquivalenceConverter):
-            CASE_SENSITIVE = True
-            SYMBOLS = {'fra': 'mycode1', 'eng': 'mycode2'}
-
-    """
-    CASE_SENSITIVE = False
-
-    def __init__(self):
-        self.codes = set()
-        self.to_symbol = {}
-        if self.CASE_SENSITIVE:
-            self.from_symbol = {}
-        else:
-            self.from_symbol = CaseInsensitiveDict()
-
-        for alpha3, symbol in self.SYMBOLS.items():
-            self.to_symbol[alpha3] = symbol
-            self.from_symbol[symbol] = (alpha3, None, None)
-            self.codes.add(symbol)
-
-    def convert(self, alpha3, country=None, script=None):
-        try:
-            return self.to_symbol[alpha3]
-        except KeyError:
-            raise LanguageConvertError(alpha3, country, script)
-
-    def reverse(self, code):
-        try:
-            return self.from_symbol[code]
-        except KeyError:
-            raise LanguageReverseError(code)
-
-
-class CountryConverter(object):
-    """A :class:`CountryConverter` supports converting an alpha2 country code
-    into a custom code
-
-    .. attribute:: codes
-
-        Set of possible custom codes
-
-    """
-    def convert(self, alpha2):
-        """Convert an alpha2 country code into a custom code
-
-        :param string alpha2: ISO-3166-1 language code
-        :return: the corresponding custom code
-        :rtype: string
-        :raise: :class:`~babelfish.exceptions.CountryConvertError`
-
-        """
-        raise NotImplementedError
-
-
-class CountryReverseConverter(CountryConverter):
-    """A :class:`CountryConverter` able to reverse a custom code into a alpha2
-    ISO-3166-1 country code
-
-    """
-    def reverse(self, code):
-        """Reverse a custom code into alpha2 code
-
-        :param string code: custom code to reverse
-        :return: the corresponding alpha2 ISO-3166-1 country code
-        :rtype: string
-        :raise: :class:`~babelfish.exceptions.CountryReverseError`
-
-        """
-        raise NotImplementedError
-
-
-class ConverterManager(object):
-    """Manager for babelfish converters behaving like a dict with lazy loading
-
-    Loading is done in this order:
-
-    * Entry point converters
-    * Registered converters
-    * Internal converters
-
-    .. attribute:: entry_point
-
-        The entry point where to look for converters
-
-    .. attribute:: internal_converters
-
-        Internal converters with entry point syntax
-
-    """
-    entry_point = ''
-    internal_converters = []
-
-    def __init__(self):
-        #: Registered converters with entry point syntax
-        self.registered_converters = []
-
-        #: Loaded converters
-        self.converters = {}
-
-    def __getitem__(self, name):
-        """Get a converter, lazy loading it if necessary"""
-        if name in self.converters:
-            return self.converters[name]
-        for ep in iter_entry_points(self.entry_point):
-            if ep.name == name:
-                self.converters[ep.name] = ep.load()()
-                return self.converters[ep.name]
-        for ep in (EntryPoint.parse(c) for c in self.registered_converters + self.internal_converters):
-            if ep.name == name:
-                # `require` argument of ep.load() is deprecated in newer versions of setuptools
-                if hasattr(ep, 'resolve'):
-                    plugin = ep.resolve()
-                elif hasattr(ep, '_load'):
-                    plugin = ep._load()
-                else:
-                    plugin = ep.load(require=False)
-                self.converters[ep.name] = plugin()
-                return self.converters[ep.name]
-        raise KeyError(name)
-
-    def __setitem__(self, name, converter):
-        """Load a converter"""
-        self.converters[name] = converter
-
-    def __delitem__(self, name):
-        """Unload a converter"""
-        del self.converters[name]
-
-    def __iter__(self):
-        """Iterator over loaded converters"""
-        return iter(self.converters)
-
-    def register(self, entry_point):
-        """Register a converter
-
-        :param string entry_point: converter to register (entry point syntax)
-        :raise: ValueError if already registered
-
-        """
-        if entry_point in self.registered_converters:
-            raise ValueError('Already registered')
-        self.registered_converters.insert(0, entry_point)
-
-    def unregister(self, entry_point):
-        """Unregister a converter
-
-        :param string entry_point: converter to unregister (entry point syntax)
-
-        """
-        self.registered_converters.remove(entry_point)
-
-    def __contains__(self, name):
-        return name in self.converters
@@ -1,17 +0,0 @@
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2013 the BabelFish authors. All rights reserved.
-# Use of this source code is governed by the 3-clause BSD license
-# that can be found in the LICENSE file.
-#
-from __future__ import unicode_literals
-from . import LanguageEquivalenceConverter
-from ..language import LANGUAGE_MATRIX
-
-
-class Alpha2Converter(LanguageEquivalenceConverter):
-    CASE_SENSITIVE = True
-    SYMBOLS = {}
-    for iso_language in LANGUAGE_MATRIX:
-        if iso_language.alpha2:
-            SYMBOLS[iso_language.alpha3] = iso_language.alpha2
@@ -1,17 +0,0 @@
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2013 the BabelFish authors. All rights reserved.
-# Use of this source code is governed by the 3-clause BSD license
-# that can be found in the LICENSE file.
-#
-from __future__ import unicode_literals
-from . import LanguageEquivalenceConverter
-from ..language import LANGUAGE_MATRIX
-
-
-class Alpha3BConverter(LanguageEquivalenceConverter):
-    CASE_SENSITIVE = True
-    SYMBOLS = {}
-    for iso_language in LANGUAGE_MATRIX:
-        if iso_language.alpha3b:
-            SYMBOLS[iso_language.alpha3] = iso_language.alpha3b
@@ -1,17 +0,0 @@
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2013 the BabelFish authors. All rights reserved.
-# Use of this source code is governed by the 3-clause BSD license
-# that can be found in the LICENSE file.
-#
-from __future__ import unicode_literals
-from . import LanguageEquivalenceConverter
-from ..language import LANGUAGE_MATRIX
-
-
-class Alpha3TConverter(LanguageEquivalenceConverter):
-    CASE_SENSITIVE = True
-    SYMBOLS = {}
-    for iso_language in LANGUAGE_MATRIX:
-        if iso_language.alpha3t:
-            SYMBOLS[iso_language.alpha3] = iso_language.alpha3t
@@ -1,31 +0,0 @@
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2013 the BabelFish authors. All rights reserved.
-# Use of this source code is governed by the 3-clause BSD license
-# that can be found in the LICENSE file.
-#
-from __future__ import unicode_literals
-from . import CountryReverseConverter, CaseInsensitiveDict
-from ..country import COUNTRY_MATRIX
-from ..exceptions import CountryConvertError, CountryReverseError
-
-
-class CountryNameConverter(CountryReverseConverter):
-    def __init__(self):
-        self.codes = set()
-        self.to_name = {}
-        self.from_name = CaseInsensitiveDict()
-        for country in COUNTRY_MATRIX:
-            self.codes.add(country.name)
-            self.to_name[country.alpha2] = country.name
-            self.from_name[country.name] = country.alpha2
-
-    def convert(self, alpha2):
-        if alpha2 not in self.to_name:
-            raise CountryConvertError(alpha2)
-        return self.to_name[alpha2]
-
-    def reverse(self, name):
-        if name not in self.from_name:
-            raise CountryReverseError(name)
-        return self.from_name[name]
@@ -1,17 +0,0 @@
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2013 the BabelFish authors. All rights reserved.
-# Use of this source code is governed by the 3-clause BSD license
-# that can be found in the LICENSE file.
-#
-from __future__ import unicode_literals
-from . import LanguageEquivalenceConverter
-from ..language import LANGUAGE_MATRIX
-
-
-class NameConverter(LanguageEquivalenceConverter):
-    CASE_SENSITIVE = False
-    SYMBOLS = {}
-    for iso_language in LANGUAGE_MATRIX:
-        if iso_language.name:
-            SYMBOLS[iso_language.alpha3] = iso_language.name
@@ -1,36 +0,0 @@
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2013 the BabelFish authors. All rights reserved.
-# Use of this source code is governed by the 3-clause BSD license
-# that can be found in the LICENSE file.
-#
-from __future__ import unicode_literals
-from . import LanguageReverseConverter, CaseInsensitiveDict
-from ..exceptions import LanguageReverseError
-from ..language import language_converters
-
-
-class OpenSubtitlesConverter(LanguageReverseConverter):
-    def __init__(self):
-        self.alpha3b_converter = language_converters['alpha3b']
-        self.alpha2_converter = language_converters['alpha2']
-        self.to_opensubtitles = {('por', 'BR'): 'pob', ('gre', None): 'ell', ('srp', None): 'scc', ('srp', 'ME'): 'mne'}
-        self.from_opensubtitles = CaseInsensitiveDict({'pob': ('por', 'BR'), 'pb': ('por', 'BR'), 'ell': ('ell', None),
-                                                       'scc': ('srp', None), 'mne': ('srp', 'ME')})
-        self.codes = (self.alpha2_converter.codes | self.alpha3b_converter.codes | set(['pob', 'pb', 'scc', 'mne']))
-
-    def convert(self, alpha3, country=None, script=None):
-        alpha3b = self.alpha3b_converter.convert(alpha3, country, script)
-        if (alpha3b, country) in self.to_opensubtitles:
-            return self.to_opensubtitles[(alpha3b, country)]
-        return alpha3b
-
-    def reverse(self, opensubtitles):
-        if opensubtitles in self.from_opensubtitles:
-            return self.from_opensubtitles[opensubtitles]
-        for conv in [self.alpha3b_converter, self.alpha2_converter]:
-            try:
-                return conv.reverse(opensubtitles)
-            except LanguageReverseError:
-                pass
-        raise LanguageReverseError(opensubtitles)
@@ -1,23 +0,0 @@
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2013 the BabelFish authors. All rights reserved.
-# Use of this source code is governed by the 3-clause BSD license
-# that can be found in the LICENSE file.
-#
-from __future__ import unicode_literals
-from . import LanguageConverter
-from ..exceptions import LanguageConvertError
-from ..language import LANGUAGE_MATRIX
-
-
-class ScopeConverter(LanguageConverter):
-    FULLNAME = {'I': 'individual', 'M': 'macrolanguage', 'S': 'special'}
-    SYMBOLS = {}
-    for iso_language in LANGUAGE_MATRIX:
-        SYMBOLS[iso_language.alpha3] = iso_language.scope
-    codes = set(SYMBOLS.values())
-
-    def convert(self, alpha3, country=None, script=None):
-        if self.SYMBOLS[alpha3] in self.FULLNAME:
-            return self.FULLNAME[self.SYMBOLS[alpha3]]
-        raise LanguageConvertError(alpha3, country, script)
@@ -1,23 +0,0 @@
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2013 the BabelFish authors. All rights reserved.
-# Use of this source code is governed by the 3-clause BSD license
-# that can be found in the LICENSE file.
-#
-from __future__ import unicode_literals
-from . import LanguageConverter
-from ..exceptions import LanguageConvertError
-from ..language import LANGUAGE_MATRIX
-
-
-class LanguageTypeConverter(LanguageConverter):
-    FULLNAME = {'A': 'ancient', 'C': 'constructed', 'E': 'extinct', 'H': 'historical', 'L': 'living', 'S': 'special'}
-    SYMBOLS = {}
-    for iso_language in LANGUAGE_MATRIX:
-        SYMBOLS[iso_language.alpha3] = iso_language.type
-    codes = set(SYMBOLS.values())
-
-    def convert(self, alpha3, country=None, script=None):
-        if self.SYMBOLS[alpha3] in self.FULLNAME:
-            return self.FULLNAME[self.SYMBOLS[alpha3]]
-        raise LanguageConvertError(alpha3, country, script)
@@ -1,104 +0,0 @@
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2013 the BabelFish authors. All rights reserved.
-# Use of this source code is governed by the 3-clause BSD license
-# that can be found in the LICENSE file.
-#
-from __future__ import unicode_literals
-from collections import namedtuple
-from functools import partial
-from pkg_resources import resource_stream  # @UnresolvedImport
-from .converters import ConverterManager
-from . import basestr
-
-
-COUNTRIES = {}
-COUNTRY_MATRIX = []
-
-#: The namedtuple used in the :data:`COUNTRY_MATRIX`
-IsoCountry = namedtuple('IsoCountry', ['name', 'alpha2'])
-
-f = resource_stream('babelfish', 'data/iso-3166-1.txt')
-f.readline()
-for l in f:
-    iso_country = IsoCountry(*l.decode('utf-8').strip().split(';'))
-    COUNTRIES[iso_country.alpha2] = iso_country.name
-    COUNTRY_MATRIX.append(iso_country)
-f.close()
-
-
-class CountryConverterManager(ConverterManager):
-    """:class:`~babelfish.converters.ConverterManager` for country converters"""
-    entry_point = 'babelfish.country_converters'
-    internal_converters = ['name = babelfish.converters.countryname:CountryNameConverter']
-
-country_converters = CountryConverterManager()
-
-
-class CountryMeta(type):
-    """The :class:`Country` metaclass
-
-    Dynamically redirect :meth:`Country.frommycode` to :meth:`Country.fromcode` with the ``mycode`` `converter`
-
-    """
-    def __getattr__(cls, name):
-        if name.startswith('from'):
-            return partial(cls.fromcode, converter=name[4:])
-        return type.__getattribute__(cls, name)
-
-
-class Country(CountryMeta(str('CountryBase'), (object,), {})):
-    """A country on Earth
-
-    A country is represented by a 2-letter code from the ISO-3166 standard
-
-    :param string country: 2-letter ISO-3166 country code
-
-    """
-    def __init__(self, country):
-        if country not in COUNTRIES:
-            raise ValueError('%r is not a valid country' % country)
-
-        #: ISO-3166 2-letter country code
-        self.alpha2 = country
-
-    @classmethod
-    def fromcode(cls, code, converter):
-        """Create a :class:`Country` by its `code` using `converter` to
-        :meth:`~babelfish.converters.CountryReverseConverter.reverse` it
-
-        :param string code: the code to reverse
-        :param string converter: name of the :class:`~babelfish.converters.CountryReverseConverter` to use
-        :return: the corresponding :class:`Country` instance
-        :rtype: :class:`Country`
-
-        """
-        return cls(country_converters[converter].reverse(code))
-
-    def __getstate__(self):
-        return self.alpha2
-
-    def __setstate__(self, state):
-        self.alpha2 = state
-
-    def __getattr__(self, name):
-        return country_converters[name].convert(self.alpha2)
-
-    def __hash__(self):
-        return hash(self.alpha2)
-
-    def __eq__(self, other):
-        if isinstance(other, basestr):
-            return str(self) == other
-        if not isinstance(other, Country):
-            return False
-        return self.alpha2 == other.alpha2
-
-    def __ne__(self, other):
-        return not self == other
-
-    def __repr__(self):
-        return '<Country [%s]>' % self
-
-    def __str__(self):
-        return self.alpha2
@@ -1,45 +0,0 @@
-#!/usr/bin/env python
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2013 the BabelFish authors. All rights reserved.
-# Use of this source code is governed by the 3-clause BSD license
-# that can be found in the LICENSE file.
-#
-from __future__ import unicode_literals
-import os.path
-import tempfile
-import zipfile
-import requests
-
-
-DATA_DIR = os.path.dirname(__file__)
-
-# iso-3166-1.txt
-print('Downloading ISO-3166-1 standard (ISO country codes)...')
-with open(os.path.join(DATA_DIR, 'iso-3166-1.txt'), 'w') as f:
-    r = requests.get('http://www.iso.org/iso/home/standards/country_codes/country_names_and_code_elements_txt.htm')
-    f.write(r.content.strip())
-
-# iso-639-3.tab
-print('Downloading ISO-639-3 standard (ISO language codes)...')
-with tempfile.TemporaryFile() as f:
-    r = requests.get('http://www-01.sil.org/iso639-3/iso-639-3_Code_Tables_20130531.zip')
-    f.write(r.content)
-    with zipfile.ZipFile(f) as z:
-        z.extract('iso-639-3.tab', DATA_DIR)
-
-# iso-15924
-print('Downloading ISO-15924 standard (ISO script codes)...')
-with tempfile.TemporaryFile() as f:
-    r = requests.get('http://www.unicode.org/iso15924/iso15924.txt.zip')
-    f.write(r.content)
-    with zipfile.ZipFile(f) as z:
-        z.extract('iso15924-utf8-20131012.txt', DATA_DIR)
-
-# opensubtitles supported languages
-print('Downloading OpenSubtitles supported languages...')
-with open(os.path.join(DATA_DIR, 'opensubtitles_languages.txt'), 'w') as f:
-    r = requests.get('http://www.opensubtitles.org/addons/export_languages.php')
-    f.write(r.content)
-
-print('Done!')
@@ -1,250 +0,0 @@
-Country Name;ISO 3166-1-alpha-2 code
-AFGHANISTAN;AF
-ÅLAND ISLANDS;AX
-ALBANIA;AL
-ALGERIA;DZ
-AMERICAN SAMOA;AS
-ANDORRA;AD
-ANGOLA;AO
-ANGUILLA;AI
-ANTARCTICA;AQ
-ANTIGUA AND BARBUDA;AG
-ARGENTINA;AR
-ARMENIA;AM
-ARUBA;AW
-AUSTRALIA;AU
-AUSTRIA;AT
-AZERBAIJAN;AZ
-BAHAMAS;BS
-BAHRAIN;BH
-BANGLADESH;BD
-BARBADOS;BB
-BELARUS;BY
-BELGIUM;BE
-BELIZE;BZ
-BENIN;BJ
-BERMUDA;BM
-BHUTAN;BT
-BOLIVIA, PLURINATIONAL STATE OF;BO
-BONAIRE, SINT EUSTATIUS AND SABA;BQ
-BOSNIA AND HERZEGOVINA;BA
-BOTSWANA;BW
-BOUVET ISLAND;BV
-BRAZIL;BR
-BRITISH INDIAN OCEAN TERRITORY;IO
-BRUNEI DARUSSALAM;BN
-BULGARIA;BG
-BURKINA FASO;BF
-BURUNDI;BI
-CAMBODIA;KH
-CAMEROON;CM
-CANADA;CA
-CAPE VERDE;CV
-CAYMAN ISLANDS;KY
-CENTRAL AFRICAN REPUBLIC;CF
-CHAD;TD
-CHILE;CL
-CHINA;CN
-CHRISTMAS ISLAND;CX
-COCOS (KEELING) ISLANDS;CC
-COLOMBIA;CO
-COMOROS;KM
-CONGO;CG
-CONGO, THE DEMOCRATIC REPUBLIC OF THE;CD
-COOK ISLANDS;CK
-COSTA RICA;CR
-CÔTE D'IVOIRE;CI
-CROATIA;HR
-CUBA;CU
-CURAÇAO;CW
-CYPRUS;CY
-CZECH REPUBLIC;CZ
-DENMARK;DK
-DJIBOUTI;DJ
-DOMINICA;DM
-DOMINICAN REPUBLIC;DO
-ECUADOR;EC
-EGYPT;EG
-EL SALVADOR;SV
-EQUATORIAL GUINEA;GQ
-ERITREA;ER
-ESTONIA;EE
-ETHIOPIA;ET
-FALKLAND ISLANDS (MALVINAS);FK
-FAROE ISLANDS;FO
-FIJI;FJ
-FINLAND;FI
-FRANCE;FR
-FRENCH GUIANA;GF
-FRENCH POLYNESIA;PF
-FRENCH SOUTHERN TERRITORIES;TF
-GABON;GA
-GAMBIA;GM
-GEORGIA;GE
-GERMANY;DE
-GHANA;GH
-GIBRALTAR;GI
-GREECE;GR
-GREENLAND;GL
-GRENADA;GD
-GUADELOUPE;GP
-GUAM;GU
-GUATEMALA;GT
-GUERNSEY;GG
-GUINEA;GN
-GUINEA-BISSAU;GW
-GUYANA;GY
-HAITI;HT
-HEARD ISLAND AND MCDONALD ISLANDS;HM
-HOLY SEE (VATICAN CITY STATE);VA
-HONDURAS;HN
-HONG KONG;HK
-HUNGARY;HU
-ICELAND;IS
-INDIA;IN
-INDONESIA;ID
-IRAN, ISLAMIC REPUBLIC OF;IR
-IRAQ;IQ
-IRELAND;IE
-ISLE OF MAN;IM
-ISRAEL;IL
-ITALY;IT
-JAMAICA;JM
-JAPAN;JP
-JERSEY;JE
-JORDAN;JO
-KAZAKHSTAN;KZ
-KENYA;KE
-KIRIBATI;KI
-KOREA, DEMOCRATIC PEOPLE'S REPUBLIC OF;KP
-KOREA, REPUBLIC OF;KR
-KUWAIT;KW
-KYRGYZSTAN;KG
-LAO PEOPLE'S DEMOCRATIC REPUBLIC;LA
-LATVIA;LV
-LEBANON;LB
-LESOTHO;LS
-LIBERIA;LR
-LIBYA;LY
-LIECHTENSTEIN;LI
-LITHUANIA;LT
-LUXEMBOURG;LU
-MACAO;MO
-MACEDONIA, THE FORMER YUGOSLAV REPUBLIC OF;MK
-MADAGASCAR;MG
-MALAWI;MW
-MALAYSIA;MY
-MALDIVES;MV
-MALI;ML
-MALTA;MT
-MARSHALL ISLANDS;MH
-MARTINIQUE;MQ
-MAURITANIA;MR
-MAURITIUS;MU
-MAYOTTE;YT
-MEXICO;MX
-MICRONESIA, FEDERATED STATES OF;FM
-MOLDOVA, REPUBLIC OF;MD
-MONACO;MC
-MONGOLIA;MN
-MONTENEGRO;ME
-MONTSERRAT;MS
-MOROCCO;MA
-MOZAMBIQUE;MZ
-MYANMAR;MM
-NAMIBIA;NA
-NAURU;NR
-NEPAL;NP
-NETHERLANDS;NL
-NEW CALEDONIA;NC
-NEW ZEALAND;NZ
-NICARAGUA;NI
-NIGER;NE
-NIGERIA;NG
-NIUE;NU
-NORFOLK ISLAND;NF
-NORTHERN MARIANA ISLANDS;MP
-NORWAY;NO
-OMAN;OM
-PAKISTAN;PK
-PALAU;PW
-PALESTINE, STATE OF;PS
-PANAMA;PA
-PAPUA NEW GUINEA;PG
-PARAGUAY;PY
-PERU;PE
-PHILIPPINES;PH
-PITCAIRN;PN
-POLAND;PL
-PORTUGAL;PT
-PUERTO RICO;PR
-QATAR;QA
-RÉUNION;RE
-ROMANIA;RO
-RUSSIAN FEDERATION;RU
-RWANDA;RW
-SAINT BARTHÉLEMY;BL
-SAINT HELENA, ASCENSION AND TRISTAN DA CUNHA;SH
-SAINT KITTS AND NEVIS;KN
-SAINT LUCIA;LC
-SAINT MARTIN (FRENCH PART);MF
-SAINT PIERRE AND MIQUELON;PM
-SAINT VINCENT AND THE GRENADINES;VC
-SAMOA;WS
-SAN MARINO;SM
-SAO TOME AND PRINCIPE;ST
-SAUDI ARABIA;SA
-SENEGAL;SN
-SERBIA;RS
-SEYCHELLES;SC
-SIERRA LEONE;SL
-SINGAPORE;SG
-SINT MAARTEN (DUTCH PART);SX
-SLOVAKIA;SK
-SLOVENIA;SI
-SOLOMON ISLANDS;SB
-SOMALIA;SO
-SOUTH AFRICA;ZA
-SOUTH GEORGIA AND THE SOUTH SANDWICH ISLANDS;GS
-SOUTH SUDAN;SS
-SPAIN;ES
-SRI LANKA;LK
-SUDAN;SD
-SURINAME;SR
-SVALBARD AND JAN MAYEN;SJ
-SWAZILAND;SZ
-SWEDEN;SE
-SWITZERLAND;CH
-SYRIAN ARAB REPUBLIC;SY
-TAIWAN, PROVINCE OF CHINA;TW
-TAJIKISTAN;TJ
-TANZANIA, UNITED REPUBLIC OF;TZ
-THAILAND;TH
-TIMOR-LESTE;TL
-TOGO;TG
-TOKELAU;TK
-TONGA;TO
-TRINIDAD AND TOBAGO;TT
-TUNISIA;TN
-TURKEY;TR
-TURKMENISTAN;TM
-TURKS AND CAICOS ISLANDS;TC
-TUVALU;TV
-UGANDA;UG
-UKRAINE;UA
-UNITED ARAB EMIRATES;AE
-UNITED KINGDOM;GB
-UNITED STATES;US
-UNITED STATES MINOR OUTLYING ISLANDS;UM
-URUGUAY;UY
-UZBEKISTAN;UZ
-VANUATU;VU
-VENEZUELA, BOLIVARIAN REPUBLIC OF;VE
-VIET NAM;VN
-VIRGIN ISLANDS, BRITISH;VG
-VIRGIN ISLANDS, U.S.;VI
-WALLIS AND FUTUNA;WF
-WESTERN SAHARA;EH
-YEMEN;YE
-ZAMBIA;ZM
-ZIMBABWE;ZW
@@ -1,176 +0,0 @@
-#
-# ISO 15924 - Codes for the representation of names of scripts
-#             Codes pour la représentation des noms d’écritures
-# Format: 
-#             Code;N°;English Name;Nom français;PVA;Date
-#
-
-Afak;439;Afaka;afaka;;2010-12-21
-Aghb;239;Caucasian Albanian;aghbanien;;2012-10-16
-Ahom;338;Ahom, Tai Ahom;âhom;;2012-11-01
-Arab;160;Arabic;arabe;Arabic;2004-05-01
-Armi;124;Imperial Aramaic;araméen impérial;Imperial_Aramaic;2009-06-01
-Armn;230;Armenian;arménien;Armenian;2004-05-01
-Avst;134;Avestan;avestique;Avestan;2009-06-01
-Bali;360;Balinese;balinais;Balinese;2006-10-10
-Bamu;435;Bamum;bamoum;Bamum;2009-06-01
-Bass;259;Bassa Vah;bassa;;2010-03-26
-Batk;365;Batak;batik;Batak;2010-07-23
-Beng;325;Bengali;bengalî;Bengali;2004-05-01
-Blis;550;Blissymbols;symboles Bliss;;2004-05-01
-Bopo;285;Bopomofo;bopomofo;Bopomofo;2004-05-01
-Brah;300;Brahmi;brahma;Brahmi;2010-07-23
-Brai;570;Braille;braille;Braille;2004-05-01
-Bugi;367;Buginese;bouguis;Buginese;2006-06-21
-Buhd;372;Buhid;bouhide;Buhid;2004-05-01
-Cakm;349;Chakma;chakma;Chakma;2012-02-06
-Cans;440;Unified Canadian Aboriginal Syllabics;syllabaire autochtone canadien unifié;Canadian_Aboriginal;2004-05-29
-Cari;201;Carian;carien;Carian;2007-07-02
-Cham;358;Cham;cham (čam, tcham);Cham;2009-11-11
-Cher;445;Cherokee;tchérokî;Cherokee;2004-05-01
-Cirt;291;Cirth;cirth;;2004-05-01
-Copt;204;Coptic;copte;Coptic;2006-06-21
-Cprt;403;Cypriot;syllabaire chypriote;Cypriot;2004-05-01
-Cyrl;220;Cyrillic;cyrillique;Cyrillic;2004-05-01
-Cyrs;221;Cyrillic (Old Church Slavonic variant);cyrillique (variante slavonne);;2004-05-01
-Deva;315;Devanagari (Nagari);dévanâgarî;Devanagari;2004-05-01
-Dsrt;250;Deseret (Mormon);déseret (mormon);Deseret;2004-05-01
-Dupl;755;Duployan shorthand, Duployan stenography;sténographie Duployé;;2010-07-18
-Egyd;070;Egyptian demotic;démotique égyptien;;2004-05-01
-Egyh;060;Egyptian hieratic;hiératique égyptien;;2004-05-01
-Egyp;050;Egyptian hieroglyphs;hiéroglyphes égyptiens;Egyptian_Hieroglyphs;2009-06-01
-Elba;226;Elbasan;elbasan;;2010-07-18
-Ethi;430;Ethiopic (Geʻez);éthiopien (geʻez, guèze);Ethiopic;2004-10-25
-Geor;240;Georgian (Mkhedruli);géorgien (mkhédrouli);Georgian;2004-05-29
-Geok;241;Khutsuri (Asomtavruli and Nuskhuri);khoutsouri (assomtavrouli et nouskhouri);Georgian;2012-10-16
-Glag;225;Glagolitic;glagolitique;Glagolitic;2006-06-21
-Goth;206;Gothic;gotique;Gothic;2004-05-01
-Gran;343;Grantha;grantha;;2009-11-11
-Grek;200;Greek;grec;Greek;2004-05-01
-Gujr;320;Gujarati;goudjarâtî (gujrâtî);Gujarati;2004-05-01
-Guru;310;Gurmukhi;gourmoukhî;Gurmukhi;2004-05-01
-Hang;286;Hangul (Hangŭl, Hangeul);hangûl (hangŭl, hangeul);Hangul;2004-05-29
-Hani;500;Han (Hanzi, Kanji, Hanja);idéogrammes han (sinogrammes);Han;2009-02-23
-Hano;371;Hanunoo (Hanunóo);hanounóo;Hanunoo;2004-05-29
-Hans;501;Han (Simplified variant);idéogrammes han (variante simplifiée);;2004-05-29
-Hant;502;Han (Traditional variant);idéogrammes han (variante traditionnelle);;2004-05-29
-Hatr;127;Hatran;hatrénien;;2012-11-01
-Hebr;125;Hebrew;hébreu;Hebrew;2004-05-01
-Hira;410;Hiragana;hiragana;Hiragana;2004-05-01
-Hluw;080;Anatolian Hieroglyphs (Luwian Hieroglyphs, Hittite Hieroglyphs);hiéroglyphes anatoliens (hiéroglyphes louvites, hiéroglyphes hittites);;2011-12-09
-Hmng;450;Pahawh Hmong;pahawh hmong;;2004-05-01
-Hrkt;412;Japanese syllabaries (alias for Hiragana + Katakana);syllabaires japonais (alias pour hiragana + katakana);Katakana_Or_Hiragana;2011-06-21
-Hung;176;Old Hungarian (Hungarian Runic);runes hongroises (ancien hongrois);;2012-10-16
-Inds;610;Indus (Harappan);indus;;2004-05-01
-Ital;210;Old Italic (Etruscan, Oscan, etc.);ancien italique (étrusque, osque, etc.);Old_Italic;2004-05-29
-Java;361;Javanese;javanais;Javanese;2009-06-01
-Jpan;413;Japanese (alias for Han + Hiragana + Katakana);japonais (alias pour han + hiragana + katakana);;2006-06-21
-Jurc;510;Jurchen;jurchen;;2010-12-21
-Kali;357;Kayah Li;kayah li;Kayah_Li;2007-07-02
-Kana;411;Katakana;katakana;Katakana;2004-05-01
-Khar;305;Kharoshthi;kharochthî;Kharoshthi;2006-06-21
-Khmr;355;Khmer;khmer;Khmer;2004-05-29
-Khoj;322;Khojki;khojkî;;2011-06-21
-Knda;345;Kannada;kannara (canara);Kannada;2004-05-29
-Kore;287;Korean (alias for Hangul + Han);coréen (alias pour hangûl + han);;2007-06-13
-Kpel;436;Kpelle;kpèllé;;2010-03-26
-Kthi;317;Kaithi;kaithî;Kaithi;2009-06-01
-Lana;351;Tai Tham (Lanna);taï tham (lanna);Tai_Tham;2009-06-01
-Laoo;356;Lao;laotien;Lao;2004-05-01
-Latf;217;Latin (Fraktur variant);latin (variante brisée);;2004-05-01
-Latg;216;Latin (Gaelic variant);latin (variante gaélique);;2004-05-01
-Latn;215;Latin;latin;Latin;2004-05-01
-Lepc;335;Lepcha (Róng);lepcha (róng);Lepcha;2007-07-02
-Limb;336;Limbu;limbou;Limbu;2004-05-29
-Lina;400;Linear A;linéaire A;;2004-05-01
-Linb;401;Linear B;linéaire B;Linear_B;2004-05-29
-Lisu;399;Lisu (Fraser);lisu (Fraser);Lisu;2009-06-01
-Loma;437;Loma;loma;;2010-03-26
-Lyci;202;Lycian;lycien;Lycian;2007-07-02
-Lydi;116;Lydian;lydien;Lydian;2007-07-02
-Mahj;314;Mahajani;mahâjanî;;2012-10-16
-Mand;140;Mandaic, Mandaean;mandéen;Mandaic;2010-07-23
-Mani;139;Manichaean;manichéen;;2007-07-15
-Maya;090;Mayan hieroglyphs;hiéroglyphes mayas;;2004-05-01
-Mend;438;Mende Kikakui;mendé kikakui;;2013-10-12
-Merc;101;Meroitic Cursive;cursif méroïtique;Meroitic_Cursive;2012-02-06
-Mero;100;Meroitic Hieroglyphs;hiéroglyphes méroïtiques;Meroitic_Hieroglyphs;2012-02-06
-Mlym;347;Malayalam;malayâlam;Malayalam;2004-05-01
-Modi;323;Modi, Moḍī;modî;;2013-10-12
-Moon;218;Moon (Moon code, Moon script, Moon type);écriture Moon;;2006-12-11
-Mong;145;Mongolian;mongol;Mongolian;2004-05-01
-Mroo;199;Mro, Mru;mro;;2010-12-21
-Mtei;337;Meitei Mayek (Meithei, Meetei);meitei mayek;Meetei_Mayek;2009-06-01
-Mult;323; Multani;multanî;;2012-11-01
-Mymr;350;Myanmar (Burmese);birman;Myanmar;2004-05-01
-Narb;106;Old North Arabian (Ancient North Arabian);nord-arabique;;2010-03-26
-Nbat;159;Nabataean;nabatéen;;2010-03-26
-Nkgb;420;Nakhi Geba ('Na-'Khi ²Ggŏ-¹baw, Naxi Geba);nakhi géba;;2009-02-23
-Nkoo;165;N’Ko;n’ko;Nko;2006-10-10
-Nshu;499;Nüshu;nüshu;;2010-12-21
-Ogam;212;Ogham;ogam;Ogham;2004-05-01
-Olck;261;Ol Chiki (Ol Cemet’, Ol, Santali);ol tchiki;Ol_Chiki;2007-07-02
-Orkh;175;Old Turkic, Orkhon Runic;orkhon;Old_Turkic;2009-06-01
-Orya;327;Oriya;oriyâ;Oriya;2004-05-01
-Osma;260;Osmanya;osmanais;Osmanya;2004-05-01
-Palm;126;Palmyrene;palmyrénien;;2010-03-26
-Pauc;263;Pau Cin Hau;paou chin haou;;2013-10-12
-Perm;227;Old Permic;ancien permien;;2004-05-01
-Phag;331;Phags-pa;’phags pa;Phags_Pa;2006-10-10
-Phli;131;Inscriptional Pahlavi;pehlevi des inscriptions;Inscriptional_Pahlavi;2009-06-01
-Phlp;132;Psalter Pahlavi;pehlevi des psautiers;;2007-11-26
-Phlv;133;Book Pahlavi;pehlevi des livres;;2007-07-15
-Phnx;115;Phoenician;phénicien;Phoenician;2006-10-10
-Plrd;282;Miao (Pollard);miao (Pollard);Miao;2012-02-06
-Prti;130;Inscriptional Parthian;parthe des inscriptions;Inscriptional_Parthian;2009-06-01
-Qaaa;900;Reserved for private use (start);réservé à l’usage privé (début);;2004-05-29
-Qabx;949;Reserved for private use (end);réservé à l’usage privé (fin);;2004-05-29
-Rjng;363;Rejang (Redjang, Kaganga);redjang (kaganga);Rejang;2009-02-23
-Roro;620;Rongorongo;rongorongo;;2004-05-01
-Runr;211;Runic;runique;Runic;2004-05-01
-Samr;123;Samaritan;samaritain;Samaritan;2009-06-01
-Sara;292;Sarati;sarati;;2004-05-29
-Sarb;105;Old South Arabian;sud-arabique, himyarite;Old_South_Arabian;2009-06-01
-Saur;344;Saurashtra;saurachtra;Saurashtra;2007-07-02
-Sgnw;095;SignWriting;SignÉcriture, SignWriting;;2006-10-10
-Shaw;281;Shavian (Shaw);shavien (Shaw);Shavian;2004-05-01
-Shrd;319;Sharada, Śāradā;charada, shard;Sharada;2012-02-06
-Sidd;302;Siddham, Siddhaṃ, Siddhamātṛkā;siddham;;2013-10-12
-Sind;318;Khudawadi, Sindhi;khoudawadî, sindhî;;2010-12-21
-Sinh;348;Sinhala;singhalais;Sinhala;2004-05-01
-Sora;398;Sora Sompeng;sora sompeng;Sora_Sompeng;2012-02-06
-Sund;362;Sundanese;sundanais;Sundanese;2007-07-02
-Sylo;316;Syloti Nagri;sylotî nâgrî;Syloti_Nagri;2006-06-21
-Syrc;135;Syriac;syriaque;Syriac;2004-05-01
-Syre;138;Syriac (Estrangelo variant);syriaque (variante estranghélo);;2004-05-01
-Syrj;137;Syriac (Western variant);syriaque (variante occidentale);;2004-05-01
-Syrn;136;Syriac (Eastern variant);syriaque (variante orientale);;2004-05-01
-Tagb;373;Tagbanwa;tagbanoua;Tagbanwa;2004-05-01
-Takr;321;Takri, Ṭākrī, Ṭāṅkrī;tâkrî;Takri;2012-02-06
-Tale;353;Tai Le;taï-le;Tai_Le;2004-10-25
-Talu;354;New Tai Lue;nouveau taï-lue;New_Tai_Lue;2006-06-21
-Taml;346;Tamil;tamoul;Tamil;2004-05-01
-Tang;520;Tangut;tangoute;;2010-12-21
-Tavt;359;Tai Viet;taï viêt;Tai_Viet;2009-06-01
-Telu;340;Telugu;télougou;Telugu;2004-05-01
-Teng;290;Tengwar;tengwar;;2004-05-01
-Tfng;120;Tifinagh (Berber);tifinagh (berbère);Tifinagh;2006-06-21
-Tglg;370;Tagalog (Baybayin, Alibata);tagal (baybayin, alibata);Tagalog;2009-02-23
-Thaa;170;Thaana;thâna;Thaana;2004-05-01
-Thai;352;Thai;thaï;Thai;2004-05-01
-Tibt;330;Tibetan;tibétain;Tibetan;2004-05-01
-Tirh;326;Tirhuta;tirhouta;;2011-12-09
-Ugar;040;Ugaritic;ougaritique;Ugaritic;2004-05-01
-Vaii;470;Vai;vaï;Vai;2007-07-02
-Visp;280;Visible Speech;parole visible;;2004-05-01
-Wara;262;Warang Citi (Varang Kshiti);warang citi;;2009-11-11
-Wole;480;Woleai;woléaï;;2010-12-21
-Xpeo;030;Old Persian;cunéiforme persépolitain;Old_Persian;2006-06-21
-Xsux;020;Cuneiform, Sumero-Akkadian;cunéiforme suméro-akkadien;Cuneiform;2006-10-10
-Yiii;460;Yi;yi;Yi;2004-05-01
-Zinh;994;Code for inherited script;codet pour écriture héritée;Inherited;2009-02-23
-Zmth;995;Mathematical notation;notation mathématique;;2007-11-26
-Zsym;996;Symbols;symboles;;2007-11-26
-Zxxx;997;Code for unwritten documents;codet pour les documents non écrits;;2011-06-21
-Zyyy;998;Code for undetermined script;codet pour écriture indéterminée;Common;2004-05-29
-Zzzz;999;Code for uncoded script;codet pour écriture non codée;Unknown;2006-10-10
@@ -1,474 +0,0 @@
-IdSubLanguage	ISO639	LanguageName	UploadEnabled	WebEnabled
-aar	aa	Afar, afar	0	0
-abk	ab	Abkhazian	0	0
-ace		Achinese	0	0
-ach		Acoli	0	0
-ada		Adangme	0	0
-ady		adyghé	0	0
-afa		Afro-Asiatic (Other)	0	0
-afh		Afrihili	0	0
-afr	af	Afrikaans	1	0
-ain		Ainu	0	0
-aka	ak	Akan	0	0
-akk		Akkadian	0	0
-alb	sq	Albanian	1	1
-ale		Aleut	0	0
-alg		Algonquian languages	0	0
-alt		Southern Altai	0	0
-amh	am	Amharic	0	0
-ang		English, Old (ca.450-1100)	0	0
-apa		Apache languages	0	0
-ara	ar	Arabic	1	1
-arc		Aramaic	0	0
-arg	an	Aragonese	0	0
-arm	hy	Armenian	1	0
-arn		Araucanian	0	0
-arp		Arapaho	0	0
-art		Artificial (Other)	0	0
-arw		Arawak	0	0
-asm	as	Assamese	0	0
-ast		Asturian, Bable	0	0
-ath		Athapascan languages	0	0
-aus		Australian languages	0	0
-ava	av	Avaric	0	0
-ave	ae	Avestan	0	0
-awa		Awadhi	0	0
-aym	ay	Aymara	0	0
-aze	az	Azerbaijani	0	0
-bad		Banda	0	0
-bai		Bamileke languages	0	0
-bak	ba	Bashkir	0	0
-bal		Baluchi	0	0
-bam	bm	Bambara	0	0
-ban		Balinese	0	0
-baq	eu	Basque	1	1
-bas		Basa	0	0
-bat		Baltic (Other)	0	0
-bej		Beja	0	0
-bel	be	Belarusian	0	0
-bem		Bemba	0	0
-ben	bn	Bengali	1	0
-ber		Berber (Other)	0	0
-bho		Bhojpuri	0	0
-bih	bh	Bihari	0	0
-bik		Bikol	0	0
-bin		Bini	0	0
-bis	bi	Bislama	0	0
-bla		Siksika	0	0
-bnt		Bantu (Other)	0	0
-bos	bs	Bosnian	1	0
-bra		Braj	0	0
-bre	br	Breton	1	0
-btk		Batak (Indonesia)	0	0
-bua		Buriat	0	0
-bug		Buginese	0	0
-bul	bg	Bulgarian	1	1
-bur	my	Burmese	1	0
-byn		Blin	0	0
-cad		Caddo	0	0
-cai		Central American Indian (Other)	0	0
-car		Carib	0	0
-cat	ca	Catalan	1	1
-cau		Caucasian (Other)	0	0
-ceb		Cebuano	0	0
-cel		Celtic (Other)	0	0
-cha	ch	Chamorro	0	0
-chb		Chibcha	0	0
-che	ce	Chechen	0	0
-chg		Chagatai	0	0
-chi	zh	Chinese	1	1
-chk		Chuukese	0	0
-chm		Mari	0	0
-chn		Chinook jargon	0	0
-cho		Choctaw	0	0
-chp		Chipewyan	0	0
-chr		Cherokee	0	0
-chu	cu	Church Slavic	0	0
-chv	cv	Chuvash	0	0
-chy		Cheyenne	0	0
-cmc		Chamic languages	0	0
-cop		Coptic	0	0
-cor	kw	Cornish	0	0
-cos	co	Corsican	0	0
-cpe		Creoles and pidgins, English based (Other)	0	0
-cpf		Creoles and pidgins, French-based (Other)	0	0
-cpp		Creoles and pidgins, Portuguese-based (Other)	0	0
-cre	cr	Cree	0	0
-crh		Crimean Tatar	0	0
-crp		Creoles and pidgins (Other)	0	0
-csb		Kashubian	0	0
-cus		Cushitic (Other)' couchitiques, autres langues	0	0
-cze	cs	Czech	1	1
-dak		Dakota	0	0
-dan	da	Danish	1	1
-dar		Dargwa	0	0
-day		Dayak	0	0
-del		Delaware	0	0
-den		Slave (Athapascan)	0	0
-dgr		Dogrib	0	0
-din		Dinka	0	0
-div	dv	Divehi	0	0
-doi		Dogri	0	0
-dra		Dravidian (Other)	0	0
-dua		Duala	0	0
-dum		Dutch, Middle (ca.1050-1350)	0	0
-dut	nl	Dutch	1	1
-dyu		Dyula	0	0
-dzo	dz	Dzongkha	0	0
-efi		Efik	0	0
-egy		Egyptian (Ancient)	0	0
-eka		Ekajuk	0	0
-elx		Elamite	0	0
-eng	en	English	1	1
-enm		English, Middle (1100-1500)	0	0
-epo	eo	Esperanto	1	0
-est	et	Estonian	1	1
-ewe	ee	Ewe	0	0
-ewo		Ewondo	0	0
-fan		Fang	0	0
-fao	fo	Faroese	0	0
-fat		Fanti	0	0
-fij	fj	Fijian	0	0
-fil		Filipino	0	0
-fin	fi	Finnish	1	1
-fiu		Finno-Ugrian (Other)	0	0
-fon		Fon	0	0
-fre	fr	French	1	1
-frm		French, Middle (ca.1400-1600)	0	0
-fro		French, Old (842-ca.1400)	0	0
-fry	fy	Frisian	0	0
-ful	ff	Fulah	0	0
-fur		Friulian	0	0
-gaa		Ga	0	0
-gay		Gayo	0	0
-gba		Gbaya	0	0
-gem		Germanic (Other)	0	0
-geo	ka	Georgian	1	1
-ger	de	German	1	1
-gez		Geez	0	0
-gil		Gilbertese	0	0
-gla	gd	Gaelic	0	0
-gle	ga	Irish	0	0
-glg	gl	Galician	1	1
-glv	gv	Manx	0	0
-gmh		German, Middle High (ca.1050-1500)	0	0
-goh		German, Old High (ca.750-1050)	0	0
-gon		Gondi	0	0
-gor		Gorontalo	0	0
-got		Gothic	0	0
-grb		Grebo	0	0
-grc		Greek, Ancient (to 1453)	0	0
-ell	el	Greek	1	1
-grn	gn	Guarani	0	0
-guj	gu	Gujarati	0	0
-gwi		Gwich´in	0	0
-hai		Haida	0	0
-hat	ht	Haitian	0	0
-hau	ha	Hausa	0	0
-haw		Hawaiian	0	0
-heb	he	Hebrew	1	1
-her	hz	Herero	0	0
-hil		Hiligaynon	0	0
-him		Himachali	0	0
-hin	hi	Hindi	1	1
-hit		Hittite	0	0
-hmn		Hmong	0	0
-hmo	ho	Hiri Motu	0	0
-hrv	hr	Croatian	1	1
-hun	hu	Hungarian	1	1
-hup		Hupa	0	0
-iba		Iban	0	0
-ibo	ig	Igbo	0	0
-ice	is	Icelandic	1	1
-ido	io	Ido	0	0
-iii	ii	Sichuan Yi	0	0
-ijo		Ijo	0	0
-iku	iu	Inuktitut	0	0
-ile	ie	Interlingue	0	0
-ilo		Iloko	0	0
-ina	ia	Interlingua (International Auxiliary Language Asso	0	0
-inc		Indic (Other)	0	0
-ind	id	Indonesian	1	1
-ine		Indo-European (Other)	0	0
-inh		Ingush	0	0
-ipk	ik	Inupiaq	0	0
-ira		Iranian (Other)	0	0
-iro		Iroquoian languages	0	0
-ita	it	Italian	1	1
-jav	jv	Javanese	0	0
-jpn	ja	Japanese	1	1
-jpr		Judeo-Persian	0	0
-jrb		Judeo-Arabic	0	0
-kaa		Kara-Kalpak	0	0
-kab		Kabyle	0	0
-kac		Kachin	0	0
-kal	kl	Kalaallisut	0	0
-kam		Kamba	0	0
-kan	kn	Kannada	0	0
-kar		Karen	0	0
-kas	ks	Kashmiri	0	0
-kau	kr	Kanuri	0	0
-kaw		Kawi	0	0
-kaz	kk	Kazakh	1	0
-kbd		Kabardian	0	0
-kha		Khasi	0	0
-khi		Khoisan (Other)	0	0
-khm	km	Khmer	1	1
-kho		Khotanese	0	0
-kik	ki	Kikuyu	0	0
-kin	rw	Kinyarwanda	0	0
-kir	ky	Kirghiz	0	0
-kmb		Kimbundu	0	0
-kok		Konkani	0	0
-kom	kv	Komi	0	0
-kon	kg	Kongo	0	0
-kor	ko	Korean	1	1
-kos		Kosraean	0	0
-kpe		Kpelle	0	0
-krc		Karachay-Balkar	0	0
-kro		Kru	0	0
-kru		Kurukh	0	0
-kua	kj	Kuanyama	0	0
-kum		Kumyk	0	0
-kur	ku	Kurdish	0	0
-kut		Kutenai	0	0
-lad		Ladino	0	0
-lah		Lahnda	0	0
-lam		Lamba	0	0
-lao	lo	Lao	0	0
-lat	la	Latin	0	0
-lav	lv	Latvian	1	0
-lez		Lezghian	0	0
-lim	li	Limburgan	0	0
-lin	ln	Lingala	0	0
-lit	lt	Lithuanian	1	0
-lol		Mongo	0	0
-loz		Lozi	0	0
-ltz	lb	Luxembourgish	1	0
-lua		Luba-Lulua	0	0
-lub	lu	Luba-Katanga	0	0
-lug	lg	Ganda	0	0
-lui		Luiseno	0	0
-lun		Lunda	0	0
-luo		Luo (Kenya and Tanzania)	0	0
-lus		lushai	0	0
-mac	mk	Macedonian	1	1
-mad		Madurese	0	0
-mag		Magahi	0	0
-mah	mh	Marshallese	0	0
-mai		Maithili	0	0
-mak		Makasar	0	0
-mal	ml	Malayalam	1	0
-man		Mandingo	0	0
-mao	mi	Maori	0	0
-map		Austronesian (Other)	0	0
-mar	mr	Marathi	0	0
-mas		Masai	0	0
-may	ms	Malay	1	1
-mdf		Moksha	0	0
-mdr		Mandar	0	0
-men		Mende	0	0
-mga		Irish, Middle (900-1200)	0	0
-mic		Mi'kmaq	0	0
-min		Minangkabau	0	0
-mis		Miscellaneous languages	0	0
-mkh		Mon-Khmer (Other)	0	0
-mlg	mg	Malagasy	0	0
-mlt	mt	Maltese	0	0
-mnc		Manchu	0	0
-mni		Manipuri	0	0
-mno		Manobo languages	0	0
-moh		Mohawk	0	0
-mol	mo	Moldavian	0	0
-mon	mn	Mongolian	1	0
-mos		Mossi	0	0
-mwl		Mirandese	0	0
-mul		Multiple languages	0	0
-mun		Munda languages	0	0
-mus		Creek	0	0
-mwr		Marwari	0	0
-myn		Mayan languages	0	0
-myv		Erzya	0	0
-nah		Nahuatl	0	0
-nai		North American Indian	0	0
-nap		Neapolitan	0	0
-nau	na	Nauru	0	0
-nav	nv	Navajo	0	0
-nbl	nr	Ndebele, South	0	0
-nde	nd	Ndebele, North	0	0
-ndo	ng	Ndonga	0	0
-nds		Low German	0	0
-nep	ne	Nepali	0	0
-new		Nepal Bhasa	0	0
-nia		Nias	0	0
-nic		Niger-Kordofanian (Other)	0	0
-niu		Niuean	0	0
-nno	nn	Norwegian Nynorsk	0	0
-nob	nb	Norwegian Bokmal	0	0
-nog		Nogai	0	0
-non		Norse, Old	0	0
-nor	no	Norwegian	1	1
-nso		Northern Sotho	0	0
-nub		Nubian languages	0	0
-nwc		Classical Newari	0	0
-nya	ny	Chichewa	0	0
-nym		Nyamwezi	0	0
-nyn		Nyankole	0	0
-nyo		Nyoro	0	0
-nzi		Nzima	0	0
-oci	oc	Occitan	1	1
-oji	oj	Ojibwa	0	0
-ori	or	Oriya	0	0
-orm	om	Oromo	0	0
-osa		Osage	0	0
-oss	os	Ossetian	0	0
-ota		Turkish, Ottoman (1500-1928)	0	0
-oto		Otomian languages	0	0
-paa		Papuan (Other)	0	0
-pag		Pangasinan	0	0
-pal		Pahlavi	0	0
-pam		Pampanga	0	0
-pan	pa	Panjabi	0	0
-pap		Papiamento	0	0
-pau		Palauan	0	0
-peo		Persian, Old (ca.600-400 B.C.)	0	0
-per	fa	Persian	1	1
-phi		Philippine (Other)	0	0
-phn		Phoenician	0	0
-pli	pi	Pali	0	0
-pol	pl	Polish	1	1
-pon		Pohnpeian	0	0
-por	pt	Portuguese	1	1
-pra		Prakrit languages	0	0
-pro		Provençal, Old (to 1500)	0	0
-pus	ps	Pushto	0	0
-que	qu	Quechua	0	0
-raj		Rajasthani	0	0
-rap		Rapanui	0	0
-rar		Rarotongan	0	0
-roa		Romance (Other)	0	0
-roh	rm	Raeto-Romance	0	0
-rom		Romany	0	0
-run	rn	Rundi	0	0
-rup		Aromanian	0	0
-rus	ru	Russian	1	1
-sad		Sandawe	0	0
-sag	sg	Sango	0	0
-sah		Yakut	0	0
-sai		South American Indian (Other)	0	0
-sal		Salishan languages	0	0
-sam		Samaritan Aramaic	0	0
-san	sa	Sanskrit	0	0
-sas		Sasak	0	0
-sat		Santali	0	0
-scc	sr	Serbian	1	1
-scn		Sicilian	0	0
-sco		Scots	0	0
-sel		Selkup	0	0
-sem		Semitic (Other)	0	0
-sga		Irish, Old (to 900)	0	0
-sgn		Sign Languages	0	0
-shn		Shan	0	0
-sid		Sidamo	0	0
-sin	si	Sinhalese	1	1
-sio		Siouan languages	0	0
-sit		Sino-Tibetan (Other)	0	0
-sla		Slavic (Other)	0	0
-slo	sk	Slovak	1	1
-slv	sl	Slovenian	1	1
-sma		Southern Sami	0	0
-sme	se	Northern Sami	0	0
-smi		Sami languages (Other)	0	0
-smj		Lule Sami	0	0
-smn		Inari Sami	0	0
-smo	sm	Samoan	0	0
-sms		Skolt Sami	0	0
-sna	sn	Shona	0	0
-snd	sd	Sindhi	0	0
-snk		Soninke	0	0
-sog		Sogdian	0	0
-som	so	Somali	0	0
-son		Songhai	0	0
-sot	st	Sotho, Southern	0	0
-spa	es	Spanish	1	1
-srd	sc	Sardinian	0	0
-srr		Serer	0	0
-ssa		Nilo-Saharan (Other)	0	0
-ssw	ss	Swati	0	0
-suk		Sukuma	0	0
-sun	su	Sundanese	0	0
-sus		Susu	0	0
-sux		Sumerian	0	0
-swa	sw	Swahili	1	0
-swe	sv	Swedish	1	1
-syr		Syriac	1	0
-tah	ty	Tahitian	0	0
-tai		Tai (Other)	0	0
-tam	ta	Tamil	1	0
-tat	tt	Tatar	0	0
-tel	te	Telugu	1	0
-tem		Timne	0	0
-ter		Tereno	0	0
-tet		Tetum	0	0
-tgk	tg	Tajik	0	0
-tgl	tl	Tagalog	1	1
-tha	th	Thai	1	1
-tib	bo	Tibetan	0	0
-tig		Tigre	0	0
-tir	ti	Tigrinya	0	0
-tiv		Tiv	0	0
-tkl		Tokelau	0	0
-tlh		Klingon	0	0
-tli		Tlingit	0	0
-tmh		Tamashek	0	0
-tog		Tonga (Nyasa)	0	0
-ton	to	Tonga (Tonga Islands)	0	0
-tpi		Tok Pisin	0	0
-tsi		Tsimshian	0	0
-tsn	tn	Tswana	0	0
-tso	ts	Tsonga	0	0
-tuk	tk	Turkmen	0	0
-tum		Tumbuka	0	0
-tup		Tupi languages	0	0
-tur	tr	Turkish	1	1
-tut		Altaic (Other)	0	0
-tvl		Tuvalu	0	0
-twi	tw	Twi	0	0
-tyv		Tuvinian	0	0
-udm		Udmurt	0	0
-uga		Ugaritic	0	0
-uig	ug	Uighur	0	0
-ukr	uk	Ukrainian	1	1
-umb		Umbundu	0	0
-und		Undetermined	0	0
-urd	ur	Urdu	1	0
-uzb	uz	Uzbek	0	0
-vai		Vai	0	0
-ven	ve	Venda	0	0
-vie	vi	Vietnamese	1	1
-vol	vo	Volapük	0	0
-vot		Votic	0	0
-wak		Wakashan languages	0	0
-wal		Walamo	0	0
-war		Waray	0	0
-was		Washo	0	0
-wel	cy	Welsh	0	0
-wen		Sorbian languages	0	0
-wln	wa	Walloon	0	0
-wol	wo	Wolof	0	0
-xal		Kalmyk	0	0
-xho	xh	Xhosa	0	0
-yao		Yao	0	0
-yap		Yapese	0	0
-yid	yi	Yiddish	0	0
-yor	yo	Yoruba	0	0
-ypk		Yupik languages	0	0
-zap		Zapotec	0	0
-zen		Zenaga	0	0
-zha	za	Zhuang	0	0
-znd		Zande	0	0
-zul	zu	Zulu	0	0
-zun		Zuni	0	0
-rum	ro	Romanian	1	1
-pob	pb	Brazilian	1	1
-mne		Montenegrin	1	0
@@ -1,85 +0,0 @@
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2013 the BabelFish authors. All rights reserved.
-# Use of this source code is governed by the 3-clause BSD license
-# that can be found in the LICENSE file.
-#
-from __future__ import unicode_literals
-
-
-class Error(Exception):
-    """Base class for all exceptions in babelfish"""
-    pass
-
-
-class LanguageError(Error, AttributeError):
-    """Base class for all language exceptions in babelfish"""
-    pass
-
-
-class LanguageConvertError(LanguageError):
-    """Exception raised by converters when :meth:`~babelfish.converters.LanguageConverter.convert` fails
-
-    :param string alpha3: alpha3 code that failed conversion
-    :param country: country code that failed conversion, if any
-    :type country: string or None
-    :param script: script code that failed conversion, if any
-    :type script: string or None
-
-    """
-    def __init__(self, alpha3, country=None, script=None):
-        self.alpha3 = alpha3
-        self.country = country
-        self.script = script
-
-    def __str__(self):
-        s = self.alpha3
-        if self.country is not None:
-            s += '-' + self.country
-        if self.script is not None:
-            s += '-' + self.script
-        return s
-
-
-class LanguageReverseError(LanguageError):
-    """Exception raised by converters when :meth:`~babelfish.converters.LanguageReverseConverter.reverse` fails
-
-    :param string code: code that failed reverse conversion
-
-    """
-    def __init__(self, code):
-        self.code = code
-
-    def __str__(self):
-        return repr(self.code)
-
-
-class CountryError(Error, AttributeError):
-    """Base class for all country exceptions in babelfish"""
-    pass
-
-
-class CountryConvertError(CountryError):
-    """Exception raised by converters when :meth:`~babelfish.converters.CountryConverter.convert` fails
-
-    :param string alpha2: alpha2 code that failed conversion
-
-    """
-    def __init__(self, alpha2):
-        self.alpha2 = alpha2
-
-    def __str__(self):
-        return self.alpha2
-
-
-class CountryReverseError(CountryError):
-    """Exception raised by converters when :meth:`~babelfish.converters.CountryReverseConverter.reverse` fails
-
-    :param string code: code that failed reverse conversion
-
-    """
-    def __init__(self, code):
-        self.code = code
-
-    def __str__(self):
-        return repr(self.code)
@@ -1,185 +0,0 @@
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2013 the BabelFish authors. All rights reserved.
-# Use of this source code is governed by the 3-clause BSD license
-# that can be found in the LICENSE file.
-#
-from __future__ import unicode_literals
-from collections import namedtuple
-from functools import partial
-from pkg_resources import resource_stream  # @UnresolvedImport
-from .converters import ConverterManager
-from .country import Country
-from .exceptions import LanguageConvertError
-from .script import Script
-from . import basestr
-
-
-LANGUAGES = set()
-LANGUAGE_MATRIX = []
-
-#: The namedtuple used in the :data:`LANGUAGE_MATRIX`
-IsoLanguage = namedtuple('IsoLanguage', ['alpha3', 'alpha3b', 'alpha3t', 'alpha2', 'scope', 'type', 'name', 'comment'])
-
-f = resource_stream('babelfish', 'data/iso-639-3.tab')
-f.readline()
-for l in f:
-    iso_language = IsoLanguage(*l.decode('utf-8').split('\t'))
-    LANGUAGES.add(iso_language.alpha3)
-    LANGUAGE_MATRIX.append(iso_language)
-f.close()
-
-
-class LanguageConverterManager(ConverterManager):
-    """:class:`~babelfish.converters.ConverterManager` for language converters"""
-    entry_point = 'babelfish.language_converters'
-    internal_converters = ['alpha2 = babelfish.converters.alpha2:Alpha2Converter',
-                           'alpha3b = babelfish.converters.alpha3b:Alpha3BConverter',
-                           'alpha3t = babelfish.converters.alpha3t:Alpha3TConverter',
-                           'name = babelfish.converters.name:NameConverter',
-                           'scope = babelfish.converters.scope:ScopeConverter',
-                           'type = babelfish.converters.type:LanguageTypeConverter',
-                           'opensubtitles = babelfish.converters.opensubtitles:OpenSubtitlesConverter']
-
-language_converters = LanguageConverterManager()
-
-
-class LanguageMeta(type):
-    """The :class:`Language` metaclass
-
-    Dynamically redirect :meth:`Language.frommycode` to :meth:`Language.fromcode` with the ``mycode`` `converter`
-
-    """
-    def __getattr__(cls, name):
-        if name.startswith('from'):
-            return partial(cls.fromcode, converter=name[4:])
-        return type.__getattribute__(cls, name)
-
-
-class Language(LanguageMeta(str('LanguageBase'), (object,), {})):
-    """A human language
-
-    A human language is composed of a language part following the ISO-639
-    standard and can be country-specific when a :class:`~babelfish.country.Country`
-    is specified.
-
-    The :class:`Language` is extensible with custom converters (see :ref:`custom_converters`)
-
-    :param string language: the language as a 3-letter ISO-639-3 code
-    :param country: the country (if any) as a 2-letter ISO-3166 code or :class:`~babelfish.country.Country` instance
-    :type country: string or :class:`~babelfish.country.Country` or None
-    :param script: the script (if any) as a 4-letter ISO-15924 code or :class:`~babelfish.script.Script` instance
-    :type script: string or :class:`~babelfish.script.Script` or None
-    :param unknown: the unknown language as a three-letters ISO-639-3 code to use as fallback
-    :type unknown: string or None
-    :raise: ValueError if the language could not be recognized and `unknown` is ``None``
-
-    """
-    def __init__(self, language, country=None, script=None, unknown=None):
-        if unknown is not None and language not in LANGUAGES:
-            language = unknown
-        if language not in LANGUAGES:
-            raise ValueError('%r is not a valid language' % language)
-        self.alpha3 = language
-        self.country = None
-        if isinstance(country, Country):
-            self.country = country
-        elif country is None:
-            self.country = None
-        else:
-            self.country = Country(country)
-        self.script = None
-        if isinstance(script, Script):
-            self.script = script
-        elif script is None:
-            self.script = None
-        else:
-            self.script = Script(script)
-
-    @classmethod
-    def fromcode(cls, code, converter):
-        """Create a :class:`Language` by its `code` using `converter` to
-        :meth:`~babelfish.converters.LanguageReverseConverter.reverse` it
-
-        :param string code: the code to reverse
-        :param string converter: name of the :class:`~babelfish.converters.LanguageReverseConverter` to use
-        :return: the corresponding :class:`Language` instance
-        :rtype: :class:`Language`
-
-        """
-        return cls(*language_converters[converter].reverse(code))
-
-    @classmethod
-    def fromietf(cls, ietf):
-        """Create a :class:`Language` by from an IETF language code
-
-        :param string ietf: the ietf code
-        :return: the corresponding :class:`Language` instance
-        :rtype: :class:`Language`
-
-        """
-        subtags = ietf.split('-')
-        language_subtag = subtags.pop(0).lower()
-        if len(language_subtag) == 2:
-            language = cls.fromalpha2(language_subtag)
-        else:
-            language = cls(language_subtag)
-        while subtags:
-            subtag = subtags.pop(0)
-            if len(subtag) == 2:
-                language.country = Country(subtag.upper())
-            else:
-                language.script = Script(subtag.capitalize())
-            if language.script is not None:
-                if subtags:
-                    raise ValueError('Wrong IETF format. Unmatched subtags: %r' % subtags)
-                break
-        return language
-
-    def __getstate__(self):
-        return self.alpha3, self.country, self.script
-
-    def __setstate__(self, state):
-        self.alpha3, self.country, self.script = state
-
-    def __getattr__(self, name):
-        alpha3 = self.alpha3
-        country = self.country.alpha2 if self.country is not None else None
-        script = self.script.code if self.script is not None else None
-        try:
-            return language_converters[name].convert(alpha3, country, script)
-        except KeyError:
-            raise AttributeError(name)
-
-    def __hash__(self):
-        return hash(str(self))
-
-    def __eq__(self, other):
-        if isinstance(other, basestr):
-            return str(self) == other
-        if not isinstance(other, Language):
-            return False
-        return (self.alpha3 == other.alpha3 and
-                self.country == other.country and
-                self.script == other.script)
-
-    def __ne__(self, other):
-        return not self == other
-
-    def __bool__(self):
-        return self.alpha3 != 'und'
-    __nonzero__ = __bool__
-
-    def __repr__(self):
-        return '<Language [%s]>' % self
-
-    def __str__(self):
-        try:
-            s = self.alpha2
-        except LanguageConvertError:
-            s = self.alpha3
-        if self.country is not None:
-            s += '-' + str(self.country)
-        if self.script is not None:
-            s += '-' + str(self.script)
-        return s
@@ -1,76 +0,0 @@
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2013 the BabelFish authors. All rights reserved.
-# Use of this source code is governed by the 3-clause BSD license
-# that can be found in the LICENSE file.
-#
-from __future__ import unicode_literals
-from collections import namedtuple
-from pkg_resources import resource_stream  # @UnresolvedImport
-from . import basestr
-
-#: Script code to script name mapping
-SCRIPTS = {}
-
-#: List of countries in the ISO-15924 as namedtuple of code, number, name, french_name, pva and date
-SCRIPT_MATRIX = []
-
-#: The namedtuple used in the :data:`SCRIPT_MATRIX`
-IsoScript = namedtuple('IsoScript', ['code', 'number', 'name', 'french_name', 'pva', 'date'])
-
-f = resource_stream('babelfish', 'data/iso15924-utf8-20131012.txt')
-f.readline()
-for l in f:
-    l = l.decode('utf-8').strip()
-    if not l or l.startswith('#'):
-        continue
-    script = IsoScript._make(l.split(';'))
-    SCRIPT_MATRIX.append(script)
-    SCRIPTS[script.code] = script.name
-f.close()
-
-
-class Script(object):
-    """A human writing system
-
-    A script is represented by a 4-letter code from the ISO-15924 standard
-
-    :param string script: 4-letter ISO-15924 script code
-
-    """
-    def __init__(self, script):
-        if script not in SCRIPTS:
-            raise ValueError('%r is not a valid script' % script)
-
-        #: ISO-15924 4-letter script code
-        self.code = script
-
-    @property
-    def name(self):
-        """English name of the script"""
-        return SCRIPTS[self.code]
-
-    def __getstate__(self):
-        return self.code
-
-    def __setstate__(self, state):
-        self.code = state
-
-    def __hash__(self):
-        return hash(self.code)
-
-    def __eq__(self, other):
-        if isinstance(other, basestr):
-            return self.code == other
-        if not isinstance(other, Script):
-            return False
-        return self.code == other.code
-
-    def __ne__(self, other):
-        return not self == other
-
-    def __repr__(self):
-        return '<Script [%s]>' % self
-
-    def __str__(self):
-        return self.code
@@ -1,368 +0,0 @@
-#!/usr/bin/env python
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2013 the BabelFish authors. All rights reserved.
-# Use of this source code is governed by the 3-clause BSD license
-# that can be found in the LICENSE file.
-#
-from __future__ import unicode_literals
-import re
-import sys
-import pickle
-from unittest import TestCase, TestSuite, TestLoader, TextTestRunner
-from pkg_resources import resource_stream  # @UnresolvedImport
-from babelfish import (LANGUAGES, Language, Country, Script, language_converters, country_converters,
-    LanguageReverseConverter, LanguageConvertError, LanguageReverseError, CountryReverseError)
-
-
-if sys.version_info[:2] <= (2, 6):
-    _MAX_LENGTH = 80
-
-    def safe_repr(obj, short=False):
-        try:
-            result = repr(obj)
-        except Exception:
-            result = object.__repr__(obj)
-        if not short or len(result) < _MAX_LENGTH:
-            return result
-        return result[:_MAX_LENGTH] + ' [truncated]...'
-
-    class _AssertRaisesContext(object):
-        """A context manager used to implement TestCase.assertRaises* methods."""
-
-        def __init__(self, expected, test_case, expected_regexp=None):
-            self.expected = expected
-            self.failureException = test_case.failureException
-            self.expected_regexp = expected_regexp
-
-        def __enter__(self):
-            return self
-
-        def __exit__(self, exc_type, exc_value, tb):
-            if exc_type is None:
-                try:
-                    exc_name = self.expected.__name__
-                except AttributeError:
-                    exc_name = str(self.expected)
-                raise self.failureException(
-                    "{0} not raised".format(exc_name))
-            if not issubclass(exc_type, self.expected):
-                # let unexpected exceptions pass through
-                return False
-            self.exception = exc_value  # store for later retrieval
-            if self.expected_regexp is None:
-                return True
-
-            expected_regexp = self.expected_regexp
-            if isinstance(expected_regexp, basestring):
-                expected_regexp = re.compile(expected_regexp)
-            if not expected_regexp.search(str(exc_value)):
-                raise self.failureException('"%s" does not match "%s"' %
-                         (expected_regexp.pattern, str(exc_value)))
-            return True
-
-    class _Py26FixTestCase(object):
-        def assertIsNone(self, obj, msg=None):
-            """Same as self.assertTrue(obj is None), with a nicer default message."""
-            if obj is not None:
-                standardMsg = '%s is not None' % (safe_repr(obj),)
-                self.fail(self._formatMessage(msg, standardMsg))
-
-        def assertIsNotNone(self, obj, msg=None):
-            """Included for symmetry with assertIsNone."""
-            if obj is None:
-                standardMsg = 'unexpectedly None'
-                self.fail(self._formatMessage(msg, standardMsg))
-
-        def assertIn(self, member, container, msg=None):
-            """Just like self.assertTrue(a in b), but with a nicer default message."""
-            if member not in container:
-                standardMsg = '%s not found in %s' % (safe_repr(member),
-                                                      safe_repr(container))
-                self.fail(self._formatMessage(msg, standardMsg))
-
-        def assertNotIn(self, member, container, msg=None):
-            """Just like self.assertTrue(a not in b), but with a nicer default message."""
-            if member in container:
-                standardMsg = '%s unexpectedly found in %s' % (safe_repr(member),
-                                                            safe_repr(container))
-                self.fail(self._formatMessage(msg, standardMsg))
-
-        def assertIs(self, expr1, expr2, msg=None):
-            """Just like self.assertTrue(a is b), but with a nicer default message."""
-            if expr1 is not expr2:
-                standardMsg = '%s is not %s' % (safe_repr(expr1),
-                                                 safe_repr(expr2))
-                self.fail(self._formatMessage(msg, standardMsg))
-
-        def assertIsNot(self, expr1, expr2, msg=None):
-            """Just like self.assertTrue(a is not b), but with a nicer default message."""
-            if expr1 is expr2:
-                standardMsg = 'unexpectedly identical: %s' % (safe_repr(expr1),)
-                self.fail(self._formatMessage(msg, standardMsg))
-
-else:
-    class _Py26FixTestCase(object):
-        pass
-
-
-class TestScript(TestCase, _Py26FixTestCase):
-    def test_wrong_script(self):
-        self.assertRaises(ValueError, lambda: Script('Azer'))
-
-    def test_eq(self):
-        self.assertEqual(Script('Latn'), Script('Latn'))
-
-    def test_ne(self):
-        self.assertNotEqual(Script('Cyrl'), Script('Latn'))
-
-    def test_hash(self):
-        self.assertEqual(hash(Script('Hira')), hash('Hira'))
-
-    def test_pickle(self):
-        self.assertEqual(pickle.loads(pickle.dumps(Script('Latn'))), Script('Latn'))
-
-
-class TestCountry(TestCase, _Py26FixTestCase):
-    def test_wrong_country(self):
-        self.assertRaises(ValueError, lambda: Country('ZZ'))
-
-    def test_eq(self):
-        self.assertEqual(Country('US'), Country('US'))
-
-    def test_ne(self):
-        self.assertNotEqual(Country('GB'), Country('US'))
-        self.assertIsNotNone(Country('US'))
-
-    def test_hash(self):
-        self.assertEqual(hash(Country('US')), hash('US'))
-
-    def test_pickle(self):
-        for country in [Country('GB'), Country('US')]:
-            self.assertEqual(pickle.loads(pickle.dumps(country)), country)
-
-    def test_converter_name(self):
-        self.assertEqual(Country('US').name, 'UNITED STATES')
-        self.assertEqual(Country.fromname('UNITED STATES'), Country('US'))
-        self.assertEqual(Country.fromcode('UNITED STATES', 'name'), Country('US'))
-        self.assertRaises(CountryReverseError, lambda: Country.fromname('ZZZZZ'))
-        self.assertEqual(len(country_converters['name'].codes), 249)
-
-
-class TestLanguage(TestCase, _Py26FixTestCase):
-    def test_languages(self):
-        self.assertEqual(len(LANGUAGES), 7874)
-
-    def test_wrong_language(self):
-        self.assertRaises(ValueError, lambda: Language('zzz'))
-
-    def test_unknown_language(self):
-        self.assertEqual(Language('zzzz', unknown='und'), Language('und'))
-
-    def test_converter_alpha2(self):
-        self.assertEqual(Language('eng').alpha2, 'en')
-        self.assertEqual(Language.fromalpha2('en'), Language('eng'))
-        self.assertEqual(Language.fromcode('en', 'alpha2'), Language('eng'))
-        self.assertRaises(LanguageReverseError, lambda: Language.fromalpha2('zz'))
-        self.assertRaises(LanguageConvertError, lambda: Language('aaa').alpha2)
-        self.assertEqual(len(language_converters['alpha2'].codes), 184)
-
-    def test_converter_alpha3b(self):
-        self.assertEqual(Language('fra').alpha3b, 'fre')
-        self.assertEqual(Language.fromalpha3b('fre'), Language('fra'))
-        self.assertEqual(Language.fromcode('fre', 'alpha3b'), Language('fra'))
-        self.assertRaises(LanguageReverseError, lambda: Language.fromalpha3b('zzz'))
-        self.assertRaises(LanguageConvertError, lambda: Language('aaa').alpha3b)
-        self.assertEqual(len(language_converters['alpha3b'].codes), 418)
-
-    def test_converter_alpha3t(self):
-        self.assertEqual(Language('fra').alpha3t, 'fra')
-        self.assertEqual(Language.fromalpha3t('fra'), Language('fra'))
-        self.assertEqual(Language.fromcode('fra', 'alpha3t'), Language('fra'))
-        self.assertRaises(LanguageReverseError, lambda: Language.fromalpha3t('zzz'))
-        self.assertRaises(LanguageConvertError, lambda: Language('aaa').alpha3t)
-        self.assertEqual(len(language_converters['alpha3t'].codes), 418)
-
-    def test_converter_name(self):
-        self.assertEqual(Language('eng').name, 'English')
-        self.assertEqual(Language.fromname('English'), Language('eng'))
-        self.assertEqual(Language.fromcode('English', 'name'), Language('eng'))
-        self.assertRaises(LanguageReverseError, lambda: Language.fromname('Zzzzzzzzz'))
-        self.assertEqual(len(language_converters['name'].codes), 7874)
-
-    def test_converter_scope(self):
-        self.assertEqual(language_converters['scope'].codes, set(['I', 'S', 'M']))
-        self.assertEqual(Language('eng').scope, 'individual')
-        self.assertEqual(Language('und').scope, 'special')
-
-    def test_converter_type(self):
-        self.assertEqual(language_converters['type'].codes, set(['A', 'C', 'E', 'H', 'L', 'S']))
-        self.assertEqual(Language('eng').type, 'living')
-        self.assertEqual(Language('und').type, 'special')
-
-    def test_converter_opensubtitles(self):
-        self.assertEqual(Language('fra').opensubtitles, Language('fra').alpha3b)
-        self.assertEqual(Language('por', 'BR').opensubtitles, 'pob')
-        self.assertEqual(Language.fromopensubtitles('fre'), Language('fra'))
-        self.assertEqual(Language.fromopensubtitles('pob'), Language('por', 'BR'))
-        self.assertEqual(Language.fromopensubtitles('pb'), Language('por', 'BR'))
-        # Montenegrin is not recognized as an ISO language (yet?) but for now it is
-        # unofficially accepted as Serbian from Montenegro
-        self.assertEqual(Language.fromopensubtitles('mne'), Language('srp', 'ME'))
-        self.assertEqual(Language.fromcode('pob', 'opensubtitles'), Language('por', 'BR'))
-        self.assertRaises(LanguageReverseError, lambda: Language.fromopensubtitles('zzz'))
-        self.assertRaises(LanguageConvertError, lambda: Language('aaa').opensubtitles)
-        self.assertEqual(len(language_converters['opensubtitles'].codes), 606)
-
-        # test with all the LANGUAGES from the opensubtitles api
-        # downloaded from: http://www.opensubtitles.org/addons/export_languages.php
-        f = resource_stream('babelfish', 'data/opensubtitles_languages.txt')
-        f.readline()
-        for l in f:
-            idlang, alpha2, _, upload_enabled, web_enabled = l.decode('utf-8').strip().split('\t')
-            if not int(upload_enabled) and not int(web_enabled):
-                # do not test LANGUAGES that are too esoteric / not widely available
-                continue
-            self.assertEqual(Language.fromopensubtitles(idlang).opensubtitles, idlang)
-            if alpha2:
-                self.assertEqual(Language.fromopensubtitles(idlang), Language.fromopensubtitles(alpha2))
-        f.close()
-
-    def test_fromietf_country_script(self):
-        language = Language.fromietf('fra-FR-Latn')
-        self.assertEqual(language.alpha3, 'fra')
-        self.assertEqual(language.country, Country('FR'))
-        self.assertEqual(language.script, Script('Latn'))
-
-    def test_fromietf_country_no_script(self):
-        language = Language.fromietf('fra-FR')
-        self.assertEqual(language.alpha3, 'fra')
-        self.assertEqual(language.country, Country('FR'))
-        self.assertIsNone(language.script)
-
-    def test_fromietf_no_country_no_script(self):
-        language = Language.fromietf('fra-FR')
-        self.assertEqual(language.alpha3, 'fra')
-        self.assertEqual(language.country, Country('FR'))
-        self.assertIsNone(language.script)
-
-    def test_fromietf_no_country_script(self):
-        language = Language.fromietf('fra-Latn')
-        self.assertEqual(language.alpha3, 'fra')
-        self.assertIsNone(language.country)
-        self.assertEqual(language.script, Script('Latn'))
-
-    def test_fromietf_alpha2_language(self):
-        language = Language.fromietf('fr-Latn')
-        self.assertEqual(language.alpha3, 'fra')
-        self.assertIsNone(language.country)
-        self.assertEqual(language.script, Script('Latn'))
-
-    def test_fromietf_wrong_language(self):
-        self.assertRaises(ValueError, lambda: Language.fromietf('xyz-FR'))
-
-    def test_fromietf_wrong_country(self):
-        self.assertRaises(ValueError, lambda: Language.fromietf('fra-YZ'))
-
-    def test_fromietf_wrong_script(self):
-        self.assertRaises(ValueError, lambda: Language.fromietf('fra-FR-Wxyz'))
-
-    def test_eq(self):
-        self.assertEqual(Language('eng'), Language('eng'))
-
-    def test_ne(self):
-        self.assertNotEqual(Language('fra'), Language('eng'))
-        self.assertIsNotNone(Language('fra'))
-
-    def test_nonzero(self):
-        self.assertFalse(bool(Language('und')))
-        self.assertTrue(bool(Language('eng')))
-
-    def test_language_hasattr(self):
-        self.assertTrue(hasattr(Language('fra'), 'alpha3'))
-        self.assertTrue(hasattr(Language('fra'), 'alpha2'))
-        self.assertFalse(hasattr(Language('bej'), 'alpha2'))
-
-    def test_country(self):
-        self.assertEqual(Language('por', 'BR').country, Country('BR'))
-        self.assertEqual(Language('eng', Country('US')).country, Country('US'))
-
-    def test_eq_with_country(self):
-        self.assertEqual(Language('eng', 'US'), Language('eng', Country('US')))
-
-    def test_ne_with_country(self):
-        self.assertNotEqual(Language('eng', 'US'), Language('eng', Country('GB')))
-
-    def test_script(self):
-        self.assertEqual(Language('srp', script='Latn').script, Script('Latn'))
-        self.assertEqual(Language('srp', script=Script('Cyrl')).script, Script('Cyrl'))
-
-    def test_eq_with_script(self):
-        self.assertEqual(Language('srp', script='Latn'), Language('srp', script=Script('Latn')))
-
-    def test_ne_with_script(self):
-        self.assertNotEqual(Language('srp', script='Latn'), Language('srp', script=Script('Cyrl')))
-
-    def test_eq_with_country_and_script(self):
-        self.assertEqual(Language('srp', 'SR', 'Latn'), Language('srp', Country('SR'), Script('Latn')))
-
-    def test_ne_with_country_and_script(self):
-        self.assertNotEqual(Language('srp', 'SR', 'Latn'), Language('srp', Country('SR'), Script('Cyrl')))
-
-    def test_hash(self):
-        self.assertEqual(hash(Language('fra')), hash('fr'))
-        self.assertEqual(hash(Language('ace')), hash('ace'))
-        self.assertEqual(hash(Language('por', 'BR')), hash('pt-BR'))
-        self.assertEqual(hash(Language('srp', script='Cyrl')), hash('sr-Cyrl'))
-        self.assertEqual(hash(Language('eng', 'US', 'Latn')), hash('en-US-Latn'))
-
-    def test_pickle(self):
-        for lang in [Language('fra'),
-                     Language('eng', 'US'),
-                     Language('srp', script='Latn'),
-                     Language('eng', 'US', 'Latn')]:
-            self.assertEqual(pickle.loads(pickle.dumps(lang)), lang)
-
-    def test_str(self):
-        self.assertEqual(Language.fromietf(str(Language('eng', 'US', 'Latn'))), Language('eng', 'US', 'Latn'))
-        self.assertEqual(Language.fromietf(str(Language('fra', 'FR'))), Language('fra', 'FR'))
-        self.assertEqual(Language.fromietf(str(Language('bel'))), Language('bel'))
-
-    def test_register_converter(self):
-        class TestConverter(LanguageReverseConverter):
-            def __init__(self):
-                self.to_test = {'fra': 'test1', 'eng': 'test2'}
-                self.from_test = {'test1': 'fra', 'test2': 'eng'}
-
-            def convert(self, alpha3, country=None, script=None):
-                if alpha3 not in self.to_test:
-                    raise LanguageConvertError(alpha3, country, script)
-                return self.to_test[alpha3]
-
-            def reverse(self, test):
-                if test not in self.from_test:
-                    raise LanguageReverseError(test)
-                return (self.from_test[test], None)
-        language = Language('fra')
-        self.assertFalse(hasattr(language, 'test'))
-        language_converters['test'] = TestConverter()
-        self.assertTrue(hasattr(language, 'test'))
-        self.assertIn('test', language_converters)
-        self.assertEqual(Language('fra').test, 'test1')
-        self.assertEqual(Language.fromtest('test2').alpha3, 'eng')
-        del language_converters['test']
-        self.assertNotIn('test', language_converters)
-        self.assertRaises(KeyError, lambda: Language.fromtest('test1'))
-        self.assertRaises(AttributeError, lambda: Language('fra').test)
-
-
-def suite():
-    suite = TestSuite()
-    suite.addTest(TestLoader().loadTestsFromTestCase(TestScript))
-    suite.addTest(TestLoader().loadTestsFromTestCase(TestCountry))
-    suite.addTest(TestLoader().loadTestsFromTestCase(TestLanguage))
-    return suite
-
-
-if __name__ == '__main__':
-    TextTestRunner().run(suite())
@@ -4,7 +4,6 @@
 # Use of this source code is governed by the 3-clause BSD license
 # that can be found in the LICENSE file.
 #
-from __future__ import unicode_literals
 from collections import namedtuple
 from functools import partial
 from pkg_resources import resource_stream  # @UnresolvedImport
@@ -4,7 +4,6 @@
 # Use of this source code is governed by the 3-clause BSD license
 # that can be found in the LICENSE file.
 #
-from __future__ import unicode_literals
 from collections import namedtuple
 from functools import partial
 from pkg_resources import resource_stream  # @UnresolvedImport
@@ -4,7 +4,6 @@
 # Use of this source code is governed by the 3-clause BSD license
 # that can be found in the LICENSE file.
 #
-from __future__ import unicode_literals
 from collections import namedtuple
 from pkg_resources import resource_stream  # @UnresolvedImport
 from . import basestr
@@ -5,26 +5,31 @@ http://www.crummy.com/software/BeautifulSoup/

 Beautiful Soup uses a pluggable XML or HTML parser to parse a
 (possibly invalid) document into a tree representation. Beautiful Soup
-provides provides methods and Pythonic idioms that make it easy to
-navigate, search, and modify the parse tree.
+provides methods and Pythonic idioms that make it easy to navigate,
+search, and modify the parse tree.

-Beautiful Soup works with Python 2.6 and up. It works better if lxml
+Beautiful Soup works with Python 2.7 and up. It works better if lxml
 and/or html5lib is installed.

 For more than you ever wanted to know about Beautiful Soup, see the
 documentation:
 http://www.crummy.com/software/BeautifulSoup/bs4/doc/
+
 """

+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
 __author__ = "Leonard Richardson (leonardr@segfault.org)"
-__version__ = "4.4.1"
-__copyright__ = "Copyright (c) 2004-2015 Leonard Richardson"
+__version__ = "4.6.0"
+__copyright__ = "Copyright (c) 2004-2017 Leonard Richardson"
 __license__ = "MIT"

 __all__ = ['BeautifulSoup']

 import os
 import re
+import traceback
 import warnings

 from .builder import builder_registry, ParserRejectedMarkup
@@ -77,7 +82,7 @@ class BeautifulSoup(Tag):

    ASCII_SPACES = '\x20\x0a\x09\x0c\x0d'

-    NO_PARSER_SPECIFIED_WARNING = "No parser was explicitly specified, so I'm using the best available %(markup_type)s parser for this system (\"%(parser)s\"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.\n\nTo get rid of this warning, change this:\n\n BeautifulSoup([your markup])\n\nto this:\n\n BeautifulSoup([your markup], \"%(parser)s\")\n"
+    NO_PARSER_SPECIFIED_WARNING = "No parser was explicitly specified, so I'm using the best available %(markup_type)s parser for this system (\"%(parser)s\"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.\n\nThe code that caused this warning is on line %(line_number)s of the file %(filename)s. To get rid of this warning, change code that looks like this:\n\n BeautifulSoup(YOUR_MARKUP})\n\nto this:\n\n BeautifulSoup(YOUR_MARKUP, \"%(parser)s\")\n"

    def __init__(self, markup="", features=None, builder=None,
                 parse_only=None, from_encoding=None, exclude_encodings=None,
@@ -137,6 +142,10 @@ class BeautifulSoup(Tag):
        from_encoding = from_encoding or deprecated_argument(
            "fromEncoding", "from_encoding")

+        if from_encoding and isinstance(markup, unicode):
+            warnings.warn("You provided Unicode markup but also provided a value for from_encoding. Your from_encoding will be ignored.")
+            from_encoding = None
+
        if len(kwargs) > 0:
            arg = kwargs.keys().pop()
            raise TypeError(
@@ -161,19 +170,29 @@ class BeautifulSoup(Tag):
                    markup_type = "XML"
                else:
                    markup_type = "HTML"
+
+                caller = traceback.extract_stack()[0]
+                filename = caller[0]
+                line_number = caller[1]
                warnings.warn(self.NO_PARSER_SPECIFIED_WARNING % dict(
+                    filename=filename,
+                    line_number=line_number,
                    parser=builder.NAME,
                    markup_type=markup_type))

        self.builder = builder
        self.is_xml = builder.is_xml
+        self.known_xml = self.is_xml
        self.builder.soup = self

        self.parse_only = parse_only

        if hasattr(markup, 'read'):        # It's a file-type object.
            markup = markup.read()
-        elif len(markup) <= 256:
+        elif len(markup) <= 256 and (
+                (isinstance(markup, bytes) and not b'<' in markup)
+                or (isinstance(markup, unicode) and not u'<' in markup)
+        ):
            # Print out warnings for a couple beginner problems
            # involving passing non-markup to Beautiful Soup.
            # Beautiful Soup will still parse the input as markup,
@@ -195,16 +214,10 @@ class BeautifulSoup(Tag):
                if isinstance(markup, unicode):
                    markup = markup.encode("utf8")
                warnings.warn(
-                    '"%s" looks like a filename, not markup. You should probably open this file and pass the filehandle into Beautiful Soup.' % markup)
-            if markup[:5] == "http:" or markup[:6] == "https:":
-                # TODO: This is ugly but I couldn't get it to work in
-                # Python 3 otherwise.
-                if ((isinstance(markup, bytes) and not b' ' in markup)
-                    or (isinstance(markup, unicode) and not u' ' in markup)):
-                    if isinstance(markup, unicode):
-                        markup = markup.encode("utf8")
-                    warnings.warn(
-                        '"%s" looks like a URL. Beautiful Soup is not an HTTP client. You should probably use an HTTP client to get the document behind the URL, and feed that document to Beautiful Soup.' % markup)
+                    '"%s" looks like a filename, not markup. You should'
+                    ' probably open this file and pass the filehandle into'
+                    ' Beautiful Soup.' % markup)
+            self._check_markup_is_url(markup)

        for (self.markup, self.original_encoding, self.declared_html_encoding,
         self.contains_replacement_characters) in (
@@ -223,15 +236,52 @@ class BeautifulSoup(Tag):
        self.builder.soup = None

    def __copy__(self):
-        return type(self)(self.encode(), builder=self.builder)
+        copy = type(self)(
+            self.encode('utf-8'), builder=self.builder, from_encoding='utf-8'
+        )
+
+        # Although we encoded the tree to UTF-8, that may not have
+        # been the encoding of the original markup. Set the copy's
+        # .original_encoding to reflect the original object's
+        # .original_encoding.
+        copy.original_encoding = self.original_encoding
+        return copy

    def __getstate__(self):
        # Frequently a tree builder can't be pickled.
        d = dict(self.__dict__)
        if 'builder' in d and not self.builder.picklable:
-            del d['builder']
+            d['builder'] = None
        return d

+    @staticmethod
+    def _check_markup_is_url(markup):
+        """ 
+        Check if markup looks like it's actually a url and raise a warning 
+        if so. Markup can be unicode or str (py2) / bytes (py3).
+        """
+        if isinstance(markup, bytes):
+            space = b' '
+            cant_start_with = (b"http:", b"https:")
+        elif isinstance(markup, unicode):
+            space = u' '
+            cant_start_with = (u"http:", u"https:")
+        else:
+            return
+
+        if any(markup.startswith(prefix) for prefix in cant_start_with):
+            if not space in markup:
+                if isinstance(markup, bytes):
+                    decoded_markup = markup.decode('utf-8', 'replace')
+                else:
+                    decoded_markup = markup
+                warnings.warn(
+                    '"%s" looks like a URL. Beautiful Soup is not an'
+                    ' HTTP client. You should probably use an HTTP client like'
+                    ' requests to get the document behind the URL, and feed'
+                    ' that document to Beautiful Soup.' % decoded_markup
+                )
+
    def _feed(self):
        # Convert the document to Unicode.
        self.builder.reset()
@@ -335,7 +385,18 @@ class BeautifulSoup(Tag):
        if parent.next_sibling:
            # This node is being inserted into an element that has
            # already been parsed. Deal with any dangling references.
-            index = parent.contents.index(o)
+            index = len(parent.contents)-1
+            while index >= 0:
+                if parent.contents[index] is o:
+                    break
+                index -= 1
+            else:
+                raise ValueError(
+                    "Error building tree: supposedly %r was inserted "
+                    "into %r after the fact, but I don't see it!" % (
+                        o, parent
+                    )
+                )
            if index == 0:
                previous_element = parent
                previous_sibling = None
@@ -387,7 +448,7 @@ class BeautifulSoup(Tag):
        """Push a start tag on to the stack.

        If this method returns None, the tag was rejected by the
-        SoupStrainer. You should proceed as if the tag had not occured
+        SoupStrainer. You should proceed as if the tag had not occurred
        in the document. For instance, if this was a self-closing tag,
        don't call handle_endtag.
        """
@@ -1,9 +1,13 @@
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
 from collections import defaultdict
 import itertools
 import sys
 from bs4.element import (
    CharsetMetaAttributeValue,
    ContentMetaAttributeValue,
+    HTMLAwareEntitySubstitution,
    whitespace_re
    )

@@ -227,9 +231,14 @@ class HTMLTreeBuilder(TreeBuilder):
    Such as which tags are empty-element tags.
    """

-    preserve_whitespace_tags = set(['pre', 'textarea'])
-    empty_element_tags = set(['br' , 'hr', 'input', 'img', 'meta',
-                              'spacer', 'link', 'frame', 'base'])
+    preserve_whitespace_tags = HTMLAwareEntitySubstitution.preserve_whitespace_tags
+    empty_element_tags = set([
+        # These are from HTML5.
+        'area', 'base', 'br', 'col', 'embed', 'hr', 'img', 'input', 'keygen', 'link', 'menuitem', 'meta', 'param', 'source', 'track', 'wbr',
+
+        # These are from HTML4, removed in HTML5.
+        'spacer', 'frame'
+    ])

    # The HTML standard defines these attributes as containing a
    # space-separated list of values, not a single value. That is,
@@ -1,9 +1,12 @@
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
 __all__ = [
    'HTML5TreeBuilder',
    ]

-from pdb import set_trace
 import warnings
+import re
 from bs4.builder import (
    PERMISSIVE,
    HTML,
@@ -15,7 +18,10 @@ from bs4.element import (
    whitespace_re,
 )
 import html5lib
-from html5lib.constants import namespaces
+from html5lib.constants import (
+    namespaces,
+    prefixes,
+    )
 from bs4.element import (
    Comment,
    Doctype,
@@ -23,6 +29,15 @@ from bs4.element import (
    Tag,
    )

+try:
+    # Pre-0.99999999
+    from html5lib.treebuilders import _base as treebuilder_base
+    new_html5lib = False
+except ImportError, e:
+    # 0.99999999 and up
+    from html5lib.treebuilders import base as treebuilder_base
+    new_html5lib = True
+
 class HTML5TreeBuilder(HTMLTreeBuilder):
    """Use html5lib to build a tree."""

@@ -47,7 +62,14 @@ class HTML5TreeBuilder(HTMLTreeBuilder):
        if self.soup.parse_only is not None:
            warnings.warn("You provided a value for parse_only, but the html5lib tree builder doesn't support parse_only. The entire document will be parsed.")
        parser = html5lib.HTMLParser(tree=self.create_treebuilder)
-        doc = parser.parse(markup, encoding=self.user_specified_encoding)
+
+        extra_kwargs = dict()
+        if not isinstance(markup, unicode):
+            if new_html5lib:
+                extra_kwargs['override_encoding'] = self.user_specified_encoding
+            else:
+                extra_kwargs['encoding'] = self.user_specified_encoding
+        doc = parser.parse(markup, **extra_kwargs)

        # Set the character encoding detected by the tokenizer.
        if isinstance(markup, unicode):
@@ -55,11 +77,17 @@ class HTML5TreeBuilder(HTMLTreeBuilder):
            # charEncoding to UTF-8 if it gets Unicode input.
            doc.original_encoding = None
        else:
-            doc.original_encoding = parser.tokenizer.stream.charEncoding[0]
+            original_encoding = parser.tokenizer.stream.charEncoding[0]
+            if not isinstance(original_encoding, basestring):
+                # In 0.99999999 and up, the encoding is an html5lib
+                # Encoding object. We want to use a string for compatibility
+                # with other tree builders.
+                original_encoding = original_encoding.name
+            doc.original_encoding = original_encoding

    def create_treebuilder(self, namespaceHTMLElements):
        self.underlying_builder = TreeBuilderForHtml5lib(
-            self.soup, namespaceHTMLElements)
+            namespaceHTMLElements, self.soup)
        return self.underlying_builder

    def test_fragment_to_document(self, fragment):
@@ -67,10 +95,14 @@ class HTML5TreeBuilder(HTMLTreeBuilder):
        return u'<html><head></head><body>%s</body></html>' % fragment


-class TreeBuilderForHtml5lib(html5lib.treebuilders._base.TreeBuilder):
+class TreeBuilderForHtml5lib(treebuilder_base.TreeBuilder):

-    def __init__(self, soup, namespaceHTMLElements):
-        self.soup = soup
+    def __init__(self, namespaceHTMLElements, soup=None):
+        if soup:
+            self.soup = soup
+        else:
+            from bs4 import BeautifulSoup
+            self.soup = BeautifulSoup("", "html.parser")
        super(TreeBuilderForHtml5lib, self).__init__(namespaceHTMLElements)

    def documentClass(self):
@@ -93,7 +125,8 @@ class TreeBuilderForHtml5lib(html5lib.treebuilders._base.TreeBuilder):
        return TextNode(Comment(data), self.soup)

    def fragmentClass(self):
-        self.soup = BeautifulSoup("")
+        from bs4 import BeautifulSoup
+        self.soup = BeautifulSoup("", "html.parser")
        self.soup.name = "[document_fragment]"
        return Element(self.soup, self.soup, None)

@@ -105,7 +138,57 @@ class TreeBuilderForHtml5lib(html5lib.treebuilders._base.TreeBuilder):
        return self.soup

    def getFragment(self):
-        return html5lib.treebuilders._base.TreeBuilder.getFragment(self).element
+        return treebuilder_base.TreeBuilder.getFragment(self).element
+
+    def testSerializer(self, element):
+        from bs4 import BeautifulSoup
+        rv = []
+        doctype_re = re.compile(r'^(.*?)(?: PUBLIC "(.*?)"(?: "(.*?)")?| SYSTEM "(.*?)")?$')
+
+        def serializeElement(element, indent=0):
+            if isinstance(element, BeautifulSoup):
+                pass
+            if isinstance(element, Doctype):
+                m = doctype_re.match(element)
+                if m:
+                    name = m.group(1)
+                    if m.lastindex > 1:
+                        publicId = m.group(2) or ""
+                        systemId = m.group(3) or m.group(4) or ""
+                        rv.append("""|%s<!DOCTYPE %s "%s" "%s">""" %
+                                  (' ' * indent, name, publicId, systemId))
+                    else:
+                        rv.append("|%s<!DOCTYPE %s>" % (' ' * indent, name))
+                else:
+                    rv.append("|%s<!DOCTYPE >" % (' ' * indent,))
+            elif isinstance(element, Comment):
+                rv.append("|%s<!-- %s -->" % (' ' * indent, element))
+            elif isinstance(element, NavigableString):
+                rv.append("|%s\"%s\"" % (' ' * indent, element))
+            else:
+                if element.namespace:
+                    name = "%s %s" % (prefixes[element.namespace],
+                                      element.name)
+                else:
+                    name = element.name
+                rv.append("|%s<%s>" % (' ' * indent, name))
+                if element.attrs:
+                    attributes = []
+                    for name, value in element.attrs.items():
+                        if isinstance(name, NamespacedAttribute):
+                            name = "%s %s" % (prefixes[name.namespace], name.name)
+                        if isinstance(value, list):
+                            value = " ".join(value)
+                        attributes.append((name, value))
+
+                    for name, value in sorted(attributes):
+                        rv.append('|%s%s="%s"' % (' ' * (indent + 2), name, value))
+                indent += 2
+                for child in element.children:
+                    serializeElement(child, indent)
+        serializeElement(element, 0)
+
+        return "\n".join(rv)

 class AttrList(object):
    def __init__(self, element):
@@ -137,9 +220,9 @@ class AttrList(object):
        return name in list(self.attrs.keys())


-class Element(html5lib.treebuilders._base.Node):
+class Element(treebuilder_base.Node):
    def __init__(self, element, soup, namespace):
-        html5lib.treebuilders._base.Node.__init__(self, element.name)
+        treebuilder_base.Node.__init__(self, element.name)
        self.element = element
        self.soup = soup
        self.namespace = namespace
@@ -158,8 +241,10 @@ class Element(html5lib.treebuilders._base.Node):
            child = node
        elif node.element.__class__ == NavigableString:
            string_child = child = node.element
+            node.parent = self
        else:
            child = node.element
+            node.parent = self

        if not isinstance(child, basestring) and child.parent is not None:
            node.element.extract()
@@ -197,6 +282,8 @@ class Element(html5lib.treebuilders._base.Node):
                most_recent_element=most_recent_element)

    def getAttributes(self):
+        if isinstance(self.element, Comment):
+            return {}
        return AttrList(self.element)

    def setAttributes(self, attributes):
@@ -224,11 +311,11 @@ class Element(html5lib.treebuilders._base.Node):
    attributes = property(getAttributes, setAttributes)

    def insertText(self, data, insertBefore=None):
+        text = TextNode(self.soup.new_string(data), self.soup)
        if insertBefore:
-            text = TextNode(self.soup.new_string(data), self.soup)
-            self.insertBefore(data, insertBefore)
+            self.insertBefore(text, insertBefore)
        else:
-            self.appendChild(data)
+            self.appendChild(text)

    def insertBefore(self, node, refNode):
        index = self.element.index(refNode.element)
@@ -250,6 +337,7 @@ class Element(html5lib.treebuilders._base.Node):
        # print "MOVE", self.element.contents
        # print "FROM", self.element
        # print "TO", new_parent.element
+
        element = self.element
        new_parent_element = new_parent.element
        # Determine what this tag's next_element will be once all the children
@@ -268,7 +356,6 @@ class Element(html5lib.treebuilders._base.Node):
            new_parents_last_descendant_next_element = new_parent_element.next_element

        to_append = element.contents
-        append_after = new_parent_element.contents
        if len(to_append) > 0:
            # Set the first child's previous_element and previous_sibling
            # to elements within the new parent
@@ -285,12 +372,19 @@ class Element(html5lib.treebuilders._base.Node):
            if new_parents_last_child:
                new_parents_last_child.next_sibling = first_child

-            # Fix the last child's next_element and next_sibling
-            last_child = to_append[-1]
-            last_child.next_element = new_parents_last_descendant_next_element
+            # Find the very last element being moved. It is now the
+            # parent's last descendant. It has no .next_sibling and
+            # its .next_element is whatever the previous last
+            # descendant had.
+            last_childs_last_descendant = to_append[-1]._last_descendant(False, True)
+
+            last_childs_last_descendant.next_element = new_parents_last_descendant_next_element
            if new_parents_last_descendant_next_element:
-                new_parents_last_descendant_next_element.previous_element = last_child
-            last_child.next_sibling = None
+                # TODO: This code has no test coverage and I'm not sure
+                # how to get html5lib to go through this path, but it's
+                # just the other side of the previous line.
+                new_parents_last_descendant_next_element.previous_element = last_childs_last_descendant
+            last_childs_last_descendant.next_sibling = None

        for child in to_append:
            child.parent = new_parent_element
@@ -324,7 +418,7 @@ class Element(html5lib.treebuilders._base.Node):

 class TextNode(Element):
    def __init__(self, element, soup):
-        html5lib.treebuilders._base.Node.__init__(self, None)
+        treebuilder_base.Node.__init__(self, None)
        self.element = element
        self.soup = soup

@@ -1,5 +1,8 @@
 """Use the HTMLParser library to parse HTML files that aren't too bad."""

+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
 __all__ = [
    'HTMLParserTreeBuilder',
    ]
@@ -49,7 +52,31 @@ from bs4.builder import (
 HTMLPARSER = 'html.parser'

 class BeautifulSoupHTMLParser(HTMLParser):
-    def handle_starttag(self, name, attrs):
+
+    def __init__(self, *args, **kwargs):
+        HTMLParser.__init__(self, *args, **kwargs)
+
+        # Keep a list of empty-element tags that were encountered
+        # without an explicit closing tag. If we encounter a closing tag
+        # of this type, we'll associate it with one of those entries.
+        #
+        # This isn't a stack because we don't care about the
+        # order. It's a list of closing tags we've already handled and
+        # will ignore, assuming they ever show up.
+        self.already_closed_empty_element = []
+    
+    def handle_startendtag(self, name, attrs):
+        # This is only called when the markup looks like
+        # <tag/>.
+
+        # is_startend() tells handle_starttag not to close the tag
+        # just because its name matches a known empty-element tag. We
+        # know that this is an empty-element tag and we want to call
+        # handle_endtag ourselves.
+        tag = self.handle_starttag(name, attrs, handle_empty_element=False)
+        self.handle_endtag(name)
+        
+    def handle_starttag(self, name, attrs, handle_empty_element=True):
        # XXX namespace
        attr_dict = {}
        for key, value in attrs:
@@ -59,10 +86,34 @@ class BeautifulSoupHTMLParser(HTMLParser):
                value = ''
            attr_dict[key] = value
            attrvalue = '""'
-        self.soup.handle_starttag(name, None, None, attr_dict)
+        #print "START", name
+        tag = self.soup.handle_starttag(name, None, None, attr_dict)
+        if tag and tag.is_empty_element and handle_empty_element:
+            # Unlike other parsers, html.parser doesn't send separate end tag
+            # events for empty-element tags. (It's handled in
+            # handle_startendtag, but only if the original markup looked like
+            # <tag/>.)
+            #
+            # So we need to call handle_endtag() ourselves. Since we
+            # know the start event is identical to the end event, we
+            # don't want handle_endtag() to cross off any previous end
+            # events for tags of this name.
+            self.handle_endtag(name, check_already_closed=False)

-    def handle_endtag(self, name):
-        self.soup.handle_endtag(name)
+            # But we might encounter an explicit closing tag for this tag
+            # later on. If so, we want to ignore it.
+            self.already_closed_empty_element.append(name)
+            
+    def handle_endtag(self, name, check_already_closed=True):
+        #print "END", name
+        if check_already_closed and name in self.already_closed_empty_element:
+            # This is a redundant end tag for an empty-element tag.
+            # We've already called handle_endtag() for it, so just
+            # check it off the list.
+            # print "ALREADY CLOSED", name
+            self.already_closed_empty_element.remove(name)
+        else:
+            self.soup.handle_endtag(name)

    def handle_data(self, data):
        self.soup.handle_data(data)
@@ -166,6 +217,7 @@ class HTMLParserTreeBuilder(HTMLTreeBuilder):
            warnings.warn(RuntimeWarning(
                "Python's built-in HTMLParser cannot parse the given document. This is not a bug in Beautiful Soup. The best solution is to install an external parser (lxml or html5lib), and use Beautiful Soup with that parser. See http://www.crummy.com/software/BeautifulSoup/bs4/doc/#installing-a-parser for help."))
            raise e
+        parser.already_closed_empty_element = []

 # Patch 3.2 versions of HTMLParser earlier than 3.2.3 to use some
 # 3.2.3 code. This ensures they don't treat markup like <p></p> as a
@@ -1,3 +1,5 @@
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
 __all__ = [
    'LXMLTreeBuilderForXML',
    'LXMLTreeBuilder',
@@ -12,6 +14,7 @@ from bs4.element import (
    Doctype,
    NamespacedAttribute,
    ProcessingInstruction,
+    XMLProcessingInstruction,
 )
 from bs4.builder import (
    FAST,
@@ -29,6 +32,7 @@ class LXMLTreeBuilderForXML(TreeBuilder):
    DEFAULT_PARSER_CLASS = etree.XMLParser

    is_xml = True
+    processing_instruction_class = XMLProcessingInstruction

    NAME = "lxml-xml"
    ALTERNATE_NAMES = ["xml"]
@@ -87,6 +91,16 @@ class LXMLTreeBuilderForXML(TreeBuilder):

        Each 4-tuple represents a strategy for parsing the document.
        """
+        # Instead of using UnicodeDammit to convert the bytestring to
+        # Unicode using different encodings, use EncodingDetector to
+        # iterate over the encodings, and tell lxml to try to parse
+        # the document as each one in turn.
+        is_html = not self.is_xml
+        if is_html:
+            self.processing_instruction_class = ProcessingInstruction
+        else:
+            self.processing_instruction_class = XMLProcessingInstruction
+
        if isinstance(markup, unicode):
            # We were given Unicode. Maybe lxml can parse Unicode on
            # this system?
@@ -98,11 +112,6 @@ class LXMLTreeBuilderForXML(TreeBuilder):
            yield (markup.encode("utf8"), "utf8",
                   document_declared_encoding, False)

-        # Instead of using UnicodeDammit to convert the bytestring to
-        # Unicode using different encodings, use EncodingDetector to
-        # iterate over the encodings, and tell lxml to try to parse
-        # the document as each one in turn.
-        is_html = not self.is_xml
        try_encodings = [user_specified_encoding, document_declared_encoding]
        detector = EncodingDetector(
            markup, try_encodings, is_html, exclude_encodings)
@@ -201,7 +210,7 @@ class LXMLTreeBuilderForXML(TreeBuilder):
    def pi(self, target, data):
        self.soup.endData()
        self.soup.handle_data(target + ' ' + data)
-        self.soup.endData(ProcessingInstruction)
+        self.soup.endData(self.processing_instruction_class)

    def data(self, content):
        self.soup.handle_data(content)
@@ -229,6 +238,7 @@ class LXMLTreeBuilder(HTMLTreeBuilder, LXMLTreeBuilderForXML):

    features = ALTERNATE_NAMES + [NAME, HTML, FAST, PERMISSIVE]
    is_xml = False
+    processing_instruction_class = ProcessingInstruction

    def default_parser(self, encoding):
        return etree.HTMLParser
@@ -6,9 +6,10 @@ necessary. It is heavily based on code from Mark Pilgrim's Universal
 Feed Parser. It works best on XML and HTML, but it does not rewrite the
 XML or HTML to reflect a new encoding; that's the tree builder's job.
 """
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
 __license__ = "MIT"

-from pdb import set_trace
 import codecs
 from htmlentitydefs import codepoint2name
 import re
@@ -309,7 +310,7 @@ class EncodingDetector:
        else:
            xml_endpos = 1024
            html_endpos = max(2048, int(len(markup) * 0.05))
-            
+
        declared_encoding = None
        declared_encoding_match = xml_encoding_re.search(markup, endpos=xml_endpos)
        if not declared_encoding_match and is_html:
@@ -346,7 +347,7 @@ class UnicodeDammit:
        self.tried_encodings = []
        self.contains_replacement_characters = False
        self.is_html = is_html
-
+        self.log = logging.getLogger(__name__)
        self.detector = EncodingDetector(
            markup, override_encodings, is_html, exclude_encodings)

@@ -376,9 +377,10 @@ class UnicodeDammit:
                if encoding != "ascii":
                    u = self._convert_from(encoding, "replace")
                if u is not None:
-                    logging.warning(
+                    self.log.warning(
                            "Some characters could not be decoded, and were "
-                            "replaced with REPLACEMENT CHARACTER.")
+                            "replaced with REPLACEMENT CHARACTER."
+                    )
                    self.contains_replacement_characters = True
                    break

@@ -734,7 +736,7 @@ class UnicodeDammit:
        0xde : b'\xc3\x9e',     # Þ
        0xdf : b'\xc3\x9f',     # ß
        0xe0 : b'\xc3\xa0',     # à
-        0xe1 : b'\xa1',     # á
+        0xe1 : b'\xa1',         # á
        0xe2 : b'\xc3\xa2',     # â
        0xe3 : b'\xc3\xa3',     # ã
        0xe4 : b'\xc3\xa4',     # ä
@@ -1,5 +1,7 @@
 """Diagnostic functions, mainly for use when doing tech support."""

+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
 __license__ = "MIT"

 import cProfile
@@ -56,7 +58,8 @@ def diagnose(data):
        data = data.read()
    elif os.path.exists(data):
        print '"%s" looks like a filename. Reading data from the file.' % data
-        data = open(data).read()
+        with open(data) as fp:
+            data = fp.read()
    elif data.startswith("http:") or data.startswith("https:"):
        print '"%s" looks like a URL. Beautiful Soup is not an HTTP client.' % data
        print "You need to use some other library to get the document behind the URL, and feed that document to Beautiful Soup."
@@ -1,8 +1,10 @@
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
 __license__ = "MIT"

-from pdb import set_trace
 import collections
 import re
+import shlex
 import sys
 import warnings
 from bs4.dammit import EntitySubstitution
@@ -99,6 +101,8 @@ class HTMLAwareEntitySubstitution(EntitySubstitution):

    preformatted_tags = set(["pre"])

+    preserve_whitespace_tags = set(['pre', 'textarea'])
+
    @classmethod
    def _substitute_if_appropriate(cls, ns, f):
        if (isinstance(ns, NavigableString)
@@ -127,8 +131,8 @@ class PageElement(object):
    # to methods like encode() and prettify():
    #
    # "html" - All Unicode characters with corresponding HTML entities
-    #   are converted to those entities on output.
-    # "minimal" - Bare ampersands and angle brackets are converted to
+    #   are converted to those entities on output. 
+   # "minimal" - Bare ampersands and angle brackets are converted to
    #   XML entities: &amp; &lt; &gt;
    # None - The null formatter. Unicode characters are never
    #   converted to entities.  This is not recommended, but it's
@@ -169,11 +173,19 @@ class PageElement(object):

        This is used when mapping a formatter name ("minimal") to an
        appropriate function (one that performs entity-substitution on
-        the contents of <script> and <style> tags, or not). It's
+        the contents of <script> and <style> tags, or not). It can be
        inefficient, but it should be called very rarely.
        """
+        if self.known_xml is not None:
+            # Most of the time we will have determined this when the
+            # document is parsed.
+            return self.known_xml
+
+        # Otherwise, it's likely that this element was created by
+        # direct invocation of the constructor from within the user's
+        # Python code.
        if self.parent is None:
-            # This is the top-level object. It should have .is_xml set
+            # This is the top-level object. It should have .known_xml set
            # from tree creation. If not, take a guess--BS is usually
            # used on HTML markup.
            return getattr(self, 'is_xml', False)
@@ -523,9 +535,16 @@ class PageElement(object):
                return ResultSet(strainer, result)
            elif isinstance(name, basestring):
                # Optimization to find all tags with a given name.
+                if name.count(':') == 1:
+                    # This is a name with a prefix.
+                    prefix, name = name.split(':', 1)
+                else:
+                    prefix = None
                result = (element for element in generator
                          if isinstance(element, Tag)
-                            and element.name == name)
+                            and element.name == name
+                          and (prefix is None or element.prefix == prefix)
+                )
                return ResultSet(strainer, result)
        results = ResultSet(strainer)
        while True:
@@ -637,7 +656,7 @@ class PageElement(object):
            return lambda el: el._attr_value_as_string(
                attribute, '').startswith(value)
        elif operator == '$':
-            # string represenation of `attribute` ends with `value`
+            # string representation of `attribute` ends with `value`
            return lambda el: el._attr_value_as_string(
                attribute, '').endswith(value)
        elif operator == '*':
@@ -677,6 +696,11 @@ class NavigableString(unicode, PageElement):
    PREFIX = ''
    SUFFIX = ''

+    # We can't tell just by looking at a string whether it's contained
+    # in an XML document or an HTML document.
+
+    known_xml = None
+
    def __new__(cls, value):
        """Create a new NavigableString.

@@ -743,10 +767,16 @@ class CData(PreformattedString):
    SUFFIX = u']]>'

 class ProcessingInstruction(PreformattedString):
+    """A SGML processing instruction."""

    PREFIX = u'<?'
    SUFFIX = u'>'

+class XMLProcessingInstruction(ProcessingInstruction):
+    """An XML processing instruction."""
+    PREFIX = u'<?'
+    SUFFIX = u'?>'
+
 class Comment(PreformattedString):

    PREFIX = u'<!--'
@@ -781,7 +811,8 @@ class Tag(PageElement):
    """Represents a found HTML tag with its attributes and contents."""

    def __init__(self, parser=None, builder=None, name=None, namespace=None,
-                 prefix=None, attrs=None, parent=None, previous=None):
+                 prefix=None, attrs=None, parent=None, previous=None,
+                 is_xml=None):
        "Basic constructor."

        if parser is None:
@@ -795,6 +826,14 @@ class Tag(PageElement):
        self.name = name
        self.namespace = namespace
        self.prefix = prefix
+        if builder is not None:
+            preserve_whitespace_tags = builder.preserve_whitespace_tags
+        else:
+            if is_xml:
+                preserve_whitespace_tags = []
+            else:
+                preserve_whitespace_tags = HTMLAwareEntitySubstitution.preserve_whitespace_tags
+        self.preserve_whitespace_tags = preserve_whitespace_tags
        if attrs is None:
            attrs = {}
        elif attrs:
@@ -805,6 +844,13 @@ class Tag(PageElement):
                attrs = dict(attrs)
        else:
            attrs = dict(attrs)
+
+        # If possible, determine ahead of time whether this tag is an
+        # XML tag.
+        if builder:
+            self.known_xml = builder.is_xml
+        else:
+            self.known_xml = is_xml
        self.attrs = attrs
        self.contents = []
        self.setup(parent, previous)
@@ -824,7 +870,7 @@ class Tag(PageElement):
        Its contents are a copy of the old Tag's contents.
        """
        clone = type(self)(None, self.builder, self.name, self.namespace,
-                           self.nsprefix, self.attrs)
+                           self.prefix, self.attrs, is_xml=self._is_xml)
        for attr in ('can_be_empty_element', 'hidden'):
            setattr(clone, attr, getattr(self, attr))
        for child in self.contents:
@@ -946,6 +992,13 @@ class Tag(PageElement):
        attribute."""
        return self.attrs.get(key, default)

+    def get_attribute_list(self, key, default=None):
+        """The same as get(), but always returns a list."""
+        value = self.get(key, default)
+        if not isinstance(value, list):
+            value = [value]
+        return value
+    
    def has_attr(self, key):
        return key in self.attrs

@@ -997,7 +1050,7 @@ class Tag(PageElement):
                    tag_name, tag_name))
            return self.find(tag_name)
        # We special case contents to avoid recursion.
-        elif not tag.startswith("__") and not tag=="contents":
+        elif not tag.startswith("__") and not tag == "contents":
            return self.find(tag)
        raise AttributeError(
            "'%s' object has no attribute '%s'" % (self.__class__, tag))
@@ -1057,10 +1110,11 @@ class Tag(PageElement):

    def _should_pretty_print(self, indent_level):
        """Should this tag be pretty-printed?"""
+
        return (
-            indent_level is not None and
-            (self.name not in HTMLAwareEntitySubstitution.preformatted_tags
-             or self._is_xml))
+            indent_level is not None
+            and self.name not in self.preserve_whitespace_tags
+        )

    def decode(self, indent_level=None,
               eventual_encoding=DEFAULT_OUTPUT_ENCODING,
@@ -1280,6 +1334,7 @@ class Tag(PageElement):

    _selector_combinators = ['>', '+', '~']
    _select_debug = False
+    quoted_colon = re.compile('"[^"]*:[^"]*"')
    def select_one(self, selector):
        """Perform a CSS selection operation on the current element."""
        value = self.select(selector, limit=1)
@@ -1305,8 +1360,7 @@ class Tag(PageElement):
                if limit and len(context) >= limit:
                    break
            return context
-
-        tokens = selector.split()
+        tokens = shlex.split(selector)
        current_context = [self]

        if tokens[-1] in self._selector_combinators:
@@ -1358,7 +1412,7 @@ class Tag(PageElement):
                    return classes.issubset(candidate.get('class', []))
                checker = classes_match

-            elif ':' in token:
+            elif ':' in token and not self.quoted_colon.search(token):
                # Pseudo-class
                tag_name, pseudo = token.split(':', 1)
                if tag_name == '':
@@ -1389,11 +1443,8 @@ class Tag(PageElement):
                            self.count += 1
                            if self.count == self.destination:
                                return True
-                            if self.count > self.destination:
-                                # Stop the generator that's sending us
-                                # these things.
-                                raise StopIteration()
-                            return False
+                            else:
+                                return False
                    checker = Counter(pseudo_value).nth_child_of_type
                else:
                    raise NotImplementedError(
@@ -1498,13 +1549,12 @@ class Tag(PageElement):
                            # don't include it in the context more than once.
                            new_context.append(candidate)
                            new_context_ids.add(id(candidate))
-                            if limit and len(new_context) >= limit:
-                                break
                    elif self._select_debug:
                        print "     FAILURE %s %s" % (candidate.name, repr(candidate.attrs))

-
            current_context = new_context
+        if limit and len(current_context) >= limit:
+            current_context = current_context[:limit]

        if self._select_debug:
            print "Final verdict:"
@@ -1662,28 +1712,22 @@ class SoupStrainer(object):
                "I don't know how to match against a %s" % markup.__class__)
        return found

-    def _matches(self, markup, match_against):
+    def _matches(self, markup, match_against, already_tried=None):
        # print u"Matching %s against %s" % (markup, match_against)
        result = False
        if isinstance(markup, list) or isinstance(markup, tuple):
            # This should only happen when searching a multi-valued attribute
            # like 'class'.
-            if (isinstance(match_against, unicode)
-                and ' ' in match_against):
-                # A bit of a special case. If they try to match "foo
-                # bar" on a multivalue attribute's value, only accept
-                # the literal value "foo bar"
-                #
-                # XXX This is going to be pretty slow because we keep
-                # splitting match_against. But it shouldn't come up
-                # too often.
-                return (whitespace_re.split(match_against) == markup)
-            else:
-                for item in markup:
-                    if self._matches(item, match_against):
-                        return True
-                return False
-
+            for item in markup:
+                if self._matches(item, match_against):
+                    return True
+            # We didn't match any particular value of the multivalue
+            # attribute, but maybe we match the attribute value when
+            # considered as a string.
+            if self._matches(' '.join(markup), match_against):
+                return True
+            return False
+        
        if match_against is True:
            # True matches any non-None value.
            return markup is not None
@@ -1693,6 +1737,7 @@ class SoupStrainer(object):

        # Custom callables take the tag as an argument, but all
        # other ways of matching match the tag name as a string.
+        original_markup = markup
        if isinstance(markup, Tag):
            markup = markup.name

@@ -1703,18 +1748,51 @@ class SoupStrainer(object):
            # None matches None, False, an empty string, an empty list, and so on.
            return not match_against

-        if isinstance(match_against, unicode):
+        if (hasattr(match_against, '__iter__')
+            and not isinstance(match_against, basestring)):
+            # We're asked to match against an iterable of items.
+            # The markup must be match at least one item in the
+            # iterable. We'll try each one in turn.
+            #
+            # To avoid infinite recursion we need to keep track of
+            # items we've already seen.
+            if not already_tried:
+                already_tried = set()
+            for item in match_against:
+                if item.__hash__:
+                    key = item
+                else:
+                    key = id(item)
+                if key in already_tried:
+                    continue
+                else:
+                    already_tried.add(key)
+                    if self._matches(original_markup, item, already_tried):
+                        return True
+            else:
+                return False
+        
+        # Beyond this point we might need to run the test twice: once against
+        # the tag's name and once against its prefixed name.
+        match = False
+        
+        if not match and isinstance(match_against, unicode):
            # Exact string match
-            return markup == match_against
+            match = markup == match_against

-        if hasattr(match_against, 'match'):
+        if not match and hasattr(match_against, 'search'):
            # Regexp match
            return match_against.search(markup)

-        if hasattr(match_against, '__iter__'):
-            # The markup must be an exact match against something
-            # in the iterable.
-            return markup in match_against
+        if (not match
+            and isinstance(original_markup, Tag)
+            and original_markup.prefix):
+            # Try the whole thing again with the prefixed tag name.
+            return self._matches(
+                original_markup.prefix + ':' + original_markup.name, match_against
+            )
+
+        return match


 class ResultSet(list):
@@ -1723,3 +1801,8 @@ class ResultSet(list):
    def __init__(self, source, result=()):
        super(ResultSet, self).__init__(result)
        self.source = source
+
+    def __getattr__(self, key):
+        raise AttributeError(
+            "ResultSet object has no attribute '%s'. You're probably treating a list of items like a single item. Did you call find_all() when you meant to call find()?" % key
+        )
@@ -1,5 +1,7 @@
 """Helper classes for tests."""

+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
 __license__ = "MIT"

 import pickle
@@ -67,6 +69,18 @@ class HTMLTreeBuilderSmokeTest(object):
    markup in these tests, there's not much room for interpretation.
    """

+    def test_empty_element_tags(self):
+        """Verify that all HTML4 and HTML5 empty element (aka void element) tags
+        are handled correctly.
+        """
+        for name in [
+                'area', 'base', 'br', 'col', 'embed', 'hr', 'img', 'input', 'keygen', 'link', 'menuitem', 'meta', 'param', 'source', 'track', 'wbr',
+                'spacer', 'frame'
+        ]:
+            soup = self.soup("")
+            new_tag = soup.new_tag(name)
+            self.assertEqual(True, new_tag.is_empty_element)
+    
    def test_pickle_and_unpickle_identity(self):
        # Pickling a tree, then unpickling it, yields a tree identical
        # to the original.
@@ -137,6 +151,14 @@ class HTMLTreeBuilderSmokeTest(object):
            markup.replace(b"\n", b""))

    def test_processing_instruction(self):
+        # We test both Unicode and bytestring to verify that
+        # process_markup correctly sets processing_instruction_class
+        # even when the markup is already Unicode and there is no
+        # need to process anything.
+        markup = u"""<?PITarget PIContent?>"""
+        soup = self.soup(markup)
+        self.assertEqual(markup, soup.decode())
+
        markup = b"""<?PITarget PIContent?>"""
        soup = self.soup(markup)
        self.assertEqual(markup, soup.encode("utf8"))
@@ -215,9 +237,22 @@ Hello, world!
        self.assertEqual(comment, baz.previous_element)

    def test_preserved_whitespace_in_pre_and_textarea(self):
-        """Whitespace must be preserved in <pre> and <textarea> tags."""
-        self.assertSoupEquals("<pre>   </pre>")
-        self.assertSoupEquals("<textarea> woo  </textarea>")
+        """Whitespace must be preserved in <pre> and <textarea> tags,
+        even if that would mean not prettifying the markup.
+        """
+        pre_markup = "<pre>   </pre>"
+        textarea_markup = "<textarea> woo\nwoo  </textarea>"
+        self.assertSoupEquals(pre_markup)
+        self.assertSoupEquals(textarea_markup)
+
+        soup = self.soup(pre_markup)
+        self.assertEqual(soup.pre.prettify(), pre_markup)
+
+        soup = self.soup(textarea_markup)
+        self.assertEqual(soup.textarea.prettify(), textarea_markup)
+
+        soup = self.soup("<textarea></textarea>")
+        self.assertEqual(soup.textarea.prettify(), "<textarea></textarea>")

    def test_nested_inline_elements(self):
        """Inline elements can be nested indefinitely."""
@@ -307,6 +342,13 @@ Hello, world!
        self.assertEqual("p", soup.p.name)
        self.assertConnectedness(soup)

+    def test_empty_element_tags(self):
+        """Verify consistent handling of empty-element tags,
+        no matter how they come in through the markup.
+        """
+        self.assertSoupEquals('<br/><br/><br/>', "<br/><br/><br/>")
+        self.assertSoupEquals('<br /><br /><br />', "<br/><br/><br/>")
+        
    def test_head_tag_between_head_and_body(self):
        "Prevent recurrence of a bug in the html5lib treebuilder."
        content = """<html><head></head>
@@ -480,7 +522,9 @@ Hello, world!
        hebrew_document = b'<html><head><title>Hebrew (ISO 8859-8) in Visual Directionality</title></head><body><h1>Hebrew (ISO 8859-8) in Visual Directionality</h1>\xed\xe5\xec\xf9</body></html>'
        soup = self.soup(
            hebrew_document, from_encoding="iso8859-8")
-        self.assertEqual(soup.original_encoding, 'iso8859-8')
+        # Some tree builders call it iso8859-8, others call it iso-8859-9.
+        # That's not a difference we really care about.
+        assert soup.original_encoding in ('iso8859-8', 'iso-8859-8')
        self.assertEqual(
            soup.encode('utf-8'),
            hebrew_document.decode("iso8859-8").encode("utf-8"))
@@ -563,6 +607,11 @@ class XMLTreeBuilderSmokeTest(object):
        soup = self.soup(markup)
        self.assertEqual(markup, soup.encode("utf8"))

+    def test_processing_instruction(self):
+        markup = b"""<?xml version="1.0" encoding="utf8"?>\n<?PITarget PIContent?>"""
+        soup = self.soup(markup)
+        self.assertEqual(markup, soup.encode("utf8"))
+
    def test_real_xhtml_document(self):
        """A real XHTML document should come out *exactly* the same as it went in."""
        markup = b"""<?xml version="1.0" encoding="utf-8"?>
@@ -639,6 +688,40 @@ class XMLTreeBuilderSmokeTest(object):
        soup = self.soup(markup)
        self.assertEqual(unicode(soup.foo), markup)

+    def test_find_by_prefixed_name(self):
+        doc = """<?xml version="1.0" encoding="utf-8"?>
+<Document xmlns="http://example.com/ns0"
+    xmlns:ns1="http://example.com/ns1"
+    xmlns:ns2="http://example.com/ns2"
+    <ns1:tag>foo</ns1:tag>
+    <ns1:tag>bar</ns1:tag>
+    <ns2:tag key="value">baz</ns2:tag>
+</Document>
+"""
+        soup = self.soup(doc)
+
+        # There are three <tag> tags.
+        self.assertEqual(3, len(soup.find_all('tag')))
+
+        # But two of them are ns1:tag and one of them is ns2:tag.
+        self.assertEqual(2, len(soup.find_all('ns1:tag')))
+        self.assertEqual(1, len(soup.find_all('ns2:tag')))
+        
+        self.assertEqual(1, len(soup.find_all('ns2:tag', key='value')))
+        self.assertEqual(3, len(soup.find_all(['ns1:tag', 'ns2:tag'])))
+        
+    def test_copy_tag_preserves_namespace(self):
+        xml = """<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
+<w:document xmlns:w="http://example.com/ns0"/>"""
+    
+        soup = self.soup(xml)
+        tag = soup.document
+        duplicate = copy.copy(tag)
+
+        # The two tags have the same namespace prefix.
+        self.assertEqual(tag.prefix, duplicate.prefix)
+
+
 class HTML5TreeBuilderSmokeTest(HTMLTreeBuilderSmokeTest):
    """Smoke test for a tree builder that supports HTML5."""

@@ -84,6 +84,33 @@ class HTML5LibBuilderSmokeTest(SoupTest, HTML5TreeBuilderSmokeTest):
        self.assertEqual(u"<body><p><em>foo</em></p><em>\n</em><p><em>bar<a></a></em></p>\n</body>", soup.body.decode())
        self.assertEqual(2, len(soup.find_all('p')))

+    def test_reparented_markup_containing_identical_whitespace_nodes(self):
+        """Verify that we keep the two whitespace nodes in this
+        document distinct when reparenting the adjacent <tbody> tags.
+        """
+        markup = '<table> <tbody><tbody><ims></tbody> </table>'
+        soup = self.soup(markup)
+        space1, space2 = soup.find_all(string=' ')
+        tbody1, tbody2 = soup.find_all('tbody')
+        assert space1.next_element is tbody1
+        assert tbody2.next_element is space2
+
+    def test_reparented_markup_containing_children(self):
+        markup = '<div><a>aftermath<p><noscript>target</noscript>aftermath</a></p></div>'
+        soup = self.soup(markup)
+        noscript = soup.noscript
+        self.assertEqual("target", noscript.next_element)
+        target = soup.find(string='target')
+
+        # The 'aftermath' string was duplicated; we want the second one.
+        final_aftermath = soup.find_all(string='aftermath')[-1]
+
+        # The <noscript> tag was moved beneath a copy of the <a> tag,
+        # but the 'target' string within is still connected to the
+        # (second) 'aftermath' string.
+        self.assertEqual(final_aftermath, target.next_element)
+        self.assertEqual(target, final_aftermath.previous_element)
+        
    def test_processing_instruction(self):
        """Processing instructions become comments."""
        markup = b"""<?PITarget PIContent?>"""
@@ -96,3 +123,8 @@ class HTML5LibBuilderSmokeTest(SoupTest, HTML5TreeBuilderSmokeTest):
        a1, a2 = soup.find_all('a')
        self.assertEqual(a1, a2)
        assert a1 is not a2
+
+    def test_foster_parenting(self):
+        markup = b"""<table><td></tbody>A"""
+        soup = self.soup(markup)
+        self.assertEqual(u"<body>A<table><tbody><tr><td></td></tr></tbody></table></body>", soup.body.decode())
@@ -29,4 +29,6 @@ class HTMLParserTreeBuilderSmokeTest(SoupTest, HTMLTreeBuilderSmokeTest):
        loaded = pickle.loads(dumped)
        self.assertTrue(isinstance(loaded.builder, type(tree.builder)))

-
+    def test_redundant_empty_element_closing_tags(self):
+        self.assertSoupEquals('<br></br><br></br><br></br>', "<br/><br/><br/>")
+        self.assertSoupEquals('</br></br></br>', "")
@@ -35,7 +35,6 @@ try:
 except ImportError, e:
    LXML_PRESENT = False

-PYTHON_2_PRE_2_7 = (sys.version_info < (2,7))
 PYTHON_3_PRE_3_2 = (sys.version_info[0] == 3 and sys.version_info < (3,2))

 class TestConstructor(SoupTest):
@@ -77,7 +76,7 @@ class TestWarnings(SoupTest):
    def test_no_warning_if_explicit_parser_specified(self):
        with warnings.catch_warnings(record=True) as w:
            soup = self.soup("<a><b></b></a>", "html.parser")
-        self.assertEquals([], w)
+        self.assertEqual([], w)

    def test_parseOnlyThese_renamed_to_parse_only(self):
        with warnings.catch_warnings(record=True) as w:
@@ -118,15 +117,34 @@ class TestWarnings(SoupTest):
            soup = self.soup(filename)
        self.assertEqual(0, len(w))

-    def test_url_warning(self):
-        with warnings.catch_warnings(record=True) as w:
-            soup = self.soup("http://www.crummy.com/")
-        msg = str(w[0].message)
-        self.assertTrue("looks like a URL" in msg)
+    def test_url_warning_with_bytes_url(self):
+        with warnings.catch_warnings(record=True) as warning_list:
+            soup = self.soup(b"http://www.crummybytes.com/")
+        # Be aware this isn't the only warning that can be raised during
+        # execution..
+        self.assertTrue(any("looks like a URL" in str(w.message) 
+            for w in warning_list))
+
+    def test_url_warning_with_unicode_url(self):
+        with warnings.catch_warnings(record=True) as warning_list:
+            # note - this url must differ from the bytes one otherwise
+            # python's warnings system swallows the second warning
+            soup = self.soup(u"http://www.crummyunicode.com/")
+        self.assertTrue(any("looks like a URL" in str(w.message) 
+            for w in warning_list))
+
+    def test_url_warning_with_bytes_and_space(self):
+        with warnings.catch_warnings(record=True) as warning_list:
+            soup = self.soup(b"http://www.crummybytes.com/ is great")
+        self.assertFalse(any("looks like a URL" in str(w.message) 
+            for w in warning_list))
+
+    def test_url_warning_with_unicode_and_space(self):
+        with warnings.catch_warnings(record=True) as warning_list:
+            soup = self.soup(u"http://www.crummyuncode.com/ is great")
+        self.assertFalse(any("looks like a URL" in str(w.message) 
+            for w in warning_list))

-        with warnings.catch_warnings(record=True) as w:
-            soup = self.soup("http://www.crummy.com/ is great")
-        self.assertEqual(0, len(w))

 class TestSelectiveParsing(SoupTest):

@@ -260,7 +278,7 @@ class TestEncodingConversion(SoupTest):
        self.assertEqual(soup_from_unicode.encode('utf-8'), self.utf8_data)

    @skipIf(
-        PYTHON_2_PRE_2_7 or PYTHON_3_PRE_3_2,
+        PYTHON_3_PRE_3_2,
        "Bad HTMLParser detected; skipping test of non-ASCII characters in attribute name.")
    def test_attribute_name_containing_unicode_characters(self):
        markup = u'<div><a \N{SNOWMAN}="snowman"></a></div>'
@@ -1,3 +1,4 @@
+
 # -*- coding: utf-8 -*-
 """Tests for Beautiful Soup's tree traversal methods.

@@ -222,7 +223,19 @@ class TestFindAllByName(TreeTest):
        self.assertSelects(
            tree.find_all(id_matches_name), ["Match 1.", "Match 2."])

+    def test_find_with_multi_valued_attribute(self):
+        soup = self.soup(
+            "<div class='a b'>1</div><div class='a c'>2</div><div class='a d'>3</div>"
+        )
+        r1 = soup.find('div', 'a d');
+        r2 = soup.find('div', re.compile(r'a d'));
+        r3, r4 = soup.find_all('div', ['a b', 'a d']);
+        self.assertEqual('3', r1.string)
+        self.assertEqual('3', r2.string)
+        self.assertEqual('1', r3.string)
+        self.assertEqual('3', r4.string)

+        
 class TestFindAllByAttribute(TreeTest):

    def test_find_all_by_attribute_name(self):
@@ -294,10 +307,10 @@ class TestFindAllByAttribute(TreeTest):
        f = tree.find_all("gar", class_=re.compile("a"))
        self.assertSelects(f, ["Found it"])

-        # Since the class is not the string "foo bar", but the two
-        # strings "foo" and "bar", this will not find anything.
+        # If the search fails to match the individual strings "foo" and "bar",
+        # it will be tried against the combined string "foo bar".
        f = tree.find_all("gar", class_=re.compile("o b"))
-        self.assertSelects(f, [])
+        self.assertSelects(f, ["Found it"])

    def test_find_all_with_non_dictionary_for_attrs_finds_by_class(self):
        soup = self.soup("<a class='bar'>Found it</a>")
@@ -335,7 +348,7 @@ class TestFindAllByAttribute(TreeTest):
        strainer = SoupStrainer(attrs={'id' : 'first'})
        self.assertSelects(tree.find_all(strainer), ['Match.'])

-    def test_find_all_with_missing_atribute(self):
+    def test_find_all_with_missing_attribute(self):
        # You can pass in None as the value of an attribute to find_all.
        # This will match tags that do not have that attribute set.
        tree = self.soup("""<a id="1">ID present.</a>
@@ -1273,6 +1286,10 @@ class TestCDAtaListAttributes(SoupTest):
        soup = self.soup("<a class='foo\tbar'>")
        self.assertEqual(b'<a class="foo bar"></a>', soup.a.encode())

+    def test_get_attribute_list(self):
+        soup = self.soup("<a id='abc def'>")
+        self.assertEqual(['abc def'], soup.a.get_attribute_list('id'))
+        
    def test_accept_charset(self):
        soup = self.soup('<form accept-charset="ISO-8859-1 UTF-8">')
        self.assertEqual(['ISO-8859-1', 'UTF-8'], soup.form['accept-charset'])
@@ -1328,6 +1345,13 @@ class TestPersistence(SoupTest):
        copied = copy.deepcopy(self.tree)
        self.assertEqual(copied.decode(), self.tree.decode())

+    def test_copy_preserves_encoding(self):
+        soup = BeautifulSoup(b'<p>&nbsp;</p>', 'html.parser')
+        encoding = soup.original_encoding
+        copy = soup.__copy__()
+        self.assertEqual(u"<p> </p>", unicode(copy))
+        self.assertEqual(encoding, copy.original_encoding)
+
    def test_unicode_pickle(self):
        # A tree containing Unicode characters can be pickled.
        html = u"<b>\N{SNOWMAN}</b>"
@@ -1676,8 +1700,8 @@ class TestSoupSelector(TreeTest):
    def setUp(self):
        self.soup = BeautifulSoup(self.HTML, 'html.parser')

-    def assertSelects(self, selector, expected_ids):
-        el_ids = [el['id'] for el in self.soup.select(selector)]
+    def assertSelects(self, selector, expected_ids, **kwargs):
+        el_ids = [el['id'] for el in self.soup.select(selector, **kwargs)]
        el_ids.sort()
        expected_ids.sort()
        self.assertEqual(expected_ids, el_ids,
@@ -1720,6 +1744,13 @@ class TestSoupSelector(TreeTest):
        for selector in ('html div', 'html body div', 'body div'):
            self.assertSelects(selector, ['data1', 'main', 'inner', 'footer'])

+
+    def test_limit(self):
+        self.assertSelects('html div', ['main'], limit=1)
+        self.assertSelects('html body div', ['inner', 'main'], limit=2)
+        self.assertSelects('body div', ['data1', 'main', 'inner', 'footer'],
+                           limit=10)
+
    def test_tag_no_match(self):
        self.assertEqual(len(self.soup.select('del')), 0)

@@ -1902,6 +1933,14 @@ class TestSoupSelector(TreeTest):
            ('div[data-tag]', ['data1'])
        )

+    def test_quoted_space_in_selector_name(self):
+        html = """<div style="display: wrong">nope</div>
+        <div style="display: right">yes</div>
+        """
+        soup = BeautifulSoup(html, 'html.parser')
+        [chosen] = soup.select('div[style="display: right"]')
+        self.assertEqual("yes", chosen.string)
+
    def test_unsupported_pseudoclass(self):
        self.assertRaises(
            NotImplementedError, self.soup.select, "a:no-such-pseudoclass")
@@ -1,3 +1,3 @@
 from .core import where, old_where

-__version__ = "2017.04.17"
+__version__ = "2018.01.18"
@@ -19,18 +19,19 @@ class DeprecatedBundleWarning(DeprecationWarning):


 def where():
-    f = os.path.split(__file__)[0]
+    f = os.path.dirname(__file__)

    return os.path.join(f, 'cacert.pem')


 def old_where():
    warnings.warn(
-        "The weak security bundle is being deprecated.",
+        "The weak security bundle has been removed. certifi.old_where() is now an alias "
+        "of certifi.where(). Please update your code to use certifi.where() instead. "
+        "certifi.old_where() will be removed in 2018.",
        DeprecatedBundleWarning
    )
-    f = os.path.split(__file__)[0]
-    return os.path.join(f, 'weak.pem')
+    return where()

 if __name__ == '__main__':
    print(where())
@@ -0,0 +1,436 @@
+"""contextlib2 - backports and enhancements to the contextlib module"""
+
+import sys
+import warnings
+from collections import deque
+from functools import wraps
+
+__all__ = ["contextmanager", "closing", "ContextDecorator", "ExitStack",
+           "redirect_stdout", "redirect_stderr", "suppress"]
+
+# Backwards compatibility
+__all__ += ["ContextStack"]
+
+class ContextDecorator(object):
+    "A base class or mixin that enables context managers to work as decorators."
+
+    def refresh_cm(self):
+        """Returns the context manager used to actually wrap the call to the
+        decorated function.
+
+        The default implementation just returns *self*.
+
+        Overriding this method allows otherwise one-shot context managers
+        like _GeneratorContextManager to support use as decorators via
+        implicit recreation.
+
+        DEPRECATED: refresh_cm was never added to the standard library's
+                    ContextDecorator API
+        """
+        warnings.warn("refresh_cm was never added to the standard library",
+                      DeprecationWarning)
+        return self._recreate_cm()
+
+    def _recreate_cm(self):
+        """Return a recreated instance of self.
+
+        Allows an otherwise one-shot context manager like
+        _GeneratorContextManager to support use as
+        a decorator via implicit recreation.
+
+        This is a private interface just for _GeneratorContextManager.
+        See issue #11647 for details.
+        """
+        return self
+
+    def __call__(self, func):
+        @wraps(func)
+        def inner(*args, **kwds):
+            with self._recreate_cm():
+                return func(*args, **kwds)
+        return inner
+
+
+class _GeneratorContextManager(ContextDecorator):
+    """Helper for @contextmanager decorator."""
+
+    def __init__(self, func, args, kwds):
+        self.gen = func(*args, **kwds)
+        self.func, self.args, self.kwds = func, args, kwds
+        # Issue 19330: ensure context manager instances have good docstrings
+        doc = getattr(func, "__doc__", None)
+        if doc is None:
+            doc = type(self).__doc__
+        self.__doc__ = doc
+        # Unfortunately, this still doesn't provide good help output when
+        # inspecting the created context manager instances, since pydoc
+        # currently bypasses the instance docstring and shows the docstring
+        # for the class instead.
+        # See http://bugs.python.org/issue19404 for more details.
+
+    def _recreate_cm(self):
+        # _GCM instances are one-shot context managers, so the
+        # CM must be recreated each time a decorated function is
+        # called
+        return self.__class__(self.func, self.args, self.kwds)
+
+    def __enter__(self):
+        try:
+            return next(self.gen)
+        except StopIteration:
+            raise RuntimeError("generator didn't yield")
+
+    def __exit__(self, type, value, traceback):
+        if type is None:
+            try:
+                next(self.gen)
+            except StopIteration:
+                return
+            else:
+                raise RuntimeError("generator didn't stop")
+        else:
+            if value is None:
+                # Need to force instantiation so we can reliably
+                # tell if we get the same exception back
+                value = type()
+            try:
+                self.gen.throw(type, value, traceback)
+                raise RuntimeError("generator didn't stop after throw()")
+            except StopIteration as exc:
+                # Suppress StopIteration *unless* it's the same exception that
+                # was passed to throw().  This prevents a StopIteration
+                # raised inside the "with" statement from being suppressed.
+                return exc is not value
+            except RuntimeError as exc:
+                # Don't re-raise the passed in exception
+                if exc is value:
+                    return False
+                # Likewise, avoid suppressing if a StopIteration exception
+                # was passed to throw() and later wrapped into a RuntimeError
+                # (see PEP 479).
+                if _HAVE_EXCEPTION_CHAINING and exc.__cause__ is value:
+                    return False
+                raise
+            except:
+                # only re-raise if it's *not* the exception that was
+                # passed to throw(), because __exit__() must not raise
+                # an exception unless __exit__() itself failed.  But throw()
+                # has to raise the exception to signal propagation, so this
+                # fixes the impedance mismatch between the throw() protocol
+                # and the __exit__() protocol.
+                #
+                if sys.exc_info()[1] is not value:
+                    raise
+
+
+def contextmanager(func):
+    """@contextmanager decorator.
+
+    Typical usage:
+
+        @contextmanager
+        def some_generator(<arguments>):
+            <setup>
+            try:
+                yield <value>
+            finally:
+                <cleanup>
+
+    This makes this:
+
+        with some_generator(<arguments>) as <variable>:
+            <body>
+
+    equivalent to this:
+
+        <setup>
+        try:
+            <variable> = <value>
+            <body>
+        finally:
+            <cleanup>
+
+    """
+    @wraps(func)
+    def helper(*args, **kwds):
+        return _GeneratorContextManager(func, args, kwds)
+    return helper
+
+
+class closing(object):
+    """Context to automatically close something at the end of a block.
+
+    Code like this:
+
+        with closing(<module>.open(<arguments>)) as f:
+            <block>
+
+    is equivalent to this:
+
+        f = <module>.open(<arguments>)
+        try:
+            <block>
+        finally:
+            f.close()
+
+    """
+    def __init__(self, thing):
+        self.thing = thing
+    def __enter__(self):
+        return self.thing
+    def __exit__(self, *exc_info):
+        self.thing.close()
+
+
+class _RedirectStream(object):
+
+    _stream = None
+
+    def __init__(self, new_target):
+        self._new_target = new_target
+        # We use a list of old targets to make this CM re-entrant
+        self._old_targets = []
+
+    def __enter__(self):
+        self._old_targets.append(getattr(sys, self._stream))
+        setattr(sys, self._stream, self._new_target)
+        return self._new_target
+
+    def __exit__(self, exctype, excinst, exctb):
+        setattr(sys, self._stream, self._old_targets.pop())
+
+
+class redirect_stdout(_RedirectStream):
+    """Context manager for temporarily redirecting stdout to another file.
+
+        # How to send help() to stderr
+        with redirect_stdout(sys.stderr):
+            help(dir)
+
+        # How to write help() to a file
+        with open('help.txt', 'w') as f:
+            with redirect_stdout(f):
+                help(pow)
+    """
+
+    _stream = "stdout"
+
+
+class redirect_stderr(_RedirectStream):
+    """Context manager for temporarily redirecting stderr to another file."""
+
+    _stream = "stderr"
+
+
+class suppress(object):
+    """Context manager to suppress specified exceptions
+
+    After the exception is suppressed, execution proceeds with the next
+    statement following the with statement.
+
+         with suppress(FileNotFoundError):
+             os.remove(somefile)
+         # Execution still resumes here if the file was already removed
+    """
+
+    def __init__(self, *exceptions):
+        self._exceptions = exceptions
+
+    def __enter__(self):
+        pass
+
+    def __exit__(self, exctype, excinst, exctb):
+        # Unlike isinstance and issubclass, CPython exception handling
+        # currently only looks at the concrete type hierarchy (ignoring
+        # the instance and subclass checking hooks). While Guido considers
+        # that a bug rather than a feature, it's a fairly hard one to fix
+        # due to various internal implementation details. suppress provides
+        # the simpler issubclass based semantics, rather than trying to
+        # exactly reproduce the limitations of the CPython interpreter.
+        #
+        # See http://bugs.python.org/issue12029 for more details
+        return exctype is not None and issubclass(exctype, self._exceptions)
+
+
+# Context manipulation is Python 3 only
+_HAVE_EXCEPTION_CHAINING = sys.version_info[0] >= 3
+if _HAVE_EXCEPTION_CHAINING:
+    def _make_context_fixer(frame_exc):
+        def _fix_exception_context(new_exc, old_exc):
+            # Context may not be correct, so find the end of the chain
+            while 1:
+                exc_context = new_exc.__context__
+                if exc_context is old_exc:
+                    # Context is already set correctly (see issue 20317)
+                    return
+                if exc_context is None or exc_context is frame_exc:
+                    break
+                new_exc = exc_context
+            # Change the end of the chain to point to the exception
+            # we expect it to reference
+            new_exc.__context__ = old_exc
+        return _fix_exception_context
+
+    def _reraise_with_existing_context(exc_details):
+        try:
+            # bare "raise exc_details[1]" replaces our carefully
+            # set-up context
+            fixed_ctx = exc_details[1].__context__
+            raise exc_details[1]
+        except BaseException:
+            exc_details[1].__context__ = fixed_ctx
+            raise
+else:
+    # No exception context in Python 2
+    def _make_context_fixer(frame_exc):
+        return lambda new_exc, old_exc: None
+
+    # Use 3 argument raise in Python 2,
+    # but use exec to avoid SyntaxError in Python 3
+    def _reraise_with_existing_context(exc_details):
+        exc_type, exc_value, exc_tb = exc_details
+        exec ("raise exc_type, exc_value, exc_tb")
+
+# Handle old-style classes if they exist
+try:
+    from types import InstanceType
+except ImportError:
+    # Python 3 doesn't have old-style classes
+    _get_type = type
+else:
+    # Need to handle old-style context managers on Python 2
+    def _get_type(obj):
+        obj_type = type(obj)
+        if obj_type is InstanceType:
+            return obj.__class__ # Old-style class
+        return obj_type # New-style class
+
+# Inspired by discussions on http://bugs.python.org/issue13585
+class ExitStack(object):
+    """Context manager for dynamic management of a stack of exit callbacks
+
+    For example:
+
+        with ExitStack() as stack:
+            files = [stack.enter_context(open(fname)) for fname in filenames]
+            # All opened files will automatically be closed at the end of
+            # the with statement, even if attempts to open files later
+            # in the list raise an exception
+
+    """
+    def __init__(self):
+        self._exit_callbacks = deque()
+
+    def pop_all(self):
+        """Preserve the context stack by transferring it to a new instance"""
+        new_stack = type(self)()
+        new_stack._exit_callbacks = self._exit_callbacks
+        self._exit_callbacks = deque()
+        return new_stack
+
+    def _push_cm_exit(self, cm, cm_exit):
+        """Helper to correctly register callbacks to __exit__ methods"""
+        def _exit_wrapper(*exc_details):
+            return cm_exit(cm, *exc_details)
+        _exit_wrapper.__self__ = cm
+        self.push(_exit_wrapper)
+
+    def push(self, exit):
+        """Registers a callback with the standard __exit__ method signature
+
+        Can suppress exceptions the same way __exit__ methods can.
+
+        Also accepts any object with an __exit__ method (registering a call
+        to the method instead of the object itself)
+        """
+        # We use an unbound method rather than a bound method to follow
+        # the standard lookup behaviour for special methods
+        _cb_type = _get_type(exit)
+        try:
+            exit_method = _cb_type.__exit__
+        except AttributeError:
+            # Not a context manager, so assume its a callable
+            self._exit_callbacks.append(exit)
+        else:
+            self._push_cm_exit(exit, exit_method)
+        return exit # Allow use as a decorator
+
+    def callback(self, callback, *args, **kwds):
+        """Registers an arbitrary callback and arguments.
+
+        Cannot suppress exceptions.
+        """
+        def _exit_wrapper(exc_type, exc, tb):
+            callback(*args, **kwds)
+        # We changed the signature, so using @wraps is not appropriate, but
+        # setting __wrapped__ may still help with introspection
+        _exit_wrapper.__wrapped__ = callback
+        self.push(_exit_wrapper)
+        return callback # Allow use as a decorator
+
+    def enter_context(self, cm):
+        """Enters the supplied context manager
+
+        If successful, also pushes its __exit__ method as a callback and
+        returns the result of the __enter__ method.
+        """
+        # We look up the special methods on the type to match the with statement
+        _cm_type = _get_type(cm)
+        _exit = _cm_type.__exit__
+        result = _cm_type.__enter__(cm)
+        self._push_cm_exit(cm, _exit)
+        return result
+
+    def close(self):
+        """Immediately unwind the context stack"""
+        self.__exit__(None, None, None)
+
+    def __enter__(self):
+        return self
+
+    def __exit__(self, *exc_details):
+        received_exc = exc_details[0] is not None
+
+        # We manipulate the exception state so it behaves as though
+        # we were actually nesting multiple with statements
+        frame_exc = sys.exc_info()[1]
+        _fix_exception_context = _make_context_fixer(frame_exc)
+
+        # Callbacks are invoked in LIFO order to match the behaviour of
+        # nested context managers
+        suppressed_exc = False
+        pending_raise = False
+        while self._exit_callbacks:
+            cb = self._exit_callbacks.pop()
+            try:
+                if cb(*exc_details):
+                    suppressed_exc = True
+                    pending_raise = False
+                    exc_details = (None, None, None)
+            except:
+                new_exc_details = sys.exc_info()
+                # simulate the stack of exceptions by setting the context
+                _fix_exception_context(new_exc_details[1], exc_details[1])
+                pending_raise = True
+                exc_details = new_exc_details
+        if pending_raise:
+            _reraise_with_existing_context(exc_details)
+        return received_exc and suppressed_exc
+
+# Preserve backwards compatibility
+class ContextStack(ExitStack):
+    """Backwards compatibility alias for ExitStack"""
+
+    def __init__(self):
+        warnings.warn("ContextStack has been renamed to ExitStack",
+                      DeprecationWarning)
+        super(ContextStack, self).__init__()
+
+    def register_exit(self, callback):
+        return self.push(callback)
+
+    def register(self, callback, *args, **kwds):
+        return self.callback(callback, *args, **kwds)
+
+    def preserve(self):
+        return self.pop_all()
@@ -0,0 +1,18 @@
+"""Provide a (g)dbm-compatible interface to bsddb.hashopen."""
+
+import sys
+import warnings
+warnings.warnpy3k("in 3.x, the dbhash module has been removed", stacklevel=2)
+try:
+    import bsddb
+except ImportError:
+    # prevent a second import of this module from spuriously succeeding
+    del sys.modules[__name__]
+    raise
+
+__all__ = ["error","open"]
+
+error = bsddb.error                     # Exported for anydbm
+
+def open(file, flag = 'r', mode=0666):
+    return bsddb.hashopen(file, flag, mode)
@@ -1,27 +0,0 @@
-Copyright (c) 2011-2014 Mike Bayer
-
-All rights reserved.
-
-Redistribution and use in source and binary forms, with or without
-modification, are permitted provided that the following conditions
-are met:
-1. Redistributions of source code must retain the above copyright
-   notice, this list of conditions and the following disclaimer.
-2. Redistributions in binary form must reproduce the above copyright
-   notice, this list of conditions and the following disclaimer in the
-   documentation and/or other materials provided with the distribution.
-3. The name of the author or contributors may not be used to endorse or
-   promote products derived from this software without specific prior
-   written permission.
-
-THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
-ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
-IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
-ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
-FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
-DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
-OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
-HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
-LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
-OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
-SUCH DAMAGE.
@@ -1,91 +0,0 @@
-dogpile.cache
-=============
-
-A caching API built around the concept of a "dogpile lock", which allows
-continued access to an expiring data value while a single thread generates a
-new value.
-
-dogpile.cache builds on the `dogpile.core <http://pypi.python.org/pypi/dogpile.core>`_
-locking system, which implements the idea of "allow one creator to write while
-others read" in the abstract.   Overall, dogpile.cache is intended as a
-replacement to the `Beaker <http://beaker.groovie.org>`_ caching system, the internals
-of which are written by the same author.   All the ideas of Beaker which "work"
-are re-implemented in dogpile.cache in a more efficient and succinct manner,
-and all the cruft (Beaker's internals were first written in 2005) relegated
-to the trash heap.
-
-Features
--------
-
-* A succinct API which encourages up-front configuration of pre-defined
-  "regions", each one defining a set of caching characteristics including
-  storage backend, configuration options, and default expiration time.
-* A standard get/set/delete API as well as a function decorator API is
-  provided.
-* The mechanics of key generation are fully customizable.   The function
-  decorator API features a pluggable "key generator" to customize how
-  cache keys are made to correspond to function calls, and an optional
-  "key mangler" feature provides for pluggable mangling of keys
-  (such as encoding, SHA-1 hashing) as desired for each region.
-* The dogpile lock, first developed as the core engine behind the Beaker
-  caching system, here vastly simplified, improved, and better tested.
-  Some key performance
-  issues that were intrinsic to Beaker's architecture, particularly that
-  values would frequently be "double-fetched" from the cache, have been fixed.
-* Backends implement their own version of a "distributed" lock, where the
-  "distribution" matches the backend's storage system.  For example, the
-  memcached backends allow all clients to coordinate creation of values
-  using memcached itself.   The dbm file backend uses a lockfile
-  alongside the dbm file.  New backends, such as a Redis-based backend,
-  can provide their own locking mechanism appropriate to the storage
-  engine.
-* Writing new backends or hacking on the existing backends is intended to be
-  routine - all that's needed are basic get/set/delete methods. A distributed
-  lock tailored towards the backend is an optional addition, else dogpile uses
-  a regular thread mutex. New backends can be registered with dogpile.cache
-  directly or made available via setuptools entry points.
-* Included backends feature three memcached backends (python-memcached, pylibmc,
-  bmemcached), a Redis backend, a backend based on Python's
-  anydbm, and a plain dictionary backend.
-* Space for third party plugins, including the first which provides the
-  dogpile.cache engine to Mako templates.
-* Python 3 compatible in place - no 2to3 required.
-
-Synopsis
--------
-
-dogpile.cache features a single public usage object known as the ``CacheRegion``.
-This object then refers to a particular ``CacheBackend``.   Typical usage
-generates a region using ``make_region()``, which can then be used at the
-module level to decorate functions, or used directly in code with a traditional
-get/set interface.   Configuration of the backend is applied to the region
-using ``configure()`` or ``configure_from_config()``, allowing deferred
-config-file based configuration to occur after modules have been imported::
-
-    from dogpile.cache import make_region
-
-    region = make_region().configure(
-        'dogpile.cache.pylibmc',
-        expiration_time = 3600,
-        arguments = {
-            'url':["127.0.0.1"],
-            'binary':True,
-            'behaviors':{"tcp_nodelay": True,"ketama":True}
-        }
-    )
-
-    @region.cache_on_arguments()
-    def load_user_info(user_id):
-        return some_database.lookup_user_by_id(user_id)
-
-
-Documentation
-------------
-
-See dogpile.cache's full documentation at
-`dogpile.cache documentation <http://dogpilecache.readthedocs.org>`_.
-
-
-
-
-
@@ -1,6 +0,0 @@
-# See http://peak.telecommunity.com/DevCenter/setuptools#namespace-packages
-try:
-    __import__('pkg_resources').declare_namespace(__name__)
-except ImportError:
-    from pkgutil import extend_path
-    __path__ = extend_path(__path__, __name__)
@@ -1,3 +0,0 @@
-__version__ = '0.5.4'
-
-from .region import CacheRegion, register_backend, make_region
@@ -1,193 +0,0 @@
-import operator
-from .compat import py3k
-
-
-class NoValue(object):
-    """Describe a missing cache value.
-
-    The :attr:`.NO_VALUE` module global
-    should be used.
-
-    """
-    @property
-    def payload(self):
-        return self
-
-    if py3k:
-        def __bool__(self): #pragma NO COVERAGE
-            return False
-    else:
-        def __nonzero__(self): #pragma NO COVERAGE
-            return False
-
-NO_VALUE = NoValue()
-"""Value returned from ``get()`` that describes
-a  key not present."""
-
-class CachedValue(tuple):
-    """Represent a value stored in the cache.
-
-    :class:`.CachedValue` is a two-tuple of
-    ``(payload, metadata)``, where ``metadata``
-    is dogpile.cache's tracking information (
-    currently the creation time).  The metadata
-    and tuple structure is pickleable, if
-    the backend requires serialization.
-
-    """
-    payload = property(operator.itemgetter(0))
-    """Named accessor for the payload."""
-
-    metadata = property(operator.itemgetter(1))
-    """Named accessor for the dogpile.cache metadata dictionary."""
-
-    def __new__(cls, payload, metadata):
-        return tuple.__new__(cls, (payload, metadata))
-
-    def __reduce__(self):
-        return CachedValue, (self.payload, self.metadata)
-
-class CacheBackend(object):
-    """Base class for backend implementations."""
-
-    key_mangler = None
-    """Key mangling function.
-
-    May be None, or otherwise declared
-    as an ordinary instance method.
-
-    """
-
-    def __init__(self, arguments): #pragma NO COVERAGE
-        """Construct a new :class:`.CacheBackend`.
-
-        Subclasses should override this to
-        handle the given arguments.
-
-        :param arguments: The ``arguments`` parameter
-         passed to :func:`.make_registry`.
-
-        """
-        raise NotImplementedError()
-
-    @classmethod
-    def from_config_dict(cls, config_dict, prefix):
-        prefix_len = len(prefix)
-        return cls(
-                dict(
-                    (key[prefix_len:], config_dict[key])
-                    for key in config_dict
-                    if key.startswith(prefix)
-                )
-            )
-
-    def get_mutex(self, key):
-        """Return an optional mutexing object for the given key.
-
-        This object need only provide an ``acquire()``
-        and ``release()`` method.
-
-        May return ``None``, in which case the dogpile
-        lock will use a regular ``threading.Lock``
-        object to mutex concurrent threads for
-        value creation.   The default implementation
-        returns ``None``.
-
-        Different backends may want to provide various
-        kinds of "mutex" objects, such as those which
-        link to lock files, distributed mutexes,
-        memcached semaphores, etc.  Whatever
-        kind of system is best suited for the scope
-        and behavior of the caching backend.
-
-        A mutex that takes the key into account will
-        allow multiple regenerate operations across
-        keys to proceed simultaneously, while a mutex
-        that does not will serialize regenerate operations
-        to just one at a time across all keys in the region.
-        The latter approach, or a variant that involves
-        a modulus of the given key's hash value,
-        can be used as a means of throttling the total
-        number of value recreation operations that may
-        proceed at one time.
-
-        """
-        return None
-
-    def get(self, key): #pragma NO COVERAGE
-        """Retrieve a value from the cache.
-
-        The returned value should be an instance of
-        :class:`.CachedValue`, or ``NO_VALUE`` if
-        not present.
-
-        """
-        raise NotImplementedError()
-
-    def get_multi(self, keys): #pragma NO COVERAGE
-        """Retrieve multiple values from the cache.
-
-        The returned value should be a list, corresponding
-        to the list of keys given.
-
-        .. versionadded:: 0.5.0
-
-        """
-        raise NotImplementedError()
-
-    def set(self, key, value): #pragma NO COVERAGE
-        """Set a value in the cache.
-
-        The key will be whatever was passed
-        to the registry, processed by the
-        "key mangling" function, if any.
-        The value will always be an instance
-        of :class:`.CachedValue`.
-
-        """
-        raise NotImplementedError()
-
-    def set_multi(self, mapping): #pragma NO COVERAGE
-        """Set multiple values in the cache.
-
-        The key will be whatever was passed
-        to the registry, processed by the
-        "key mangling" function, if any.
-        The value will always be an instance
-        of :class:`.CachedValue`.
-
-        .. versionadded:: 0.5.0
-
-        """
-        raise NotImplementedError()
-
-    def delete(self, key): #pragma NO COVERAGE
-        """Delete a value from the cache.
-
-        The key will be whatever was passed
-        to the registry, processed by the
-        "key mangling" function, if any.
-
-        The behavior here should be idempotent,
-        that is, can be called any number of times
-        regardless of whether or not the
-        key exists.
-        """
-        raise NotImplementedError()
-
-    def delete_multi(self, keys): #pragma NO COVERAGE
-        """Delete multiple values from the cache.
-
-        The key will be whatever was passed
-        to the registry, processed by the
-        "key mangling" function, if any.
-
-        The behavior here should be idempotent,
-        that is, can be called any number of times
-        regardless of whether or not the
-        key exists.
-
-        .. versionadded:: 0.5.0
-
-        """
-        raise NotImplementedError()
@@ -1,10 +0,0 @@
-from dogpile.cache.region import register_backend
-
-register_backend("dogpile.cache.null", "dogpile.cache.backends.null", "NullBackend")
-register_backend("dogpile.cache.dbm", "dogpile.cache.backends.file", "DBMBackend")
-register_backend("dogpile.cache.pylibmc", "dogpile.cache.backends.memcached", "PylibmcBackend")
-register_backend("dogpile.cache.bmemcached", "dogpile.cache.backends.memcached", "BMemcachedBackend")
-register_backend("dogpile.cache.memcached", "dogpile.cache.backends.memcached", "MemcachedBackend")
-register_backend("dogpile.cache.memory", "dogpile.cache.backends.memory", "MemoryBackend")
-register_backend("dogpile.cache.memory_pickle", "dogpile.cache.backends.memory", "MemoryPickleBackend")
-register_backend("dogpile.cache.redis", "dogpile.cache.backends.redis", "RedisBackend")
@@ -1,441 +0,0 @@
-"""
-File Backends
------------------
-
-Provides backends that deal with local filesystem access.
-
-"""
-
-from __future__ import with_statement
-from dogpile.cache.api import CacheBackend, NO_VALUE
-from contextlib import contextmanager
-from dogpile.cache import compat
-from dogpile.cache import util
-import os
-
-__all__ = 'DBMBackend', 'FileLock', 'AbstractFileLock'
-
-class DBMBackend(CacheBackend):
-    """A file-backend using a dbm file to store keys.
-
-    Basic usage::
-
-        from dogpile.cache import make_region
-
-        region = make_region().configure(
-            'dogpile.cache.dbm',
-            expiration_time = 3600,
-            arguments = {
-                "filename":"/path/to/cachefile.dbm"
-            }
-        )
-
-    DBM access is provided using the Python ``anydbm`` module,
-    which selects a platform-specific dbm module to use.
-    This may be made to be more configurable in a future
-    release.
-
-    Note that different dbm modules have different behaviors.
-    Some dbm implementations handle their own locking, while
-    others don't.  The :class:`.DBMBackend` uses a read/write
-    lockfile by default, which is compatible even with those
-    DBM implementations for which this is unnecessary,
-    though the behavior can be disabled.
-
-    The DBM backend by default makes use of two lockfiles.
-    One is in order to protect the DBM file itself from
-    concurrent writes, the other is to coordinate
-    value creation (i.e. the dogpile lock).  By default,
-    these lockfiles use the ``flock()`` system call
-    for locking; this is **only available on Unix
-    platforms**.   An alternative lock implementation, such as one
-    which is based on threads or uses a third-party system
-    such as `portalocker <https://pypi.python.org/pypi/portalocker>`_,
-    can be dropped in using the ``lock_factory`` argument
-    in conjunction with the :class:`.AbstractFileLock` base class.
-
-    Currently, the dogpile lock is against the entire
-    DBM file, not per key.   This means there can
-    only be one "creator" job running at a time
-    per dbm file.
-
-    A future improvement might be to have the dogpile lock
-    using a filename that's based on a modulus of the key.
-    Locking on a filename that uniquely corresponds to the
-    key is problematic, since it's not generally safe to
-    delete lockfiles as the application runs, implying an
-    unlimited number of key-based files would need to be
-    created and never deleted.
-
-    Parameters to the ``arguments`` dictionary are
-    below.
-
-    :param filename: path of the filename in which to
-     create the DBM file.  Note that some dbm backends
-     will change this name to have additional suffixes.
-    :param rw_lockfile: the name of the file to use for
-     read/write locking.  If omitted, a default name
-     is used by appending the suffix ".rw.lock" to the
-     DBM filename.  If False, then no lock is used.
-    :param dogpile_lockfile: the name of the file to use
-     for value creation, i.e. the dogpile lock.  If
-     omitted, a default name is used by appending the
-     suffix ".dogpile.lock" to the DBM filename. If
-     False, then dogpile.cache uses the default dogpile
-     lock, a plain thread-based mutex.
-    :param lock_factory: a function or class which provides
-     for a read/write lock.  Defaults to :class:`.FileLock`.
-     Custom implementations need to implement context-manager
-     based ``read()`` and ``write()`` functions - the
-     :class:`.AbstractFileLock` class is provided as a base class
-     which provides these methods based on individual read/write lock
-     functions.  E.g. to replace the lock with the dogpile.core
-     :class:`.ReadWriteMutex`::
-
-        from dogpile.core.readwrite_lock import ReadWriteMutex
-        from dogpile.cache.backends.file import AbstractFileLock
-
-        class MutexLock(AbstractFileLock):
-            def __init__(self, filename):
-                self.mutex = ReadWriteMutex()
-
-            def acquire_read_lock(self, wait):
-                ret = self.mutex.acquire_read_lock(wait)
-                return wait or ret
-
-            def acquire_write_lock(self, wait):
-                ret = self.mutex.acquire_write_lock(wait)
-                return wait or ret
-
-            def release_read_lock(self):
-                return self.mutex.release_read_lock()
-
-            def release_write_lock(self):
-                return self.mutex.release_write_lock()
-
-        from dogpile.cache import make_region
-
-        region = make_region().configure(
-            "dogpile.cache.dbm",
-            expiration_time=300,
-            arguments={
-                "filename": "file.dbm",
-                "lock_factory": MutexLock
-            }
-        )
-
-     While the included :class:`.FileLock` uses ``os.flock()``, a
-     windows-compatible implementation can be built using a library
-     such as `portalocker <https://pypi.python.org/pypi/portalocker>`_.
-
-     .. versionadded:: 0.5.2
-
-
-
-    """
-    def __init__(self, arguments):
-        self.filename = os.path.abspath(
-                            os.path.normpath(arguments['filename'])
-                        )
-        dir_, filename = os.path.split(self.filename)
-
-        self.lock_factory = arguments.get("lock_factory", FileLock)
-        self._rw_lock = self._init_lock(
-                                arguments.get('rw_lockfile'),
-                                ".rw.lock", dir_, filename)
-        self._dogpile_lock = self._init_lock(
-                                arguments.get('dogpile_lockfile'),
-                                ".dogpile.lock",
-                                dir_, filename,
-                                util.KeyReentrantMutex.factory)
-
-        # TODO: make this configurable
-        if compat.py3k:
-            import dbm
-        else:
-            import anydbm as dbm
-        self.dbmmodule = dbm
-        self._init_dbm_file()
-
-    def _init_lock(self, argument, suffix, basedir, basefile, wrapper=None):
-        if argument is None:
-            lock = self.lock_factory(os.path.join(basedir, basefile + suffix))
-        elif argument is not False:
-            lock = self.lock_factory(
-                        os.path.abspath(
-                            os.path.normpath(argument)
-                        ))
-        else:
-            return None
-        if wrapper:
-            lock = wrapper(lock)
-        return lock
-
-    def _init_dbm_file(self):
-        exists = os.access(self.filename, os.F_OK)
-        if not exists:
-            for ext in ('db', 'dat', 'pag', 'dir'):
-                if os.access(self.filename + os.extsep + ext, os.F_OK):
-                    exists = True
-                    break
-        if not exists:
-            fh = self.dbmmodule.open(self.filename, 'c')
-            fh.close()
-
-    def get_mutex(self, key):
-        # using one dogpile for the whole file.   Other ways
-        # to do this might be using a set of files keyed to a
-        # hash/modulus of the key.   the issue is it's never
-        # really safe to delete a lockfile as this can
-        # break other processes trying to get at the file
-        # at the same time - so handling unlimited keys
-        # can't imply unlimited filenames
-        if self._dogpile_lock:
-            return self._dogpile_lock(key)
-        else:
-            return None
-
-    @contextmanager
-    def _use_rw_lock(self, write):
-        if self._rw_lock is None:
-            yield
-        elif write:
-            with self._rw_lock.write():
-                yield
-        else:
-            with self._rw_lock.read():
-                yield
-
-    @contextmanager
-    def _dbm_file(self, write):
-        with self._use_rw_lock(write):
-            dbm = self.dbmmodule.open(self.filename,
-                                "w" if write else "r")
-            yield dbm
-            dbm.close()
-
-    def get(self, key):
-        with self._dbm_file(False) as dbm:
-            if hasattr(dbm, 'get'):
-                value = dbm.get(key, NO_VALUE)
-            else:
-                # gdbm objects lack a .get method
-                try:
-                    value = dbm[key]
-                except KeyError:
-                    value = NO_VALUE
-            if value is not NO_VALUE:
-                value = compat.pickle.loads(value)
-            return value
-
-    def get_multi(self, keys):
-        return [self.get(key) for key in keys]
-
-    def set(self, key, value):
-        with self._dbm_file(True) as dbm:
-            dbm[key] = compat.pickle.dumps(value)
-
-    def set_multi(self, mapping):
-        with self._dbm_file(True) as dbm:
-            for key,value in mapping.items():
-                dbm[key] = compat.pickle.dumps(value)
-
-    def delete(self, key):
-        with self._dbm_file(True) as dbm:
-            try:
-                del dbm[key]
-            except KeyError:
-                pass
-
-    def delete_multi(self, keys):
-        with self._dbm_file(True) as dbm:
-            for key in keys:
-                try:
-                    del dbm[key]
-                except KeyError:
-                    pass
-
-class AbstractFileLock(object):
-    """Coordinate read/write access to a file.
-
-    typically is a file-based lock but doesn't necessarily have to be.
-
-    The default implementation here is :class:`.FileLock`.
-
-    Implementations should provide the following methods::
-
-        * __init__()
-        * acquire_read_lock()
-        * acquire_write_lock()
-        * release_read_lock()
-        * release_write_lock()
-
-    The ``__init__()`` method accepts a single argument "filename", which
-    may be used as the "lock file", for those implementations that use a lock
-    file.
-
-    Note that multithreaded environments must provide a thread-safe
-    version of this lock.  The recommended approach for file-descriptor-based
-    locks is to use a Python ``threading.local()`` so that a unique file descriptor
-    is held per thread.  See the source code of :class:`.FileLock` for an
-    implementation example.
-
-
-    """
-
-    def __init__(self, filename):
-        """Constructor, is given the filename of a potential lockfile.
-
-        The usage of this filename is optional and no file is
-        created by default.
-
-        Raises ``NotImplementedError`` by default, must be
-        implemented by subclasses.
-        """
-        raise NotImplementedError()
-
-    def acquire(self, wait=True):
-        """Acquire the "write" lock.
-
-        This is a direct call to :meth:`.AbstractFileLock.acquire_write_lock`.
-
-        """
-        return self.acquire_write_lock(wait)
-
-    def release(self):
-        """Release the "write" lock.
-
-        This is a direct call to :meth:`.AbstractFileLock.release_write_lock`.
-
-        """
-        self.release_write_lock()
-
-    @contextmanager
-    def read(self):
-        """Provide a context manager for the "read" lock.
-
-        This method makes use of :meth:`.AbstractFileLock.acquire_read_lock`
-        and :meth:`.AbstractFileLock.release_read_lock`
-
-        """
-
-        self.acquire_read_lock(True)
-        try:
-            yield
-        finally:
-            self.release_read_lock()
-
-    @contextmanager
-    def write(self):
-        """Provide a context manager for the "write" lock.
-
-        This method makes use of :meth:`.AbstractFileLock.acquire_write_lock`
-        and :meth:`.AbstractFileLock.release_write_lock`
-
-        """
-
-        self.acquire_write_lock(True)
-        try:
-            yield
-        finally:
-            self.release_write_lock()
-
-    @property
-    def is_open(self):
-        """optional method."""
-        raise NotImplementedError()
-
-    def acquire_read_lock(self, wait):
-        """Acquire a 'reader' lock.
-
-        Raises ``NotImplementedError`` by default, must be
-        implemented by subclasses.
-        """
-        raise NotImplementedError()
-
-    def acquire_write_lock(self, wait):
-        """Acquire a 'write' lock.
-
-        Raises ``NotImplementedError`` by default, must be
-        implemented by subclasses.
-        """
-        raise NotImplementedError()
-
-    def release_read_lock(self):
-        """Release a 'reader' lock.
-
-        Raises ``NotImplementedError`` by default, must be
-        implemented by subclasses.
-        """
-        raise NotImplementedError()
-
-    def release_write_lock(self):
-        """Release a 'writer' lock.
-
-        Raises ``NotImplementedError`` by default, must be
-        implemented by subclasses.
-        """
-        raise NotImplementedError()
-
-class FileLock(AbstractFileLock):
-    """Use lockfiles to coordinate read/write access to a file.
-
-    Only works on Unix systems, using
-    `fcntl.flock() <http://docs.python.org/library/fcntl.html>`_.
-
-    """
-
-    def __init__(self, filename):
-        self._filedescriptor = compat.threading.local()
-        self.filename = filename
-
-    @util.memoized_property
-    def _module(self):
-        import fcntl
-        return fcntl
-
-    @property
-    def is_open(self):
-        return hasattr(self._filedescriptor, 'fileno')
-
-    def acquire_read_lock(self, wait):
-        return self._acquire(wait, os.O_RDONLY, self._module.LOCK_SH)
-
-    def acquire_write_lock(self, wait):
-        return self._acquire(wait, os.O_WRONLY, self._module.LOCK_EX)
-
-    def release_read_lock(self):
-        self._release()
-
-    def release_write_lock(self):
-        self._release()
-
-    def _acquire(self, wait, wrflag, lockflag):
-        wrflag |= os.O_CREAT
-        fileno = os.open(self.filename, wrflag)
-        try:
-            if not wait:
-                lockflag |= self._module.LOCK_NB
-            self._module.flock(fileno, lockflag)
-        except IOError:
-            os.close(fileno)
-            if not wait:
-                # this is typically
-                # "[Errno 35] Resource temporarily unavailable",
-                # because of LOCK_NB
-                return False
-            else:
-                raise
-        else:
-            self._filedescriptor.fileno = fileno
-            return True
-
-    def _release(self):
-        try:
-            fileno = self._filedescriptor.fileno
-        except AttributeError:
-            return
-        else:
-            self._module.flock(fileno, self._module.LOCK_UN)
-            os.close(fileno)
-            del self._filedescriptor.fileno
@@ -1,332 +0,0 @@
-"""
-Memcached Backends
------------------
-
-Provides backends for talking to `memcached <http://memcached.org>`_.
-
-"""
-
-from dogpile.cache.api import CacheBackend, NO_VALUE
-from dogpile.cache import compat
-from dogpile.cache import util
-import random
-import time
-
-__all__ = 'GenericMemcachedBackend', 'MemcachedBackend',\
-          'PylibmcBackend', 'BMemcachedBackend', 'MemcachedLock'
-
-class MemcachedLock(object):
-    """Simple distributed lock using memcached.
-
-    This is an adaptation of the lock featured at
-    http://amix.dk/blog/post/19386
-
-    """
-
-    def __init__(self, client_fn, key):
-        self.client_fn = client_fn
-        self.key = "_lock" + key
-
-    def acquire(self, wait=True):
-        client = self.client_fn()
-        i = 0
-        while True:
-            if client.add(self.key, 1):
-                return True
-            elif not wait:
-                return False
-            else:
-                sleep_time = (((i+1)*random.random()) + 2**i) / 2.5
-                time.sleep(sleep_time)
-            if i < 15:
-                i += 1
-
-    def release(self):
-        client = self.client_fn()
-        client.delete(self.key)
-
-class GenericMemcachedBackend(CacheBackend):
-    """Base class for memcached backends.
-
-    This base class accepts a number of paramters
-    common to all backends.
-
-    :param url: the string URL to connect to.  Can be a single
-     string or a list of strings.  This is the only argument
-     that's required.
-    :param distributed_lock: boolean, when True, will use a
-     memcached-lock as the dogpile lock (see :class:`.MemcachedLock`).
-     Use this when multiple
-     processes will be talking to the same memcached instance.
-     When left at False, dogpile will coordinate on a regular
-     threading mutex.
-    :param memcached_expire_time: integer, when present will
-     be passed as the ``time`` parameter to ``pylibmc.Client.set``.
-     This is used to set the memcached expiry time for a value.
-
-     .. note::
-
-         This parameter is **different** from Dogpile's own
-         ``expiration_time``, which is the number of seconds after
-         which Dogpile will consider the value to be expired.
-         When Dogpile considers a value to be expired,
-         it **continues to use the value** until generation
-         of a new value is complete, when using
-         :meth:`.CacheRegion.get_or_create`.
-         Therefore, if you are setting ``memcached_expire_time``, you'll
-         want to make sure it is greater than ``expiration_time``
-         by at least enough seconds for new values to be generated,
-         else the value won't be available during a regeneration,
-         forcing all threads to wait for a regeneration each time
-         a value expires.
-
-    The :class:`.GenericMemachedBackend` uses a ``threading.local()``
-    object to store individual client objects per thread,
-    as most modern memcached clients do not appear to be inherently
-    threadsafe.
-
-    In particular, ``threading.local()`` has the advantage over pylibmc's
-    built-in thread pool in that it automatically discards objects
-    associated with a particular thread when that thread ends.
-
-    """
-
-    set_arguments = {}
-    """Additional arguments which will be passed
-    to the :meth:`set` method."""
-
-    def __init__(self, arguments):
-        self._imports()
-        # using a plain threading.local here.   threading.local
-        # automatically deletes the __dict__ when a thread ends,
-        # so the idea is that this is superior to pylibmc's
-        # own ThreadMappedPool which doesn't handle this
-        # automatically.
-        self.url = util.to_list(arguments['url'])
-        self.distributed_lock = arguments.get('distributed_lock', False)
-        self.memcached_expire_time = arguments.get(
-            'memcached_expire_time', 0)
-
-    def _imports(self):
-        """client library imports go here."""
-        raise NotImplementedError()
-
-    def _create_client(self):
-        """Creation of a Client instance goes here."""
-        raise NotImplementedError()
-
-    @util.memoized_property
-    def _clients(self):
-        backend = self
-        class ClientPool(compat.threading.local):
-            def __init__(self):
-                self.memcached = backend._create_client()
-
-        return ClientPool()
-
-    @property
-    def client(self):
-        """Return the memcached client.
-
-        This uses a threading.local by
-        default as it appears most modern
-        memcached libs aren't inherently
-        threadsafe.
-
-        """
-        return self._clients.memcached
-
-    def get_mutex(self, key):
-        if self.distributed_lock:
-            return MemcachedLock(lambda: self.client, key)
-        else:
-            return None
-
-    def get(self, key):
-        value = self.client.get(key)
-        if value is None:
-            return NO_VALUE
-        else:
-            return value
-
-    def get_multi(self, keys):
-        values = self.client.get_multi(keys)
-        return [
-            NO_VALUE if key not in values
-            else values[key] for key in keys
-        ]
-
-    def set(self, key, value):
-        self.client.set(key,
-            value,
-            **self.set_arguments
-        )
-
-    def set_multi(self, mapping):
-        self.client.set_multi(mapping,
-            **self.set_arguments
-        )
-
-    def delete(self, key):
-        self.client.delete(key)
-
-    def delete_multi(self, keys):
-        self.client.delete_multi(keys)
-
-class MemcacheArgs(object):
-    """Mixin which provides support for the 'time' argument to set(),
-    'min_compress_len' to other methods.
-
-    """
-    def __init__(self, arguments):
-        self.min_compress_len = arguments.get('min_compress_len', 0)
-
-        self.set_arguments = {}
-        if "memcached_expire_time" in arguments:
-            self.set_arguments["time"] =\
-            arguments["memcached_expire_time"]
-        if "min_compress_len" in arguments:
-            self.set_arguments["min_compress_len"] =\
-            arguments["min_compress_len"]
-        super(MemcacheArgs, self).__init__(arguments)
-
-class PylibmcBackend(MemcacheArgs, GenericMemcachedBackend):
-    """A backend for the
-    `pylibmc <http://sendapatch.se/projects/pylibmc/index.html>`_
-    memcached client.
-
-    A configuration illustrating several of the optional
-    arguments described in the pylibmc documentation::
-
-        from dogpile.cache import make_region
-
-        region = make_region().configure(
-            'dogpile.cache.pylibmc',
-            expiration_time = 3600,
-            arguments = {
-                'url':["127.0.0.1"],
-                'binary':True,
-                'behaviors':{"tcp_nodelay": True,"ketama":True}
-            }
-        )
-
-    Arguments accepted here include those of
-    :class:`.GenericMemcachedBackend`, as well as
-    those below.
-
-    :param binary: sets the ``binary`` flag understood by
-     ``pylibmc.Client``.
-    :param behaviors: a dictionary which will be passed to
-     ``pylibmc.Client`` as the ``behaviors`` parameter.
-    :param min_compress_len: Integer, will be passed as the
-     ``min_compress_len`` parameter to the ``pylibmc.Client.set``
-     method.
-
-    """
-
-    def __init__(self, arguments):
-        self.binary = arguments.get('binary', False)
-        self.behaviors = arguments.get('behaviors', {})
-        super(PylibmcBackend, self).__init__(arguments)
-
-
-    def _imports(self):
-        global pylibmc
-        import pylibmc
-
-    def _create_client(self):
-        return pylibmc.Client(self.url,
-            binary=self.binary,
-            behaviors=self.behaviors
-        )
-
-class MemcachedBackend(MemcacheArgs, GenericMemcachedBackend):
-    """A backend using the standard `Python-memcached <http://www.tummy.com/Community/software/python-memcached/>`_
-    library.
-
-    Example::
-
-        from dogpile.cache import make_region
-
-        region = make_region().configure(
-            'dogpile.cache.memcached',
-            expiration_time = 3600,
-            arguments = {
-                'url':"127.0.0.1:11211"
-            }
-        )
-
-    """
-    def _imports(self):
-        global memcache
-        import memcache
-
-    def _create_client(self):
-        return memcache.Client(self.url)
-
-class BMemcachedBackend(GenericMemcachedBackend):
-    """A backend for the
-    `python-binary-memcached <https://github.com/jaysonsantos/python-binary-memcached>`_
-    memcached client.
-
-    This is a pure Python memcached client which
-    includes the ability to authenticate with a memcached
-    server using SASL.
-
-    A typical configuration using username/password::
-
-        from dogpile.cache import make_region
-
-        region = make_region().configure(
-            'dogpile.cache.bmemcached',
-            expiration_time = 3600,
-            arguments = {
-                'url':["127.0.0.1"],
-                'username':'scott',
-                'password':'tiger'
-            }
-        )
-
-    Arguments which can be passed to the ``arguments``
-    dictionary include:
-
-    :param username: optional username, will be used for
-     SASL authentication.
-    :param password: optional password, will be used for
-     SASL authentication.
-
-    """
-    def __init__(self, arguments):
-        self.username = arguments.get('username', None)
-        self.password = arguments.get('password', None)
-        super(BMemcachedBackend, self).__init__(arguments)
-
-    def _imports(self):
-        global bmemcached
-        import bmemcached
-
-        class RepairBMemcachedAPI(bmemcached.Client):
-            """Repairs BMemcached's non-standard method
-            signatures, which was fixed in BMemcached
-            ef206ed4473fec3b639e.
-
-            """
-
-            def add(self, key, value):
-                try:
-                    return super(RepairBMemcachedAPI, self).add(key, value)
-                except ValueError:
-                    return False
-
-        self.Client = RepairBMemcachedAPI
-
-    def _create_client(self):
-        return self.Client(self.url,
-            username=self.username,
-            password=self.password
-        )
-
-    def delete_multi(self, keys):
-        """python-binary-memcached api does not implements delete_multi"""
-        for key in keys:
-            self.delete(key)
@@ -1,122 +0,0 @@
-"""
-Memory Backends
---------------
-
-Provides simple dictionary-based backends.
-
-The two backends are :class:`.MemoryBackend` and :class:`.MemoryPickleBackend`;
-the latter applies a serialization step to cached values while the former
-places the value as given into the dictionary.
-
-"""
-
-from dogpile.cache.api import CacheBackend, NO_VALUE
-from dogpile.cache.compat import pickle
-
-class MemoryBackend(CacheBackend):
-    """A backend that uses a plain dictionary.
-
-    There is no size management, and values which
-    are placed into the dictionary will remain
-    until explicitly removed.   Note that
-    Dogpile's expiration of items is based on
-    timestamps and does not remove them from
-    the cache.
-
-    E.g.::
-
-        from dogpile.cache import make_region
-
-        region = make_region().configure(
-            'dogpile.cache.memory'
-        )
-
-
-    To use a Python dictionary of your choosing,
-    it can be passed in with the ``cache_dict``
-    argument::
-
-        my_dictionary = {}
-        region = make_region().configure(
-            'dogpile.cache.memory',
-            arguments={
-                "cache_dict":my_dictionary
-            }
-        )
-
-
-    """
-    pickle_values = False
-
-    def __init__(self, arguments):
-        self._cache = arguments.pop("cache_dict", {})
-
-    def get(self, key):
-        value = self._cache.get(key, NO_VALUE)
-        if value is not NO_VALUE and self.pickle_values:
-            value = pickle.loads(value)
-        return value
-
-    def get_multi(self, keys):
-        ret = [self._cache.get(key, NO_VALUE)
-            for key in keys]
-        if self.pickle_values:
-            ret = [
-                    pickle.loads(value)
-                    if value is not NO_VALUE else value
-                    for value in ret
-                ]
-        return ret
-
-    def set(self, key, value):
-        if self.pickle_values:
-            value = pickle.dumps(value, pickle.HIGHEST_PROTOCOL)
-        self._cache[key] = value
-
-    def set_multi(self, mapping):
-        pickle_values = self.pickle_values
-        for key, value in mapping.items():
-            if pickle_values:
-                value = pickle.dumps(value, pickle.HIGHEST_PROTOCOL)
-            self._cache[key] = value
-
-    def delete(self, key):
-        self._cache.pop(key, None)
-
-    def delete_multi(self, keys):
-        for key in keys:
-            self._cache.pop(key, None)
-
-
-class MemoryPickleBackend(MemoryBackend):
-    """A backend that uses a plain dictionary, but serializes objects on
-    :meth:`.MemoryBackend.set` and deserializes :meth:`.MemoryBackend.get`.
-
-    E.g.::
-
-        from dogpile.cache import make_region
-
-        region = make_region().configure(
-            'dogpile.cache.memory_pickle'
-        )
-
-    The usage of pickle to serialize cached values allows an object
-    as placed in the cache to be a copy of the original given object, so
-    that any subsequent changes to the given object aren't reflected
-    in the cached value, thus making the backend behave the same way
-    as other backends which make use of serialization.
-
-    The serialization is performed via pickle, and incurs the same
-    performance hit in doing so as that of other backends; in this way
-    the :class:`.MemoryPickleBackend` performance is somewhere in between
-    that of the pure :class:`.MemoryBackend` and the remote server oriented
-    backends such as that of Memcached or Redis.
-
-    Pickle behavior here is the same as that of the Redis backend, using
-    either ``cPickle`` or ``pickle`` and specifying ``HIGHEST_PROTOCOL``
-    upon serialize.
-
-    .. versionadded:: 0.5.3
-
-    """
-    pickle_values = True
@@ -1,62 +0,0 @@
-"""
-Null Backend
-------------
-
-The Null backend does not do any caching at all.  It can be
-used to test behavior without caching, or as a means of disabling
-caching for a region that is otherwise used normally.
-
-.. versionadded:: 0.5.4
-
-"""
-
-from dogpile.cache.api import CacheBackend, NO_VALUE
-
-
-__all__ = ['NullBackend']
-
-
-class NullLock(object):
-    def acquire(self):
-        pass
-
-    def release(self):
-        pass
-
-
-class NullBackend(CacheBackend):
-    """A "null" backend that effectively disables all cache operations.
-
-    Basic usage::
-
-        from dogpile.cache import make_region
-
-        region = make_region().configure(
-            'dogpile.cache.null'
-        )
-
-    """
-
-    def __init__(self, arguments):
-        pass
-
-    def get_mutex(self, key):
-        return NullLock()
-
-    def get(self, key):
-        return NO_VALUE
-
-    def get_multi(self, keys):
-        return [NO_VALUE for k in keys]
-
-    def set(self, key, value):
-        pass
-
-    def set_multi(self, mapping):
-        pass
-
-    def delete(self, key):
-        pass
-
-    def delete_multi(self, keys):
-        pass
@@ -1,181 +0,0 @@
-"""
-Redis Backends
------------------
-
-Provides backends for talking to `Redis <http://redis.io>`_.
-
-"""
-
-from __future__ import absolute_import
-from dogpile.cache.api import CacheBackend, NO_VALUE
-from dogpile.cache.compat import pickle, u
-
-redis = None
-
-__all__ = 'RedisBackend',
-
-
-class RedisBackend(CacheBackend):
-    """A `Redis <http://redis.io/>`_ backend, using the
-    `redis-py <http://pypi.python.org/pypi/redis/>`_ backend.
-
-    Example configuration::
-
-        from dogpile.cache import make_region
-
-        region = make_region().configure(
-            'dogpile.cache.redis',
-            arguments = {
-                'host': 'localhost',
-                'port': 6379,
-                'db': 0,
-                'redis_expiration_time': 60*60*2,   # 2 hours
-                'distributed_lock':True
-                }
-        )
-
-    Arguments accepted in the arguments dictionary:
-
-    :param url: string. If provided, will override separate host/port/db
-     params.  The format is that accepted by ``StrictRedis.from_url()``.
-
-     .. versionadded:: 0.4.1
-
-    :param host: string, default is ``localhost``.
-
-    :param password: string, default is no password.
-
-     .. versionadded:: 0.4.1
-
-    :param port: integer, default is ``6379``.
-
-    :param db: integer, default is ``0``.
-
-    :param redis_expiration_time: integer, number of seconds after setting
-     a value that Redis should expire it.  This should be larger than dogpile's
-     cache expiration.  By default no expiration is set.
-
-    :param distributed_lock: boolean, when True, will use a
-     redis-lock as the dogpile lock.
-     Use this when multiple
-     processes will be talking to the same redis instance.
-     When left at False, dogpile will coordinate on a regular
-     threading mutex.
-
-    :param lock_timeout: integer, number of seconds after acquiring a lock that
-     Redis should expire it.  This argument is only valid when
-     ``distributed_lock`` is ``True``.
-
-     .. versionadded:: 0.5.0
-
-    :param socket_timeout: float, seconds for socket timeout.
-     Default is None (no timeout).
-
-     .. versionadded:: 0.5.4
-
-    :param lock_sleep: integer, number of seconds to sleep when failed to
-     acquire a lock.  This argument is only valid when
-     ``distributed_lock`` is ``True``.
-
-     .. versionadded:: 0.5.0
-
-    :param connection_pool: ``redis.ConnectionPool`` object.  If provided,
-     this object supersedes other connection arguments passed to the
-     ``redis.StrictRedis`` instance, including url and/or host as well as
-     socket_timeout, and will be passed to ``redis.StrictRedis`` as the
-     source of connectivity.
-
-     .. versionadded:: 0.5.4
-
-
-    """
-
-    def __init__(self, arguments):
-        self._imports()
-        self.url = arguments.pop('url', None)
-        self.host = arguments.pop('host', 'localhost')
-        self.password = arguments.pop('password', None)
-        self.port = arguments.pop('port', 6379)
-        self.db = arguments.pop('db', 0)
-        self.distributed_lock = arguments.get('distributed_lock', False)
-        self.socket_timeout = arguments.pop('socket_timeout', None)
-
-        self.lock_timeout = arguments.get('lock_timeout', None)
-        self.lock_sleep = arguments.get('lock_sleep', 0.1)
-
-        self.redis_expiration_time = arguments.pop('redis_expiration_time', 0)
-        self.connection_pool = arguments.get('connection_pool', None)
-        self.client = self._create_client()
-
-    def _imports(self):
-        # defer imports until backend is used
-        global redis
-        import redis
-
-    def _create_client(self):
-        if self.connection_pool is not None:
-            # the connection pool already has all other connection
-            # options present within, so here we disregard socket_timeout
-            # and others.
-            return redis.StrictRedis(connection_pool=self.connection_pool)
-
-        args = {}
-        if self.socket_timeout:
-            args['socket_timeout'] = self.socket_timeout
-
-        if self.url is not None:
-            args.update(url=self.url)
-            return redis.StrictRedis.from_url(**args)
-        else:
-            args.update(
-                host=self.host, password=self.password,
-                port=self.port, db=self.db
-            )
-            return redis.StrictRedis(**args)
-
-
-    def get_mutex(self, key):
-        if self.distributed_lock:
-            return self.client.lock(u('_lock{0}').format(key),
-                                    self.lock_timeout, self.lock_sleep)
-        else:
-            return None
-
-    def get(self, key):
-        value = self.client.get(key)
-        if value is None:
-            return NO_VALUE
-        return pickle.loads(value)
-
-    def get_multi(self, keys):
-        values = self.client.mget(keys)
-        return [pickle.loads(v) if v is not None else NO_VALUE
-                  for v in values]
-
-    def set(self, key, value):
-        if self.redis_expiration_time:
-            self.client.setex(key, self.redis_expiration_time,
-                              pickle.dumps(value, pickle.HIGHEST_PROTOCOL))
-        else:
-            self.client.set(key, pickle.dumps(value, pickle.HIGHEST_PROTOCOL))
-
-    def set_multi(self, mapping):
-        mapping = dict(
-            (k, pickle.dumps(v, pickle.HIGHEST_PROTOCOL))
-            for k, v in mapping.items()
-        )
-
-        if not self.redis_expiration_time:
-            self.client.mset(mapping)
-        else:
-            pipe = self.client.pipeline()
-            for key, value in mapping.items():
-                pipe.setex(key, self.redis_expiration_time, value)
-            pipe.execute()
-
-    def delete(self, key):
-        self.client.delete(key)
-
-    def delete_multi(self, keys):
-        self.client.delete(*keys)
-
@@ -1,68 +0,0 @@
-import sys
-
-
-py2k = sys.version_info < (3, 0)
-py3k = sys.version_info >= (3, 0)
-py32 = sys.version_info >= (3, 2)
-py27 = sys.version_info >= (2, 7)
-jython = sys.platform.startswith('java')
-win32 = sys.platform.startswith('win')
-
-try:
-    import threading
-except ImportError:
-    import dummy_threading as threading
-
-
-if py3k: # pragma: no cover
-    string_types = str,
-    text_type = str
-    string_type = str
-
-    if py32:
-        callable = callable
-    else:
-        def callable(fn):
-            return hasattr(fn, '__call__')
-
-    def u(s):
-        return s
-
-    def ue(s):
-        return s
-
-    import configparser
-    import io
-    import _thread as thread
-else:
-    string_types = basestring,
-    text_type = unicode
-    string_type = str
-
-    def u(s):
-        return unicode(s, "utf-8")
-
-    def ue(s):
-        return unicode(s, "unicode_escape")
-
-    import ConfigParser as configparser
-    import StringIO as io
-
-
-    callable = callable
-    import thread
-
-
-if py3k or jython:
-    import pickle
-else:
-    import cPickle as pickle
-
-
-def timedelta_total_seconds(td):
-    if py27:
-        return td.total_seconds()
-    else:
-        return (td.microseconds + (td.seconds + td.days * 24 * 3600) * 1e6) / 1e6
-
-
@@ -1,17 +0,0 @@
-"""Exception classes for dogpile.cache."""
-
-
-class DogpileCacheException(Exception):
-    """Base Exception for dogpile.cache exceptions to inherit from."""
-
-
-class RegionAlreadyConfigured(DogpileCacheException):
-    """CacheRegion instance is already configured."""
-
-
-class RegionNotConfigured(DogpileCacheException):
-    """CacheRegion instance has not been configured."""
-
-
-class ValidationError(DogpileCacheException):
-    """Error validating a value or option."""
@@ -1,87 +0,0 @@
-"""
-Mako Integration
----------------
-
-dogpile.cache includes a `Mako <http://www.makotemplates.org>`_ plugin that replaces `Beaker <http://beaker.groovie.org>`_
-as the cache backend.
-Setup a Mako template lookup using the "dogpile.cache" cache implementation
-and a region dictionary::
-
-    from dogpile.cache import make_region
-    from mako.lookup import TemplateLookup
-
-    my_regions = {
-        "local":make_region().configure(
-                    "dogpile.cache.dbm",
-                    expiration_time=360,
-                    arguments={"filename":"file.dbm"}
-                ),
-        "memcached":make_region().configure(
-                    "dogpile.cache.pylibmc",
-                    expiration_time=3600,
-                    arguments={"url":["127.0.0.1"]}
-                )
-    }
-
-    mako_lookup = TemplateLookup(
-        directories=["/myapp/templates"],
-        cache_impl="dogpile.cache",
-        cache_args={
-            'regions':my_regions
-        }
-    )
-
-To use the above configuration in a template, use the ``cached=True`` argument on any
-Mako tag which accepts it, in conjunction with the name of the desired region
-as the ``cache_region`` argument::
-
-    <%def name="mysection()" cached="True" cache_region="memcached">
-        some content that's cached
-    </%def>
-
-
-"""
-from mako.cache import CacheImpl
-
-class MakoPlugin(CacheImpl):
-    """A Mako ``CacheImpl`` which talks to dogpile.cache."""
-
-    def __init__(self, cache):
-        super(MakoPlugin, self).__init__(cache)
-        try:
-            self.regions = self.cache.template.cache_args['regions']
-        except KeyError:
-            raise KeyError(
-                "'cache_regions' argument is required on the "
-                "Mako Lookup or Template object for usage "
-                "with the dogpile.cache plugin.")
-
-    def _get_region(self, **kw):
-        try:
-            region = kw['region']
-        except KeyError:
-            raise KeyError(
-                "'cache_region' argument must be specified with 'cache=True'"
-                "within templates for usage with the dogpile.cache plugin.")
-        try:
-            return self.regions[region]
-        except KeyError:
-            raise KeyError("No such region '%s'" % region)
-
-    def get_and_replace(self, key, creation_function, **kw):
-        expiration_time = kw.pop("timeout", None)
-        return self._get_region(**kw).get_or_create(key, creation_function,
-                                            expiration_time=expiration_time)
-
-    def get_or_create(self, key, creation_function, **kw):
-        return self.get_and_replace(key, creation_function, **kw)
-
-    def put(self, key, value, **kw):
-        self._get_region(**kw).put(key, value)
-
-    def get(self, key, **kw):
-        expiration_time = kw.pop("timeout", None)
-        return self._get_region(**kw).get(key, expiration_time=expiration_time)
-
-    def invalidate(self, key, **kw):
-        self._get_region(**kw).delete(key)
@@ -1,93 +0,0 @@
-"""
-Proxy Backends
------------------
-
-Provides a utility and a decorator class that allow for modifying the behavior
-of different backends without altering the class itself or having to extend the
-base backend.
-
-.. versionadded:: 0.5.0  Added support for the :class:`.ProxyBackend` class.
-
-"""
-
-from .api import CacheBackend
-
-class ProxyBackend(CacheBackend):
-    """A decorator class for altering the functionality of backends.
-
-    Basic usage::
-
-        from dogpile.cache import make_region
-        from dogpile.cache.proxy import ProxyBackend
-
-        class MyFirstProxy(ProxyBackend):
-            def get(self, key):
-                # ... custom code goes here ...
-                return self.proxied.get(key)
-
-            def set(self, key, value):
-                # ... custom code goes here ...
-                self.proxied.set(key)
-
-        class MySecondProxy(ProxyBackend):
-            def get(self, key):
-                # ... custom code goes here ...
-                return self.proxied.get(key)
-
-
-        region = make_region().configure(
-            'dogpile.cache.dbm',
-            expiration_time = 3600,
-            arguments = {
-                "filename":"/path/to/cachefile.dbm"
-            },
-            wrap = [ MyFirstProxy, MySecondProxy ]
-        )
-
-    Classes that extend :class:`.ProxyBackend` can be stacked
-    together.  The ``.proxied`` property will always
-    point to either the concrete backend instance or
-    the next proxy in the chain that a method can be
-    delegated towards.
-
-    .. versionadded:: 0.5.0
-
-    """
-
-    def __init__(self, *args, **kwargs):
-        self.proxied = None
-
-    def wrap(self, backend):
-        ''' Take a backend as an argument and setup the self.proxied property.
-        Return an object that be used as a backend by a :class:`.CacheRegion`
-        object.
-        '''
-        assert(isinstance(backend, CacheBackend) or isinstance(backend, ProxyBackend))
-        self.proxied = backend
-        return self
-
-    #
-    # Delegate any functions that are not already overridden to
-    # the proxies backend
-    #
-    def get(self, key):
-        return self.proxied.get(key)
-
-    def set(self, key, value):
-        self.proxied.set(key, value)
-
-    def delete(self, key):
-        self.proxied.delete(key)
-
-    def get_multi(self, keys):
-        return self.proxied.get_multi(keys)
-
-    def set_multi(self, keys):
-        self.proxied.set_multi(keys)
-
-    def delete_multi(self, keys):
-        self.proxied.delete_multi(keys)
-
-    def get_mutex(self, key):
-        return self.proxied.get_mutex(key)
-
@@ -1,189 +0,0 @@
-from hashlib import sha1
-import inspect
-import re
-import collections
-from . import compat
-
-
-def coerce_string_conf(d):
-    result = {}
-    for k, v in d.items():
-        if not isinstance(v, compat.string_types):
-            result[k] = v
-            continue
-
-        v = v.strip()
-        if re.match(r'^[-+]?\d+$', v):
-            result[k] = int(v)
-        elif re.match(r'^[-+]?(?:\d+(?:\.\d*)?|\.\d+)(?:[eE][-+]?\d+)?$', v):
-            result[k] = float(v)
-        elif v.lower() in ('false', 'true'):
-            result[k] = v.lower() == 'true'
-        elif v == 'None':
-            result[k] = None
-        else:
-            result[k] = v
-    return result
-
-class PluginLoader(object):
-    def __init__(self, group):
-        self.group = group
-        self.impls = {}
-
-    def load(self, name):
-        if name in self.impls:
-            return self.impls[name]()
-        else:  # pragma NO COVERAGE
-            import pkg_resources
-            for impl in pkg_resources.iter_entry_points(
-                                self.group,
-                                name):
-                self.impls[name] = impl.load
-                return impl.load()
-            else:
-                raise Exception(
-                        "Can't load plugin %s %s" %
-                        (self.group, name))
-
-    def register(self, name, modulepath, objname):
-        def load():
-            mod = __import__(modulepath)
-            for token in modulepath.split(".")[1:]:
-                mod = getattr(mod, token)
-            return getattr(mod, objname)
-        self.impls[name] = load
-
-
-def function_key_generator(namespace, fn, to_str=compat.string_type):
-    """Return a function that generates a string
-    key, based on a given function as well as
-    arguments to the returned function itself.
-
-    This is used by :meth:`.CacheRegion.cache_on_arguments`
-    to generate a cache key from a decorated function.
-
-    It can be replaced using the ``function_key_generator``
-    argument passed to :func:`.make_region`.
-
-    """
-
-    if namespace is None:
-        namespace = '%s:%s' % (fn.__module__, fn.__name__)
-    else:
-        namespace = '%s:%s|%s' % (fn.__module__, fn.__name__, namespace)
-
-    args = inspect.getargspec(fn)
-    has_self = args[0] and args[0][0] in ('self', 'cls')
-    def generate_key(*args, **kw):
-        if kw:
-            raise ValueError(
-                    "dogpile.cache's default key creation "
-                    "function does not accept keyword arguments.")
-        if has_self:
-            args = args[1:]
-
-        return namespace + "|" + " ".join(map(to_str, args))
-    return generate_key
-
-def function_multi_key_generator(namespace, fn, to_str=compat.string_type):
-
-    if namespace is None:
-        namespace = '%s:%s' % (fn.__module__, fn.__name__)
-    else:
-        namespace = '%s:%s|%s' % (fn.__module__, fn.__name__, namespace)
-
-    args = inspect.getargspec(fn)
-    has_self = args[0] and args[0][0] in ('self', 'cls')
-    def generate_keys(*args, **kw):
-        if kw:
-            raise ValueError(
-                    "dogpile.cache's default key creation "
-                    "function does not accept keyword arguments.")
-        if has_self:
-            args = args[1:]
-        return [namespace + "|" + key for key in map(to_str, args)]
-    return generate_keys
-
-def sha1_mangle_key(key):
-    """a SHA1 key mangler."""
-
-    return sha1(key).hexdigest()
-
-def length_conditional_mangler(length, mangler):
-    """a key mangler that mangles if the length of the key is
-    past a certain threshold.
-
-    """
-    def mangle(key):
-        if len(key) >= length:
-            return mangler(key)
-        else:
-            return key
-    return mangle
-
-class memoized_property(object):
-    """A read-only @property that is only evaluated once."""
-    def __init__(self, fget, doc=None):
-        self.fget = fget
-        self.__doc__ = doc or fget.__doc__
-        self.__name__ = fget.__name__
-
-    def __get__(self, obj, cls):
-        if obj is None:
-            return self
-        obj.__dict__[self.__name__] = result = self.fget(obj)
-        return result
-
-def to_list(x, default=None):
-    """Coerce to a list."""
-    if x is None:
-        return default
-    if not isinstance(x, (list, tuple)):
-        return [x]
-    else:
-        return x
-
-
-class KeyReentrantMutex(object):
-
-    def __init__(self, key, mutex, keys):
-        self.key = key
-        self.mutex = mutex
-        self.keys = keys
-
-    @classmethod
-    def factory(cls, mutex):
-        # this collection holds zero or one
-        # thread idents as the key; a set of
-        # keynames held as the value.
-        keystore = collections.defaultdict(set)
-        def fac(key):
-            return KeyReentrantMutex(key, mutex, keystore)
-        return fac
-
-    def acquire(self, wait=True):
-        current_thread = compat.threading.current_thread().ident
-        keys = self.keys.get(current_thread)
-        if keys is not None and \
-            self.key not in keys:
-            # current lockholder, new key. add it in
-            keys.add(self.key)
-            return True
-        elif self.mutex.acquire(wait=wait):
-            # after acquire, create new set and add our key
-            self.keys[current_thread].add(self.key)
-            return True
-        else:
-            return False
-
-    def release(self):
-        current_thread = compat.threading.current_thread().ident
-        keys = self.keys.get(current_thread)
-        assert keys is not None, "this thread didn't do the acquire"
-        assert self.key in keys, "No acquire held for key '%s'" % self.key
-        keys.remove(self.key)
-        if not keys:
-            # when list of keys empty, remove
-            # the thread ident and unlock.
-            del self.keys[current_thread]
-            self.mutex.release()
@@ -1,11 +0,0 @@
-from .dogpile import NeedRegenerationException, Lock
-from .nameregistry import NameRegistry
-from .readwrite_lock import ReadWriteMutex
-from .legacy import Dogpile, SyncReaderDogpile
-
-__all__ = [
-        'Dogpile', 'SyncReaderDogpile', 'NeedRegenerationException',
-        'NameRegistry', 'ReadWriteMutex', 'Lock']
-
-__version__ = '0.4.1'
-
@@ -1,162 +0,0 @@
-import time
-import logging
-
-log = logging.getLogger(__name__)
-
-class NeedRegenerationException(Exception):
-    """An exception that when raised in the 'with' block,
-    forces the 'has_value' flag to False and incurs a
-    regeneration of the value.
-
-    """
-
-NOT_REGENERATED = object()
-
-class Lock(object):
-    """Dogpile lock class.
-
-    Provides an interface around an arbitrary mutex
-    that allows one thread/process to be elected as
-    the creator of a new value, while other threads/processes
-    continue to return the previous version
-    of that value.
-
-    .. versionadded:: 0.4.0
-        The :class:`.Lock` class was added as a single-use object
-        representing the dogpile API without dependence on
-        any shared state between multiple instances.
-
-    :param mutex: A mutex object that provides ``acquire()``
-     and ``release()`` methods.
-    :param creator: Callable which returns a tuple of the form
-     (new_value, creation_time).  "new_value" should be a newly
-     generated value representing completed state.  "creation_time"
-     should be a floating point time value which is relative
-     to Python's ``time.time()`` call, representing the time
-     at which the value was created.  This time value should
-     be associated with the created value.
-    :param value_and_created_fn: Callable which returns
-     a tuple of the form (existing_value, creation_time).  This
-     basically should return what the last local call to the ``creator()``
-     callable has returned, i.e. the value and the creation time,
-     which would be assumed here to be from a cache.  If the
-     value is not available, the :class:`.NeedRegenerationException`
-     exception should be thrown.
-    :param expiretime: Expiration time in seconds.  Set to
-     ``None`` for never expires.  This timestamp is compared
-     to the creation_time result and ``time.time()`` to determine if
-     the value returned by value_and_created_fn is "expired".
-    :param async_creator: A callable.  If specified, this callable will be
-     passed the mutex as an argument and is responsible for releasing the mutex
-     after it finishes some asynchronous value creation.  The intent is for
-     this to be used to defer invocation of the creator callable until some
-     later time.
-
-     .. versionadded:: 0.4.1 added the async_creator argument.
-
-    """
-
-    def __init__(self,
-            mutex,
-            creator,
-            value_and_created_fn,
-            expiretime,
-            async_creator=None,
-        ):
-        self.mutex = mutex
-        self.creator = creator
-        self.value_and_created_fn = value_and_created_fn
-        self.expiretime = expiretime
-        self.async_creator = async_creator
-
-    def _is_expired(self, createdtime):
-        """Return true if the expiration time is reached, or no
-        value is available."""
-
-        return not self._has_value(createdtime) or \
-            (
-                self.expiretime is not None and
-                time.time() - createdtime > self.expiretime
-            )
-
-    def _has_value(self, createdtime):
-        """Return true if the creation function has proceeded
-        at least once."""
-        return createdtime > 0
-
-    def _enter(self):
-        value_fn = self.value_and_created_fn
-
-        try:
-            value = value_fn()
-            value, createdtime = value
-        except NeedRegenerationException:
-            log.debug("NeedRegenerationException")
-            value = NOT_REGENERATED
-            createdtime = -1
-
-        generated = self._enter_create(createdtime)
-
-        if generated is not NOT_REGENERATED:
-            generated, createdtime = generated
-            return generated
-        elif value is NOT_REGENERATED:
-            try:
-                value, createdtime = value_fn()
-                return value
-            except NeedRegenerationException:
-                raise Exception("Generation function should "
-                            "have just been called by a concurrent "
-                            "thread.")
-        else:
-            return value
-
-    def _enter_create(self, createdtime):
-
-        if not self._is_expired(createdtime):
-            return NOT_REGENERATED
-
-        async = False
-
-        if self._has_value(createdtime):
-            if not self.mutex.acquire(False):
-                log.debug("creation function in progress "
-                            "elsewhere, returning")
-                return NOT_REGENERATED
-        else:
-            log.debug("no value, waiting for create lock")
-            self.mutex.acquire()
-
-        try:
-            log.debug("value creation lock %r acquired" % self.mutex)
-
-            # see if someone created the value already
-            try:
-                value, createdtime = self.value_and_created_fn()
-            except NeedRegenerationException:
-                pass
-            else:
-                if not self._is_expired(createdtime):
-                    log.debug("value already present")
-                    return value, createdtime
-                elif self.async_creator:
-                    log.debug("Passing creation lock to async runner")
-                    self.async_creator(self.mutex)
-                    async = True
-                    return value, createdtime
-
-            log.debug("Calling creation function")
-            created = self.creator()
-            return created
-        finally:
-            if not async:
-                self.mutex.release()
-                log.debug("Released creation lock")
-
-
-    def __enter__(self):
-        return self._enter()
-
-    def __exit__(self, type, value, traceback):
-        pass
-
@@ -1,154 +0,0 @@
-from __future__ import with_statement
-
-from .util import threading
-from .readwrite_lock import ReadWriteMutex
-from .dogpile import Lock
-import time
-import contextlib
-
-class Dogpile(object):
-    """Dogpile lock class.
-
-    .. deprecated:: 0.4.0
-        The :class:`.Lock` object specifies the full
-        API of the :class:`.Dogpile` object in a single way,
-        rather than providing multiple modes of usage which
-        don't necessarily work in the majority of cases.
-        :class:`.Dogpile` is now a wrapper around the :class:`.Lock` object
-        which provides dogpile.core's original usage pattern.
-        This usage pattern began as something simple, but was
-        not of general use in real-world caching environments without
-        several extra complicating factors; the :class:`.Lock`
-        object presents the "real-world" API more succinctly,
-        and also fixes a cross-process concurrency issue.
-
-    :param expiretime: Expiration time in seconds.  Set to
-     ``None`` for never expires.
-    :param init: if True, set the 'createdtime' to the
-     current time.
-    :param lock: a mutex object that provides
-     ``acquire()`` and ``release()`` methods.
-
-    """
-    def __init__(self, expiretime, init=False, lock=None):
-        """Construct a new :class:`.Dogpile`.
-
-        """
-        if lock:
-            self.dogpilelock = lock
-        else:
-            self.dogpilelock = threading.Lock()
-
-        self.expiretime = expiretime
-        if init:
-            self.createdtime = time.time()
-
-    createdtime = -1
-    """The last known 'creation time' of the value,
-    stored as an epoch (i.e. from ``time.time()``).
-
-    If the value here is -1, it is assumed the value
-    should recreate immediately.
-
-    """
-
-    def acquire(self, creator,
-                        value_fn=None,
-                        value_and_created_fn=None):
-        """Acquire the lock, returning a context manager.
-
-        :param creator: Creation function, used if this thread
-         is chosen to create a new value.
-
-        :param value_fn: Optional function that returns
-         the value from some datasource.  Will be returned
-         if regeneration is not needed.
-
-        :param value_and_created_fn: Like value_fn, but returns a tuple
-         of (value, createdtime).  The returned createdtime
-         will replace the "createdtime" value on this dogpile
-         lock.   This option removes the need for the dogpile lock
-         itself to remain persistent across usages; another
-         dogpile can come along later and pick up where the
-         previous one left off.
-
-        """
-
-        if value_and_created_fn is None:
-            if value_fn is None:
-                def value_and_created_fn():
-                    return None, self.createdtime
-            else:
-                def value_and_created_fn():
-                    return value_fn(), self.createdtime
-
-            def creator_wrapper():
-                value = creator()
-                self.createdtime = time.time()
-                return value, self.createdtime
-        else:
-            def creator_wrapper():
-                value = creator()
-                self.createdtime = time.time()
-                return value
-
-        return Lock(
-            self.dogpilelock,
-            creator_wrapper,
-            value_and_created_fn,
-            self.expiretime
-        )
-
-    @property
-    def is_expired(self):
-        """Return true if the expiration time is reached, or no
-        value is available."""
-
-        return not self.has_value or \
-            (
-                self.expiretime is not None and
-                time.time() - self.createdtime > self.expiretime
-            )
-
-    @property
-    def has_value(self):
-        """Return true if the creation function has proceeded
-        at least once."""
-        return self.createdtime > 0
-
-
-class SyncReaderDogpile(Dogpile):
-    """Provide a read-write lock function on top of the :class:`.Dogpile`
-    class.
-
-    .. deprecated:: 0.4.0
-        The :class:`.ReadWriteMutex` object can be used directly.
-
-    """
-    def __init__(self, *args, **kw):
-        super(SyncReaderDogpile, self).__init__(*args, **kw)
-        self.readwritelock = ReadWriteMutex()
-
-    @contextlib.contextmanager
-    def acquire_write_lock(self):
-        """Return the "write" lock context manager.
-
-        This will provide a section that is mutexed against
-        all readers/writers for the dogpile-maintained value.
-
-        """
-
-        self.readwritelock.acquire_write_lock()
-        try:
-            yield
-        finally:
-            self.readwritelock.release_write_lock()
-
-    @contextlib.contextmanager
-    def acquire(self, *arg, **kw):
-        with super(SyncReaderDogpile, self).acquire(*arg, **kw) as value:
-            self.readwritelock.acquire_read_lock()
-            try:
-                yield value
-            finally:
-                self.readwritelock.release_read_lock()
@@ -1,83 +0,0 @@
-from .util import threading
-import weakref
-
-class NameRegistry(object):
-    """Generates and return an object, keeping it as a
-    singleton for a certain identifier for as long as its
-    strongly referenced.
-
-    e.g.::
-
-        class MyFoo(object):
-            "some important object."
-            def __init__(self, identifier):
-                self.identifier = identifier
-
-        registry = NameRegistry(MyFoo)
-
-        # thread 1:
-        my_foo = registry.get("foo1")
-
-        # thread 2
-        my_foo = registry.get("foo1")
-
-    Above, ``my_foo`` in both thread #1 and #2 will
-    be *the same object*.   The constructor for
-    ``MyFoo`` will be called once, passing the
-    identifier ``foo1`` as the argument.
-
-    When thread 1 and thread 2 both complete or
-    otherwise delete references to ``my_foo``, the
-    object is *removed* from the :class:`.NameRegistry` as
-    a result of Python garbage collection.
-
-    :param creator: A function that will create a new
-     value, given the identifier passed to the :meth:`.NameRegistry.get`
-     method.
-
-    """
-    _locks = weakref.WeakValueDictionary()
-    _mutex = threading.RLock()
-
-    def __init__(self, creator):
-        """Create a new :class:`.NameRegistry`.
-
-
-        """
-        self._values = weakref.WeakValueDictionary()
-        self._mutex = threading.RLock()
-        self.creator = creator
-
-    def get(self, identifier, *args, **kw):
-        """Get and possibly create the value.
-
-        :param identifier: Hash key for the value.
-         If the creation function is called, this identifier
-         will also be passed to the creation function.
-        :param \*args, \**kw: Additional arguments which will
-         also be passed to the creation function if it is
-         called.
-
-        """
-        try:
-            if identifier in self._values:
-                return self._values[identifier]
-            else:
-                return self._sync_get(identifier, *args, **kw)
-        except KeyError:
-            return self._sync_get(identifier, *args, **kw)
-
-    def _sync_get(self, identifier, *args, **kw):
-        self._mutex.acquire()
-        try:
-            try:
-                if identifier in self._values:
-                    return self._values[identifier]
-                else:
-                    self._values[identifier] = value = self.creator(identifier, *args, **kw)
-                    return value
-            except KeyError:
-                self._values[identifier] = value = self.creator(identifier, *args, **kw)
-                return value
-        finally:
-            self._mutex.release()
@@ -1,130 +0,0 @@
-from .util import threading
-
-import logging
-log = logging.getLogger(__name__)
-
-class LockError(Exception):
-    pass
-
-class ReadWriteMutex(object):
-    """A mutex which allows multiple readers, single writer.
-    
-    :class:`.ReadWriteMutex` uses a Python ``threading.Condition``
-    to provide this functionality across threads within a process.
-    
-    The Beaker package also contained a file-lock based version
-    of this concept, so that readers/writers could be synchronized
-    across processes with a common filesystem.  A future Dogpile 
-    release may include this additional class at some point.
-    
-    """
-
-    def __init__(self):
-        # counts how many asynchronous methods are executing
-        self.async = 0
-
-        # pointer to thread that is the current sync operation
-        self.current_sync_operation = None
-
-        # condition object to lock on
-        self.condition = threading.Condition(threading.Lock())
-
-    def acquire_read_lock(self, wait = True):
-        """Acquire the 'read' lock."""
-        self.condition.acquire()
-        try:
-            # see if a synchronous operation is waiting to start
-            # or is already running, in which case we wait (or just
-            # give up and return)
-            if wait:
-                while self.current_sync_operation is not None:
-                    self.condition.wait()
-            else:
-                if self.current_sync_operation is not None:
-                    return False
-
-            self.async += 1
-            log.debug("%s acquired read lock", self)
-        finally:
-            self.condition.release()
-
-        if not wait: 
-            return True
-
-    def release_read_lock(self):
-        """Release the 'read' lock."""
-        self.condition.acquire()
-        try:
-            self.async -= 1
-
-            # check if we are the last asynchronous reader thread 
-            # out the door.
-            if self.async == 0:
-                # yes. so if a sync operation is waiting, notifyAll to wake
-                # it up
-                if self.current_sync_operation is not None:
-                    self.condition.notifyAll()
-            elif self.async < 0:
-                raise LockError("Synchronizer error - too many "
-                                "release_read_locks called")
-            log.debug("%s released read lock", self)
-        finally:
-            self.condition.release()
-
-    def acquire_write_lock(self, wait = True):
-        """Acquire the 'write' lock."""
-        self.condition.acquire()
-        try:
-            # here, we are not a synchronous reader, and after returning,
-            # assuming waiting or immediate availability, we will be.
-
-            if wait:
-                # if another sync is working, wait
-                while self.current_sync_operation is not None:
-                    self.condition.wait()
-            else:
-                # if another sync is working,
-                # we dont want to wait, so forget it
-                if self.current_sync_operation is not None:
-                    return False
-
-            # establish ourselves as the current sync 
-            # this indicates to other read/write operations
-            # that they should wait until this is None again
-            self.current_sync_operation = threading.currentThread()
-
-            # now wait again for asyncs to finish
-            if self.async > 0:
-                if wait:
-                    # wait
-                    self.condition.wait()
-                else:
-                    # we dont want to wait, so forget it
-                    self.current_sync_operation = None
-                    return False
-            log.debug("%s acquired write lock", self)
-        finally:
-            self.condition.release()
-
-        if not wait: 
-            return True
-
-    def release_write_lock(self):
-        """Release the 'write' lock."""
-        self.condition.acquire()
-        try:
-            if self.current_sync_operation is not threading.currentThread():
-                raise LockError("Synchronizer error - current thread doesn't "
-                                "have the write lock")
-
-            # reset the current sync operation so 
-            # another can get it
-            self.current_sync_operation = None
-
-            # tell everyone to get ready
-            self.condition.notifyAll()
-
-            log.debug("%s released write lock", self)
-        finally:
-            # everyone go !!
-            self.condition.release()
@@ -1,8 +0,0 @@
-import sys
-py3k = sys.version_info >= (3, 0)
-
-try:
-    import threading
-except ImportError:
-    import dummy_threading as threading
-
--- a/Show More
+++ b/Show More