Compare commits

..

68 Commits

Author SHA1 Message Date
Diaoul ea1fe66884 Update documentation 2015-03-04 23:42:26 +01:00
Diaoul 2786f66c2a Update release notes 2015-03-04 23:41:34 +01:00
Diaoul eef70c4ed8 Bump version to 0.7.5 2015-03-04 20:47:37 +01:00
Diaoul 253b8ac69e Add python 2.6 to classifiers and travis 2015-03-04 20:47:08 +01:00
Diaoul e48bf7de02 Add PyCharm to gitignore 2015-03-04 20:46:32 +01:00
Antoine Bertin 8c040f2197 Merge pull request #404 from caronc/0.7.x
0.7.x
2015-03-04 20:31:06 +01:00
Chris Caron 407cbe1679 series/movie title sanitization refactored 2015-02-01 15:46:43 -05:00
Chris Caron 16ddb8a02e addic7ed timeout reverted back to 10 second timeout 2015-02-01 14:48:14 -05:00
Chris Caron c74aa01c09 random user agent determined at runtime; default user-agent reference centralized 2015-02-01 14:46:35 -05:00
Chris Caron 1b7291e45c Merge branch '0.7.x' of github:caronc/subliminal into 0.7.x 2015-01-30 16:38:45 -05:00
Chris Caron a86694d8d1 updated requirements so that Travis CI doesn't haul in guessit v0.10 2015-01-30 16:38:16 -05:00
Chris Caron e45fe70c95 updated requirements 2015-01-28 19:29:51 -05:00
Chris Caron 6c2833612c Merge branch '0.7.x' of github:caronc/subliminal into 0.7.x 2015-01-28 16:48:44 -05:00
Chris Caron 3acbefb9c2 fixes #428; I'm not proud of this fix; but it resolves the issue.
Hopefully the Addic7ed administrator can respond to my email inquiring as to why they are blocking us by the User-Agent string.

Automation is the 21st century of the internet; No one wants to click 8
times past banners just to get a 1KB (in size) subtitle. Most people
have Ad blocking software and don't even see these banners anyway. There
are many other ways to get people on board with helping them out
financially (if that's what this is about), and at the same time
accomodate those who've automated their service. I will roll back this
commit when we can come to a better resolution.
2015-01-28 16:48:26 -05:00
Chris Caron a25640b57f fixes #425; I'm not proud of this fix; but it resolves the issue.
Hopefully the Addic7ed administrator can respond to my email inquiring as to why they are blocking us by the User-Agent string.

Automation is the 21st century of the internet; No one wants to click 8
times past banners just to get a 1KB (in size) subtitle. Most people
have Ad blocking software and don't even see these banners anyway. There
are many other ways to get people on board with helping them out
financially (if that's what this is about), and at the same time
accomodate those who've automated their service. I will roll back this
commit when we can come to a better resolution.
2015-01-25 19:57:26 -05:00
Chris Caron d45e3b0941 improved logging for provider debugging 2015-01-25 19:47:53 -05:00
Chris Caron e6bd7040e3 scan_video() now takes pre-guessed video as optional input. This grants
the flexibility for 3rd party apps that wrap the subliminal framework to
do their own guess management. The default behaviour of this fuction
will remain the same as it always has. This new feature will only kick
in if the video is specified to be used instead.
2014-12-07 17:12:08 -05:00
Chris Caron 7fef3cfc93 scoring adjustments moved into compute_score() and some logging cleanup 2014-11-29 20:16:37 -05:00
Chris Caron bc575ea9b2 comparison functions added to help with 3rd party filtering 2014-11-29 20:15:28 -05:00
Chris Caron 3c7634fef2 eliminated bierdopje entry point to fix testing issue 2014-11-11 14:22:17 -05:00
Chris Caron d635269aa0 removed bierdopje provider since it's not referenced anymore anyway (tests for it were removed in an earlier commit) 2014-11-11 14:07:49 -05:00
Chris Caron 90d06e2072 logging output slighly adjusted (added some clarity and removed some unnecessary entries when not debugging) 2014-11-11 14:06:09 -05:00
Chris Caron 87aefa3a4d Massive overhaul on testing to make it Python v2.6 compatible 2014-11-11 13:17:51 -05:00
Chris Caron 6cdf18961e Fixed TVSubtitles.net matching 2014-11-11 13:16:57 -05:00
Chris Caron 827c75f092 better handling of duplicate download prevention 2014-11-11 13:16:12 -05:00
Chris Caron 1e9588e5b8 Added support for titles that contain quotes 2014-10-18 16:18:42 -04:00
Chris Caron 4702284e87 podnapisi website changes maded in Aug 2014 broke this provider in subliminal. Provider now supports new page layout 2014-09-18 10:03:20 -04:00
Chris Caron 88a2cb9681 added graceful handling of subtitle providers that are simply offline or unavailable. 2014-09-18 10:02:22 -04:00
Chris Caron 3b52a9385f added ability to prioritize multiple matched subtitles; Download Hearing Impaired (HI) first before non-HI or vs versa. This is kind of an extension to the bestscore enhancment already added in a previous commit. 2014-09-18 10:00:21 -04:00
Chris Caron 73a4e5a3af subliminal bugfix to prevent multiple matched subtitles from different providers being downloaded. Just download 1 (the best matched); bugfix 2014-09-18 09:52:48 -04:00
Chris Caron 71d206ad46 allow searching for subtitles by best score; not exclusively hearing impaired (HI) or non-HI 2014-09-18 09:51:39 -04:00
Chris Caron 14c7443635 applied guessit v0.7 support 2014-09-18 09:50:56 -04:00
Chris Caron e6dc714f7a Eliminated Dict Comprehensions (PEP 274) references to allow subliminal to work with python v2.4+ 2014-09-18 09:48:22 -04:00
Chris Caron 3b832a4564 updated guessit and babelfish minimum requirements 2014-09-18 09:47:50 -04:00
Antoine Bertin a6abc268a5 Release 0.7.4 2014-01-27 22:21:49 +01:00
Antoine Bertin 2dea4fef44 Update requirements
Exclude newest releases of guessit and babelfish
2014-01-27 22:20:21 +01:00
Antoine Bertin fe76634d02 Release 0.7.3 2013-11-22 21:05:49 +01:00
Antoine Bertin b499540bed Sync README and documentation 2013-11-22 20:47:56 +01:00
Antoine Bertin 7b4a9c2060 Fix wrong error catched for babelfish 0.4.0 2013-11-21 23:52:06 +01:00
Antoine Bertin a84cc80a88 Ignore IDE error in cli 2013-11-21 23:51:25 +01:00
Antoine Bertin 241cea9729 Fix podnapisi tests 2013-11-21 23:31:43 +01:00
Antoine Bertin 4b83ddc63e Improve assertions in tests 2013-11-21 23:31:02 +01:00
Antoine Bertin 0b431fbb8d Add Podnapisi to the list of providers in documentation 2013-11-21 00:38:01 +01:00
Antoine Bertin 3736d921a1 Update dogpile.cache to 0.5.2 and use a MutexLock in cli 2013-11-21 00:12:53 +01:00
Antoine Bertin 380fb28d2e Update to babelfish 0.4.0 2013-11-20 22:40:06 +01:00
Antoine Bertin 64c0ee4ccf Add setuptools to dev-requirements.txt 2013-11-20 22:38:55 +01:00
Antoine Bertin 5977bf69fb Improve embedded subtitles language detection 2013-11-14 22:08:48 +01:00
CelestianX 179ae6a24e Updated README with proper information
Section relative to the library was invalid. Missing references and
arguments.
2013-11-14 21:55:24 +01:00
Antoine Bertin 16942ec4c7 Switch to 0.7.3 2013-11-14 21:53:36 +01:00
Antoine Bertin 6f5378ea40 Be more permissive in subtitle validation 2013-11-14 21:51:39 +01:00
Antoine Bertin 02ee2039f4 Skip empty language in addic7ed 2013-11-14 21:51:04 +01:00
Antoine Bertin e7f89c1a19 Release 0.7.2 2013-11-10 11:06:46 +01:00
Antoine Bertin be4f9d92eb Remove unused import 2013-11-10 11:03:37 +01:00
Antoine Bertin ca63b97e79 Parse IETF language format in cli 2013-11-10 10:25:00 +01:00
Antoine Bertin c1ed4a0232 Fix exception handling when validating subtitle 2013-11-10 10:24:23 +01:00
Antoine Bertin b826a0bf08 Use debug level for subtitle track detection 2013-11-10 10:23:41 +01:00
Antoine Bertin fd0d87d719 Use info level when skipping providers 2013-11-10 10:23:24 +01:00
Antoine Bertin 094373f3c1 Add podnapisi provider 2013-11-10 10:22:33 +01:00
Antoine Bertin 5dac623c9f Update to babelfish 0.3.0 2013-11-09 20:25:39 +01:00
Antoine Bertin 57d1e772ec Use more list comprehension 2013-11-09 18:09:19 +01:00
Antoine Bertin 983efbfd9b Reduce debug logging in opensubtitles 2013-11-09 18:06:52 +01:00
Antoine Bertin 1f11e293c1 Add missing docstring 2013-11-09 18:06:17 +01:00
Antoine Bertin bfd278ae1c Change Subtitle repr language format 2013-11-09 18:05:48 +01:00
Antoine Bertin dbe1b9d2af Update guessit requirement 2013-11-09 03:04:13 +01:00
Antoine Bertin d71bc4bf09 Set CLI default cache expiration time to 30 days 2013-11-07 00:33:38 +01:00
Antoine Bertin faf2e1dfa4 Add a CACHE_VERSION to force cache reloading on version change 2013-11-07 00:26:06 +01:00
Antoine Bertin 93360aa1bb Fix find_show_id in tvsubtitles for ambiguous series 2013-11-06 21:08:09 +01:00
Antoine Bertin 8df7780ef9 Switch to 0.7.2 2013-11-06 00:42:44 +01:00
29 changed files with 894 additions and 525 deletions
+3
View File
@@ -34,6 +34,9 @@ pip-log.txt
# Rope
.ropeproject
# PyCharm
.idea
# Sphinx
docs/_build
+1
View File
@@ -1,6 +1,7 @@
language: python
python:
- "2.6"
- "2.7"
install:
+40
View File
@@ -1,6 +1,46 @@
Changelog
=========
0.7.5
-----
**release date:** 2015-03-04
* Update requirements
* Remove BierDopje provider
* Add pre-guessed video optional argument in scan_video
* Improve hearing impaired support
* Fix TVSubtitles and Podnapisi providers
0.7.4
-----
**release date:** 2014-01-27
* Fix requirements for guessit and babelfish
0.7.3
-----
**release date:** 2013-11-22
* Fix windows compatibility
* Improve subtitle validation
* Improve embedded subtitle languages detection
* Improve unittests
0.7.2
-----
**release date:** 2013-11-10
* Fix TVSubtitles for ambiguous series
* Add a CACHE_VERSION to force cache reloading on version change
* Set CLI default cache expiration time to 30 days
* Add podnapisi provider
* Support script for languages e.g. Latn, Cyrl
* Improve logging levels
* Fix subtitle validation in some rare cases
0.7.1
-----
**release date:** 2013-11-06
+2 -2
View File
@@ -17,8 +17,8 @@ Subliminal uses multiple providers to give users a vast choice and have a better
best matching subtitles. Providers are extensible through a dedicated entry point.
* Addic7ed
* BierDopje
* OpenSubtitles
* Podnapisi
* TheSubDB
* TvSubtitles
@@ -45,7 +45,7 @@ skipping videos that already have subtitles whether they are embedded or not::
subliminal.cache_region.configure('dogpile.cache.dbm', arguments={'filename': '/path/to/cachefile.dbm'})
# scan for videos in the folder and their subtitles
videos = scan_videos(['/path/to/video/folder'], subtitles=True, embedded_subtitles=True)
videos = subliminal.scan_videos(['/path/to/video/folder'], subtitles=True, embedded_subtitles=True, age=timedelta(weeks=1))
# download
subliminal.download_best_subtitles(videos, {Language('eng'), Language('fra')}, age=timedelta(week=1))
+1
View File
@@ -2,3 +2,4 @@ sympy>=0.7.3
sphinx>=1.1.3
sphinxcontrib-programoutput>=0.8
Sphinx-PyPI-upload>=0.2.1
setuptools>=1.4
+6 -2
View File
@@ -2,6 +2,10 @@ Cache
=====
.. module:: subliminal.cache
.. autodata:: region
.. autodata:: CACHE_VERSION
.. autofunction:: subliminal_key_generator
.. data:: region
Refer to `dogpile.cache's documentation <http://dogpilecache.readthedocs.org>`_ to see how to configure a region
The dogpile.cache region
Refer to `dogpile.cache's documentation <http://dogpilecache.readthedocs.org>`_ to see how to configure the region
+2 -2
View File
@@ -17,8 +17,8 @@ Subliminal uses multiple providers to give users a vast choice and have a better
best matching subtitles. Providers are extensible through a dedicated entry point.
* Addic7ed
* BierDopje
* OpenSubtitles
* Podnapisi
* TheSubDB
* TvSubtitles
@@ -47,7 +47,7 @@ skipping videos that already have subtitles whether they are embedded or not::
subliminal.cache_region.configure('dogpile.cache.dbm', arguments={'filename': '/path/to/cachefile.dbm'})
# scan for videos in the folder and their subtitles
videos = scan_videos(['/path/to/video/folder'], subtitles=True, embedded_subtitles=True)
videos = subliminal.scan_videos(['/path/to/video/folder'], subtitles=True, embedded_subtitles=True, age=timedelta(weeks=1))
# download
subliminal.download_best_subtitles(videos, {Language('eng'), Language('fra')}, age=timedelta(week=1))
+3 -3
View File
@@ -1,9 +1,9 @@
beautifulsoup4>=4.3.2
guessit>=0.6.1
guessit>=0.7,<0.10
requests>=2.0.1
enzyme>=0.4.0
html5lib>=0.99
dogpile.cache>=0.5.1
babelfish>=0.2.1
dogpile.cache>=0.5.2
babelfish>=0.5.0
charade>=1.0.3
pysrt>=0.5.0
+6 -4
View File
@@ -4,7 +4,7 @@ from setuptools import setup, find_packages
setup(name='subliminal',
version='0.7.1',
version='0.7.5',
license='MIT',
description='Subtitles, faster than your thoughts',
long_description=open('README.rst').read() + '\n\n' + open('HISTORY.rst').read(),
@@ -19,6 +19,7 @@ setup(name='subliminal',
'Operating System :: OS Independent',
'Programming Language :: Python',
'Programming Language :: Python :: 2',
'Programming Language :: Python :: 2.6',
'Programming Language :: Python :: 2.7',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.3',
@@ -27,12 +28,13 @@ setup(name='subliminal',
entry_points={
'console_scripts': ['subliminal = subliminal.cli:subliminal'],
'subliminal.providers': ['addic7ed = subliminal.providers.addic7ed:Addic7edProvider',
'bierdopje = subliminal.providers.bierdopje:BierDopjeProvider',
'opensubtitles = subliminal.providers.opensubtitles:OpenSubtitlesProvider',
'podnapisi = subliminal.providers.podnapisi:PodnapisiProvider',
'thesubdb = subliminal.providers.thesubdb:TheSubDBProvider',
'tvsubtitles = subliminal.providers.tvsubtitles:TVsubtitlesProvider'],
'babelfish.converters': ['addic7ed = subliminal.converters.addic7ed:Addic7edConverter',
'tvsubtitles = subliminal.converters.tvsubtitles:TVsubtitlesConverter']
'babelfish.language_converters': ['addic7ed = subliminal.converters.addic7ed:Addic7edConverter',
'podnapisi = subliminal.converters.podnapisi:PodnapisiConverter',
'tvsubtitles = subliminal.converters.tvsubtitles:TVsubtitlesConverter']
},
install_requires=open('requirements.txt').readlines(),
test_suite='subliminal.tests.suite')
+6 -3
View File
@@ -1,16 +1,19 @@
# -*- coding: utf-8 -*-
__title__ = 'subliminal'
__version__ = '0.7.1'
__version__ = '0.7.5'
__author__ = 'Antoine Bertin'
__license__ = 'MIT'
__copyright__ = 'Copyright 2013 Antoine Bertin'
import logging
from .api import PROVIDERS_ENTRY_POINT, list_subtitles, download_subtitles, download_best_subtitles
from .cache import region as cache_region
from .cache import MutexLock, region as cache_region
from .exceptions import Error, ProviderError, ProviderConfigurationError, ProviderNotAvailable, InvalidSubtitle
from .subtitle import Subtitle
from .video import VIDEO_EXTENSIONS, SUBTITLE_EXTENSIONS, Video, Episode, Movie, scan_videos, scan_video
class NullHandler(logging.Handler):
def emit(self, record):
pass
logging.getLogger(__name__).addHandler(logging.NullHandler())
logging.getLogger(__name__).addHandler(NullHandler())
+86 -28
View File
@@ -6,9 +6,10 @@ import logging
import operator
import babelfish
import pkg_resources
from os.path import basename
from .exceptions import ProviderNotAvailable, InvalidSubtitle
from .subtitle import get_subtitle_path
from socket import error as socket_error
logger = logging.getLogger(__name__)
@@ -47,11 +48,11 @@ def list_subtitles(videos, languages, providers=None, provider_configs=None):
Provider = provider_entry_point.load()
provider_languages = Provider.languages & languages - subtitle_languages
if not provider_languages:
logger.debug('Skipping provider %r: no language to search for', provider_entry_point.name)
logger.info('Skipping provider %r: no language to search for', provider_entry_point.name)
continue
provider_videos = [v for v in videos if Provider.check(v)]
if not provider_videos:
logger.debug('Skipping provider %r: no video to search for', provider_entry_point.name)
logger.info('Skipping provider %r: no video to search for', provider_entry_point.name)
continue
# list subtitles with the provider
@@ -67,16 +68,21 @@ def list_subtitles(videos, languages, providers=None, provider_configs=None):
provider_entry_point.name, provider_video, provider_video_languages)
try:
provider_subtitles = provider.list_subtitles(provider_video, provider_video_languages)
except ProviderNotAvailable:
except ProviderNotAvailable as err:
logger.warning('Provider %r is not available, discarding it', provider_entry_point.name)
logger.debug('ProviderNotAvailable error: %r', str(err))
break
except:
logger.exception('Unexpected error in provider %r', provider_entry_point.name)
continue
logger.info('Found %d subtitles', len(provider_subtitles))
logger.info('Found %d subtitle(s) on %s' % (
len(provider_subtitles),
provider_entry_point.name,
))
subtitles[provider_video].extend(provider_subtitles)
except ProviderNotAvailable:
except ProviderNotAvailable as err:
logger.warning('Provider %r is not available, discarding it', provider_entry_point.name)
logger.debug('ProviderNotAvailable error: %r', str(err))
return subtitles
@@ -92,15 +98,19 @@ def download_subtitles(subtitles, provider_configs=None, single=False):
"""
provider_configs = provider_configs or {}
discarded_providers = set()
providers_by_name = {ep.name: ep.load() for ep in pkg_resources.iter_entry_points(PROVIDERS_ENTRY_POINT)}
providers_by_name = dict([(ep.name, ep.load()) for ep in pkg_resources.iter_entry_points(PROVIDERS_ENTRY_POINT)])
initialized_providers = {}
downloaded_subtitles = collections.defaultdict(list)
fetched_subtitles = set()
try:
for video, video_subtitles in subtitles.items():
languages = {subtitle.language for subtitle in video_subtitles}
languages = set([subtitle.language for subtitle in video_subtitles])
downloaded_languages = set()
for subtitle in video_subtitles:
# filter
if subtitle.language in downloaded_languages:
logger.debug('Skipping subtitle: %r already downloaded', subtitle.language)
continue
if subtitle.provider_name in discarded_providers:
logger.debug('Skipping subtitle from discarded provider %r', subtitle.provider_name)
@@ -113,19 +123,35 @@ def download_subtitles(subtitles, provider_configs=None, single=False):
provider = providers_by_name[subtitle.provider_name](**provider_configs.get(subtitle.provider_name, {}))
try:
provider.initialize()
except ProviderNotAvailable:
except ProviderNotAvailable as err:
logger.warning('Provider %r is not available, discarding it', subtitle.provider_name)
logger.debug('ProviderNotAvailable error: %r', str(err))
discarded_providers.add(subtitle.provider_name)
continue
except socket_error as err:
logger.warning('Provider %r is not responding, discarding it', subtitle.provider_name)
logger.debug('Provider socket error: %r', str(err))
discarded_providers.add(subtitle.provider_name)
continue
except:
logger.exception('Unexpected error in provider %r', subtitle.provider_name)
discarded_providers.add(subtitle.provider_name)
continue
initialized_providers[subtitle.provider_name] = provider
# download subtitles
subtitle_path = get_subtitle_path(video.name, None if single else subtitle.language)
if basename(subtitle_path) in fetched_subtitles:
logger.debug('Skipping subtitle already retrieved %r', basename(subtitle_path))
continue
logger.info('Downloading subtitle %r into %r', subtitle, subtitle_path)
try:
subtitle_text = provider.download_subtitle(subtitle)
except ProviderNotAvailable:
downloaded_subtitles[video].append(subtitle)
except ProviderNotAvailable as err:
logger.warning('Provider %r is not available, discarding it', subtitle.provider_name)
logger.debug('ProviderNotAvailable error: %r', str(err))
discarded_providers.add(subtitle.provider_name)
continue
except InvalidSubtitle:
@@ -136,8 +162,9 @@ def download_subtitles(subtitles, provider_configs=None, single=False):
continue
with io.open(subtitle_path, 'w', encoding='utf-8') as f:
f.write(subtitle_text)
downloaded_languages.add(subtitle.language)
if single or downloaded_languages == languages:
downloaded_languages.add(subtitle.language)
fetched_subtitles.add(basename(subtitle_path))
if single or sorted(downloaded_languages) == sorted(languages):
break
finally: # terminate providers
for (provider_name, provider) in initialized_providers.items():
@@ -147,10 +174,11 @@ def download_subtitles(subtitles, provider_configs=None, single=False):
logger.warning('Provider %r is not available, unable to terminate', provider_name)
except:
logger.exception('Unexpected error in provider %r', provider_name)
return downloaded_subtitles
def download_best_subtitles(videos, languages, providers=None, provider_configs=None, single=False, min_score=0,
hearing_impaired=False):
hearing_impaired=False, hi_score_adjust=0):
"""Download the best subtitles for `videos` with the given `languages` using the specified `providers`
:param videos: videos to download subtitles for
@@ -164,11 +192,13 @@ def download_best_subtitles(videos, languages, providers=None, provider_configs=
:param bool single: download with .srt extension if `True`, add language identifier otherwise
:param int min_score: minimum score for subtitles to download
:param bool hearing_impaired: download hearing impaired subtitles
:param int hi_score_adjust: Adjust hearing_impaired_scores if matched.
"""
provider_configs = provider_configs or {}
discarded_providers = set()
downloaded_subtitles = collections.defaultdict(list)
fetched_subtitles = set()
# filter videos
videos = [v for v in videos if v.subtitle_languages & languages < languages
and (not single or babelfish.Language('und') not in v.subtitle_languages)]
@@ -187,24 +217,34 @@ def download_best_subtitles(videos, languages, providers=None, provider_configs=
logger.debug('Skipping provider %r: no language to search for', provider_entry_point.name)
continue
if not [v for v in videos if Provider.check(v)]:
logger.debug('Skipping provider %r: no video to search for', provider_entry_point.name)
logger.debug('Skipping provider %r: video type not hosted here.', provider_entry_point.name)
continue
provider = Provider(**provider_configs.get(provider_entry_point.name, {}))
try:
provider.initialize()
except ProviderNotAvailable:
except ProviderNotAvailable as err:
logger.warning('Provider %r is not available, discarding it', provider_entry_point.name)
logger.debug('ProviderNotAvailable error: %r', str(err))
continue
except socket_error as err:
logger.warning('Provider %r is not responding, discarding it', provider_entry_point.name)
logger.debug('Provider socket error: %r', str(err))
continue
except:
logger.exception('Unexpected error in provider %r', provider_entry_point.name)
continue
initialized_providers[provider_entry_point.name] = provider
try:
for video in videos:
# search for subtitles
subtitles = []
downloaded_languages = set()
for provider_name, provider in initialized_providers.items():
if provider.check(video):
if provider_name in discarded_providers:
logger.debug('Skipping discarded provider %r', provider_name)
continue
provider_video_languages = provider.languages & languages - video.subtitle_languages
if not provider_video_languages:
logger.debug('Skipping provider %r: no language to search for for video %r', provider_name,
@@ -214,30 +254,38 @@ def download_best_subtitles(videos, languages, providers=None, provider_configs=
provider_name, video, provider_video_languages)
try:
provider_subtitles = provider.list_subtitles(video, provider_video_languages)
except ProviderNotAvailable:
except ProviderNotAvailable as err:
logger.warning('Provider %r is not available, discarding it', provider_name)
logger.debug('ProviderNotAvailable error: %r', str(err))
discarded_providers.add(provider_name)
continue
except:
logger.exception('Unexpected error in provider %r', provider_name)
continue
logger.info('Found %d subtitles', len(provider_subtitles))
logger.info('Found %d subtitle(s) on %s' % (
len(provider_subtitles),
provider_name,
))
subtitles.extend(provider_subtitles)
# find the best subtitles and download them
downloaded_languages = video.subtitle_languages.copy()
for subtitle, score in sorted([(s, s.compute_score(video)) for s in subtitles],
key=operator.itemgetter(1), reverse=True):
for subtitle, score in sorted([(s, s.compute_score(video, hi_score_adjust)) \
for s in subtitles], key=operator.itemgetter(1), reverse=True):
# filter
if subtitle.provider_name in discarded_providers:
logger.debug('Skipping subtitle from discarded provider %r', subtitle.provider_name)
continue
if subtitle.hearing_impaired != hearing_impaired:
logger.debug('Skipping subtitle: hearing impaired != %r', hearing_impaired)
continue
if hearing_impaired is not None:
if subtitle.hearing_impaired != hearing_impaired:
logger.debug('Skipping subtitle: hearing impaired != %r', hearing_impaired)
continue
if score < min_score:
logger.debug('Skipping subtitle: score < %d', min_score)
continue
if subtitle.language in downloaded_languages:
logger.debug('Skipping subtitle: %r already downloaded', subtitle.language)
continue
@@ -245,12 +293,17 @@ def download_best_subtitles(videos, languages, providers=None, provider_configs=
# download
provider = initialized_providers[subtitle.provider_name]
subtitle_path = get_subtitle_path(video.name, None if single else subtitle.language)
if basename(subtitle_path) in fetched_subtitles:
logger.debug('Skipping subtitle already retrieved %r', basename(subtitle_path))
continue
logger.info('Downloading subtitle %r with score %d into %r', subtitle, score, subtitle_path)
try:
subtitle_text = provider.download_subtitle(subtitle)
downloaded_subtitles[video].append(subtitle)
except ProviderNotAvailable:
except ProviderNotAvailable as err:
logger.warning('Provider %r is not available, discarding it', subtitle.provider_name)
logger.debug('ProviderNotAvailable error: %r', str(err))
discarded_providers.add(subtitle.provider_name)
continue
except InvalidSubtitle:
@@ -261,16 +314,21 @@ def download_best_subtitles(videos, languages, providers=None, provider_configs=
continue
with io.open(subtitle_path, 'w', encoding='utf-8') as f:
f.write(subtitle_text)
downloaded_languages.add(subtitle.language)
if single or downloaded_languages >= languages:
logger.debug('All languages downloaded')
downloaded_languages.add(subtitle.language)
fetched_subtitles.add(basename(subtitle_path))
if single or sorted(downloaded_languages) == sorted(languages):
break
finally: # terminate providers
for (provider_name, provider) in initialized_providers.items():
try:
provider.terminate()
except ProviderNotAvailable:
except ProviderNotAvailable as err:
logger.warning('Provider %r is not available, unable to terminate', provider_name)
logger.debug('ProviderNotAvailable error: %r', str(err))
except socket_error as err:
logger.warning('Provider %r is not available, unable to terminate', provider_name)
logger.debug('Provider socket error: %r', str(err))
except:
logger.exception('Unexpected error in provider %r', provider_name)
return downloaded_subtitles
+53 -3
View File
@@ -1,6 +1,56 @@
# -*- coding: utf-8 -*-
import dogpile.cache
import inspect
from dogpile.cache import make_region # @UnresolvedImport
from dogpile.cache.backends.file import AbstractFileLock # @UnresolvedImport
from dogpile.cache.compat import string_type # @UnresolvedImport
from dogpile.core.readwrite_lock import ReadWriteMutex # @UnresolvedImport
#: The subliminal's dogpile.cache region
region = dogpile.cache.make_region()
#: Subliminal's cache version
CACHE_VERSION = 2
def subliminal_key_generator(namespace, fn, to_str=string_type):
"""Add a :data:`CACHE_VERSION` to dogpile.cache's default function_key_generator"""
if namespace is None:
namespace = '%d:%s:%s' % (CACHE_VERSION, fn.__module__, fn.__name__)
else:
namespace = '%d:%s:%s|%s' % (CACHE_VERSION, fn.__module__, fn.__name__, namespace)
args = inspect.getargspec(fn)
has_self = args[0] and args[0][0] in ('self', 'cls')
def generate_key(*args, **kw):
if kw:
raise ValueError('Keyword arguments not supported')
if has_self:
args = args[1:]
return namespace + '|' + ' '.join(map(to_str, args))
return generate_key
class MutexLock(AbstractFileLock):
""":class:`MutexLock` is a thread-based rw lock based on :class:`dogpile.core.ReadWriteMutex`"""
def __init__(self, filename):
self.mutex = ReadWriteMutex()
def acquire_read_lock(self, wait):
ret = self.mutex.acquire_read_lock(wait)
return wait or ret
def acquire_write_lock(self, wait):
ret = self.mutex.acquire_write_lock(wait)
return wait or ret
def release_read_lock(self):
return self.mutex.release_read_lock()
def release_write_lock(self):
return self.mutex.release_write_lock()
#: The dogpile.cache region (long-lived)
region = make_region(function_key_generator=subliminal_key_generator)
#: The dogpile.cache region for :meth:`~subliminal.providers.Provider.query` (short-lived)
query_region = make_region(function_key_generator=subliminal_key_generator)
+8 -17
View File
@@ -9,7 +9,7 @@ import sys
import babelfish
import guessit
import pkg_resources
from subliminal import (__version__, PROVIDERS_ENTRY_POINT, cache_region, Video, Episode, Movie, scan_videos,
from subliminal import (__version__, PROVIDERS_ENTRY_POINT, cache_region, MutexLock, Video, Episode, Movie, scan_videos,
download_best_subtitles)
try:
import colorlog
@@ -29,7 +29,7 @@ def subliminal():
required_arguments_group = parser.add_argument_group('required arguments')
required_arguments_group.add_argument('paths', nargs='+', metavar='PATH', help='path to video file or folder')
required_arguments_group.add_argument('-l', '--languages', nargs='+', required=True, metavar='LANGUAGE',
help='wanted languages as alpha2 code (ISO-639-1)')
help='wanted languages as IETF codes e.g. fr, pt-BR, sr-Cyrl ')
# configuration
configuration_group = parser.add_argument_group('configuration')
@@ -52,11 +52,6 @@ def subliminal():
filtering_group.add_argument('-f', '--force', action='store_true',
help='force subtitle download for videos with existing subtitles')
# addic7ed
addic7ed_group = parser.add_argument_group('addic7ed')
addic7ed_group.add_argument('--addic7ed-username', metavar='USERNAME', help='username for addic7ed provider')
addic7ed_group.add_argument('--addic7ed-password', metavar='PASSWORD', help='password for addic7ed provider')
# output
output_group = parser.add_argument_group('output')
output_exclusive_group = output_group.add_mutually_exclusive_group()
@@ -81,16 +76,16 @@ def subliminal():
# parse languages
try:
args.languages = {babelfish.Language.fromalpha2(l) for l in args.languages}
args.languages = set( babelfish.Language.fromietf(l) for l in args.languages )
except babelfish.Error:
parser.error('argument -l/--languages: codes are not ISO-639-1: %r' % args.languages)
parser.error('argument -l/--languages: codes are not IETF: %r' % args.languages)
# parse age
if args.age is not None:
match = re.match(r'^(?:(?P<weeks>\d+?)w)?(?:(?P<days>\d+?)d)?(?:(?P<hours>\d+?)h)?$', args.age)
if not match:
parser.error('argument -a/--age: invalid age: %r' % args.age)
args.age = datetime.timedelta(**{k: int(v) for k, v in match.groupdict(0).items()})
args.age = datetime.timedelta(**dict([(k, int(v)) for k, v in match.groupdict(0).items()]))
# parse cache-file
args.cache_file = os.path.abspath(os.path.expanduser(args.cache_file))
@@ -100,11 +95,6 @@ def subliminal():
# parse provider configs
provider_configs = {}
if (args.addic7ed_username is not None and args.addic7ed_password is None
or args.addic7ed_username is None and args.addic7ed_password is not None):
parser.error('argument --addic7ed-username/--addic7ed-password: both arguments are required or none')
if args.addic7ed_username is not None and args.addic7ed_password is not None:
provider_configs['addic7ed'] = {'username': args.addic7ed_username, 'password': args.addic7ed_password}
# parse color
if args.color and colorlog is None:
@@ -138,14 +128,15 @@ def subliminal():
logging.getLogger('subliminal.api').setLevel(logging.INFO)
# configure cache
cache_region.configure('dogpile.cache.dbm', arguments={'filename': args.cache_file})
cache_region.configure('dogpile.cache.dbm', expiration_time=datetime.timedelta(days=30), # @UndefinedVariable
arguments={'filename': args.cache_file, 'lock_factory': MutexLock})
# scan videos
videos = scan_videos([p for p in args.paths if os.path.exists(p)], subtitles=not args.force,
embedded_subtitles=not args.force, age=args.age)
# guess videos
videos.extend([Video.fromguess(os.path.split(p)[1], guessit.guess_file_info(p, 'autodetect')) for p in args.paths
videos.extend([Video.fromguess(os.path.split(p)[1], guessit.guess_file_info(p, info=['filename'])) for p in args.paths
if not os.path.exists(p)])
# download best subtitles
+19 -17
View File
@@ -1,29 +1,31 @@
# -*- coding: utf-8 -*-
from __future__ import unicode_literals
from babelfish.converters.name import NameConverter
from babelfish import LanguageReverseConverter, language_converters
class Addic7edConverter(NameConverter):
class Addic7edConverter(LanguageReverseConverter):
def __init__(self):
super(Addic7edConverter, self).__init__()
self.from_addic7ed = {'Català': ('cat', None), 'Chinese (Simplified)': ('zho', None),
'Chinese (Traditional)': ('zho', None), 'Euskera': ('eus', None),
'Galego': ('glg', None), 'Greek': ('ell', None),
'Malay': ('msa', None), 'Portuguese (Brazilian)': ('por', 'BR'),
'Serbian (Cyrillic)': ('srp', None), 'Serbian (Latin)': ('srp', None),
'Spanish (Latin America)': ('spa', None), 'Spanish (Spain)': ('spa', None)}
self.to_addic7ed = {('cat', None): 'Català', ('zho', None): 'Chinese (Simplified)',
('eus', None): 'Euskera', ('glg', None): 'Galego',
('ell', None): 'Greek', ('msa', None): 'Malay',
('por', 'BR'): 'Portuguese (Brazilian)', ('srp', None): 'Serbian (Cyrillic)'}
self.codes |= set(self.from_addic7ed.keys())
self.name_converter = language_converters['name']
self.from_addic7ed = {'Català': ('cat',), 'Chinese (Simplified)': ('zho',), 'Chinese (Traditional)': ('zho',),
'Euskera': ('eus',), 'Galego': ('glg',), 'Greek': ('ell',), 'Malay': ('msa',),
'Portuguese (Brazilian)': ('por', 'BR'), 'Serbian (Cyrillic)': ('srp', None, 'Cyrl'),
'Serbian (Latin)': ('srp',), 'Spanish (Latin America)': ('spa',),
'Spanish (Spain)': ('spa',)}
self.to_addic7ed = {('cat',): 'Català', ('zho',): 'Chinese (Simplified)', ('eus',): 'Euskera',
('glg',): 'Galego', ('ell',): 'Greek', ('msa',): 'Malay',
('por', 'BR'): 'Portuguese (Brazilian)', ('srp', None, 'Cyrl'): 'Serbian (Cyrillic)'}
self.codes = self.name_converter.codes | set(self.from_addic7ed.keys())
def convert(self, alpha3, country=None):
def convert(self, alpha3, country=None, script=None):
if (alpha3, country, script) in self.to_addic7ed:
return self.to_addic7ed[(alpha3, country, script)]
if (alpha3, country) in self.to_addic7ed:
return self.to_addic7ed[(alpha3, country)]
return super(Addic7edConverter, self).convert(alpha3, country)
if (alpha3,) in self.to_addic7ed:
return self.to_addic7ed[(alpha3,)]
return self.name_converter.convert(alpha3, country, script)
def reverse(self, addic7ed):
if addic7ed in self.from_addic7ed:
return self.from_addic7ed[addic7ed]
return super(Addic7edConverter, self).reverse(addic7ed)
return self.name_converter.reverse(addic7ed)
+32
View File
@@ -0,0 +1,32 @@
# -*- coding: utf-8 -*-
from __future__ import unicode_literals
from babelfish import LanguageReverseConverter, LanguageConvertError, LanguageReverseError
class PodnapisiConverter(LanguageReverseConverter):
def __init__(self):
self.from_podnapisi = {2: ('eng',), 28: ('spa',), 26: ('pol',), 36: ('srp',), 1: ('slv',), 38: ('hrv',),
9: ('ita',), 8: ('fra',), 48: ('por', 'BR'), 23: ('nld',), 12: ('ara',), 13: ('ron',),
33: ('bul',), 32: ('por',), 16: ('ell',), 15: ('hun',), 31: ('fin',), 30: ('tur',),
7: ('ces',), 25: ('swe',), 27: ('rus',), 24: ('dan',), 22: ('heb',), 51: ('vie',),
52: ('fas',), 5: ('deu',), 14: ('spa', 'AR'), 54: ('ind',), 47: ('srp', None, 'Cyrl'),
3: ('nor',), 20: ('est',), 10: ('bos',), 17: ('zho',), 37: ('slk',), 35: ('mkd',),
11: ('jpn',), 4: ('kor',), 29: ('sqi',), 6: ('isl',), 19: ('lit',), 46: ('ukr',),
44: ('tha',), 53: ('cat',), 56: ('sin',), 21: ('lav',), 40: ('cmn',), 55: ('msa',),
42: ('hin',), 50: ('bel',)}
self.to_podnapisi = dict([(v, k) for k, v in self.from_podnapisi.items()])
self.codes = set(self.from_podnapisi.keys())
def convert(self, alpha3, country=None, script=None):
if (alpha3,) in self.to_podnapisi:
return self.to_podnapisi[(alpha3,)]
if (alpha3, country) in self.to_podnapisi:
return self.to_podnapisi[(alpha3, country)]
if (alpha3, country, script) in self.to_podnapisi:
return self.to_podnapisi[(alpha3, country, script)]
raise LanguageConvertError(alpha3, country, script)
def reverse(self, podnapisi):
if podnapisi not in self.from_podnapisi:
raise LanguageReverseError(podnapisi)
return self.from_podnapisi[podnapisi]
+12 -10
View File
@@ -1,22 +1,24 @@
# -*- coding: utf-8 -*-
from __future__ import unicode_literals
from babelfish.converters.alpha2 import Alpha2Converter
from babelfish import LanguageReverseConverter, language_converters
class TVsubtitlesConverter(Alpha2Converter):
class TVsubtitlesConverter(LanguageReverseConverter):
def __init__(self):
super(TVsubtitlesConverter, self).__init__()
self.from_tvsubtitles = {'br': ('por', 'BR'), 'ua': ('ukr', None), 'gr': ('ell', None), 'cn': ('zho', None),
'jp': ('jpn', None), 'cz': ('ces', None)}
self.to_tvsubtitles = {v: k for k, v in self.from_tvsubtitles}
self.codes |= set(self.from_tvsubtitles.keys())
self.alpha2_converter = language_converters['alpha2']
self.from_tvsubtitles = {'br': ('por', 'BR'), 'ua': ('ukr',), 'gr': ('ell',), 'cn': ('zho',), 'jp': ('jpn',),
'cz': ('ces',)}
self.to_tvsubtitles = set([(v, k) for k, v in self.from_tvsubtitles])
self.codes = self.alpha2_converter.codes | set(self.from_tvsubtitles.keys())
def convert(self, alpha3, country=None):
def convert(self, alpha3, country=None, script=None):
if (alpha3, country) in self.to_tvsubtitles:
return self.to_tvsubtitles[(alpha3, country)]
return super(TVsubtitlesConverter, self).convert(alpha3, country)
if (alpha3,) in self.to_tvsubtitles:
return self.to_tvsubtitles[(alpha3,)]
return self.alpha2_converter.convert(alpha3, country, script)
def reverse(self, tvsubtitles):
if tvsubtitles in self.from_tvsubtitles:
return self.from_tvsubtitles[tvsubtitles]
return super(TVsubtitlesConverter, self).reverse(tvsubtitles)
return self.alpha2_converter.reverse(tvsubtitles)
+25
View File
@@ -2,7 +2,25 @@
from __future__ import unicode_literals
import babelfish
from ..video import Episode, Movie
from .. import __version__
from random import randint
# Agent List
AGENT_LIST = (
'Mozilla/5.0 (Windows NT 6.3; rv:36.0) Gecko/20100101 Firefox/36.0',
'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.1 Safari/537.36',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10; rv:33.0) Gecko/20100101 Firefox/33.0',
'Mozilla/5.0 (X11; Linux i586; rv:31.0) Gecko/20100101 Firefox/31.0',
'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20130401 Firefox/31.0',
'Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0',
'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.0 Safari/537.36',
'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.0 Safari/537.36',
'Mozilla/5.0 (compatible, MSIE 11, Windows NT 6.3; Trident/7.0; rv:11.0) like Gecko',
'Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.75.14 (KHTML, like Gecko) Version/7.0.3 Safari/7046A194A',
'Mozilla/5.0 (iPad; CPU OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5355d Safari/8536.25',
)
class Provider(object):
"""Base class for providers
@@ -22,6 +40,13 @@ class Provider(object):
#: Required hash, if any
required_hash = None
# Returns a random agent to use from the list above
random_user_agent = AGENT_LIST[randint(0, len(AGENT_LIST)-1)]
# Defines the ideal user agent to use for all providers otherwise
primary_user_agent = 'Subliminal/%s' % __version__
def __init__(self, **kwargs):
pass
+16 -41
View File
@@ -6,10 +6,9 @@ import bs4
import charade
import requests
from . import Provider
from .. import __version__
from ..cache import region
from ..exceptions import ProviderConfigurationError, ProviderNotAvailable, InvalidSubtitle
from ..subtitle import Subtitle, is_valid_subtitle
from ..subtitle import Subtitle, is_valid_subtitle, sanitize_string
from ..video import Episode
@@ -53,49 +52,20 @@ class Addic7edSubtitle(Subtitle):
class Addic7edProvider(Provider):
languages = {babelfish.Language('por', 'BR')} | {babelfish.Language(l)
languages = set([babelfish.Language('por', 'BR')]) | set([babelfish.Language(l)
for l in ['ara', 'aze', 'ben', 'bos', 'bul', 'cat', 'ces', 'dan', 'deu', 'ell', 'eng', 'eus', 'fas',
'fin', 'fra', 'glg', 'heb', 'hrv', 'hun', 'hye', 'ind', 'ita', 'jpn', 'kor', 'mkd', 'msa',
'nld', 'nor', 'pol', 'por', 'ron', 'rus', 'slk', 'slv', 'spa', 'sqi', 'srp', 'swe', 'tha',
'tur', 'ukr', 'vie', 'zho']}
'tur', 'ukr', 'vie', 'zho']])
video_types = (Episode,)
server = 'http://www.addic7ed.com'
def __init__(self, username=None, password=None):
if username is not None and password is None or username is None and password is not None:
raise ProviderConfigurationError('Username and password must be specified')
self.username = username
self.password = password
self.logged_in = False
def initialize(self):
self.session = requests.Session()
self.session.headers = {'User-Agent': 'Subliminal/%s' % __version__}
# login
if self.username is not None and self.password is not None:
logger.debug('Logging in')
data = {'username': self.username, 'password': self.password, 'Submit': 'Log in'}
try:
r = self.session.post(self.server + '/dologin.php', data, timeout=10, allow_redirects=False)
except requests.Timeout:
raise ProviderNotAvailable('Timeout after 10 seconds')
if r.status_code == 302:
logger.info('Logged in')
self.logged_in = True
else:
logger.error('Failed to login')
def terminate(self):
# logout
if self.logged_in:
try:
r = self.session.get(self.server + '/logout.php', timeout=10)
logger.info('Logged out')
except requests.Timeout:
raise ProviderNotAvailable('Timeout after 10 seconds')
if r.status_code != 200:
raise ProviderNotAvailable('Request failed with status code %d' % r.status_code)
self.session.close()
self.session.headers = {
'User-Agent': self.random_user_agent,
'Referer': self.server,
}
def get(self, url, params=None):
"""Make a GET request on `url` with the given parameters
@@ -126,7 +96,8 @@ class Addic7edProvider(Provider):
soup = self.get('/shows.php')
show_ids = {}
for html_show in soup.select('td.version > h3 > a[href^="/show/"]'):
show_ids[html_show.string.lower()] = int(html_show['href'][6:])
show_ids[sanitize_string(html_show.string)] = \
int(html_show['href'][6:])
return show_ids
@region.cache_on_arguments()
@@ -150,10 +121,11 @@ class Addic7edProvider(Provider):
def query(self, series, season):
show_ids = self.get_show_ids()
if series.lower() in show_ids:
show_id = show_ids[series.lower()]
sanitized_series = sanitize_string(series)
if sanitized_series in show_ids:
show_id = show_ids[sanitized_series]
else:
show_id = self.find_show_id(series.lower())
show_id = self.find_show_id(sanitized_series)
if show_id is None:
return []
params = {'show_id': show_id, 'season': season}
@@ -166,6 +138,9 @@ class Addic7edProvider(Provider):
if cells[5].string != 'Completed':
logger.debug('Skipping incomplete subtitle')
continue
if not cells[3].string:
logger.debug('Skipping empty language')
continue
subtitles.append(Addic7edSubtitle(babelfish.Language.fromaddic7ed(cells[3].string), series, season,
int(cells[1].string), cells[2].string, cells[4].string,
bool(cells[6].string), cells[9].a['href'], link))
-138
View File
@@ -1,138 +0,0 @@
# -*- coding: utf-8 -*-
from __future__ import unicode_literals
import logging
import urllib
import babelfish
import charade
import guessit
import requests
import xml.etree.ElementTree
from . import Provider
from .. import __version__
from ..cache import region
from ..exceptions import InvalidSubtitle, ProviderNotAvailable, ProviderError
from ..subtitle import Subtitle, is_valid_subtitle, compute_guess_matches
from ..video import Episode
logger = logging.getLogger(__name__)
class BierDopjeSubtitle(Subtitle):
provider_name = 'bierdopje'
def __init__(self, language, season, episode, tvdb_id, series, filename, download_link):
super(BierDopjeSubtitle, self).__init__(language)
self.season = season
self.episode = episode
self.tvdb_id = tvdb_id
self.series = series
self.filename = filename
self.download_link = download_link
def compute_matches(self, video):
matches = set()
# tvdb_id
if video.tvdb_id and self.tvdb_id == video.tvdb_id:
matches.add('tvdb_id')
# series
if video.series and self.series == video.series:
matches.add('series')
# season
if video.season and self.season == video.season:
matches.add('season')
# episode
if video.episode and self.episode == video.episode:
matches.add('episode')
matches |= compute_guess_matches(video, guessit.guess_episode_info(self.filename + '.mkv'))
return matches
class BierDopjeProvider(Provider):
languages = {babelfish.Language(l) for l in ['eng', 'nld']}
video_types = (Episode,)
def initialize(self):
self.session = requests.Session()
self.session.headers = {'User-Agent': 'Subliminal/%s' % __version__}
def terminate(self):
self.session.close()
def get(self, url, **params):
"""Make a GET request on the `url` formatted with `**params`
:param string url: API part of the URL to reach without the leading slash
:param \*\*params: format specs for the `url`
:return: the response
:rtype: :class:`xml.etree.ElementTree.Element`
:raise: :class:`~subliminal.exceptions.ProviderNotAvailable`
"""
try:
r = self.session.get('http://api.bierdopje.com/A2B638AC5D804C2E/' + url.format(**params), timeout=10)
except requests.Timeout:
raise ProviderNotAvailable('Timeout after 10 seconds')
if r.status_code == 429:
raise ProviderNotAvailable('Too Many Requests')
elif r.status_code != 200:
raise ProviderError('Request failed with status code %d' % r.status_code)
return xml.etree.ElementTree.fromstring(r.content)
@region.cache_on_arguments()
def find_show_id(self, series):
"""Find the show id from series name
:param string series: series of the episode
:return: show id
:rtype: int
"""
logger.debug('Searching for series %r', series)
root = self.get('FindShowByName/{series}', series=urllib.quote(series))
if root.find('response/status').text == 'false':
logger.info('Series %r not found', series)
return None
return int(root.find('response/results/result[1]/showid').text)
def query(self, language, season, episode, tvdb_id=None, series=None):
params = {'language': language.alpha2, 'season': season, 'episode': episode}
if tvdb_id is not None:
params['showid'] = tvdb_id
params['istvdbid'] = 'true'
elif series is not None:
show_id = self.find_show_id(series)
if show_id is None:
return []
params['showid'] = show_id
params['istvdbid'] = 'false'
else:
raise ValueError('Missing parameter tvdb_id or series')
logger.debug('Searching subtitles %r', params)
root = self.get('GetAllSubsFor/{showid}/{season}/{episode}/{language}/{istvdbid}', **params)
if root.find('response/status').text == 'false':
logger.debug('No subtitle found')
return []
logger.debug('Found subtitles %r', root.find('response/results'))
return [BierDopjeSubtitle(language, season, episode, tvdb_id, series, result.find('filename').text,
result.find('downloadlink').text) for result in root.find('response/results')]
def list_subtitles(self, video, languages):
subtitles = []
for language in languages:
subtitles.extend(self.query(language, video.season, video.episode, video.tvdb_id, video.series))
return subtitles
def download_subtitle(self, subtitle):
try:
r = self.session.get(subtitle.download_link, timeout=10)
except requests.Timeout:
raise ProviderNotAvailable('Timeout after 10 seconds')
if r.status_code == 429:
raise ProviderNotAvailable('Too Many Requests')
elif r.status_code != 200:
raise ProviderError('Request failed with status code %d' % r.status_code)
subtitle_text = r.content.decode(charade.detect(r.content)['encoding'], 'replace')
if not is_valid_subtitle(subtitle_text):
raise InvalidSubtitle
return subtitle_text
+13 -8
View File
@@ -13,6 +13,7 @@ from . import Provider
from .. import __version__
from ..exceptions import ProviderError, ProviderNotAvailable, InvalidSubtitle
from ..subtitle import Subtitle, is_valid_subtitle, compute_guess_matches
from ..subtitle import sanitize_string
from ..video import Episode, Movie
@@ -23,8 +24,8 @@ class OpenSubtitlesSubtitle(Subtitle):
provider_name = 'opensubtitles'
series_re = re.compile('^"(?P<series_name>.*)" (?P<series_title>.*)$')
def __init__(self, language, hearing_impaired, id, matched_by, movie_kind, hash, movie_name, movie_release_name, movie_year,
movie_imdb_id, series_season, series_episode):
def __init__(self, language, hearing_impaired, id, matched_by, movie_kind, hash, movie_name, movie_release_name, # @ReservedAssignment
movie_year, movie_imdb_id, series_season, series_episode):
super(OpenSubtitlesSubtitle, self).__init__(language, hearing_impaired)
self.id = id
self.matched_by = matched_by
@@ -50,7 +51,9 @@ class OpenSubtitlesSubtitle(Subtitle):
# episode
if isinstance(video, Episode) and self.movie_kind == 'episode':
# series
if video.series and self.series_name.lower() == video.series.lower():
if video.series and \
sanitize_string(self.series_name) == \
sanitize_string(video.series):
matches.add('series')
# season
if video.season and self.series_season == video.season:
@@ -77,13 +80,15 @@ class OpenSubtitlesSubtitle(Subtitle):
if video.imdb_id and self.movie_imdb_id == video.imdb_id:
matches.add('imdb_id')
# title
if video.title and self.movie_name.lower() == video.title.lower():
if video.title and \
sanitize_string(self.movie_name) == \
sanitize_string(video.title):
matches.add('title')
return matches
class OpenSubtitlesProvider(Provider):
languages = {babelfish.Language.fromopensubtitles(l) for l in babelfish.CONVERTERS['opensubtitles'].codes}
languages = set([babelfish.Language.fromopensubtitles(l) for l in babelfish.language_converters['opensubtitles'].codes])
def __init__(self):
self.server = xmlrpclib.ServerProxy('http://api.opensubtitles.org/xml-rpc')
@@ -106,7 +111,7 @@ class OpenSubtitlesProvider(Provider):
if response['status'] != '200 OK':
raise ProviderError('Logout failed with status %r' % response['status'])
def query(self, languages, hash=None, size=None, imdb_id=None, query=None):
def query(self, languages, hash=None, size=None, imdb_id=None, query=None): # @ReservedAssignment
searches = []
if hash and size:
searches.append({'moviehash': hash, 'moviebytesize': str(size)})
@@ -128,7 +133,6 @@ class OpenSubtitlesProvider(Provider):
if not response['data']:
logger.debug('No subtitle found')
return []
logger.debug('Found subtitles %r', response['data'])
return [OpenSubtitlesSubtitle(babelfish.Language.fromopensubtitles(r['SubLanguageID']),
bool(int(r['SubHearingImpaired'])), r['IDSubtitleFile'], r['MatchedBy'],
r['MovieKind'], r['MovieHash'], r['MovieName'], r['MovieReleaseName'],
@@ -141,7 +145,8 @@ class OpenSubtitlesProvider(Provider):
query = None
if ('opensubtitles' not in video.hashes or not video.size) and not video.imdb_id:
query = video.name.split(os.sep)[-1]
return self.query(languages, hash=video.hashes.get('opensubtitles'), size=video.size, imdb_id=video.imdb_id, query=query)
return self.query(languages, hash=video.hashes.get('opensubtitles'), size=video.size, imdb_id=video.imdb_id,
query=query)
def download_subtitle(self, subtitle):
try:
+208
View File
@@ -0,0 +1,208 @@
# -*- coding: utf-8 -*-
from __future__ import unicode_literals
import io
import logging
import re
import contextlib
import xml.etree.ElementTree
import zipfile
import babelfish
import bs4
import charade
import guessit
import requests
from . import Provider
from ..exceptions import InvalidSubtitle, ProviderNotAvailable, ProviderError
from ..subtitle import Subtitle, is_valid_subtitle, compute_guess_matches
from ..subtitle import sanitize_string
from ..video import Episode, Movie
logger = logging.getLogger(__name__)
URL_RE = re.compile(
'^((http[s]?|ftp):\/)?\/?([^:\/\s]+)(:([^\/]*))?((\/\w+)*\/)' + \
'([\w\-\.]+[^#?\s]+)(\?([^#]*))?(#(.*))?$',
)
class PodnapisiSubtitle(Subtitle):
provider_name = 'podnapisi'
def __init__(self, language, id, releases, hearing_impaired, link, series=None, season=None, episode=None, # @ReservedAssignment
title=None, year=None):
super(PodnapisiSubtitle, self).__init__(language, hearing_impaired)
self.id = id
self.releases = releases
self.hearing_impaired = hearing_impaired
self.link = '/ppodnapisi' + link
self.series = series
self.season = season
self.episode = episode
self.title = title
self.year = year
def compute_matches(self, video):
matches = set()
# episode
if isinstance(video, Episode):
# series
if video.series and \
sanitize_string(self.series) == \
sanitize_string(video.series):
matches.add('series')
# season
if video.season and self.season == video.season:
matches.add('season')
# episode
if video.episode and self.episode == video.episode:
matches.add('episode')
# guess
for release in self.releases:
matches |= compute_guess_matches(video, guessit.guess_episode_info(release + '.mkv'))
# movie
elif isinstance(video, Movie):
# title
if video.title and \
sanitize_string(self.title) == \
sanitize_string(video.title):
matches.add('title')
# year
if video.year and self.year == video.year:
matches.add('year')
# guess
for release in self.releases:
matches |= compute_guess_matches(video, guessit.guess_movie_info(release + '.mkv'))
return matches
class PodnapisiProvider(Provider):
languages = set([babelfish.Language.frompodnapisi(l) for l in babelfish.language_converters['podnapisi'].codes])
video_types = (Episode, Movie)
server = 'http://simple.podnapisi.net'
pre_link_re = re.compile('^.*(?P<link>/ppodnapisi/predownload/i/\d+/k/.*$)')
link_re = re.compile('^.*(?P<link>/[a-zA-Z]{2}/ppodnapisi/download/i/\d+/k/.*$)')
def initialize(self):
self.session = requests.Session()
self.session.headers = {'User-Agent': self.primary_user_agent }
def terminate(self):
self.session.close()
def get(self, url, params=None, headers=None, is_xml=True):
"""Make a GET request on `url` with the given parameters
:param string url: part of the URL to reach with the leading slash
:param dict params: params of the request
:param dict headers: headers of the request
:param bool xml: whether the response content is XML or not
:return: the response
:rtype: :class:`xml.etree.ElementTree.Element` or :class:`bs4.BeautifulSoup`
:raise: :class:`~subliminal.exceptions.ProviderNotAvailable`
"""
prefix_url = ''
url_result = URL_RE.search(url)
if url_result and url_result.group(2) is None:
prefix_url = self.server
try:
r = self.session.get(
prefix_url + url, params=params,
headers=headers,
timeout=10,
)
except requests.Timeout:
raise ProviderNotAvailable('Timeout after 10 seconds')
if r.status_code != 200:
raise ProviderNotAvailable('Request failed with status code %d' % r.status_code)
if is_xml:
return xml.etree.ElementTree.fromstring(r.content)
else:
return bs4.BeautifulSoup(r.content, ['permissive'])
def query(self, language, series=None, season=None, episode=None, title=None, year=None):
params = {'sXML': 1, 'sJ': language.podnapisi}
if series and season and episode:
params['sK'] = series
params['sTS'] = season
params['sTE'] = episode
elif title:
params['sK'] = title
if year:
params['sY'] = year
else:
raise ValueError('Missing parameters series and season and episode or title')
logger.debug('Searching episode %r', params)
subtitles = []
while True:
root = self.get('/ppodnapisi/search', params)
if not int(root.find('pagination/results').text):
logger.debug('No subtitle found')
break
if series and season and episode:
try:
subtitles.extend([PodnapisiSubtitle(language, int(s.find('id').text), s.find('release').text.split(),
'h' in (s.find('flags').text or ''), s.find('url').text[38:],
series=series, season=season, episode=episode)
for s in root.findall('subtitle')])
except AttributeError:
# there simply wasn't enough information in the TV Show
# gracefully handle this instead of crashing :)
break
elif title:
try:
subtitles.extend([PodnapisiSubtitle(language, int(s.find('id').text), s.find('release').text.split(),
'h' in (s.find('flags').text or ''), s.find('url').text[38:],
title=title, year=year)
for s in root.findall('subtitle')])
except AttributeError:
# there simply wasn't enough information in the movie
# gracefully handle this instead of crashing :)
break
if int(root.find('pagination/current').text) >= int(root.find('pagination/count').text):
break
params['page'] = int(root.find('pagination/current').text) + 1
return subtitles
def list_subtitles(self, video, languages):
if isinstance(video, Episode):
return [s for l in languages for s in self.query(l, series=video.series, season=video.season,
episode=video.episode)]
elif isinstance(video, Movie):
return [s for l in languages for s in self.query(l, title=video.title, year=video.year)]
def download_subtitle(self, subtitle):
soup = self.get(subtitle.link, is_xml=False)
pre_link = soup.find('a', href=self.pre_link_re)
if not pre_link:
raise ProviderError('Cannot find the pre-download link')
pre_link = self.server + \
self.pre_link_re.match(pre_link['href']).group('link')
# Continue following the link
soup = self.get(
pre_link,
headers={
'Referer': self.server,
},
is_xml=False,
)
link = soup.find('a', href=self.link_re)
if not link:
raise ProviderError('Cannot find the download link')
try:
r = self.session.get(self.server + self.link_re.match(link['href']).group('link'), timeout=10)
except requests.Timeout:
raise ProviderNotAvailable('Timeout after 10 seconds')
if r.status_code != 200:
raise ProviderNotAvailable('Request failed with status code %d' % r.status_code)
with contextlib.closing(zipfile.ZipFile(io.BytesIO(r.content))) as zf:
if len(zf.namelist()) > 1:
raise ProviderError('More than one file to unzip')
subtitle_bytes = zf.read(zf.namelist()[0])
subtitle_text = subtitle_bytes.decode(charade.detect(subtitle_bytes)['encoding'], 'replace')
if not is_valid_subtitle(subtitle_text):
raise InvalidSubtitle
return subtitle_text
+4 -4
View File
@@ -16,7 +16,7 @@ logger = logging.getLogger(__name__)
class TheSubDBSubtitle(Subtitle):
provider_name = 'thesubdb'
def __init__(self, language, hash):
def __init__(self, language, hash): # @ReservedAssignment
super(TheSubDBSubtitle, self).__init__(language)
self.hash = hash
@@ -29,7 +29,7 @@ class TheSubDBSubtitle(Subtitle):
class TheSubDBProvider(Provider):
languages = {babelfish.Language.fromalpha2(l) for l in ['en', 'es', 'fr', 'it', 'nl', 'pl', 'pt', 'ro', 'sv', 'tr']}
languages = set([babelfish.Language.fromalpha2(l) for l in ['en', 'es', 'fr', 'it', 'nl', 'pl', 'pt', 'ro', 'sv', 'tr']])
required_hash = 'thesubdb'
def initialize(self):
@@ -55,7 +55,7 @@ class TheSubDBProvider(Provider):
raise ProviderNotAvailable('Timeout after 10 seconds')
return r
def query(self, hash):
def query(self, hash): # @ReservedAssignment
params = {'action': 'search', 'hash': hash}
logger.debug('Searching subtitles %r', params)
r = self.get(params)
@@ -65,7 +65,7 @@ class TheSubDBProvider(Provider):
elif r.status_code != 200:
raise ProviderError('Request failed with status code %d' % r.status_code)
return [TheSubDBSubtitle(language, hash) for language in
{babelfish.Language.fromalpha2(l) for l in r.content.split(',')}]
set([babelfish.Language.fromalpha2(l) for l in r.content.split(',')])]
def list_subtitles(self, video, languages):
return [s for s in self.query(video.hashes['thesubdb']) if s.language in languages]
+39 -10
View File
@@ -3,18 +3,19 @@ from __future__ import unicode_literals
import io
import logging
import re
import contextlib
import zipfile
import babelfish
import bs4
import charade
import requests
from . import Provider
from .. import __version__
from ..cache import region
from ..exceptions import InvalidSubtitle, ProviderNotAvailable, ProviderError
from ..subtitle import Subtitle, is_valid_subtitle
from ..subtitle import Subtitle, is_valid_subtitle, sanitize_string
from ..video import Episode
IGNORE_DATEMATCH=re.compile('^(.*)[ \t0-9-._)(]*$')
logger = logging.getLogger(__name__)
@@ -22,7 +23,7 @@ logger = logging.getLogger(__name__)
class TVsubtitlesSubtitle(Subtitle):
provider_name = 'tvsubtitles'
def __init__(self, language, series, season, episode, id, rip, release):
def __init__(self, language, series, season, episode, id, rip, release): # @ReservedAssignment
super(TVsubtitlesSubtitle, self).__init__(language)
self.series = series
self.season = season
@@ -56,17 +57,18 @@ class TVsubtitlesSubtitle(Subtitle):
class TVsubtitlesProvider(Provider):
languages = {babelfish.Language('por', 'BR')} | {babelfish.Language(l)
languages = set([babelfish.Language('por', 'BR')]) | set([babelfish.Language(l)
for l in ['ara', 'bul', 'ces', 'dan', 'deu', 'ell', 'eng', 'fin', 'fra', 'hun', 'ita', 'jpn', 'kor',
'nld', 'pol', 'por', 'ron', 'rus', 'spa', 'swe', 'tur', 'ukr', 'zho']}
'nld', 'pol', 'por', 'ron', 'rus', 'spa', 'swe', 'tur', 'ukr', 'zho']])
video_types = (Episode,)
server = 'http://www.tvsubtitles.net'
episode_id_re = re.compile('^episode-(\d+)\.html$')
subtitle_re = re.compile('^\/subtitle-(\d+)\.html$')
episode_id_re = re.compile('^episode-\d+\.html$')
subtitle_re = re.compile('^\/subtitle-\d+\.html$')
link_re = re.compile('^(?P<series>.+) \(\d{4}-\d{4}\)$')
def initialize(self):
self.session = requests.Session()
self.session.headers = {'User-Agent': 'Subliminal/%s' % __version__}
self.session.headers = {'User-Agent': self.primary_user_agent }
def terminate(self):
self.session.close()
@@ -77,6 +79,7 @@ class TVsubtitlesProvider(Provider):
:param string url: part of the URL to reach with the leading slash
:param dict params: params of the request
:param dict data: data of the request
:param string method: method of the request
:return: the response
:rtype: :class:`bs4.BeautifulSoup`
:raise: :class:`~subliminal.exceptions.ProviderNotAvailable`
@@ -103,9 +106,35 @@ class TVsubtitlesProvider(Provider):
logger.debug('Searching series %r', data)
soup = self.request('/search.php', data=data, method='POST')
links = soup.select('div.left li div a[href^="/tvshow-"]')
sanitized_series = IGNORE_DATEMATCH.match(
sanitize_string(series).replace('.', ' ').strip(),
)
if not sanitized_series:
sanitized_series = sanitize_string(series)\
.replace('.', ' ').strip()
else:
sanitized_series = sanitized_series.group(1)
if not links:
logger.info('Series %r not found', series)
return None
for link in links:
match = self.link_re.match(link.string)
if not match:
logger.warning('Could not parse %r', link.string)
continue
show = IGNORE_DATEMATCH.match(
sanitize_string(match.group('series'))\
.replace('.', ' ').strip(),
)
if not show:
logger.warning('Could not postparse %r', match.group('series'))
continue
show = show.group(1)
if show == sanitized_series:
return int(link['href'][8:-5])
return int(links[0]['href'][8:-5])
@region.cache_on_arguments()
@@ -130,7 +159,7 @@ class TVsubtitlesProvider(Provider):
return episode_ids
def query(self, series, season, episode):
show_id = self.find_show_id(series.lower())
show_id = self.find_show_id(series)
if show_id is None:
return []
episode_ids = self.find_episode_ids(show_id, season)
@@ -156,7 +185,7 @@ class TVsubtitlesProvider(Provider):
raise ProviderNotAvailable('Timeout after 10 seconds')
if r.status_code != 200:
raise ProviderNotAvailable('Request failed with status code %d' % r.status_code)
with zipfile.ZipFile(io.BytesIO(r.content)) as zf:
with contextlib.closing(zipfile.ZipFile(io.BytesIO(r.content))) as zf:
if len(zf.namelist()) > 1:
raise ProviderError('More than one file to unzip')
subtitle_bytes = zf.read(zf.namelist()[0])
+1 -1
View File
@@ -6,7 +6,7 @@ from sympy import Eq, symbols, solve
# Symbols
release_group, resolution, video_codec, audio_codec = symbols('release_group resolution video_codec audio_codec')
imdb_id, hash, title, series, tvdb_id, season, episode = symbols('imdb_id hash title series tvdb_id season episode')
imdb_id, hash, title, series, tvdb_id, season, episode = symbols('imdb_id hash title series tvdb_id season episode') # @ReservedAssignment
year = symbols('year')
+44 -10
View File
@@ -4,11 +4,15 @@ import logging
import os.path
import babelfish
import pysrt
import re
from .video import Episode, Movie
logger = logging.getLogger(__name__)
#: The following characters are always stripped
IGNORED_CHARACTERS_RE = re.compile('[!@#$\'"]')
class Subtitle(object):
"""Base class for subtitle
@@ -33,7 +37,7 @@ class Subtitle(object):
"""
raise NotImplementedError
def compute_score(self, video):
def compute_score(self, video, hi_score_adjust=0):
"""Compute the score of the subtitle against the `video`
There are equivalent matches so that a provider can match one element or its equivalent. This is
@@ -47,6 +51,7 @@ class Subtitle(object):
:param video: the video to compute the score against
:type video: :class:`~subliminal.video.Video`
:param hi_score_adjust: adjust hearing impaired matched videos by this value
:return: score of the subtitle
:rtype: int
@@ -62,20 +67,45 @@ class Subtitle(object):
# remove equivalences
if isinstance(video, Episode):
if 'imdb_id' in matches:
matches -= {'series', 'tvdb_id', 'season', 'episode', 'title'}
matches -= set(['series', 'tvdb_id', 'season', 'episode', 'title'])
if 'tvdb_id' in matches:
matches -= {'series'}
matches -= set(['series',])
if 'title' in matches:
matches -= {'season', 'episode'}
matches -= set(['season', 'episode'])
# add other scores
score += sum((video.scores[match] for match in matches))
logger.info('Computed score %d with matches %r', score, initial_matches)
score += sum([video.scores[match] for match in matches])
# Adjust scoring if hearing impaired subtitles are detected
if self.hearing_impaired and hi_score_adjust != 0:
logger.debug('Hearing impaired subtitle score adjusted ' + \
'by %d' % hi_score_adjust)
# Priortization (adjust score)
score += hi_score_adjust
logger.debug('Computed score %d with matches %r', score, initial_matches)
return score
def __repr__(self):
return '<%s [%r]>' % (self.__class__.__name__, self.language)
return '<%s [%s]>' % (self.__class__.__name__, self.language)
def sanitize_string(str_in):
"""
Sanitizes a string passed into it by eliminating characters that might
otherwise cause issues when attempting to locate a match on websites by
striping out any special characters and forcing a consistent string that
can be used for caching too.
:param string str_in: the string to sanitize
:return: sanitized string
:rtype: string
"""
if not isinstance(str_in, basestring):
# handle int, float, etc
str_in = str(str_in)
return IGNORED_CHARACTERS_RE.sub('', str_in).lower().strip()
def get_subtitle_path(video_path, language=None):
"""Create the subtitle path from the given `video_path` and `language`
@@ -90,7 +120,7 @@ def get_subtitle_path(video_path, language=None):
if language is not None:
try:
return subtitle_path + '.%s.%s' % (language.alpha2, 'srt')
except babelfish.ConvertError:
except babelfish.LanguageConvertError:
return subtitle_path + '.%s.%s' % (language.alpha3, 'srt')
return subtitle_path + '.srt'
@@ -105,8 +135,12 @@ def is_valid_subtitle(subtitle_text):
try:
pysrt.from_string(subtitle_text, error_handling=pysrt.ERROR_RAISE)
return True
except pysrt.Error:
return False
except pysrt.Error as e:
if e.args[0] > 80:
return True
except:
logger.exception('Unexpected error when validating subtitle')
return False
def compute_guess_matches(video, guess):
+1 -1
View File
@@ -6,7 +6,7 @@ from subliminal import cache_region
from . import test_providers, test_subliminal
cache_region.configure('dogpile.cache.memory', expiration_time=60 * 30)
cache_region.configure('dogpile.cache.memory', expiration_time=60 * 30) # @UndefinedVariable
suite = TestSuite([test_providers.suite(), test_subliminal.suite()])
+159 -165
View File
@@ -25,135 +25,66 @@ class Addic7edProviderTestCase(ProviderTestCase):
def test_find_show_id(self):
with self.Provider() as provider:
show_id = provider.find_show_id('The Big Bang')
self.assertTrue(show_id == 126)
self.assertEqual(show_id, 126)
def test_find_show_id_error(self):
with self.Provider() as provider:
show_id = provider.find_show_id('the big how i met your mother')
self.assertTrue(show_id is None)
self.assertEqual(show_id, None)
def test_get_show_ids(self):
with self.Provider() as provider:
show_ids = provider.get_show_ids()
self.assertTrue('the big bang theory' in show_ids and show_ids['the big bang theory'] == 126)
self.assertTrue('the big bang theory' in show_ids)
self.assertEqual(show_ids['the big bang theory'], 126)
def test_query_episode_0(self):
video = EPISODES[0]
languages = {Language('tur'), Language('rus'), Language('heb'), Language('ita'), Language('fra'),
languages = set([Language('rus'), Language('heb'), Language('ita'), Language('fra'),
Language('ron'), Language('nld'), Language('eng'), Language('deu'), Language('ell'),
Language('por', 'BR'), Language('bul')}
matches = {frozenset(['episode', 'release_group', 'title', 'series', 'resolution', 'season']),
Language('por', 'BR'), Language('bul')])
matches = set([frozenset(['episode', 'release_group', 'title', 'series', 'resolution', 'season']),
frozenset(['series', 'resolution', 'season']),
frozenset(['series', 'episode', 'season', 'title']),
frozenset(['series', 'release_group', 'season']),
frozenset(['series', 'episode', 'season', 'release_group', 'title']),
frozenset(['series', 'season'])}
frozenset(['series', 'season'])])
with self.Provider() as provider:
subtitles = provider.query(video.series, video.season)
self.assertTrue({frozenset(subtitle.compute_matches(video)) for subtitle in subtitles} == matches)
self.assertTrue({subtitle.language for subtitle in subtitles} == languages)
self.assertTrue(matches - set([frozenset(subtitle.compute_matches(video)) for subtitle in subtitles]) == set([]))
self.assertTrue(languages - set([subtitle.language for subtitle in subtitles]) == set([]))
def test_query_episode_1(self):
video = EPISODES[1]
languages = {Language('ind'), Language('spa'), Language('hrv'), Language('ita'), Language('fra'),
languages = set([Language('ind'), Language('spa'), Language('hrv'), Language('ita'), Language('fra'),
Language('cat'), Language('ell'), Language('nld'), Language('eng'), Language('fas'),
Language('por'), Language('nor'), Language('deu'), Language('ron'), Language('por', 'BR'),
Language('bul')}
matches = {frozenset(['series', 'episode', 'resolution', 'season', 'title']),
Language('bul')])
matches = set([frozenset(['series', 'episode', 'resolution', 'season', 'title']),
frozenset(['series', 'resolution', 'season']),
frozenset(['series', 'episode', 'season', 'title']),
frozenset(['series', 'release_group', 'season']),
frozenset(['series', 'resolution', 'release_group', 'season']),
frozenset(['series', 'episode', 'season', 'release_group', 'title']),
frozenset(['series', 'season'])}
frozenset(['series', 'season'])])
with self.Provider() as provider:
subtitles = provider.query(video.series, video.season)
self.assertTrue({frozenset(subtitle.compute_matches(video)) for subtitle in subtitles} == matches)
self.assertTrue({subtitle.language for subtitle in subtitles} == languages)
self.assertTrue(matches - set([frozenset(subtitle.compute_matches(video)) for subtitle in subtitles]) == set([]))
self.assertTrue(languages - set([subtitle.language for subtitle in subtitles]) == set([]))
def test_list_subtitles(self):
video = EPISODES[0]
languages = {Language('eng'), Language('fra')}
matches = {frozenset(['series', 'episode', 'season', 'release_group', 'title']),
frozenset(['series', 'episode', 'season', 'title'])}
languages = set([Language('eng'), Language('fra')])
matches = set([frozenset(['series', 'episode', 'season', 'release_group', 'title']),
frozenset(['series', 'episode', 'season', 'title'])])
with self.Provider() as provider:
subtitles = provider.list_subtitles(video, languages)
self.assertTrue({frozenset(subtitle.compute_matches(video)) for subtitle in subtitles} == matches)
self.assertTrue({subtitle.language for subtitle in subtitles} == languages)
self.assertTrue(matches - set([frozenset(subtitle.compute_matches(video)) for subtitle in subtitles]) == set([]))
self.assertTrue(languages - set([subtitle.language for subtitle in subtitles]) == set([]))
def test_download_subtitle(self):
video = EPISODES[0]
languages = {Language('eng'), Language('fra')}
with self.Provider() as provider:
subtitles = provider.list_subtitles(video, languages)
subtitle_text = provider.download_subtitle(subtitles[0])
self.assertTrue(is_valid_subtitle(subtitle_text))
class BierDopjeProviderTestCase(ProviderTestCase):
provider_name = 'bierdopje'
def test_find_show_id(self):
with self.Provider() as provider:
show_id = provider.find_show_id('The Big Bang')
self.assertTrue(show_id == 9203)
def test_find_show_id_error(self):
with self.Provider() as provider:
show_id = provider.find_show_id('the big how i met your mother')
self.assertTrue(show_id is None)
def test_query_episode_0(self):
video = EPISODES[0]
language = Language('eng')
matches = {frozenset(['series', 'video_codec', 'resolution', 'episode', 'season']),
frozenset(['season', 'video_codec', 'episode', 'series']),
frozenset(['episode', 'video_codec', 'season', 'series', 'resolution', 'release_group'])}
with self.Provider() as provider:
subtitles = provider.query(language, video.season, video.episode, series=video.series)
self.assertTrue({frozenset(subtitle.compute_matches(video)) for subtitle in subtitles} == matches)
self.assertTrue({subtitle.language for subtitle in subtitles} == {language})
def test_query_episode_1(self):
video = EPISODES[1]
language = Language('nld')
matches = {frozenset(['series', 'video_codec', 'resolution', 'episode', 'season']),
frozenset(['season', 'video_codec', 'episode', 'series']),
frozenset(['series', 'episode', 'season']),
frozenset(['season', 'video_codec', 'episode', 'release_group', 'series']),
frozenset(['episode', 'video_codec', 'season', 'series', 'resolution', 'release_group'])}
with self.Provider() as provider:
subtitles = provider.query(language, video.season, video.episode, series=video.series)
self.assertTrue({frozenset(subtitle.compute_matches(video)) for subtitle in subtitles} == matches)
self.assertTrue({subtitle.language for subtitle in subtitles} == {language})
def test_query_episode_0_tvdb_id(self):
video = EPISODES[0]
language = Language('eng')
matches = {frozenset(['video_codec', 'tvdb_id', 'episode', 'season', 'series']),
frozenset(['episode', 'video_codec', 'series', 'season', 'tvdb_id', 'resolution', 'release_group']),
frozenset(['episode', 'series', 'video_codec', 'tvdb_id', 'resolution', 'season'])}
with self.Provider() as provider:
subtitles = provider.query(language, video.season, video.episode, tvdb_id=video.tvdb_id)
self.assertTrue({frozenset(subtitle.compute_matches(video)) for subtitle in subtitles} == matches)
self.assertTrue({subtitle.language for subtitle in subtitles} == {language})
def test_list_subtitles(self):
video = EPISODES[1]
languages = {Language('eng'), Language('nld')}
matches = {frozenset(['series', 'video_codec', 'tvdb_id', 'episode', 'season']),
frozenset(['episode', 'video_codec', 'season', 'series', 'tvdb_id', 'resolution', 'release_group']),
frozenset(['season', 'tvdb_id', 'episode', 'series']),
frozenset(['episode', 'video_codec', 'season', 'series', 'tvdb_id', 'resolution']),
frozenset(['episode', 'video_codec', 'season', 'series', 'tvdb_id', 'release_group'])}
with self.Provider() as provider:
subtitles = provider.list_subtitles(video, languages)
self.assertTrue({frozenset(subtitle.compute_matches(video)) for subtitle in subtitles} == matches)
self.assertTrue({subtitle.language for subtitle in subtitles} == languages)
def test_download_subtitle(self):
video = EPISODES[0]
languages = {Language('eng'), Language('nld')}
languages = set([Language('eng'), Language('fra')])
with self.Provider() as provider:
subtitles = provider.list_subtitles(video, languages)
subtitle_text = provider.download_subtitle(subtitles[0])
@@ -165,113 +96,165 @@ class OpenSubtitlesProviderTestCase(ProviderTestCase):
def test_query_movie_0_query(self):
video = MOVIES[0]
languages = {Language('eng')}
matches = {frozenset([]), frozenset(['imdb_id', 'resolution', 'title', 'year']),
languages = set([Language('eng'), ])
matches = set([frozenset([]), frozenset(['imdb_id', 'resolution', 'title', 'year']),
frozenset(['imdb_id', 'title', 'year']),
frozenset(['imdb_id', 'video_codec', 'title', 'year']),
frozenset(['imdb_id', 'resolution', 'title', 'video_codec', 'year']),
frozenset(['imdb_id', 'title', 'year', 'video_codec', 'resolution', 'release_group'])}
frozenset(['imdb_id', 'title', 'year', 'video_codec', 'resolution', 'release_group'])])
with self.Provider() as provider:
subtitles = provider.query(languages, query=video.title)
self.assertTrue({frozenset(subtitle.compute_matches(video)) for subtitle in subtitles} == matches)
self.assertTrue({subtitle.language for subtitle in subtitles} == languages)
self.assertTrue(matches - set([frozenset(subtitle.compute_matches(video)) for subtitle in subtitles]) == set([]))
self.assertTrue(languages - set([subtitle.language for subtitle in subtitles]) == set([]))
def test_query_episode_0_query(self):
video = EPISODES[0]
languages = {Language('eng')}
matches = {frozenset(['series', 'episode', 'season', 'imdb_id']),
languages = set([Language('eng'), ])
matches = set([frozenset(['series', 'episode', 'season', 'imdb_id']),
frozenset(['series', 'imdb_id', 'video_codec', 'episode', 'season']),
frozenset(['episode', 'title', 'series', 'imdb_id', 'video_codec', 'season'])}
frozenset(['episode', 'title', 'series', 'imdb_id', 'video_codec', 'season'])])
with self.Provider() as provider:
subtitles = provider.query(languages, query=video.name.split(os.sep)[-1])
self.assertTrue({frozenset(subtitle.compute_matches(video)) for subtitle in subtitles} == matches)
self.assertTrue({subtitle.language for subtitle in subtitles} == languages)
self.assertTrue(matches - set([frozenset(subtitle.compute_matches(video)) for subtitle in subtitles]) == set([]))
self.assertTrue(languages - set([subtitle.language for subtitle in subtitles]) == set([]))
def test_query_episode_1_query(self):
video = EPISODES[1]
languages = {Language('eng'), Language('fra')}
matches = {frozenset(['episode', 'title', 'series', 'imdb_id', 'video_codec', 'season']),
languages = set([Language('eng'), Language('fra')])
matches = set([frozenset(['episode', 'title', 'series', 'imdb_id', 'video_codec', 'season']),
frozenset(['series', 'imdb_id', 'title', 'episode', 'season']),
frozenset(['series', 'imdb_id', 'video_codec', 'episode', 'season']),
frozenset(['episode', 'video_codec', 'series', 'imdb_id', 'resolution', 'season']),
frozenset(['series', 'imdb_id', 'resolution', 'episode', 'season']),
frozenset(['series', 'episode', 'season', 'imdb_id'])}
frozenset(['series', 'episode', 'season', 'imdb_id'])])
with self.Provider() as provider:
subtitles = provider.query(languages, query=video.name.split(os.sep)[-1])
self.assertTrue({frozenset(subtitle.compute_matches(video)) for subtitle in subtitles} == matches)
self.assertTrue({subtitle.language for subtitle in subtitles} == languages)
self.assertTrue(matches - set([frozenset(subtitle.compute_matches(video)) for subtitle in subtitles]) == set([]))
self.assertTrue(languages - set([subtitle.language for subtitle in subtitles]) == set([]))
def test_query_movie_0_imdb_id(self):
video = MOVIES[0]
languages = {Language('eng'), Language('fra')}
matches = {frozenset(['imdb_id', 'video_codec', 'title', 'year']),
languages = set([Language('eng'), Language('fra')])
matches = set([frozenset(['imdb_id', 'video_codec', 'title', 'year']),
frozenset(['imdb_id', 'resolution', 'title', 'video_codec', 'year']),
frozenset(['imdb_id', 'title', 'year', 'video_codec', 'resolution', 'release_group']),
frozenset(['imdb_id', 'title', 'year']),
frozenset(['imdb_id', 'resolution', 'title', 'year'])}
frozenset(['imdb_id', 'resolution', 'title', 'year'])])
with self.Provider() as provider:
subtitles = provider.query(languages, imdb_id=video.imdb_id)
self.assertTrue({frozenset(subtitle.compute_matches(video)) for subtitle in subtitles} == matches)
self.assertTrue({subtitle.language for subtitle in subtitles} == languages)
self.assertTrue(matches - set([frozenset(subtitle.compute_matches(video)) for subtitle in subtitles]) == set([]))
self.assertTrue(languages - set([subtitle.language for subtitle in subtitles]) == set([]))
def test_query_episode_0_imdb_id(self):
video = EPISODES[0]
languages = {Language('eng'), Language('fra')}
matches = {frozenset(['series', 'episode', 'season', 'imdb_id']),
languages = set([Language('eng'), Language('fra')])
matches = set([frozenset(['series', 'episode', 'season', 'imdb_id']),
frozenset(['episode', 'release_group', 'video_codec', 'series', 'imdb_id', 'resolution', 'season']),
frozenset(['series', 'imdb_id', 'video_codec', 'episode', 'season']),
frozenset(['episode', 'title', 'series', 'imdb_id', 'video_codec', 'season'])}
frozenset(['episode', 'title', 'series', 'imdb_id', 'video_codec', 'season'])])
with self.Provider() as provider:
subtitles = provider.query(languages, imdb_id=video.imdb_id)
self.assertTrue({frozenset(subtitle.compute_matches(video)) for subtitle in subtitles} == matches)
self.assertTrue({subtitle.language for subtitle in subtitles} == languages)
self.assertTrue(matches - set([frozenset(subtitle.compute_matches(video)) for subtitle in subtitles]) == set([]))
self.assertTrue(languages - set([subtitle.language for subtitle in subtitles]) == set([]))
def test_query_movie_0_hash(self):
video = MOVIES[0]
languages = {Language('eng')}
matches = {frozenset(['hash', 'title', 'video_codec', 'year', 'resolution', 'imdb_id']),
languages = set([Language('eng'), ])
matches = set([frozenset(['hash', 'title', 'video_codec', 'year', 'resolution', 'imdb_id']),
frozenset(['hash', 'title', 'video_codec', 'year', 'resolution', 'release_group', 'imdb_id']),
frozenset(['year', 'video_codec', 'imdb_id', 'hash', 'title']),
frozenset([]),
frozenset(['year', 'resolution', 'imdb_id', 'hash', 'title']),
frozenset(['year', 'imdb_id', 'hash', 'title'])}
frozenset(['year', 'imdb_id', 'hash', 'title'])])
with self.Provider() as provider:
subtitles = provider.query(languages, hash=video.hashes['opensubtitles'], size=video.size)
self.assertTrue({frozenset(subtitle.compute_matches(video)) for subtitle in subtitles} == matches)
self.assertTrue({subtitle.language for subtitle in subtitles} == languages)
self.assertTrue(matches - set([frozenset(subtitle.compute_matches(video)) for subtitle in subtitles]) == set([]))
self.assertTrue(languages - set([subtitle.language for subtitle in subtitles]) == set([]))
def test_query_episode_0_hash(self):
video = EPISODES[0]
languages = {Language('eng')}
matches = {frozenset(['series', 'hash']),
languages = set([Language('eng'), ])
matches = set([frozenset(['series', 'hash']),
frozenset(['episode', 'season', 'series', 'imdb_id', 'video_codec', 'hash']),
frozenset(['series', 'episode', 'season', 'hash', 'imdb_id']),
frozenset(['series', 'resolution', 'hash', 'video_codec'])}
frozenset(['series', 'resolution', 'hash', 'video_codec'])])
with self.Provider() as provider:
subtitles = provider.query(languages, hash=video.hashes['opensubtitles'], size=video.size)
self.assertTrue({frozenset(subtitle.compute_matches(video)) for subtitle in subtitles} == matches)
self.assertTrue({subtitle.language for subtitle in subtitles} == languages)
self.assertTrue(matches - set([frozenset(subtitle.compute_matches(video)) for subtitle in subtitles]) == set([]))
self.assertTrue(languages - set([subtitle.language for subtitle in subtitles]) == set([]))
def test_list_subtitles(self):
video = MOVIES[0]
languages = {Language('eng'), Language('fra')}
matches = {frozenset(['title', 'video_codec', 'year', 'resolution', 'release_group', 'imdb_id']),
languages = set([Language('eng'), Language('fra')])
matches = set([frozenset(['title', 'video_codec', 'year', 'resolution', 'release_group', 'imdb_id']),
frozenset(['imdb_id', 'year', 'title']),
frozenset(['year', 'video_codec', 'imdb_id', 'resolution', 'title']),
frozenset(['hash', 'title', 'video_codec', 'year', 'resolution', 'release_group', 'imdb_id']),
frozenset(['year', 'video_codec', 'imdb_id', 'hash', 'title']),
frozenset([]),
frozenset(['year', 'resolution', 'imdb_id', 'hash', 'title']),
frozenset(['hash', 'title', 'video_codec', 'year', 'resolution', 'imdb_id']),
frozenset(['year', 'imdb_id', 'hash', 'title']),
frozenset(['video_codec', 'imdb_id', 'year', 'title']),
frozenset(['year', 'imdb_id', 'resolution', 'title'])}
frozenset(['year', 'imdb_id', 'resolution', 'title'])])
with self.Provider() as provider:
subtitles = provider.list_subtitles(video, languages)
self.assertTrue({frozenset(subtitle.compute_matches(video)) for subtitle in subtitles} == matches)
self.assertTrue({subtitle.language for subtitle in subtitles} == languages)
self.assertTrue(matches - set([frozenset(subtitle.compute_matches(video)) for subtitle in subtitles]) == set([]))
self.assertTrue(languages - set([subtitle.language for subtitle in subtitles]) == set([]))
def test_download_subtitle(self):
video = MOVIES[0]
languages = {Language('eng'), Language('fra')}
languages = set([Language('eng'), Language('fra')])
with self.Provider() as provider:
subtitles = provider.list_subtitles(video, languages)
subtitle_text = provider.download_subtitle(subtitles[0])
self.assertTrue(is_valid_subtitle(subtitle_text))
class PodnapisiProviderTestCase(ProviderTestCase):
provider_name = 'podnapisi'
def test_query_movie_0(self):
video = MOVIES[0]
language = Language('eng')
matches = set([frozenset(['video_codec', 'title', 'resolution', 'year']),
frozenset(['title', 'resolution', 'year']),
frozenset(['video_codec', 'title', 'year']),
frozenset(['title', 'year']),
frozenset(['video_codec', 'title', 'resolution', 'release_group', 'year']),
frozenset(['video_codec', 'title', 'resolution', 'audio_codec', 'year'])])
with self.Provider() as provider:
subtitles = provider.query(language, title=video.title, year=video.year)
self.assertTrue(matches - set([frozenset(subtitle.compute_matches(video)) for subtitle in subtitles]) == set([]))
self.assertTrue([Language, ], [subtitle.language for subtitle in subtitles])
def test_query_episode_0(self):
video = EPISODES[0]
language = Language('eng')
matches = set([frozenset(['episode', 'series', 'season', 'video_codec', 'resolution', 'release_group']),
frozenset(['season', 'video_codec', 'episode', 'resolution', 'series'])])
with self.Provider() as provider:
subtitles = provider.query(language, series=video.series, season=video.season, episode=video.episode)
self.assertTrue(matches - set([frozenset(subtitle.compute_matches(video)) for subtitle in subtitles]) == set([]))
self.assertTrue([Language, ], [subtitle.language for subtitle in subtitles])
def test_list_subtitles(self):
video = MOVIES[0]
languages = set([Language('eng'), Language('fra')])
matches = set([frozenset(['video_codec', 'title', 'resolution', 'year']),
frozenset(['title', 'resolution', 'year']),
frozenset(['video_codec', 'title', 'year']),
frozenset(['title', 'year']),
frozenset(['video_codec', 'title', 'resolution', 'release_group', 'year']),
frozenset(['video_codec', 'title', 'resolution', 'audio_codec', 'year'])])
with self.Provider() as provider:
subtitles = provider.list_subtitles(video, languages)
self.assertTrue(matches - set([frozenset(subtitle.compute_matches(video)) for subtitle in subtitles]) == set([]))
self.assertTrue(languages - set([subtitle.language for subtitle in subtitles]) == set([]))
def test_download_subtitle(self):
video = MOVIES[0]
languages = set([Language('eng'), Language('fra')])
with self.Provider() as provider:
subtitles = provider.list_subtitles(video, languages)
subtitle_text = provider.download_subtitle(subtitles[0])
@@ -283,34 +266,34 @@ class TheSubDBProviderTestCase(ProviderTestCase):
def test_query_episode_0(self):
video = EPISODES[0]
languages = {Language('eng'), Language('spa'), Language('por')}
matches = {frozenset(['hash'])}
languages = set([Language('eng'), Language('spa'), Language('por')])
matches = set([frozenset(['hash']), ])
with self.Provider() as provider:
subtitles = provider.query(video.hashes['thesubdb'])
self.assertTrue({frozenset(subtitle.compute_matches(video)) for subtitle in subtitles} == matches)
self.assertTrue({subtitle.language for subtitle in subtitles} == languages)
self.assertTrue(matches - set([frozenset(subtitle.compute_matches(video)) for subtitle in subtitles]) == set([]))
self.assertTrue(languages - set([subtitle.language for subtitle in subtitles]) == set([]))
def test_query_episode_1(self):
video = EPISODES[1]
languages = {Language('eng'), Language('por')}
matches = {frozenset(['hash'])}
languages = set([Language('eng'), Language('por')])
matches = set([frozenset(['hash']), ])
with self.Provider() as provider:
subtitles = provider.query(video.hashes['thesubdb'])
self.assertTrue({frozenset(subtitle.compute_matches(video)) for subtitle in subtitles} == matches)
self.assertTrue({subtitle.language for subtitle in subtitles} == languages)
self.assertTrue(matches - set([frozenset(subtitle.compute_matches(video)) for subtitle in subtitles]) == set([]))
self.assertTrue(languages - set([subtitle.language for subtitle in subtitles]) == set([]))
def test_list_subtitles(self):
video = MOVIES[0]
languages = {Language('eng'), Language('por')}
matches = {frozenset(['hash'])}
languages = set([Language('eng'), Language('por')])
matches = set([frozenset(['hash']), ])
with self.Provider() as provider:
subtitles = provider.list_subtitles(video, languages)
self.assertTrue({frozenset(subtitle.compute_matches(video)) for subtitle in subtitles} == matches)
self.assertTrue({subtitle.language for subtitle in subtitles} == languages)
self.assertTrue(matches - set([frozenset(subtitle.compute_matches(video)) for subtitle in subtitles]) == set([]))
self.assertTrue(languages - set([subtitle.language for subtitle in subtitles]) == set([]))
def test_download_subtitle(self):
video = MOVIES[0]
languages = {Language('eng'), Language('por')}
languages = (Language('eng'), Language('por'), )
with self.Provider() as provider:
subtitles = provider.list_subtitles(video, languages)
subtitle_text = provider.download_subtitle(subtitles[0])
@@ -323,52 +306,63 @@ class TVsubtitlesProviderTestCase(ProviderTestCase):
def test_find_show_id(self):
with self.Provider() as provider:
show_id = provider.find_show_id('The Big Bang')
self.assertTrue(show_id == 154)
self.assertEqual(show_id, 154)
def test_find_show_id_ambiguous(self):
with self.Provider() as provider:
show_id = provider.find_show_id('New Girl')
self.assertEqual(show_id, 977)
def test_find_show_id_no_dots(self):
with self.Provider() as provider:
show_id = provider.find_show_id('Marvel\'s Agents of S H I E L D')
self.assertEqual(show_id, 1340)
def test_find_show_id_error(self):
with self.Provider() as provider:
show_id = provider.find_show_id('the big gaming')
self.assertTrue(show_id is None)
self.assertEqual(show_id, None)
def test_find_episode_ids(self):
with self.Provider() as provider:
episode_ids = provider.find_episode_ids(154, 5)
self.assertTrue(set(episode_ids.keys()) == set(range(1, 25)))
self.assertEqual(set(episode_ids.keys()), set(range(1, 25)))
def test_query_episode_0(self):
video = EPISODES[0]
languages = {Language('fra'), Language('por'), Language('hun'), Language('ron'), Language('eng')}
matches = {frozenset(['series', 'episode', 'season', 'video_codec']),
frozenset(['series', 'episode', 'season'])}
languages = set([Language('fra'), Language('por'), Language('hun'), Language('ron'), Language('eng')])
matches = set([frozenset(['series', 'episode', 'resolution', 'season']),
frozenset(['series', 'episode', 'season'])])
with self.Provider() as provider:
subtitles = provider.query(video.series, video.season, video.episode)
self.assertTrue({frozenset(subtitle.compute_matches(video)) for subtitle in subtitles} == matches)
self.assertTrue({subtitle.language for subtitle in subtitles} == languages)
self.assertTrue(matches - set([frozenset(subtitle.compute_matches(video)) for subtitle in subtitles]) == set([]))
self.assertTrue(languages - set([subtitle.language for subtitle in subtitles]) == set([]))
def test_query_episode_1(self):
video = EPISODES[1]
languages = {Language('fra'), Language('ell'), Language('ron'), Language('eng'), Language('hun'),
Language('por'), Language('por', 'BR')}
matches = {frozenset(['series', 'episode', 'resolution', 'season']),
languages = set([Language('fra'), Language('ell'), Language('ron'), Language('eng'), Language('hun'),
Language('por'), Language('por', 'BR')])
matches = set([frozenset(['series', 'episode', 'resolution', 'season']),
frozenset(['series', 'episode', 'season', 'video_codec']),
frozenset(['series', 'episode', 'season'])}
frozenset(['series', 'episode', 'season'])])
with self.Provider() as provider:
subtitles = provider.query(video.series, video.season, video.episode)
self.assertTrue({frozenset(subtitle.compute_matches(video)) for subtitle in subtitles} == matches)
self.assertTrue({subtitle.language for subtitle in subtitles} == languages)
self.assertTrue(matches - set([frozenset(subtitle.compute_matches(video)) for subtitle in subtitles]) == set([]))
self.assertTrue(languages - set([subtitle.language for subtitle in subtitles]) == set([]))
def test_list_subtitles(self):
video = EPISODES[0]
languages = {Language('eng'), Language('fra')}
matches = {frozenset(['series', 'episode', 'season'])}
languages = set([Language('eng'), Language('fra')])
matches = set([frozenset(['series', 'episode', 'resolution', 'season']),
frozenset([u'series', u'episode', u'season'])])
with self.Provider() as provider:
subtitles = provider.list_subtitles(video, languages)
self.assertTrue({frozenset(subtitle.compute_matches(video)) for subtitle in subtitles} == matches)
self.assertTrue({subtitle.language for subtitle in subtitles} == languages)
self.assertTrue(matches - set([frozenset(subtitle.compute_matches(video)) for subtitle in subtitles]) == set([]))
self.assertTrue(languages - set([subtitle.language for subtitle in subtitles]) == set([]))
def test_download_subtitle(self):
video = EPISODES[0]
languages = {Language('hun')}
languages = (Language('hun'), )
with self.Provider() as provider:
subtitles = provider.list_subtitles(video, languages)
subtitle_text = provider.download_subtitle(subtitles[0])
@@ -378,8 +372,8 @@ class TVsubtitlesProviderTestCase(ProviderTestCase):
def suite():
suite = TestSuite()
suite.addTest(TestLoader().loadTestsFromTestCase(Addic7edProviderTestCase))
suite.addTest(TestLoader().loadTestsFromTestCase(BierDopjeProviderTestCase))
suite.addTest(TestLoader().loadTestsFromTestCase(OpenSubtitlesProviderTestCase))
suite.addTest(TestLoader().loadTestsFromTestCase(PodnapisiProviderTestCase))
suite.addTest(TestLoader().loadTestsFromTestCase(TheSubDBProviderTestCase))
suite.addTest(TestLoader().loadTestsFromTestCase(TVsubtitlesProviderTestCase))
return suite
+48 -45
View File
@@ -5,7 +5,7 @@ import os
import shutil
from unittest import TestCase, TestSuite, TestLoader, TextTestRunner
from babelfish import Language
from subliminal import list_subtitles, download_subtitles, download_best_subtitles, scan_video, scan_videos
from subliminal import list_subtitles, download_subtitles, download_best_subtitles, scan_video
from subliminal.tests.common import MOVIES, EPISODES
@@ -21,30 +21,30 @@ class ApiTestCase(TestCase):
def test_list_subtitles_movie_0(self):
videos = [MOVIES[0]]
languages = {Language('eng')}
languages = set([ Language('eng'), ])
subtitles = list_subtitles(videos, languages)
self.assertTrue(len(subtitles) == len(videos))
self.assertEqual(len(subtitles), len(videos))
self.assertTrue(len(subtitles[videos[0]]) > 0)
def test_list_subtitles_movie_0_por_br(self):
videos = [MOVIES[0]]
languages = {Language('por', 'BR')}
languages = set([Language('por', 'BR'), ])
subtitles = list_subtitles(videos, languages)
self.assertTrue(len(subtitles) == len(videos))
self.assertEqual(len(subtitles), len(videos))
self.assertTrue(len(subtitles[videos[0]]) > 0)
def test_list_subtitles_episodes(self):
videos = [EPISODES[0], EPISODES[1]]
languages = {Language('eng'), Language('fra')}
languages = set([Language('eng'), Language('fra')])
subtitles = list_subtitles(videos, languages)
self.assertTrue(len(subtitles) == len(videos))
self.assertEqual(len(subtitles), len(videos))
self.assertTrue(len(subtitles[videos[0]]) > 0)
def test_download_subtitles(self):
videos = [EPISODES[0], EPISODES[1]]
for video in videos:
video.name = os.path.join(TEST_DIR, video.name.split(os.sep)[-1])
languages = {Language('eng'), Language('fra')}
languages = set([Language('eng'), Language('fra')])
subtitles = list_subtitles(videos, languages)
download_subtitles(subtitles)
for video in videos:
@@ -55,7 +55,7 @@ class ApiTestCase(TestCase):
videos = [EPISODES[0], EPISODES[1]]
for video in videos:
video.name = os.path.join(TEST_DIR, video.name.split(os.sep)[-1])
languages = {Language('eng'), Language('fra')}
languages = set([Language('eng'), Language('fra')])
subtitles = list_subtitles(videos, languages)
download_subtitles(subtitles, single=True)
for video in videos:
@@ -65,10 +65,11 @@ class ApiTestCase(TestCase):
videos = [EPISODES[0], EPISODES[1]]
for video in videos:
video.name = os.path.join(TEST_DIR, video.name.split(os.sep)[-1])
languages = {Language('eng'), Language('fra')}
languages = set([Language('eng'), Language('fra')])
subtitles = download_best_subtitles(videos, languages)
for video in videos:
self.assertTrue(video in subtitles and len(subtitles[video]) == 2)
self.assertTrue(video in subtitles)
self.assertTrue(len(subtitles[video]) == 2)
self.assertTrue(os.path.exists(os.path.splitext(video.name)[0] + '.en.srt'))
self.assertTrue(os.path.exists(os.path.splitext(video.name)[0] + '.fr.srt'))
@@ -76,27 +77,28 @@ class ApiTestCase(TestCase):
videos = [EPISODES[0], EPISODES[1]]
for video in videos:
video.name = os.path.join(TEST_DIR, video.name.split(os.sep)[-1])
languages = {Language('eng'), Language('fra')}
languages = set([Language('eng'), Language('fra')])
subtitles = download_best_subtitles(videos, languages, single=True)
for video in videos:
self.assertTrue(video in subtitles and len(subtitles[video]) == 1)
self.assertTrue(video in subtitles)
self.assertEqual(len(subtitles[video]), 1)
self.assertTrue(os.path.exists(os.path.splitext(video.name)[0] + '.srt'))
def test_download_best_subtitles_min_score(self):
videos = [MOVIES[0]]
for video in videos:
video.name = os.path.join(TEST_DIR, video.name.split(os.sep)[-1])
languages = {Language('eng'), Language('fra')}
languages = set([Language('eng'), Language('fra')])
subtitles = download_best_subtitles(videos, languages, min_score=1000)
self.assertTrue(len(subtitles) == 0)
self.assertEqual(len(subtitles), 0)
def test_download_best_subtitles_hearing_impaired(self):
videos = [MOVIES[0]]
for video in videos:
video.name = os.path.join(TEST_DIR, video.name.split(os.sep)[-1])
languages = {Language('eng')}
languages = set([Language('eng'), ])
subtitles = download_best_subtitles(videos, languages, hearing_impaired=True)
self.assertTrue(subtitles[videos[0]][0].hearing_impaired == True)
self.assertTrue(subtitles[videos[0]][0].hearing_impaired)
class VideoTestCase(TestCase):
@@ -111,47 +113,47 @@ class VideoTestCase(TestCase):
def test_scan_video_movie(self):
video = MOVIES[0]
scanned_video = scan_video(os.path.join(TEST_DIR, os.path.split(video.name)[1]))
self.assertTrue(scanned_video.name == os.path.join(TEST_DIR, os.path.split(video.name)[1]))
self.assertTrue(scanned_video.title.lower() == video.title.lower())
self.assertTrue(scanned_video.year == video.year)
self.assertTrue(scanned_video.video_codec == video.video_codec)
self.assertTrue(scanned_video.resolution == video.resolution)
self.assertTrue(scanned_video.release_group == video.release_group)
self.assertTrue(scanned_video.subtitle_languages == set())
self.assertTrue(scanned_video.hashes == {})
self.assertTrue(scanned_video.audio_codec is None)
self.assertTrue(scanned_video.imdb_id is None)
self.assertTrue(scanned_video.size == 0)
self.assertEqual(scanned_video.name, os.path.join(TEST_DIR, os.path.split(video.name)[1]))
self.assertEqual(scanned_video.title.lower(), video.title.lower())
self.assertEqual(scanned_video.year, video.year)
self.assertEqual(scanned_video.video_codec, video.video_codec)
self.assertEqual(scanned_video.resolution, video.resolution)
self.assertEqual(scanned_video.release_group, video.release_group)
self.assertEqual(scanned_video.subtitle_languages, set())
self.assertEqual(scanned_video.hashes, {})
self.assertEqual(scanned_video.audio_codec, None)
self.assertEqual(scanned_video.imdb_id, None)
self.assertEqual(scanned_video.size, 0)
def test_scan_video_episode(self):
video = EPISODES[0]
scanned_video = scan_video(os.path.join(TEST_DIR, os.path.split(video.name)[1]))
self.assertTrue(scanned_video.name == os.path.join(TEST_DIR, os.path.split(video.name)[1]))
self.assertTrue(scanned_video.series == video.series)
self.assertTrue(scanned_video.season == video.season)
self.assertTrue(scanned_video.episode == video.episode)
self.assertTrue(scanned_video.video_codec == video.video_codec)
self.assertTrue(scanned_video.resolution == video.resolution)
self.assertTrue(scanned_video.release_group == video.release_group)
self.assertTrue(scanned_video.subtitle_languages == set())
self.assertTrue(scanned_video.hashes == {})
self.assertTrue(scanned_video.title is None)
self.assertTrue(scanned_video.tvdb_id is None)
self.assertTrue(scanned_video.imdb_id is None)
self.assertTrue(scanned_video.audio_codec is None)
self.assertTrue(scanned_video.size == 0)
self.assertEqual(scanned_video.name, os.path.join(TEST_DIR, os.path.split(video.name)[1]))
self.assertEqual(scanned_video.series, video.series)
self.assertEqual(scanned_video.season, video.season)
self.assertEqual(scanned_video.episode, video.episode)
self.assertEqual(scanned_video.video_codec, video.video_codec)
self.assertEqual(scanned_video.resolution, video.resolution)
self.assertEqual(scanned_video.release_group, video.release_group)
self.assertEqual(scanned_video.subtitle_languages, set())
self.assertEqual(scanned_video.hashes, {})
self.assertEqual(scanned_video.title, None)
self.assertEqual(scanned_video.tvdb_id, None)
self.assertEqual(scanned_video.imdb_id, None)
self.assertEqual(scanned_video.audio_codec, None)
self.assertEqual(scanned_video.size, 0)
def test_scan_video_subtitle_language_und(self):
video = EPISODES[0]
open(os.path.join(TEST_DIR, os.path.splitext(os.path.split(video.name)[1])[0]) + '.srt', 'w').close()
scanned_video = scan_video(os.path.join(TEST_DIR, os.path.split(video.name)[1]))
self.assertTrue(scanned_video.subtitle_languages == {Language('und')})
self.assertEqual(scanned_video.subtitle_languages, set([Language('und'), ]))
def test_scan_video_subtitles_language_eng(self):
video = EPISODES[0]
open(os.path.join(TEST_DIR, os.path.splitext(os.path.split(video.name)[1])[0]) + '.en.srt', 'w').close()
scanned_video = scan_video(os.path.join(TEST_DIR, os.path.split(video.name)[1]))
self.assertTrue(scanned_video.subtitle_languages == {Language('eng')})
self.assertEqual(scanned_video.subtitle_languages, set([Language('eng'), ]))
def test_scan_video_subtitles_languages(self):
video = EPISODES[0]
@@ -159,7 +161,8 @@ class VideoTestCase(TestCase):
open(os.path.join(TEST_DIR, os.path.splitext(os.path.split(video.name)[1])[0]) + '.fr.srt', 'w').close()
open(os.path.join(TEST_DIR, os.path.splitext(os.path.split(video.name)[1])[0]) + '.srt', 'w').close()
scanned_video = scan_video(os.path.join(TEST_DIR, os.path.split(video.name)[1]))
self.assertTrue(scanned_video.subtitle_languages == {Language('eng'), Language('fra'), Language('und')})
self.assertEqual(scanned_video.subtitle_languages, set([Language('eng'), Language('fra'), Language('und')]))
def suite():
suite = TestSuite()
+56 -11
View File
@@ -71,6 +71,10 @@ class Video(object):
def __hash__(self):
return hash(self.name)
def __eq__(self, other):
return self.__class__.__name__ == other.__class__.__name__\
and self.name == other.name
class Episode(Video):
"""Episode :class:`Video`
@@ -112,6 +116,18 @@ class Episode(Video):
def __repr__(self):
return '<%s [%r, %rx%r]>' % (self.__class__.__name__, self.series, self.season, self.episode)
def __hash__(self):
return hash((
self.series,
self.season,
self.episode,
))
def __eq__(self, other):
return self.__class__.__name__ == other.__class__.__name__\
and self.series == other.series\
and self.season == other.season\
and self.episode == other.episode
class Movie(Video):
"""Movie :class:`Video`
@@ -147,6 +163,18 @@ class Movie(Video):
return '<%s [%r]>' % (self.__class__.__name__, self.title)
return '<%s [%r, %r]>' % (self.__class__.__name__, self.title, self.year)
def __hash__(self):
if self.year is None:
return hash((
self.title,
self.year,
))
return hash(self.title)
def __eq__(self, other):
return self.__class__.__name__ == other.__class__.__name__\
and self.title == other.title\
and self.year == other.year
def scan_subtitle_languages(path):
"""Search for subtitles with alpha2 extension from a video `path` and return their language
@@ -156,7 +184,7 @@ def scan_subtitle_languages(path):
:rtype: set
"""
language_extensions = tuple('.' + c for c in babelfish.CONVERTERS['alpha2'].codes)
language_extensions = tuple('.' + c for c in babelfish.language_converters['alpha2'].codes)
dirpath, filename = os.path.split(path)
subtitles = set()
for p in os.listdir(dirpath):
@@ -169,12 +197,14 @@ def scan_subtitle_languages(path):
return subtitles
def scan_video(path, subtitles=True, embedded_subtitles=True):
def scan_video(path, subtitles=True, embedded_subtitles=True, video=None):
"""Scan a video and its subtitle languages from a video `path`
:param string path: absolute path to the video
:param bool subtitles: scan for subtitles with the same name
:param bool embedded_subtitles: scan for embedded subtitles
:parm :class:`Video`: optionally specify a video if you've already detected on
by other means.
:return: the scanned video
:rtype: :class:`Video`
:raise: ValueError if cannot guess enough information from the path
@@ -182,7 +212,12 @@ def scan_video(path, subtitles=True, embedded_subtitles=True):
"""
dirpath, filename = os.path.split(path)
logger.info('Scanning video %r in %r', filename, dirpath)
video = Video.fromguess(path, guessit.guess_file_info(path, 'autodetect'))
if not video:
video = Video.fromguess(
path,
guessit.guess_file_info(path, info=['filename']),
)
video.size = os.path.getsize(path)
if video.size > 10485760:
logger.debug('Size is %d', video.size)
@@ -239,14 +274,24 @@ def scan_video(path, subtitles=True, embedded_subtitles=True):
if embedded_subtitles:
embedded_subtitle_languages = set()
for st in mkv.subtitle_tracks:
try:
embedded_subtitle_languages.add(babelfish.Language.fromalpha3b(st.language or 'und'))
except babelfish.Error:
logger.error('Embedded subtitle language %r is not a valid language', st.language)
if st.language:
try:
embedded_subtitle_languages.add(babelfish.Language.fromalpha3b(st.language))
except babelfish.Error:
logger.error('Embedded subtitle track language %r is not a valid language', st.language)
embedded_subtitle_languages.add(babelfish.Language('und'))
elif st.name:
try:
embedded_subtitle_languages.add(babelfish.Language.fromname(st.name))
except babelfish.Error:
logger.error('Embedded subtitle track name %r is not a valid language', st.name)
embedded_subtitle_languages.add(babelfish.Language('und'))
else:
embedded_subtitle_languages.add(babelfish.Language('und'))
logger.debug('Found embedded subtitle %r with enzyme', embedded_subtitle_languages)
video.subtitle_languages |= embedded_subtitle_languages
else:
logger.info('MKV has no subtitle track')
logger.debug('MKV has no subtitle track')
except enzyme.Error:
logger.error('Parsing video metadata with enzyme failed')
return video
@@ -282,12 +327,12 @@ def scan_videos(paths, subtitles=True, embedded_subtitles=True, age=None):
for dirpath, dirnames, filenames in os.walk(path):
# skip badly encoded directories
if isinstance(dirpath, bytes):
logger.error('Skipping badly encoded directory %r', dirpath.decode('utf-8', errors='replace'))
logger.error('Skipping badly encoded directory %r', dirpath.decode('utf-8', 'replace'))
continue
# skip badly encoded and hidden sub directories
for dirname in list(dirnames):
if isinstance(dirname, bytes):
logger.error('Skipping badly encoded dirname %r in %r', dirname.decode('utf-8', errors='replace'),
logger.error('Skipping badly encoded dirname %r in %r', dirname.decode('utf-8', 'replace'),
dirpath)
dirnames.remove(dirname)
elif dirname.startswith('.'):
@@ -297,7 +342,7 @@ def scan_videos(paths, subtitles=True, embedded_subtitles=True, age=None):
for filename in filenames:
# skip badly encoded files
if isinstance(filename, bytes):
logger.error('Skipping badly encoded filename %r in %r', filename.decode('utf-8', errors='replace'),
logger.error('Skipping badly encoded filename %r in %r', filename.decode('utf-8', 'replace'),
dirpath)
continue
# filter videos