mirror of
https://github.com/dgtlmoon/changedetection.io.git
synced 2026-11-18 20:54:26 +00:00
Example plugins - API fetcher and check stats, watch list notes hook, update_finalize fix (#4622)
* Example plugins - API fetcher and check stats examples, watch list notes hook, update_finalize fix New hook watch_list_extra_text(watch) (ported from browser-ui): short plain-text notes under each watch in the watch list, escaped when rendered. New example plugins, each `pip install .`-able and importing only from plugin_api: - fetcher/api_fetcher: register_content_fetcher, plugin_settings_tab, fetcher_status_icon, plugin_static_path - lifecycle/check_stats: update_handler_alter, update_finalize, watch_list_extra_text, ui_edit_stats_extras, get_html_head_extras, plugin_static_path - processor/wordcounter: also shows "Found: N words" in the watch list via watch_list_extra_text Fixes found by the examples: - update_finalize always got update_handler=None after a successful check (the worker deleted the handler before the finally block captured it), so update_handler_alter + update_finalize couldn't be paired - The API rejected plugin fetchers in 'fetch_backend' (hardcoded pattern), now built from installed fetchers - fetcher_status_icon docstring said it returns HTML, it returns a dict plugin_api 0.3.0: exports Fetcher and load_plugin_settings. Tests check each example end to end, including that it's listed under Settings > Info. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Example plugins - Rename the fetcher example to fetcher/example_browser_fetcher "Example browser fetcher": an extra fetcher that gets pages through its own API. Package cdio_example_browser_fetcher, fetch method html_example_browser_fetcher. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Example browser fetcher - Return a screenshot (random noise PNG, standard library only) Shows the screenshot side of a fetcher: supports_screenshots = True and self.screenshot set to PNG bytes, saved as last-screenshot.png. The test checks it's served as a valid 240x160 PNG. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
c0b9320ef3
commit
b40193b080
22 files changed
+618
-30
No files matched your search
@@ -5,7 +5,7 @@ active until you install it.
|
||||
|
||||
To install one, run `pip install .` from its directory, in the same Python environment that runs
|
||||
changedetection.io, then restart changedetection.io. Plugins are picked up through the `changedetectionio`
|
||||
entry point in their `pyproject.toml`. They need plugin API 0.2.0 or newer
|
||||
entry point in their `pyproject.toml`. They need plugin API 0.3.0 or newer
|
||||
(`changedetectionio.plugin_api.PLUGIN_API_VERSION`).
|
||||
|
||||
Plugins import only from `changedetectionio.plugin_api`, everything else in the package is internal and
|
||||
@@ -21,6 +21,7 @@ the page and reports a change when the count changes. The `min_word_length` sett
|
||||
**Hooks**
|
||||
- `register_processor` - adds `word_counter` as a processor
|
||||
- `get_api_watch_processor_state` - shows the count as `processor_state_word_counter` in the API
|
||||
- `watch_list_extra_text` - "Found: 5 words" under the watch in the watch list
|
||||
|
||||
**Also shows** `processor.py` (the check itself) and `api.yaml` (documents `processor_config_word_counter`
|
||||
and `processor_state_word_counter` in the API spec).
|
||||
@@ -43,10 +44,60 @@ curl "http://localhost:5000/api/v1/watch/YOUR-UUID" -H "x-api-key: YOUR_API_KEY"
|
||||
|
||||
---
|
||||
|
||||
## Example browser fetcher - `fetcher/example_browser_fetcher`
|
||||
|
||||
A new fetch method, "Example browser fetcher", that gets pages through its own API instead of fetching
|
||||
them directly, for example a remote browser or rendering service.
|
||||
|
||||
**Hooks**
|
||||
- `register_content_fetcher` - adds the `html_example_browser_fetcher` fetch method
|
||||
- `plugin_settings_tab` - an "Example browser fetcher" tab in Settings for the API URL and key
|
||||
- `fetcher_status_icon` - the icon next to watches using it in the watch list
|
||||
- `plugin_static_path` - serves that icon
|
||||
|
||||
**Also shows** `fetcher.py`, a `Fetcher` subclass, reading its settings with `load_plugin_settings` and
|
||||
returning a screenshot (random noise here, a real browser service would send its own).
|
||||
|
||||
**Install**
|
||||
```bash
|
||||
cd changedetectionio/example_plugins/fetcher/example_browser_fetcher
|
||||
pip install .
|
||||
```
|
||||
|
||||
**Try it:** in Settings > "Example browser fetcher", set the API URL with `{url}` where the page URL goes,
|
||||
for example `https://api.example.com/fetch?url={url}`, and your key (sent as `X-Api-Key`). Then pick
|
||||
"Example browser fetcher" as a watch's fetch method, or `"fetch_backend": "html_example_browser_fetcher"`
|
||||
in the API.
|
||||
|
||||
---
|
||||
|
||||
## Check stats - `lifecycle/check_stats`
|
||||
|
||||
Times every check. Shows "Last check took 1.23s" under each watch in the watch list, and a small table on
|
||||
the edit page's Stats tab. Stats are kept in memory, so they start again after a restart.
|
||||
|
||||
**Hooks**
|
||||
- `update_handler_alter` - runs as a check starts, notes the time
|
||||
- `update_finalize` - runs when the check is over (worked or not), records how long it took
|
||||
- `watch_list_extra_text` - the note under the watch in the watch list
|
||||
- `ui_edit_stats_extras` - the table on the edit page's Stats tab
|
||||
- `get_html_head_extras` - links its CSS on every page
|
||||
- `plugin_static_path` - serves that CSS
|
||||
|
||||
**Install**
|
||||
```bash
|
||||
cd changedetectionio/example_plugins/lifecycle/check_stats
|
||||
pip install .
|
||||
```
|
||||
|
||||
**Try it:** open the watch list after the next checks have run.
|
||||
|
||||
---
|
||||
|
||||
## Writing your own
|
||||
|
||||
Copy an example's directory, then rename the package directory and its `name` / entry point in
|
||||
`pyproject.toml`. For a processor, also change `PROCESSOR_NAME` and the two schema names in `api.yaml`.
|
||||
`pyproject.toml`. Once installed, it's listed under Settings > Info with the first line of its docstring. For a processor, also change `PROCESSOR_NAME` and the two schema names in `api.yaml`.
|
||||
Every hook and its arguments are documented on `ChangeDetectionSpec` in
|
||||
[`pluggy_interface.py`](../pluggy_interface.py).
|
||||
|
||||
|
||||
+55
@@ -0,0 +1,55 @@
|
||||
"""
|
||||
Example plugin - an extra browser fetcher that gets pages through its own API.
|
||||
|
||||
Configure it under Settings > "Example browser fetcher", then pick "Example browser fetcher" as a
|
||||
watch's fetch method. Install it with `pip install .` from the directory above this one, see
|
||||
example_plugins/README.md. It is not active until installed (or registered).
|
||||
"""
|
||||
import os
|
||||
|
||||
from changedetectionio.plugin_api import hookimpl
|
||||
|
||||
# Names your settings file ({datastore}/cdio_example_browser_fetcher.json) and the settings tab
|
||||
PLUGIN_ID = 'cdio_example_browser_fetcher'
|
||||
# Fetcher names must start with 'html_', this is what's stored in a watch's 'fetch_backend'
|
||||
FETCHER_NAME = 'html_example_browser_fetcher'
|
||||
|
||||
# Set at startup by changedetection.io (inject_datastore_into_plugins), the fetcher reads its
|
||||
# settings with it
|
||||
datastore = None
|
||||
|
||||
|
||||
@hookimpl
|
||||
def register_content_fetcher():
|
||||
from .fetcher import fetcher
|
||||
return (FETCHER_NAME, fetcher)
|
||||
|
||||
|
||||
@hookimpl
|
||||
def plugin_settings_tab():
|
||||
from wtforms import Form, StringField
|
||||
|
||||
class SettingsForm(Form):
|
||||
# These share the settings page form with everything else, so give them unique names
|
||||
example_browser_fetcher_url = StringField(
|
||||
'API URL',
|
||||
description='Use {url} where the page URL goes, for example https://api.example.com/fetch?url={url}')
|
||||
example_browser_fetcher_key = StringField(
|
||||
'API key', description='Sent to the API in the X-Api-Key header')
|
||||
|
||||
return {'plugin_id': PLUGIN_ID, 'tab_label': 'Example browser fetcher', 'form_class': SettingsForm}
|
||||
|
||||
|
||||
@hookimpl
|
||||
def fetcher_status_icon(fetcher_name):
|
||||
"""The icon next to watches using this fetcher in the watch list."""
|
||||
if fetcher_name != FETCHER_NAME:
|
||||
return None
|
||||
return {'filename': 'cdio_example_browser_fetcher.svg', 'alt': 'Example browser fetcher', 'title': 'Fetched with the example browser fetcher',
|
||||
'group': 'plugin', 'style': 'height: 1em;'}
|
||||
|
||||
|
||||
@hookimpl
|
||||
def plugin_static_path():
|
||||
"""Files in here are served at /static/plugin/<filename>."""
|
||||
return os.path.join(os.path.dirname(__file__), 'static')
|
||||
+75
@@ -0,0 +1,75 @@
|
||||
import asyncio
|
||||
from urllib.parse import quote
|
||||
|
||||
from changedetectionio.plugin_api import Fetcher, load_plugin_settings
|
||||
|
||||
|
||||
def noise_png(width=240, height=160):
|
||||
"""A PNG of random noise, standing in for a real page screenshot. Standard library only."""
|
||||
import os
|
||||
import struct
|
||||
import zlib
|
||||
|
||||
def chunk(kind, data):
|
||||
return struct.pack('>I', len(data)) + kind + data + struct.pack('>I', zlib.crc32(kind + data) & 0xffffffff)
|
||||
|
||||
# Each row: filter type 0, then width RGB pixels
|
||||
pixels = b''.join(b'\x00' + os.urandom(width * 3) for _ in range(height))
|
||||
return (b'\x89PNG\r\n\x1a\n'
|
||||
+ chunk(b'IHDR', struct.pack('>IIBBBBB', width, height, 8, 2, 0, 0, 0)) # 8 bit RGB
|
||||
+ chunk(b'IDAT', zlib.compress(pixels))
|
||||
+ chunk(b'IEND', b''))
|
||||
|
||||
|
||||
class fetcher(Fetcher):
|
||||
fetcher_description = 'Example browser fetcher'
|
||||
# Tells changedetection.io this fetcher returns a screenshot (see _fetch), so it's shown and saved
|
||||
supports_screenshots = True
|
||||
|
||||
def __init__(self, proxy_override=None, custom_browser_connection_url=None, **kwargs):
|
||||
super().__init__(**kwargs)
|
||||
|
||||
async def run(self, url=None, timeout=None, request_headers=None, **kwargs):
|
||||
# requests is blocking, keep it off the event loop
|
||||
await asyncio.get_event_loop().run_in_executor(None, lambda: self._fetch(url, timeout, request_headers))
|
||||
|
||||
def _fetch(self, url, timeout, request_headers):
|
||||
import requests
|
||||
from . import PLUGIN_ID, datastore
|
||||
|
||||
settings = load_plugin_settings(datastore.datastore_path, PLUGIN_ID) if datastore else {}
|
||||
api_url = (settings.get('example_browser_fetcher_url') or '').strip()
|
||||
if '{url}' not in api_url:
|
||||
raise Exception("Example browser fetcher isn't set up, add an API URL with {url} in Settings")
|
||||
|
||||
headers = dict(request_headers or {})
|
||||
if settings.get('example_browser_fetcher_key'):
|
||||
headers['X-Api-Key'] = settings['example_browser_fetcher_key']
|
||||
|
||||
r = requests.get(api_url.replace('{url}', quote(url, safe='')), headers=headers, timeout=timeout or 60)
|
||||
if not r.ok:
|
||||
raise Exception(f"Example browser fetcher API returned HTTP {r.status_code}")
|
||||
|
||||
# What a fetcher must set for the processor
|
||||
self.status_code = r.status_code
|
||||
self.headers = dict(r.headers)
|
||||
self.content = r.text
|
||||
self.raw_content = r.content
|
||||
# A real browser service would return its screenshot of the page, saved as last-screenshot.png
|
||||
# (so return PNG bytes). This example just makes some noise.
|
||||
self.screenshot = noise_png()
|
||||
|
||||
async def quit(self, watch=None):
|
||||
pass
|
||||
|
||||
def get_error(self):
|
||||
return self.error
|
||||
|
||||
def get_last_status_code(self):
|
||||
return self.status_code
|
||||
|
||||
def screenshot_step(self, step_n):
|
||||
return None
|
||||
|
||||
def is_ready(self):
|
||||
return True
|
||||
+1
@@ -0,0 +1 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M18 10h-1.26A8 8 0 1 0 9 20h9a5 5 0 0 0 0-10z"/></svg>
|
||||
|
After Width: | Height: | Size: 222 B |
@@ -0,0 +1,23 @@
|
||||
[build-system]
|
||||
requires = ["setuptools>=77"] # 77+ for the SPDX license string
|
||||
build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "cdio-example-browser-fetcher"
|
||||
version = "0.1.0"
|
||||
description = "Example changedetection.io plugin: an extra browser fetcher that gets pages through its own API"
|
||||
license = "Apache-2.0"
|
||||
requires-python = ">=3.11"
|
||||
# No dependency on changedetection.io itself: install this into the same environment that runs it
|
||||
# (it needs plugin API 0.3.0 or newer, changedetectionio.plugin_api.PLUGIN_API_VERSION)
|
||||
dependencies = []
|
||||
|
||||
# This is what makes changedetection.io load the plugin at startup
|
||||
[project.entry-points.changedetectionio]
|
||||
cdio_example_browser_fetcher = "cdio_example_browser_fetcher"
|
||||
|
||||
[tool.setuptools]
|
||||
packages = ["cdio_example_browser_fetcher"]
|
||||
|
||||
[tool.setuptools.package-data]
|
||||
cdio_example_browser_fetcher = ["static/*"]
|
||||
+84
@@ -0,0 +1,84 @@
|
||||
"""
|
||||
Example plugin - times every check and shows the result in the watch list and on the edit page.
|
||||
|
||||
Install it with `pip install .` from the directory above this one, see example_plugins/README.md.
|
||||
It is not active until installed (or registered), nothing scans example_plugins/ for plugins.
|
||||
|
||||
The stats are kept in memory only, so they start empty again after a restart.
|
||||
"""
|
||||
import os
|
||||
import time
|
||||
|
||||
from markupsafe import escape
|
||||
|
||||
from changedetectionio.plugin_api import hookimpl
|
||||
|
||||
# Per watch uuid: {'duration': seconds, 'ok': bool, 'error': str or None, 'finished': unix time, 'count': int}
|
||||
STATS = {}
|
||||
|
||||
# Where the handler keeps its start time between the two check hooks below
|
||||
_STARTED_ATTR = '_cdio_example_check_stats_started'
|
||||
|
||||
|
||||
@hookimpl
|
||||
def update_handler_alter(update_handler, watch, datastore):
|
||||
"""Called once a check is about to start, before the page is fetched."""
|
||||
setattr(update_handler, _STARTED_ATTR, time.monotonic())
|
||||
# None = keep the handler as it is, you could also return a wrapped one
|
||||
return None
|
||||
|
||||
|
||||
@hookimpl
|
||||
def update_finalize(update_handler, watch, datastore, processing_exception):
|
||||
"""Called once the check is over, whether it worked or not."""
|
||||
started = getattr(update_handler, _STARTED_ATTR, None) if update_handler else None
|
||||
if started is None or not watch:
|
||||
return
|
||||
previous = STATS.get(watch.get('uuid'), {})
|
||||
STATS[watch.get('uuid')] = {
|
||||
'duration': time.monotonic() - started,
|
||||
'ok': processing_exception is None,
|
||||
'error': str(processing_exception) if processing_exception else None,
|
||||
'finished': time.time(),
|
||||
'count': previous.get('count', 0) + 1,
|
||||
}
|
||||
|
||||
|
||||
@hookimpl
|
||||
def watch_list_extra_text(watch):
|
||||
"""A short note under the watch in the watch list. Called for every row, so in memory only."""
|
||||
stats = STATS.get(watch.get('uuid'))
|
||||
if not stats:
|
||||
return None
|
||||
if stats['ok']:
|
||||
return f"Last check took {stats['duration']:.2f}s"
|
||||
return f"Last check failed after {stats['duration']:.2f}s"
|
||||
|
||||
|
||||
@hookimpl
|
||||
def ui_edit_stats_extras(watch):
|
||||
"""A small table on the edit page's Stats tab. This is raw HTML, so escape anything you include."""
|
||||
stats = STATS.get(watch.get('uuid'))
|
||||
if not stats:
|
||||
return '<p class="cdio-example-check-stats">No checks timed since the last restart.</p>'
|
||||
rows = [
|
||||
('Checks timed since restart', stats['count']),
|
||||
('Last check took', f"{stats['duration']:.2f}s"),
|
||||
('Last check', 'OK' if stats['ok'] else f"Failed: {stats['error']}"),
|
||||
]
|
||||
body = ''.join(f'<tr><td>{escape(label)}</td><td>{escape(value)}</td></tr>' for label, value in rows)
|
||||
return f'<table class="pure-table cdio-example-check-stats"><tbody>{body}</tbody></table>'
|
||||
|
||||
|
||||
@hookimpl
|
||||
def plugin_static_path():
|
||||
"""Files in here are served at /static/plugin/<filename>."""
|
||||
return os.path.join(os.path.dirname(__file__), 'static')
|
||||
|
||||
|
||||
@hookimpl
|
||||
def get_html_head_extras():
|
||||
"""Added inside <head> on every page."""
|
||||
from flask import url_for
|
||||
css = url_for('static_content', group='plugin', filename='cdio_example_check_stats.css')
|
||||
return f'<link rel="stylesheet" href="{css}">'
|
||||
+4
@@ -0,0 +1,4 @@
|
||||
/* Served by the plugin's plugin_static_path(), linked from get_html_head_extras() */
|
||||
.cdio-example-check-stats td:first-child {
|
||||
font-weight: bold;
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
[build-system]
|
||||
requires = ["setuptools>=77"] # 77+ for the SPDX license string
|
||||
build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "cdio-example-check-stats"
|
||||
version = "0.1.0"
|
||||
description = "Example changedetection.io plugin: times every check, shows it in the watch list and edit page"
|
||||
license = "Apache-2.0"
|
||||
requires-python = ">=3.11"
|
||||
# No dependency on changedetection.io itself: install this into the same environment that runs it
|
||||
# (it needs plugin API 0.3.0 or newer, changedetectionio.plugin_api.PLUGIN_API_VERSION)
|
||||
dependencies = []
|
||||
|
||||
# This is what makes changedetection.io load the plugin at startup
|
||||
[project.entry-points.changedetectionio]
|
||||
cdio_example_check_stats = "cdio_example_check_stats"
|
||||
|
||||
[tool.setuptools]
|
||||
packages = ["cdio_example_check_stats"]
|
||||
|
||||
[tool.setuptools.package-data]
|
||||
cdio_example_check_stats = ["static/*"]
|
||||
+9
@@ -25,6 +25,15 @@ def register_processor():
|
||||
}
|
||||
|
||||
|
||||
@hookimpl
|
||||
def watch_list_extra_text(watch):
|
||||
"""A note under the watch in the watch list. Called for every row, so in memory only."""
|
||||
if watch.get('processor') != PROCESSOR_NAME:
|
||||
return None
|
||||
word_count = (watch.get(f'processor_state_{PROCESSOR_NAME}') or {}).get('word_count')
|
||||
return f"Found: {word_count} words" if word_count is not None else None
|
||||
|
||||
|
||||
@hookimpl
|
||||
def get_api_watch_processor_state(watch, processor_name, context):
|
||||
"""processor_state_word_counter in GET /api/v1/watch/<uuid> and GET /api/v1/watch, see api.yaml"""
|
||||
|
||||
Reference in new issue
Block a user