Bump the bundled free-tier DB-IP IP-to-Country Lite database from the
2025-12 release to the latest available 2026-06 release, and update the
geodb register path accordingly.
Source: https://db-ip.com/db/download/ip-to-country (CC-BY-4.0)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Streaming a large-topic send paginated subscribers 1000 at a time, so a
500K-recipient topic issued ~500 subscriber-page queries plus their per-page
target resolutions. Raise MESSAGE_RECIPIENTS_PAGE_SIZE to 5000 (APP_LIMIT_COUNT),
cutting topic-path round-trips ~5x with no extra memory pressure (bounded
concurrency still caps in-flight sends). The streaming test fixture now sizes
itself to the page-size constant so it keeps exercising multi-page pagination.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Fiber-based SwooleCoroutinePolyfill shadowed real Swoole symbols and
poisoned PHPStan, and driving the worker through real Swoole coroutines
(Co\run/batch) inside the shared unit process contaminated and crashed
later, unrelated tests. The native extension is now available, so delete
the polyfill (and its PHPStan exclude) and test the worker's logic by
invoking its component methods directly with zero coroutines.
Reflection invokes the private methods without setAccessible (deprecated
on PHP 8.5), and a TestableMessaging subclass zeroes retryDelay so the
backoff never sleeps. Coverage: isRetryableError classification,
retrySend (throttle-then-success, partial-throttle scoping, non-retryable
terminal, exhausted retries, error-cap with exact delivered accounting,
isolated expired-token cleanup failure) and streamRecipients (cursor
pagination by topicInternalId, no targets-subquery read, no runaway
limit, per-page dedup, user/direct-target resolution). The bounded-
concurrency Semaphore + batch() integration stays covered by the e2e
Messaging suite, where executing real coroutines is safe.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Locking decorator serialized every Redis command on the single
consumer connection shared by all consume coroutines. On an idle queue
each waiting coroutine holds the mutex for a full 2s blocking pop, so a
finished job's completion ops (remove job key, increment stats) starve
behind the poppers: a 10ms cache purge took 22-24s to ack, and Storage's
testFilePreviewCache observed stale previews because purges landed tens
of seconds after enqueue.
Pass a Broker\Pool as the adapter consumer instead: each consume
coroutine leases its own connection from the consumer pool for the
lifetime of its loop, eliminating the shared-connection mutex entirely.
The Locking wrapper in the pool factory is reverted, and consumer pools
get +2 headroom above the coroutine count for queue-size telemetry
reads. Purge ack latency drops to 4-58ms.
Known tradeoff: Broker\Pool::close() is a no-op, so SIGTERM no longer
breaks the blocking pop early; workers rely on the process kill that
already follows.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two CI failure causes:
- Connection pools are sized for HTTP-server worker counts and came out
at 1 per worker process; 8 consume coroutines exhausted them
(Pool 'cache_redis_main' is empty). Floor the pool size at
_APP_WORKER_MAX_COROUTINES.
- The mails and messaging workers disable the Swoole TCP hook at
runtime to work around the TLS handshake crash (swoole-src#4909,
#7914). The flag is process-global, so after the first job every
blocking Redis pop freezes the coroutine scheduler and deliveries
arrive minutes late. Both workers stay at 1 coroutine, matching
pre-coroutine behavior.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Coroutine-safety audit of all worker actions (instance state mutated
during action() is shared across consume coroutines):
- audits: batches logs across messages ($logs/$lastTriggeredTime) with
Commit/NoCommit ack semantics tied to the shared batch; concurrent
flushes lose or duplicate audit entries
- webhooks: $errors buffer is reset per job and drives the
retry/disable path; concurrent jobs wipe or steal each other's errors
- migrations: project DB callables are rebound on the instance per
message; a concurrent message swaps database handles mid-migration
Messaging cached a project-scoped Local device unkeyed, reusing the
first project's storage path for every subsequent message; create it
per call instead.
Deletes, mails, certificates, executions, functions, builds and
screenshots keep all per-message state in locals and stay at 8.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
StatsUsage batches metrics across messages on shared instance state
($stats/$keys/$projects/$statDocuments); concurrent consume coroutines
race the flush in commitToDB, losing aggregates on reset and
double-applying upsertDocumentsWithIncrease on overlapping flushes.
StatsResources resets its shared $documents buffer at the start of each
job, so a second coroutine starting mid-count wipes the first one's
batch and the projects' metrics interleave into one write.
Both stay at 1 coroutine until accumulation moves to per-message state.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Only the databases worker is ordering-sensitive (schema operations
must apply in sequence); every other worker handles independent jobs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Workers run a single process (_APP_WORKERS_NUM=1) with concurrency
provided by coroutines instead. I/O-bound workers get 8 coroutines;
CPU-heavy, stateful, or ordering-sensitive workers (databases, builds,
migrations, stats-resources, stats-usage) stay serial at 1.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Upgrade utopia-php/queue to 0.19.0 and utopia-php/platform to 1.0.0-rc4,
enabling workers to consume multiple jobs concurrently via Swoole
coroutines within a single worker process.
- Workers pass a maxCoroutines option to the queue adapter, configured
via the new _APP_WORKER_MAX_COROUTINES env var (default 1)
- Per-message DI resources now live in a coroutine-scoped context
container (Server::context()) instead of the shared container
- Consumer Redis connections are wrapped in the Locking decorator so
concurrent coroutines can safely share one connection
- Add a functions worker load test: 30 async 1s executions drain in
31.1s at 1 coroutine, 10.4s at 5, 7.3s at 10
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The test polyfill declares stand-in Swoole\Coroutine and Swoole\Coroutine\Channel
classes behind a runtime extension check. PHPStan ignores the guard and parsed
those minimal declarations, shadowing the real swoole/ide-helper stubs and
reporting false 'undefined static method' errors codebase-wide (e.g. getContext,
join, create, Channel::close in app/realtime.php). Exclude the polyfill from
analysis so the ide-helper stubs are used. Verified: Checks/Analyze clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Tests/Unit runs in the Appwrite container (PHP 8.5) with the native Swoole
extension, where these tests' real Co\run()/batch() scheduler runs in-process
and destabilises later coroutine-sensitive tests in the shared unit suite
(the run aborted mid-suite at ~126/644 inside the Network/Fetcher tests).
Skip them under native Swoole; they still run locally via the Fiber polyfill,
and the real worker path is covered by the e2e Messaging suite.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The CI unit-test job runs inside the Appwrite container on PHP 8.5, where
ReflectionMethod::setAccessible() is deprecated, and the suite fails on
triggered deprecations. It is also a no-op on PHP 8.1+ (reflection invokes
private methods without it), so it was both unnecessary and the cause of the
red Tests/Unit check. Reverts the setAccessible addition from 7c8d7dcb80.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Swoole\Coroutine::sleep() rejects 0/negative values (CI runs unit tests with
the real Swoole extension, where the Fiber test polyfill is inert). The retry
tests override the base delay to 0 for speed, which made sleep(0.0) abort the
process under real Swoole. Guard the call; production delays are always >= 1s.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Apply five reviewer fixes to the large-topic send path without touching
the adapter-agnostic messaging library:
- Empty recipient pages no longer count as recipients: a page that
resolves to zero identifiers is skipped instead of running batch([])
and masking the "No valid recipients found." outcome.
- Best-effort expired-device-token cleanup is isolated from the provider
send: the send try/catch wraps only adapter->send(), results are
processed outside it, and the findOne/updateDocument cleanup has its
own try/catch so a transient DB error can never be misattributed as a
send failure or trigger a spurious retry. Delivery accounting stays
exact across attempts.
- resolveProvider caches the fallback provider under the requested id so
a topic full of targets pointing at one disabled/missing provider does
not re-query the providers collection once per page.
- The test Swoole channel polyfill enforces capacity (push returns false
when full) so a saturated semaphore surfaces Contention and the
scheduler defers the task, making the concurrency cap genuinely
bounded under the polyfill.
- ReflectionMethod::setAccessible(true) added for explicitness/back-compat.
Adds a regression test proving a thrown expired-token cleanup is
swallowed and the real provider failure is recorded instead.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Provider rate limiting and transient errors previously failed a whole
batch outright. Wrap the adapter send in a backoff/retry loop that
partitions per-recipient results: delivered recipients are summed and
never retried, retryable failures (throttle/rate-limit/quota/503/timeout
phrasing) are re-sent with only the still-failing identifiers in `to`,
and permanent failures stay terminal. Exponential backoff with jitter
runs on the non-blocking Swoole coroutine sleep so sibling sends keep
progressing; Retry-After is intentionally not honored to keep the
messaging library adapter-agnostic.
Accounting stays exact: recipients is the original batch size and
delivered is summed across attempts without double-counting.
Regression test for the enable-guard fix: PATCH {enabled:true} on an SES
provider whose fromEmail is still empty must return 400 provider_missing_credentials
rather than enabling a provider that would fail every send at the AWS API.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The enable-guard used array_key_exists, which is true even for empty-string
values — options['fromEmail'] is always stored on create, so a provider could
be enabled with an empty sender address. Use !empty() on the values in both the
create and update guards. Also corrects a misleading test comment.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wire the Amazon SES adapter shipped in utopia-php/messaging 1.0.0 into the
messaging service, mirroring the existing Sendgrid/Mailgun providers.
- Bump utopia-php/messaging 0.22.* -> 1.0.* (the 1.0.0 src is byte-identical
to 0.22.3 apart from the new SES adapter, so the bump is non-breaking)
- Add POST/PATCH /v1/messaging/providers/ses routes (createSesProvider /
updateSesProvider), storing accessKey/secretKey/region under credentials and
fromName/fromEmail/replyToName/replyToEmail under options, with the same
enable-guard and sparse-update semantics as the other email providers
- Map the 'ses' provider to the SES adapter in the messaging worker
- Add SES reference docs and isolated e2e create/update/delete coverage
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Large external sends (e.g. a 500K-recipient topic email) OOM'd the
messaging worker: sendExternalMessage read $topic->getAttribute('targets'),
triggering the subQueryTopicTargets filter that loads up to
APP_LIMIT_SUBSCRIBERS_SUBQUERY (1M) subscribers plus a giant targets IN()
into memory, then fanned every adapter-sized chunk out through a single
unbounded Swoole\Coroutine\batch() — spawning unbounded concurrent sends.
Rework the external send path into a single-job streaming model:
- Recipients are streamed in bounded pages (MESSAGE_RECIPIENTS_PAGE_SIZE)
via cursor pagination. Topics walk the subscribers collection by
topicInternalId and resolve each page's targets by $sequence, so the
topic's targets subquery is never materialized. Users/direct targets
paginate the targets collection. Peak memory is O(page), not O(topic).
- Send concurrency is bounded with Utopia\Lock\Semaphore
(MESSAGE_SEND_CONCURRENCY permits): each adapter batch send is wrapped in
$semaphore->withLock() and awaited per page with Swoole's native batch().
Excess coroutines block on acquire instead of all running at once.
- Results accumulate across pages: an exact deliveredTotal/failedTotal
partition (failed = recipients - delivered) and a delivery-errors list
capped at MESSAGE_DELIVERY_ERRORS_LIMIT, so a 500K send never collects
one error string per recipient.
- An idempotency guard skips messages already in a terminal status, so
queue redelivery does not reprocess a completed send.
Only existing message fields (status, deliveredTotal, deliveryErrors) are
written — no schema, migration, model, or queue-payload changes. Internal
SMS and other transactional paths are untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>