Commit Graph
35500 Commits
Author SHA1 Message Date
Prem Palanisamy 57d21e283d Relax migration smoke status counters assertion 2026-06-03 09:57:07 +01:00
Prem Palanisamy a0256d7cbf Expose project blocks and console access time 2026-06-03 09:46:27 +01:00
Prem Palanisamy 2bc1fbedd7 Expose project region in response model 2026-06-03 09:34:23 +01:00
Prem Palanisamy e80ad301f6 Use released migration dependencies 2026-06-03 08:34:31 +01:00
Prem Palanisamy 46dc30db73 Merge remote-tracking branch 'origin/1.9.x' into add-api-key-migration
# Conflicts:
#	composer.json
#	composer.lock
2026-06-03 08:21:08 +01:00
Damodar Lohani 20f579d6d4 Merge pull request #12473 from appwrite/chore/upgrade-dbip-2026-06
Upgrade DB-IP Country Lite database to 2026-06
2026-06-03 10:58:54 +05:45
Damodar LohaniandClaude Opus 4.8 79a617eb4b Upgrade DB-IP Country Lite database to 2026-06
Bump the bundled free-tier DB-IP IP-to-Country Lite database from the
2025-12 release to the latest available 2026-06 release, and update the
geodb register path accordingly.

Source: https://db-ip.com/db/download/ip-to-country (CC-BY-4.0)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 04:36:06 +00:00
Chirag Aggarwal 6e600bad6b Enable RunsOn OTEL for E2E jobs (#12472) 2026-06-03 08:14:29 +05:30
Jake Barnby a6ead889ac Merge pull request #12465 from appwrite/feat-messaging-scale-fanout-19x
perf(messaging): stream large-topic sends and bound fan-out concurrency
2026-06-03 14:36:54 +12:00
Jake Barnby 78e858136b Merge branch '1.9.x' into feat-messaging-scale-fanout-19x 2026-06-03 13:45:14 +12:00
Jake BarnbyandClaude Opus 4.8 2084b21ae3 perf(messaging): raise recipient page size to 5000 to cut DB round-trips
Streaming a large-topic send paginated subscribers 1000 at a time, so a
500K-recipient topic issued ~500 subscriber-page queries plus their per-page
target resolutions. Raise MESSAGE_RECIPIENTS_PAGE_SIZE to 5000 (APP_LIMIT_COUNT),
cutting topic-path round-trips ~5x with no extra memory pressure (bounded
concurrency still caps in-flight sends). The streaming test fixture now sizes
itself to the page-size constant so it keeps exercising multi-page pagination.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 13:41:19 +12:00
Jake BarnbyandClaude Opus 4.8 58d7ae9194 test(messaging): rewrite fan-out tests as coroutine-free unit tests
The Fiber-based SwooleCoroutinePolyfill shadowed real Swoole symbols and
poisoned PHPStan, and driving the worker through real Swoole coroutines
(Co\run/batch) inside the shared unit process contaminated and crashed
later, unrelated tests. The native extension is now available, so delete
the polyfill (and its PHPStan exclude) and test the worker's logic by
invoking its component methods directly with zero coroutines.

Reflection invokes the private methods without setAccessible (deprecated
on PHP 8.5), and a TestableMessaging subclass zeroes retryDelay so the
backoff never sleeps. Coverage: isRetryableError classification,
retrySend (throttle-then-success, partial-throttle scoping, non-retryable
terminal, exhausted retries, error-cap with exact delivered accounting,
isolated expired-token cleanup failure) and streamRecipients (cursor
pagination by topicInternalId, no targets-subquery read, no runaway
limit, per-page dedup, user/direct-target resolution). The bounded-
concurrency Semaphore + batch() integration stays covered by the e2e
Messaging suite, where executing real coroutines is safe.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 12:42:32 +12:00
Luke B. Silver 61e189fb4e Merge pull request #12468 from appwrite/feat/coroutine-workers
Add coroutine concurrency to queue workers
2026-06-02 22:58:49 +01:00
loks0nandClaude Opus 4.8 78769b7f13 Lease one broker connection per consume coroutine instead of sharing one
The Locking decorator serialized every Redis command on the single
consumer connection shared by all consume coroutines. On an idle queue
each waiting coroutine holds the mutex for a full 2s blocking pop, so a
finished job's completion ops (remove job key, increment stats) starve
behind the poppers: a 10ms cache purge took 22-24s to ack, and Storage's
testFilePreviewCache observed stale previews because purges landed tens
of seconds after enqueue.

Pass a Broker\Pool as the adapter consumer instead: each consume
coroutine leases its own connection from the consumer pool for the
lifetime of its loop, eliminating the shared-connection mutex entirely.
The Locking wrapper in the pool factory is reverted, and consumer pools
get +2 headroom above the coroutine count for queue-size telemetry
reads. Purge ack latency drops to 4-58ms.

Known tradeoff: Broker\Pool::close() is a no-op, so SIGTERM no longer
breaks the blocking pop early; workers rely on the process kill that
already follows.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-02 22:37:37 +01:00
loks0nandClaude Opus 4.8 4baaa7d93f Fix coroutine starvation: scale pools to coroutine count, serialize SMTP workers
Two CI failure causes:

- Connection pools are sized for HTTP-server worker counts and came out
  at 1 per worker process; 8 consume coroutines exhausted them
  (Pool 'cache_redis_main' is empty). Floor the pool size at
  _APP_WORKER_MAX_COROUTINES.

- The mails and messaging workers disable the Swoole TCP hook at
  runtime to work around the TLS handshake crash (swoole-src#4909,
  #7914). The flag is process-global, so after the first job every
  blocking Redis pop freezes the coroutine scheduler and deliveries
  arrive minutes late. Both workers stay at 1 coroutine, matching
  pre-coroutine behavior.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-02 22:03:10 +01:00
loks0nandClaude Opus 4.8 b53268a4dd Remove functions load test harness
The throughput numbers live in the PR; the harness itself is not
CI material.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-02 21:40:48 +01:00
loks0nandClaude Opus 4.8 7889a1448f Serialize workers with shared accumulator state; fix Messaging local device cache
Coroutine-safety audit of all worker actions (instance state mutated
during action() is shared across consume coroutines):

- audits: batches logs across messages ($logs/$lastTriggeredTime) with
  Commit/NoCommit ack semantics tied to the shared batch; concurrent
  flushes lose or duplicate audit entries
- webhooks: $errors buffer is reset per job and drives the
  retry/disable path; concurrent jobs wipe or steal each other's errors
- migrations: project DB callables are rebound on the instance per
  message; a concurrent message swaps database handles mid-migration

Messaging cached a project-scoped Local device unkeyed, reusing the
first project's storage path for every subsequent message; create it
per call instead.

Deletes, mails, certificates, executions, functions, builds and
screenshots keep all per-message state in locals and stay at 8.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-02 21:38:26 +01:00
loks0nandClaude Opus 4.8 ead860457c Keep stats workers serial: shared accumulator state is not coroutine-safe
StatsUsage batches metrics across messages on shared instance state
($stats/$keys/$projects/$statDocuments); concurrent consume coroutines
race the flush in commitToDB, losing aggregates on reset and
double-applying upsertDocumentsWithIncrease on overlapping flushes.

StatsResources resets its shared $documents buffer at the start of each
job, so a second coroutine starting mid-count wipes the first one's
batch and the projects' metrics interleave into one write.

Both stay at 1 coroutine until accumulation moves to per-message state.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-02 21:35:08 +01:00
loks0nandClaude Opus 4.8 e3afc66bac Run all workers at 8 coroutines except databases
Only the databases worker is ordering-sensitive (schema operations
must apply in sequence); every other worker handles independent jobs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-02 21:33:16 +01:00
loks0nandClaude Opus 4.8 2e49713194 Set explicit per-worker process and coroutine counts in compose
Workers run a single process (_APP_WORKERS_NUM=1) with concurrency
provided by coroutines instead. I/O-bound workers get 8 coroutines;
CPU-heavy, stateful, or ordering-sensitive workers (databases, builds,
migrations, stats-resources, stats-usage) stay serial at 1.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-02 21:31:17 +01:00
loks0nandClaude Opus 4.8 2191a332ef Add coroutine concurrency to queue workers
Upgrade utopia-php/queue to 0.19.0 and utopia-php/platform to 1.0.0-rc4,
enabling workers to consume multiple jobs concurrently via Swoole
coroutines within a single worker process.

- Workers pass a maxCoroutines option to the queue adapter, configured
  via the new _APP_WORKER_MAX_COROUTINES env var (default 1)
- Per-message DI resources now live in a coroutine-scoped context
  container (Server::context()) instead of the shared container
- Consumer Redis connections are wrapped in the Locking decorator so
  concurrent coroutines can safely share one connection
- Add a functions worker load test: 30 async 1s executions drain in
  31.1s at 1 coroutine, 10.4s at 5, 7.3s at 10

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-02 21:23:31 +01:00
Jake BarnbyandClaude Opus 4.8 977cbd41da chore(messaging): exclude Swoole coroutine polyfill from PHPStan
The test polyfill declares stand-in Swoole\Coroutine and Swoole\Coroutine\Channel
classes behind a runtime extension check. PHPStan ignores the guard and parsed
those minimal declarations, shadowing the real swoole/ide-helper stubs and
reporting false 'undefined static method' errors codebase-wide (e.g. getContext,
join, create, Channel::close in app/realtime.php). Exclude the polyfill from
analysis so the ide-helper stubs are used. Verified: Checks/Analyze clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 04:58:52 +12:00
Jake BarnbyandClaude Opus 4.8 c5551258fe test(messaging): run fan-out coroutine tests only without native Swoole
Tests/Unit runs in the Appwrite container (PHP 8.5) with the native Swoole
extension, where these tests' real Co\run()/batch() scheduler runs in-process
and destabilises later coroutine-sensitive tests in the shared unit suite
(the run aborted mid-suite at ~126/644 inside the Network/Fetcher tests).
Skip them under native Swoole; they still run locally via the Fiber polyfill,
and the real worker path is covered by the e2e Messaging suite.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 04:45:32 +12:00
Torsten Dittmann 3927d52be1 Merge pull request #12466 from appwrite/fix/storage-chunked-upload-permissions 2026-06-02 20:43:12 +04:00
Jake BarnbyandClaude Opus 4.8 e2278a7370 test(messaging): drop deprecated ReflectionMethod::setAccessible call
The CI unit-test job runs inside the Appwrite container on PHP 8.5, where
ReflectionMethod::setAccessible() is deprecated, and the suite fails on
triggered deprecations. It is also a no-op on PHP 8.1+ (reflection invokes
private methods without it), so it was both unnecessary and the cause of the
red Tests/Unit check. Reverts the setAccessible addition from 7c8d7dcb80.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 04:35:12 +12:00
Torsten Dittmann 2d7f3d4cba test(storage): guard final chunk upload assertion 2026-06-02 20:22:44 +04:00
Jake BarnbyandClaude Opus 4.8 7247b0effd fix(messaging): skip non-positive retry backoff sleep
Swoole\Coroutine::sleep() rejects 0/negative values (CI runs unit tests with
the real Swoole extension, where the Fiber test polyfill is inert). The retry
tests override the base delay to 0 for speed, which made sleep(0.0) abort the
process under real Swoole. Guard the call; production delays are always >= 1s.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 04:22:36 +12:00
Torsten Dittmann 06c722e1e4 test(storage): cover final chunk upload without read permission 2026-06-02 20:16:57 +04:00
Jake BarnbyandClaude Opus 4.8 7c8d7dcb80 fix(messaging): address fanout scale review comments
Apply five reviewer fixes to the large-topic send path without touching
the adapter-agnostic messaging library:

- Empty recipient pages no longer count as recipients: a page that
  resolves to zero identifiers is skipped instead of running batch([])
  and masking the "No valid recipients found." outcome.
- Best-effort expired-device-token cleanup is isolated from the provider
  send: the send try/catch wraps only adapter->send(), results are
  processed outside it, and the findOne/updateDocument cleanup has its
  own try/catch so a transient DB error can never be misattributed as a
  send failure or trigger a spurious retry. Delivery accounting stays
  exact across attempts.
- resolveProvider caches the fallback provider under the requested id so
  a topic full of targets pointing at one disabled/missing provider does
  not re-query the providers collection once per page.
- The test Swoole channel polyfill enforces capacity (push returns false
  when full) so a saturated semaphore surfaces Contention and the
  scheduler defers the task, making the concurrency cap genuinely
  bounded under the polyfill.
- ReflectionMethod::setAccessible(true) added for explicitness/back-compat.

Adds a regression test proving a thrown expired-token cleanup is
swallowed and the real provider failure is recorded instead.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 04:13:49 +12:00
Torsten Dittmann ccd1a073b6 fix(storage): resume chunked uploads without file read permission 2026-06-02 20:08:26 +04:00
Jake Barnby 2cd1abc273 feat(messaging): retry throttled recipients with backoff in send path
Provider rate limiting and transient errors previously failed a whole
batch outright. Wrap the adapter send in a backoff/retry loop that
partitions per-recipient results: delivered recipients are summed and
never retried, retryable failures (throttle/rate-limit/quota/503/timeout
phrasing) are re-sent with only the still-failing identifiers in `to`,
and permanent failures stay terminal. Exponential backoff with jitter
runs on the non-blocking Swoole coroutine sleep so sibling sends keep
progressing; Retry-After is intentionally not honored to keep the
messaging library adapter-agnostic.

Accounting stays exact: recipients is the original batch size and
delivered is summed across attempts without double-counting.
2026-06-03 03:57:39 +12:00
Jake Barnby c9fc025500 Merge pull request #12464 from appwrite/feat-ses-support
feat(messaging): add Amazon SES email provider
2026-06-03 03:42:13 +12:00
Jake BarnbyandClaude Opus 4.8 5e07e9fc6e test(messaging): cover enabling SES provider with empty fromEmail
Regression test for the enable-guard fix: PATCH {enabled:true} on an SES
provider whose fromEmail is still empty must return 400 provider_missing_credentials
rather than enabling a provider that would fail every send at the AWS API.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 03:23:26 +12:00
Jake BarnbyandClaude Opus 4.8 1bfb0b115c fix(messaging): require non-empty SES fromEmail and credentials to enable
The enable-guard used array_key_exists, which is true even for empty-string
values — options['fromEmail'] is always stored on create, so a provider could
be enabled with an empty sender address. Use !empty() on the values in both the
create and update guards. Also corrects a misleading test comment.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 03:20:17 +12:00
Jake BarnbyandClaude Opus 4.8 b28633086b feat(messaging): add Amazon SES email provider
Wire the Amazon SES adapter shipped in utopia-php/messaging 1.0.0 into the
messaging service, mirroring the existing Sendgrid/Mailgun providers.

- Bump utopia-php/messaging 0.22.* -> 1.0.* (the 1.0.0 src is byte-identical
  to 0.22.3 apart from the new SES adapter, so the bump is non-breaking)
- Add POST/PATCH /v1/messaging/providers/ses routes (createSesProvider /
  updateSesProvider), storing accessKey/secretKey/region under credentials and
  fromName/fromEmail/replyToName/replyToEmail under options, with the same
  enable-guard and sparse-update semantics as the other email providers
- Map the 'ses' provider to the SES adapter in the messaging worker
- Add SES reference docs and isolated e2e create/update/delete coverage

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 03:09:45 +12:00
Jake BarnbyandClaude Opus 4.8 da061ba7d2 fix(messaging): stream large-topic sends and bound send concurrency
Large external sends (e.g. a 500K-recipient topic email) OOM'd the
messaging worker: sendExternalMessage read $topic->getAttribute('targets'),
triggering the subQueryTopicTargets filter that loads up to
APP_LIMIT_SUBSCRIBERS_SUBQUERY (1M) subscribers plus a giant targets IN()
into memory, then fanned every adapter-sized chunk out through a single
unbounded Swoole\Coroutine\batch() — spawning unbounded concurrent sends.

Rework the external send path into a single-job streaming model:

- Recipients are streamed in bounded pages (MESSAGE_RECIPIENTS_PAGE_SIZE)
  via cursor pagination. Topics walk the subscribers collection by
  topicInternalId and resolve each page's targets by $sequence, so the
  topic's targets subquery is never materialized. Users/direct targets
  paginate the targets collection. Peak memory is O(page), not O(topic).
- Send concurrency is bounded with Utopia\Lock\Semaphore
  (MESSAGE_SEND_CONCURRENCY permits): each adapter batch send is wrapped in
  $semaphore->withLock() and awaited per page with Swoole's native batch().
  Excess coroutines block on acquire instead of all running at once.
- Results accumulate across pages: an exact deliveredTotal/failedTotal
  partition (failed = recipients - delivered) and a delivery-errors list
  capped at MESSAGE_DELIVERY_ERRORS_LIMIT, so a 500K send never collects
  one error string per recipient.
- An idempotency guard skips messages already in a terminal status, so
  queue redelivery does not reprocess a completed send.

Only existing message fields (status, deliveredTotal, deliveryErrors) are
written — no schema, migration, model, or queue-payload changes. Internal
SMS and other transactional paths are untouched.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 02:10:47 +12:00
Jake Barnby 69b49c7228 Merge pull request #12427 from appwrite/appwrite-embeddings
refactor: update embedding model and endpoint to use 'nomic' and 'app…
2026-06-02 20:34:21 +12:00
ArnabChatterjee20k 32759377b1 updated composer lock 2026-06-02 13:30:17 +05:30
Jake Barnby d24679ab08 Merge branch '1.9.x' into appwrite-embeddings 2026-06-02 19:06:50 +12:00
Chirag Aggarwal ab9052d52d Merge pull request #12459 from appwrite/fix/telemetry-04-deps
Update telemetry dependency stack
2026-06-02 10:45:35 +05:30
Chirag Aggarwal 4fad6b781a Update telemetry dependency stack 2026-06-02 09:53:08 +05:30
Damodar Lohani f6e293874b Merge pull request #12456 from appwrite/fix-chunked-upload-bucketid-missing-event
fix: reset queued event on non-final chunk in createFile
2026-06-02 08:26:01 +05:45
Damodar LohaniandClaude Opus 4.8 47e2ccd96a fix: reset queued event on non-final chunk in createFile
On chunked uploads, the createFile route registers a templated event
label `buckets.[bucketId].files.[fileId].create` during init. On an
intermediate (non-final) chunk the request returns 201 but the event
params are never set, so App::shutdown() calls Event::generateEvents()
with an unpopulated [bucketId] template and throws "[bucketId] is
missing from the params".

Reset the queued event in the non-final-chunk branch so no templated
event is generated until the upload completes, matching the reset()
already used by other early-exit paths in the same closure.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-02 02:24:30 +00:00
Luke B. Silver db59f5d338 Merge pull request #12455 from appwrite/chore/remove-pgetparamsvalues
refactor: decompose the shared API shutdown hook + drop deprecated Route::getParamsValues()
2026-06-01 20:57:08 +01:00
loks0nandClaude Opus 4.8 f0192c6561 Split abuse and audit flows into their own shutdown hooks
Break the remaining shared 'api' shutdown hook into focused hooks: one for
event/realtime/function/webhook dispatch, one for abuse-reset handling, and
one for audit logging. Each injects only what it uses, so the wide
shared injection list is gone.

Registered in the original execution order (events -> abuse -> audit ->
cache -> usage) so behaviour is unchanged; the pre-existing 'session'
group shutdown hook is unaffected. Applied early-return guards to the
events and abuse hooks to flatten their nesting.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-01 17:15:29 +01:00
loks0nandClaude Opus 4.8 2e5a29c650 Extract request-completed/usage into its own shutdown hook; tighten cache hook
Move the RequestCompleted dispatch and usage-metrics publishing out of the
large shared shutdown hook into a dedicated Http::shutdown() hook that
injects only what it needs (request, response, project, user, usage,
publisherForUsage, authorization, bus, apiKey). Drop the now-unused
usage/publisherForUsage/bus/apiKey/authorization injections from the main
hook. Registered after the cache hook so cache writes keep happening
before usage publishing, as in the original single-hook ordering.

Also tighten the cache hook with guard clauses: hoist the cache/key/
signature setup, make the create path save-and-return explicitly instead
of relying on the empty-document signature mismatch, resolve the resource
labels lazily inside the create branch (so the param reconstruction and
label parsing only run on a cache miss, not on every cached hit), and
collapse the trailing save into the existing-entry path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-01 16:56:46 +01:00
loks0nandClaude Opus 4.8 c9059c75a4 Use injected 'params' directly for audit resource labels
audits.resource labels only reference URL path params (resource IDs), so
the audit code can use the injected 'params' local directly instead of
reconstructing the full resolved param set. The path+query/body+default
reconstruction now lives only in the cache hook, where the screenshot
cache.resource label genuinely needs query params.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-01 16:46:17 +01:00
loks0nandClaude Opus 4.8 cd0936c721 Extract response caching into its own shutdown hook
Move the cache.resource / cache.resourceType handling out of the large
shared shutdown hook into a dedicated Http::shutdown() hook in the 'api'
group. It injects only what caching needs (route, params, request,
response, project, user, dbForProject, authorization) and guards with
early returns, leaving the main hook focused on events, audit, and usage.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-01 16:42:39 +01:00
loks0nandClaude Opus 4.8 0bf406e597 Tighten param reconstruction
Use Request::getParam() for the query/body+default fallback instead of a
cached getParams() copy, dropping the temporary and shortening the diff.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-01 16:18:00 +01:00
loks0nandClaude Opus 4.8 ad7a76abbb Use injected 'params' local instead of getPathValues reconstruction
The http server exposes the route's path values as an injectable 'params'
local, so inject it directly rather than rebuilding them via
Router::preparePath() + Route::getPathValues(). Query/body params and
declared defaults are still merged in to reproduce the full resolved
param set (path values take precedence), matching prior getParamsValues()
behaviour.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-01 16:07:27 +01:00