Drop the 18 lost-update endpoint locks (Project/* settings + Projects
team/update). Those address a different bug class (read-modify-write
races) than the manager-flagged production problem (regions slow queries
to platform from thundering herd on accessedAt writes).
Kept:
- distributedLock + distributedLockOrFail factories on the per-request
container, GENERAL_RESOURCE_LOCKED exception, 409 mapping
- 4 thundering-herd sites: cache-invalidation in shared/api.php (3) +
router projects.accessedAt in general.php (1)
Dropped:
- 18 endpoint OrFail wires
- testConcurrentTogglesAllPersist + Swoole-cURL test client patches
- dev/test-distributed-lock.sh smoke script
The previous shape required every caller to thread `log: $log, logger: $logger`
as named args into each `distributedLock(...)` invocation, plus inject `log`
and `logger` into the surrounding action just to forward them to the lock.
Across 21 call sites this added ~100 LOC of pure plumbing.
The cause: the lock factory was registered on the global container in
`app/init/resources.php`, where per-request resources like `log` aren't
visible. That forced the factory to expose its inner closure with optional
`?Log $log = null, ?Logger $logger = null` params, which every caller had
to satisfy.
Move the lock factory + its `lockErrorReporter`/`lockTargetOf` helpers from
the global container to the per-request container (`resources/request.php`),
and add `'log'` + `'logger'` to the factory's dep list. The factory closure
now runs per-request and closes over the per-request `Log`/`Logger`. Inner
closure returned to callers no longer needs the optional params, and call
sites drop the named args entirely.
Knock-on cleanup:
- Drop `->inject('log')`, `->inject('logger')`, the corresponding action
params, and `use Utopia\Logger\{Log,Logger}` imports from 19 endpoint
files where they were only there for the lock
- Drop the same plumbing from `app/controllers/shared/api.php` (3 lock call
sites)
- Drop just the Logger plumbing from `app/controllers/general.php` (router
function + 3 callbacks); `Log` is kept because it's used elsewhere in
that file
- Net 120 LOC removed across 23 files
No behavior change: the lock factories still produce the same closures
(skip-on-contention `distributedLock`, blocking-with-409 `distributedLockOrFail`).
The static lockErrorReporter rate limiter (1 push per 60s per
`(action, target)` bucket) continues to work — it lives on a closure-static
in the helper, which is independent of where the helper is constructed.
Verified end-to-end: testConcurrentTogglesAllPersist passes 4/5 (the cold-
start race flake is the same one we've consistently seen and is orthogonal
to lock changes).
Every request that arrives via a custom-domain rule (router path) reads
the project's `accessedAt` timestamp and, if the throttle window
(`APP_PROJECT_ACCESS`) has elapsed, writes a fresh value. With concurrent
traffic across multiple pods, this is a per-row hot RMW that loses
updates silently — the surviving timestamp depends on which pod's write
landed last.
Wrap the read-modify-write in `distributedLock('lock:platform:projects:{id}')`
(skip-on-contention variant). Every concurrent pod would write the same
throttled value, so losing the race is correct: the winner's update covers
ours.
Wires `distributedLock` and `?Logger` through:
- `router()` function signature (app/controllers/general.php:70)
- the three Http::init / Http::get callbacks that invoke router():
`*` catch-all (init), `/robots.txt`, `/humans.txt`
Two related cloud-only RMW sites (`teams.accessedAt`,
`projects.mcpAccessedAt`) live in `appwrite-labs/cloud` and need a
follow-up PR there. They depend on this branch reaching 1.9.x so the
`distributedLock` DI resource is available downstream.
The pilot on `Project/Services/Update.php` (commit fb0d43daf3) fixed the
silent lost-update on the `services` JSON map. The same read-modify-write
pattern exists on 17 other endpoints — all writing to the SAME `projects`
document, mostly via sparse `updateDocument`. Concurrent writes silently
overwrite each other, both within an endpoint (two toggles to different
auth methods) and across endpoints (parallel SMTP update + service toggle).
The worst offender is `Projects/Update.php` (`PATCH /v1/projects/:projectId`):
it calls `setAttribute` 11 times then writes the WHOLE document back, so a
concurrent write to *any* attribute on the project doc — services, auths,
smtp, templates, mockNumbers — gets clobbered.
Wrap every read-modify-write window with `distributedLockOrFail` keyed on
`lock:platform:projects:{$projectId}`. The single shared key serializes
both same-endpoint and cross-endpoint races on the same project.
Endpoints now under the lock:
Projects module (console-managed, plural):
- Projects/Update (full doc setAttribute chain)
- Projects/Team/Update (sparse + computed permissions; cascade
to installations/repositories/vcsComments
runs after release — separate collections)
Project module (current-project, singular):
- AuthMethods/Update (auths[authKey])
- Protocols/Update (apis[protocolId])
- SMTP/Update (smtp map; PHPMailer probe inside lock —
10s default TTL covers Timeout=5)
- Templates/Email/Update (templates['email.X-locale'])
- MockPhone/{Create,Delete,Update} (auths['mockNumbers'][])
- Policies/MembershipPrivacy (auths['memberships*'])
- Policies/PasswordDictionary (auths['passwordDictionary'])
- Policies/PasswordHistory (auths['passwordHistory'])
- Policies/PasswordPersonalData (auths['personalDataCheck'])
- Policies/SessionAlert (auths['sessionAlerts'])
- Policies/SessionDuration (auths['duration'])
- Policies/SessionInvalidation (auths['invalidateSessions'])
- Policies/SessionLimit (auths['maxSessions'])
- Policies/UserLimit (auths['limit'])
Each endpoint:
1. Injects `distributedLockOrFail`, `log`, `logger`
2. Re-reads the project document inside the lock so the baseline reflects
any update that landed between request init and lock acquisition
3. Throws `GENERAL_RESOURCE_LOCKED` (HTTP 409) on contention timeout
SMTP/Update and Templates/Email/Update additionally refactor variable-
variables (`${$key}`) to an explicit input array — PHPStan can't trace
dynamic references through closure `use` clauses.
`utopia-php/lock` v0.2.0 was published this week and provides the same
Redis SET-NX-EX + Lua-compare-and-delete primitive we built locally as
`premtsd-code/lock`. Drop the dev-preview package in favor of the
official Utopia PHP library.
- composer: replace `premtsd-code/lock` with `utopia-php/lock` 0.2.*
(still via VCS — not on Packagist yet)
- resources.php: rewire both factory variants
- `Lock + Adapter\Redis` → `Distributed`
- `acquire()` → `tryAcquire()` for skip variant
- `acquire(blocking: true, waitTimeout)` → `acquire($waitTimeout)` for
OrFail variant
- `LockAcquireException` → `\RedisException`
- `(int) $ttl` cast — utopia-php/lock takes seconds as int
- docker-compose: thread `_APP_LOCKING_ENABLED` into the appwrite
service environment so the kill switch documented in
`app/config/variables.php` is actually usable from `.env`
Verified end-to-end on local stack:
- positive case (locking enabled): 5/5 testConcurrentTogglesAllPersist
pass, lock keys observed in `redis-cli MONITOR` with concurrent SET
NX contention
- negative case (locking disabled): 1/3 detect lost updates as before
Remove query param fallback for impersonateEmail and impersonatePhone
to avoid PII exposure in server logs, browser history, and Referer
headers. Only impersonateUserId (an opaque internal ID) is safe to
pass via URL query param.
Allow impersonation to be specified via URL query params
(?impersonateUserId, ?impersonateEmail, ?impersonatePhone) as a
fallback to the existing headers, enabling Console to embed
impersonation in direct file/image URLs where headers cannot be set.
Lock backend errors (Redis/Dragonfly unreachable) and release errors
(TTL expired or backend dropped while held) were previously visible only
in the lock.attempts counter and Console::warning lines. They now also
push a structured Log entry through the configured logger adapter, so
operators using Sentry/Raygun/AppSignal/LogOwl get first-class events
for these specific failure modes.
Pattern matches Embeddings/Text/Create.php exactly:
- Action injects 'log' (per-request Log object) and 'logger'
(?Logger, nullable when _APP_LOGGING_CONFIG unset).
- Helper mutates the per-request $log instead of constructing a
fresh one — preserves the per-request context Embeddings expects.
- Same field set: namespace='http', server, version, type,
setMessage, setAction, setEnvironment, addTag('code', ...),
addExtra('file' / 'line' / 'trace').
- Defensive try/catch around addLog() so logging failures don't
break fail-open.
Lock-specific tags added for slicing in Sentry:
- lock.target — collection name (projects, keys, users, ...).
Bounded set, safe for high-cardinality stores.
- lock.key_pattern — full key with the trailing document ID
stripped (lock:platform:projects:* not lock:platform:projects:abc).
Prevents unbounded log cardinality from per-document IDs.
Rate limiting via per-pod static buckets, 60s window per
(action, target) combo. During a 5-minute Dragonfly outage, a fleet
of N pods produces at most N events/min, well within Sentry's dedup
tolerance. Static state is per-Swoole-worker; coroutines may race
on the bucket boundary but the worst case is one duplicate report.
Type level set to Log::TYPE_WARNING (not ERROR): fail-open means the
request still succeeds, so this is degraded operation, not a failed
request.
Deliberately NOT reported to Sentry:
- 409 GENERAL_RESOURCE_LOCKED (normal user-facing concurrency)
- skip-on-contention events (idempotent fan-out by design)
- acquire retry conflicts (internal loop)
- destructor cleanups (have an expected baseline rate; the
lock.attempts counter aggregates them better than Sentry would)
Factory signature change: distributedLock and distributedLockOrFail
now accept ?Log and ?Logger as optional named args at call time
(rather than capturing Logger at factory-build time). The factory
closure runs once at boot but the per-request Log resource is
fresh per request — capturing at boot would have given stale state.
Existing call sites threaded log: $log, logger: $logger. Sites that
don't (workers, CLI tasks) get null and just log to Console as
before.
Two complementary tests for the lost-update bug the lock fixes, plus
the test-Client patches needed to make Swoole-coroutine HTTP work.
- tests/e2e/Services/Project/ServicesBase.php :: testConcurrentTogglesAllPersist
Fires N parallel PATCHes via Swoole\Coroutine\run + SWOOLE_HOOK_CURL,
then refetches the project and asserts (successCount == enabledCount).
With the lock disabled (_APP_LOCKING_ENABLED=disabled) sparse
updateDocument() calls overwrite each other and the assertion fails —
proving the test detects the bug.
- dev/test-distributed-lock.sh
Same proof via curl + bash background jobs. Runnable outside the
PHPUnit suite for manual verification or load-style sweeps. Reads
APPWRITE_ENDPOINT / APPWRITE_PROJECT_ID / APPWRITE_API_KEY.
- tests/e2e/Client.php (test util):
* Skip CURLOPT_PATH_AS_IS (option 234) when SWOOLE_HOOK_CURL is
active. Swoole's emulated cURL doesn't support it; setting it
fatal-errors as soon as any test enables the cURL hook.
* Don't redundantly set CURLOPT_NOBODY=false on non-HEAD requests
(false is cURL's default). Under Swoole's hooked cURL this
strips the body of PATCH/PUT requests, hitting the framework's
404 fallback instead of the intended route.
Both changes preserve native (non-hooked) cURL behavior unchanged.
They unblock any future test that wants real parallel HTTP via
Swoole\Coroutine\run + the cURL hook.
Both follow the same proof-of-bug pattern: run with locking enabled
(must pass) AND with it disabled (must fail). Verified locally against
the running stack:
_APP_LOCKING_ENABLED=enabled -> PASS (15 assertions)
_APP_LOCKING_ENABLED=disabled -> FAIL (successCount=5 enabledCount=4)
Three P1 issues flagged on the initial commit:
1. Lock key in updateProjectService used "platform:project:{id}" —
missing the "lock:" namespace prefix and using singular "project"
instead of the conventional plural collection name. The factory's
`lockTargetOf` extracts segment [2] as the telemetry target, so
the broken key was emitting the project ID itself as the target
attribute (cardinality blowup, broken dashboards). Fixed to
"lock:platform:projects:{id}" matching the convention used in
shared/api.php.
2. The 409 contention exception embedded the raw Redis lock key in
its user-facing message, leaking internal collection names and
the locking namespace to API clients. Removed the custom message
so the catalog default ("The requested resource is currently
being modified...") is used. Telemetry already carries the
target collection for operator-side observability.
3. _APP_LOCKING_ENABLED variable doc had `introduction: '1.10.0'`
on a 1.9.x-targeted PR. Corrected to '1.9.3' (next 1.9.x patch).
Prem Palanisamy
Harsh Mahajan
harsh mahajan
Matej Bačo
Matej Bačo
Damodar Lohani
Damodar Lohani