Lock backend errors (Redis/Dragonfly unreachable) and release errors
(TTL expired or backend dropped while held) were previously visible only
in the lock.attempts counter and Console::warning lines. They now also
push a structured Log entry through the configured logger adapter, so
operators using Sentry/Raygun/AppSignal/LogOwl get first-class events
for these specific failure modes.
Pattern matches Embeddings/Text/Create.php exactly:
- Action injects 'log' (per-request Log object) and 'logger'
(?Logger, nullable when _APP_LOGGING_CONFIG unset).
- Helper mutates the per-request $log instead of constructing a
fresh one — preserves the per-request context Embeddings expects.
- Same field set: namespace='http', server, version, type,
setMessage, setAction, setEnvironment, addTag('code', ...),
addExtra('file' / 'line' / 'trace').
- Defensive try/catch around addLog() so logging failures don't
break fail-open.
Lock-specific tags added for slicing in Sentry:
- lock.target — collection name (projects, keys, users, ...).
Bounded set, safe for high-cardinality stores.
- lock.key_pattern — full key with the trailing document ID
stripped (lock:platform:projects:* not lock:platform:projects:abc).
Prevents unbounded log cardinality from per-document IDs.
Rate limiting via per-pod static buckets, 60s window per
(action, target) combo. During a 5-minute Dragonfly outage, a fleet
of N pods produces at most N events/min, well within Sentry's dedup
tolerance. Static state is per-Swoole-worker; coroutines may race
on the bucket boundary but the worst case is one duplicate report.
Type level set to Log::TYPE_WARNING (not ERROR): fail-open means the
request still succeeds, so this is degraded operation, not a failed
request.
Deliberately NOT reported to Sentry:
- 409 GENERAL_RESOURCE_LOCKED (normal user-facing concurrency)
- skip-on-contention events (idempotent fan-out by design)
- acquire retry conflicts (internal loop)
- destructor cleanups (have an expected baseline rate; the
lock.attempts counter aggregates them better than Sentry would)
Factory signature change: distributedLock and distributedLockOrFail
now accept ?Log and ?Logger as optional named args at call time
(rather than capturing Logger at factory-build time). The factory
closure runs once at boot but the per-request Log resource is
fresh per request — capturing at boot would have given stale state.
Existing call sites threaded log: $log, logger: $logger. Sites that
don't (workers, CLI tasks) get null and just log to Console as
before.
Two complementary tests for the lost-update bug the lock fixes, plus
the test-Client patches needed to make Swoole-coroutine HTTP work.
- tests/e2e/Services/Project/ServicesBase.php :: testConcurrentTogglesAllPersist
Fires N parallel PATCHes via Swoole\Coroutine\run + SWOOLE_HOOK_CURL,
then refetches the project and asserts (successCount == enabledCount).
With the lock disabled (_APP_LOCKING_ENABLED=disabled) sparse
updateDocument() calls overwrite each other and the assertion fails —
proving the test detects the bug.
- dev/test-distributed-lock.sh
Same proof via curl + bash background jobs. Runnable outside the
PHPUnit suite for manual verification or load-style sweeps. Reads
APPWRITE_ENDPOINT / APPWRITE_PROJECT_ID / APPWRITE_API_KEY.
- tests/e2e/Client.php (test util):
* Skip CURLOPT_PATH_AS_IS (option 234) when SWOOLE_HOOK_CURL is
active. Swoole's emulated cURL doesn't support it; setting it
fatal-errors as soon as any test enables the cURL hook.
* Don't redundantly set CURLOPT_NOBODY=false on non-HEAD requests
(false is cURL's default). Under Swoole's hooked cURL this
strips the body of PATCH/PUT requests, hitting the framework's
404 fallback instead of the intended route.
Both changes preserve native (non-hooked) cURL behavior unchanged.
They unblock any future test that wants real parallel HTTP via
Swoole\Coroutine\run + the cURL hook.
Both follow the same proof-of-bug pattern: run with locking enabled
(must pass) AND with it disabled (must fail). Verified locally against
the running stack:
_APP_LOCKING_ENABLED=enabled -> PASS (15 assertions)
_APP_LOCKING_ENABLED=disabled -> FAIL (successCount=5 enabledCount=4)
Three P1 issues flagged on the initial commit:
1. Lock key in updateProjectService used "platform:project:{id}" —
missing the "lock:" namespace prefix and using singular "project"
instead of the conventional plural collection name. The factory's
`lockTargetOf` extracts segment [2] as the telemetry target, so
the broken key was emitting the project ID itself as the target
attribute (cardinality blowup, broken dashboards). Fixed to
"lock:platform:projects:{id}" matching the convention used in
shared/api.php.
2. The 409 contention exception embedded the raw Redis lock key in
its user-facing message, leaking internal collection names and
the locking namespace to API clients. Removed the custom message
so the catalog default ("The requested resource is currently
being modified...") is used. Telemetry already carries the
target collection for operator-side observability.
3. _APP_LOCKING_ENABLED variable doc had `introduction: '1.10.0'`
on a 1.9.x-targeted PR. Corrected to '1.9.3' (next 1.9.x patch).
Adds two DI factories and wires them where coordination is needed:
- distributedLock — skip on contention, void return. For idempotent
fan-out where N pods doing the same write is wasteful but losing
the race is correct.
- distributedLockOrFail — blocking acquire (3s default) then throws
GENERAL_RESOURCE_LOCKED (HTTP 409) on contention. For
read-modify-write on shared mutable state where a silent skip
would drop a user's change.
Both factories: _APP_LOCKING_ENABLED kill switch (set 'disabled' for
fail-open), fail-open on Redis-unreachable, and a lock.attempts
telemetry counter sliced by outcome and target collection.
Wired sites:
- shared/api.php × 3 (distributedLock): keys.accessedAt + sdks,
projects.accessedAt, users.accessedAt. Reduces redundant writes
and cache-purge fan-out under request bursts on the same project.
- Project/Services/Update.php × 1 (distributedLockOrFail): the
services map toggle. Re-reads inside the lock so the baseline
reflects concurrent updates. Two simultaneous toggles to
different services no longer lose one of them.
Lock key namespace: lock:platform:{collection}:{id}.
Dep: premtsd-code/lock pinned to a specific commit as a development
preview. Migration to utopia-php/lock is a follow-up once that
package is published.
A phpspy profile of a production databases worker showed the V20
backwards-compat request filter accounting for ~40% of in-request
samples on `databases.listDocuments` traffic. Two compounding causes:
1. `Request::getParams()` re-ran the entire filter chain on every
invocation. The framework and app call `getParams()` several times
per request (route param binding, `cacheIdentifier()`, action
injection, logging), so V20's recursive schema walk executed N times
with identical inputs.
2. Inside `V20::getRelatedCollectionKeys`, the `databases/$databaseId`
document was fetched at every recursion frame (up to
RELATION_MAX_DEPTH = 3), and sibling relationships pointing at the
same related collection each did their own `getDocument` call.
This commit:
- Memoizes the post-filter params on `Request`. The cache is
invalidated by `addFilter`, `resetFilters`, and `setRoute`. `Request`
is constructed per HTTP request (app/http.php), so the memo is
naturally request-scoped. Helps every request filter version, not
just V20.
- Splits V20's walk into an entry point that resolves the database
namespace once and a pure recursive helper.
- Caches the collection `attributes` array per
`(databaseNamespace, collectionId)` on the filter instance, so shared
related collections collapse to one `getDocument` call. Missing or
errored lookups are cached as `null` to avoid retry storms.
Prem Palanisamy
ArnabChatterjee20k
ArnabChatterjee20k
Matej Bačo
Matej Bačo
Luke B. Silver
Chirag Aggarwal