Two new E2E tests exercising the schema-tolerance UpdateInPlace path
added in utopia-php/migration's DestinationAppwrite.
testAppwriteMigrationUpsertUpdatesContainerMetadata (positive):
- Fresh migration copies source database + table + column + row to dest.
- Mutates source database name (PUT /databases/:id) and table
name/permissions/rowSecurity/enabled (PUT /tablesdb/:db/tables/:id).
- One-second sleep before mutation ensures source's $updatedAt is
strictly greater than dest's at second granularity (strtotime
comparison).
- Upsert re-migration asserts:
- 'completed' status.
- dest database name matches source's new name.
- dest table name / enabled / rowSecurity match source's new values.
- child row's 'name' attribute is untouched — UpdateInPlace only
rewrites container metadata, not rows.
testAppwriteMigrationSkipPreservesContainerDrift (negative):
- Fresh migration, then mutate BOTH dest (simulating ops tightening
permissions post-migration) and source (divergence).
- Skip re-migration asserts dest kept its tightened values — Skip's
strict "don't touch" contract protects dev→prod cutover workflows
from accidentally wiping ops-side drift on schema re-sync.
Both tests use performMigrationSync for strict 'completed' assertions.
Runtime ~18s combined. Existing testAppwriteMigrationRowsOnDuplicate
and testAppwriteMigrationReRunIsIdempotent regression-tested locally.
Picks up the UpdateInPlace branch — database/table metadata drift is
now reconciled on Upsert-newer (renames, enable toggles, table
permissions / documentSecurity) via updateDocument, without touching
child rows.
Picks up the PR #171 refactor + unit tests:
- resolveSchemaAction decision point consolidation
- deleteAttributeCompletely primitive (two-way cleanup in one place)
- Hardened sourceIsNewer against MySQL zero-date sentinel
- 14 unit tests locking the decision matrix
Branch is iterating on utopia-php/migration's re-migration tolerance.
Other test matrices (unit, general, abuse, screenshots, benchmark, and
every other e2e service) add ~30+ minutes to CI without exercising code
this PR touches. Restrict the matrix to the Migrations service and skip
the unrelated test jobs until the migration work is ready to merge.
All jobs marked with 'TEMP:' comments + 'if: false' — revert to the
full matrix before merging to main.
Static analysis (lint, phpstan, composer audit, specs, locale, security)
still runs on every PR push.
utopia-php/migration's DestinationAppwrite now handles schema tolerance on
re-migration (PR #171 on feat/skip-duplicates): it pre-checks destination
`_metadata` for each database / table / column / index and tolerates in
Skip/Upsert mode. Re-runs no longer produce schema-level errors, so the
E2E tests can drop the status-tolerant workaround and assert strict
'completed' outcomes.
Changes:
- composer.json: pin utopia-php/migration to dev-feat/skip-duplicates (aliased
to 1.9.99 for stability resolution). Will be replaced with a fixed 1.10.0
tag once the migration PR lands.
- testAppwriteMigrationRowsOnDuplicate: replace the tolerant
runMigrationAssertingRowSuccess helper with performMigrationSync on the
Skip and Upsert re-runs. Asserts 'completed' status on every run,
destination row content matches the expected value per mode (Mutated
preserved on Skip, Original restored on Upsert). Helper method removed.
- testAppwriteMigrationReRunIsIdempotent (new): seeds two rows on source,
runs the migration three times back-to-back (fresh, Skip re-run, Upsert
re-run) against unchanged source data, asserts strict 'completed' on every
run and row content is stable across all three. Exercises the
schema-tolerance path end-to-end: every database/table/column on
destination already exists with a matching spec, so DestinationAppwrite's
pre-check returns Tolerate for every resource.
A phpspy profile of a production databases worker showed the V20
backwards-compat request filter accounting for ~40% of in-request
samples on `databases.listDocuments` traffic. Two compounding causes:
1. `Request::getParams()` re-ran the entire filter chain on every
invocation. The framework and app call `getParams()` several times
per request (route param binding, `cacheIdentifier()`, action
injection, logging), so V20's recursive schema walk executed N times
with identical inputs.
2. Inside `V20::getRelatedCollectionKeys`, the `databases/$databaseId`
document was fetched at every recursion frame (up to
RELATION_MAX_DEPTH = 3), and sibling relationships pointing at the
same related collection each did their own `getDocument` call.
This commit:
- Memoizes the post-filter params on `Request`. The cache is
invalidated by `addFilter`, `resetFilters`, and `setRoute`. `Request`
is constructed per HTTP request (app/http.php), so the memo is
naturally request-scoped. Helps every request filter version, not
just V20.
- Splits V20's walk into an entry point that resolves the database
namespace once and a pure recursive helper.
- Caches the collection `attributes` array per
`(databaseNamespace, collectionId)` on the filter instance, so shared
related collections collapse to one `getDocument` call. Missing or
errored lookups are cached as `null` to avoid retry storms.
- Updated span logging keys to use camelCase for uniformity across connection and message events.
- Added checks to ensure project and user IDs are only logged if they are not empty, enhancing data integrity.
- Improved error handling and logging structure to maintain consistency in telemetry data.
- Introduced new arrays to capture subscribed channels and passed queries during connection and message events.
- Enhanced span logging to include details about channels and queries for better monitoring and analysis.
- Updated telemetry data structure to reflect the new metrics, improving traceability of realtime interactions.
- Introduced span logging for connection open and close events, capturing metrics such as inbound and outbound bytes, subscription counts, and response codes.
- Enhanced error handling with logging of exceptions during connection lifecycle.
- Updated the structure of the telemetry data to include project and user IDs for better traceability.
Matej Bačo
Matej Bačo
Luke B. Silver
Prem Palanisamy
Chirag Aggarwal
ArnabChatterjee20k
ArnabChatterjee20k