Commit Graph

813 Commits

Author SHA1 Message Date
Chirag Aggarwal 65d1e58e34 refactor: migrate delete queue publisher 2026-05-14 14:40:11 +05:30
Jake Barnby a4d1043684 Merge pull request #12302 from appwrite/fix-transaction-deletes
fix: match transactions collection name in deletes worker
2026-05-14 17:48:18 +12:00
Jake Barnby 10eae0ab87 fix: match transactions collection name in deletes worker
The DELETE_TYPE_TRANSACTION constant was 'transaction' (singular), so
the worker's inner switch on $document->getCollection() (which returns
'transactions') never matched and every transaction deletion fell to
the default branch, logging "No lazy delete operation available for
document of type: transactions". Renamed to DELETE_TYPE_TRANSACTIONS
to align with every other constant used in that switch.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-14 13:58:40 +12:00
Chirag Aggarwal 85e2cf7d8b (refactor): Migrate queueForFunctions to FunctionPublisher and FunctionMessage 2026-05-13 18:28:05 +05:30
loks0n a0c4b7695e chore: normalize Span keys and centralize span lifecycle
Span attribute keys are now snake_case with dots only for child
relationships. Worker span lifecycle moved to app/worker.php; selective
trace filtering moved to the exporter sampler in app/init/span.php so
handlers only call Span::add.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-13 10:29:42 +01:00
Jake Barnby be2bb984c5 Merge pull request #12194 from appwrite/feat-insights-module
feat(insights): add insights module with CTA framework
2026-05-13 17:34:56 +12:00
Jake Barnby b058eb31f4 (refactor): inline CTAs as JSON on insights collection 2026-05-13 17:10:33 +12:00
Luke B. Silver 35941ddc34 Merge pull request #12220 from appwrite/feat/utopia-http-resources-context
Migrate to utopia-php/http resources/context API
2026-05-12 23:01:29 +01:00
Jake Barnby 304b0dab35 (fix): address greptile P1 review comments
- Add skipFilters to Reports/Get.php (was the only endpoint still
  triggering the full N+1 subquery cascade)
- Scale CTA batch limit dynamically (insightCount * MAX_CTA_COUNT)
  instead of fixed APP_LIMIT_SUBQUERY to prevent silent truncation
- Revert deleteReport to callback-based pagination so CTAs are not
  orphaned when a report has more than APP_LIMIT_SUBQUERY insights
- Add explicit prefix lengths (700) to _key_project_resource and
  _key_project_parent_resource indexes to stay under InnoDB 3072-byte limit
- Validate CTA service/method against ADVISOR_CTA_SERVICES and
  ADVISOR_CTA_METHODS enums in the CTAs validator

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-13 01:42:59 +12:00
Jake Barnby 9da4a3260d (refactor): improve advisor module perf, security, and maintainability
- Fix N+1 in Reports/XList (51→4 queries) via skipFilters + batch fetch
- Add skipFilters to Reports/Delete and cursor fetch (avoid loading all
  nested insights/CTAs just for ownership check)
- Fix N+1 in deleteReport worker (flat CTA deletion instead of per-insight)
- Add advisor entity cleanup on project deletion (reports, insights, CTAs)
- Remove resourceInternalId, parentResourceInternalId, $permissions from
  Insight response model (internal IDs leak DB internals, permissions unused)
- Remove dead subQueryInsightCTAs filter registration
- Remove stale enum-value comments from platform schema
- Fix _key_dismissedAt index to include projectInternalId
- Fix scope category from 'Other' to 'Advisor'
- Switch action base class from Utopia\Platform\Action to Appwrite\Platform\Action

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-13 01:13:37 +12:00
Jake Barnby f9ff1166b9 (perf): batch-fetch CTAs across an insight set to collapse N+1 2026-05-12 23:24:26 +12:00
Jake Barnby 4cd35e4d93 Merge branch '1.9.x' into feat-insights-module 2026-05-12 22:59:34 +12:00
Matej Bačo 5e1439f64e Merge pull request #12276 from appwrite/feat-rework-project-response-model
Feat: Rework project response model
2026-05-12 12:57:16 +02:00
loks0n 4bdcdb6f96 Migrate to utopia-php/http resources/context API
Adopts the new split DI containers in utopia-php/http: `resources()` for
boot-time wiring (shared across requests) and `context()` for per-request
state. Replaces the removed `getResource()`/`setResource()`/`getContainer()`
helpers throughout the HTTP entry point, controllers, GraphQL layer, and
installer.

Bumps the dependency chain accordingly: utopia-php/http to the dev branch
(aliased to 0.34.25 to satisfy platform's exact pin), servers 0.4.*,
queue 0.18.*, and pulled-along cli/platform/database upgrades.

Also tightens app/init/resources/request.php by collapsing single-return
factories to arrow functions.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 11:41:42 +01:00
copilot-swe-agent[bot] 63df500b6f refactor(constants): remove Advisor alias prefixes for insight/report enums
Agent-Logs-Url: https://github.com/appwrite/appwrite/sessions/55c56d91-4db4-4501-b5cb-42159dbc9778

Co-authored-by: abnegate <5857008+abnegate@users.noreply.github.com>
2026-05-12 08:09:18 +00:00
Jake Barnby 53d18a0d08 refactor(advisor): rename module to Advisor (service-named, top-level)
The module's namespace and directory now match the top-level service
name (`advisor`) instead of one of its resource names (`insights`):

- src/Appwrite/Platform/Modules/Insights -> .../Modules/Advisor
- src/Appwrite/Insights -> src/Appwrite/Advisor
- tests/unit/Insights -> tests/unit/Advisor
- Route group label flipped from `'insights'` to `'advisor'`
- Section-header comments aligned

Resource names (`insights`, `reports`, `insightCTAs`) and the
`Insight*`/`Report` response models stay — those are the resources the
service exposes, not the service itself.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 23:40:06 +12:00
Matej Bačo 2da400d5b4 Bump version to introduce response filter (for project get) 2026-05-11 12:53:01 +02:00
Matej Bačo 443ebac71c Finish project response model 2026-05-11 11:56:22 +02:00
Jake Barnby db4cb5361d fix(databases): fall back to project DSN when database attr is empty
`getDatabasesDB` used `??` to fall back from a database doc's `database`
attribute to the project DSN, but `??` only triggers on null. Migration
destinations end up with an empty-string `database` (the value is copied
from the source DB but isn't a valid DSN on the destination's pool),
which slipped past the fallback and surfaced as a 500 with
`new DSN('mysql://')` in the catch block. Use elvis so empty strings
fall back too.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 19:12:48 +12:00
copilot-swe-agent[bot] 49bf1198b2 refactor(advisor): polish constants aliases and insight lookup helper
Agent-Logs-Url: https://github.com/appwrite/appwrite/sessions/517a4586-d3e8-40b3-a3a9-f2d2ca82b0a0

Co-authored-by: abnegate <5857008+abnegate@users.noreply.github.com>
2026-05-11 02:08:29 +00:00
copilot-swe-agent[bot] bfd6cebbb0 fix(advisor): address review comments on tests, naming, docs, and get insight
Agent-Logs-Url: https://github.com/appwrite/appwrite/sessions/517a4586-d3e8-40b3-a3a9-f2d2ca82b0a0

Co-authored-by: abnegate <5857008+abnegate@users.noreply.github.com>
2026-05-11 01:59:31 +00:00
Jake Barnby 8fa9ead279 chore: merge 1.9.x into feat-insights-module
Resolves conflict in app/init/constants.php (kept 4327 cache buster, took 1.9.4 stable version).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 17:59:01 +12:00
Jake Barnby 83c0a32494 fix(advisor): make reports.target a TEXT column
target stores free-form URLs or resource IDs. URLs in the wild can
exceed the prior 2048-char cap, so switch the column to VAR_TEXT
(65535). The _key_project_target index already declares an explicit
700-char prefix length, so indexing still works on both MariaDB and
MongoDB.

Bump APP_CACHE_BUSTER for the schema change.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 12:42:29 +12:00
Jake Barnby b830c08257 fix(advisor): use VAR_TEXT for free-form fields
Switch markdown summaries, JSON CTA params, and the embedded
insights/ctas subQuery payloads from sized VAR_STRING to VAR_TEXT.
None of these fields are indexed and they all hold free-form data, so
removing the artificial size cap means we never have to revisit the
limit when an analyzer surfaces a longer report.

- reports.summary, insights.summary: markdown, no upper bound enforced
- reports.insights, insights.ctas: subQuery virtuals carrying full
  child rows
- insightCTAs.params: JSON parameter blob for arbitrary CTA actions

Bump APP_CACHE_BUSTER for the schema change.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 12:30:45 +12:00
Jake Barnby b747c97561 fix(advisor): address review comments on scopes and schema
- Register dedicated reports.write scope and switch deleteReport to it
  so cloud can issue narrowly-scoped delete keys without granting
  insights.write.
- Make insights.parentResourceInternalId optional with null default to
  match its companion parentResourceType/parentResourceId fields and
  unblock insights with no parent (e.g. database-level performance
  insights).
- Tighten Insight.reportId model description: insights always belong to
  a report, ad-hoc insights are not supported.
- Add reports.write to default test API key and admin role so existing
  e2e tests using serverHeaders() can hit the delete endpoint.
- Bump APP_CACHE_BUSTER for the schema change.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 12:18:39 +12:00
Chirag Aggarwal 34075322d7 Migrate mails and messaging queues to publishers 2026-05-08 14:32:11 +05:30
copilot-swe-agent[bot] 56b9de097f fix(advisor): restore report delete with async nested cleanup
Agent-Logs-Url: https://github.com/appwrite/appwrite/sessions/6496395d-5cbf-42ba-a3e0-d3c9e6ca901c

Co-authored-by: abnegate <5857008+abnegate@users.noreply.github.com>
2026-05-08 06:54:49 +00:00
copilot-swe-agent[bot] 6d0eab2583 refactor(advisor): make insights API read-only in CE
Agent-Logs-Url: https://github.com/appwrite/appwrite/sessions/8d7897b5-ac68-487d-954a-be717380bf66

Co-authored-by: abnegate <5857008+abnegate@users.noreply.github.com>
2026-05-08 06:07:23 +00:00
Chirag Aggarwal 8ebf4fee46 Use storage cache callback max age 2026-05-08 11:18:55 +05:30
Chirag Aggarwal 9337daf7ac Preserve default storage cache hit max age 2026-05-08 11:15:17 +05:30
Chirag Aggarwal bf76736971 Split storage cache control context 2026-05-08 09:57:33 +05:30
Chirag Aggarwal 6abd88b8f1 Type storage cache control config 2026-05-08 09:44:00 +05:30
Chirag Aggarwal 7b5a5b8d19 Rename storage cache control hook 2026-05-08 08:44:35 +05:30
Chirag Aggarwal 20ab5b026e Add cached response cache-control callback 2026-05-08 08:33:37 +05:30
loks0n 7d0843cc7f Migrate queueForBuilds to publisherForBuilds pattern
Replaces the stateful Appwrite\Event\Build queue class with a stateless
BuildPublisher and BuildMessage DTO, matching the publisher pattern used
by audits, certificates, executions, migrations, screenshots, stats, and
usage. Call sites now enqueue messages directly instead of mutating a
shared event object and relying on the API shutdown hook.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-07 13:23:21 +01:00
Matej Bačo ec927104fe Merge pull request #12221 from appwrite/fix-variables-api-feature-parity
Feat: Variables API feature parity
2026-05-06 11:27:32 +02:00
Jake Barnby 6c6782dd28 Merge remote-tracking branch 'origin/1.9.x' into pr-12194-feat-insights-module
# Conflicts:
#	app/config/scopes/project.php
2026-05-06 20:33:40 +12:00
Matej Bačo 1ccf058c14 Fix failing tests 2026-05-06 10:19:30 +02:00
Jake Barnby c5dfc42a60 refactor(insights): rename ctas collection to insightCTAs
Disambiguate the platform-level collection name. Field/request-param
remains `ctas` (the embedded array on the insight response).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-06 18:12:17 +12:00
Jake Barnby 5404bfec75 refactor(insights): promote CTAs to own collection with backref
Embedding CTAs as a 16384-byte JSON blob on `insights` was the wrong
shape — they're real documents with their own lifecycle. Move them out.

Schema:
- New platform `ctas` collection. Each row carries `projectInternalId`,
  `projectId`, `insightInternalId`, `insightId` (backref), plus the
  CTA fields: `key`, `label`, `service`, `method`, `params`.
- Indexes: `(projectInternalId, insightInternalId)` for the subquery
  lookup and a UNIQUE `(insightInternalId, key)` so the per-insight
  uniqueness invariant lives at the DB layer (not just in PHP).
- The `ctas` field on `insights` becomes a virtual attribute backed by
  a new `subQueryInsightCTAs` filter that joins child docs at read
  time. Consumers still get CTAs embedded on the insight response —
  one round-trip from their perspective.
- The CTA descriptor's within-insight identifier renamed `id` → `key`
  (clashed with the document `$id`). Validator updated.

Endpoints:
- Manager Create now persists CTAs as separate `ctas` documents after
  the parent insight, then re-fetches the insight so the response
  carries the freshly-joined CTA list.
- User Update trimmed to user-controlled state only (`severity`,
  `status`). `title`, `summary`, `payload`, `ctas`, and `analyzedAt`
  are analyzer-controlled — analyzers re-ingest by deleting and
  POSTing again to the manager endpoint.
- Insight Delete cascades to CTAs.
- Report Delete cascades through Insights → CTAs.

Response model:
- InsightCTA gains the standard document headers (`$id`,
  `$createdAt`, `$updatedAt`) and an `insightId` backref. The
  caller-supplied identifier is now `key`.

Tests:
- E2E sampleCTA factory uses `key` everywhere; testCreate asserts the
  freshly-created CTA carries `$id`, `$createdAt`, `insightId`, and
  the right shape.
- Dropped the testUpdate*CTA* tests — user Update no longer accepts
  CTAs. testDismissViaUpdate now depends on testUpdate directly.
- Unit tests rewritten to validate `key` instead of `id`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-06 18:07:49 +12:00
Jake Barnby a1f64c6f71 refactor(insights): split CTA action into service + method
- InsightCTA model now exposes `service` (SDK namespace) and `method`
  (function name) instead of a single `action` string. Drops the
  string-splitting burden on every consumer and lets the console reach
  the right SDK method directly.

- Validator requires both `service` and `method` non-empty; same
  16-entry max still enforced.

- Endpoint normalization (Create + Update) splits the new shape into
  the persisted CTA descriptor.

- Constants split: INSIGHT_CTA_SERVICE_* (databases / tablesDB /
  documentsDB / vectorsDB) and INSIGHT_CTA_METHOD_* (createIndex).

- Insight model + InsightCTA model docs updated with the new field
  semantics and per-engine examples.

- E2E factory `sampleCTA($id, $engine)` emits the correct service and
  engine-appropriate params keys (tableId/columns for tablesDB;
  collectionId/attributes everywhere else). Engine matrix asserts
  `service` and `method` independently.

- Added e2e + unit coverage for the new failure modes (missing
  service, missing method, empty service, empty method).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-06 16:35:33 +12:00
Jake Barnby 8f79379b6e test(insights): full e2e + per-engine CTA action mapping
- Restructure InsightsBase trait with explicit helper methods
  (createInsight/getInsight/listInsights/updateInsight/deleteInsight,
  createReport/getReport/listReports/updateReport/deleteReport, plus
  sampleInsight/sampleCTA factories) — same shape ProxyBase uses.

- Add coverage for: report CRUD + duplicate-id rejection, invalid type
  rejection, list filtering by all allowed attributes, cursor
  pagination + missing-cursor, update preserving untouched fields,
  CTA validation edge cases (duplicate ids, empty fields, count > 16),
  dismissal round-trip + status filter, report cascade delete,
  unauthorized access (no server key), empty-result list.

- Engine-specific insight types (tablesDBIndex, documentsDBIndex,
  vectorsDBIndex, plus the legacy databaseIndex) so the CTA's `action`
  can map to the matching public API: databases.createIndex,
  tablesDB.createIndex, documentsDB.createIndex,
  vectorsDB.createIndex. dataProvider drives the engine matrix and
  asserts the right action lands in the persisted CTA. Constants for
  each action name live in app/init/constants.php.

- InsightCTA model docs spell out which action belongs to which engine
  and that the params keys differ between APIs (tableId/columns for
  tablesDB vs collectionId/attributes for the legacy / DocumentsDB /
  VectorsDB APIs).

- Insight model `type` description now lists every engine variant.

- CTAsTest gains coverage for object-shaped params, empty-action and
  empty-label rejection, and the default 16-entry cap.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-06 16:25:49 +12:00
Jake Barnby 00565ea471 refactor(insights): metadata-only CTAs, platform DB, reports parent
Address review feedback on PR #12194:

- Pivot CTAs to pure descriptors (id/label/action/params). Drop the
  server-side execution layer: Action interface, registry, the
  databases.indexes.create CTA action, the params validator, the
  /v1/insights/:id/ctas/:id/executions endpoint, the InsightCTAExecution
  model, the INSIGHT_CTA_* errors, and the corresponding events. The
  console invokes the existing public API directly with the descriptor's
  action + params.

- Restore Databases\Indexes\Action.php to its pre-CTA shape and inline
  the index-create body back into Create.php (the createIndex helper
  was added solely for CTA reuse).

- Move insights collection from project DB to platform DB and add a
  parent reports collection alongside it. Insights carry projectId /
  projectInternalId for tenant scoping and an optional reportId for
  grouping. List endpoints filter by projectInternalId; Get/Update/
  Delete also enforce project ownership before touching the document.

- New Reports module with full CRUD (Create/Get/XList/Update/Delete),
  Report response model, Reports query validator, REPORT_NOT_FOUND /
  REPORT_ALREADY_EXISTS errors, reports.read / reports.write scopes,
  and reports.* event tree. Delete cascades to child insights.

- Update.php now mutates the loaded document via setAttribute (instead
  of passing a partial new Document), reuses CTAsValidator (instead of
  the looser ArrayList<JSON> + isset check), and rejects duplicate CTA
  ids.

- Create.php enforces unique CTA ids during normalization.

- CTAsValidator gained a configurable maxCount (default 16) so the
  Create path matches the Update path and the DB column size, and
  oversized payloads return a clean 400.

- Validator\Queries\Insights adds status and reportId to
  ALLOWED_ATTRIBUTES so dismissal / report workflows are filterable.

- Realtime channel parser guards $parts[1] for both insights and
  reports event names.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-06 13:46:07 +12:00
Matej Bačo 305e8e1ec7 Add request filter 2026-05-05 16:24:41 +02:00
ArnabChatterjee20k bca876acaf Merge remote-tracking branch 'origin/1.9.x' into big-int 2026-05-05 18:55:11 +05:30
Jake Barnby 69c637c72d style(insights): use aliased import for DatabasesIndexesCreate registry entry
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 15:01:31 +12:00
Jake Barnby 242c7f75ad refactor(insights): nest databases create-index CTA under resource path
Move the action class from Action/DatabasesCreateIndex.php to
Action/Databases/Indexes/Create.php so the directory mirrors the
underlying resource hierarchy. Action name follows: databases.createIndex
becomes databases.indexes.create, with the constant renamed to
INSIGHT_CTA_ACTION_DATABASES_INDEXES_CREATE.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 14:57:28 +12:00
Jake Barnby f779c7aa3b refactor(insights): collapse dismissal into a status field on update
Dismissal was a sub-resource (POST /v1/insights/:id/dismissals) but a
dismissal is just a state transition, not a thing the client creates.
Drop the dedicated endpoint and add a `status` enum (`active` |
`dismissed`) to the insights schema, settable via the existing PATCH
update route. The server still derives `dismissedAt` and `dismissedBy`
on transition for audit/sorting, but the client-facing API is just a
single status toggle.

- Schema: add `status` attribute (default `active`)
- Constants: add `INSIGHT_STATUSES`
- Update endpoint: accept `status` param, derive dismissedAt/By on
  active <-> dismissed transitions
- Response model: add `status` rule
- Drop Insights/Dismissal/Create.php, the createInsightDismissal SDK
  method, the `insights.[id].dismissals.create` event, and the
  `insight.dismissal.create` audit
- E2E: replace testCreateDismissal with testDismissViaUpdate covering
  both directions of the toggle

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 14:36:31 +12:00
Jake Barnby cd539d972a refactor(insights): capitalise CTA acronym in identifiers
Project-specific override of the default camelCase-acronyms convention:
namespaces, class names, file paths, and SDK method names use `CTA` in
all caps. Touches all insights surfaces — directories, response models,
validators, container resource keys, and SDK method names like
`createInsightCTAExecution`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 14:27:45 +12:00
Jake Barnby d188bd7a2e refactor(insights): register CTA registry on the global $register
Persist the insight CTA registry across requests by attaching it to the
boot-time global $register, mirroring the pattern used for `geodb`,
`passwordsDictionary`, `hooks`, etc. The container resource now looks
the registry up from `$register` instead of rebuilding it on every
request.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 14:22:10 +12:00