Compare commits

...
Author SHA1 Message Date
Prem Palanisamy 3f2f41be60 Add dev key migration support and console project scopes 2026-03-16 22:42:42 +00:00
Prem Palanisamy 210989bd94 Add dev key migration support 2026-03-16 21:07:49 +00:00
Prem Palanisamy b9274e97e1 Pass projectInternalId to destination migration 2026-03-14 23:10:54 +00:00
Prem Palanisamy 93cfb8fd8c Merge remote-tracking branch 'origin/sync-mongodb' into feat-platform-db-access 2026-03-14 11:10:40 +00:00
ArnabChatterjee20k 64dc730a3b refactor: remove debug output and enhance collection creation test with eventual assertion 2026-03-11 16:52:11 +05:30
ArnabChatterjee20k 3096404db5 added log vectordb colleciton creation 2026-03-11 16:23:13 +05:30
ArnabChatterjee20kandGitHub 6818f0b1c3 Merge branch '1.8.x' into sync-mongodb 2026-03-11 15:52:11 +05:30
ArnabChatterjee20k 3ddc073874 updated ollama 2026-03-11 15:51:27 +05:30
ArnabChatterjee20k 4f3468de7b Merge remote-tracking branch 'origin/1.8.x' into sync-mongodb 2026-03-11 14:36:30 +05:30
ArnabChatterjee20k 7a9d7202e5 updated docker file 2026-03-11 14:24:59 +05:30
ArnabChatterjee20k 09bfab03a7 updated ollama 2026-03-11 13:56:48 +05:30
ArnabChatterjee20k 52cd8b7992 updating ollama 2026-03-11 13:47:26 +05:30
ArnabChatterjee20kandGitHub 1cb7ea49ec Merge branch '1.8.x' into sync-mongodb 2026-03-10 18:43:50 +05:30
ArnabChatterjee20k 9989820e17 Fix database type handling and update dependencies in composer files 2026-03-10 18:43:17 +05:30
ArnabChatterjee20k 798c605492 updated postgres 2026-03-10 17:43:34 +05:30
ArnabChatterjee20k ffa4645d25 Refactor database adapter initialization for attribute support
- Moved the `setSupportForAttributes` method call from the adapter initialization to the database resource creation in `getDatabasesDB`.
- Updated the logic to ensure attribute support is set correctly based on the database type.
2026-03-10 17:32:38 +05:30
ArnabChatterjee20k ca0656ce35 Refactor PostgreSQL setup in Docker configuration
- Updated docker-compose.yml to use a pre-built PostgreSQL image instead of a custom build.
- Removed obsolete PostgreSQL Dockerfiles from the project.
- Updated composer.lock with a new content hash.
2026-03-10 16:43:04 +05:30
ArnabChatterjee20k e4178d9ae6 Merge remote-tracking branch 'origin/sync-mongodb' into sync-mongodb 2026-03-10 16:34:10 +05:30
ArnabChatterjee20k 843dfa5c6c Update OLLAMA_KEEP_ALIVE setting in docker-compose.yml to disable keep-alive 2026-03-10 15:53:51 +05:30
ArnabChatterjee20k 52e2276445 Update dependencies and refactor code for clarity
- Changed version constraint for `spomky-labs/otphp` in composer.json to allow any 11.x version.
- Refactored `getDatabasesDBForProject` method in Migrations class to use the shorthand syntax for callable.
- Updated return values in Format class to use more descriptive types for index creation.
- Corrected parameter documentation in Attributes class for clarity and consistency.
2026-03-10 15:49:03 +05:30
ArnabChatterjee20k 9f78fc5341 updated env 2026-03-10 15:41:38 +05:30
ArnabChatterjee20k bf0d49cc76 Refactor embedding timeout and clean database logic
- Updated embedding agent timeout to be configurable via environment variable.
- Removed commented code in XList for clarity.
- Refactored database cleaning logic into separate methods for better readability and maintainability.
2026-03-10 15:41:32 +05:30
ArnabChatterjee20k b5b2338c82 pr comments addressed
* reverted install php
* deleted databasebase from tablesdb
2026-03-10 15:39:10 +05:30
ArnabChatterjee20k cef4d899e3 Add LEGACY constant and update database type references
- Introduced a new constant `LEGACY` in Constants.php.
- Updated `databaseType` in Action classes to use `LEGACY` instead of `TABLESDB`.
- Cleaned up duplicate parameter definition in Create.php for transactionId.
2026-03-10 14:38:14 +05:30
ArnabChatterjee20k f6655343c7 pr comments addressed
* Added new exception `MIGRATION_DATABASE_TYPE_UNSUPPORTED` with proper error metadata and HTTP 400 response.
* Replaced generic CSV database type errors with the new migration-specific exception for clearer error handling.
* Added support for `DOCUMENTSDB` in migration transfer resource service mapping.
* Fixed Appwrite report initialization by correctly injecting `getDatabasesDB`.
* Updated database adapter initialization to conditionally disable attribute support for `DOCUMENTSDB`.
* Moved `setSupportForAttributes` logic from pool initialization to database resource creation.
* Removed duplicate `getDatabasesDB` resource definition and redundant database event listener setup.
* Cleaned up unused variables and minor code inconsistencies.
* Fixed docblock formatting in `TransactionState`.
* Adjusted metrics handling in VectorDB embeddings text creation (removed unnecessary trigger/reset flow).
2026-03-10 14:30:57 +05:30
Jake BarnbyandGitHub 4efababbb4 Merge pull request #11486 from appwrite/vectorsdb 2026-03-10 15:15:21 +13:00
Jake BarnbyandGitHub 5a258b9da1 Merge branch '1.8.x' into sync-mongodb 2026-03-10 15:11:54 +13:00
ArnabChatterjee20k ef248bd5fb fixed usage vectordb model issue on graphql 2026-03-09 17:30:42 +05:30
ArnabChatterjee20k 9c737d906b fixed vectordb db creation issue 2026-03-09 16:40:18 +05:30
ArnabChatterjee20k b8d2f7ece5 updated 2026-03-09 16:10:49 +05:30
ArnabChatterjee20k 5990422a2d updated lock 2026-03-09 15:55:33 +05:30
ArnabChatterjee20k ec500939df updated lock 2026-03-09 15:54:48 +05:30
ArnabChatterjee20k fddaebf254 Merge branch 'sync-mongodb' into vectorsdb 2026-03-09 15:51:30 +05:30
ArnabChatterjee20k d6bc9f120e Merge remote-tracking branch 'origin/1.8.x' into sync-mongodb 2026-03-09 15:50:48 +05:30
ArnabChatterjee20k 1705266779 remove debug logging from database creation process 2026-03-09 15:48:46 +05:30
ArnabChatterjee20k 97d64c3e01 fixing the dsn issue 2026-03-09 15:10:05 +05:30
ArnabChatterjee20k f839cf79d7 added logs for checking dsns on ci 2026-03-09 14:31:23 +05:30
ArnabChatterjee20k 844891c0ae updated 2026-03-09 14:29:39 +05:30
ArnabChatterjee20k c417206e7c updated migration 2026-03-09 14:22:26 +05:30
ArnabChatterjee20k 5326e6571b updated migration 2026-03-09 14:07:29 +05:30
ArnabChatterjee20k 681d930da3 renamed vectordb to vectordb 2026-03-09 13:51:48 +05:30
ArnabChatterjee20k f102419d7c updated format 2026-03-05 14:03:17 +05:30
ArnabChatterjee20k a863ac773a updated index specs 2026-03-05 13:30:54 +05:30
ArnabChatterjee20k 26e92c6a8a Merge remote-tracking branch 'origin/1.8.x' into sync-mongodb 2026-03-05 12:32:19 +05:30
ArnabChatterjee20k 9f5172c1e9 updated migration 2026-03-05 12:30:30 +05:30
ArnabChatterjee20k 519ade1e51 updated realtime to include break 2026-03-05 12:22:45 +05:30
ArnabChatterjee20k 356cd0287c updated index types on vectordb
* merged all indexes together on the vectordb route so that the indexes on the sdk generator doesn't get overriden and fallback to the database adapter to know index is correct or not
2026-03-05 12:21:46 +05:30
ArnabChatterjee20k e84a26be76 Merge remote-tracking branch 'origin/1.8.x' into sync-mongodb 2026-03-05 12:11:44 +05:30
Prem Palanisamy abbc5f9097 Change console-key endpoint from GET to POST 2026-03-03 20:33:11 +00:00
Prem Palanisamy c6f4fc041f Add unit tests for scopedProjectId and source fields in Key::decode() 2026-03-03 18:58:41 +00:00
Prem Palanisamy cdddd438f5 Update composer.lock for utopia-php/migration dev branch 2026-03-03 16:54:17 +00:00
Prem Palanisamy d51130ec8f Add E2E test for platform migration 2026-03-03 16:21:36 +00:00
Prem Palanisamy 01c2d0f04b Add expire field to migration console key response 2026-03-03 15:31:44 +00:00
Prem Palanisamy 01ccee520b Add platform migration support 2026-03-03 11:20:32 +00:00
Prem Palanisamy f25d3df7af Add platform DB access via console-key endpoint 2026-03-03 11:19:13 +00:00
ArnabChatterjee20k d6c8aeee2a Merge remote-tracking branch 'origin/1.8.x' into sync-mongodb 2026-03-02 12:49:01 +05:30
ArnabChatterjee20k 8c3f11b3d5 updated tests 2026-03-02 12:25:40 +05:30
ArnabChatterjee20k e37a9ad059 updated compose 2026-02-27 22:25:08 +05:30
ArnabChatterjee20k 9fc11ce475 updated tests and docker compose 2026-02-27 19:28:24 +05:30
ArnabChatterjee20k 9a32c59340 added vectordb tests 2026-02-27 19:27:24 +05:30
ArnabChatterjee20k c78d19de0d updated 2026-02-27 18:47:46 +05:30
ArnabChatterjee20k 1353f1f56c added status 2026-02-27 18:44:59 +05:30
ArnabChatterjee20k ee34d83cd8 added output of not starting 2026-02-27 18:40:36 +05:30
ArnabChatterjee20k d68449d25b reset 2026-02-27 18:34:18 +05:30
ArnabChatterjee20k d34c83a798 updated tests 2026-02-27 18:25:33 +05:30
ArnabChatterjee20k 6e485bcd5a updated tests file to see the logs of appwrite 2026-02-27 18:12:00 +05:30
ArnabChatterjee20k bab3a49b0b updated 2026-02-27 18:01:06 +05:30
ArnabChatterjee20k 55796eca5c updated lock 2026-02-27 17:21:03 +05:30
ArnabChatterjee20k abf497d509 Merge remote-tracking branch 'origin/1.8.x' into sync-mongodb 2026-02-27 17:19:31 +05:30
ArnabChatterjee20k a76a5cda38 Refactor code structure for improved readability and maintainability 2026-02-27 17:15:00 +05:30
ArnabChatterjee20k 377a9d915e updated env 2026-02-27 13:09:56 +05:30
ArnabChatterjee20k c7b3972fa0 fixed event 2026-02-26 17:12:44 +05:30
ArnabChatterjee20k fda53c0520 updated composer 2026-02-26 17:07:24 +05:30
ArnabChatterjee20k f2e789537c updated lock 2026-02-26 16:34:42 +05:30
ArnabChatterjee20k 3287ef6a57 Merge remote-tracking branch 'origin/1.8.x' into sync-mongodb 2026-02-26 16:33:56 +05:30
ArnabChatterjee20k 74b3d8f493 updated lock 2026-02-26 16:22:59 +05:30
ArnabChatterjee20k e0972f4636 updated lock 2026-02-25 16:50:13 +05:30
ArnabChatterjee20k 42f914f6aa fixed merge conflicts 2026-02-25 16:49:31 +05:30
ArnabChatterjee20k 58f4fff864 added 2026-02-25 16:20:44 +05:30
ArnabChatterjee20k 681c9743b1 added controllers 2026-02-25 16:09:28 +05:30
ArnabChatterjee20k cd2d2a1238 added configs 2026-02-25 16:09:16 +05:30
ArnabChatterjee20k 905763e755 added tests 2026-02-25 16:08:40 +05:30
ArnabChatterjee20k 2550482432 updated docs 2026-02-25 16:07:19 +05:30
ArnabChatterjee20k 8d028066ae update transaction not to start if there is no operation in the logs 2026-02-25 15:44:14 +05:30
ArnabChatterjee20k bd4adad545 refractored transaction update 2026-02-25 12:49:27 +05:30
ArnabChatterjee20k c40f9b1de0 addressed comments 2026-02-24 18:23:52 +05:30
ArnabChatterjee20k 8237250453 refactor: streamline database DSN handling and remove deprecated attributes 2026-02-24 17:40:50 +05:30
ArnabChatterjee20k c8a7ed2022 updated migration to remove the dependency on dsn at project level attribute 2026-02-24 10:43:10 +05:30
ArnabChatterjee20k 4847c5a308 changed deps so that create document depends on test create collection 2026-02-23 13:27:01 +05:30
ArnabChatterjee20k f6985bae41 updated lock 2026-02-23 13:15:52 +05:30
ArnabChatterjee20k cc9df4f5c5 Merge remote-tracking branch 'origin/1.8.x' into documents-db-api 2026-02-23 13:09:49 +05:30
ArnabChatterjee20k 83b7b4e599 updated 2026-02-23 13:05:12 +05:30
ArnabChatterjee20k fdc573009c updated tests 2026-02-20 16:53:46 +05:30
ArnabChatterjee20k 51d769b0a1 Merge remote-tracking branch 'origin/1.8.x' into documents-db-api 2026-02-20 13:58:56 +05:30
ArnabChatterjee20k f6b04d05b2 updated composer lock 2026-02-18 16:10:13 +05:30
ArnabChatterjee20k 2926412604 Merge remote-tracking branch 'origin/1.8.x' into documents-db-api 2026-02-18 16:07:44 +05:30
ArnabChatterjee20k ee21b1feda updated composer 2026-02-18 16:06:31 +05:30
ArnabChatterjee20k 2d0140f705 Merge remote-tracking branch 'origin/1.8.x' into documents-db-api 2026-02-18 16:01:55 +05:30
ArnabChatterjee20k 183bd4143c updated tests 2026-02-16 18:36:42 +05:30
ArnabChatterjee20k 57ac67a106 linting and updated timeouts 2026-02-16 16:23:38 +05:30
ArnabChatterjee20k 4b6d809a3b updated docker compose 2026-02-16 16:09:15 +05:30
ArnabChatterjee20k 0221556ae7 Merge remote-tracking branch 'origin/documents-db-api' into documents-db-api 2026-02-16 15:16:51 +05:30
ArnabChatterjee20k 6a1955d20d updated tests 2026-02-16 15:16:04 +05:30
ArnabChatterjee20kandGitHub 8c7d2c58f5 Merge branch '1.8.x' into documents-db-api 2026-02-16 14:30:53 +05:30
ArnabChatterjee20k 5b7746b464 empty 2026-02-16 14:12:46 +05:30
ArnabChatterjee20k e6e0cef91e linting 2026-02-16 13:55:34 +05:30
ArnabChatterjee20k bdc4c4a18d Merge remote-tracking branch 'origin/documents-db-api' into documents-db-api 2026-02-16 13:53:21 +05:30
ArnabChatterjee20k 49efa02e09 synced documents db and vectordb with 1.8.x 2026-02-16 13:52:16 +05:30
ArnabChatterjee20k fd96bcbd3f Merge remote-tracking branch 'origin/1.8.x' into documents-db-api 2026-02-16 12:20:53 +05:30
Jake BarnbyandGitHub 9647c3fa7d Merge pull request #11310 from appwrite/database-type-filters 2026-02-12 09:38:45 +00:00
Darshan 1e411bbef5 update: lint. 2026-02-12 13:25:44 +05:30
Darshan 7c47a4db72 add: filtering. 2026-02-12 13:22:07 +05:30
ArnabChatterjee20k ae33104ecf added admin auth to the index creation 2026-02-11 18:19:46 +05:30
ArnabChatterjee20k b90a9e8cab added admin auth to the index creation 2026-02-11 16:49:33 +05:30
ArnabChatterjee20k 4024cf0492 Merge remote-tracking branch 'origin/1.8.x' into documents-db-api 2026-02-11 16:22:37 +05:30
ArnabChatterjee20k e07f9a7b49 added admin auth to vector db 2026-02-11 14:04:14 +05:30
ArnabChatterjee20k 61f778196b Merge remote-tracking branch 'origin/1.8.x' into documents-db-api 2026-02-11 13:58:43 +05:30
ArnabChatterjee20k 93897f86d0 Merge remote-tracking branch 'origin/1.8.x' into documents-db-api 2026-02-10 15:34:51 +05:30
ArnabChatterjee20k 095e08441f added get databases db in the document logs 2026-02-10 13:33:29 +05:30
ArnabChatterjee20k da63463a3c updated migration 2026-02-10 11:20:53 +05:30
ArnabChatterjee20k 8e140bae6c updated project and migraiton worker 2026-02-09 17:43:42 +05:30
ArnabChatterjee20k 3b2eb02f8a Merge branch '1.8.x' into documents-db-api 2026-02-09 17:27:09 +05:30
ArnabChatterjee20k e53f5b4023 add admin permission to create 2026-01-28 19:06:19 +05:30
ArnabChatterjee20k 02cddb5c08 updated the format specs 2026-01-28 10:55:19 +05:30
ArnabChatterjee20k cb4d11a1d4 updated 2026-01-27 18:13:27 +05:30
ArnabChatterjee20k 139535e83c updated 2026-01-27 17:18:24 +05:30
ArnabChatterjee20k b9e95f6d82 updated 2026-01-27 17:17:23 +05:30
ArnabChatterjee20k 79cb668a50 updated composer lock 2026-01-27 16:34:34 +05:30
ArnabChatterjee20k 8078516e2f fixed composer lock 2026-01-27 16:32:59 +05:30
ArnabChatterjee20k 1729bd23b2 Merge remote-tracking branch 'origin/1.8.x' into documents-db-api 2026-01-27 16:29:25 +05:30
ArnabChatterjee20k ac7fc7ae69 linting 2026-01-22 12:47:26 +05:30
ArnabChatterjee20k 4b570dc3f0 updetad pools 2026-01-22 12:41:18 +05:30
ArnabChatterjee20k aa3d66df7b Merge branch 'new-pool-adapter' into documents-db-api 2026-01-22 12:34:22 +05:30
ArnabChatterjee20k a16e05ed4d updated timeout 2026-01-21 13:36:35 +05:30
ArnabChatterjee20k 84dc5682d5 updated auth param 2026-01-21 12:31:42 +05:30
ArnabChatterjee20k a112de41c5 updated ollama timeout 2026-01-21 12:23:29 +05:30
ArnabChatterjee20k 02bba85fe7 updated migration tests for adding endpoint 2026-01-21 12:21:37 +05:30
ArnabChatterjee20k 56f68d33df updated migration 2026-01-21 11:44:13 +05:30
ArnabChatterjee20k d4edf9ed13 updated vectordb test 2026-01-20 20:10:57 +05:30
ArnabChatterjee20k 12b38e2a27 commenting mongodb out for now to check the issues on the ci 2026-01-20 19:45:19 +05:30
ArnabChatterjee20k 9f1807a128 linting and fixing resources timeout and validator 2026-01-20 19:41:52 +05:30
ArnabChatterjee20k 6bdfcc3fb5 fixed endpoint in migration 2026-01-20 18:44:54 +05:30
ArnabChatterjee20k d8dd1918c4 synced vectordb 2026-01-20 15:45:12 +05:30
ArnabChatterjee20k 376c5e47f5 added missing injections 2026-01-20 14:01:37 +05:30
ArnabChatterjee20k edfb9e0bd7 updated resources to use the listeners based on the databasetype 2026-01-20 13:56:52 +05:30
ArnabChatterjee20k eab4c7f28a Merge remote-tracking branch 'origin/1.8.x' into documents-db-api 2026-01-20 13:56:32 +05:30
ArnabChatterjee20k c361340252 linting 2026-01-19 18:45:05 +05:30
ArnabChatterjee20k 41e197003e fixed merge conflicts 2026-01-19 18:43:30 +05:30
ArnabChatterjee20k c7ba37ec49 fixed merge conflicts 2026-01-19 14:35:31 +05:30
ArnabChatterjee20k ba82aab563 Merge remote-tracking branch 'origin/1.8.x' into documents-db-api 2026-01-19 13:22:53 +05:30
ArnabChatterjee20k f580dc9145 Merge remote-tracking branch 'upstream/1.8.x' into documents-db-api 2026-01-13 14:32:46 +05:30
Jake BarnbyandGitHub 421dca2f99 Merge pull request #10653 from appwrite/vector-db-api
vectordb api endpoints
2025-12-10 10:59:58 +00:00
ArnabChatterjee20k 991b2b1c3b updated delete worker for vectordb and updated tests 2025-12-09 18:44:26 +05:30
ArnabChatterjee20k d22743a1cf Merge remote-tracking branch 'upstream/documents-db-api' into vector-db-api 2025-12-09 18:37:09 +05:30
ArnabChatterjee20k a2039c0b76 added deletion test for multidb 2025-12-09 18:34:25 +05:30
ArnabChatterjee20k c549ddd149 updated project deletion worker to delete per database 2025-12-09 18:27:10 +05:30
ArnabChatterjee20k f3721f712a fixed missing project usage fields 2025-12-09 15:38:43 +05:30
ArnabChatterjee20k 480757dcfe added usage test for the vectordb 2025-12-09 12:07:26 +05:30
ArnabChatterjee20k fbaeff8920 Merge remote-tracking branch 'upstream/documents-db-api' into vector-db-api 2025-12-08 20:31:07 +05:30
ArnabChatterjee20k de88ec4065 updated missing skip metrics and tablesdb tests 2025-12-08 20:30:37 +05:30
ArnabChatterjee20k f149ba33d4 updated stats for vectordb 2025-12-08 20:27:30 +05:30
ArnabChatterjee20k b15f2eed8f Merge remote-tracking branch 'upstream/documents-db-api' into vector-db-api 2025-12-08 19:32:21 +05:30
ArnabChatterjee20k ec63dc22d4 updated controllers 2025-12-08 19:09:54 +05:30
ArnabChatterjee20k 7c7b98399c linting 2025-12-08 17:51:01 +05:30
ArnabChatterjee20k 9d679e726e updated composer lock 2025-12-08 17:46:00 +05:30
ArnabChatterjee20k 070ec03c0c Merge remote-tracking branch 'upstream/1.8.x' into documents-db-api 2025-12-08 17:31:04 +05:30
ArnabChatterjee20k be89224c36 updated console client test 2025-12-05 19:14:29 +05:30
ArnabChatterjee20k 8edd3c87cb added documentsdb total reads and writes in the project response 2025-12-05 19:09:35 +05:30
ArnabChatterjee20k 9ac3cd73a8 Refactor database metrics handling for DocumentsDB
- Introduced new methods to retrieve metrics for write and read operations based on database type.
- Updated bulk delete, update, upsert, and create actions to use the new metric methods.
- Enhanced document retrieval and update actions to utilize the new metric structure.
- Added support for DocumentsDB metrics in usage statistics and response models.
- Created new response models for DocumentsDB usage and updated existing models to accommodate new metrics.
- Adjusted migration and stats resources to include DocumentsDB metrics.
- Updated tests to validate the new metrics and ensure correct functionality for DocumentsDB usage.
2025-12-05 18:58:13 +05:30
ArnabChatterjee20k 203170c36d updated database and collection stats 2025-12-04 15:57:18 +05:30
ArnabChatterjee20k 6f5bfdb726 updated stats 2025-12-03 20:18:53 +05:30
b499857871 Update src/Appwrite/Utopia/Database/Validator/Queries/Base.php
Co-authored-by: Jake Barnby <jakeb994@gmail.com>
2025-12-03 19:12:56 +05:30
1388b0ca21 Update app/config/variables.php
Co-authored-by: Jake Barnby <jakeb994@gmail.com>
2025-12-03 19:12:38 +05:30
ArnabChatterjee20k af095bdf4a updated stats 2025-12-03 19:11:46 +05:30
ArnabChatterjee20k 7af361fa16 * added logger in the create text embedding
* aadded error text metric
2025-12-03 13:38:24 +05:30
ArnabChatterjee20k 069f231b3a updated composer lock 2025-12-02 18:40:28 +05:30
ArnabChatterjee20k def028e970 Merge remote-tracking branch 'upstream/documents-db-api' into vector-db-api 2025-12-02 18:38:21 +05:30
ArnabChatterjee20k 16f49d13c8 updated composer lock 2025-12-02 18:36:29 +05:30
ArnabChatterjee20k e72d7979e0 updated composer lock 2025-12-02 18:31:44 +05:30
ArnabChatterjee20k ca4c7b5361 added stats usage for text embeddings 2025-12-02 18:13:49 +05:30
ArnabChatterjee20k 7ad6bb048c merged 1.8.x 2025-12-02 16:35:40 +05:30
ArnabChatterjee20k d2c9ac079a Refactor VectorDB Embeddings API and Update Dimension Handling
- Updated the Create.php file to streamline the embedding creation process by removing unnecessary authentication checks and simplifying the parameter structure.
- Changed the parameter from 'documents' to 'texts' for clarity and adjusted the embedding model handling.
- Modified the response model to include error handling for embedding generation.
- Updated the Embedding response model to reflect changes in dimension naming and added error message handling.
- Refactored VectorDBCollection model to replace 'dimensions' with 'dimension' for consistency across the codebase.
- Adjusted all relevant tests to accommodate the new parameter names and response structures, ensuring comprehensive coverage for the updated functionality.
2025-12-02 14:20:25 +05:30
ArnabChatterjee20k abe1257bd5 Merge remote-tracking branch 'upstream/documents-db-api' into vector-db-api 2025-11-21 20:17:59 +05:30
ArnabChatterjee20k 3db22fd2db fixed enum route 2025-11-21 20:03:44 +05:30
ArnabChatterjee20k a423067b36 Merge remote-tracking branch 'upstream/1.8.x' into documents-db-api 2025-11-21 19:53:52 +05:30
ArnabChatterjee20k 33aab0357f updated tests 2025-11-21 19:52:28 +05:30
ArnabChatterjee20k 34e9c69a62 updated tests condition 2025-11-21 19:11:26 +05:30
ArnabChatterjee20k fdf5ca0067 updated tests condition 2025-11-21 19:08:35 +05:30
ArnabChatterjee20k 6eb17b847f updated composer 2025-11-21 18:06:32 +05:30
ArnabChatterjee20k 918ebb8ecc Add VectorDB text embedding creation endpoint and update related tests 2025-11-21 18:00:51 +05:30
ArnabChatterjee20k c0ac52810e Merge remote-tracking branch 'upstream/documents-db-api' into vector-db-api 2025-11-21 17:25:18 +05:30
ArnabChatterjee20k b658cddc71 updated get project dsn to be of the project db type(dedicate, shared) so that databasesdb and projectdb are not in conflict 2025-11-21 17:22:19 +05:30
ArnabChatterjee20k fe753db78c Merge remote-tracking branch 'upstream/documents-db-api' into vector-db-api 2025-11-20 18:57:38 +05:30
ArnabChatterjee20k 2f1e47e35a updated migrations 2025-11-20 18:56:14 +05:30
ArnabChatterjee20k c3ad337398 updated migrations for vectordb importing/exporting in csv endpoints 2025-11-20 18:54:41 +05:30
ArnabChatterjee20k cc759868b8 Merge remote-tracking branch 'upstream/documents-db-api' into vector-db-api 2025-11-20 15:51:01 +05:30
ArnabChatterjee20k 1004435e9e skipped operator tests for mongodb 2025-11-20 14:07:05 +05:30
ArnabChatterjee20k a9e749a81f updated migration service extractor for csv import/export 2025-11-20 14:06:42 +05:30
ArnabChatterjee20k 553fe15ba9 resolved merge conflicts 2025-11-19 20:01:02 +05:30
ArnabChatterjee20k fec758d237 Merge remote-tracking branch 'upstream/1.8.x' into documents-db-api 2025-11-19 20:00:17 +05:30
ArnabChatterjee20k 70a1d5d0f7 updated composer lock 2025-11-14 21:33:17 +05:30
ArnabChatterjee20k c66308e4e9 Merge remote-tracking branch 'upstream/documents-db-api' into vector-db-api 2025-11-14 21:28:40 +05:30
ArnabChatterjee20k 21aae468f2 updated database type function 2025-11-14 19:35:53 +05:30
ArnabChatterjee20k f007b83ccc updated composer 2025-11-14 19:18:03 +05:30
ArnabChatterjee20k 28be12d827 Merge remote-tracking branch 'upstream/1.8.x' into documents-db-api 2025-11-14 19:16:08 +05:30
ArnabChatterjee20k 93bd6b39a4 updated migrations + index finding issue in vectordb 2025-11-13 16:47:00 +05:30
ArnabChatterjee20k 75045ffd58 added constants for the vector dimension 2025-11-12 16:28:58 +05:30
ArnabChatterjee20k ed8419986a linting 2025-11-12 16:23:59 +05:30
ArnabChatterjee20k 954011b2c4 added transactions for vectordb 2025-11-12 16:21:27 +05:30
ArnabChatterjee20k c3f221f1fb added timestamp mutation test 2025-11-07 13:13:13 +05:30
ArnabChatterjee20k de59f53de3 reverted error tracing 2025-11-06 18:25:01 +05:30
ArnabChatterjee20k 04b169fab8 updated composer packages 2025-11-06 18:01:01 +05:30
ArnabChatterjee20k 857ec3b099 reverted docker compose 2025-11-06 17:54:21 +05:30
ArnabChatterjee20k b1dd377b67 Merge branch 'documents-db-api' into vector-db-api 2025-11-06 17:52:14 +05:30
ArnabChatterjee20k 81128e8446 reverted changes 2025-11-06 17:51:33 +05:30
ArnabChatterjee20k 86371e2082 Merge branch 'documents-db-api' into vector-db-api 2025-11-06 17:50:14 +05:30
ArnabChatterjee20k bd6f62dd68 updated realtime, create endpoint , resources, registers 2025-11-06 17:48:39 +05:30
ArnabChatterjee20k d2275c05fb updated tests 2025-11-05 20:34:12 +05:30
ArnabChatterjee20k 4988881568 updated method names and sdk namespaces for vectordb 2025-11-03 11:17:57 +05:30
ArnabChatterjee20k f9be1ddfc5 added vector embedding creation + update endpoint 2025-10-31 19:07:34 +05:30
ArnabChatterjee20k 60e95458a7 vector db endpoints
* configs and default attributes + indexes
* separate collection creation setup
* fix -> database metadata fix on collection creation
2025-10-31 18:01:47 +05:30
ArnabChatterjee20k 2bb6b0a2a9 updated database dsn for create in the create database 2025-10-31 16:45:01 +05:30
ArnabChatterjee20k 3311bd27cb Merge branch 'documents-db-api' into vector-db-api 2025-10-31 16:35:00 +05:30
ArnabChatterjee20k 5d9ef0dcca added metadata create on database 2025-10-31 16:33:54 +05:30
ArnabChatterjee20k 7ddecdd9ef Merge branch 'documents-db-api' into vector-db-api 2025-10-31 10:45:58 +05:30
ArnabChatterjee20k c8f966b0e5 added the default database type for the transactions context 2025-10-30 13:44:26 +05:30
ArnabChatterjee20k 3d2f21be7c updated composer lock 2025-10-30 13:02:49 +05:30
ArnabChatterjee20k 3e876f5bd8 fixed merge conflicts 2025-10-30 12:57:41 +05:30
ArnabChatterjee20k a5e56438c7 Merge remote-tracking branch 'upstream/1.8.x' into documents-db-api 2025-10-30 12:35:57 +05:30
ArnabChatterjee20k 4e79259c4e Refactor database handling to streamline DSN retrieval for documents and tables databases 2025-10-30 12:27:52 +05:30
ArnabChatterjee20k fae79e5c3f updated tests from feat-mongodb branch 2025-10-29 13:49:18 +05:30
ArnabChatterjee20k b169be9dda updated migration merge conflicts 2025-10-29 11:41:46 +05:30
ArnabChatterjee20k 8f4496346d Merge remote-tracking branch 'upstream/1.8.x' into documents-db-api 2025-10-28 18:41:03 +05:30
ArnabChatterjee20k f306c03030 updated tests and index creation 2025-10-28 18:32:17 +05:30
ArnabChatterjee20k a8469a9b92 updated composer lock 2025-10-27 18:10:41 +05:30
ArnabChatterjee20k b63510acb5 Merge remote-tracking branch 'upstream/1.8.x' into documents-db-api 2025-10-27 18:10:08 +05:30
ArnabChatterjee20k 3e612ec88d flask test fixing via retry 2025-10-27 18:08:44 +05:30
ArnabChatterjee20k 1be36ae552 updated deps 2025-10-27 16:19:04 +05:30
ArnabChatterjee20k 445b657207 Merge remote-tracking branch 'upstream/1.8.x' into documents-db-api 2025-10-27 16:15:10 +05:30
ArnabChatterjee20k 5098973e0d reverted longtext resource file 2025-10-17 18:14:16 +05:30
ArnabChatterjee20k d0976ff429 updated registers config 2025-10-17 14:16:45 +05:30
ArnabChatterjee20k 157783f782 merged feat-mongodb 2025-10-17 13:58:27 +05:30
ArnabChatterjee20k e3a70373e1 updated docuemntsdb transaction path names 2025-10-16 22:19:53 +05:30
ArnabChatterjee20k a6e5563f8e reverted docker compose 2025-10-16 21:51:56 +05:30
ArnabChatterjee20k 470d84a843 * updated transactions
* updated tests
2025-10-16 21:45:30 +05:30
ArnabChatterjee20k 80037f7e8d updated http php for mongodb setup 2025-10-16 15:57:13 +05:30
ArnabChatterjee20k cc848c90a7 updated lock file 2025-10-16 14:25:22 +05:30
ArnabChatterjee20k e0c293c5b4 Merge remote-tracking branch 'upstream/1.8.x' into documents-db-api 2025-10-16 14:25:14 +05:30
ArnabChatterjee20k 2d82013261 updated docker compose setup for mongodb which was causing issue for mongo transactions 2025-10-16 14:22:28 +05:30
ArnabChatterjee20k 5a6561f158 vectordb (in progress)
* pools , env setup
* collections, db endpoints
2025-10-16 12:53:39 +05:30
ArnabChatterjee20k f28588ecb6 updated migration 2025-10-15 14:24:59 +05:30
ArnabChatterjee20k 6579ea5b56 Merge remote-tracking branch 'upstream/1.8.x' into documents-db-api 2025-10-15 14:16:49 +05:30
ArnabChatterjee20k 9373688be2 updated migration worker to pass the dsn resolver for the destination database 2025-10-15 10:33:46 +05:30
ArnabChatterjee20k 2cec06be18 Merge remote-tracking branch 'upstream/1.8.x' into documents-db-api 2025-10-14 21:18:38 +05:30
ArnabChatterjee20k 3a60bb306c * updated migration tests
* database dsn fetching
* removed dsn from response models
2025-10-14 21:13:26 +05:30
ArnabChatterjee20k e2ddb0d6d0 updated migration 2025-10-14 13:23:48 +05:30
Darshan 0049265c8a Merge remote-tracking branch 'origin/documents-db-api' into documents-db-api 2025-10-14 12:49:41 +05:30
Darshan 1f27867565 update: method name. 2025-10-14 12:49:34 +05:30
ArnabChatterjee20k faf0cdec35 changed default fallbackForDB to mariadb 2025-10-14 12:46:01 +05:30
ArnabChatterjee20k 6073eab496 * updated env to have mongodb support for the tablesdb/legacy
* setAttributeSupport to support context schema and schemaless in mongodb
* updated the connection registers
2025-10-13 15:47:40 +05:30
ArnabChatterjee20k 50ddc611a4 updated migrations 2025-10-12 20:49:02 +05:30
ArnabChatterjee20k 0ef2cfa753 reverted docker compose 2025-10-12 20:26:39 +05:30
ArnabChatterjee20k 790e503ad9 updated composer lock 2025-10-12 20:19:57 +05:30
ArnabChatterjee20k f6dbc0c882 Merge remote-tracking branch 'upstream/feat-mongodb' into documents-db-api 2025-10-12 20:17:53 +05:30
ArnabChatterjee20k a6c226aac9 removed the documentsdb changes from the migration tests 2025-10-10 20:38:54 +05:30
ArnabChatterjee20k 2d41a3353d * migration fix for tablesdb, legacy with multitype callback
* updated patch script
2025-10-10 20:29:05 +05:30
ArnabChatterjee20k 34e4208b3e addressed comments
* renamed getDatabaseDB to getDatabasesDB
* renamed dbForDatabase to dbForDatabases
* removed call_user_func and using newer callable syntax
2025-10-10 20:00:43 +05:30
ArnabChatterjee20k d51486c62a * fixed stats usage events(hacky fix)
* updated redundant routes(happened during merge conflicts)
2025-10-10 19:39:59 +05:30
ArnabChatterjee20k f62f00d4d9 Merge remote-tracking branch 'upstream/documents-db-api' into documents-db-api 2025-10-10 15:26:33 +05:30
ArnabChatterjee20k 44e6ef3146 event generation fix 2025-10-10 15:24:42 +05:30
Jake BarnbyandGitHub dea49d54c9 Merge pull request #10628 from appwrite/fix-enums
Update: enums
2025-10-10 22:16:45 +13:00
Darshan 8a9523fa48 update: enums. 2025-10-10 14:40:26 +05:30
Darshan 5dba0183c0 update: enums. 2025-10-10 14:40:01 +05:30
ArnabChatterjee20k 0108f86180 linting + merge conflicts resolving 2025-10-10 13:39:16 +05:30
ArnabChatterjee20k bc71d83697 empty commit 2025-10-10 10:58:11 +05:30
ArnabChatterjee20k 0754c6263e merged changes from 1.8.x 2025-10-10 10:55:48 +05:30
ArnabChatterjee20k 2c037803f0 updated sdk names and enums for documentsdb 2025-10-09 18:41:10 +05:30
ArnabChatterjee20k 48410d0cf6 * added getDatabaseDB in stats worker, migration worker
* added prefixed events for multitype db support with backward compat
* added realtime tests, migration tests
2025-10-09 18:26:05 +05:30
ArnabChatterjee20k 66b0f3c9df * added waiting time in usage test so that aggregator can work
* added documentsdb usage test
2025-10-09 18:24:04 +05:30
ArnabChatterjee20k d8910eea73 fix - updated documentsdb sdk namespace to documentsDB 2025-10-09 10:30:57 +05:30
ArnabChatterjee20k f7f965a812 * added prefixed channels for realtime channels along with backward compatibility
* added documentsdb tests
2025-10-09 00:35:38 +05:30
ArnabChatterjee20k d806bdbca7 added project region in the get database dsn 2025-10-08 11:17:12 +05:30
ArnabChatterjee20k 19b1196633 * new db pool implementation for connecting to db type
* fixed migration tests
2025-10-08 11:09:32 +05:30
ArnabChatterjee20k 88f2b1ad50 added patch script for adding database dsn in the databases collection 2025-10-07 13:18:11 +05:30
ArnabChatterjee20k f5ea19b15d updated tests 2025-10-07 12:45:21 +05:30
ArnabChatterjee20k d2645ecfb1 Merge remote-tracking branch 'upstream/1.8.x' into documents-db-api 2025-10-07 10:47:26 +05:30
ArnabChatterjee20k 44b75d50f4 revmoved databasse logs xlist action for documentsdb 2025-10-03 19:56:20 +05:30
ArnabChatterjee20k df23aacbc0 updated routes and endpoints 2025-10-03 19:51:00 +05:30
ArnabChatterjee20k 7d4dfe06e2 * removed dbForDocumentsRecords
* using callback based db resolution
* updated env
2025-10-03 19:50:29 +05:30
ArnabChatterjee20k e50af4c981 add references docs for the documentsdb 2025-10-02 19:24:35 +05:30
ArnabChatterjee20k 4fdab6fd52 updated database db resolution in the resources itself instead of switch in the worker action 2025-10-02 18:58:11 +05:30
ArnabChatterjee20k c54450f582 updated composer 2025-10-02 18:57:15 +05:30
ArnabChatterjee20k 6bf9693146 Merge remote-tracking branch 'upstream/1.8.x' into documents-db-api 2025-10-02 18:56:48 +05:30
ArnabChatterjee20k 422fd8b157 updated get endpoint 2025-09-30 20:02:30 +05:30
ArnabChatterjee20k bf061b460b added documentsdb in the github actions 2025-09-30 19:29:36 +05:30
ArnabChatterjee20k a035760b00 updated injections and reverted reduntant changes 2025-09-30 19:15:43 +05:30
ArnabChatterjee20k 85c0454c49 linting 2025-09-30 18:45:53 +05:30
ArnabChatterjee20k 7bfb0401c6 linting 2025-09-30 18:44:27 +05:30
ArnabChatterjee20k c097f4a450 updated composer lock 2025-09-30 18:43:02 +05:30
ArnabChatterjee20k a3faa108c5 Merge remote-tracking branch 'upstream/1.8.x' into documents-db-api 2025-09-30 18:42:18 +05:30
ArnabChatterjee20k c89b76ba80 *updated tests
*updated create documents to use the on next callback
2025-09-30 18:19:44 +05:30
ArnabChatterjee20k 80636e8a91 updated timeouts 2025-09-30 00:03:12 +05:30
ArnabChatterjee20k 206845e106 updated tests and delete delete documents 2025-09-28 14:43:44 +05:30
ArnabChatterjee20k 64d015065f * added new databasetype dsn
* updated worker to use the dsn to choose the database resource
2025-09-26 19:52:25 +05:30
ArnabChatterjee20k 56af56e341 updated endpoints 2025-09-26 18:01:39 +05:30
ArnabChatterjee20k 3086e1cbc1 updated increment, and decrements routes 2025-09-25 19:58:59 +05:30
ArnabChatterjee20k b032a55045 updated the endpoints 2025-09-25 00:48:09 +05:30
ArnabChatterjee20k d71c56b11f * Migrating endpoints
* New adapter resolution and injection
* Removed attribute creation for documentsdb
2025-09-23 23:15:46 +05:30
ArnabChatterjee20k 98ee242812 migrated tests 2025-09-20 18:21:00 +05:30
ArnabChatterjee20k 79267c8123 migrated endpoints 2025-09-20 18:20:50 +05:30
ArnabChatterjee20k d2ad0e7c74 documentsdb database creation endpoints 2025-09-19 19:44:46 +05:30
258 changed files with 23363 additions and 1681 deletions
+13 -4
View File
@@ -39,14 +39,23 @@ _APP_REDIS_HOST=redis
_APP_REDIS_PORT=6379
_APP_REDIS_PASS=
_APP_REDIS_USER=
COMPOSE_PROFILES=mongodb
_APP_DB_ADAPTER=mongodb
_APP_DB_HOST=mongodb
_APP_DB_PORT=27017
COMPOSE_PROFILES=mariadb,mongodb,postgresql
_APP_DB_ADAPTER=mariadb
_APP_DB_HOST=mariadb
_APP_DB_PORT=3306
_APP_DB_SCHEMA=appwrite
_APP_DB_USER=user
_APP_DB_PASS=password
_APP_DB_ROOT_PASS=rootsecretpassword
_APP_DB_ADAPTER_DOCUMENTSDB=mongodb
_APP_DB_HOST_DOCUMENTSDB=mongodb
_APP_DB_PORT_DOCUMENTSDB=27017
_APP_DB_ADAPTER_VECTORSDB=postgresql
_APP_DB_HOST_VECTORSDB=postgresql
_APP_DB_PORT_VECTORSDB=5432
_APP_EMBEDDING_MODELS=embeddinggemma
_APP_EMBEDDING_ENDPOINT='http://ollama:11434/api/embed'
_APP_EMBEDDING_TIMEOUT=30000
_APP_STORAGE_DEVICE=Local
_APP_STORAGE_S3_ACCESS_KEY=
_APP_STORAGE_S3_SECRET=
+2
View File
@@ -4,6 +4,7 @@
$common = include __DIR__ . '/collections/common.php';
$projects = include __DIR__ . '/collections/projects.php';
$databases = include __DIR__ . '/collections/databases.php';
$vectorsdb = include __DIR__ . '/collections/vectorsdb.php';
$platform = include __DIR__ . '/collections/platform.php';
$logs = include __DIR__ . '/collections/logs.php';
@@ -26,6 +27,7 @@ unset($common['files']);
$collections = [
'buckets' => $buckets,
'databases' => $databases,
'vectorsdb' => $vectorsdb,
'projects' => array_merge_recursive($projects, $common),
'console' => array_merge_recursive($platform, $common),
'logs' => $logs,
+9
View File
@@ -61,6 +61,15 @@ return [
'array' => false,
'filters' => [],
],
[
'$id' => ID::custom('database'),
'type' => Database::VAR_STRING,
'size' => 128,
'required' => true,
'signed' => true,
'array' => false,
'filters' => [],
]
],
'indexes' => [
[
+165
View File
@@ -0,0 +1,165 @@
<?php
use Utopia\Database\Database;
use Utopia\Database\Helpers\ID;
return [
'collections' => [
'$collection' => ID::custom('databases'),
'$id' => ID::custom('collections'),
'name' => 'Collections',
'attributes' => [
[
'$id' => ID::custom('databaseInternalId'),
'type' => Database::VAR_STRING,
'format' => '',
'size' => Database::LENGTH_KEY,
'signed' => true,
'required' => true,
'default' => null,
'array' => false,
'filters' => [],
],
[
'$id' => ID::custom('databaseId'),
'type' => Database::VAR_STRING,
'signed' => true,
'size' => Database::LENGTH_KEY,
'format' => '',
'filters' => [],
'required' => true,
'default' => null,
'array' => false,
],
[
'$id' => ID::custom('name'),
'type' => Database::VAR_STRING,
'size' => 256,
'required' => true,
'signed' => true,
'array' => false,
'filters' => [],
],
[
'$id' => ID::custom('dimension'),
'type' => Database::VAR_INTEGER,
'size' => 0,
'required' => true,
'signed' => false,
'array' => false,
'filters' => [],
],
[
'$id' => ID::custom('enabled'),
'type' => Database::VAR_BOOLEAN,
'signed' => true,
'size' => 0,
'format' => '',
'filters' => [],
'required' => true,
'default' => null,
'array' => false,
],
[
'$id' => ID::custom('documentSecurity'),
'type' => Database::VAR_BOOLEAN,
'signed' => true,
'size' => 0,
'format' => '',
'filters' => [],
'required' => true,
'default' => null,
'array' => false,
],
[
'$id' => ID::custom('attributes'),
'type' => Database::VAR_STRING,
'size' => 1000000,
'required' => false,
'signed' => true,
'array' => false,
'filters' => ['subQueryAttributes'],
],
[
'$id' => ID::custom('indexes'),
'type' => Database::VAR_STRING,
'size' => 1000000,
'required' => false,
'signed' => true,
'array' => false,
'filters' => ['subQueryIndexes'],
],
[
'$id' => ID::custom('search'),
'type' => Database::VAR_STRING,
'format' => '',
'size' => 16384,
'signed' => true,
'required' => false,
'default' => null,
'array' => false,
'filters' => [],
],
],
'defaultAttributes' => [
[
'$id' => ID::custom('embeddings'),
'type' => Database::VAR_VECTOR,
'required' => true,
'signed' => false,
'array' => false,
'filters' => [],
],
[
'$id' => ID::custom('metadata'),
'type' => Database::VAR_OBJECT,
'default' => [],
'required' => false,
'size' => 0,
'signed' => false,
'array' => false,
'filters' => [],
],
],
'indexes' => [
[
'$id' => ID::custom('_fulltext_search'),
'type' => Database::INDEX_FULLTEXT,
'attributes' => ['search'],
'lengths' => [],
'orders' => [],
],
[
'$id' => ID::custom('_key_name'),
'type' => Database::INDEX_KEY,
'attributes' => ['name'],
'lengths' => [256],
'orders' => [Database::ORDER_ASC],
],
[
'$id' => ID::custom('_key_enabled'),
'type' => Database::INDEX_KEY,
'attributes' => ['enabled'],
'lengths' => [],
'orders' => [Database::ORDER_ASC],
],
[
'$id' => ID::custom('_key_documentSecurity'),
'type' => Database::INDEX_KEY,
'attributes' => ['documentSecurity'],
'lengths' => [],
'orders' => [Database::ORDER_ASC],
],
],
'defaultIndexes' => [
// not creating default indexes on the embeddings as it depends on the type of query users using the most
[
'$id' => ID::custom('_key_metadata'),
'type' => Database::INDEX_OBJECT,
'attributes' => ['metadata'],
'lengths' => [],
'orders' => [],
],
]
]
];
+5
View File
@@ -1201,6 +1201,11 @@ return [
'description' => 'Migration is already in progress. You can check the status of the migration in your Appwrite Console\'s "Settings" > "Migrations".',
'code' => 409,
],
Exception::MIGRATION_DATABASE_TYPE_UNSUPPORTED => [
'name' => Exception::MIGRATION_DATABASE_TYPE_UNSUPPORTED,
'description' => 'The specified database type is not supported for CSV import or export operations.',
'code' => 400,
],
/** Realtime */
Exception::REALTIME_MESSAGE_FORMAT_INVALID => [
+9
View File
@@ -0,0 +1,9 @@
<?php
// Resource types available for console project migration keys
return [
'platforms' => 'platforms.read',
'devKeys' => 'devKeys.read',
'devKeysWrite' => 'devKeys.write',
];
+90 -7
View File
@@ -1,5 +1,6 @@
<?php
use Ahc\Jwt\JWT;
use Appwrite\Event\Event;
use Appwrite\Event\Migration;
use Appwrite\Extend\Exception;
@@ -14,7 +15,9 @@ use Appwrite\Utopia\Response;
use Utopia\Compression\Algorithms\GZIP;
use Utopia\Compression\Algorithms\Zstd;
use Utopia\Compression\Compression;
use Utopia\Config\Config;
use Utopia\Database\Database;
use Utopia\Database\DateTime;
use Utopia\Database\Document;
use Utopia\Database\Exception\Order as OrderException;
use Utopia\Database\Exception\Query as QueryException;
@@ -43,6 +46,16 @@ use Utopia\Validator\WhiteList;
include_once __DIR__ . '/../shared/api.php';
function getDatabaseTransferResourceServices(string $databaseType)
{
return match($databaseType) {
DATABASE_TYPE_LEGACY,
DATABASE_TYPE_TABLESDB => Transfer::GROUP_DATABASES_TABLES_DB,
DATABASE_TYPE_VECTORSDB => Transfer::GROUP_DATABASES_VECTOR_DB,
DATABASE_TYPE_DOCUMENTSDB => Transfer::GROUP_DATABASES_DOCUMENTS_DB
};
}
Http::post('/v1/migrations/appwrite')
->groups(['api', 'migrations'])
->desc('Create Appwrite migration')
@@ -427,8 +440,16 @@ Http::post('/v1/migrations/csv/imports')
throw new \Exception('Unable to copy file');
}
// getting databasetype
$resources = explode(':', $resourceId);
$databaseId = $resources[0];
$database = $authorization->skip(fn () => $dbForProject->getDocument('databases', $databaseId));
$databaseType = $database->getAttribute('type');
if (!in_array($databaseType, CSV_ALLOWED_DATABASE_TYPES)) {
throw new Exception(Exception::MIGRATION_DATABASE_TYPE_UNSUPPORTED, 'Database type not supported for csv');
}
$fileSize = $deviceForMigrations->getFileSize($newPath);
$resources = Transfer::extractServices([Transfer::GROUP_DATABASES]);
$resources = Transfer::extractServices([getDatabaseTransferResourceServices($databaseType)]);
$migration = $dbForProject->createDocument('migrations', new Document([
'$id' => $migrationId,
@@ -557,13 +578,23 @@ Http::post('/v1/migrations/csv/exports')
throw new Exception(Exception::GENERAL_QUERY_INVALID, $validator->getDescription());
}
// getting databasetype
$resources = explode(':', $resourceId);
$databaseId = $resources[0];
$database = $authorization->skip(fn () => $dbForProject->getDocument('databases', $databaseId));
$databaseType = $database->getAttribute('type');
if (!in_array($databaseType, CSV_ALLOWED_DATABASE_TYPES)) {
throw new Exception(Exception::MIGRATION_DATABASE_TYPE_UNSUPPORTED, 'Database type not supported for csv');
}
$resources = Transfer::extractServices([getDatabaseTransferResourceServices($databaseType)]);
$migration = $dbForProject->createDocument('migrations', new Document([
'$id' => ID::unique(),
'status' => 'pending',
'stage' => 'init',
'source' => Appwrite::getName(),
'destination' => CSV::getName(),
'resources' => Transfer::extractServices([Transfer::GROUP_DATABASES]),
'resources' => $resources,
'resourceId' => $resourceId,
'resourceType' => Resource::TYPE_DATABASE,
'statusCounters' => '{}',
@@ -691,6 +722,59 @@ Http::get('/v1/migrations/:migrationId')
$response->dynamic($migration, Response::MODEL_MIGRATION);
});
Http::post('/v1/migrations/appwrite/console-key')
->groups(['api', 'migrations'])
->desc('Create console API key for migration')
->label('scope', 'migrations.write')
->label('sdk', new Method(
namespace: 'migrations',
group: null,
name: 'createAppwriteConsoleKey',
description: '/docs/references/migrations/migration-appwrite-console-key.md',
auth: [AuthType::KEY],
responses: [
new SDKResponse(
code: Response::STATUS_CODE_OK,
model: Response::MODEL_MIGRATION_KEY,
)
]
))
->param('resources', [], new ArrayList(new WhiteList(\array_keys(Config::getParam('consoleProjectScopes')))), 'List of resource types to request access for.', true)
->inject('response')
->inject('project')
->action(function (array $resources, Response $response, Document $project) {
$consoleProjectScopes = Config::getParam('consoleProjectScopes');
$scopes = empty($resources)
? \array_values($consoleProjectScopes)
: \array_values(\array_intersect_key($consoleProjectScopes, \array_flip($resources)));
if (empty($scopes)) {
throw new Exception(Exception::GENERAL_UNAUTHORIZED_SCOPE);
}
$jwt = new JWT(System::getEnv('_APP_OPENSSL_KEY_V1'), 'HS256', APP_CONSOLE_KEY_TTL, 0);
$consoleKey = $jwt->encode([
'projectId' => 'console',
'name' => 'Migration Settings Key',
'source' => KEY_SOURCE_MIGRATION,
'scopes' => $scopes,
'disabledMetrics' => [
METRIC_DATABASES_OPERATIONS_READS,
METRIC_DATABASES_OPERATIONS_WRITES,
METRIC_NETWORK_REQUESTS,
METRIC_NETWORK_INBOUND,
METRIC_NETWORK_OUTBOUND,
],
'scopedProjectId' => $project->getId(),
]);
$response->dynamic(new Document([
'key' => API_KEY_DYNAMIC . '_' . $consoleKey,
'expire' => DateTime::addSeconds(new \DateTime(), APP_CONSOLE_KEY_TTL),
]), Response::MODEL_MIGRATION_KEY);
});
Http::get('/v1/migrations/appwrite/report')
->groups(['api', 'migrations'])
->desc('Get Appwrite migration report')
@@ -713,12 +797,11 @@ Http::get('/v1/migrations/appwrite/report')
->param('projectID', '', new Text(512), "Source's Project ID")
->param('key', '', new Text(512), "Source's API Key")
->inject('response')
->inject('dbForProject')
->inject('project')
->inject('user')
->action(function (array $resources, string $endpoint, string $projectID, string $key, Response $response) {
->inject('getDatabasesDB')
->action(function (array $resources, string $endpoint, string $projectID, string $key, Response $response, callable $getDatabasesDB) {
try {
$appwrite = new Appwrite($projectID, $endpoint, $key);
$appwrite = new Appwrite($projectID, $endpoint, $key, $getDatabasesDB);
$report = $appwrite->report($resources);
} catch (\Throwable $e) {
throw new Exception(
+60
View File
@@ -62,16 +62,33 @@ Http::get('/v1/project/usage')
METRIC_EXECUTIONS_MB_SECONDS,
METRIC_BUILDS_MB_SECONDS,
METRIC_DOCUMENTS,
METRIC_DOCUMENTS_DOCUMENTSDB,
METRIC_DATABASES,
METRIC_DATABASES_DOCUMENTSDB,
METRIC_USERS,
METRIC_BUCKETS,
METRIC_FILES_STORAGE,
METRIC_DATABASES_STORAGE,
METRIC_DATABASES_STORAGE_DOCUMENTSDB,
METRIC_DEPLOYMENTS_STORAGE,
METRIC_BUILDS_STORAGE,
METRIC_DATABASES_OPERATIONS_READS,
METRIC_DATABASES_OPERATIONS_READS_DOCUMENTSDB,
METRIC_DATABASES_OPERATIONS_WRITES,
METRIC_DATABASES_OPERATIONS_WRITES_DOCUMENTSDB,
METRIC_FILES_IMAGES_TRANSFORMED,
// VectorsDB totals
METRIC_DATABASES_VECTORSDB,
METRIC_COLLECTIONS_VECTORSDB,
METRIC_DOCUMENTS_VECTORSDB,
METRIC_DATABASES_STORAGE_VECTORSDB,
METRIC_DATABASES_OPERATIONS_READS_VECTORSDB,
METRIC_DATABASES_OPERATIONS_WRITES_VECTORSDB,
// Embeddings totals
METRIC_EMBEDDINGS_TEXT,
METRIC_EMBEDDINGS_TEXT_TOTAL_TOKENS,
METRIC_EMBEDDINGS_TEXT_TOTAL_DURATION,
METRIC_EMBEDDINGS_TEXT_TOTAL_ERROR
],
'period' => [
METRIC_NETWORK_REQUESTS,
@@ -80,11 +97,26 @@ Http::get('/v1/project/usage')
METRIC_USERS,
METRIC_EXECUTIONS,
METRIC_DATABASES_STORAGE,
METRIC_DATABASES_STORAGE_DOCUMENTSDB,
METRIC_EXECUTIONS_MB_SECONDS,
METRIC_BUILDS_MB_SECONDS,
METRIC_DATABASES_OPERATIONS_READS,
METRIC_DATABASES_OPERATIONS_READS_DOCUMENTSDB,
METRIC_DATABASES_OPERATIONS_WRITES,
METRIC_DATABASES_OPERATIONS_WRITES_DOCUMENTSDB,
METRIC_FILES_IMAGES_TRANSFORMED,
// VectorsDB time series
METRIC_DATABASES_VECTORSDB,
METRIC_COLLECTIONS_VECTORSDB,
METRIC_DOCUMENTS_VECTORSDB,
METRIC_DATABASES_STORAGE_VECTORSDB,
METRIC_DATABASES_OPERATIONS_READS_VECTORSDB,
METRIC_DATABASES_OPERATIONS_WRITES_VECTORSDB,
// Embeddings time series
METRIC_EMBEDDINGS_TEXT,
METRIC_EMBEDDINGS_TEXT_TOTAL_TOKENS,
METRIC_EMBEDDINGS_TEXT_TOTAL_DURATION,
METRIC_EMBEDDINGS_TEXT_TOTAL_ERROR
]
];
@@ -357,8 +389,11 @@ Http::get('/v1/project/usage')
'buildsMbSecondsTotal' => $total[METRIC_BUILDS_MB_SECONDS],
'documentsTotal' => $total[METRIC_DOCUMENTS],
'rowsTotal' => $total[METRIC_DOCUMENTS],
'documentsdbDocumentsTotal' => $total[METRIC_DOCUMENTS_DOCUMENTSDB],
'databasesTotal' => $total[METRIC_DATABASES],
'documentsdbTotal' => $total[METRIC_DATABASES_DOCUMENTSDB],
'databasesStorageTotal' => $total[METRIC_DATABASES_STORAGE],
'documentsdbDatabasesStorageTotal' => $total[METRIC_DATABASES_STORAGE_DOCUMENTSDB],
'usersTotal' => $total[METRIC_USERS],
'bucketsTotal' => $total[METRIC_BUCKETS],
'filesStorageTotal' => $total[METRIC_FILES_STORAGE],
@@ -367,10 +402,27 @@ Http::get('/v1/project/usage')
'deploymentsStorageTotal' => $total[METRIC_DEPLOYMENTS_STORAGE],
'databasesReadsTotal' => $total[METRIC_DATABASES_OPERATIONS_READS],
'databasesWritesTotal' => $total[METRIC_DATABASES_OPERATIONS_WRITES],
'documentsdbDatabasesReadsTotal' => $total[METRIC_DATABASES_OPERATIONS_READS_DOCUMENTSDB],
'documentsdbDatabasesWritesTotal' => $total[METRIC_DATABASES_OPERATIONS_WRITES_DOCUMENTSDB],
'vectorsdbDatabasesTotal' => $total[METRIC_DATABASES_VECTORSDB] ?? 0,
'vectorsdbCollectionsTotal' => $total[METRIC_COLLECTIONS_VECTORSDB] ?? 0,
'vectorsdbDocumentsTotal' => $total[METRIC_DOCUMENTS_VECTORSDB] ?? 0,
'vectorsdbDatabasesStorageTotal' => $total[METRIC_DATABASES_STORAGE_VECTORSDB] ?? 0,
'vectorsdbDatabasesReadsTotal' => $total[METRIC_DATABASES_OPERATIONS_READS_VECTORSDB] ?? 0,
'vectorsdbDatabasesWritesTotal' => $total[METRIC_DATABASES_OPERATIONS_WRITES_VECTORSDB] ?? 0,
'executionsBreakdown' => $executionsBreakdown,
'bucketsBreakdown' => $bucketsBreakdown,
'databasesReads' => $usage[METRIC_DATABASES_OPERATIONS_READS],
'databasesWrites' => $usage[METRIC_DATABASES_OPERATIONS_WRITES],
'documentsdbDatabasesReads' => $usage[METRIC_DATABASES_OPERATIONS_READS_DOCUMENTSDB],
'documentsdbDatabasesWrites' => $usage[METRIC_DATABASES_OPERATIONS_WRITES_DOCUMENTSDB],
'documentsdbDatabasesStorage' => $usage[METRIC_DATABASES_STORAGE_DOCUMENTSDB],
'vectorsdbDatabases' => $usage[METRIC_DATABASES_VECTORSDB] ?? [],
'vectorsdbCollections' => $usage[METRIC_COLLECTIONS_VECTORSDB] ?? [],
'vectorsdbDocuments' => $usage[METRIC_DOCUMENTS_VECTORSDB] ?? [],
'vectorsdbDatabasesStorage' => $usage[METRIC_DATABASES_STORAGE_VECTORSDB] ?? [],
'vectorsdbDatabasesReads' => $usage[METRIC_DATABASES_OPERATIONS_READS_VECTORSDB] ?? [],
'vectorsdbDatabasesWrites' => $usage[METRIC_DATABASES_OPERATIONS_WRITES_VECTORSDB] ?? [],
'databasesStorageBreakdown' => $databasesStorageBreakdown,
'executionsMbSecondsBreakdown' => $executionsMbSecondsBreakdown,
'buildsMbSecondsBreakdown' => $buildsMbSecondsBreakdown,
@@ -380,6 +432,14 @@ Http::get('/v1/project/usage')
'authPhoneCountryBreakdown' => $authPhoneCountryBreakdown,
'imageTransformations' => $usage[METRIC_FILES_IMAGES_TRANSFORMED],
'imageTransformationsTotal' => $total[METRIC_FILES_IMAGES_TRANSFORMED],
'embeddingsText' => $usage[METRIC_EMBEDDINGS_TEXT] ?? [],
'embeddingsTextTokens' => $usage[METRIC_EMBEDDINGS_TEXT_TOTAL_TOKENS] ?? [],
'embeddingsTextDuration' => $usage[METRIC_EMBEDDINGS_TEXT_TOTAL_DURATION] ?? [],
'embeddingsTextErrors' => $usage[METRIC_EMBEDDINGS_TEXT_TOTAL_ERROR] ?? [],
'embeddingsTextTotal' => $total[METRIC_EMBEDDINGS_TEXT] ?? 0,
'embeddingsTextTokensTotal' => $total[METRIC_EMBEDDINGS_TEXT_TOTAL_TOKENS] ?? 0,
'embeddingsTextDurationTotal' => $total[METRIC_EMBEDDINGS_TEXT_TOTAL_DURATION] ?? 0,
'embeddingsTextErrorsTotal' => $total[METRIC_EMBEDDINGS_TEXT_TOTAL_ERROR] ?? 0,
]), Response::MODEL_USAGE_PROJECT);
});
+23
View File
@@ -338,6 +338,23 @@ Http::init()
$scopes = \array_unique($scopes);
// Migration-sourced keys are scoped to a single project's console
// endpoints (e.g. /v1/projects/:projectId/platforms). Verify the
// URL :projectId matches the token's scopedProjectId.
if (!empty($apiKey) && $apiKey->getSource() === KEY_SOURCE_MIGRATION) {
$scopedProjectId = $apiKey->getScopedProjectId();
if (empty($scopedProjectId)) {
throw new Exception(Exception::GENERAL_UNAUTHORIZED_SCOPE);
}
$pathValues = $route->getPathValues($request);
if (($pathValues['projectId'] ?? '') !== $scopedProjectId) {
throw new Exception(Exception::GENERAL_UNAUTHORIZED_SCOPE);
}
}
$authorization->addRole($role);
foreach ($user->getRoles($authorization) as $authRole) {
$authorization->addRole($authRole);
@@ -470,6 +487,12 @@ Http::init()
->action(function (Http $utopia, Request $request, Response $response, Document $project, Document $user, Event $queueForEvents, Messaging $queueForMessaging, Audit $queueForAudits, Delete $queueForDeletes, EventDatabase $queueForDatabase, Build $queueForBuilds, StatsUsage $queueForStatsUsage, Func $queueForFunctions, Mail $queueForMails, Database $dbForProject, callable $timelimit, Document $resourceToken, string $mode, ?Key $apiKey, array $plan, Document $devKey, Telemetry $telemetry, array $platform, Authorization $authorization) {
$route = $utopia->getRoute();
$path = $route->getMatchedPath();
$databaseType = match (true) {
str_contains($path, '/documentsdb') => DATABASE_TYPE_DOCUMENTSDB,
str_contains($path, '/vectorsdb') => DATABASE_TYPE_VECTORSDB,
default => '',
};
if (
array_key_exists('rest', $project->getAttribute('apis', []))
+19 -1
View File
@@ -196,6 +196,8 @@ include __DIR__ . '/controllers/general.php';
function createDatabase(Http $app, string $resourceKey, string $dbName, array $collections, mixed $pools, ?callable $extraSetup = null): void
{
$max = 15;
$sleep = 2;
$max = 15;
$sleep = 2;
$attempts = 0;
@@ -409,13 +411,29 @@ $http->on(Constant::EVENT_START, function (Server $http) use ($payloadSize, $tot
});
$projectCollections = $collections['projects'];
$sharedTables = \explode(',', System::getEnv('_APP_DATABASE_SHARED_TABLES', ''));
$sharedTablesV1 = \explode(',', System::getEnv('_APP_DATABASE_SHARED_TABLES_V1', ''));
$sharedTablesV2 = \array_diff($sharedTables, $sharedTablesV1);
$documentsSharedTables = \explode(',', System::getEnv('_APP_DATABASE_DOCUMENTSDB_SHARED_TABLES', ''));
$documentsSharedTablesV1 = \explode(',', System::getEnv('_APP_DATABASE_DOCUMENTSDB_SHARED_TABLES_V1', ''));
$documentsSharedTablesV2 = \array_diff($documentsSharedTables, $documentsSharedTablesV1);
$vectorSharedTables = \explode(',', System::getEnv('_APP_DATABASE_VECTORSDB_SHARED_TABLES', ''));
$vectorSharedTablesV1 = \explode(',', System::getEnv('_APP_DATABASE_VECTORSDB_SHARED_TABLES_V1', ''));
$vectorSharedTablesV2 = \array_diff($vectorSharedTables, $vectorSharedTablesV1);
$cache = $app->getResource('cache');
foreach ($sharedTablesV2 as $hostname) {
// All shared tables V2 pools that need project metadata collections
$sharedTablesV2All = \array_values(\array_unique(\array_filter([
...$sharedTablesV2,
...$documentsSharedTablesV2,
...$vectorSharedTablesV2,
])));
foreach ($sharedTablesV2All as $hostname) {
Span::init('database.setup');
Span::add('database.hostname', $hostname);
+1
View File
@@ -25,6 +25,7 @@ Config::load('roles', __DIR__ . '/../config/roles.php', $configAdapter); // Use
Config::load('projectScopes', __DIR__ . '/../config/scopes/project.php', $configAdapter);
Config::load('organizationScopes', __DIR__ . '/../config/scopes/organization.php', $configAdapter);
Config::load('accountScopes', __DIR__ . '/../config/scopes/account.php', $configAdapter);
Config::load('consoleProjectScopes', __DIR__ . '/../config/scopes/consoleProject.php', $configAdapter);
Config::load('services', __DIR__ . '/../config/services.php', $configAdapter); // List of services
Config::load('variables', __DIR__ . '/../config/variables.php', $configAdapter); // List of env variables
Config::load('regions', __DIR__ . '/../config/regions.php', $configAdapter); // List of available regions
+56
View File
@@ -41,6 +41,7 @@ const APP_LIMIT_LIST_DEFAULT = 25; // Default maximum number of items to return
const APP_LIMIT_DATABASE_BATCH = 100; // Default maximum batch size for database operations
const APP_LIMIT_DATABASE_TRANSACTION = 100; // Default maximum operations per transaction
const APP_KEY_ACCESS = 24 * 60 * 60; // 24 hours
const APP_CONSOLE_KEY_TTL = 120; // 2 minutes
const APP_USER_ACCESS = 24 * 60 * 60; // 24 hours
const APP_PROJECT_ACCESS = 24 * 60 * 60; // 24 hours
const APP_RESOURCE_TOKEN_ACCESS = 24 * 60 * 60; // 24 hours
@@ -256,6 +257,8 @@ const API_KEY_STANDARD = 'standard';
const API_KEY_DYNAMIC = 'dynamic';
const API_KEY_ORGANIZATION = 'organization';
const API_KEY_ACCOUNT = 'account';
// API key source identifiers
const KEY_SOURCE_MIGRATION = 'migration';
// Usage metrics
const METRIC_TEAMS = 'teams';
const METRIC_USERS = 'users';
@@ -288,6 +291,45 @@ const METRIC_DATABASES_OPERATIONS_READS = 'databases.operations.reads';
const METRIC_DATABASE_ID_OPERATIONS_READS = '{databaseInternalId}.databases.operations.reads';
const METRIC_DATABASES_OPERATIONS_WRITES = 'databases.operations.writes';
const METRIC_DATABASE_ID_OPERATIONS_WRITES = '{databaseInternalId}.databases.operations.writes';
// documentsdb
const METRIC_DATABASES_DOCUMENTSDB = 'documentsdb.databases';
const METRIC_COLLECTIONS_DOCUMENTSDB = 'documentsdb.collections';
const METRIC_DATABASES_STORAGE_DOCUMENTSDB = 'documentsdb.databases.storage';
const METRIC_DATABASE_ID_COLLECTIONS_DOCUMENTSDB = 'documentsdb.{databaseInternalId}.collections';
const METRIC_DATABASE_ID_STORAGE_DOCUMENTSDB = 'documentsdb.{databaseInternalId}.databases.storage';
const METRIC_DOCUMENTS_DOCUMENTSDB = 'documentsdb.documents';
const METRIC_DATABASE_ID_DOCUMENTS_DOCUMENTSDB = 'documentsdb.{databaseInternalId}.documents';
const METRIC_DATABASE_ID_COLLECTION_ID_DOCUMENTS_DOCUMENTSDB = 'documentsdb.{databaseInternalId}.{collectionInternalId}.documents';
const METRIC_DATABASE_ID_COLLECTION_ID_STORAGE_DOCUMENTSDB = 'documentsdb.{databaseInternalId}.{collectionInternalId}.databases.storage';
const METRIC_DATABASES_OPERATIONS_READS_DOCUMENTSDB = 'documentsdb.databases.operations.reads';
const METRIC_DATABASE_ID_OPERATIONS_READS_DOCUMENTSDB = 'documentsdb.{databaseInternalId}.databases.operations.reads';
const METRIC_DATABASES_OPERATIONS_WRITES_DOCUMENTSDB = 'documentsdb.databases.operations.writes';
const METRIC_DATABASE_ID_OPERATIONS_WRITES_DOCUMENTSDB = 'documentsdb.{databaseInternalId}.databases.operations.writes';
// vectorsdb
const METRIC_DATABASES_VECTORSDB = 'vectorsdb.databases';
const METRIC_COLLECTIONS_VECTORSDB = 'vectorsdb.collections';
const METRIC_DATABASES_STORAGE_VECTORSDB = 'vectorsdb.databases.storage';
const METRIC_DATABASE_ID_COLLECTIONS_VECTORSDB = 'vectorsdb.{databaseInternalId}.collections';
const METRIC_DATABASE_ID_STORAGE_VECTORSDB = 'vectorsdb.{databaseInternalId}.databases.storage';
const METRIC_DOCUMENTS_VECTORSDB = 'vectorsdb.documents';
const METRIC_DATABASE_ID_DOCUMENTS_VECTORSDB = 'vectorsdb.{databaseInternalId}.documents';
const METRIC_DATABASE_ID_COLLECTION_ID_DOCUMENTS_VECTORSDB = 'vectorsdb.{databaseInternalId}.{collectionInternalId}.documents';
const METRIC_DATABASE_ID_COLLECTION_ID_STORAGE_VECTORSDB = 'vectorsdb.{databaseInternalId}.{collectionInternalId}.databases.storage';
const METRIC_DATABASES_OPERATIONS_READS_VECTORSDB = 'vectorsdb.databases.operations.reads';
const METRIC_DATABASE_ID_OPERATIONS_READS_VECTORSDB = 'vectorsdb.{databaseInternalId}.databases.operations.reads';
const METRIC_DATABASES_OPERATIONS_WRITES_VECTORSDB = 'vectorsdb.databases.operations.writes';
const METRIC_DATABASE_ID_OPERATIONS_WRITES_VECTORSDB = 'vectorsdb.{databaseInternalId}.databases.operations.writes';
const METRIC_EMBEDDINGS_TEXT = 'embeddings.text';
const METRIC_EMBEDDINGS_MODEL_TEXT = 'embeddings.text.{embeddingModel}';
const METRIC_EMBEDDINGS_TEXT_TOTAL_ERROR = 'embeddings.text.totalErrors';
const METRIC_EMBEDDINGS_MODEL_TEXT_TOTAL_ERROR = 'embeddings.text.{embeddingModel}.totalErrors';
const METRIC_EMBEDDINGS_TEXT_TOTAL_DURATION = 'embeddings.text.totalDuration';
const METRIC_EMBEDDINGS_MODEL_TEXT_TOTAL_DURATION = 'embeddings.text.{embeddingModel}.totalDuration';
const METRIC_EMBEDDINGS_TEXT_TOTAL_TOKENS = 'embeddings.text.totalTokens';
const METRIC_EMBEDDINGS_MODEL_TEXT_TOTAL_TOKENS = 'embeddings.text.{embeddingModel}.totalTokens';
const METRIC_BUCKETS = 'buckets';
const METRIC_FILES = 'files';
const METRIC_FILES_STORAGE = 'files.storage';
@@ -380,6 +422,7 @@ const RESOURCE_TYPE_SUBSCRIBERS = 'subscribers';
const RESOURCE_TYPE_MESSAGES = 'messages';
const RESOURCE_TYPE_EXECUTIONS = 'executions';
const RESOURCE_TYPE_VCS = 'vcs';
const RESOURCE_TYPE_EMBEDDINGS_TEXT = 'embeddingsText';
// Resource types for Tokens
const TOKENS_RESOURCE_TYPE_FILES = 'files';
@@ -401,3 +444,16 @@ const CACHE_RECONNECT_RETRY_DELAY = 1000;
// Project status
const PROJECT_STATUS_ACTIVE = 'active';
// Database types
const DATABASE_TYPE_LEGACY = 'legacy';
const DATABASE_TYPE_TABLESDB = 'tablesdb';
const DATABASE_TYPE_DOCUMENTSDB = 'documentsdb';
const DATABASE_TYPE_VECTORSDB = 'vectorsdb';
// CSV import/export allowed database types
const CSV_ALLOWED_DATABASE_TYPES = [
DATABASE_TYPE_LEGACY,
DATABASE_TYPE_TABLESDB,
DATABASE_TYPE_VECTORSDB
];
+24
View File
@@ -22,6 +22,7 @@ use Appwrite\Utopia\Response\Model\AttributeLine;
use Appwrite\Utopia\Response\Model\AttributeList;
use Appwrite\Utopia\Response\Model\AttributeLongtext;
use Appwrite\Utopia\Response\Model\AttributeMediumtext;
use Appwrite\Utopia\Response\Model\AttributeObject;
use Appwrite\Utopia\Response\Model\AttributePoint;
use Appwrite\Utopia\Response\Model\AttributePolygon;
use Appwrite\Utopia\Response\Model\AttributeRelationship;
@@ -29,6 +30,7 @@ use Appwrite\Utopia\Response\Model\AttributeString;
use Appwrite\Utopia\Response\Model\AttributeText;
use Appwrite\Utopia\Response\Model\AttributeURL;
use Appwrite\Utopia\Response\Model\AttributeVarchar;
use Appwrite\Utopia\Response\Model\AttributeVector;
use Appwrite\Utopia\Response\Model\AuthProvider;
use Appwrite\Utopia\Response\Model\BaseList;
use Appwrite\Utopia\Response\Model\Branch;
@@ -65,6 +67,7 @@ use Appwrite\Utopia\Response\Model\DetectionRuntime;
use Appwrite\Utopia\Response\Model\DetectionVariable;
use Appwrite\Utopia\Response\Model\DevKey;
use Appwrite\Utopia\Response\Model\Document as ModelDocument;
use Appwrite\Utopia\Response\Model\Embedding;
use Appwrite\Utopia\Response\Model\Error;
use Appwrite\Utopia\Response\Model\ErrorDev;
use Appwrite\Utopia\Response\Model\Execution;
@@ -98,6 +101,7 @@ use Appwrite\Utopia\Response\Model\MFARecoveryCodes;
use Appwrite\Utopia\Response\Model\MFAType;
use Appwrite\Utopia\Response\Model\Migration;
use Appwrite\Utopia\Response\Model\MigrationFirebaseProject;
use Appwrite\Utopia\Response\Model\MigrationKey;
use Appwrite\Utopia\Response\Model\MigrationReport;
use Appwrite\Utopia\Response\Model\Mock;
use Appwrite\Utopia\Response\Model\MockNumber;
@@ -136,6 +140,8 @@ use Appwrite\Utopia\Response\Model\UsageBuckets;
use Appwrite\Utopia\Response\Model\UsageCollection;
use Appwrite\Utopia\Response\Model\UsageDatabase;
use Appwrite\Utopia\Response\Model\UsageDatabases;
use Appwrite\Utopia\Response\Model\UsageDocumentsDB;
use Appwrite\Utopia\Response\Model\UsageDocumentsDBs;
use Appwrite\Utopia\Response\Model\UsageFunction;
use Appwrite\Utopia\Response\Model\UsageFunctions;
use Appwrite\Utopia\Response\Model\UsageProject;
@@ -144,9 +150,12 @@ use Appwrite\Utopia\Response\Model\UsageSites;
use Appwrite\Utopia\Response\Model\UsageStorage;
use Appwrite\Utopia\Response\Model\UsageTable;
use Appwrite\Utopia\Response\Model\UsageUsers;
use Appwrite\Utopia\Response\Model\UsageVectorsDB;
use Appwrite\Utopia\Response\Model\UsageVectorsDBs;
use Appwrite\Utopia\Response\Model\User;
use Appwrite\Utopia\Response\Model\Variable;
use Appwrite\Utopia\Response\Model\VcsContent;
use Appwrite\Utopia\Response\Model\VectorsDBCollection;
use Appwrite\Utopia\Response\Model\Webhook;
// General
@@ -211,9 +220,12 @@ Response::setModel(new BaseList('Migrations List', Response::MODEL_MIGRATION_LIS
Response::setModel(new BaseList('Migrations Firebase Projects List', Response::MODEL_MIGRATION_FIREBASE_PROJECT_LIST, 'projects', Response::MODEL_MIGRATION_FIREBASE_PROJECT));
Response::setModel(new BaseList('Specifications List', Response::MODEL_SPECIFICATION_LIST, 'specifications', Response::MODEL_SPECIFICATION));
Response::setModel(new BaseList('VCS Content List', Response::MODEL_VCS_CONTENT_LIST, 'contents', Response::MODEL_VCS_CONTENT));
Response::setModel(new BaseList('VectorsDB Collections List', Response::MODEL_VECTORSDB_COLLECTION_LIST, 'collections', Response::MODEL_VECTORSDB_COLLECTION));
Response::setModel(new BaseList('Embedding list', Response::MODEL_EMBEDDING_LIST, 'embeddings', Response::MODEL_EMBEDDING));
// Entities
Response::setModel(new Database());
Response::setModel(new Embedding());
// Collection API Models
Response::setModel(new Collection());
@@ -237,6 +249,17 @@ Response::setModel(new AttributeText());
Response::setModel(new AttributeMediumtext());
Response::setModel(new AttributeLongtext());
// DocumentsDB API Models
Response::setModel(new UsageDocumentsDBs());
Response::setModel(new UsageDocumentsDB());
// VectorsDB API Models
Response::setModel(new VectorsDBCollection());
Response::setModel(new AttributeObject());
Response::setModel(new AttributeVector());
Response::setModel(new UsageVectorsDBs());
Response::setModel(new UsageVectorsDB());
// Table API Models
Response::setModel(new Table());
Response::setModel(new Column());
@@ -357,6 +380,7 @@ Response::setModel(new Subscriber());
Response::setModel(new Target());
Response::setModel(new Migration());
Response::setModel(new MigrationReport());
Response::setModel(new MigrationKey());
Response::setModel(new MigrationFirebaseProject());
// Tests (keep last)
+31 -3
View File
@@ -160,7 +160,6 @@ $register->set('pools', function () {
'pass' => System::getEnv('_APP_DB_PASS', ''),
'path' => System::getEnv('_APP_DB_SCHEMA', ''),
]);
$fallbackForRedis = 'redis_main=' . AppwriteURL::unparse([
'scheme' => 'redis',
'host' => System::getEnv('_APP_REDIS_HOST', 'redis'),
@@ -169,6 +168,23 @@ $register->set('pools', function () {
'pass' => System::getEnv('_APP_REDIS_PASS', ''),
]);
$fallbackForDocumentsDB = 'db_main=' . AppwriteURL::unparse([
'scheme' => System::getEnv('_APP_DB_ADAPTER_DOCUMENTSDB', 'mongodb'),
'host' => System::getEnv('_APP_DB_HOST_DOCUMENTSDB', 'mongodb'),
'port' => System::getEnv('_APP_DB_PORT_DOCUMENTSDB', '27017'),
'user' => System::getEnv('_APP_DB_USER', ''),
'pass' => System::getEnv('_APP_DB_PASS', ''),
'path' => System::getEnv('_APP_DB_SCHEMA', ''),
]);
$fallbackForVectorsDB = 'db_main=' . AppwriteURL::unparse([
'scheme' => System::getEnv('_APP_DB_ADAPTER_VECTORSDB', 'postgresql'),
'host' => System::getEnv('_APP_DB_HOST_VECTORSDB', 'postgresql'),
'port' => System::getEnv('_APP_DB_PORT_VECTORSDB', '5432'),
'user' => System::getEnv('_APP_DB_USER', ''),
'pass' => System::getEnv('_APP_DB_PASS', ''),
'path' => System::getEnv('_APP_DB_SCHEMA', ''),
]);
$connections = [
'console' => [
'type' => 'database',
@@ -180,13 +196,25 @@ $register->set('pools', function () {
'type' => 'database',
'dsns' => $fallbackForDB,
'multiple' => true,
'schemes' => ['mariadb', 'mongodb', 'mysql', 'postgresql'],
'schemes' => ['mongodb','mariadb', 'mysql','postgresql'],
],
'documentsdb' => [
'type' => 'database',
'dsns' => System::getEnv('_APP_CONNECTIONS_DATABASE_DOCUMENTSDB', $fallbackForDocumentsDB),
'multiple' => true,
'schemes' => ['mongodb'],
],
'vectorsdb' => [
'type' => 'database',
'dsns' => System::getEnv('_APP_CONNECTIONS_DATABASE_VECTORSDB', $fallbackForVectorsDB),
'multiple' => true,
'schemes' => ['postgresql'],
],
'logs' => [
'type' => 'database',
'dsns' => System::getEnv('_APP_CONNECTIONS_DB_LOGS', $fallbackForDB),
'multiple' => false,
'schemes' => ['mariadb', 'mongodb', 'mysql', 'postgresql'],
'schemes' => ['mongodb','mariadb', 'mysql','postgresql'],
],
'publisher' => [
'type' => 'publisher',
+186 -12
View File
@@ -31,6 +31,8 @@ use Appwrite\Utopia\Request;
use Appwrite\Utopia\Response;
use Executor\Executor;
use Utopia\Abuse\Adapters\TimeLimit\Redis as TimeLimitRedis;
use Utopia\Agents\Adapters\Ollama;
use Utopia\Agents\Agent;
use Utopia\Audit\Adapter\Database as AdapterDatabase;
use Utopia\Audit\Audit;
use Utopia\Auth\Hashes\Argon2;
@@ -550,7 +552,7 @@ Http::setResource('authorization', function () {
return new Authorization();
}, []);
Http::setResource('dbForProject', function (Group $pools, Database $dbForPlatform, Cache $cache, Document $project, Response $response, Publisher $publisher, Publisher $publisherFunctions, Publisher $publisherWebhooks, Event $queueForEvents, Func $queueForFunctions, Webhook $queueForWebhooks, Realtime $queueForRealtime, StatsUsage $queueForStatsUsage, Authorization $authorization) {
Http::setResource('dbForProject', function (Group $pools, Database $dbForPlatform, Cache $cache, Document $project, Response $response, Publisher $publisher, Publisher $publisherFunctions, Publisher $publisherWebhooks, Event $queueForEvents, Func $queueForFunctions, Webhook $queueForWebhooks, Realtime $queueForRealtime, StatsUsage $queueForStatsUsage, Authorization $authorization, Request $request) {
if ($project->isEmpty() || $project->getId() === 'console') {
return $dbForPlatform;
}
@@ -658,7 +660,31 @@ Http::setResource('dbForProject', function (Group $pools, Database $dbForPlatfor
$dbForProject->getCache()->purge($cacheKey);
};
$usageDatabaseListener = function (string $event, Document $document, StatsUsage $queueForStatsUsage) {
/**
* Prefix metrics with database type when applicable.
* Avoids prefixing for legacy and tablesdb types to preserve historical metrics.
*/
$getDatabaseTypePrefixedMetric = function (string $databaseType, string $metric): string {
if (
$databaseType === '' ||
$databaseType === DATABASE_TYPE_LEGACY ||
$databaseType === DATABASE_TYPE_TABLESDB
) {
return $metric;
}
return $databaseType . '.' . $metric;
};
// Determine database type from request path, similar to api.php
$path = $request->getURI();
$databaseType = match (true) {
str_contains($path, '/documentsdb') => DATABASE_TYPE_DOCUMENTSDB,
str_contains($path, '/vectorsdb') => DATABASE_TYPE_VECTORSDB,
default => '',
};
$usageDatabaseListener = function (string $event, Document $document, StatsUsage $queueForStatsUsage) use ($getDatabaseTypePrefixedMetric, $databaseType) {
$value = 1;
switch ($event) {
@@ -690,7 +716,8 @@ Http::setResource('dbForProject', function (Group $pools, Database $dbForPlatfor
$queueForStatsUsage->addMetric(METRIC_SESSIONS, $value); //per project
break;
case $document->getCollection() === 'databases': // databases
$queueForStatsUsage->addMetric(METRIC_DATABASES, $value); // per project
$metric = $getDatabaseTypePrefixedMetric($databaseType, METRIC_DATABASES);
$queueForStatsUsage->addMetric($metric, $value); // per project
if ($event === Database::EVENT_DOCUMENT_DELETE) {
$queueForStatsUsage->addReduce($document);
@@ -699,9 +726,11 @@ Http::setResource('dbForProject', function (Group $pools, Database $dbForPlatfor
case str_starts_with($document->getCollection(), 'database_') && !str_contains($document->getCollection(), 'collection'): //collections
$parts = explode('_', $document->getCollection());
$databaseInternalId = $parts[1] ?? 0;
$collectionMetric = $getDatabaseTypePrefixedMetric($databaseType, METRIC_COLLECTIONS);
$databaseIdCollectionMetric = $getDatabaseTypePrefixedMetric($databaseType, METRIC_DATABASE_ID_COLLECTIONS);
$queueForStatsUsage
->addMetric(METRIC_COLLECTIONS, $value) // per project
->addMetric(str_replace('{databaseInternalId}', $databaseInternalId, METRIC_DATABASE_ID_COLLECTIONS), $value);
->addMetric($collectionMetric, $value) // per project
->addMetric(str_replace('{databaseInternalId}', $databaseInternalId, $databaseIdCollectionMetric), $value);
if ($event === Database::EVENT_DOCUMENT_DELETE) {
$queueForStatsUsage->addReduce($document);
@@ -711,10 +740,13 @@ Http::setResource('dbForProject', function (Group $pools, Database $dbForPlatfor
$parts = explode('_', $document->getCollection());
$databaseInternalId = $parts[1] ?? 0;
$collectionInternalId = $parts[3] ?? 0;
$documentsMetric = $getDatabaseTypePrefixedMetric($databaseType, METRIC_DOCUMENTS);
$databaseIdDocumentsMetric = $getDatabaseTypePrefixedMetric($databaseType, METRIC_DATABASE_ID_DOCUMENTS);
$databaseIdCollectionIdDocumentsMetric = $getDatabaseTypePrefixedMetric($databaseType, METRIC_DATABASE_ID_COLLECTION_ID_DOCUMENTS);
$queueForStatsUsage
->addMetric(METRIC_DOCUMENTS, $value) // per project
->addMetric(str_replace('{databaseInternalId}', $databaseInternalId, METRIC_DATABASE_ID_DOCUMENTS), $value) // per database
->addMetric(str_replace(['{databaseInternalId}', '{collectionInternalId}'], [$databaseInternalId, $collectionInternalId], METRIC_DATABASE_ID_COLLECTION_ID_DOCUMENTS), $value); // per collection
->addMetric($documentsMetric, $value) // per project
->addMetric(str_replace('{databaseInternalId}', $databaseInternalId, $databaseIdDocumentsMetric), $value) // per database
->addMetric(str_replace(['{databaseInternalId}', '{collectionInternalId}'], [$databaseInternalId, $collectionInternalId], $databaseIdCollectionIdDocumentsMetric), $value); // per collection
break;
case $document->getCollection() === 'buckets': //buckets
$queueForStatsUsage
@@ -772,6 +804,9 @@ Http::setResource('dbForProject', function (Group $pools, Database $dbForPlatfor
$queueForWebhooks = new Webhook($publisherWebhooks);
$queueForRealtime = new Realtime();
// to get consumed in the stats worker without calling to database
$queueForStatsUsage->setContext('database', new Document(['type' => $databaseType]));
$database
->on(Database::EVENT_DOCUMENT_CREATE, 'calculate-usage', fn ($event, $document) => $usageDatabaseListener($event, $document, $queueForStatsUsage))
@@ -795,7 +830,7 @@ Http::setResource('dbForProject', function (Group $pools, Database $dbForPlatfor
return $database;
}, ['pools', 'dbForPlatform', 'cache', 'project', 'response', 'publisher', 'publisherFunctions', 'publisherWebhooks', 'queueForEvents', 'queueForFunctions', 'queueForWebhooks', 'queueForRealtime', 'queueForStatsUsage', 'authorization']);
}, ['pools', 'dbForPlatform', 'cache', 'project', 'response', 'publisher', 'publisherFunctions', 'publisherWebhooks', 'queueForEvents', 'queueForFunctions', 'queueForWebhooks', 'queueForRealtime', 'queueForStatsUsage', 'authorization', 'request']);
Http::setResource('dbForPlatform', function (Group $pools, Cache $cache, Authorization $authorization) {
@@ -816,6 +851,138 @@ Http::setResource('dbForPlatform', function (Group $pools, Cache $cache, Authori
return $database;
}, ['pools', 'cache', 'authorization']);
Http::setResource('getDatabasesDB', function (Group $pools, Cache $cache, Document $project, Request $request, StatsUsage $queueForStatsUsage, Authorization $authorization) {
return function (Document $database) use ($pools, $cache, $project, $request, $queueForStatsUsage, $authorization): Database {
$databaseDSN = $database->getAttribute('database', '');
$databaseType = $database->getAttribute('type', '');
try {
$databaseDSN = new DSN($databaseDSN);
} catch (\InvalidArgumentException) {
// for old databases migrated through patch script
// databaseDSN determines the adapter
$databaseDSN = new DSN('mysql://'.$databaseDSN);
}
try {
$dsn = new DSN($project->getAttribute('database'));
} catch (\InvalidArgumentException) {
// TODO: Temporary until all projects are using shared tables
$dsn = new DSN('mysql://' . $project->getAttribute('database'));
}
$pool = $pools->get($databaseDSN->getHost());
$adapter = new DatabasePool($pool);
$database = new Database($adapter, $cache);
$sharedTables = \explode(',', System::getEnv('_APP_DATABASE_SHARED_TABLES', ''));
$database
->setDatabase(APP_DATABASE)
->setAuthorization($authorization)
->setMetadata('host', \gethostname())
->setMetadata('project', $project->getId())
->setTimeout(APP_DATABASE_TIMEOUT_MILLISECONDS_API)
->setMaxQueryValues(APP_DATABASE_QUERY_MAX_VALUES);
// inside pools authorization needs to be set first
$database->getAdapter()->setSupportForAttributes($databaseType !== DOCUMENTSDB);
if (\in_array($dsn->getHost(), $sharedTables)) {
$database
->setSharedTables(true)
->setTenant((int)$project->getSequence())
->setNamespace($dsn->getParam('namespace'));
} else {
$database
->setSharedTables(false)
->setTenant(null)
->setNamespace('_' . $project->getSequence());
}
$timeout = \intval($request->getHeader('x-appwrite-timeout'));
if (!empty($timeout) && Http::isDevelopment()) {
$database->setTimeout($timeout);
}
// Register database event listeners for usage stats collection
$documentsMetric = METRIC_DOCUMENTS;
$databaseIdDocumentsMetric = METRIC_DATABASE_ID_DOCUMENTS;
$databaseIdCollectionIdDocumentsMetric = METRIC_DATABASE_ID_COLLECTION_ID_DOCUMENTS;
if ($databaseType !== DATABASE_TYPE_LEGACY && $databaseType !== DATABASE_TYPE_TABLESDB) {
$documentsMetric = $databaseType. '.' .$documentsMetric;
$databaseIdDocumentsMetric = $databaseType. '.' .$databaseIdDocumentsMetric;
$databaseIdCollectionIdDocumentsMetric = $databaseType . '.' .$databaseIdCollectionIdDocumentsMetric;
}
$database
->on(Database::EVENT_DOCUMENT_CREATE, 'calculate-usage', function ($event, $document) use ($queueForStatsUsage, $documentsMetric, $databaseIdDocumentsMetric, $databaseIdCollectionIdDocumentsMetric) {
$value = 1;
if (str_starts_with($document->getCollection(), 'database_') && str_contains($document->getCollection(), '_collection_')) {
$parts = explode('_', $document->getCollection());
$databaseInternalId = $parts[1] ?? 0;
$collectionInternalId = $parts[3] ?? 0;
$queueForStatsUsage
->addMetric($documentsMetric, $value) // per project
->addMetric(str_replace('{databaseInternalId}', $databaseInternalId, $databaseIdDocumentsMetric), $value) // per database
->addMetric(str_replace(['{databaseInternalId}', '{collectionInternalId}'], [$databaseInternalId, $collectionInternalId], $databaseIdCollectionIdDocumentsMetric), $value); // per collection
}
})
->on(Database::EVENT_DOCUMENT_DELETE, 'calculate-usage', function ($event, $document) use ($queueForStatsUsage, $documentsMetric, $databaseIdDocumentsMetric, $databaseIdCollectionIdDocumentsMetric) {
$value = -1;
if (str_starts_with($document->getCollection(), 'database_') && str_contains($document->getCollection(), '_collection_')) {
$parts = explode('_', $document->getCollection());
$databaseInternalId = $parts[1] ?? 0;
$collectionInternalId = $parts[3] ?? 0;
$queueForStatsUsage
->addMetric($documentsMetric, $value) // per project
->addMetric(str_replace('{databaseInternalId}', $databaseInternalId, $databaseIdDocumentsMetric), $value) // per database
->addMetric(str_replace(['{databaseInternalId}', '{collectionInternalId}'], [$databaseInternalId, $collectionInternalId], $databaseIdCollectionIdDocumentsMetric), $value); // per collection
}
})
->on(Database::EVENT_DOCUMENTS_CREATE, 'calculate-usage', function ($event, $document) use ($queueForStatsUsage, $documentsMetric, $databaseIdDocumentsMetric, $databaseIdCollectionIdDocumentsMetric) {
$value = $document->getAttribute('modified', 0);
if (str_starts_with($document->getCollection(), 'database_') && str_contains($document->getCollection(), '_collection_')) {
$parts = explode('_', $document->getCollection());
$databaseInternalId = $parts[1] ?? 0;
$collectionInternalId = $parts[3] ?? 0;
$queueForStatsUsage
->addMetric($documentsMetric, $value) // per project
->addMetric(str_replace('{databaseInternalId}', $databaseInternalId, $databaseIdDocumentsMetric), $value) // per database
->addMetric(str_replace(['{databaseInternalId}', '{collectionInternalId}'], [$databaseInternalId, $collectionInternalId], $databaseIdCollectionIdDocumentsMetric), $value); // per collection
}
})
->on(Database::EVENT_DOCUMENTS_DELETE, 'calculate-usage', function ($event, $document) use ($queueForStatsUsage, $documentsMetric, $databaseIdDocumentsMetric, $databaseIdCollectionIdDocumentsMetric) {
$value = -1 * $document->getAttribute('modified', 0);
if (str_starts_with($document->getCollection(), 'database_') && str_contains($document->getCollection(), '_collection_')) {
$parts = explode('_', $document->getCollection());
$databaseInternalId = $parts[1] ?? 0;
$collectionInternalId = $parts[3] ?? 0;
$queueForStatsUsage
->addMetric($documentsMetric, $value) // per project
->addMetric(str_replace('{databaseInternalId}', $databaseInternalId, $databaseIdDocumentsMetric), $value) // per database
->addMetric(str_replace(['{databaseInternalId}', '{collectionInternalId}'], [$databaseInternalId, $collectionInternalId], $databaseIdCollectionIdDocumentsMetric), $value); // per collection
}
})
->on(Database::EVENT_DOCUMENTS_UPSERT, 'calculate-usage', function ($event, $document) use ($queueForStatsUsage, $documentsMetric, $databaseIdDocumentsMetric, $databaseIdCollectionIdDocumentsMetric) {
$value = $document->getAttribute('created', 0);
if (str_starts_with($document->getCollection(), 'database_') && str_contains($document->getCollection(), '_collection_')) {
$parts = explode('_', $document->getCollection());
$databaseInternalId = $parts[1] ?? 0;
$collectionInternalId = $parts[3] ?? 0;
$queueForStatsUsage
->addMetric($documentsMetric, $value) // per project
->addMetric(str_replace('{databaseInternalId}', $databaseInternalId, $databaseIdDocumentsMetric), $value) // per database
->addMetric(str_replace(['{databaseInternalId}', '{collectionInternalId}'], [$databaseInternalId, $collectionInternalId], $databaseIdCollectionIdDocumentsMetric), $value); // per collection
}
});
return $database;
};
}, ['pools','cache','project','request','queueForStatsUsage','authorization']);
Http::setResource('getProjectDB', function (Group $pools, Database $dbForPlatform, $cache, Authorization $authorization) {
$databases = [];
@@ -1433,9 +1600,9 @@ Http::setResource('resourceToken', function ($project, $dbForProject, $request,
return new Document([]);
}, ['project', 'dbForProject', 'request', 'authorization']);
Http::setResource('transactionState', function (Database $dbForProject, Authorization $authorization) {
return new TransactionState($dbForProject, $authorization);
}, ['dbForProject', 'authorization']);
Http::setResource('transactionState', function (Database $dbForProject, Authorization $authorization, callable $getDatabasesDB) {
return new TransactionState($dbForProject, $authorization, $getDatabasesDB);
}, ['dbForProject', 'authorization', 'getDatabasesDB']);
Http::setResource('executionsRetentionCount', function (Document $project, array $plan) {
if ($project->getId() === 'console' || empty($plan)) {
@@ -1444,3 +1611,10 @@ Http::setResource('executionsRetentionCount', function (Document $project, array
return (int) ($plan['executionsRetentionCount'] ?? 100);
}, ['project', 'plan']);
Http::setResource('embeddingAgent', function ($register) {
$adapter = new Ollama();
$adapter->setEndpoint(System::getEnv('_APP_EMBEDDING_ENDPOINT', 'http://ollama:11434/api/embed'));
$adapter->setTimeout((int) System::getEnv('_APP_EMBEDDING_TIMEOUT', '30000'));
return new Agent($adapter);
}, ['register']);
+54
View File
@@ -219,6 +219,60 @@ Server::setResource('getLogsDB', function (Group $pools, Cache $cache, Authoriza
};
}, ['pools', 'cache', 'authorization']);
Server::setResource('getDatabasesDB', function (Cache $cache, Registry $register, Document $project, Authorization $authorization) {
return function (Document $database, ?Document $projectDocument = null) use ($cache, $register, $project, $authorization): Database {
$projectDocument ??= $project;
$databaseDSN = $database->getAttribute('database', '');
$databaseType = $database->getAttribute('type', '');
// Backwardscompatibility: older or seeded legacy databases may not have a DSN stored
// in the "database" attribute. In that case, fall back to the project's database DSN.
if ($databaseDSN === '') {
$databaseDSN = $projectDocument->getAttribute('database', '');
}
try {
$databaseDSN = new DSN($databaseDSN);
} catch (\InvalidArgumentException) {
$databaseDSN = new DSN('mysql://'.$databaseDSN);
}
try {
$dsn = new DSN($projectDocument->getAttribute('database'));
} catch (\InvalidArgumentException) {
// Temporary fallback until all projects use shared tables
$dsn = new DSN('mysql://' . $projectDocument->getAttribute('database'));
}
$pools = $register->get('pools');
$pool = $pools->get($databaseDSN->getHost());
$adapter = new DatabasePool($pool);
$database = new Database($adapter, $cache);
$database
->setDatabase(APP_DATABASE)
->setAuthorization($authorization);
$database->getAdapter()->setSupportForAttributes($databaseType !== DOCUMENTSDB);
$sharedTables = \explode(',', System::getEnv('_APP_DATABASE_SHARED_TABLES', ''));
if (\in_array($dsn->getHost(), $sharedTables, true)) {
$database
->setSharedTables(true)
->setTenant((int) $projectDocument->getSequence())
->setNamespace($dsn->getParam('namespace'));
} else {
$database
->setSharedTables(false)
->setTenant(null)
->setNamespace('_' . $projectDocument->getSequence());
}
$database->setTimeout(APP_DATABASE_TIMEOUT_MILLISECONDS_WORKER);
return $database;
};
}, ['cache', 'register', 'project', 'authorization']);
Server::setResource('abuseRetention', function () {
return time() - (int) System::getEnv('_APP_MAINTENANCE_RETENTION_ABUSE', 86400); // 1 day
});
+2 -3
View File
@@ -49,7 +49,6 @@
"appwrite/php-runtimes": "0.19.*",
"appwrite/php-clamav": "2.0.*",
"utopia-php/abuse": "1.2.*",
"utopia-php/agents": "1.2.*",
"utopia-php/analytics": "0.15.*",
"utopia-php/audit": "2.2.*",
"utopia-php/auth": "0.5.*",
@@ -59,6 +58,7 @@
"utopia-php/config": "1.*",
"utopia-php/console": "0.1.*",
"utopia-php/database": "5.*",
"utopia-php/agents": "1.*",
"utopia-php/detector": "0.2.*",
"utopia-php/domains": "1.*",
"utopia-php/emails": "0.6.*",
@@ -70,7 +70,7 @@
"utopia-php/locale": "0.8.*",
"utopia-php/logger": "0.6.*",
"utopia-php/messaging": "0.20.*",
"utopia-php/migration": "1.7.*",
"utopia-php/migration": "dev-add-dev-key-migration as 1.8.0",
"utopia-php/platform": "0.7.*",
"utopia-php/pools": "1.*",
"utopia-php/span": "1.1.*",
@@ -109,7 +109,6 @@
},
"config": {
"platform": {
"php": "8.3"
},
"allow-plugins": {
"php-http/discovery": true,
Generated
+18 -12
View File
@@ -4,7 +4,7 @@
"Read more about it at https://getcomposer.org/doc/01-basic-usage.md#installing-dependencies",
"This file is @generated automatically"
],
"content-hash": "b99693284208ff3d006260a089a4f7b9",
"content-hash": "a487482e621dfe0514f1fdcd78e6b70a",
"packages": [
{
"name": "adhocore/jwt",
@@ -4517,16 +4517,16 @@
},
{
"name": "utopia-php/migration",
"version": "1.7.0",
"version": "dev-add-dev-key-migration",
"source": {
"type": "git",
"url": "https://github.com/utopia-php/migration.git",
"reference": "97583ae502e40621ea91a71de19d053c5ae2e558"
"reference": "8577c62a77415ced8b70713901d00750c83495a9"
},
"dist": {
"type": "zip",
"url": "https://api.github.com/repos/utopia-php/migration/zipball/97583ae502e40621ea91a71de19d053c5ae2e558",
"reference": "97583ae502e40621ea91a71de19d053c5ae2e558",
"url": "https://api.github.com/repos/utopia-php/migration/zipball/8577c62a77415ced8b70713901d00750c83495a9",
"reference": "8577c62a77415ced8b70713901d00750c83495a9",
"shasum": ""
},
"require": {
@@ -4566,9 +4566,9 @@
],
"support": {
"issues": "https://github.com/utopia-php/migration/issues",
"source": "https://github.com/utopia-php/migration/tree/1.7.0"
"source": "https://github.com/utopia-php/migration/tree/add-dev-key-migration"
},
"time": "2026-03-10T06:36:27+00:00"
"time": "2026-03-16T21:06:31+00:00"
},
{
"name": "utopia-php/mongo",
@@ -9105,9 +9105,18 @@
"time": "2024-03-07T20:33:40+00:00"
}
],
"aliases": [],
"aliases": [
{
"package": "utopia-php/migration",
"version": "dev-add-dev-key-migration",
"alias": "1.8.0",
"alias_normalized": "1.8.0.0"
}
],
"minimum-stability": "dev",
"stability-flags": {},
"stability-flags": {
"utopia-php/migration": 20
},
"prefer-stable": true,
"prefer-lowest": false,
"platform": {
@@ -9128,8 +9137,5 @@
"platform-dev": {
"ext-fileinfo": "*"
},
"platform-overrides": {
"php": "8.3"
},
"plugin-api-version": "2.9.0"
}
+56 -9
View File
@@ -104,6 +104,7 @@ services:
- ${_APP_DB_HOST:-mongodb}
- redis
- coredns
- ollama
# - clamav
entrypoint:
- php
@@ -152,6 +153,12 @@ services:
- _APP_DB_SCHEMA
- _APP_DB_USER
- _APP_DB_PASS
- _APP_DB_ADAPTER_VECTORSDB
- _APP_DB_HOST_VECTORSDB
- _APP_DB_PORT_VECTORSDB
- _APP_DB_SCHEMA_VECTORSDB
- _APP_DB_USER_VECTORSDB
- _APP_DB_PASS_VECTORSDB
- _APP_SMTP_HOST
- _APP_SMTP_PORT
- _APP_SMTP_SECURE
@@ -288,6 +295,7 @@ services:
depends_on:
- ${_APP_DB_HOST:-mongodb}
- redis
- ollama
environment:
- _APP_ENV
- _APP_WORKER_PER_CORE
@@ -304,6 +312,12 @@ services:
- _APP_DB_SCHEMA
- _APP_DB_USER
- _APP_DB_PASS
- _APP_DB_ADAPTER_VECTORSDB
- _APP_DB_HOST_VECTORSDB
- _APP_DB_PORT_VECTORSDB
- _APP_DB_SCHEMA_VECTORSDB
- _APP_DB_USER_VECTORSDB
- _APP_DB_PASS_VECTORSDB
- _APP_USAGE_STATS
- _APP_LOGGING_CONFIG
- _APP_LOGGING_CONFIG_REALTIME
@@ -323,6 +337,7 @@ services:
depends_on:
- redis
- ${_APP_DB_HOST:-mongodb}
- ollama
environment:
- _APP_ENV
- _APP_WORKER_PER_CORE
@@ -356,6 +371,7 @@ services:
- ${_APP_DB_HOST:-mongodb}
- request-catcher-sms
- request-catcher-webhook
- ollama
environment:
- _APP_ENV
- _APP_WORKER_PER_CORE
@@ -386,6 +402,7 @@ services:
depends_on:
- redis
- ${_APP_DB_HOST:-mongodb}
- ollama
volumes:
- appwrite-uploads:/storage/uploads:rw
- appwrite-cache:/storage/cache:rw
@@ -395,6 +412,7 @@ services:
- appwrite-certificates:/storage/certificates:rw
- ./app:/usr/src/code/app
- ./src:/usr/src/code/src
environment:
- _APP_ENV
- _APP_WORKER_PER_CORE
@@ -451,9 +469,11 @@ services:
volumes:
- ./app:/usr/src/code/app
- ./src:/usr/src/code/src
depends_on:
- redis
- ${_APP_DB_HOST:-mongodb}
- ollama
environment:
- _APP_ENV
- _APP_WORKER_PER_CORE
@@ -469,6 +489,12 @@ services:
- _APP_DB_SCHEMA
- _APP_DB_USER
- _APP_DB_PASS
- _APP_DB_ADAPTER_VECTORSDB
- _APP_DB_HOST_VECTORSDB
- _APP_DB_PORT_VECTORSDB
- _APP_DB_SCHEMA_VECTORSDB
- _APP_DB_USER_VECTORSDB
- _APP_DB_PASS_VECTORSDB
- _APP_LOGGING_CONFIG
- _APP_WORKERS_NUM
- _APP_QUEUE_NAME
@@ -490,6 +516,7 @@ services:
depends_on:
- redis
- ${_APP_DB_HOST:-mongodb}
- ollama
environment:
- _APP_ENV
- _APP_WORKER_PER_CORE
@@ -622,6 +649,7 @@ services:
depends_on:
- redis
- ${_APP_DB_HOST:-mongodb}
- ollama
volumes:
- appwrite-config:/storage/config:rw
- appwrite-certificates:/storage/certificates:rw
@@ -841,6 +869,7 @@ services:
- ./app:/usr/src/code/app
- ./src:/usr/src/code/src
- ./tests:/usr/src/code/tests
depends_on:
- ${_APP_DB_HOST:-mongodb}
environment:
@@ -1037,6 +1066,7 @@ services:
depends_on:
- redis
- ${_APP_DB_HOST:-mongodb}
- ollama
environment:
- _APP_ENV
- _APP_WORKER_PER_CORE
@@ -1070,6 +1100,7 @@ services:
depends_on:
- ${_APP_DB_HOST:-mongodb}
- redis
- ollama
environment:
- _APP_ENV
- _APP_WORKER_PER_CORE
@@ -1100,6 +1131,7 @@ services:
depends_on:
- ${_APP_DB_HOST:-mongodb}
- redis
- ollama
environment:
- _APP_ENV
- _APP_WORKER_PER_CORE
@@ -1129,6 +1161,7 @@ services:
depends_on:
- ${_APP_DB_HOST:-mongodb}
- redis
- ollama
environment:
- _APP_ENV
- _APP_WORKER_PER_CORE
@@ -1220,7 +1253,6 @@ services:
start_period: 5s
mariadb:
profiles: ["mariadb"]
image: mariadb:10.11 # fix issues when upgrading using: mysql_upgrade -u root -p
container_name: appwrite-mariadb
<<: *x-logging
@@ -1239,7 +1271,6 @@ services:
command: "mysqld --innodb-flush-method=fsync"
mongodb:
profiles: ["mongodb"]
image: mongo:8.2.5
container_name: appwrite-mongodb
<<: *x-logging
@@ -1276,7 +1307,6 @@ services:
start_period: 30s
appwrite-mongo-express:
profiles: ["mongodb"]
image: mongo-express
container_name: appwrite-mongo-express
networks:
@@ -1291,25 +1321,41 @@ services:
- mongodb
postgresql:
profiles: ["postgresql"]
build:
context: ./tests/resources/postgresql
args:
POSTGRES_VERSION: 17
image: appwrite/postgres:0.1.0
container_name: appwrite-postgresql
<<: *x-logging
networks:
- appwrite
volumes:
- appwrite-postgresql:/var/lib/postgresql/data:rw
- appwrite-postgresql:/var/lib/postgresql/18/data:rw
ports:
- "5432:5432"
environment:
- POSTGRES_DB=${_APP_DB_SCHEMA}
- POSTGRES_USER=${_APP_DB_USER}
- POSTGRES_PASSWORD=${_APP_DB_PASS}
healthcheck:
test: ["CMD-SHELL", "pg_isready -U ${_APP_DB_USER} -d ${_APP_DB_SCHEMA}"]
interval: 5s
timeout: 5s
retries: 10
start_period: 10s
command: "postgres"
ollama:
image: appwrite/ollama:0.1.1
container_name: ollama
ports:
- "11434:11434"
restart: unless-stopped
environment:
MODELS: ${_APP_EMBEDDING_MODELS:-embeddinggemma}
OLLAMA_KEEP_ALIVE: 24h
volumes:
- appwrite-models:/root/.ollama
networks:
- appwrite
redis:
image: redis:7.4.7-alpine
<<: *x-logging
@@ -1549,3 +1595,4 @@ volumes:
appwrite-sites:
appwrite-builds:
appwrite-config:
appwrite-models:
@@ -0,0 +1 @@
Create a new Collection. Before using this route, you should create a new database resource using either a [server integration](https://appwrite.io/docs/server/databases#documentsDBCreateCollection) API or directly from your database console.
@@ -0,0 +1 @@
Create a new Document. Before using this route, you should create a new collection resource using either a [server integration](https://appwrite.io/docs/server/databases#documentsDBCreateCollection) API or directly from your database console.
@@ -0,0 +1 @@
Create new Documents. Before using this route, you should create a new collection resource using either a [server integration](https://appwrite.io/docs/server/databases#documentsDBCreateCollection) API or directly from your database console.
@@ -0,0 +1,2 @@
Creates an index on the attributes listed. Your index should include all the attributes you will query in a single request.
Attributes can be `key`, `fulltext`, and `unique`.
+1
View File
@@ -0,0 +1 @@
Create a new Database.
@@ -0,0 +1 @@
Decrement a specific column of a row by a given value.
@@ -0,0 +1 @@
Delete a collection by its unique ID. Only users with write permissions have access to delete this resource.
@@ -0,0 +1 @@
Delete a document by its unique ID.
@@ -0,0 +1 @@
Bulk delete documents using queries, if no queries are passed then all documents are deleted.
@@ -0,0 +1 @@
Delete an index.
+1
View File
@@ -0,0 +1 @@
Delete a database by its unique ID. Only API keys with with databases.write scope can delete a database.
@@ -0,0 +1 @@
Get the collection activity logs list by its unique ID.
@@ -0,0 +1 @@
Get usage metrics and statistics for a collection. Returning the total number of documents. The response includes both current totals and historical data over time. Use the optional range parameter to specify the time window for historical data: 24h (last 24 hours), 30d (last 30 days), or 90d (last 90 days). If not specified, range defaults to 30 days.
@@ -0,0 +1 @@
Get a collection by its unique ID. This endpoint response returns a JSON object with the collection metadata.
@@ -0,0 +1 @@
Get usage metrics and statistics for a database. You can view the total number of collections, documents, and storage usage. The response includes both current totals and historical data over time. Use the optional range parameter to specify the time window for historical data: 24h (last 24 hours), 30d (last 30 days), or 90d (last 90 days). If not specified, range defaults to 30 days.
@@ -0,0 +1 @@
Get the document activity logs list by its unique ID.
@@ -0,0 +1 @@
Get a document by its unique ID. This endpoint response returns a JSON object with the document data.
+1
View File
@@ -0,0 +1 @@
Get index by ID.
+1
View File
@@ -0,0 +1 @@
Get the database activity logs list by its unique ID.
+1
View File
@@ -0,0 +1 @@
Get a database by its unique ID. This endpoint response returns a JSON object with the database metadata.
@@ -0,0 +1 @@
Increment a specific column of a row by a given value.
@@ -0,0 +1 @@
List attributes in the collection.
@@ -0,0 +1 @@
Get a list of all collections that belong to the provided databaseId. You can use the search parameter to filter your results.
@@ -0,0 +1 @@
Get a list of all the user's documents in a given collection. You can use the query params to filter your results.
@@ -0,0 +1 @@
List indexes in the collection.
@@ -0,0 +1 @@
List usage metrics and statistics for all databases in the project. You can view the total number of databases, collections, documents, and storage usage. The response includes both current totals and historical data over time. Use the optional range parameter to specify the time window for historical data: 24h (last 24 hours), 30d (last 30 days), or 90d (last 90 days). If not specified, range defaults to 30 days.
+1
View File
@@ -0,0 +1 @@
Get a list of all databases from the current Appwrite project. You can use the search parameter to filter your results.
@@ -0,0 +1 @@
Update a collection by its unique ID.
@@ -0,0 +1 @@
Update a document by its unique ID. Using the patch method you can pass only specific fields that will get updated.
@@ -0,0 +1 @@
Update all documents that match your queries, if no queries are submitted then all documents are updated. You can pass only specific fields to be updated.
+1
View File
@@ -0,0 +1 @@
Update a database by its unique ID.
@@ -0,0 +1 @@
Create or update a Document. Before using this route, you should create a new collection resource using either a [server integration](https://appwrite.io/docs/server/databases#documentsDBCreateCollection) API or directly from your database console.
@@ -0,0 +1 @@
Create or update Documents. Before using this route, you should create a new collection resource using either a [server integration](https://appwrite.io/docs/server/databases#documentsDBCreateCollection) API or directly from your database console.
@@ -0,0 +1 @@
Generate a temporary console-scoped API key for migrating project settings (platforms, keys, schedules).
Executable → Regular
View File
Executable → Regular
View File
+1 -1
View File
@@ -15,4 +15,4 @@ adminDb.createUser({
roles: [
{ role: 'readWrite', db: database }
]
});
});
+24 -2
View File
@@ -28,6 +28,8 @@ class Key
protected bool $projectCheckDisabled = false,
protected bool $previewAuthDisabled = false,
protected bool $deploymentStatusIgnored = false,
protected string $scopedProjectId = '',
protected string $source = '',
) {
}
@@ -103,6 +105,16 @@ class Key
return $this->projectCheckDisabled;
}
public function getScopedProjectId(): string
{
return $this->scopedProjectId;
}
public function getSource(): string
{
return $this->source;
}
/**
* Decode the given secret key into a Key object, containing the project ID, type, role, scopes, and name.
* Can be a stored API key or a dynamic key (JWT).
@@ -161,7 +173,15 @@ class Key
$projectCheckDisabled = $payload['projectCheckDisabled'] ?? false;
$previewAuthDisabled = $payload['previewAuthDisabled'] ?? false;
$deploymentStatusIgnored = $payload['deploymentStatusIgnored'] ?? false;
$scopes = \array_merge($payload['scopes'] ?? [], $scopes);
$scopedProjectId = $payload['scopedProjectId'] ?? '';
$source = $payload['source'] ?? '';
// Keys with a scoped project are restricted — only use explicit JWT scopes
if (!empty($scopedProjectId)) {
$scopes = $payload['scopes'] ?? [];
} else {
$scopes = \array_merge($payload['scopes'] ?? [], $scopes);
}
if (!$projectCheckDisabled && $projectId !== $project->getId()) {
return $guestKey;
@@ -181,7 +201,9 @@ class Key
$bannerDisabled,
$projectCheckDisabled,
$previewAuthDisabled,
$deploymentStatusIgnored
$deploymentStatusIgnored,
$scopedProjectId,
$source
);
case API_KEY_STANDARD:
$key = $project->find(
+27 -14
View File
@@ -21,17 +21,23 @@ class TransactionState
{
private Database $dbForProject;
private Authorization $authorization;
/** @var Authorization $authorization */
public function __construct(Database $dbForProject, Authorization $authorization)
/**
* @var callable(Document $database): Database
*/
private mixed $getDatabasesDB;
public function __construct(Database $dbForProject, Authorization $authorization, callable $getDatabasesDB)
{
$this->dbForProject = $dbForProject;
$this->authorization = $authorization;
$this->getDatabasesDB = $getDatabasesDB;
}
/**
* Get a document with transaction-aware logic
*
* @param Document $database Target database document
* @param string $collectionId Collection ID
* @param string $documentId Document ID
* @param string|null $transactionId Optional transaction ID
@@ -42,13 +48,15 @@ class TransactionState
* @throws Timeout
*/
public function getDocument(
Document $database,
string $collectionId,
string $documentId,
?string $transactionId = null,
array $queries = []
): Document {
$dbForDatabases = ($this->getDatabasesDB)($database);
if ($transactionId === null) {
return $this->dbForProject->getDocument($collectionId, $documentId, $queries);
return $dbForDatabases->getDocument($collectionId, $documentId, $queries);
}
$state = $this->getTransactionState($transactionId);
@@ -66,7 +74,7 @@ class TransactionState
if ($docState['action'] === 'update' || $docState['action'] === 'upsert') {
// Merge with committed version
$committedDoc = $this->dbForProject->getDocument($collectionId, $documentId, $queries);
$committedDoc = $dbForDatabases->getDocument($collectionId, $documentId, $queries);
if (!$committedDoc->isEmpty()) {
foreach ($docState['document']->getAttributes() as $key => $value) {
if ($key !== '$id') {
@@ -80,13 +88,13 @@ class TransactionState
}
}
}
return $this->dbForProject->getDocument($collectionId, $documentId, $queries);
return $dbForDatabases->getDocument($collectionId, $documentId, $queries);
}
/**
* List documents with transaction-aware logic
*
* @param Document $database Target database document
* @param string $collectionId Collection ID
* @param string|null $transactionId Optional transaction ID
* @param array $queries Optional query filters
@@ -96,17 +104,19 @@ class TransactionState
* @throws Timeout
*/
public function listDocuments(
Document $database,
string $collectionId,
?string $transactionId = null,
array $queries = []
): array {
$dbForDatabases = ($this->getDatabasesDB)($database);
// If no transaction, use normal database retrieval
if ($transactionId === null) {
return $this->dbForProject->find($collectionId, $queries);
return $dbForDatabases->find($collectionId, $queries);
}
$state = $this->getTransactionState($transactionId);
$committedDocs = $this->dbForProject->find($collectionId, $queries);
$committedDocs = $dbForDatabases->find($collectionId, $queries);
$documentMap = [];
// Build map of committed documents
@@ -147,6 +157,7 @@ class TransactionState
/**
* Count documents with transaction-aware logic
*
* @param Document $database Target database document
* @param string $collectionId Collection ID
* @param string|null $transactionId Optional transaction ID
* @param array $queries Optional query filters
@@ -156,23 +167,23 @@ class TransactionState
* @throws Timeout
*/
public function countDocuments(
Document $database,
string $collectionId,
?string $transactionId = null,
array $queries = []
): int {
$dbForDatabases = ($this->getDatabasesDB)($database);
if ($transactionId === null) {
return $this->dbForProject->count($collectionId, $queries, APP_LIMIT_COUNT);
return $dbForDatabases->count($collectionId, $queries, APP_LIMIT_COUNT);
}
$state = $this->getTransactionState($transactionId);
$baseCount = $this->dbForProject->count($collectionId, $queries, APP_LIMIT_COUNT);
$baseCount = $dbForDatabases->count($collectionId, $queries, APP_LIMIT_COUNT);
if (!isset($state[$collectionId])) {
return $baseCount;
}
$committedDocs = $this->dbForProject->find($collectionId, $queries);
$committedDocs = $dbForDatabases->find($collectionId, $queries);
$committedDocIds = [];
foreach ($committedDocs as $doc) {
$committedDocIds[$doc->getId()] = true;
@@ -214,17 +225,19 @@ class TransactionState
/**
* Check if a document exists with transaction-aware logic
*
* @param Document $database Target database document
* @param string $collectionId Collection ID
* @param string $documentId Document ID
* @param string|null $transactionId Optional transaction ID
* @return bool True if document exists
*/
public function documentExists(
Document $database,
string $collectionId,
string $documentId,
?string $transactionId = null
): bool {
$doc = $this->getDocument($collectionId, $documentId, $transactionId);
$doc = $this->getDocument($database, $collectionId, $documentId, $transactionId);
return !$doc->isEmpty();
}
+9 -2
View File
@@ -519,6 +519,7 @@ class Event
* @param string $pattern
* @param array $params
* @param ?Document $database
* @param ?Document $database
* @return array
* @throws \InvalidArgumentException
*/
@@ -533,7 +534,7 @@ class Event
$parsed = self::parseEventPattern($pattern);
// to switch the resource types from databases to the required prefix
// eg; all databases events get fired with databases. prefix which mainly depicts legacy type
// so a projection from databases to the actual prefix
// so a projection from databases to the actual prefix(documentsdb, vectorsdb,etc)
if ((str_contains($pattern, 'databases.') && $database && $database->getAttribute('type') !== 'legacy')) {
$parsed = self::getDatabaseTypeEvents($database, $parsed);
}
@@ -695,7 +696,6 @@ class Event
)
) {
$pairedEvents = [];
foreach ($events as $event) {
$pairedEvents[] = $event;
// tablesdb needs databases event with tables and collections
@@ -745,6 +745,13 @@ class Event
'attributes' => 'columns',
];
break;
case 'documentsdb':
case 'vectorsdb':
// sending the type itself(eg: documentsdb, vectorsdb)
$eventMap = [
'databases' => $database->getAttribute('type')
];
break;
}
foreach ($event as $eventKey => $eventValue) {
if (isset($eventMap[$eventValue])) {
+1
View File
@@ -84,6 +84,7 @@ class StatsUsage extends Event
}
return true;
}),
'context' => $this->context,
];
}
+1
View File
@@ -339,6 +339,7 @@ class Exception extends \Exception
public const string MIGRATION_ALREADY_EXISTS = 'migration_already_exists';
public const string MIGRATION_IN_PROGRESS = 'migration_in_progress';
public const string MIGRATION_PROVIDER_ERROR = 'migration_provider_error';
public const string MIGRATION_DATABASE_TYPE_UNSUPPORTED = 'migration_database_type_unsupported';
/** Realtime */
public const string REALTIME_MESSAGE_FORMAT_INVALID = 'realtime_message_format_invalid';
+25 -6
View File
@@ -492,6 +492,8 @@ class Realtime extends MessagingAdapter
break;
case 'databases':
case 'tablesdb':
case 'documentsdb':
case 'vectorsdb':
$resource = $parts[4] ?? '';
if (in_array($resource, ['columns', 'attributes', 'indexes'])) {
$channels[] = 'console';
@@ -511,12 +513,20 @@ class Realtime extends MessagingAdapter
$resourceId = $tableId ?: $collectionId;
$channels = [];
// sending legacy + tablesdb events to both legacy and tablesdb
$channels = array_values(array_unique(array_merge(
self::getDatabaseChannels('legacy', $database->getId(), $resourceId, $payload->getId(), 'databases'),
self::getDatabaseChannels('tablesdb', $database->getId(), $resourceId, $payload->getId(), 'databases'),
self::getDatabaseChannels('tablesdb', $database->getId(), $resourceId, $payload->getId())
)));
switch ($parts[0]) {
case 'databases':
case 'tablesdb':
// sending legacy + tablesdb events to both legacy and tablesdb
$channels = array_values(array_unique(array_merge(
self::getDatabaseChannels('legacy', $database->getId(), $resourceId, $payload->getId(), 'databases'),
self::getDatabaseChannels('tablesdb', $database->getId(), $resourceId, $payload->getId(), 'databases'),
self::getDatabaseChannels('tablesdb', $database->getId(), $resourceId, $payload->getId())
)));
break;
default:
// only prefixed events
$channels = array_values(self::getDatabaseChannels($parts[0], $database->getId(), $resourceId, $payload->getId()));
}
$roles = $collection->getAttribute('documentSecurity', false)
? \array_merge($collection->getRead(), $payload->getRead())
@@ -582,6 +592,7 @@ class Realtime extends MessagingAdapter
* @param string $resourceId The collection/table ID
* @param string $payloadId The document/row ID
* @param string $prefixOverride Override the channel prefix when different API types share the same terminology but need different prefixes
* (e.g., 'databases' and 'documentsdb' use same terminology but need different prefixes)
* @return array Array of channel names
*/
private static function getDatabaseChannels(
@@ -615,6 +626,13 @@ class Realtime extends MessagingAdapter
$channels[] = "{$basePrefix}.{$databaseId}.tables.{$resourceId}.rows.{$payloadId}";
break;
case 'documentsdb':
case 'vectorsdb':
$channels[] = 'documents';
$channels[] = "{$basePrefix}.{$databaseId}.collections.{$resourceId}.documents";
$channels[] = "{$basePrefix}.{$databaseId}.collections.{$resourceId}.documents.{$payloadId}";
break;
default:
$basePrefix = 'databases';
$channels[] = 'documents';
@@ -623,6 +641,7 @@ class Realtime extends MessagingAdapter
break;
}
return $channels;
}
}
@@ -22,3 +22,11 @@ const INDEX = 'index';
const DOCUMENTS = 'document';
const ATTRIBUTES = 'attribute';
const COLLECTIONS = 'collection';
const LEGACY = 'legacy';
const TABLESDB = 'tablesdb';
const DOCUMENTSDB = 'documentsdb';
const VECTORSDB = 'vectorsdb';
const MIN_VECTOR_DIMENSION = 1;
const MAX_VECTOR_DIMENSION = 16000;
@@ -10,7 +10,7 @@ use Utopia\Database\Operator;
class Action extends AppwriteAction
{
private string $context = 'legacy';
private string $context = DATABASE_TYPE_LEGACY;
public function getDatabaseType(): string
{
@@ -20,7 +20,13 @@ class Action extends AppwriteAction
public function setHttpPath(string $path): AppwriteAction
{
if (\str_contains($path, '/tablesdb')) {
$this->context = 'tablesdb';
$this->context = DATABASE_TYPE_TABLESDB;
}
if (\str_contains($path, '/documentsdb')) {
$this->context = DATABASE_TYPE_DOCUMENTSDB;
}
if (\str_contains($path, '/vectorsdb')) {
$this->context = DATABASE_TYPE_VECTORSDB;
}
return parent::setHttpPath($path);
}
@@ -15,6 +15,8 @@ abstract class Action extends UtopiaAction
*/
private ?string $context = COLLECTIONS;
private ?string $databaseType = LEGACY;
/**
* Get the response model used in the SDK and HTTP responses.
*/
@@ -24,6 +26,9 @@ abstract class Action extends UtopiaAction
{
if (\str_contains($path, '/tablesdb')) {
$this->context = TABLES;
$this->databaseType = TABLESDB;
} elseif (\str_contains($path, '/vectorsdb')) {
$this->databaseType = VECTORSDB;
}
return parent::setHttpPath($path);
}
@@ -36,6 +41,14 @@ abstract class Action extends UtopiaAction
return $this->context;
}
/**
* Get the current API database type.
*/
protected function getDatabaseType(): string
{
return $this->databaseType;
}
/**
* Get the key used in event parameters (e.g., 'collectionId' or 'tableId').
*/
@@ -84,12 +84,13 @@ class Create extends Action
->param('indexes', [], new ArrayList(new JSON(), APP_LIMIT_ARRAY_PARAMS_SIZE), 'Array of index definitions to create. Each index should contain: key (string), type (string: key, fulltext, unique, spatial), attributes (array of attribute keys), orders (array of ASC/DESC, optional), and lengths (array of integers, optional).', true)
->inject('response')
->inject('dbForProject')
->inject('getDatabasesDB')
->inject('queueForEvents')
->inject('authorization')
->callback($this->action(...));
}
public function action(string $databaseId, string $collectionId, string $name, ?array $permissions, bool $documentSecurity, bool $enabled, array $attributes, array $indexes, UtopiaResponse $response, Database $dbForProject, Event $queueForEvents, Authorization $authorization): void
public function action(string $databaseId, string $collectionId, string $name, ?array $permissions, bool $documentSecurity, bool $enabled, array $attributes, array $indexes, UtopiaResponse $response, Database $dbForProject, callable $getDatabasesDB, Event $queueForEvents, Authorization $authorization): void
{
$database = $authorization->skip(fn () => $dbForProject->getDocument('databases', $databaseId));
@@ -121,12 +122,18 @@ class Create extends Action
throw new Exception(Exception::DATABASE_NOT_FOUND, params: [$databaseId]);
}
/**
* @var Database $dbForDatabases
*/
$dbForDatabases = $getDatabasesDB($database);
$collectionKey = 'database_' . $database->getSequence() . '_collection_' . $collection->getSequence();
$databaseKey = 'database_' . $database->getSequence();
$attributesValidator = new AttributesValidator(
APP_LIMIT_ARRAY_PARAMS_SIZE,
$dbForProject->getAdapter()->getSupportForSpatialAttributes()
$dbForDatabases->getAdapter()->getSupportForSpatialAttributes(),
$dbForDatabases->getAdapter()->getSupportForAttributes()
);
if (!$attributesValidator->isValid($attributes)) {
@@ -155,7 +162,7 @@ class Create extends Action
}
// Validate indexes
$indexesValidator = new IndexesValidator($dbForProject->getLimitForIndexes());
$indexesValidator = new IndexesValidator($dbForDatabases->getLimitForIndexes());
if (!$indexesValidator->isValid($indexes)) {
$dbForProject->deleteDocument($databaseKey, $collection->getId());
throw new Exception(Exception::GENERAL_ARGUMENT_INVALID, $indexesValidator->getDescription());
@@ -178,21 +185,23 @@ class Create extends Action
$indexValidator = new IndexValidator(
$collectionAttributes,
[],
$dbForProject->getAdapter()->getMaxIndexLength(),
$dbForProject->getAdapter()->getInternalIndexesKeys(),
$dbForProject->getAdapter()->getSupportForIndexArray(),
$dbForProject->getAdapter()->getSupportForSpatialIndexNull(),
$dbForProject->getAdapter()->getSupportForSpatialIndexOrder(),
$dbForProject->getAdapter()->getSupportForVectors(),
$dbForProject->getAdapter()->getSupportForAttributes(),
$dbForProject->getAdapter()->getSupportForMultipleFulltextIndexes(),
$dbForProject->getAdapter()->getSupportForIdenticalIndexes(),
$dbForProject->getAdapter()->getSupportForObjectIndexes(),
$dbForProject->getAdapter()->getSupportForTrigramIndex(),
$dbForProject->getAdapter()->getSupportForSpatialAttributes(),
$dbForProject->getAdapter()->getSupportForIndex(),
$dbForProject->getAdapter()->getSupportForUniqueIndex(),
$dbForProject->getAdapter()->getSupportForFulltextIndex(),
$dbForDatabases->getAdapter()->getMaxIndexLength(),
$dbForDatabases->getAdapter()->getInternalIndexesKeys(),
$dbForDatabases->getAdapter()->getSupportForIndexArray(),
$dbForDatabases->getAdapter()->getSupportForSpatialIndexNull(),
$dbForDatabases->getAdapter()->getSupportForSpatialIndexOrder(),
$dbForDatabases->getAdapter()->getSupportForVectors(),
$dbForDatabases->getAdapter()->getSupportForAttributes(),
$dbForDatabases->getAdapter()->getSupportForMultipleFulltextIndexes(),
$dbForDatabases->getAdapter()->getSupportForIdenticalIndexes(),
$dbForDatabases->getAdapter()->getSupportForObjectIndexes(),
$dbForDatabases->getAdapter()->getSupportForTrigramIndex(),
$dbForDatabases->getAdapter()->getSupportForSpatialAttributes(),
$dbForDatabases->getAdapter()->getSupportForIndex(),
$dbForDatabases->getAdapter()->getSupportForUniqueIndex(),
$dbForDatabases->getAdapter()->getSupportForFulltextIndex(),
$dbForDatabases->getAdapter()->getSupportForTTLIndexes(),
$dbForDatabases->getAdapter()->getSupportForObject(),
);
foreach ($collectionIndexes as $indexDoc) {
@@ -203,7 +212,7 @@ class Create extends Action
}
try {
$dbForProject->createCollection(
$dbForDatabases->createCollection(
id: $collectionKey,
attributes: $collectionAttributes,
indexes: $collectionIndexes,
@@ -62,13 +62,14 @@ class Delete extends Action
->param('collectionId', '', fn (Database $dbForProject) => new UID($dbForProject->getAdapter()->getMaxUIDLength()), 'Collection ID.', false, ['dbForProject'])
->inject('response')
->inject('dbForProject')
->inject('getDatabasesDB')
->inject('queueForDatabase')
->inject('queueForEvents')
->inject('authorization')
->callback($this->action(...));
}
public function action(string $databaseId, string $collectionId, UtopiaResponse $response, Database $dbForProject, EventDatabase $queueForDatabase, Event $queueForEvents, Authorization $authorization): void
public function action(string $databaseId, string $collectionId, UtopiaResponse $response, Database $dbForProject, callable $getDatabasesDB, EventDatabase $queueForDatabase, Event $queueForEvents, Authorization $authorization): void
{
$database = $authorization->skip(fn () => $dbForProject->getDocument('databases', $databaseId));
if ($database->isEmpty()) {
@@ -85,7 +86,8 @@ class Delete extends Action
throw new Exception(Exception::GENERAL_SERVER_ERROR, "Failed to remove $type from DB");
}
$dbForProject->purgeCachedCollection('database_' . $database->getSequence() . '_collection_' . $collection->getSequence());
$dbForDatabases = $getDatabasesDB($database);
$dbForDatabases->purgeCachedCollection('database_' . $database->getSequence() . '_collection_' . $collection->getSequence());
$queueForDatabase
->setType(DATABASE_TYPE_DELETE_COLLECTION)
@@ -17,6 +17,7 @@ abstract class Action extends DatabasesAction
* @var string|null The current context (either 'row' or 'document')
*/
private ?string $context = DOCUMENTS;
private ?string $databaseType = DATABASE_TYPE_LEGACY;
/**
* Get the response model used in the SDK and HTTP responses.
@@ -27,6 +28,10 @@ abstract class Action extends DatabasesAction
{
if (str_contains($path, '/tablesdb/')) {
$this->context = ROWS;
} elseif (str_contains($path, '/documentsdb/')) {
$this->databaseType = DATABASE_TYPE_DOCUMENTSDB;
} elseif (str_contains($path, '/vectorsdb/')) {
$this->databaseType = DATABASE_TYPE_VECTORSDB;
}
$contextId = '$' . $this->getCollectionsEventsContext() . 'Id';
@@ -45,6 +50,39 @@ abstract class Action extends DatabasesAction
return parent::setHttpPath($path);
}
protected function getDatabasesOperationReadMetric(): string
{
if ($this->databaseType === DATABASE_TYPE_LEGACY || $this->databaseType === DATABASE_TYPE_TABLESDB) {
return METRIC_DATABASES_OPERATIONS_READS;
}
return $this->databaseType.'.'.METRIC_DATABASES_OPERATIONS_READS;
}
protected function getDatabasesIdOperationReadMetric(): string
{
if ($this->databaseType === DATABASE_TYPE_LEGACY || $this->databaseType === DATABASE_TYPE_TABLESDB) {
return METRIC_DATABASE_ID_OPERATIONS_READS;
}
return $this->databaseType.'.'.METRIC_DATABASE_ID_OPERATIONS_READS;
}
protected function getDatabasesOperationWriteMetric(): string
{
if ($this->databaseType === DATABASE_TYPE_LEGACY || $this->databaseType === DATABASE_TYPE_TABLESDB) {
return METRIC_DATABASES_OPERATIONS_WRITES;
}
return $this->databaseType.'.'.METRIC_DATABASES_OPERATIONS_WRITES;
}
protected function getDatabasesIdOperationWriteMetric(): string
{
if ($this->databaseType === DATABASE_TYPE_LEGACY || $this->databaseType === DATABASE_TYPE_TABLESDB) {
return METRIC_DATABASE_ID_OPERATIONS_WRITES;
}
return $this->databaseType.'.'.METRIC_DATABASE_ID_OPERATIONS_WRITES;
}
/**
* Get the plural of the given name.
*
@@ -82,6 +82,7 @@ class Decrement extends Action
->param('transactionId', null, fn (Database $dbForProject) => new Nullable(new UID($dbForProject->getAdapter()->getMaxUIDLength())), 'Transaction ID for staging the operation.', true, ['dbForProject'])
->inject('response')
->inject('dbForProject')
->inject('getDatabasesDB')
->inject('queueForEvents')
->inject('queueForStatsUsage')
->inject('plan')
@@ -89,7 +90,7 @@ class Decrement extends Action
->callback($this->action(...));
}
public function action(string $databaseId, string $collectionId, string $documentId, string $attribute, int|float $value, int|float|null $min, ?string $transactionId, UtopiaResponse $response, Database $dbForProject, Event $queueForEvents, StatsUsage $queueForStatsUsage, array $plan, Authorization $authorization): void
public function action(string $databaseId, string $collectionId, string $documentId, string $attribute, int|float $value, int|float|null $min, ?string $transactionId, UtopiaResponse $response, Database $dbForProject, callable $getDatabasesDB, Event $queueForEvents, StatsUsage $queueForStatsUsage, array $plan, Authorization $authorization): void
{
$isAPIKey = User::isApp($authorization->getRoles());
$isPrivilegedUser = User::isPrivileged($authorization->getRoles());
@@ -170,8 +171,9 @@ class Decrement extends Action
return;
}
$dbForDatabases = $getDatabasesDB($database);
try {
$document = $dbForProject->decreaseDocumentAttribute(
$document = $dbForDatabases->decreaseDocumentAttribute(
collection: 'database_' . $database->getSequence() . '_collection_' . $collection->getSequence(),
id: $documentId,
attribute: $attribute,
@@ -201,8 +203,8 @@ class Decrement extends Action
);
$queueForStatsUsage
->addMetric(METRIC_DATABASES_OPERATIONS_WRITES, 1)
->addMetric(str_replace('{databaseInternalId}', $database->getSequence(), METRIC_DATABASE_ID_OPERATIONS_WRITES), 1);
->addMetric($this->getDatabasesOperationWriteMetric(), 1)
->addMetric(str_replace('{databaseInternalId}', $database->getSequence(), $this->getDatabasesIdOperationWriteMetric()), 1);
$queueForEvents
->setParam('databaseId', $databaseId)
@@ -82,6 +82,7 @@ class Increment extends Action
->param('transactionId', null, fn (Database $dbForProject) => new Nullable(new UID($dbForProject->getAdapter()->getMaxUIDLength())), 'Transaction ID for staging the operation.', true, ['dbForProject'])
->inject('response')
->inject('dbForProject')
->inject('getDatabasesDB')
->inject('queueForEvents')
->inject('queueForStatsUsage')
->inject('plan')
@@ -89,7 +90,7 @@ class Increment extends Action
->callback($this->action(...));
}
public function action(string $databaseId, string $collectionId, string $documentId, string $attribute, int|float $value, int|float|null $max, ?string $transactionId, UtopiaResponse $response, Database $dbForProject, Event $queueForEvents, StatsUsage $queueForStatsUsage, array $plan, Authorization $authorization): void
public function action(string $databaseId, string $collectionId, string $documentId, string $attribute, int|float $value, int|float|null $max, ?string $transactionId, UtopiaResponse $response, Database $dbForProject, callable $getDatabasesDB, Event $queueForEvents, StatsUsage $queueForStatsUsage, array $plan, Authorization $authorization): void
{
$isAPIKey = User::isApp($authorization->getRoles());
$isPrivilegedUser = User::isPrivileged($authorization->getRoles());
@@ -170,8 +171,9 @@ class Increment extends Action
return;
}
$dbForDatabases = $getDatabasesDB($database);
try {
$document = $dbForProject->increaseDocumentAttribute(
$document = $dbForDatabases->increaseDocumentAttribute(
collection: 'database_' . $database->getSequence() . '_collection_' . $collection->getSequence(),
id: $documentId,
attribute: $attribute,
@@ -201,8 +203,8 @@ class Increment extends Action
);
$queueForStatsUsage
->addMetric(METRIC_DATABASES_OPERATIONS_WRITES, 1)
->addMetric(str_replace('{databaseInternalId}', $database->getSequence(), METRIC_DATABASE_ID_OPERATIONS_WRITES), 1);
->addMetric($this->getDatabasesOperationWriteMetric(), 1)
->addMetric(str_replace('{databaseInternalId}', $database->getSequence(), $this->getDatabasesIdOperationWriteMetric()), 1);
$queueForEvents
->setParam('databaseId', $databaseId)
@@ -76,6 +76,7 @@ class Delete extends Action
->param('transactionId', null, fn (Database $dbForProject) => new Nullable(new UID($dbForProject->getAdapter()->getMaxUIDLength())), 'Transaction ID for staging the operation.', true, ['dbForProject'])
->inject('response')
->inject('dbForProject')
->inject('getDatabasesDB')
->inject('queueForStatsUsage')
->inject('queueForEvents')
->inject('queueForRealtime')
@@ -86,7 +87,7 @@ class Delete extends Action
->callback($this->action(...));
}
public function action(string $databaseId, string $collectionId, array $queries, ?string $transactionId, UtopiaResponse $response, Database $dbForProject, StatsUsage $queueForStatsUsage, Event $queueForEvents, Event $queueForRealtime, Event $queueForFunctions, Event $queueForWebhooks, array $plan, EventProcessor $eventProcessor): void
public function action(string $databaseId, string $collectionId, array $queries, ?string $transactionId, UtopiaResponse $response, Database $dbForProject, callable $getDatabasesDB, StatsUsage $queueForStatsUsage, Event $queueForEvents, Event $queueForRealtime, Event $queueForFunctions, Event $queueForWebhooks, array $plan, EventProcessor $eventProcessor): void
{
$database = $dbForProject->getDocument('databases', $databaseId);
if ($database->isEmpty()) {
@@ -163,10 +164,11 @@ class Delete extends Action
return;
}
$dbForDatabases = $getDatabasesDB($database);
$documents = [];
try {
$modified = $dbForProject->deleteDocuments(
$modified = $dbForDatabases->deleteDocuments(
'database_' . $database->getSequence() . '_collection_' . $collection->getSequence(),
$queries,
onNext: function (Document $document) use ($plan, &$documents) {
@@ -189,12 +191,12 @@ class Delete extends Action
}
$queueForStatsUsage
->addMetric(METRIC_DATABASES_OPERATIONS_WRITES, \max(1, $modified))
->addMetric(str_replace('{databaseInternalId}', $database->getSequence(), METRIC_DATABASE_ID_OPERATIONS_WRITES), \max(1, $modified));
->addMetric($this->getDatabasesOperationWriteMetric(), \max(1, $modified))
->addMetric(str_replace('{databaseInternalId}', $database->getSequence(), $this->getDatabasesIdOperationWriteMetric()), \max(1, $modified));
$response->dynamic(new Document([
'total' => $modified,
$this->getSDKGroup() => $documents,
$this->getSDKGroup() => $documents
]), $this->getResponseModel());
$this->triggerBulk(
@@ -80,6 +80,7 @@ class Update extends Action
->param('transactionId', null, fn (Database $dbForProject) => new Nullable(new UID($dbForProject->getAdapter()->getMaxUIDLength())), 'Transaction ID for staging the operation.', true, ['dbForProject'])
->inject('response')
->inject('dbForProject')
->inject('getDatabasesDB')
->inject('queueForStatsUsage')
->inject('queueForEvents')
->inject('queueForRealtime')
@@ -90,7 +91,7 @@ class Update extends Action
->callback($this->action(...));
}
public function action(string $databaseId, string $collectionId, string|array $data, array $queries, ?string $transactionId, UtopiaResponse $response, Database $dbForProject, StatsUsage $queueForStatsUsage, Event $queueForEvents, Event $queueForRealtime, Event $queueForFunctions, Event $queueForWebhooks, array $plan, EventProcessor $eventProcessor): void
public function action(string $databaseId, string $collectionId, string|array $data, array $queries, ?string $transactionId, UtopiaResponse $response, Database $dbForProject, callable $getDatabasesDB, StatsUsage $queueForStatsUsage, Event $queueForEvents, Event $queueForRealtime, Event $queueForFunctions, Event $queueForWebhooks, array $plan, EventProcessor $eventProcessor): void
{
$data = \is_string($data)
? \json_decode($data, true)
@@ -189,11 +190,12 @@ class Update extends Action
return;
}
$dbForDatabases = $getDatabasesDB($database);
$documents = [];
try {
$modified = $dbForProject->withPreserveDates(function () use ($plan, &$documents, $dbForProject, $database, $collection, $data, $queries) {
return $dbForProject->updateDocuments(
$modified = $dbForDatabases->withPreserveDates(function () use ($plan, &$documents, $dbForDatabases, $database, $collection, $data, $queries) {
return $dbForDatabases->updateDocuments(
'database_' . $database->getSequence() . '_collection_' . $collection->getSequence(),
new Document($data),
$queries,
@@ -220,8 +222,8 @@ class Update extends Action
}
$queueForStatsUsage
->addMetric(METRIC_DATABASES_OPERATIONS_WRITES, \max(1, $modified))
->addMetric(str_replace('{databaseInternalId}', $database->getSequence(), METRIC_DATABASE_ID_OPERATIONS_WRITES), \max(1, $modified));
->addMetric($this->getDatabasesOperationWriteMetric(), \max(1, $modified))
->addMetric(str_replace('{databaseInternalId}', $database->getSequence(), $this->getDatabasesIdOperationWriteMetric()), \max(1, $modified));
$response->dynamic(new Document([
'total' => $modified,
@@ -58,7 +58,7 @@ class Upsert extends Action
group: $this->getSDKGroup(),
name: self::getName(),
description: '/docs/references/databases/upsert-documents.md',
auth: [AuthType::ADMIN, AuthType::KEY],
auth: [AuthType::ADMIN, AuthType::SESSION, AuthType::KEY, AuthType::JWT],
responses: [
new SDKResponse(
code: SwooleResponse::STATUS_CODE_CREATED,
@@ -78,6 +78,7 @@ class Upsert extends Action
->param('transactionId', null, fn (Database $dbForProject) => new Nullable(new UID($dbForProject->getAdapter()->getMaxUIDLength())), 'Transaction ID for staging the operation.', true, ['dbForProject'])
->inject('response')
->inject('dbForProject')
->inject('getDatabasesDB')
->inject('queueForStatsUsage')
->inject('queueForEvents')
->inject('queueForRealtime')
@@ -88,7 +89,7 @@ class Upsert extends Action
->callback($this->action(...));
}
public function action(string $databaseId, string $collectionId, array $documents, ?string $transactionId, UtopiaResponse $response, Database $dbForProject, StatsUsage $queueForStatsUsage, Event $queueForEvents, Event $queueForRealtime, Event $queueForFunctions, Event $queueForWebhooks, array $plan, EventProcessor $eventProcessor): void
public function action(string $databaseId, string $collectionId, array $documents, ?string $transactionId, UtopiaResponse $response, Database $dbForProject, callable $getDatabasesDB, StatsUsage $queueForStatsUsage, Event $queueForEvents, Event $queueForRealtime, Event $queueForFunctions, Event $queueForWebhooks, array $plan, EventProcessor $eventProcessor): void
{
$database = $dbForProject->getDocument('databases', $databaseId);
if ($database->isEmpty()) {
@@ -165,11 +166,12 @@ class Upsert extends Action
return;
}
$dbForDatabases = $getDatabasesDB($database);
$upserted = [];
try {
$modified = $dbForProject->withPreserveDates(function () use ($dbForProject, $database, $collection, $documents, $plan, &$upserted) {
return $dbForProject->upsertDocuments(
$modified = $dbForDatabases->withPreserveDates(function () use ($dbForDatabases, $database, $collection, $documents, $plan, &$upserted) {
return $dbForDatabases->upsertDocuments(
'database_' . $database->getSequence() . '_collection_' . $collection->getSequence(),
$documents,
onNext: function (Document $document) use ($plan, &$upserted) {
@@ -195,8 +197,8 @@ class Upsert extends Action
}
$queueForStatsUsage
->addMetric(METRIC_DATABASES_OPERATIONS_WRITES, \max(1, $modified))
->addMetric(str_replace('{databaseInternalId}', $database->getSequence(), METRIC_DATABASE_ID_OPERATIONS_WRITES), \max(1, $modified));
->addMetric($this->getDatabasesOperationWriteMetric(), \max(1, $modified))
->addMetric(str_replace('{databaseInternalId}', $database->getSequence(), $this->getDatabasesIdOperationWriteMetric()), \max(1, $modified));
$response->dynamic(new Document([
'total' => $modified,
@@ -85,7 +85,7 @@ class Create extends Action
new Parameter('documentId', optional: false),
new Parameter('data', optional: false),
new Parameter('permissions', optional: true),
new Parameter('transactionId', optional: true),
new Parameter('transactionId', optional: true)
],
deprecated: new Deprecated(
since: '1.8.0',
@@ -110,7 +110,7 @@ class Create extends Action
new Parameter('databaseId', optional: false),
new Parameter('collectionId', optional: false),
new Parameter('documents', optional: false),
new Parameter('transactionId', optional: true),
new Parameter('transactionId', optional: true)
],
deprecated: new Deprecated(
since: '1.8.0',
@@ -127,6 +127,7 @@ class Create extends Action
->param('transactionId', null, fn (Database $dbForProject) => new Nullable(new UID($dbForProject->getAdapter()->getMaxUIDLength())), 'Transaction ID for staging the operation.', true, ['dbForProject'])
->inject('response')
->inject('dbForProject')
->inject('getDatabasesDB')
->inject('user')
->inject('queueForEvents')
->inject('queueForStatsUsage')
@@ -138,7 +139,7 @@ class Create extends Action
->inject('eventProcessor')
->callback($this->action(...));
}
public function action(string $databaseId, string $documentId, string $collectionId, string|array $data, ?array $permissions, ?array $documents, ?string $transactionId, UtopiaResponse $response, Database $dbForProject, Document $user, Event $queueForEvents, StatsUsage $queueForStatsUsage, Event $queueForRealtime, Event $queueForFunctions, Event $queueForWebhooks, array $plan, Authorization $authorization, EventProcessor $eventProcessor): void
public function action(string $databaseId, string $documentId, string $collectionId, string|array $data, ?array $permissions, ?array $documents, ?string $transactionId, UtopiaResponse $response, Database $dbForProject, callable $getDatabasesDB, Document $user, Event $queueForEvents, StatsUsage $queueForStatsUsage, Event $queueForRealtime, Event $queueForFunctions, Event $queueForWebhooks, array $plan, Authorization $authorization, EventProcessor $eventProcessor): void
{
$data = \is_string($data)
? \json_decode($data, true)
@@ -447,11 +448,12 @@ class Create extends Action
return;
}
$dbForDatabases = $getDatabasesDB($database);
try {
$created = [];
$dbForProject->withPreserveDates(
function () use (&$created, $dbForProject, $database, $collection, $documents) {
$dbForProject->createDocuments(
$dbForDatabases->withPreserveDates(
function () use (&$created, $dbForDatabases, $database, $collection, $documents) {
$dbForDatabases->createDocuments(
'database_' . $database->getSequence() . '_collection_' . $collection->getSequence(),
$documents,
onNext: function ($doc) use (&$created) {
@@ -490,15 +492,15 @@ class Create extends Action
}
$queueForStatsUsage
->addMetric(METRIC_DATABASES_OPERATIONS_WRITES, \max(1, $operations))
->addMetric(str_replace('{databaseInternalId}', $database->getSequence(), METRIC_DATABASE_ID_OPERATIONS_WRITES), \max(1, $operations)); // per collection
->addMetric($this->getDatabasesOperationWriteMetric(), \max(1, $operations))
->addMetric(str_replace('{databaseInternalId}', $database->getSequence(), $this->getDatabasesIdOperationWriteMetric()), \max(1, $operations)); // per collection
$response->setStatusCode(SwooleResponse::STATUS_CODE_CREATED);
if ($isBulk) {
$response->dynamic(new Document([
'total' => count($created),
$this->getSdkGroup() => $created
$this->getSDKGroup() => $created
]), $this->getBulkResponseModel());
$this->triggerBulk(
@@ -79,6 +79,7 @@ class Delete extends Action
->inject('requestTimestamp')
->inject('response')
->inject('dbForProject')
->inject('getDatabasesDB')
->inject('queueForEvents')
->inject('queueForStatsUsage')
->inject('transactionState')
@@ -95,6 +96,7 @@ class Delete extends Action
?\DateTime $requestTimestamp,
UtopiaResponse $response,
Database $dbForProject,
callable $getDatabasesDB,
Event $queueForEvents,
StatsUsage $queueForStatsUsage,
TransactionState $transactionState,
@@ -116,14 +118,15 @@ class Delete extends Action
throw new Exception($this->getParentNotFoundException(), params: [$collectionId]);
}
$dbForDatabases = $getDatabasesDB($database);
// Read permission should not be required for delete
$collectionTableId = 'database_' . $database->getSequence() . '_collection_' . $collection->getSequence();
if ($transactionId !== null) {
// Use transaction-aware document retrieval to see changes from same transaction
$document = $transactionState->getDocument($collectionTableId, $documentId, $transactionId);
$document = $transactionState->getDocument($database, $collectionTableId, $documentId, $transactionId);
} else {
$document = $authorization->skip(fn () => $dbForProject->getDocument($collectionTableId, $documentId));
$document = $authorization->skip(fn () => $dbForDatabases->getDocument($collectionTableId, $documentId));
}
if ($document->isEmpty()) {
@@ -187,8 +190,8 @@ class Delete extends Action
}
try {
$dbForProject->withRequestTimestamp($requestTimestamp, function () use ($dbForProject, $database, $collection, $documentId) {
$dbForProject->deleteDocument(
$dbForDatabases->withRequestTimestamp($requestTimestamp, function () use ($dbForDatabases, $database, $collection, $documentId) {
$dbForDatabases->deleteDocument(
'database_' . $database->getSequence() . '_collection_' . $collection->getSequence(),
$documentId
);
@@ -211,8 +214,8 @@ class Delete extends Action
);
$queueForStatsUsage
->addMetric(METRIC_DATABASES_OPERATIONS_WRITES, 1)
->addMetric(str_replace('{databaseInternalId}', $database->getSequence(), METRIC_DATABASE_ID_OPERATIONS_WRITES), 1); // per collection
->addMetric($this->getDatabasesOperationWriteMetric(), 1)
->addMetric(str_replace('{databaseInternalId}', $database->getSequence(), $this->getDatabasesIdOperationWriteMetric()), 1); // per collection
$response->addHeader('X-Debug-Operations', 1);
@@ -68,13 +68,14 @@ class Get extends Action
->param('transactionId', null, fn (Database $dbForProject) => new Nullable(new UID($dbForProject->getAdapter()->getMaxUIDLength())), 'Transaction ID to read uncommitted changes within the transaction.', true, ['dbForProject'])
->inject('response')
->inject('dbForProject')
->inject('getDatabasesDB')
->inject('queueForStatsUsage')
->inject('transactionState')
->inject('authorization')
->callback($this->action(...));
}
public function action(string $databaseId, string $collectionId, string $documentId, array $queries, ?string $transactionId, UtopiaResponse $response, Database $dbForProject, StatsUsage $queueForStatsUsage, TransactionState $transactionState, Authorization $authorization): void
public function action(string $databaseId, string $collectionId, string $documentId, array $queries, ?string $transactionId, UtopiaResponse $response, Database $dbForProject, callable $getDatabasesDB, StatsUsage $queueForStatsUsage, TransactionState $transactionState, Authorization $authorization): void
{
$isAPIKey = User::isApp($authorization->getRoles());
$isPrivilegedUser = User::isPrivileged($authorization->getRoles());
@@ -86,6 +87,7 @@ class Get extends Action
$collection = $authorization->skip(fn () => $dbForProject->getDocument('database_' . $database->getSequence(), $collectionId));
$dbForDatabases = $getDatabasesDB($database);
if ($collection->isEmpty() || (!$collection->getAttribute('enabled', false) && !$isAPIKey && !$isPrivilegedUser)) {
throw new Exception($this->getParentNotFoundException(), params: [$collectionId]);
}
@@ -99,16 +101,17 @@ class Get extends Action
try {
$selects = Query::groupByType($queries)['selections'] ?? [];
$collectionTableId = 'database_' . $database->getSequence() . '_collection_' . $collection->getSequence();
$collectionTableId = 'database_' . $database->getSequence() . '_collection_' . $collection->getSequence();
// Use transaction-aware document retrieval if transactionId is provided
if ($transactionId !== null) {
$document = $transactionState->getDocument($collectionTableId, $documentId, $transactionId, $queries);
$document = $transactionState->getDocument($database, $collectionTableId, $documentId, $transactionId, $queries);
} elseif (! empty($selects)) {
// has selects, allow relationship on documents!
$document = $dbForProject->getDocument($collectionTableId, $documentId, $queries);
$document = $dbForDatabases->getDocument($collectionTableId, $documentId, $queries);
} else {
// has no selects, disable relationship looping on documents!
$document = $dbForProject->skipRelationships(fn () => $dbForProject->getDocument($collectionTableId, $documentId, $queries));
$document = $dbForDatabases->skipRelationships(fn () => $dbForDatabases->getDocument($collectionTableId, $documentId, $queries));
}
} catch (QueryException $e) {
throw new Exception(Exception::GENERAL_QUERY_INVALID, $e->getMessage());
@@ -131,8 +134,8 @@ class Get extends Action
);
$queueForStatsUsage
->addMetric(METRIC_DATABASES_OPERATIONS_READS, max($operations, 1))
->addMetric(str_replace('{databaseInternalId}', $database->getSequence(), METRIC_DATABASE_ID_OPERATIONS_READS), $operations);
->addMetric($this->getDatabasesOperationReadMetric(), max($operations, 1))
->addMetric(str_replace('{databaseInternalId}', $database->getSequence(), $this->getDatabasesIdOperationReadMetric()), $operations);
$response->addHeader('X-Debug-Operations', $operations);
@@ -70,6 +70,7 @@ class XList extends Action
->param('queries', [], new Queries([new Limit(), new Offset()]), 'Array of query strings generated using the Query class provided by the SDK. [Learn more about queries](https://appwrite.io/docs/queries). Only supported methods are limit and offset', true)
->inject('response')
->inject('dbForProject')
->inject('getDatabasesDB')
->inject('locale')
->inject('geodb')
->inject('authorization')
@@ -77,7 +78,7 @@ class XList extends Action
->callback($this->action(...));
}
public function action(string $databaseId, string $collectionId, string $documentId, array $queries, UtopiaResponse $response, Database $dbForProject, Locale $locale, Reader $geodb, Authorization $authorization, Audit $audit): void
public function action(string $databaseId, string $collectionId, string $documentId, array $queries, UtopiaResponse $response, Database $dbForProject, callable $getDatabasesDB, Locale $locale, Reader $geodb, Authorization $authorization, Audit $audit): void
{
$database = $authorization->skip(fn () => $dbForProject->getDocument('databases', $databaseId));
if ($database->isEmpty()) {
@@ -89,7 +90,8 @@ class XList extends Action
throw new Exception($this->getParentNotFoundException(), params: [$collectionId]);
}
$document = $dbForProject->getDocument('database_' . $database->getSequence() . '_collection_' . $collection->getSequence(), $documentId);
$dbForDatabases = $getDatabasesDB($database);
$document = $dbForDatabases->getDocument('database_' . $database->getSequence() . '_collection_' . $collection->getSequence(), $documentId);
if ($document->isEmpty()) {
throw new Exception($this->getNotFoundException(), params: [$documentId]);
}
@@ -83,6 +83,7 @@ class Update extends Action
->inject('requestTimestamp')
->inject('response')
->inject('dbForProject')
->inject('getDatabasesDB')
->inject('queueForEvents')
->inject('queueForStatsUsage')
->inject('transactionState')
@@ -91,7 +92,7 @@ class Update extends Action
->callback($this->action(...));
}
public function action(string $databaseId, string $collectionId, string $documentId, string|array $data, ?array $permissions, ?string $transactionId, ?\DateTime $requestTimestamp, UtopiaResponse $response, Database $dbForProject, Event $queueForEvents, StatsUsage $queueForStatsUsage, TransactionState $transactionState, array $plan, Authorization $authorization): void
public function action(string $databaseId, string $collectionId, string $documentId, string|array $data, ?array $permissions, ?string $transactionId, ?\DateTime $requestTimestamp, UtopiaResponse $response, Database $dbForProject, callable $getDatabasesDB, Event $queueForEvents, StatsUsage $queueForStatsUsage, TransactionState $transactionState, array $plan, Authorization $authorization): void
{
$data = (\is_string($data)) ? \json_decode($data, true) : $data; // Cast to JSON array
@@ -118,15 +119,16 @@ class Update extends Action
$data = $this->parseOperators($data, $collection);
}
$dbForDatabases = $getDatabasesDB($database);
// Read permission should not be required for update
/** @var Document $document */
$collectionTableId = 'database_' . $database->getSequence() . '_collection_' . $collection->getSequence();
if ($transactionId !== null) {
// Use transaction-aware document retrieval to see changes from same transaction
$document = $transactionState->getDocument($collectionTableId, $documentId, $transactionId);
$document = $transactionState->getDocument($database, $collectionTableId, $documentId, $transactionId);
} else {
$document = $authorization->skip(fn () => $dbForProject->getDocument($collectionTableId, $documentId));
$document = $authorization->skip(fn () => $dbForDatabases->getDocument($collectionTableId, $documentId));
}
if ($document->isEmpty()) {
@@ -247,8 +249,8 @@ class Update extends Action
$setCollection($collection, $newDocument);
$queueForStatsUsage
->addMetric(METRIC_DATABASES_OPERATIONS_WRITES, max($operations, 1))
->addMetric(str_replace('{databaseInternalId}', $database->getSequence(), METRIC_DATABASE_ID_OPERATIONS_WRITES), $operations);
->addMetric($this->getDatabasesOperationWriteMetric(), max($operations, 1))
->addMetric(str_replace('{databaseInternalId}', $database->getSequence(), $this->getDatabasesIdOperationWriteMetric()), $operations);
// Handle transaction staging
if ($transactionId !== null) {
@@ -319,9 +321,9 @@ class Update extends Action
try {
$document = $dbForProject->withRequestTimestamp(
$document = $dbForDatabases->withRequestTimestamp(
$requestTimestamp,
fn () => $dbForProject->withPreserveDates(fn () => $dbForProject->updateDocument(
fn () => $dbForDatabases->withPreserveDates(fn () => $dbForDatabases->updateDocument(
'database_' . $database->getSequence() . '_collection_' . $collection->getSequence(),
$document->getId(),
$newDocument
@@ -87,6 +87,7 @@ class Upsert extends Action
->inject('response')
->inject('user')
->inject('dbForProject')
->inject('getDatabasesDB')
->inject('queueForEvents')
->inject('queueForStatsUsage')
->inject('transactionState')
@@ -95,7 +96,7 @@ class Upsert extends Action
->callback($this->action(...));
}
public function action(string $databaseId, string $collectionId, string $documentId, string|array $data, ?array $permissions, ?string $transactionId, ?\DateTime $requestTimestamp, UtopiaResponse $response, Document $user, Database $dbForProject, Event $queueForEvents, StatsUsage $queueForStatsUsage, TransactionState $transactionState, array $plan, Authorization $authorization): void
public function action(string $databaseId, string $collectionId, string $documentId, string|array $data, ?array $permissions, ?string $transactionId, ?\DateTime $requestTimestamp, UtopiaResponse $response, Document $user, Database $dbForProject, callable $getDatabasesDB, Event $queueForEvents, StatsUsage $queueForStatsUsage, TransactionState $transactionState, array $plan, Authorization $authorization): void
{
$data = (\is_string($data)) ? \json_decode($data, true) : $data; // Cast to JSON array
@@ -124,6 +125,7 @@ class Upsert extends Action
$data = $this->parseOperators($data, $collection);
}
$dbForDatabases = $getDatabasesDB($database);
$allowedPermissions = [
Database::PERMISSION_READ,
Database::PERMISSION_UPDATE,
@@ -134,13 +136,15 @@ class Upsert extends Action
$collectionTableId = 'database_' . $database->getSequence() . '_collection_' . $collection->getSequence();
$collectionTableId = 'database_' . $database->getSequence() . '_collection_' . $collection->getSequence();
// If no permission, upsert permission from the old document if present (update scenario) else add default permission (create scenario)
if (\is_null($permissions)) {
if ($transactionId !== null) {
// Use transaction-aware document retrieval to see changes from same transaction
$oldDocument = $transactionState->getDocument($collectionTableId, $documentId, $transactionId);
$oldDocument = $transactionState->getDocument($database, $collectionTableId, $documentId, $transactionId);
} else {
$oldDocument = $authorization->skip(fn () => $dbForProject->getDocument($collectionTableId, $documentId));
$oldDocument = $authorization->skip(fn () => $dbForDatabases->getDocument($collectionTableId, $documentId));
}
if ($oldDocument->isEmpty()) {
if (!empty($user->getId())) {
@@ -182,7 +186,7 @@ class Upsert extends Action
$newDocument = new Document($data);
$operations = 0;
$setCollection = (function (Document $collection, Document $document) use ($isAPIKey, $isPrivilegedUser, &$setCollection, $dbForProject, $database, &$operations, $authorization) {
$setCollection = (function (Document $collection, Document $document) use ($isAPIKey, $isPrivilegedUser, &$setCollection, $dbForProject, $dbForDatabases, $database, &$operations, $authorization) {
$operations++;
$relationships = \array_filter(
@@ -226,7 +230,7 @@ class Upsert extends Action
if ($relation instanceof Document) {
$relation = $this->removeReadonlyAttributes($relation, $isAPIKey || $isPrivilegedUser);
$oldDocument = $authorization->skip(fn () => $dbForProject->getDocument(
$oldDocument = $authorization->skip(fn () => $dbForDatabases->getDocument(
'database_' . $database->getSequence() . '_collection_' . $relatedCollection->getSequence(),
$relation->getId()
));
@@ -257,8 +261,8 @@ class Upsert extends Action
$setCollection($collection, $newDocument);
$queueForStatsUsage
->addMetric(METRIC_DATABASES_OPERATIONS_WRITES, \max(1, $operations))
->addMetric(str_replace('{databaseInternalId}', $database->getSequence(), METRIC_DATABASE_ID_OPERATIONS_WRITES), \max(1, $operations));
->addMetric($this->getDatabasesOperationWriteMetric(), \max(1, $operations))
->addMetric(str_replace('{databaseInternalId}', $database->getSequence(), $this->getDatabasesIdOperationWriteMetric()), \max(1, $operations));
// Handle transaction staging
if ($transactionId !== null) {
@@ -327,8 +331,8 @@ class Upsert extends Action
$upserted = [];
try {
$dbForProject->withPreserveDates(function () use (&$upserted, $dbForProject, $database, $collection, $newDocument) {
return $dbForProject->upsertDocuments(
$dbForDatabases->withPreserveDates(function () use (&$upserted, $dbForDatabases, $database, $collection, $newDocument) {
return $dbForDatabases->upsertDocuments(
'database_' . $database->getSequence() . '_collection_' . $collection->getSequence(),
[$newDocument],
onNext: function (Document $document) use (&$upserted) {
@@ -351,9 +355,9 @@ class Upsert extends Action
if (empty($upserted[0])) {
if ($transactionId !== null) {
// For transactions, get the document with transaction changes applied
$upserted[0] = $transactionState->getDocument($collectionTableId, $documentId, $transactionId);
$upserted[0] = $transactionState->getDocument($database, $collectionTableId, $documentId, $transactionId);
} else {
$upserted[0] = $dbForProject->getDocument($collectionTableId, $documentId);
$upserted[0] = $dbForDatabases->getDocument($collectionTableId, $documentId);
}
}
@@ -75,13 +75,14 @@ class XList extends Action
->inject('response')
->inject('dbForProject')
->inject('user')
->inject('getDatabasesDB')
->inject('queueForStatsUsage')
->inject('transactionState')
->inject('authorization')
->callback($this->action(...));
}
public function action(string $databaseId, string $collectionId, array $queries, ?string $transactionId, bool $includeTotal, int $ttl, UtopiaResponse $response, Database $dbForProject, Document $user, StatsUsage $queueForStatsUsage, TransactionState $transactionState, Authorization $authorization): void
public function action(string $databaseId, string $collectionId, array $queries, ?string $transactionId, bool $includeTotal, int $ttl, UtopiaResponse $response, Database $dbForProject, Document $user, callable $getDatabasesDB, StatsUsage $queueForStatsUsage, TransactionState $transactionState, Authorization $authorization): void
{
$isAPIKey = User::isApp($authorization->getRoles());
$isPrivilegedUser = User::isPrivileged($authorization->getRoles());
@@ -102,6 +103,7 @@ class XList extends Action
throw new Exception(Exception::GENERAL_QUERY_INVALID, $e->getMessage());
}
$dbForDatabases = $getDatabasesDB($database);
$cursor = Query::getCursorQueries($queries, false);
$cursor = \reset($cursor);
@@ -113,7 +115,7 @@ class XList extends Action
$documentId = $cursor->getValue();
$cursorDocument = $authorization->skip(fn () => $dbForProject->getDocument('database_' . $database->getSequence() . '_collection_' . $collection->getSequence(), $documentId));
$cursorDocument = $authorization->skip(fn () => $dbForDatabases->getDocument('database_' . $database->getSequence() . '_collection_' . $collection->getSequence(), $documentId));
if ($cursorDocument->isEmpty()) {
$type = ucfirst($this->getContext());
@@ -126,11 +128,10 @@ class XList extends Action
try {
$selectQueries = Query::groupByType($queries)['selections'] ?? [];
$collectionTableId = 'database_' . $database->getSequence() . '_collection_' . $collection->getSequence();
// Use transaction-aware document retrieval if transactionId is provided
if ($transactionId !== null) {
$documents = $transactionState->listDocuments($collectionTableId, $transactionId, $queries);
$total = $includeTotal ? $transactionState->countDocuments($collectionTableId, $transactionId, $queries) : 0;
$documents = $transactionState->listDocuments($database, $collectionTableId, $transactionId, $queries);
$total = $includeTotal ? $transactionState->countDocuments($database, $collectionTableId, $transactionId, $queries) : 0;
} elseif (! empty($selectQueries)) {
if ((int)$ttl > 0) {
@@ -169,7 +170,7 @@ class XList extends Action
}, $cachedDocuments);
$documentsCacheHit = true;
} else {
$documents = $dbForProject->find($collectionTableId, $queries);
$documents = $dbForDatabases->find($collectionTableId, $queries);
// Convert Document objects to arrays for caching
$documentsArray = \array_map(function ($doc) {
@@ -195,15 +196,15 @@ class XList extends Action
} else {
// has selects, allow relationship on documents
$documents = $dbForProject->find($collectionTableId, $queries);
$total = $includeTotal ? $dbForProject->count($collectionTableId, $queries, APP_LIMIT_COUNT) : 0;
$documents = $dbForDatabases->find($collectionTableId, $queries);
$total = $includeTotal ? $dbForDatabases->count($collectionTableId, $queries, APP_LIMIT_COUNT) : 0;
}
} else {
// has no selects, disable relationship loading on documents
/* @type Document[] $documents */
$documents = $dbForProject->skipRelationships(fn () => $dbForProject->find($collectionTableId, $queries));
$total = $includeTotal ? $dbForProject->count($collectionTableId, $queries, APP_LIMIT_COUNT) : 0;
$documents = $dbForDatabases->skipRelationships(fn () => $dbForDatabases->find($collectionTableId, $queries));
$total = $includeTotal ? $dbForDatabases->count($collectionTableId, $queries, APP_LIMIT_COUNT) : 0;
}
} catch (OrderException $e) {
$documents = $this->isCollectionsAPI() ? 'documents' : 'rows';
@@ -229,8 +230,8 @@ class XList extends Action
}
$queueForStatsUsage
->addMetric(METRIC_DATABASES_OPERATIONS_READS, max($operations, 1))
->addMetric(str_replace('{databaseInternalId}', $database->getSequence(), METRIC_DATABASE_ID_OPERATIONS_READS), $operations);
->addMetric($this->getDatabasesOperationReadMetric(), max($operations, 1))
->addMetric(str_replace('{databaseInternalId}', $database->getSequence(), $this->getDatabasesIdOperationReadMetric()), $operations);
$response->dynamic(new Document([
'total' => $total,
@@ -77,13 +77,14 @@ class Create extends Action
->param('lengths', [], new ArrayList(new Nullable(new Integer()), APP_LIMIT_ARRAY_PARAMS_SIZE), 'Length of index. Maximum of ' . APP_LIMIT_ARRAY_PARAMS_SIZE, optional: true)
->inject('response')
->inject('dbForProject')
->inject('getDatabasesDB')
->inject('queueForDatabase')
->inject('queueForEvents')
->inject('authorization')
->callback($this->action(...));
}
public function action(string $databaseId, string $collectionId, string $key, string $type, array $attributes, array $orders, array $lengths, UtopiaResponse $response, Database $dbForProject, EventDatabase $queueForDatabase, Event $queueForEvents, Authorization $authorization): void
public function action(string $databaseId, string $collectionId, string $key, string $type, array $attributes, array $orders, array $lengths, UtopiaResponse $response, Database $dbForProject, callable $getDatabasesDB, EventDatabase $queueForDatabase, Event $queueForEvents, Authorization $authorization): void
{
$db = $authorization->skip(fn () => $dbForProject->getDocument('databases', $databaseId));
@@ -103,7 +104,9 @@ class Create extends Action
Query::equal('databaseInternalId', [$db->getSequence()])
], 61);
$limit = $dbForProject->getLimitForIndexes();
$dbForDatabases = $getDatabasesDB($db);
$limit = $dbForDatabases->getLimitForIndexes();
if ($count >= $limit) {
throw new Exception($this->getLimitException(), params: [$collectionId]);
@@ -145,32 +148,35 @@ class Create extends Action
];
$contextType = $this->getParentContext();
foreach ($attributes as $i => $attribute) {
$attributeIndex = \array_search($attribute, array_column($oldAttributes, 'key'));
if ($dbForDatabases->getAdapter()->getSupportForAttributes()) {
foreach ($attributes as $i => $attribute) {
// find attribute metadata in collection document
$attributeIndex = \array_search($attribute, array_column($oldAttributes, 'key'));
if ($attributeIndex === false) {
throw new Exception($this->getParentUnknownException(), params: [$attribute]);
}
if ($attributeIndex === false) {
throw new Exception($this->getParentUnknownException(), params: [$attribute]);
}
$attributeStatus = $oldAttributes[$attributeIndex]['status'];
$attributeType = $oldAttributes[$attributeIndex]['type'];
$attributeArray = $oldAttributes[$attributeIndex]['array'] ?? false;
$attributeStatus = $oldAttributes[$attributeIndex]['status'];
$attributeType = $oldAttributes[$attributeIndex]['type'];
$attributeArray = $oldAttributes[$attributeIndex]['array'] ?? false;
if ($attributeType === Database::VAR_RELATIONSHIP) {
throw new Exception($this->getParentInvalidTypeException(), "Cannot create an index for a relationship $contextType: " . $oldAttributes[$attributeIndex]['key']);
}
if ($attributeType === Database::VAR_RELATIONSHIP) {
throw new Exception($this->getParentInvalidTypeException(), "Cannot create an index for a relationship $contextType: " . $oldAttributes[$attributeIndex]['key']);
}
if ($attributeStatus !== 'available') {
throw new Exception($this->getParentNotAvailableException(), params: [$oldAttributes[$attributeIndex]['key']]);
}
if ($attributeStatus !== 'available') {
throw new Exception($this->getParentNotAvailableException(), params: [$oldAttributes[$attributeIndex]['key']]);
}
if (empty($lengths[$i])) {
$lengths[$i] = null;
}
if (empty($lengths[$i])) {
$lengths[$i] = null;
}
if ($attributeArray === true) {
// Because of a bug in MySQL, we cannot create indexes on array attributes for now, otherwise queries break.
throw new Exception(Exception::INDEX_INVALID, 'Creating indexes on array attributes is not currently supported.');
if ($attributeArray === true) {
// Because of a bug in MySQL, we cannot create indexes on array attributes for now, otherwise queries break.
throw new Exception(Exception::INDEX_INVALID, 'Creating indexes on array attributes is not currently supported.');
}
}
}
@@ -191,21 +197,23 @@ class Create extends Action
$validator = new IndexValidator(
$collection->getAttribute('attributes'),
$collection->getAttribute('indexes'),
$dbForProject->getAdapter()->getMaxIndexLength(),
$dbForProject->getAdapter()->getInternalIndexesKeys(),
$dbForProject->getAdapter()->getSupportForIndexArray(),
$dbForProject->getAdapter()->getSupportForSpatialIndexNull(),
$dbForProject->getAdapter()->getSupportForSpatialIndexOrder(),
$dbForProject->getAdapter()->getSupportForVectors(),
$dbForProject->getAdapter()->getSupportForAttributes(),
$dbForProject->getAdapter()->getSupportForMultipleFulltextIndexes(),
$dbForProject->getAdapter()->getSupportForIdenticalIndexes(),
$dbForProject->getAdapter()->getSupportForObjectIndexes(),
$dbForProject->getAdapter()->getSupportForTrigramIndex(),
$dbForProject->getAdapter()->getSupportForSpatialAttributes(),
$dbForProject->getAdapter()->getSupportForIndex(),
$dbForProject->getAdapter()->getSupportForUniqueIndex(),
$dbForProject->getAdapter()->getSupportForFulltextIndex(),
$dbForDatabases->getAdapter()->getMaxIndexLength(),
$dbForDatabases->getAdapter()->getInternalIndexesKeys(),
$dbForDatabases->getAdapter()->getSupportForIndexArray(),
$dbForDatabases->getAdapter()->getSupportForSpatialIndexNull(),
$dbForDatabases->getAdapter()->getSupportForSpatialIndexOrder(),
$dbForDatabases->getAdapter()->getSupportForVectors(),
$dbForDatabases->getAdapter()->getSupportForAttributes(),
$dbForDatabases->getAdapter()->getSupportForMultipleFulltextIndexes(),
$dbForDatabases->getAdapter()->getSupportForIdenticalIndexes(),
$dbForDatabases->getAdapter()->getSupportForObjectIndexes(),
$dbForDatabases->getAdapter()->getSupportForTrigramIndex(),
$dbForDatabases->getAdapter()->getSupportForSpatialAttributes(),
$dbForDatabases->getAdapter()->getSupportForIndex(),
$dbForDatabases->getAdapter()->getSupportForUniqueIndex(),
$dbForDatabases->getAdapter()->getSupportForFulltextIndex(),
$dbForDatabases->getAdapter()->getSupportForTTLIndexes(),
$dbForDatabases->getAdapter()->getSupportForObject()
);
if (!$validator->isValid($index)) {
@@ -70,12 +70,13 @@ class Update extends Action
->param('enabled', true, new Boolean(), 'Is collection enabled? When set to \'disabled\', users cannot access the collection but Server SDKs with and API key can still read and write to the collection. No data is lost when this is toggled.', true)
->inject('response')
->inject('dbForProject')
->inject('getDatabasesDB')
->inject('queueForEvents')
->inject('authorization')
->callback($this->action(...));
}
public function action(string $databaseId, string $collectionId, ?string $name, ?array $permissions, bool $documentSecurity, bool $enabled, UtopiaResponse $response, Database $dbForProject, Event $queueForEvents, Authorization $authorization): void
public function action(string $databaseId, string $collectionId, ?string $name, ?array $permissions, bool $documentSecurity, bool $enabled, UtopiaResponse $response, Database $dbForProject, callable $getDatabasesDB, Event $queueForEvents, Authorization $authorization): void
{
$database = $authorization->skip(fn () => $dbForProject->getDocument('databases', $databaseId));
if ($database->isEmpty()) {
@@ -110,7 +111,8 @@ class Update extends Action
->setAttribute('search', \implode(' ', [$collectionId, $searchName]))
);
$dbForProject->updateCollection('database_' . $database->getSequence() . '_collection_' . $collection->getSequence(), $permissions, $documentSecurity);
$dbForDatabases = $getDatabasesDB($database);
$dbForDatabases->updateCollection('database_' . $database->getSequence() . '_collection_' . $collection->getSequence(), $permissions, $documentSecurity);
$queueForEvents
->setContext('database', $database)
@@ -31,6 +31,11 @@ class Get extends Action
return UtopiaResponse::MODEL_USAGE_COLLECTION;
}
protected function getMetric(): string
{
return METRIC_DATABASE_ID_COLLECTION_ID_DOCUMENTS;
}
public function __construct()
{
$this
@@ -64,14 +69,16 @@ class Get extends Action
->inject('response')
->inject('dbForProject')
->inject('authorization')
->inject('getDatabasesDB')
->callback($this->action(...));
}
public function action(string $databaseId, string $range, string $collectionId, UtopiaResponse $response, Database $dbForProject, Authorization $authorization): void
public function action(string $databaseId, string $range, string $collectionId, UtopiaResponse $response, Database $dbForProject, Authorization $authorization, callable $getDatabasesDB): void
{
$database = $dbForProject->getDocument('databases', $databaseId);
$collectionDocument = $dbForProject->getDocument('database_' . $database->getSequence(), $collectionId);
$collection = $dbForProject->getCollection('database_' . $database->getSequence() . '_collection_' . $collectionDocument->getSequence());
$dbForDatabases = $getDatabasesDB($database);
$collection = $dbForDatabases->getCollection('database_' . $database->getSequence() . '_collection_' . $collectionDocument->getSequence());
if ($collection->isEmpty()) {
throw new Exception($this->getNotFoundException(), params: [$collectionId]);
@@ -81,7 +88,7 @@ class Get extends Action
$stats = $usage = [];
$days = $periods[$range];
$metrics = [
str_replace(['{databaseInternalId}', '{collectionInternalId}'], [$database->getSequence(), $collectionDocument->getSequence()], METRIC_DATABASE_ID_COLLECTION_ID_DOCUMENTS),
str_replace(['{databaseInternalId}', '{collectionInternalId}'], [$database->getSequence(), $collectionDocument->getSequence()], $this->getMetric()),
];
$authorization->skip(function () use ($dbForProject, $days, $metrics, &$stats) {
@@ -19,7 +19,9 @@ use Utopia\Database\Exception\Index as IndexException;
use Utopia\Database\Exception\Limit as LimitException;
use Utopia\Database\Exception\Structure as StructureException;
use Utopia\Database\Helpers\ID;
use Utopia\DSN\DSN;
use Utopia\Http\Adapter\Swoole\Response as SwooleResponse;
use Utopia\System\System;
use Utopia\Validator\Boolean;
use Utopia\Validator\Text;
@@ -30,6 +32,121 @@ class Create extends Action
return 'createDatabase';
}
protected function getDatabaseDSN(Document $project): string
{
// TODO: use database worker for for creating the v2 schema if not present
// it is considered that the v2 metadata schema is already created during server start in the http.php
return $this->constructDatabaseDSNFromProjectDatabase($this->getDatabaseType(), $project->getAttribute('region'), $project->getAttribute('database'));
}
private function constructDatabaseDSNFromProjectDatabase(string $databasetype, $region, ?string $dsn = null): string
{
$databases = [];
$databaseKeys = [];
/**
* @var string|null $databaseOverride
*/
$databaseOverride = '';
$dbScheme = '';
$databaseSharedTables = [];
$databaseSharedTablesV1 = [];
$databaseSharedTablesV2 = [];
$projectSharedTables = [];
$projectSharedTablesV1 = [];
$projectSharedTablesV2 = [];
switch ($databasetype) {
case DOCUMENTSDB:
$databases = Config::getParam('pools-documentsdb', []);
$databaseKeys = System::getEnv('_APP_DATABASE_DOCUMENTSDB_KEYS', '');
$databaseOverride = System::getEnv('_APP_DATABASE_DOCUMENTSDB_OVERRIDE');
$dbScheme = System::getEnv('_APP_DB_HOST_DOCUMENTSDB', 'mongodb');
$databaseSharedTables = \explode(',', System::getEnv('_APP_DATABASE_DOCUMENTSDB_SHARED_TABLES', ''));
$databaseSharedTablesV1 = \explode(',', System::getEnv('_APP_DATABASE_DOCUMENTSDB_SHARED_TABLES_V1', ''));
break;
case VECTORSDB:
$databases = Config::getParam('pools-vectorsdb', []);
$databaseKeys = System::getEnv('_APP_DATABASE_VECTORSDB_KEYS', '');
$databaseOverride = System::getEnv('_APP_DATABASE_VECTORSDB_OVERRIDE');
$dbScheme = System::getEnv('_APP_DB_HOST_VECTORSDB', 'postgresql');
$databaseSharedTables = \explode(',', System::getEnv('_APP_DATABASE_VECTORSDB_SHARED_TABLES', ''));
$databaseSharedTablesV1 = \explode(',', System::getEnv('_APP_DATABASE_VECTORSDB_SHARED_TABLES_V1', ''));
break;
default:
// legacy/tablesdb
// it is already created during create project
return $dsn;
}
$isSharedTablesV1 = false;
$isSharedTablesV2 = false;
if (!empty($dsn)) {
try {
$parsedDsn = new DSN($dsn);
$dsnHost = $parsedDsn->getHost();
} catch (\InvalidArgumentException) {
$dsnHost = $dsn;
}
$projectSharedTables = \explode(',', System::getEnv('_APP_DATABASE_SHARED_TABLES', ''));
$projectSharedTablesV1 = \explode(',', System::getEnv('_APP_DATABASE_SHARED_TABLES_V1', ''));
$projectSharedTablesV2 = \array_diff($projectSharedTables, $projectSharedTablesV1);
$isSharedTablesV1 = \in_array($dsnHost, $projectSharedTablesV1);
$isSharedTablesV2 = \in_array($dsnHost, $projectSharedTablesV2);
}
if ($region !== 'default') {
$keys = explode(',', $databaseKeys);
$databases = array_filter($keys, function ($value) use ($region) {
return str_contains($value, $region);
});
}
$databaseSharedTablesV2 = \array_diff($databaseSharedTables, $databaseSharedTablesV1);
$index = \array_search($databaseOverride, $databases);
if ($index !== false) {
$selectedDsn = $databases[$index];
} else {
if (!empty($dsn)) {
$beforeFilter = \array_values($databases);
if ($isSharedTablesV1) {
$databases = array_filter($databases, fn ($value) => \in_array($value, $databaseSharedTablesV1));
} elseif ($isSharedTablesV2) {
$databases = array_filter($databases, fn ($value) => \in_array($value, $databaseSharedTablesV2));
} else {
$databases = array_filter($databases, fn ($value) => !\in_array($value, $databaseSharedTables));
}
}
$selectedDsn = !empty($databases) ? $databases[array_rand($databases)] : '';
}
if (\in_array($selectedDsn, $databaseSharedTables)) {
$schema = 'appwrite';
$database = 'appwrite';
$namespace = System::getEnv('_APP_DATABASE_SHARED_NAMESPACE', '');
$selectedDsn = $schema . '://' . $selectedDsn . '?database=' . $database;
if (!empty($namespace)) {
$selectedDsn .= '&namespace=' . $namespace;
}
}
try {
new DSN($selectedDsn);
} catch (\InvalidArgumentException) {
$selectedDsn = $dbScheme.'://' . $selectedDsn;
}
return $selectedDsn;
}
protected function getDatabaseCollection()
{
return match ($this->getDatabaseType()) {
'vectorsdb' => (Config::getParam('collections', [])['vectorsdb'] ?? [])['collections'] ?? [],
default => (Config::getParam('collections', [])['databases'] ?? [])['collections'] ?? [],
};
}
public function __construct()
{
$this
@@ -65,13 +182,15 @@ class Create extends Action
->param('databaseId', '', fn (Database $dbForProject) => new CustomId(false, $dbForProject->getAdapter()->getMaxUIDLength()), 'Unique Id. Choose a custom ID or generate a random ID with `ID.unique()`. Valid chars are a-z, A-Z, 0-9, period, hyphen, and underscore. Can\'t start with a special char. Max length is 36 chars.', false, ['dbForProject'])
->param('name', '', new Text(128), 'Database name. Max length: 128 chars.')
->param('enabled', true, new Boolean(), 'Is the database enabled? When set to \'disabled\', users cannot access the database but Server SDKs with an API key can still read and write to the database. No data is lost when this is toggled.', true)
->inject('project')
->inject('response')
->inject('dbForProject')
->inject('getDatabasesDB')
->inject('queueForEvents')
->callback($this->action(...));
}
public function action(string $databaseId, string $name, bool $enabled, UtopiaResponse $response, Database $dbForProject, Event $queueForEvents): void
public function action(string $databaseId, string $name, bool $enabled, Document $project, UtopiaResponse $response, Database $dbForProject, callable $getDatabasesDB, Event $queueForEvents): void
{
$databaseId = $databaseId == 'unique()' ? ID::unique() : $databaseId;
@@ -82,6 +201,7 @@ class Create extends Action
'enabled' => $enabled,
'search' => implode(' ', [$databaseId, $name]),
'type' => $this->getDatabaseType(),
'database' => $this->getDatabaseDSN($project)
]));
} catch (DuplicateException) {
throw new Exception(Exception::DATABASE_ALREADY_EXISTS, params: [$databaseId]);
@@ -91,7 +211,7 @@ class Create extends Action
$database = $dbForProject->getDocument('databases', $databaseId);
$collections = (Config::getParam('collections', [])['databases'] ?? [])['collections'] ?? [];
$collections = $this->getDatabaseCollection();
if (empty($collections)) {
throw new Exception(Exception::GENERAL_SERVER_ERROR, 'The "collections" collection is not configured.');
}
@@ -10,11 +10,45 @@ abstract class Action extends DatabasesAction
* The current API context (either 'table' or 'collection').
*/
private ?string $context = COLLECTIONS;
private ?string $databaseType = LEGACY;
public function getDatabaseType(): string
{
return $this->databaseType;
}
protected function getDatabasesOperationWriteMetric(): string
{
if ($this->databaseType === LEGACY || $this->databaseType === TABLESDB) {
return METRIC_DATABASES_OPERATIONS_WRITES;
}
return $this->databaseType.'.'.METRIC_DATABASES_OPERATIONS_WRITES;
}
protected function getDatabasesIdOperationWriteMetric(): string
{
if ($this->databaseType === LEGACY || $this->databaseType === TABLESDB) {
return METRIC_DATABASE_ID_OPERATIONS_WRITES;
}
return $this->databaseType.'.'.METRIC_DATABASE_ID_OPERATIONS_WRITES;
}
public function setHttpPath(string $path): DatabasesAction
{
if (\str_contains($path, '/tablesdb')) {
$this->context = TABLES;
switch (true) {
case str_contains($path, '/tablesdb'):
$this->context = TABLES;
$this->databaseType = TABLESDB;
break;
case str_contains($path, '/documentsdb'):
$this->context = COLLECTIONS;
$this->databaseType = DOCUMENTSDB;
break;
case str_contains($path, '/vectorsdb'):
$this->context = COLLECTIONS;
$this->databaseType = VECTORSDB;
break;
}
return parent::setHttpPath($path);
}
@@ -148,7 +148,7 @@ class Create extends Action
$collectionKey = 'database_' . $database->getSequence() . '_collection_' . $collection->getSequence();
$isDependant = isset($dependants[$collectionKey][$documentId]);
$document = $transactionState->getDocument($collectionKey, $documentId, $transactionId);
$document = $transactionState->getDocument($database, $collectionKey, $documentId, $transactionId);
if ($document->isEmpty() && !$isDependant && $operation['action'] !== 'upsert') {
throw new Exception(Exception::DOCUMENT_NOT_FOUND, params: [$documentId]);
}
@@ -67,8 +67,10 @@ class Update extends Action
->param('transactionId', '', fn (Database $dbForProject) => new UID($dbForProject->getAdapter()->getMaxUIDLength()), 'Transaction ID.', false, ['dbForProject'])
->param('commit', false, new Boolean(), 'Commit transaction?', true)
->param('rollback', false, new Boolean(), 'Rollback transaction?', true)
->inject('project')
->inject('response')
->inject('dbForProject')
->inject('getDatabasesDB')
->inject('user')
->inject('transactionState')
->inject('queueForDeletes')
@@ -88,6 +90,7 @@ class Update extends Action
* @param bool $rollback
* @param UtopiaResponse $response
* @param Database $dbForProject
* @param callable $getDatabasesDB
* @param Document $user
* @param TransactionState $transactionState
* @param Delete $queueForDeletes
@@ -106,7 +109,7 @@ class Update extends Action
* @throws Structure
* @throws \Utopia\Http\Exception
*/
public function action(string $transactionId, bool $commit, bool $rollback, UtopiaResponse $response, Database $dbForProject, Document $user, TransactionState $transactionState, Delete $queueForDeletes, Event $queueForEvents, StatsUsage $queueForStatsUsage, Event $queueForRealtime, Event $queueForFunctions, Event $queueForWebhooks, Authorization $authorization, EventProcessor $eventProcessor): void
public function action(string $transactionId, bool $commit, bool $rollback, Document $project, UtopiaResponse $response, Database $dbForProject, callable $getDatabasesDB, Document $user, TransactionState $transactionState, Delete $queueForDeletes, Event $queueForEvents, StatsUsage $queueForStatsUsage, Event $queueForRealtime, Event $queueForFunctions, Event $queueForWebhooks, Authorization $authorization, EventProcessor $eventProcessor): void
{
if (!$commit && !$rollback) {
throw new Exception(Exception::GENERAL_BAD_REQUEST, 'Either commit or rollback must be true');
@@ -135,14 +138,52 @@ class Update extends Action
}
if ($commit) {
$operations = [];
$totalOperations = 0;
$databaseOperations = [];
$currentDocumentId = null;
$firstOperation = $authorization->skip(fn () => $dbForProject->findOne('transactionLogs', [
Query::equal('transactionInternalId', [$transaction->getSequence()]),
Query::orderAsc(),
]));
if ($firstOperation->isEmpty()) {
$transaction = $authorization->skip(fn () => $dbForProject->updateDocument(
'transactions',
$transactionId,
new Document(['status' => 'committed'])
));
$queueForDeletes
->setType(DELETE_TYPE_DOCUMENT)
->setDocument($transaction);
$response
->setStatusCode(SwooleResponse::STATUS_CODE_OK)
->dynamic($transaction, $this->getResponseModel());
return;
}
$databaseDoc = null;
switch ($this->getDatabaseType()) {
case DATABASE_TYPE_DOCUMENTSDB:
case DATABASE_TYPE_VECTORSDB:
$databaseDoc = $authorization->skip(fn () => $dbForProject->findOne('databases', [
Query::equal('$sequence', [$firstOperation['databaseInternalId']])
]));
break;
default:
// Legacy/tablesdb: use project-level database
$databaseDoc = new Document(['database' => $project->getAttribute('database')]);
break;
}
$dbForDatabases = $getDatabasesDB($databaseDoc);
try {
$dbForProject->withTransaction(function () use ($dbForProject, $transactionState, $queueForDeletes, $transactionId, &$transaction, &$operations, &$totalOperations, &$databaseOperations, &$currentDocumentId, $queueForEvents, $queueForStatsUsage, $queueForRealtime, $queueForFunctions, $queueForWebhooks, $authorization) {
$dbForDatabases->withTransaction(function () use ($dbForDatabases, $dbForProject, $transactionState, $queueForDeletes, $transactionId, &$transaction, &$operations, &$totalOperations, &$databaseOperations, &$currentDocumentId, $queueForEvents, $queueForStatsUsage, $queueForRealtime, $queueForFunctions, $queueForWebhooks, $authorization) {
$authorization->skip(fn () => $dbForProject->updateDocument('transactions', $transactionId, new Document([
'status' => 'committing',
])));
@@ -182,7 +223,7 @@ class Update extends Action
}
if ($action === 'delete' && $documentId && empty($data)) {
$doc = $dbForProject->getDocument($collectionId, $documentId);
$doc = $dbForDatabases->getDocument($collectionId, $documentId);
if (!$doc->isEmpty()) {
$operation['data'] = $doc->getArrayCopy();
$data = $operation['data'];
@@ -196,40 +237,40 @@ class Update extends Action
switch ($action) {
case 'create':
$this->handleCreateOperation($dbForProject, $collectionId, $documentId, $data, $createdAt, $state);
$this->handleCreateOperation($dbForDatabases, $collectionId, $documentId, $data, $createdAt, $state);
break;
case 'update':
$this->handleUpdateOperation($dbForProject, $collectionId, $documentId, $data, $createdAt, $state);
$this->handleUpdateOperation($dbForDatabases, $collectionId, $documentId, $data, $createdAt, $state);
break;
case 'upsert':
$this->handleUpsertOperation($dbForProject, $collectionId, $documentId, $data, $createdAt, $state);
$this->handleUpsertOperation($dbForDatabases, $collectionId, $documentId, $data, $createdAt, $state);
break;
case 'delete':
$this->handleDeleteOperation($dbForProject, $collectionId, $documentId, $createdAt, $state);
$this->handleDeleteOperation($dbForDatabases, $collectionId, $documentId, $createdAt, $state);
break;
case 'increment':
$this->handleIncrementOperation($dbForProject, $collectionId, $documentId, $data, $createdAt, $state);
$this->handleIncrementOperation($dbForDatabases, $collectionId, $documentId, $data, $createdAt, $state);
break;
case 'decrement':
$this->handleDecrementOperation($dbForProject, $collectionId, $documentId, $data, $createdAt, $state);
$this->handleDecrementOperation($dbForDatabases, $collectionId, $documentId, $data, $createdAt, $state);
break;
case 'bulkCreate':
$count = $this->handleBulkCreateOperation($dbForProject, $collectionId, $data, $createdAt, $state);
$count = $this->handleBulkCreateOperation($dbForDatabases, $collectionId, $data, $createdAt, $state);
$totalOperations += $count;
$databaseOperations[$databaseInternalId] = ($databaseOperations[$databaseInternalId] ?? 0) + $count;
break;
case 'bulkUpdate':
$count = $this->handleBulkUpdateOperation($dbForProject, $transactionState, $collectionId, $data, $createdAt, $state);
$count = $this->handleBulkUpdateOperation($dbForDatabases, $transactionState, $collectionId, $data, $createdAt, $state);
$totalOperations += $count;
$databaseOperations[$databaseInternalId] = ($databaseOperations[$databaseInternalId] ?? 0) + $count;
break;
case 'bulkUpsert':
$count = $this->handleBulkUpsertOperation($dbForProject, $transactionState, $collectionId, $data, $createdAt, $state);
$count = $this->handleBulkUpsertOperation($dbForDatabases, $transactionState, $collectionId, $data, $createdAt, $state);
$totalOperations += $count;
$databaseOperations[$databaseInternalId] = ($databaseOperations[$databaseInternalId] ?? 0) + $count;
break;
case 'bulkDelete':
$count = $this->handleBulkDeleteOperation($dbForProject, $transactionState, $collectionId, $data, $createdAt, $state);
$count = $this->handleBulkDeleteOperation($dbForDatabases, $transactionState, $collectionId, $data, $createdAt, $state);
$totalOperations += $count;
$databaseOperations[$databaseInternalId] = ($databaseOperations[$databaseInternalId] ?? 0) + $count;
break;
@@ -280,15 +321,16 @@ class Update extends Action
}
$queueForStatsUsage
->addMetric(METRIC_DATABASES_OPERATIONS_WRITES, $totalOperations);
->addMetric($this->getDatabasesOperationWriteMetric(), $totalOperations);
foreach ($databaseOperations as $sequence => $count) {
$queueForStatsUsage->addMetric(
str_replace('{databaseInternalId}', $sequence, METRIC_DATABASE_ID_OPERATIONS_WRITES),
str_replace('{databaseInternalId}', $sequence, $this->getDatabasesIdOperationWriteMetric()),
$count
);
}
$dbCache = [];
foreach ($operations as $operation) {
$databaseInternalId = $operation['databaseInternalId'];
$collectionInternalId = $operation['collectionInternalId'];
@@ -301,6 +343,16 @@ class Update extends Action
$data = $data->getArrayCopy();
}
// using a dbCache so only one time database is set with databaseInternalId
if (!isset($dbCache[$databaseInternalId])) {
$databaseDoc = $authorization->skip(fn () => $dbForProject->findOne('databases', [
Query::equal('$sequence', [$databaseInternalId])
]));
$dbCache[$databaseInternalId] = $getDatabasesDB($databaseDoc);
}
$dbForDatabases = $dbCache[$databaseInternalId];
$database = $authorization->skip(fn () => $dbForProject->findOne('databases', [
Query::equal('$sequence', [$databaseInternalId])
]));
@@ -330,7 +382,7 @@ class Update extends Action
$eventAction = 'create';
$docId = $documentId ?? $data['$id'] ?? null;
if ($docId) {
$doc = $dbForProject->getDocument($collectionId, $docId);
$doc = $dbForDatabases->getDocument($collectionId, $docId);
if (!$doc->isEmpty()) {
$documentsToTrigger[] = $doc;
}
@@ -341,7 +393,7 @@ class Update extends Action
case 'decrement':
$eventAction = 'update';
if ($documentId) {
$doc = $dbForProject->getDocument($collectionId, $documentId);
$doc = $dbForDatabases->getDocument($collectionId, $documentId);
if (!$doc->isEmpty()) {
$documentsToTrigger[] = $doc;
}
@@ -357,7 +409,7 @@ class Update extends Action
$eventAction = 'update';
$docId = $documentId ?? $data['$id'] ?? null;
if ($docId) {
$doc = $dbForProject->getDocument($collectionId, $docId);
$doc = $dbForDatabases->getDocument($collectionId, $docId);
if (!$doc->isEmpty()) {
$documentsToTrigger[] = $doc;
}
@@ -26,6 +26,43 @@ class Get extends Action
return 'getDatabaseUsage';
}
protected $databaseType = DATABASE_TYPE_LEGACY;
public function setHttpPath(string $path): Action
{
$this->databaseType = match (true) {
str_contains($path, '/documentsdb') => DATABASE_TYPE_DOCUMENTSDB,
str_contains($path, '/vectorsdb') => DATABASE_TYPE_VECTORSDB,
default => DATABASE_TYPE_LEGACY,
};
return parent::setHttpPath($path);
}
protected function getMetrics(): array
{
$metrics = [
METRIC_DATABASE_ID_COLLECTIONS,
METRIC_DATABASE_ID_DOCUMENTS,
METRIC_DATABASE_ID_STORAGE,
METRIC_DATABASE_ID_OPERATIONS_READS,
METRIC_DATABASE_ID_OPERATIONS_WRITES
];
if ($this->databaseType === DATABASE_TYPE_LEGACY || $this->databaseType === DATABASE_TYPE_TABLESDB) {
return $metrics;
}
return array_map(
fn ($metric) => "{$this->databaseType}.{$metric}",
$metrics
);
}
protected function getResponseModel(): string
{
return UtopiaResponse::MODEL_USAGE_DATABASE;
}
public function __construct()
{
$this
@@ -74,13 +111,10 @@ class Get extends Action
$periods = Config::getParam('usage', []);
$stats = $usage = [];
$days = $periods[$range];
$metrics = [
str_replace('{databaseInternalId}', $database->getSequence(), METRIC_DATABASE_ID_COLLECTIONS),
str_replace('{databaseInternalId}', $database->getSequence(), METRIC_DATABASE_ID_DOCUMENTS),
str_replace('{databaseInternalId}', $database->getSequence(), METRIC_DATABASE_ID_STORAGE),
str_replace('{databaseInternalId}', $database->getSequence(), METRIC_DATABASE_ID_OPERATIONS_READS),
str_replace('{databaseInternalId}', $database->getSequence(), METRIC_DATABASE_ID_OPERATIONS_WRITES)
];
$metrics = array_map(
fn ($metric) => str_replace('{databaseInternalId}', $database->getSequence(), $metric),
$this->getMetrics()
);
$authorization->skip(function () use ($dbForProject, $days, $metrics, &$stats) {
foreach ($metrics as $metric) {
@@ -142,6 +176,6 @@ class Get extends Action
'storage' => $usage[$metrics[2]]['data'],
'databaseReads' => $usage[$metrics[3]]['data'],
'databaseWrites' => $usage[$metrics[4]]['data'],
]), UtopiaResponse::MODEL_USAGE_DATABASE);
]), $this->getResponseModel());
}
}
@@ -24,6 +24,43 @@ class XList extends Action
return 'listDatabaseUsage';
}
protected $databaseType = DATABASE_TYPE_LEGACY;
public function setHttpPath(string $path): Action
{
$this->databaseType = match (true) {
str_contains($path, '/documentsdb') => DATABASE_TYPE_DOCUMENTSDB,
str_contains($path, '/vectorsdb') => DATABASE_TYPE_VECTORSDB,
default => DATABASE_TYPE_LEGACY,
};
return parent::setHttpPath($path);
}
protected function getMetrics(): array
{
$metrics = [
METRIC_DATABASES,
METRIC_COLLECTIONS,
METRIC_DOCUMENTS,
METRIC_DATABASES_STORAGE,
METRIC_DATABASES_OPERATIONS_READS,
METRIC_DATABASES_OPERATIONS_WRITES,
];
if ($this->databaseType === DATABASE_TYPE_LEGACY || $this->databaseType === DATABASE_TYPE_TABLESDB) {
return $metrics;
}
return array_map(
fn ($metric) => "{$this->databaseType}.{$metric}",
$metrics
);
}
protected function getResponseModel(): string
{
return UtopiaResponse::MODEL_USAGE_DATABASES;
}
public function __construct()
{
$this
@@ -66,14 +103,7 @@ class XList extends Action
$periods = Config::getParam('usage', []);
$stats = $usage = [];
$days = $periods[$range];
$metrics = [
METRIC_DATABASES,
METRIC_COLLECTIONS,
METRIC_DOCUMENTS,
METRIC_DATABASES_STORAGE,
METRIC_DATABASES_OPERATIONS_READS,
METRIC_DATABASES_OPERATIONS_WRITES,
];
$metrics = $this->getMetrics();
$authorization->skip(function () use ($dbForProject, $days, $metrics, &$stats) {
foreach ($metrics as $metric) {
@@ -136,6 +166,6 @@ class XList extends Action
'storage' => $usage[$metrics[3]]['data'],
'databasesReads' => $usage[$metrics[4]]['data'],
'databasesWrites' => $usage[$metrics[5]]['data'],
]), UtopiaResponse::MODEL_USAGE_DATABASES);
]), $this->getResponseModel());
}
}
@@ -17,7 +17,6 @@ use Utopia\Database\Exception\Query as QueryException;
use Utopia\Database\Query;
use Utopia\Database\Validator\Query\Cursor;
use Utopia\Http\Adapter\Swoole\Response as SwooleResponse;
use Utopia\Platform\Action;
use Utopia\Validator\Boolean;
use Utopia\Validator\Text;
@@ -92,6 +91,8 @@ class XList extends Action
$cursor->setValue($cursorDocument);
}
$queries[] = Query::equal('type', [$this->getDatabaseType()]);
try {
$databases = $dbForProject->find('databases', $queries);
$total = $includeTotal ? $dbForProject->count('databases', $queries, APP_LIMIT_COUNT) : 0;
@@ -0,0 +1,75 @@
<?php
namespace Appwrite\Platform\Modules\Databases\Http\DocumentsDB\Collections;
use Appwrite\Platform\Modules\Databases\Http\Databases\Collections\Create as CollectionCreate;
use Appwrite\SDK\AuthType;
use Appwrite\SDK\ContentType;
use Appwrite\SDK\Method;
use Appwrite\SDK\Response as SDKResponse;
use Appwrite\Utopia\Database\Validator\CustomId;
use Appwrite\Utopia\Response as UtopiaResponse;
use Utopia\Database\Database;
use Utopia\Database\Validator\Permissions;
use Utopia\Database\Validator\UID;
use Utopia\Http\Adapter\Swoole\Response as SwooleResponse;
use Utopia\Validator\ArrayList;
use Utopia\Validator\Boolean;
use Utopia\Validator\JSON;
use Utopia\Validator\Nullable;
use Utopia\Validator\Text;
class Create extends CollectionCreate
{
public static function getName(): string
{
return 'createDocumentsDBCollection';
}
protected function getResponseModel(): string
{
return UtopiaResponse::MODEL_COLLECTION;
}
public function __construct()
{
$this
->setHttpMethod(self::HTTP_REQUEST_METHOD_POST)
->setHttpPath('/v1/documentsdb/:databaseId/collections')
->desc('Create collection')
->groups(['api', 'database'])
->label('event', 'databases.[databaseId].collections.[collectionId].create')
->label('scope', 'collections.write')
->label('resourceType', RESOURCE_TYPE_DATABASES)
->label('audits.event', 'collections.create')
->label('audits.resource', 'database/{request.databaseId}/collection/{response.$id}')
->label('sdk', new Method(
namespace: 'documentsDB',
group: 'collections',
name: 'createCollection',
description: '/docs/references/documentsdb/create-collection.md',
auth: [AuthType::ADMIN, AuthType::KEY],
responses: [
new SDKResponse(
code: SwooleResponse::STATUS_CODE_CREATED,
model: $this->getResponseModel(),
)
],
contentType: ContentType::JSON
))
->param('databaseId', '', fn (Database $dbForProject) => new UID($dbForProject->getAdapter()->getMaxUIDLength()), 'Database ID.', false, ['dbForProject'])
->param('collectionId', '', fn (Database $dbForProject) => new CustomId(false, $dbForProject->getAdapter()->getMaxUIDLength()), 'Unique Id. Choose a custom ID or generate a random ID with `ID.unique()`. Valid chars are a-z, A-Z, 0-9, period, hyphen, and underscore. Can\'t start with a special char. Max length is 36 chars.', false, ['dbForProject'])
->param('name', '', new Text(128), 'Collection name. Max length: 128 chars.')
->param('permissions', null, new Nullable(new Permissions(APP_LIMIT_ARRAY_PARAMS_SIZE)), 'An array of permissions strings. By default, no user is granted with any permissions. [Learn more about permissions](https://appwrite.io/docs/permissions).', true)
->param('documentSecurity', false, new Boolean(true), 'Enables configuring permissions for individual documents. A user needs one of document or collection level permissions to access a document. [Learn more about permissions](https://appwrite.io/docs/permissions).', true)
->param('enabled', true, new Boolean(), 'Is collection enabled? When set to \'disabled\', users cannot access the collection but Server SDKs with and API key can still read and write to the collection. No data is lost when this is toggled.', true)
->param('attributes', [], new ArrayList(new JSON(), APP_LIMIT_ARRAY_PARAMS_SIZE), 'Array of attribute definitions to create. Each attribute should contain: key (string), type (string: string, integer, float, boolean, datetime, relationship), size (integer, required for string type), required (boolean, optional), default (mixed, optional), array (boolean, optional), and type-specific options.', true)
->param('indexes', [], new ArrayList(new JSON(), APP_LIMIT_ARRAY_PARAMS_SIZE), 'Array of index definitions to create. Each index should contain: key (string), type (string: key, fulltext, unique, spatial), attributes (array of attribute keys), orders (array of ASC/DESC, optional), and lengths (array of integers, optional).', true)
->inject('response')
->inject('dbForProject')
->inject('getDatabasesDB')
->inject('queueForEvents')
->inject('authorization')
->callback($this->action(...));
}
}
@@ -0,0 +1,62 @@
<?php
namespace Appwrite\Platform\Modules\Databases\Http\DocumentsDB\Collections;
use Appwrite\Platform\Modules\Databases\Http\Databases\Collections\Delete as CollectionDelete;
use Appwrite\SDK\AuthType;
use Appwrite\SDK\ContentType;
use Appwrite\SDK\Method;
use Appwrite\SDK\Response as SDKResponse;
use Appwrite\Utopia\Response as UtopiaResponse;
use Utopia\Database\Validator\UID;
use Utopia\Http\Adapter\Swoole\Response as SwooleResponse;
class Delete extends CollectionDelete
{
public static function getName(): string
{
return 'deleteDocumentsDBCollection';
}
protected function getResponseModel(): string
{
return UtopiaResponse::MODEL_COLLECTION;
}
public function __construct()
{
$this
->setHttpMethod(self::HTTP_REQUEST_METHOD_DELETE)
->setHttpPath('/v1/documentsdb/:databaseId/collections/:collectionId')
->desc('Delete collection')
->groups(['api', 'database', 'schema'])
->label('scope', 'collections.write')
->label('resourceType', RESOURCE_TYPE_DATABASES)
->label('event', 'databases.[databaseId].collections.[collectionId].delete')
->label('audits.event', 'collection.delete')
->label('audits.resource', 'database/{request.databaseId}/collection/{request.collectionId}')
->label('sdk', new Method(
namespace: 'documentsDB',
group: 'collections',
name: 'deleteCollection',
description: '/docs/references/documentsdb/delete-collection.md',
auth: [AuthType::ADMIN, AuthType::KEY],
responses: [
new SDKResponse(
code: SwooleResponse::STATUS_CODE_NOCONTENT,
model: UtopiaResponse::MODEL_NONE,
)
],
contentType: ContentType::NONE
))
->param('databaseId', '', new UID(), 'Database ID.')
->param('collectionId', '', new UID(), 'Collection ID.')
->inject('response')
->inject('dbForProject')
->inject('getDatabasesDB')
->inject('queueForDatabase')
->inject('queueForEvents')
->inject('authorization')
->callback($this->action(...));
}
}
@@ -0,0 +1,73 @@
<?php
namespace Appwrite\Platform\Modules\Databases\Http\DocumentsDB\Collections\Documents\Attribute;
use Appwrite\Platform\Modules\Databases\Http\Databases\Collections\Documents\Attribute\Decrement as DecrementDocumentAttribute;
use Appwrite\SDK\AuthType;
use Appwrite\SDK\ContentType;
use Appwrite\SDK\Method;
use Appwrite\SDK\Response as SDKResponse;
use Appwrite\Utopia\Response as UtopiaResponse;
use Utopia\Database\Validator\Key;
use Utopia\Database\Validator\UID;
use Utopia\Http\Adapter\Swoole\Response as SwooleResponse;
use Utopia\Validator\Numeric;
class Decrement extends DecrementDocumentAttribute
{
public static function getName(): string
{
return 'decrementDocumentsDBDocumentAttribute';
}
protected function getResponseModel(): string
{
return UtopiaResponse::MODEL_DOCUMENT;
}
public function __construct()
{
$this
->setHttpMethod(self::HTTP_REQUEST_METHOD_PATCH)
->setHttpPath('/v1/documentsdb/:databaseId/collections/:collectionId/documents/:documentId/:attribute/decrement')
->desc('Decrement document attribute')
->groups(['api', 'database'])
->label('event', 'documentsdb.[databaseId].collections.[collectionId].documents.[documentId].update')
->label('scope', 'documents.write')
->label('resourceType', RESOURCE_TYPE_DATABASES)
->label('audits.event', 'documents.update')
->label('audits.resource', 'database/{request.databaseId}/collection/{request.collectionId}')
->label('abuse-key', 'ip:{ip},method:{method},url:{url},userId:{userId}')
->label('abuse-limit', APP_LIMIT_WRITE_RATE_DEFAULT * 2)
->label('abuse-time', APP_LIMIT_WRITE_RATE_PERIOD_DEFAULT)
->label('sdk', new Method(
namespace: 'documentsDB',
group: $this->getSdkGroup(),
name: 'decrementDocumentAttribute',
description: '/docs/references/documentsdb/decrement-document-attribute.md',
auth: [AuthType::SESSION, AuthType::JWT, AuthType::ADMIN, AuthType::KEY],
responses: [
new SDKResponse(
code: SwooleResponse::STATUS_CODE_OK,
model: $this->getResponseModel(),
)
],
contentType: ContentType::JSON
))
->param('databaseId', '', new UID(), 'Database ID.')
->param('collectionId', '', new UID(), 'Collection ID.')
->param('documentId', '', new UID(), 'Document ID.')
->param('attribute', '', new Key(), 'Attribute key.')
->param('value', 1, new Numeric(), 'Value to decrement the attribute by. The value must be a number.', true)
->param('min', null, new Numeric(), 'Minimum value for the attribute. If the current value is lesser than this value, an exception will be thrown.', true)
->param('transactionId', null, new UID(), 'Transaction ID for staging the operation.', true)
->inject('response')
->inject('dbForProject')
->inject('getDatabasesDB')
->inject('queueForEvents')
->inject('queueForStatsUsage')
->inject('plan')
->inject('authorization')
->callback($this->action(...));
}
}
@@ -0,0 +1,73 @@
<?php
namespace Appwrite\Platform\Modules\Databases\Http\DocumentsDB\Collections\Documents\Attribute;
use Appwrite\Platform\Modules\Databases\Http\Databases\Collections\Documents\Attribute\Increment as IncrementDocumentAttribute;
use Appwrite\SDK\AuthType;
use Appwrite\SDK\ContentType;
use Appwrite\SDK\Method;
use Appwrite\SDK\Response as SDKResponse;
use Appwrite\Utopia\Response as UtopiaResponse;
use Utopia\Database\Validator\Key;
use Utopia\Database\Validator\UID;
use Utopia\Http\Adapter\Swoole\Response as SwooleResponse;
use Utopia\Validator\Numeric;
class Increment extends IncrementDocumentAttribute
{
public static function getName(): string
{
return 'incrementDocumentsDBDocumentAttribute';
}
protected function getResponseModel(): string
{
return UtopiaResponse::MODEL_DOCUMENT;
}
public function __construct()
{
$this
->setHttpMethod(self::HTTP_REQUEST_METHOD_PATCH)
->setHttpPath('/v1/documentsdb/:databaseId/collections/:collectionId/documents/:documentId/:attribute/increment')
->desc('Increment document attribute')
->groups(['api', 'database'])
->label('event', 'documentsdb.[databaseId].collections.[collectionId].documents.[documentId].update')
->label('scope', 'documents.write')
->label('resourceType', RESOURCE_TYPE_DATABASES)
->label('audits.event', 'documents.update')
->label('audits.resource', 'database/{request.databaseId}/collection/{request.collectionId}')
->label('abuse-key', 'ip:{ip},method:{method},url:{url},userId:{userId}')
->label('abuse-limit', APP_LIMIT_WRITE_RATE_DEFAULT * 2)
->label('abuse-time', APP_LIMIT_WRITE_RATE_PERIOD_DEFAULT)
->label('sdk', new Method(
namespace: 'documentsDB',
group: $this->getSdkGroup(),
name: 'incrementDocumentAttribute',
description: '/docs/references/documentsdb/increment-document-attribute.md',
auth: [AuthType::SESSION, AuthType::JWT, AuthType::ADMIN, AuthType::KEY],
responses: [
new SDKResponse(
code: SwooleResponse::STATUS_CODE_OK,
model: $this->getResponseModel(),
)
],
contentType: ContentType::JSON
))
->param('databaseId', '', new UID(), 'Database ID.')
->param('collectionId', '', new UID(), 'Collection ID.')
->param('documentId', '', new UID(), 'Document ID.')
->param('attribute', '', new Key(), 'Attribute key.')
->param('value', 1, new Numeric(), 'Value to increment the attribute by. The value must be a number.', true)
->param('max', null, new Numeric(), 'Maximum value for the attribute. If the current value is greater than this value, an error will be thrown.', true)
->param('transactionId', null, new UID(), 'Transaction ID for staging the operation.', true)
->inject('response')
->inject('dbForProject')
->inject('getDatabasesDB')
->inject('queueForEvents')
->inject('queueForStatsUsage')
->inject('plan')
->inject('authorization')
->callback($this->action(...));
}
}
@@ -0,0 +1,72 @@
<?php
namespace Appwrite\Platform\Modules\Databases\Http\DocumentsDB\Collections\Documents\Bulk;
use Appwrite\Platform\Modules\Databases\Http\Databases\Collections\Documents\Bulk\Delete as DocumentsDelete;
use Appwrite\SDK\AuthType;
use Appwrite\SDK\ContentType;
use Appwrite\SDK\Method;
use Appwrite\SDK\Response as SDKResponse;
use Appwrite\Utopia\Response as UtopiaResponse;
use Utopia\Database\Validator\UID;
use Utopia\Http\Adapter\Swoole\Response as SwooleResponse;
use Utopia\Validator\ArrayList;
use Utopia\Validator\Text;
class Delete extends DocumentsDelete
{
public static function getName(): string
{
return 'deleteDocumentsDBDocuments';
}
protected function getResponseModel(): string
{
return UtopiaResponse::MODEL_DOCUMENT_LIST;
}
public function __construct()
{
$this
->setHttpMethod(self::HTTP_REQUEST_METHOD_DELETE)
->setHttpPath('/v1/documentsdb/:databaseId/collections/:collectionId/documents')
->desc('Delete documents')
->groups(['api', 'database'])
->label('scope', 'documents.write')
->label('resourceType', RESOURCE_TYPE_DATABASES)
->label('audits.event', 'documents.delete')
->label('audits.resource', 'database/{request.databaseId}/collection/{request.collectionId}')
->label('abuse-key', 'ip:{ip},method:{method},url:{url},userId:{userId}')
->label('abuse-limit', APP_LIMIT_WRITE_RATE_DEFAULT)
->label('abuse-time', APP_LIMIT_WRITE_RATE_PERIOD_DEFAULT)
->label('sdk', new Method(
namespace: 'documentsDB',
group: $this->getSdkGroup(),
name: 'deleteDocuments',
description: '/docs/references/documentsdb/delete-documents.md',
auth: [AuthType::ADMIN, AuthType::KEY],
responses: [
new SDKResponse(
code: SwooleResponse::STATUS_CODE_OK,
model: $this->getResponseModel(),
)
],
contentType: ContentType::JSON
))
->param('databaseId', '', new UID(), 'Database ID.')
->param('collectionId', '', new UID(), 'Collection ID. You can create a new collection using the Database service [server integration](https://appwrite.io/docs/server/databases#databasesCreateCollection).')
->param('queries', [], new ArrayList(new Text(APP_LIMIT_ARRAY_ELEMENT_SIZE), APP_LIMIT_ARRAY_PARAMS_SIZE), 'Array of query strings generated using the Query class provided by the SDK. [Learn more about queries](https://appwrite.io/docs/queries). Maximum of ' . APP_LIMIT_ARRAY_PARAMS_SIZE . ' queries are allowed, each ' . APP_LIMIT_ARRAY_ELEMENT_SIZE . ' characters long.', true)
->param('transactionId', null, new UID(), 'Transaction ID for staging the operation.', true)
->inject('response')
->inject('dbForProject')
->inject('getDatabasesDB')
->inject('queueForStatsUsage')
->inject('queueForEvents')
->inject('queueForRealtime')
->inject('queueForFunctions')
->inject('queueForWebhooks')
->inject('plan')
->inject('eventProcessor')
->callback($this->action(...));
}
}
@@ -0,0 +1,74 @@
<?php
namespace Appwrite\Platform\Modules\Databases\Http\DocumentsDB\Collections\Documents\Bulk;
use Appwrite\Platform\Modules\Databases\Http\Databases\Collections\Documents\Bulk\Update as DocumentsUpdate;
use Appwrite\SDK\AuthType;
use Appwrite\SDK\ContentType;
use Appwrite\SDK\Method;
use Appwrite\SDK\Response as SDKResponse;
use Appwrite\Utopia\Response as UtopiaResponse;
use Utopia\Database\Validator\UID;
use Utopia\Http\Adapter\Swoole\Response as SwooleResponse;
use Utopia\Validator\ArrayList;
use Utopia\Validator\JSON;
use Utopia\Validator\Text;
class Update extends DocumentsUpdate
{
public static function getName(): string
{
return 'updateDocumentsDBDocuments';
}
protected function getResponseModel(): string
{
return UtopiaResponse::MODEL_DOCUMENT_LIST;
}
public function __construct()
{
$this
->setHttpMethod(self::HTTP_REQUEST_METHOD_PATCH)
->setHttpPath('/v1/documentsdb/:databaseId/collections/:collectionId/documents')
->desc('Update documents')
->groups(['api', 'database'])
->label('scope', 'documents.write')
->label('resourceType', RESOURCE_TYPE_DATABASES)
->label('audits.event', 'documents.update')
->label('audits.resource', 'database/{request.databaseId}/collection/{request.collectionId}')
->label('abuse-key', 'ip:{ip},method:{method},url:{url},userId:{userId}')
->label('abuse-limit', APP_LIMIT_WRITE_RATE_DEFAULT * 2)
->label('abuse-time', APP_LIMIT_WRITE_RATE_PERIOD_DEFAULT)
->label('sdk', new Method(
namespace: 'documentsDB',
group: $this->getSdkGroup(),
name: 'updateDocuments',
description: '/docs/references/documentsdb/update-documents.md',
auth: [AuthType::ADMIN, AuthType::KEY],
responses: [
new SDKResponse(
code: SwooleResponse::STATUS_CODE_OK,
model: $this->getResponseModel(),
)
],
contentType: ContentType::JSON
))
->param('databaseId', '', new UID(), 'Database ID.')
->param('collectionId', '', new UID(), 'Collection ID.')
->param('data', [], new JSON(), 'Document data as JSON object. Include only attribute and value pairs to be updated.', true)
->param('queries', [], new ArrayList(new Text(APP_LIMIT_ARRAY_ELEMENT_SIZE), APP_LIMIT_ARRAY_PARAMS_SIZE), 'Array of query strings generated using the Query class provided by the SDK. [Learn more about queries](https://appwrite.io/docs/queries). Maximum of ' . APP_LIMIT_ARRAY_PARAMS_SIZE . ' queries are allowed, each ' . APP_LIMIT_ARRAY_ELEMENT_SIZE . ' characters long.', true)
->param('transactionId', null, new UID(), 'Transaction ID for staging the operation.', true)
->inject('response')
->inject('dbForProject')
->inject('getDatabasesDB')
->inject('queueForStatsUsage')
->inject('queueForEvents')
->inject('queueForRealtime')
->inject('queueForFunctions')
->inject('queueForWebhooks')
->inject('plan')
->inject('eventProcessor')
->callback($this->action(...));
}
}
@@ -0,0 +1,74 @@
<?php
namespace Appwrite\Platform\Modules\Databases\Http\DocumentsDB\Collections\Documents\Bulk;
use Appwrite\Platform\Modules\Databases\Http\Databases\Collections\Documents\Bulk\Upsert as DocumentsUpsert;
use Appwrite\SDK\AuthType;
use Appwrite\SDK\ContentType;
use Appwrite\SDK\Method;
use Appwrite\SDK\Response as SDKResponse;
use Appwrite\Utopia\Response as UtopiaResponse;
use Utopia\Database\Validator\UID;
use Utopia\Http\Adapter\Swoole\Response as SwooleResponse;
use Utopia\Validator\ArrayList;
use Utopia\Validator\JSON;
class Upsert extends DocumentsUpsert
{
public static function getName(): string
{
return 'upsertDocumentsDBDocuments';
}
protected function getResponseModel(): string
{
return UtopiaResponse::MODEL_DOCUMENT_LIST;
}
public function __construct()
{
$this
->setHttpMethod(self::HTTP_REQUEST_METHOD_PUT)
->setHttpPath('/v1/documentsdb/:databaseId/collections/:collectionId/documents')
->desc('Upsert documents')
->groups(['api', 'database'])
->label('scope', 'documents.write')
->label('resourceType', RESOURCE_TYPE_DATABASES)
->label('audits.event', 'document.create')
->label('audits.resource', 'database/{request.databaseId}/collection/{request.collectionId}')
->label('abuse-key', 'ip:{ip},method:{method},url:{url},userId:{userId}')
->label('abuse-limit', APP_LIMIT_WRITE_RATE_DEFAULT * 2)
->label('abuse-time', APP_LIMIT_WRITE_RATE_PERIOD_DEFAULT)
->label('sdk', [
new Method(
namespace: 'documentsDB',
group: $this->getSdkGroup(),
name: 'upsertDocuments',
description: '/docs/references/documentsdb/upsert-documents.md',
auth: [AuthType::ADMIN, AuthType::KEY],
responses: [
new SDKResponse(
code: SwooleResponse::STATUS_CODE_CREATED,
model: $this->getResponseModel(),
)
],
contentType: ContentType::JSON,
)
])
->param('databaseId', '', new UID(), 'Database ID.')
->param('collectionId', '', new UID(), 'Collection ID.')
->param('documents', [], fn (array $plan) => new ArrayList(new JSON(), $plan['databasesBatchSize'] ?? APP_LIMIT_DATABASE_BATCH), 'Array of document data as JSON objects. May contain partial documents.', false, ['plan'])
->param('transactionId', null, new UID(), 'Transaction ID for staging the operation.', true)
->inject('response')
->inject('dbForProject')
->inject('getDatabasesDB')
->inject('queueForStatsUsage')
->inject('queueForEvents')
->inject('queueForRealtime')
->inject('queueForFunctions')
->inject('queueForWebhooks')
->inject('plan')
->inject('eventProcessor')
->callback($this->action(...));
}
}
@@ -0,0 +1,116 @@
<?php
namespace Appwrite\Platform\Modules\Databases\Http\DocumentsDB\Collections\Documents;
use Appwrite\Platform\Modules\Databases\Http\Databases\Collections\Documents\Create as DocumentCreate;
use Appwrite\SDK\AuthType;
use Appwrite\SDK\ContentType;
use Appwrite\SDK\Method;
use Appwrite\SDK\Parameter;
use Appwrite\SDK\Response as SDKResponse;
use Appwrite\Utopia\Database\Validator\CustomId;
use Appwrite\Utopia\Response as UtopiaResponse;
use Utopia\Database\Database;
use Utopia\Database\Validator\Permissions;
use Utopia\Database\Validator\UID;
use Utopia\Http\Adapter\Swoole\Response as SwooleResponse;
use Utopia\Validator\ArrayList;
use Utopia\Validator\JSON;
class Create extends DocumentCreate
{
public static function getName(): string
{
return 'createDocumentsDBDocument';
}
protected function getResponseModel(): string
{
return UtopiaResponse::MODEL_DOCUMENT;
}
protected function getBulkResponseModel(): string
{
return UtopiaResponse::MODEL_DOCUMENT_LIST;
}
public function __construct()
{
$this
->setHttpMethod(self::HTTP_REQUEST_METHOD_POST)
->setHttpPath('/v1/documentsdb/:databaseId/collections/:collectionId/documents')
->desc('Create document')
->groups(['api', 'database'])
->label('scope', 'documents.write')
->label('resourceType', RESOURCE_TYPE_DATABASES)
->label('audits.event', 'document.create')
->label('audits.resource', 'database/{request.databaseId}/collection/{request.collectionId}')
->label('abuse-key', 'ip:{ip},method:{method},url:{url},userId:{userId}')
->label('abuse-limit', APP_LIMIT_WRITE_RATE_DEFAULT * 2)
->label('abuse-time', APP_LIMIT_WRITE_RATE_PERIOD_DEFAULT)
->label('sdk', [
new Method(
namespace: 'documentsDB',
group: $this->getSdkGroup(),
name: 'createDocument',
desc: 'Create document',
description: '/docs/references/documentsdb/create-document.md',
auth: [AuthType::ADMIN, AuthType::SESSION, AuthType::KEY, AuthType::JWT],
responses: [
new SDKResponse(
code: SwooleResponse::STATUS_CODE_CREATED,
model: $this->getResponseModel(),
)
],
contentType: ContentType::JSON,
parameters: [
new Parameter('databaseId', optional: false),
new Parameter('collectionId', optional: false),
new Parameter('documentId', optional: false),
new Parameter('data', optional: false),
new Parameter('permissions', optional: true),
]
),
new Method(
namespace: 'documentsDB',
group: $this->getSdkGroup(),
name: 'createDocuments',
desc: 'Create documents',
description: '/docs/references/documentsdb/create-documents.md',
auth: [AuthType::ADMIN, AuthType::SESSION, AuthType::KEY, AuthType::JWT],
responses: [
new SDKResponse(
code: SwooleResponse::STATUS_CODE_CREATED,
model: $this->getBulkResponseModel(),
)
],
contentType: ContentType::JSON,
parameters: [
new Parameter('databaseId', optional: false),
new Parameter('collectionId', optional: false),
new Parameter('documents', optional: false),
]
)
])
->param('databaseId', '', new UID(), 'Database ID.')
->param('documentId', '', new CustomId(), 'Document ID. Choose a custom ID or generate a random ID with `ID.unique()`. Valid chars are a-z, A-Z, 0-9, period, hyphen, and underscore. Can\'t start with a special char. Max length is 36 chars.', true)
->param('collectionId', '', new UID(), 'Collection ID. You can create a new collection using the Database service [server integration](https://appwrite.io/docs/server/databases#databasesCreateCollection). Make sure to define attributes before creating documents.')
->param('data', [], new JSON(), 'Document data as JSON object.', true, example: '{"username":"walter.obrien","email":"walter.obrien@example.com","fullName":"Walter O\'Brien","age":30,"isAdmin":false}')
->param('permissions', null, new Permissions(APP_LIMIT_ARRAY_PARAMS_SIZE, [Database::PERMISSION_READ, Database::PERMISSION_UPDATE, Database::PERMISSION_DELETE, Database::PERMISSION_WRITE]), 'An array of permissions strings. By default, only the current user is granted all permissions. [Learn more about permissions](https://appwrite.io/docs/permissions).', true)
->param('documents', [], fn (array $plan) => new ArrayList(new JSON(), $plan['databasesBatchSize'] ?? APP_LIMIT_DATABASE_BATCH), 'Array of documents data as JSON objects.', true, ['plan'])
->param('transactionId', null, new UID(), 'Transaction ID for staging the operation.', true)
->inject('response')
->inject('dbForProject')
->inject('getDatabasesDB')
->inject('user')
->inject('queueForEvents')
->inject('queueForStatsUsage')
->inject('queueForRealtime')
->inject('queueForFunctions')
->inject('queueForWebhooks')
->inject('plan')
->inject('authorization')
->inject('eventProcessor')
->callback($this->action(...));
}
}
@@ -0,0 +1,76 @@
<?php
namespace Appwrite\Platform\Modules\Databases\Http\DocumentsDB\Collections\Documents;
use Appwrite\Platform\Modules\Databases\Http\Databases\Collections\Documents\Delete as DocumentDelete;
use Appwrite\SDK\AuthType;
use Appwrite\SDK\ContentType;
use Appwrite\SDK\Method;
use Appwrite\SDK\Response as SDKResponse;
use Appwrite\Utopia\Response as UtopiaResponse;
use Utopia\Database\Validator\UID;
use Utopia\Http\Adapter\Swoole\Response as SwooleResponse;
class Delete extends DocumentDelete
{
public static function getName(): string
{
return 'deleteDocumentsDBDocument';
}
/**
* Same explanation as the parent action.
*
* 1. `SDKResponse` uses `UtopiaResponse::MODEL_NONE`.
* 2. But we later need the actual return type for events queue below!
*/
protected function getResponseModel(): string
{
return UtopiaResponse::MODEL_DOCUMENT;
}
public function __construct()
{
$this
->setHttpMethod(self::HTTP_REQUEST_METHOD_DELETE)
->setHttpPath('/v1/documentsdb/:databaseId/collections/:collectionId/documents/:documentId')
->desc('Delete document')
->groups(['api', 'database'])
->label('scope', 'documents.write')
->label('resourceType', RESOURCE_TYPE_DATABASES)
->label('event', 'databases.[databaseId].collections.[collectionId].documents.[documentId].delete')
->label('audits.event', 'document.delete')
->label('audits.resource', 'database/{request.databaseId}/collection/{request.collectionId}/document/{request.documentId}')
->label('abuse-key', 'ip:{ip},method:{method},url:{url},userId:{userId}')
->label('abuse-limit', APP_LIMIT_WRITE_RATE_DEFAULT)
->label('abuse-time', APP_LIMIT_WRITE_RATE_PERIOD_DEFAULT)
->label('sdk', new Method(
namespace: 'documentsDB',
group: $this->getSdkGroup(),
name: 'deleteDocument',
description: '/docs/references/documentsdb/delete-document.md',
auth: [AuthType::ADMIN, AuthType::SESSION, AuthType::KEY, AuthType::JWT],
responses: [
new SDKResponse(
code: SwooleResponse::STATUS_CODE_NOCONTENT,
model: UtopiaResponse::MODEL_NONE,
)
],
contentType: ContentType::NONE
))
->param('databaseId', '', new UID(), 'Database ID.')
->param('collectionId', '', new UID(), 'Collection ID. You can create a new collection using the Database service [server integration](https://appwrite.io/docs/server/databases#databasesCreateCollection).')
->param('documentId', '', new UID(), 'Document ID.')
->param('transactionId', null, new UID(), 'Transaction ID for staging the operation.', true)
->inject('requestTimestamp')
->inject('response')
->inject('dbForProject')
->inject('getDatabasesDB')
->inject('queueForEvents')
->inject('queueForStatsUsage')
->inject('transactionState')
->inject('plan')
->inject('authorization')
->callback($this->action(...));
}
}
@@ -0,0 +1,64 @@
<?php
namespace Appwrite\Platform\Modules\Databases\Http\DocumentsDB\Collections\Documents;
use Appwrite\Platform\Modules\Databases\Http\Databases\Collections\Documents\Get as DocumentGet;
use Appwrite\SDK\AuthType;
use Appwrite\SDK\ContentType;
use Appwrite\SDK\Method;
use Appwrite\SDK\Response as SDKResponse;
use Appwrite\Utopia\Response as UtopiaResponse;
use Utopia\Database\Validator\UID;
use Utopia\Http\Adapter\Swoole\Response as SwooleResponse;
use Utopia\Validator\ArrayList;
use Utopia\Validator\Text;
class Get extends DocumentGet
{
public static function getName(): string
{
return 'getDocumentsDBDocument';
}
protected function getResponseModel(): string
{
return UtopiaResponse::MODEL_DOCUMENT;
}
public function __construct()
{
$this
->setHttpMethod(self::HTTP_REQUEST_METHOD_GET)
->setHttpPath('/v1/documentsdb/:databaseId/collections/:collectionId/documents/:documentId')
->desc('Get document')
->groups(['api', 'database'])
->label('scope', 'documents.read')
->label('resourceType', RESOURCE_TYPE_DATABASES)
->label('sdk', new Method(
namespace: 'documentsDB',
group: $this->getSdkGroup(),
name: 'getDocument',
description: '/docs/references/documentsdb/get-document.md',
auth: [AuthType::ADMIN, AuthType::SESSION, AuthType::KEY, AuthType::JWT],
responses: [
new SDKResponse(
code: SwooleResponse::STATUS_CODE_OK,
model: $this->getResponseModel(),
)
],
contentType: ContentType::JSON
))
->param('databaseId', '', new UID(), 'Database ID.')
->param('collectionId', '', new UID(), 'Collection ID. You can create a new collection using the Database service [server integration](https://appwrite.io/docs/server/databases#databasesCreateCollection).')
->param('documentId', '', new UID(), 'Document ID.')
->param('queries', [], new ArrayList(new Text(APP_LIMIT_ARRAY_ELEMENT_SIZE), APP_LIMIT_ARRAY_PARAMS_SIZE), 'Array of query strings generated using the Query class provided by the SDK. [Learn more about queries](https://appwrite.io/docs/queries). Maximum of ' . APP_LIMIT_ARRAY_PARAMS_SIZE . ' queries are allowed, each ' . APP_LIMIT_ARRAY_ELEMENT_SIZE . ' characters long.', true)
->param('transactionId', null, new UID(), 'Transaction ID to read uncommitted changes within the transaction.', true)
->inject('response')
->inject('dbForProject')
->inject('getDatabasesDB')
->inject('queueForStatsUsage')
->inject('transactionState')
->inject('authorization')
->callback($this->action(...));
}
}

Some files were not shown because too many files have changed in this diff Show More