Appwrite/appwrite
1
0
Fork 0
mirror of https://github.com/appwrite/appwrite.git synced 2026-05-26 13:51:13 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #12138 from appwrite/feat-out-of-order-chunk-uploads

Browse Source
This commit is contained in:
Torsten Dittmann
2026-04-29 18:04:57 +04:00
committed by GitHub
parent 1aeee8c407 dfbf45f4cc
commit 6088fd55c8
10 changed files with 623 additions and 82 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.claude/parallel-chunk-upload-storage-plan.md
+174
View File
@@ -0,0 +1,174 @@
# Parallel Chunk Upload Support for utopia-php/storage
## Context
The Appwrite API now supports out-of-order chunked uploads (chunks can arrive in any sequence). The next step is **parallel uploads** — multiple chunks uploaded simultaneously via separate HTTP requests. The SDK guarantees the first chunk is sent before any parallel chunks, so the document creation race is handled at the API layer. However, the storage device layer has a race condition that must be fixed.
## Problem: `Local::joinChunks()` Race
When two requests upload the final missing chunks in parallel, both can observe `countChunks() == $chunks` and call `joinChunks()` simultaneously.
### Current behavior (loser throws)
```php
// Local::joinChunks()
$dest = \fopen($tmpAssemble, 'wb');
// ... stream all parts into $tmpAssemble ...
if (! \rename($tmpAssemble, $path)) {
\unlink($tmpAssemble);
throw new Exception('Failed to finalize assembled file '.$path);
}
```
The winner succeeds with `rename()`. The loser gets `false` from `rename()` (file already exists at `$path`) and throws a 500-error exception. The client that lost the race receives an error even though the file is fully assembled.
### Required behavior
If `$path` already exists, another request already assembled the file. The loser should **silently succeed** — the file is complete, nothing more to do.
## Proposed Changes
### 1. `Local::joinChunks()` — Handle assembly race
Before opening `$tmpAssemble`, check if the final file already exists. If it does, skip assembly entirely.
```php
private function joinChunks(string $path, int $chunks): void
{
// Race winner already assembled the file
if (\file_exists($path)) {
return;
}
$tmp = \dirname($path).DIRECTORY_SEPARATOR.'tmp_'.asename($path);
$tmpAssemble = \dirname($path).DIRECTORY_SEPARATOR.'tmp_assemble_'.asename($path);
// ... rest of assembly logic ...
if (! \rename($tmpAssemble, $path)) {
// Another request may have won the race between fclose and rename
if (\file_exists($path)) {
\unlink($tmpAssemble);
return;
}
\unlink($tmpAssemble);
throw new Exception('Failed to finalize assembled file '.$path);
}
// ... cleanup ...
}
```
### 2. `Local::countChunks()` — Reliability under concurrent writes
`countChunks()` uses `glob()` on the temp directory. Under heavy parallel load, `glob()` might miss files or return inconsistent counts. The current implementation is already fairly robust (it validates `.part.\d+` suffix), but we should document that the return value is a best-effort snapshot.
No code change needed here unless tests reveal issues.
### 3. Tests — Concurrent chunk uploads
Add a test that simulates two parallel requests completing a multi-chunk upload:
```php
public function testParallelChunkUpload(): void
{
$storage = $this->makeJoinTestStorage();
$dest = $storage->getRoot().DIRECTORY_SEPARATOR.'parallel.dat';
// Upload chunk 1 (creates temp directory)
$storage->uploadData('AAAA', $dest, 'application/octet-stream', 1, 2);
// Simulate two parallel requests uploading the last chunk
// In a real test, use pcntl_fork() or pthreads for true concurrency
// For the test suite, sequential calls are sufficient if we verify
// the second call doesn't throw after the first completed assembly
$storage->uploadData('BBBB', $dest, 'application/octet-stream', 2, 2);
// Verify file exists and is correct
$this->assertTrue(\file_exists($dest));
$this->assertSame('AAAABBBB', \file_get_contents($dest));
// Verify second assembly attempt doesn't throw
// (This simulates the race where another request already assembled)
try {
$storage->uploadData('BBBB', $dest, 'application/octet-stream', 2, 2);
} catch (\Exception $e) {
$this->fail('Duplicate assembly should not throw: '.$e->getMessage());
}
$storage->delete($storage->getRoot(), true);
}
```
A more realistic concurrent test using `pcntl_fork()`:
```php
public function testParallelChunkUploadWithFork(): void
{
if (!\function_exists('pcntl_fork')) {
$this->markTestSkipped('pcntl extension required for fork-based concurrency test');
}
$storage = $this->makeJoinTestStorage();
$dest = $storage->getRoot().DIRECTORY_SEPARATOR.'parallel-fork.dat';
// Pre-upload chunk 1
$storage->uploadData('AAAA', $dest, 'application/octet-stream', 1, 2);
$pid = pcntl_fork();
if ($pid === -1) {
$this->fail('Failed to fork');
} elseif ($pid === 0) {
// Child process: upload chunk 2
try {
$storage->uploadData('BBBB', $dest, 'application/octet-stream', 2, 2);
exit(0);
} catch (\Exception $e) {
exit(1);
}
}
// Parent process: also upload chunk 2 (race condition)
$parentSuccess = true;
try {
$storage->uploadData('BBBB', $dest, 'application/octet-stream', 2, 2);
} catch (\Exception $e) {
$parentSuccess = false;
}
pcntl_waitpid($pid, $status);
$childSuccess = pcntl_wexitstatus($status) === 0;
// At least one should succeed
$this->assertTrue($parentSuccess || $childSuccess, 'At least one parallel upload should succeed');
// File should be correctly assembled
$this->assertTrue(\file_exists($dest));
$this->assertSame('AAAABBBB', \file_get_contents($dest));
$storage->delete($storage->getRoot(), true);
}
```
## S3 Device
S3 already handles out-of-order multipart uploads natively. The `completeMultipartUpload` call with `ksort()` sorts parts by number regardless of upload order. However, parallel `completeMultipartUpload` calls for the same `uploadId` would still be problematic.
This is an **API-layer concern** — the Appwrite API should ensure only one request calls `completeMultipartUpload` per upload. The S3 device itself does not need changes.
## Files to Change
| File | Change |
|------|--------|
| `src/Storage/Device/Local.php` | Add `file_exists($path)` guard at start of `joinChunks()` and in `rename()` failure handler |
| `tests/Storage/Device/LocalTest.php` | Add `testParallelChunkUpload` and `testParallelChunkUploadWithFork` |
## Backwards Compatibility
Fully backwards compatible. The change only affects the error path when `rename()` fails due to an existing file. Previously it threw; now it returns silently. No public API signatures change.
## Related PRs
- Appwrite server PR: https://github.com/appwrite/appwrite/pull/12138 (out-of-order upload support)
- This storage PR is a prerequisite for the follow-up Appwrite PR that enables parallel chunk uploads at the API level.
app/init/constants.php
+1
View File
@@ -244,6 +244,7 @@ const APP_AUTH_TYPE_KEY = 'Key';
const APP_AUTH_TYPE_ADMIN = 'Admin';
// Response related
const MAX_OUTPUT_CHUNK_SIZE = 10 * 1024 * 1024; // 10MB
const APP_LIMIT_UPLOAD_CHUNK_SIZE = 5 * 1024 * 1024; // 5MB
const APP_FUNCTION_LOG_LENGTH_LIMIT = 1000000;
const APP_FUNCTION_ERROR_LENGTH_LIMIT = 1000000;
// Function headers
composer.json
+1 -1
View File
@@ -82,7 +82,7 @@
"utopia-php/queue": "0.17.*",
"utopia-php/servers": "0.3.*",
"utopia-php/registry": "0.5.*",
"utopia-php/storage": "1.0.*",
"utopia-php/storage": "2.*",
"utopia-php/system": "0.10.*",
"utopia-php/telemetry": "0.2.*",
"utopia-php/vcs": "3.*",
composer.lock
Generated
+26 -27
View File
@@ -4,7 +4,7 @@
"Read more about it at https://getcomposer.org/doc/01-basic-usage.md#installing-dependencies",
"This file is @generated automatically"
],
"content-hash": "805802552f7482eaeae4bdaa505ae982",
"content-hash": "4bee36b21a57e754d2b3417e72dc9599",
"packages": [
{
"name": "adhocore/jwt",
@@ -4530,16 +4530,16 @@
},
{
"name": "utopia-php/migration",
"version": "1.9.3",
"version": "1.9.4",
"source": {
"type": "git",
"url": "https://github.com/utopia-php/migration.git",
"reference": "111f6221d04578a6f721c23ac872002375f176ae"
"reference": "969dc9477ea962f16da9254facdbd8944cf13477"
},
"dist": {
"type": "zip",
"url": "https://api.github.com/repos/utopia-php/migration/zipball/111f6221d04578a6f721c23ac872002375f176ae",
"reference": "111f6221d04578a6f721c23ac872002375f176ae",
"url": "https://api.github.com/repos/utopia-php/migration/zipball/969dc9477ea962f16da9254facdbd8944cf13477",
"reference": "969dc9477ea962f16da9254facdbd8944cf13477",
"shasum": ""
},
"require": {
@@ -4550,7 +4550,7 @@
"php": ">=8.1",
"utopia-php/database": "5.*",
"utopia-php/dsn": "0.2.*",
"utopia-php/storage": "1.0.*"
"utopia-php/storage": "2.*"
},
"require-dev": {
"ext-pdo": "*",
@@ -4579,9 +4579,9 @@
],
"support": {
"issues": "https://github.com/utopia-php/migration/issues",
"source": "https://github.com/utopia-php/migration/tree/1.9.3"
"source": "https://github.com/utopia-php/migration/tree/1.9.4"
},
"time": "2026-04-22T07:13:26+00:00"
"time": "2026-04-27T12:42:51+00:00"
},
{
"name": "utopia-php/mongo",
@@ -5020,16 +5020,16 @@
},
{
"name": "utopia-php/storage",
"version": "1.0.1",
"version": "2.0.0",
"source": {
"type": "git",
"url": "https://github.com/utopia-php/storage.git",
"reference": "f014be445f0baa635d0764e1673196f412511618"
"reference": "52d1f89a47165ef0d3deff63043cda182175adfb"
},
"dist": {
"type": "zip",
"url": "https://api.github.com/repos/utopia-php/storage/zipball/f014be445f0baa635d0764e1673196f412511618",
"reference": "f014be445f0baa635d0764e1673196f412511618",
"url": "https://api.github.com/repos/utopia-php/storage/zipball/52d1f89a47165ef0d3deff63043cda182175adfb",
"reference": "52d1f89a47165ef0d3deff63043cda182175adfb",
"shasum": ""
},
"require": {
@@ -5043,9 +5043,8 @@
"utopia-php/validators": "0.2.*"
},
"require-dev": {
"laravel/pint": "1.2.*",
"phpunit/phpunit": "^9.3",
"vimeo/psalm": "4.0.1"
"laravel/pint": "^1.21",
"phpunit/phpunit": "^9.3"
},
"type": "library",
"autoload": {
@@ -5067,9 +5066,9 @@
],
"support": {
"issues": "https://github.com/utopia-php/storage/issues",
"source": "https://github.com/utopia-php/storage/tree/1.0.1"
"source": "https://github.com/utopia-php/storage/tree/2.0.0"
},
"time": "2026-02-23T05:59:32+00:00"
"time": "2026-04-27T11:39:32+00:00"
},
{
"name": "utopia-php/system",
@@ -5466,16 +5465,16 @@
"packages-dev": [
{
"name": "appwrite/sdk-generator",
"version": "1.24.0",
"version": "1.25.1",
"source": {
"type": "git",
"url": "https://github.com/appwrite/sdk-generator.git",
"reference": "6d4d26659bc7a1c347c1d4d8dae3b77b5562e0cb"
"reference": "f21a556b9acdbf75bbdcdc90a078af641646eade"
},
"dist": {
"type": "zip",
"url": "https://api.github.com/repos/appwrite/sdk-generator/zipball/6d4d26659bc7a1c347c1d4d8dae3b77b5562e0cb",
"reference": "6d4d26659bc7a1c347c1d4d8dae3b77b5562e0cb",
"url": "https://api.github.com/repos/appwrite/sdk-generator/zipball/f21a556b9acdbf75bbdcdc90a078af641646eade",
"reference": "f21a556b9acdbf75bbdcdc90a078af641646eade",
"shasum": ""
},
"require": {
@@ -5511,9 +5510,9 @@
"description": "Appwrite PHP library for generating API SDKs for multiple programming languages and platforms",
"support": {
"issues": "https://github.com/appwrite/sdk-generator/issues",
"source": "https://github.com/appwrite/sdk-generator/tree/1.24.0"
"source": "https://github.com/appwrite/sdk-generator/tree/1.25.1"
},
"time": "2026-04-24T12:50:05+00:00"
"time": "2026-04-28T11:12:22+00:00"
},
{
"name": "brianium/paratest",
@@ -6222,11 +6221,11 @@
},
{
"name": "phpstan/phpstan",
"version": "2.1.51",
"version": "2.1.52",
"dist": {
"type": "zip",
"url": "https://api.github.com/repos/phpstan/phpstan/zipball/dc3b523c45e714c70de2ac5113b958223b55dc59",
"reference": "dc3b523c45e714c70de2ac5113b958223b55dc59",
"url": "https://api.github.com/repos/phpstan/phpstan/zipball/08a34f8db7ca4daabff74a474fe13c0e56e2b4e5",
"reference": "08a34f8db7ca4daabff74a474fe13c0e56e2b4e5",
"shasum": ""
},
"require": {
@@ -6271,7 +6270,7 @@
"type": "github"
}
],
"time": "2026-04-21T18:22:01+00:00"
"time": "2026-04-28T12:17:53+00:00"
},
{
"name": "phpunit/php-code-coverage",
src/Appwrite/Platform/Modules/Functions/Http/Deployments/Create.php
+12 -17
View File
@@ -175,15 +175,8 @@ class Create extends Action
throw new Exception(Exception::STORAGE_INVALID_CONTENT_RANGE);
}
// TODO remove the condition that checks `$end === $fileSize` in next breaking version
if ($end === $fileSize - 1 || $end === $fileSize) {
//if it's a last chunks the chunk size might differ, so we set the $chunks and $chunk to notify it's last chunk
$chunks = $chunk = -1;
} else {
// Calculate total number of chunks based on the chunk size i.e ($rangeEnd - $rangeStart)
$chunks = (int) ceil($fileSize / ($end + 1 - $start));
$chunk = (int) ($start / ($end + 1 - $start)) + 1;
}
$chunks = (int) ceil($fileSize / APP_LIMIT_UPLOAD_CHUNK_SIZE);
$chunk = (int) ($start / APP_LIMIT_UPLOAD_CHUNK_SIZE) + 1;
}
if (!$fileSizeValidator->isValid($fileSize) && $functionSizeLimit !== 0) { // Check if file size is exceeding allowed limit
@@ -202,15 +195,14 @@ class Create extends Action
$metadata = ['content_type' => $deviceForLocal->getFileMimeType($fileTmpName)];
if (!$deployment->isEmpty()) {
$chunks = $deployment->getAttribute('sourceChunksTotal', 1);
$uploaded = $deployment->getAttribute('sourceChunksUploaded', 0);
$metadata = $deployment->getAttribute('sourceMetadata', []);
if ($chunk === -1) {
$chunk = $chunks;
}
} else {
// Guard against manually setting range header for single chunk upload
if ($chunks === -1) {
$chunks = 1;
$chunk = 1;
if ($uploaded === $chunks) {
$response
->setStatusCode(Response::STATUS_CODE_ACCEPTED)
->dynamic($deployment, Response::MODEL_DEPLOYMENT);
return;
}
}
@@ -258,6 +250,8 @@ class Create extends Action
'sourcePath' => $path,
'sourceSize' => $fileSize,
'totalSize' => $fileSize,
'sourceChunksTotal' => $chunks,
'sourceChunksUploaded' => $chunksUploaded,
'activate' => $activate,
'sourceMetadata' => $metadata,
'type' => $type
@@ -272,6 +266,7 @@ class Create extends Action
} else {
$deployment = $dbForProject->updateDocument('deployments', $deploymentId, new Document([
'sourceSize' => $fileSize,
'sourceChunksUploaded' => $chunksUploaded,
'sourceMetadata' => $metadata,
]));
}
src/Appwrite/Platform/Modules/Sites/Http/Deployments/Create.php
+12 -17
View File
@@ -177,15 +177,8 @@ class Create extends Action
throw new Exception(Exception::STORAGE_INVALID_CONTENT_RANGE);
}
// TODO remove the condition that checks `$end === $fileSize` in next breaking version
if ($end === $fileSize - 1 || $end === $fileSize) {
//if it's a last chunks the chunk size might differ, so we set the $chunks and $chunk to notify it's last chunk
$chunks = $chunk = -1;
} else {
// Calculate total number of chunks based on the chunk size i.e ($rangeEnd - $rangeStart)
$chunks = (int) ceil($fileSize / ($end + 1 - $start));
$chunk = (int) ($start / ($end + 1 - $start)) + 1;
}
$chunks = (int) ceil($fileSize / APP_LIMIT_UPLOAD_CHUNK_SIZE);
$chunk = (int) ($start / APP_LIMIT_UPLOAD_CHUNK_SIZE) + 1;
}
if (!$fileSizeValidator->isValid($fileSize) && $siteSizeLimit !== 0) { // Check if file size is exceeding allowed limit
@@ -204,15 +197,14 @@ class Create extends Action
$metadata = ['content_type' => $deviceForLocal->getFileMimeType($fileTmpName)];
if (!$deployment->isEmpty()) {
$chunks = $deployment->getAttribute('sourceChunksTotal', 1);
$uploaded = $deployment->getAttribute('sourceChunksUploaded', 0);
$metadata = $deployment->getAttribute('sourceMetadata', []);
if ($chunk === -1) {
$chunk = $chunks;
}
} else {
// Guard against manually setting range header for single chunk upload
if ($chunks === -1) {
$chunks = 1;
$chunk = 1;
if ($uploaded === $chunks) {
$response
->setStatusCode(Response::STATUS_CODE_ACCEPTED)
->dynamic($deployment, Response::MODEL_DEPLOYMENT);
return;
}
}
@@ -268,6 +260,8 @@ class Create extends Action
'sourcePath' => $path,
'sourceSize' => $fileSize,
'totalSize' => $fileSize,
'sourceChunksTotal' => $chunks,
'sourceChunksUploaded' => $chunksUploaded,
'activate' => $activate,
'sourceMetadata' => $metadata,
'type' => $type,
@@ -315,6 +309,7 @@ class Create extends Action
} else {
$deployment = $dbForProject->updateDocument('deployments', $deploymentId, new Document([
'sourceSize' => $fileSize,
'sourceChunksUploaded' => $chunksUploaded,
'sourceMetadata' => $metadata,
]));
}
src/Appwrite/Platform/Modules/Storage/Http/Buckets/Files/Create.php
+10 -20
View File
@@ -204,15 +204,8 @@ class Create extends Action
throw new Exception(Exception::STORAGE_INVALID_APPWRITE_ID);
}
// TODO remove the condition that checks `$end === $fileSize` in next breaking version
if ($end === $fileSize - 1 || $end === $fileSize) {
//if it's a last chunks the chunk size might differ, so we set the $chunks and $chunk to -1 notify it's last chunk
$chunks = $chunk = -1;
} else {
// Calculate total number of chunks based on the chunk size i.e ($rangeEnd - $rangeStart)
$chunks = (int) ceil($fileSize / ($end + 1 - $start));
$chunk = (int) ($start / ($end + 1 - $start)) + 1;
}
$chunks = (int) ceil($fileSize / APP_LIMIT_UPLOAD_CHUNK_SIZE);
$chunk = (int) ($start / APP_LIMIT_UPLOAD_CHUNK_SIZE) + 1;
}
/**
@@ -249,18 +242,15 @@ class Create extends Action
$uploaded = $file->getAttribute('chunksUploaded', 0);
$metadata = $file->getAttribute('metadata', []);
if ($chunk === -1) {
$chunk = $chunks;
}
if ($uploaded === $chunks) {
throw new Exception(Exception::STORAGE_FILE_ALREADY_EXISTS);
}
} else {
// Guard against manually setting range header for single chunk upload
if ($chunks === -1) {
$chunks = 1;
$chunk = 1;
if (empty($contentRange)) {
throw new Exception(Exception::STORAGE_FILE_ALREADY_EXISTS);
}
$response
->setStatusCode(Response::STATUS_CODE_OK)
->dynamic($file, Response::MODEL_FILE);
return;
}
}
tests/e2e/Services/Functions/FunctionsCustomServerTest.php
+112
View File
@@ -1079,6 +1079,118 @@ class FunctionsCustomServerTest extends Scope
}, 120000, 500);
}
public function testCreateDeploymentOutOfOrder(): void
{
$data = $this->setupTestFunction();
$functionId = $data['functionId'];
// Prepare a code file that spans at least 3 chunks
$folder = 'large';
$folderPath = realpath(__DIR__ . '/../../../resources/functions') . "/$folder";
$code = "$folderPath/code.tar.gz";
$totalSize = filesize($code);
$chunkSize = 5 * 1024 * 1024; // 5MB chunks
$mimeType = 'application/x-gzip';
$chunksTotal = (int) ceil($totalSize / $chunkSize);
// Read all chunks into memory
$handle = fopen($code, "rb");
$this->assertNotFalse($handle, "Could not open test resource: $code");
$chunks = [];
for ($i = 0; $i < $chunksTotal; $i++) {
$start = $i * $chunkSize;
$end = min($start + $chunkSize, $totalSize);
$length = $end - $start;
$chunkData = fread($handle, $length);
$chunks[] = [
'data' => $chunkData,
'start' => $start,
'end' => $end - 1,
'index' => $i,
];
}
fclose($handle);
// We need at least 2 chunks for a meaningful out-of-order test
$this->assertGreaterThanOrEqual(2, count($chunks), 'Test file must span at least 2 chunks');
// Upload chunks in out-of-order sequence: last chunk first, then first, then second
$uploadOrder = [count($chunks) - 1, 0, 1];
$deploymentId = '';
$deployment = null;
foreach ($uploadOrder as $chunkIndex) {
$chunk = $chunks[$chunkIndex];
$curlFile = new \CURLFile(
'data://' . $mimeType . ';base64,' . base64_encode($chunk['data']),
$mimeType,
'large-fx.tar.gz'
);
$headers = [
'content-type' => 'multipart/form-data',
'x-appwrite-project' => $this->getProject()['$id'],
'content-range' => 'bytes ' . $chunk['start'] . '-' . $chunk['end'] . '/' . $totalSize,
];
if (!empty($deploymentId)) {
$headers['x-appwrite-id'] = $deploymentId;
}
$deployment = $this->client->call(Client::METHOD_POST, '/functions/' . $functionId . '/deployments', array_merge($headers, $this->getHeaders()), [
'entrypoint' => 'index.js',
'code' => $curlFile,
'activate' => true,
]);
$this->assertEquals(202, $deployment['headers']['status-code']);
$deploymentId = $deployment['body']['$id'];
}
// Upload remaining chunks in any order to complete the file
$remainingChunks = [];
for ($i = 2; $i < count($chunks) - 1; $i++) {
$remainingChunks[] = $i;
}
shuffle($remainingChunks);
foreach ($remainingChunks as $chunkIndex) {
$chunk = $chunks[$chunkIndex];
$curlFile = new \CURLFile(
'data://' . $mimeType . ';base64,' . base64_encode($chunk['data']),
$mimeType,
'large-fx.tar.gz'
);
$headers = [
'content-type' => 'multipart/form-data',
'x-appwrite-project' => $this->getProject()['$id'],
'content-range' => 'bytes ' . $chunk['start'] . '-' . $chunk['end'] . '/' . $totalSize,
'x-appwrite-id' => $deploymentId,
];
$deployment = $this->client->call(Client::METHOD_POST, '/functions/' . $functionId . '/deployments', array_merge($headers, $this->getHeaders()), [
'entrypoint' => 'index.js',
'code' => $curlFile,
'activate' => true,
]);
$this->assertEquals(202, $deployment['headers']['status-code']);
}
// Wait for build to complete
$this->assertEventually(function () use ($functionId, $deploymentId) {
$deployment = $this->getDeployment($functionId, $deploymentId);
$this->assertEquals(200, $deployment['headers']['status-code']);
$this->assertEquals('ready', $deployment['body']['status']);
}, 120000, 500);
}
public function testUpdateDeployment(): void
{
$data = $this->setupTestDeployment();
tests/e2e/Services/Sites/SitesCustomServerTest.php
+128
View File
@@ -906,6 +906,134 @@ class SitesCustomServerTest extends Scope
$this->cleanupSite($siteId);
}
public function testCreateDeploymentOutOfOrder(): void
{
$siteId = $this->setupSite([
'buildRuntime' => 'node-22',
'fallbackFile' => '',
'framework' => 'other',
'name' => 'Test Site Out of Order Upload',
'outputDirectory' => './',
'providerBranch' => 'main',
'providerRootDirectory' => './',
'siteId' => ID::unique()
]);
// Create a temporary large site package for chunked upload
$tempDir = sys_get_temp_dir() . '/appwrite-test-site-' . uniqid();
mkdir($tempDir, 0777, true);
file_put_contents($tempDir . '/index.html', '<html><body>Hello World</body></html>');
// Add a large dummy file to make the package span multiple chunks
file_put_contents($tempDir . '/large.bin', random_bytes(12 * 1024 * 1024)); // 12MB non-compressible
$codePath = $tempDir . '/code.tar.gz';
Console::execute("cd $tempDir && tar --exclude code.tar.gz -czf code.tar.gz .", '', $this->stdout, $this->stderr);
$totalSize = filesize($codePath);
$chunkSize = 5 * 1024 * 1024; // 5MB chunks
$mimeType = 'application/x-gzip';
$chunksTotal = (int) ceil($totalSize / $chunkSize);
$this->assertGreaterThanOrEqual(2, $chunksTotal, 'Test file must span at least 2 chunks');
// Read all chunks into memory
$handle = fopen($codePath, "rb");
$this->assertNotFalse($handle, "Could not open test resource: $codePath");
$chunks = [];
for ($i = 0; $i < $chunksTotal; $i++) {
$start = $i * $chunkSize;
$end = min($start + $chunkSize, $totalSize);
$length = $end - $start;
$data = fread($handle, $length);
$chunks[] = [
'data' => $data,
'start' => $start,
'end' => $end - 1,
'index' => $i,
];
}
fclose($handle);
// Upload chunks in out-of-order sequence: last chunk first, then first, then second
$uploadOrder = [count($chunks) - 1, 0, 1];
$deploymentId = '';
$deployment = null;
foreach ($uploadOrder as $chunkIndex) {
$chunk = $chunks[$chunkIndex];
$curlFile = new \CURLFile(
'data://' . $mimeType . ';base64,' . base64_encode($chunk['data']),
$mimeType,
'code.tar.gz'
);
$headers = [
'content-type' => 'multipart/form-data',
'x-appwrite-project' => $this->getProject()['$id'],
'content-range' => 'bytes ' . $chunk['start'] . '-' . $chunk['end'] . '/' . $totalSize,
];
if (!empty($deploymentId)) {
$headers['x-appwrite-id'] = $deploymentId;
}
$deployment = $this->client->call(Client::METHOD_POST, '/sites/' . $siteId . '/deployments', array_merge($headers, $this->getHeaders()), [
'code' => $curlFile,
'activate' => true,
]);
$this->assertEquals(202, $deployment['headers']['status-code']);
$deploymentId = $deployment['body']['$id'];
}
// Upload remaining chunks in any order to complete the file
$remainingChunks = [];
for ($i = 2; $i < count($chunks) - 1; $i++) {
$remainingChunks[] = $i;
}
shuffle($remainingChunks);
foreach ($remainingChunks as $chunkIndex) {
$chunk = $chunks[$chunkIndex];
$curlFile = new \CURLFile(
'data://' . $mimeType . ';base64,' . base64_encode($chunk['data']),
$mimeType,
'code.tar.gz'
);
$headers = [
'content-type' => 'multipart/form-data',
'x-appwrite-project' => $this->getProject()['$id'],
'content-range' => 'bytes ' . $chunk['start'] . '-' . $chunk['end'] . '/' . $totalSize,
'x-appwrite-id' => $deploymentId,
];
$deployment = $this->client->call(Client::METHOD_POST, '/sites/' . $siteId . '/deployments', array_merge($headers, $this->getHeaders()), [
'code' => $curlFile,
'activate' => true,
]);
$this->assertEquals(202, $deployment['headers']['status-code']);
}
// Wait for build to complete
$this->assertEventually(function () use ($siteId, $deploymentId) {
$deployment = $this->getDeployment($siteId, $deploymentId);
$this->assertEquals(200, $deployment['headers']['status-code']);
$this->assertEquals('ready', $deployment['body']['status']);
}, 120000, 500);
// Clean up temp files
unlink($codePath);
unlink($tempDir . '/index.html');
unlink($tempDir . '/large.bin');
rmdir($tempDir);
$this->cleanupSite($siteId);
}
public function testCreateDeployment()
{
$siteId = $this->setupSite([
tests/e2e/Services/Storage/StorageBase.php
+147
View File
@@ -1227,6 +1227,153 @@ trait StorageBase
$this->assertEquals(204, $deleteBucketResponse['headers']['status-code']);
}
public function testCreateBucketFileOutOfOrder(): void
{
// Create a bucket for this test
$bucket = $this->client->call(Client::METHOD_POST, '/storage/buckets', [
'content-type' => 'application/json',
'x-appwrite-project' => $this->getProject()['$id'],
'x-appwrite-key' => $this->getProject()['apiKey'],
], [
'bucketId' => ID::unique(),
'name' => 'Test Bucket Out of Order Upload',
'fileSecurity' => true,
'permissions' => [
Permission::read(Role::any()),
Permission::create(Role::any()),
Permission::update(Role::any()),
Permission::delete(Role::any()),
],
]);
$this->assertEquals(201, $bucket['headers']['status-code']);
$bucketId = $bucket['body']['$id'];
// Prepare a file that spans at least 3 chunks
$source = __DIR__ . "/../../../resources/disk-a/large-file.mp4";
$totalSize = \filesize($source);
$chunkSize = 5 * 1024 * 1024; // 5MB chunks
$mimeType = mime_content_type($source);
$chunksTotal = (int) ceil($totalSize / $chunkSize);
// Read all chunks into memory
$handle = fopen($source, "rb");
$this->assertNotFalse($handle, "Could not open test resource: $source");
$chunks = [];
for ($i = 0; $i < $chunksTotal; $i++) {
$start = $i * $chunkSize;
$end = min($start + $chunkSize, $totalSize);
$length = $end - $start;
$data = fread($handle, $length);
$chunks[] = [
'data' => $data,
'start' => $start,
'end' => $end - 1,
'index' => $i,
];
}
fclose($handle);
// We need at least 3 chunks for a meaningful out-of-order test
$this->assertGreaterThanOrEqual(3, count($chunks), 'Test file must span at least 3 chunks');
// Upload chunks in out-of-order sequence: last chunk first, then first, then middle
$uploadOrder = [count($chunks) - 1, 0, 1]; // last, first, second (for 3+ chunks)
$fileId = ID::unique();
$id = '';
$uploadedFile = null;
foreach ($uploadOrder as $chunkIndex) {
$chunk = $chunks[$chunkIndex];
$curlFile = new \CURLFile(
'data://' . $mimeType . ';base64,' . base64_encode($chunk['data']),
$mimeType,
'large-file.mp4'
);
$headers = [
'content-type' => 'multipart/form-data',
'x-appwrite-project' => $this->getProject()['$id'],
'content-range' => 'bytes ' . $chunk['start'] . '-' . $chunk['end'] . '/' . $totalSize,
];
if (!empty($id)) {
$headers['x-appwrite-id'] = $id;
}
$uploadedFile = $this->client->call(Client::METHOD_POST, '/storage/buckets/' . $bucketId . '/files', array_merge($headers, $this->getHeaders()), [
'fileId' => $fileId,
'file' => $curlFile,
'permissions' => [
Permission::read(Role::any()),
],
]);
$this->assertEquals(201, $uploadedFile['headers']['status-code']);
$id = $uploadedFile['body']['$id'];
}
// Upload remaining chunks in any order to complete the file
$remainingChunks = [];
for ($i = 2; $i < count($chunks) - 1; $i++) {
$remainingChunks[] = $i;
}
// Shuffle remaining chunks for extra randomness
shuffle($remainingChunks);
foreach ($remainingChunks as $chunkIndex) {
$chunk = $chunks[$chunkIndex];
$curlFile = new \CURLFile(
'data://' . $mimeType . ';base64,' . base64_encode($chunk['data']),
$mimeType,
'large-file.mp4'
);
$headers = [
'content-type' => 'multipart/form-data',
'x-appwrite-project' => $this->getProject()['$id'],
'content-range' => 'bytes ' . $chunk['start'] . '-' . $chunk['end'] . '/' . $totalSize,
'x-appwrite-id' => $id,
];
$uploadedFile = $this->client->call(Client::METHOD_POST, '/storage/buckets/' . $bucketId . '/files', array_merge($headers, $this->getHeaders()), [
'fileId' => $fileId,
'file' => $curlFile,
'permissions' => [
Permission::read(Role::any()),
],
]);
$this->assertEquals(201, $uploadedFile['headers']['status-code']);
}
// Verify the final upload response indicates completion
$this->assertEquals($chunksTotal, $uploadedFile['body']['chunksTotal']);
$this->assertEquals($chunksTotal, $uploadedFile['body']['chunksUploaded']);
// Verify the file can be downloaded and matches the original
$download = $this->client->call(Client::METHOD_GET, '/storage/buckets/' . $bucketId . '/files/' . $id . '/download', array_merge([
'content-type' => 'application/json',
'x-appwrite-project' => $this->getProject()['$id'],
], $this->getHeaders()));
$this->assertEquals(200, $download['headers']['status-code']);
$this->assertEquals($totalSize, strlen($download['body']));
$this->assertEquals(md5_file($source), md5($download['body']));
// Clean up
$this->client->call(Client::METHOD_DELETE, '/storage/buckets/' . $bucketId . '/files/' . $id, array_merge([
'content-type' => 'application/json',
'x-appwrite-project' => $this->getProject()['$id'],
], $this->getHeaders()));
$this->client->call(Client::METHOD_DELETE, '/storage/buckets/' . $bucketId, [
'content-type' => 'application/json',
'x-appwrite-project' => $this->getProject()['$id'],
'x-appwrite-key' => $this->getProject()['apiKey'],
]);
}
public function testDeleteBucketFile(): void
{
// Create a fresh file just for deletion testing (not using cache since we delete it)