Files
Ben Irvin bac520e9f8 fix(data-transfer): fix large transfer crashes; show transfer progress (#23479)
* enhancement: update progress per chunk

* fix: check stageprogress exists to make ts happy

* chore: split progress tracker into two methods

* test: fix lint

* enhancement: display readable times

* fix: speed indication for assets

* fix: restore speed indicator and fix double counting

* chore: clean up code

* fix: data transfer memory leak

* fix: yarn.lock

* chore: fix memory logging

* ci: complex project remote transfer

* enhancement: optimized transfer mode

* test(cli): data transfer and env passthrough

* chore: only send one message warning of legacy mode

* enhancement: show transfer totals and estimated time

* test(cli): fix deterministic transfer test files

* fix: push and pull shared utils backwards compatibility

* fix(data-transfer): extract legacy asset chunk parsing and tighten transfer logging/test coverage

* enhancement: checksum negotiation

* enhancement: show skipped file warnings on client

* fix: transfer diagnostics

* test: fix open handle

* fix: clear stall timeout for assets

* chore: fix misleading comments and variables

* test: fix misleading test

* test: fix typo

* test: make checks deterministic, less flaky

* enhancement(data-transfer): speed up asset totals; widen assets start reply window on remote pull

* fix(data-transfer): harden WebSocket JSON serialization for transfer frames

* fix(data-transfer): more transfer hardening

* test: fix test imports

* fix: await async write

* fix(data-transfer): resolve push transfer deadlock and harden async writes

- Extract createAssetsDestinationWritable so Writable callbacks run before
  uploadStream completes (same WS batch as PassThrough chunks).
- Add writable-async-write (write callback + drain/finished race; avoid hang
  on destroy).
- Wire push/pull, remote-source, file & directory sources to shared write().
- Fire-and-forget pull flush: Promise.resolve(flush).catch(onError); guard
  missing stream inside try.
- Add regression tests (assets writable, writable-async-write, handler checks).

* fix(data-transfer): write push stream batches sequentially

Use a for-loop with await write() instead of Promise.all over msg.data
so non-asset stages respect one in-flight write per objectMode Writable
and backpressure from writable-async-write.

- Validate minChunksForBackpressure in assertReadStreamBackpressure
- Add engine test for non-Buffer asset chunk byte progress (counts as 1)
- Assert push.ts keeps sequential msg.data handling in static handler test

* fix(data-transfer): align push streamAsset with remote-source and harden tests

- Push handler: combine stream/end under one branch, error when start is missing
  or action is invalid; shorten stage write comments.
- Engine version-matching tests: use a fresh createDestination() per engine so
  parallel transfers do not share destination writables (MaxListeners warnings).
- File destination tests: mock createWriteStream with a new Writable per call.
- CLI transfer tests: mock progress.stream so transfer::finish runs after transfer
  and clears the progress setInterval (fixes Jest worker hang).
- Misc test cleanup: assets-destination timeout clearTimeout, collect listeners,
  writable-async-write teardown; tighten push/static test descriptions.

* test: remove parity test

* fix(data-transfer): harden collect() and stabilize transfer tests

- collect(): settle once, remove listeners on resolve/reject, avoid double completion
- engine tests: add expectHeapGrowthWithinNoise for heap smoke checks
- CLI transfer tests: console spies in beforeAll; jest.restoreAllMocks in afterAll
- stream test: remove removeAllListeners workaround
2026-04-20 16:08:05 +02:00

69 lines
1.8 KiB
JavaScript

'use strict';
/**
* Jest / execa timeouts for remote transfer CLI e2e (scale with TRANSFER_CLI_MEDIA_*).
* Override: CLI_TRANSFER_REMOTE_*_TIMEOUT_MS (legacy CLI_TRANSFER_PULL_* for runner/jest).
*/
const parseNonNegativeInt = (value, fallback) => {
const n = parseInt(value ?? '', 10);
return Number.isFinite(n) && n >= 0 ? n : fallback;
};
const totalSeededMediaBytes = () => {
const bytes = parseNonNegativeInt(process.env.TRANSFER_CLI_MEDIA_BYTES, 2048);
const count = parseNonNegativeInt(process.env.TRANSFER_CLI_MEDIA_COUNT, 2);
return bytes * count;
};
const STRAPI_DOMAIN_DEFAULT_RUNNER_MS = 30 * 60 * 1000;
/**
* Outer execa(Jest) budget. Non-strapi CLI domains stay at 2 minutes.
* @param {string} [domainName]
*/
function runnerTimeoutMs(domainName) {
const explicit =
process.env.CLI_TRANSFER_REMOTE_RUNNER_TIMEOUT_MS ||
process.env.CLI_TRANSFER_PULL_RUNNER_TIMEOUT_MS;
if (explicit) {
return parseNonNegativeInt(explicit, STRAPI_DOMAIN_DEFAULT_RUNNER_MS);
}
if (domainName !== 'strapi') {
return 2 * 60 * 1000;
}
const total = totalSeededMediaBytes();
if (total > 100 * 1024 * 1024) {
return 4 * 60 * 60 * 1000;
}
if (total > 10 * 1024 * 1024) {
return 90 * 60 * 1000;
}
return STRAPI_DOMAIN_DEFAULT_RUNNER_MS;
}
function jestSuiteTimeoutMs() {
const explicit =
process.env.CLI_TRANSFER_REMOTE_JEST_TIMEOUT_MS ||
process.env.CLI_TRANSFER_PULL_JEST_TIMEOUT_MS;
if (explicit) {
return parseNonNegativeInt(explicit, 10 * 60 * 1000);
}
const total = totalSeededMediaBytes();
if (total > 100 * 1024 * 1024) {
return 4 * 60 * 60 * 1000;
}
if (total > 10 * 1024 * 1024) {
return 90 * 60 * 1000;
}
return 10 * 60 * 1000;
}
module.exports = {
totalSeededMediaBytes,
runnerTimeoutMs,
jestSuiteTimeoutMs,
};