fix(builds): cap the billed build duration at the build timeout (#13430)

* fix(builds): cap the billed build duration at the build timeout

Jobs::duration() measures wall clock from buildStartedAt, which the first
log callback stamps — so a build that never streamed a line falls back to
the deployment's creation time and measures its whole queue wait. That
value is what Usage\Build::publish() bills, at memory x duration x cpus,
and it is billed for 'failed' exactly as for 'ready'.

The executor backend could not produce this: it measured microtime()
around the build's own execution, so queue wait was structurally
unbillable. Since builds moved to the jobs-service the mean billed
duration of a *failed* build has gone 111s (Jun) -> 157s (Jul) -> 491s
(Aug) while successful builds held at ~52s, and failed builds are now 57%
of all billed build compute fleet-wide.

One production project shows the shape plainly: 2,286 of its 2,311 failed
August builds have buildStartedAt NULL, empty buildLogs, "exit code -1",
and durations at 904-912s. On s-2vcpu-2gb that is ~1.0 GB-hour billed per
build that never ran — 1,869 of its 1,878 billed GB-hours for the month.

Clamp to _APP_COMPUTE_BUILD_TIMEOUT, the same ceiling Deployments hands
the jobs-service as timeoutSeconds: no job outlives it, so nothing past it
can have been build time. Guarded so a 0 or negative value leaves the
measurement alone rather than zeroing every build.

This is a ceiling, not the whole fix — a starved build still bills the
full timeout. Billing 0 when buildStartedAt is NULL, and the build fan-out
that starves them, are follow-ups.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* test(builds): drop the build duration cap test

Removed at request; the cap in Jobs::duration() is unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Luke B. SilverandClaude Opus 5 authored and GitHub committed 2026-08-31 12:58:02 +01:00
1 parent 3fa2e2d420
commit 11e8f837ca
1 file changed
+15 -1
@@ -559,6 +559,13 @@ class Jobs extends Action
* buildStartedAt (stamped by the first log callback) can be missing when a
* terminal callback finalizes first — fall back to the deployment's
* creation time rather than reporting 0.
*
* Clamped to _APP_COMPUTE_BUILD_TIMEOUT, the same ceiling Deployments hands
* the jobs-service as timeoutSeconds. Neither bound above is the
* orchestrator's: a build that waited for a runner without streaming a log
* line never got buildStartedAt, so the fallback measures its whole queue
* wait — and this value is what bills, at memory x duration x cpus. No job
* outlives the timeout, so nothing past it can have been build time.
*/
private function duration(Document $deployment): int
{
@@ -573,7 +580,14 @@ class Jobs extends Action
return 0;
}
return (int) \ceil(\max(0.0, \microtime(true) - $started));
$elapsed = (int) \ceil(\max(0.0, \microtime(true) - $started));
// Set by the operator on every deployed environment; guarded so a 0 or
// negative value leaves the measurement alone rather than zeroing every
// build's duration.
$timeout = (int) System::getEnv('_APP_COMPUTE_BUILD_TIMEOUT', 900);
return $timeout > 0 ? \min($timeout, $elapsed) : $elapsed;
}
/**