Lease one broker connection per consume coroutine instead of sharing one

The Locking decorator serialized every Redis command on the single
consumer connection shared by all consume coroutines. On an idle queue
each waiting coroutine holds the mutex for a full 2s blocking pop, so a
finished job's completion ops (remove job key, increment stats) starve
behind the poppers: a 10ms cache purge took 22-24s to ack, and Storage's
testFilePreviewCache observed stale previews because purges landed tens
of seconds after enqueue.

Pass a Broker\Pool as the adapter consumer instead: each consume
coroutine leases its own connection from the consumer pool for the
lifetime of its loop, eliminating the shared-connection mutex entirely.
The Locking wrapper in the pool factory is reverted, and consumer pools
get +2 headroom above the coroutine count for queue-size telemetry
reads. Purge ack latency drops to 4-58ms.

Known tradeoff: Broker\Pool::close() is a no-op, so SIGTERM no longer
breaks the blocking pop early; workers rely on the process kill that
already follows.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
loks0nandClaude Opus 4.8 committed 2026-06-02 22:37:37 +01:00
1 parent 4baaa7d93f
commit 78769b7f13
2 files changed
+15 -11

No files matched your search

+8 -10
View File
@@ -344,9 +344,13 @@ $register->set('pools', function () {
// PubSub workers hold one long-lived subscribed connection and also need
// spare capacity for publishes from the same process.
$connectionPoolSize = $type === 'pubsub'
? max(2, $poolSize)
: $poolSize;
// Consumer pools lease one connection per consume coroutine for the
// lifetime of the worker, plus headroom for queue-size reads.
$connectionPoolSize = match ($type) {
'pubsub' => max(2, $poolSize),
'consumer' => $poolSize + 2,
default => $poolSize,
};
$pool = new Pool($poolAdapter, $name, $connectionPoolSize, function () use ($type, $resource, $dsn) {
// Get Adapter
@@ -368,17 +372,11 @@ $register->set('pools', function () {
default => null
};
case 'publisher':
case 'consumer':
return match ($dsn->getScheme()) {
'redis' => new Queue\Broker\Redis(new Queue\Connection\Redis($dsn->getHost(), $dsn->getPort())),
default => null
};
case 'consumer':
// Consumers are shared by multiple coroutines within a worker
// process, so serialize commands on the single connection.
return match ($dsn->getScheme()) {
'redis' => new Queue\Broker\Redis(new Queue\Connection\Locking(new Queue\Connection\Redis($dsn->getHost(), $dsn->getPort()))),
default => null
};
case 'cache':
$adapter = match ($dsn->getScheme()) {
'redis' => new RedisCache($resource()),
+7 -1
View File
@@ -83,8 +83,14 @@ if (\str_starts_with($workerName, 'databases')) {
/** @var \Utopia\Pools\Group $pools */
$pools = $container->get('pools');
// Each consume coroutine leases its own broker connection from the pool for
// the lifetime of its consume loop, so concurrent coroutines never share a
// connection. The publisher side backs the queue-depth telemetry gauge.
$adapter = new Swoole(
$pools->get('consumer')->pop()->getResource(),
new BrokerPool(
publisher: $pools->get('consumer'),
consumer: $pools->get('consumer'),
),
System::getEnv('_APP_WORKERS_NUM', 1),
$queueName,
maxCoroutines: (int) System::getEnv('_APP_WORKER_MAX_COROUTINES', 1),