MJ Convento

/

Cache stampedes, and the Lock component doing the boring part

1,762 words, about 8 min read

At 06:14 on a Tuesday our Redis primary failed over. The failover took nine seconds. The four minutes after it were the expensive part: p99 on the operations dashboard went from 180 ms to 9.2 seconds, 41 requests hit the 30 second FPM timeout, and MySQL ran the same shipment aggregate 200-odd times in parallel because every single one of those requests found a cold key and decided it was the one who should warm it.

I wrote the cache in front of that query in 2019. It had a lock in it. The lock was the problem.

The 23 lines I deleted

Here is roughly what the 2019 version did, reconstructed from the diff. Symfony 4, phpredis, no components involved:

// src/Cache/DashboardCache.php, the 2019 shape
$value = $this->redis->get($key);
if (false !== $value) {
    return unserialize($value);
}

// only one worker should compute this
if (!$this->redis->set($key.':lock', 1, ['nx', 'ex' => 30])) {
    // someone else is computing, give up and return nothing useful
    return null;
}

$value = $this->computeAggregate();
$this->redis->setex($key, 900, serialize($value));
$this->redis->del($key.':lock');

return $value;

Two bugs, and I only knew about one of them. The one I knew about: if a worker dies between set and del, the key stays locked for the full 30 seconds and every request in that window gets a null and renders an empty panel. We shipped during a deploy once, SIGKILL landed on a worker mid-compute, and support got seven tickets in a minute about the dashboard being blank.

The one I did not know about is worse. The aggregate takes 3.4 seconds when the database is idle. Under load it has gone past 30 seconds. When that happens the lock expires while the first worker is still computing, a second worker acquires it, and then the first worker finishes and calls del on a lock it no longer owns. Now there are two writers and no lock at all. I never saw that in a log because nothing in that code knew who owned the key.

That is the thing the Lock component gets right and my version did not. A lock is owned by a Key, not by a name, and the docs are explicit that the true owner is whoever shares the same Key instance. So release() releases your lock, and isAcquired() tells you whether you still hold it. Internally the component signals a lost race by throwing SymfonyComponentLockExceptionLockConflictedException, which Lock::acquire() and Lock::acquireRead() catch in non blocking mode and turn into a false return. You only see the exception yourself when you go looking for it.

The get() callback already does most of this

We are on Symfony 7.1 now, and the first thing that went was the read path. The Cache Contracts have exactly two methods, get() and delete(), and there is no set() because get() both gets and sets. The signature, from the contracts trait:

public function get(
    string $key,
    callable $callback,
    ?float $beta = null,
    ?array &$metadata = null,
): mixed

Two kinds of stampede protection come with that, and neither is opt in. The first is locking: only one process per host computes a given key at a time. The second is early expiration, which is the parameter most people scroll past.

The locking is worth understanding before you rely on it, because it is not a distributed lock and it is quietly absent in places. LockRegistry takes crc32 of the cache key, modulo the number of files in a hard coded list, and flocks that file. The list is 25 source files inside the Cache component’s own Adapter directory, which means the concurrency cap is 25 keys in flight per host, and two unrelated keys can collide on the same file. The class comment says 20. The array has 25 entries. Somebody renamed adapters and did not update the docblock.

More importantly: it is disabled on Windows, and it is disabled when PHP_SAPI is cli, phpdbg or embed. Every warmup command, every worker, every cron you have gets no cache level locking at all. I found this out the way you would expect, which is by watching two overlapping cron runs both compute the same key.

Beta, and why the expensive keys move first

Early expiration is the part that actually fixed our Tuesday. Instead of every request waiting for the expiry cliff, the pool rolls a die on each hit and occasionally serves one unlucky caller a fake miss while everyone else keeps getting the cached value. The rule lives in the same contracts trait and is short enough to read:

$expiry <= $now - $ctime / 1000 * $beta * log(random_int(1, PHP_INT_MAX) / PHP_INT_MAX)

$ctime is how many milliseconds the callback took last time, stored on the item as metadata alongside the expiry. The logarithm of a uniform number in (0, 1] is negative, so the whole right hand term pushes the deadline earlier, scaled by how expensive the item was to build. Cheap items drift a little. A 3.4 second aggregate drifts a lot. The default beta is 1.0, 0 disables it, INF forces a recompute, and a negative value throws. When an item is picked you get an info level log line: Item "{key}" elected for early recomputation {delta}s before its expiration. Grepping for that string is how I convinced myself it was working.

Our version of the read path, with tags, which is the whole of it now:

use SymfonyContractsCacheItemInterface;
use SymfonyContractsCacheTagAwareCacheInterface;

public function __construct(
    private TagAwareCacheInterface $dashboardCache,
) {
}

public function aggregate(int $hubId): DashboardAggregate
{
    return $this->dashboardCache->get(
        'dashboard.aggregate.'.$hubId,
        function (ItemInterface $item) use ($hubId): DashboardAggregate {
            $item->expiresAfter(900);
            $item->tag(['dashboard', 'hub-'.$hubId]);

            return $this->computeAggregate($hubId);
        },
        beta: 1.0,
    );
}

Tags replaced a key registry I had also hand rolled: a Redis set of every cache key touching a hub, read and deleted on write. Tagging an item and calling invalidateTags() does the same job, and on Redis you want RedisTagAwareAdapter rather than wrapping the plain adapter, because 6.1 made tag versions an integral part of the item value and the invalidation check stopped being a second round trip per item. The generic TagAwareAdapter still exists and still works, and it can keep items in one pool and tags in another, which is useful if your values are large and your tags need to be fast.

One correction to something I said in a review in July. I described the get() callback as new. It is not new at all: the Cache CHANGELOG puts CacheInterface and probabilistic early expiration in 4.2, and says then that it “should become the preferred way to use a cache”. I had been ignoring it for six years.

What the Lock component is still for

Because cache locking is per host and off under CLI, the jobs still need real locks. The nightly recompute is the case I care about: it holds a lock for minutes, across three app servers, and must not run twice.

use SymfonyComponentLockLockFactory;

$lock = $this->lockFactory->createLock('dashboard-recompute', ttl: 60);

if (!$lock->acquire()) {
    $this->logger->info('recompute already running elsewhere, exiting');

    return Command::SUCCESS;
}

try {
    foreach ($this->hubs() as $hub) {
        $this->recompute($hub);
        $lock->refresh();
    }

    if (!$lock->isAcquired()) {
        throw new RuntimeException('lost the recompute lock mid run');
    }

    $this->connection->commit();
} finally {
    $lock->release();
}

A short TTL plus refresh() inside the loop is the pattern the docs recommend, and it is the direct answer to my 2019 bug: a 60 second TTL means a dead worker frees the resource in a minute, and refreshing every iteration means a slow run never loses it. refresh() also takes a one off TTL argument if a single step is unusually long. The store matters here too, and the docs carry a capability table: Redis is remote, expiring, shareable and serializable, while FlockStore is local and does not expire at all. We use Redis for this and semaphores for nothing, which is the opposite of the framework default.

The other thing I took from the component is shared locks. The invoice export reads a snapshot that the recompute rewrites, and it does not need exclusivity with other readers. acquireRead() from SharedLockInterface gives a read lock that can later be promoted with acquire(), and shared locks and createLockFromKey() both arrived in 5.2, along with the removal of the old RetryTillSaveStore, whose retry logic moved into Lock itself. If the store does not implement the shared interface, the component silently gives you a write lock instead, which is safe and slow and worth knowing before you benchmark.

The numbers, and what is still open

We forced a cache flush in production on a Thursday afternoon at 54 rps on that endpoint to see what would happen. p99 went from 190 ms to 260 ms for about 20 seconds, no timeouts, and MySQL logged the aggregate query 3 times rather than 200. The recompute command has not double run since June. Net deletion across the two PRs was 210 lines, most of it the key registry.

Still open: we compute early expirations synchronously, so the unlucky caller still pays the 3.4 seconds. The Cache component has had a Messenger integration for computing values in a worker since 5.2, with an EarlyExpirationHandler that 6.4 moved onto AsMessageHandler, and on paper that means the elected request returns the stale value immediately and a consumer refreshes it. I have it running in staging. I do not yet trust it with a queue that can back up, because a backed up queue turns early expiration into no expiration until the value actually dies, and then we are back to Tuesday.

Sources

  • The Cache Component, Symfony 7.1: the Cache Contracts versus PSR-6, and the two built in stampede defences (locking and probabilistic early expiration) with the beta parameter.
  • SymfonyContractsCacheCacheTrait: the exact get() signature, the early expiration formula using expiry and ctime metadata, the beta validation, and the recomputation log message.
  • SymfonyComponentCacheTraitsContractsTrait: where the callback wrapper is installed, and where it is switched off for the cli, phpdbg and embed SAPIs.
  • SymfonyComponentCacheLockRegistry: the flock based per host lock, the hard coded file list that sets the concurrency cap, and the Windows exclusion.
  • Cache Invalidation, Symfony 7.1: ItemInterface::tag(), TagAwareCacheInterface::invalidateTags(), and the tag aware adapters including the split items and tags setup.
  • Cache component CHANGELOG: CacheInterface and early expiration in 4.2, the Messenger integration in 5.2, tag versions folded into item values in 6.1, and the 6.4 EarlyExpirationHandler change.
  • The Lock Component, Symfony 7.1: expiring locks and TTL choice, refresh(), isAcquired() and lock ownership, shared locks via acquireRead(), and the store capability table.
  • Lock component CHANGELOG: shared lock support and LockFactory::createLockFromKey() in 5.2, and the removal of RetryTillSaveStore once its logic moved into Lock.