Statamic background recache smooths out bulk cache invalidations

Statamic has a page caching feature built in called half measure. Without it, each page is built on every visit. Even in perfect conditions that is slow. Under load, with a complex page, it can tip into timeouts.

Half measure solves this by saving a pre-built version of every page in a cache, so the page doesn't need to be built from scratch for every visitor.

This is great for keeping Statamic websites running fast, but presents a problem when updates are made on a production website.

The problem

Let's say the site owner updates the title of the latest blog post. When half measure is used, only the blog entry page is flushed from the static cache.

By default, it's only cached again when the page is visited. In isolation this isn't a big deal, especially if the page is light and doesn't require much server effort to rebuild.

But what happens if a website shows the latest three entries from the blog collection on the home page (very common). Statamic doesn't know about these, so we have to tell Statamic to flush the cache on other pages when a blog entry is updated.

We can do this by using the cache invalidation config, this is what it looks like:

'invalidation' => [
    'rules' => [
        'blog' => [
                'urls' => [
                    '/',                                   
                ]
            ],    
    ],
],

This tells Statamic to flush the static cache of the home page whenever a blog entry is edited. Good, but now we have two pages that aren't cached. Statamic will only recache these pages when someone visits them.

This is a very simple example, it's not uncommon to show content from multiple collections in many locations throughout a website.

In addition to the home page, we'd almost certainly want to invalidate the /blog home page and we'd want to update the list of latest entries in the sidebar of the /services and /products collections. That means we'd need to add some extra invalidation rules

'invalidation' => [

    'rules' => [
        'blog' => [
                'urls' => [
                    '/', 
                    '/blog',
                    '/services',                                    
                    '/services/*', 
                    '/products',
                    '/products/*',                                   
                ]
            ],    
    ],

],

Let's say the /services and /products collections have 40 entries each, when we save our single blog entry, we've told Statamic to flush the cache from 84 pages. None of these will be cached again until each one is visited.

If these pages are complicated and require a bit of server effort to rebuild the cache on first visit that could be enough to temporally trigger a timeout, it could run out of PHP memory.

Memory errors usually resolve themselves, but can last a while depending on how much traffic is hitting the server and how much the server is doing, it could be processing multiple queues. Which would only add to the strain.

Real word example

I've seen timeouts triggered by third-party services pinging random uncached pages from three different IP addresses at exactly the same time. This triggered an error because the server was trying to rebuild the page and deliver an uncached page at the same time, traffic was high on the server and it timed out.

Protect the page cache

Long story short, we want to avoid doing anything on a production site that clears page cache, in a way that requires pages to be visited to rebuild the cache. Because we can't be sure all the pages will be revisited in a reasonable timeframe.

Also, this happens every time any entry is edited, it could be a news, case studies or FAQs entry. It's entirely possible an entry could be edited multiple times per week or day. Each time, using the techniques above, the cache for multiple pages is flushed, making the site slow for a first time visitor to each individual page.

Statamic background recache

Half measure keeps Statamic sites fast, and invalidation rules provide a way to flush static cache on certain paths, but those fixes create a problem.

It's possible hundreds of pages could be uncached every time someone edits a single entry in Statamic. All this happens in the background without the editor knowing.

Until Statamic 6 there wasn't a good way to rebuild the cache in an intelligent way. Background recache allows us to configure Statamic to not flush the cache in one go.

Rebuild and swap, not remove and wait

Background recache changes what happens to those same pages, whether that is one page or a hundred. The rules still pick the exact same URLs. What changes is how the process is handled.

With it off, Statamic removes each affected page and lets the next visitor pay for the rebuild. With it on;

• Statamic re-requests each page on a queue,

• renders it fresh,

• and overwrites the cached copy in place.

The old version keeps serving the whole time, so nobody waits for a render. There is a brief window where the listing still shows the old title while the rebuild runs, then the fresh copy swaps in. Stale for a moment is better than cold and slow.

A cached page is never cleared and rebuilt cold. It is rebuilt quietly in the background and swapped in.

How to switch it on

This assumes you are already using half measure caching. Make sure background_recache is in static_caching.php it's new for Statamic 6, so if you've upgraded from an earlier version it might not be there.

// config/statamic/static_caching.php
'background_recache' => env('STATAMIC_BACKGROUND_RECACHE', false),

Add this to your env file

STATAMIC_BACKGROUND_RECACHE=true

Second, give the rebuild jobs somewhere to run. Background recache pushes a job per page onto Statamic's warm queue, so point that queue at a real connection and run a worker for it:

STATAMIC_STATIC_WARM_QUEUE=warming
STATAMIC_STATIC_WARM_QUEUE_CONNECTION=redis

Don't forget to run the config:clear command after making edits to env

php artisan config:clear

I use Ploi to manage my servers, and it has a nice interface for creating queues.

Keep that worker deliberately small. Every rebuild goes back through PHP-FPM to render, the same workers serving live visitors. One warm worker means rebuilds happen steadily and can never starve live traffic for CPU. Let it take a little longer. Slow and quiet beats fast and falling over.

The caveats worth knowing

Background recache has a few limits worth knowing before you lean on it:

• It only refreshes pages that were already cached. A brand-new page nobody has visited has nothing to rebuild, so it still caches on first visit.

• It depends on a live worker. If the warm queue worker is down, rebuild jobs pile up and pages quietly stop refreshing.

• It does not cover a full cache clear. Background recache only runs on content saves made in Statamic.

• A deploy that runs cache:clear flushes everything in one go, and leaves the site cold. Pair it with a static:warm step that rebuilds in place, ideally stop the deploy clearing the cache at all.

Why this is maintenance, not a one-off

Background recache is one checkbox, but making it protect a site is not. It needs the right strategy, a healthy queue worker, and a deploy pipeline that warms pages instead of wiping them.

That ongoing attention, the gap between a setting that exists and a setting that works, is the heart of a website maintenance plan. Prevention over reaction. The best caching problem is the one your visitors never see.