Warming Statamic static cache behind basic auth

Staging sites are typically hidden behind a password (basic auth) and are mostly used to show clients development work for sign off before pushing to the production site. However, recently one of my clients has been creating content on the staging site and noticed the cached pages didn't update.

Getting Statamic to clear static cache behind basic auth takes a bit of additional configuration.

When I run a deployment part of the script clears site cache. On a staging site I have a special variation of my production script that passes the basic auth login details, so the cache can be warmed as part of the deployment process.

For a staging site it's not strictly necessary, since the only visitors are users with the basic auth logins. But I prefer the staging site to mirror production as close as possible.

[Lead image: a browser basic-auth prompt sitting in front of a staging URL. Add to public/assets/ if one is available.]

How the Statamic static cache gets warmed

I run Statamic half measure static caching with background recache enabled. Two things trigger a page to be warmed.

• On deploy, the deploy script runs statamic:static:warm --queue, which queues a job per URL to render the whole site.

• On save, an editor updates content in the Control Panel and Statamic queues a background re-cache job for the affected URLs.

The important detail is how a page gets warmed. Statamic does not render it in-process. It fires a real HTTP request back at the page URL, via Guzzle, and stores the response. Guzzle throws on any non-2xx response by default.

Now add basic auth to the mix. My staging sites sit behind an .htpasswd prompt so it stays out of search engines and hidden from casual viewers.

The problem: deploy warming worked, edits did not

On deploy, the warm request hit basic auth, got a 401, Guzzle threw, and the job failed, until I passed credentials to the command. On save, the background re-cache jobs also 401d, but there was no obvious place to give them credentials, so they kept failing. Edits never re-warmed. The page only refreshed when a logged-in browser happened to visit it next.

The symptoms were easy to miss:

• The queue log filled with StaticWarmJob ... FAIL, each failing in around 28ms. Far too fast to be a real render. That is just the round trip for a 401.

• The Control Panel Static Page Cache utility showed Cached Pages: 1. The cache never filled.

A save deletes the cached page, but the job meant to rebuild it never gets past the login prompt.

Why the deploy could be fixed but edits could not

The statamic:static:warm command has built-in support for basic auth via --user and --password. When you pass both, it builds a Guzzle config with 'auth' => [user, password] and serialises that into every queued job, so each warm request authenticates. The deploy fix is simply to pass credentials. Read them from .env and hand them to the command:

# Load basic-auth creds from .env
export $(grep -E '^STATIC_WARM_(USER|PASSWORD)=' .env | xargs)

php artisan statamic:static:warm --queue \
  --user="$STATIC_WARM_USER" \
  --password="$STATIC_WARM_PASSWORD"

Exporting the variables alone does nothing. The command only reads the options, not the environment, so you have to actually pass them to the static:warm command.

The background re-cache path is different. Those jobs are dispatched deep inside Statamic with an empty client config, and there is no option to inject credentials. So they cannot authenticate through basic auth. That is the gap the NGINX change closes.

The fix: let the re-cache requests through basic auth

Every background re-cache request carries a secret query parameter:

https://staging.demo.com/some-page?__recache=YOUR_RECACHE_TOKEN

Because only your server knows it, you can safely tell NGINX to skip basic auth when a request carries the valid token. The auth_basic directive accepts a variable, and the literal value "off" disables auth for that request. A map sets the variable based on whether the token is present.

The token is the credential. If NGINX sees it, the request goes straight past the login prompt.

Step by step

1. Get the re-cache token

On the server:

php artisan statamic:static:recache-token --raw

If you have not set STATAMIC_RECACHE_TOKEN, Statamic derives a token from your APP_KEY automatically. Copy the value it prints.

Tip: set an explicit STATAMIC_RECACHE_TOKEN in .env so the token is stable and does not change if APP_KEY is ever rotated.

2. Confirm background re-caching is on

grep STATAMIC_BACKGROUND_RECACHE .env   # expect =true

If it is not enabled, a save just deletes the cached page instead of re-warming it, and there is nothing to authenticate.

3. Add the map at http scope

The map directive must live at http scope, outside any server { } block. On a Ploi site the main vhost file works. Make sure to set map_hash_bucket_size to 128, the re-cache token is a 64-character sha256, which overflows NGINX's default hash bucket.

Put map_hash_bucket_size and the map above the server { line:

# Add this
map_hash_bucket_size 128;

map $arg___recache $app_auth_realm {
    default              "Authentication Required";   # visitors  -> basic auth
    "YOUR_RECACHE_TOKEN" "off";                       # valid token -> skip auth
}

4. Point basic auth at the variable

Find the config that currently sets your basic auth. On Ploi it is a generated partial, for example /etc/nginx/ploi/staging.demo.com/server/auth.conf. Change the auth_basic line from the literal realm string to the variable, and leave auth_basic_user_file alone:

auth_basic              $app_auth_realm;
auth_basic_user_file    "/home/ploi/staging.demo.com/.htpasswd";

This is the wire that connects the map to the auth. Without it, the map is defined but never used.

5. Test and reload

sudo nginx -t            # must pass
sudo systemctl reload nginx

6. Verify at the HTTP layer

# no token -> still challenged
# expect 401
curl -sI https://staging.demo.com/ | head -1                              

# with token -> waved through
# expect 200
curl -sI "https://staging.demo.com/?__recache=YOUR_RECACHE_TOKEN" | head -1

7. Verify the real path

Save any entry in the Control Panel, then watch the queue:

watch -n 2 'php artisan queue:monitor redis:warming'

Warm jobs should now show DONE instead of FAIL, and the Static Page Cache utility Cached Pages count should start climbing.

Gotchas worth knowing

• $arg___recache has three underscores. NGINX exposes query args as $arg_<name>, and the parameter is __recache, so $arg_ plus __recache gives $arg___recache.

• auth_basic "off"; is a magic value. It is NGINX built-in way to disable auth for a request. Anything else, including an empty string, is treated as a realm.

• map_hash_bucket_size 128; is required. Without it you get could not build map_hash error.

• The map must be http scope. Put it inside a server { } or location { } block and NGINX errors with "map" directive is not allowed here.

• The token appears in query strings, so it can show up in access logs. For staging that is an acceptable trade-off. Rotate the token if it ever leaks.

Re-apply the auth.conf edit if you regenerate auth in Ploi

If your basic auth was created through the Ploi UI, the auth.conf file is generated and managed by Ploi. The moment you toggle the Authentication feature in the UI, Ploi regenerates that file and overwrites your auth_basic $app_auth_realm; change back to the literal realm string, silently breaking the bypass.

So if re-cache jobs suddenly start failing again after you have touched Ploi auth settings, check that auth.conf still reads auth_basic $app_auth_realm; first. The map in the main vhost is safer, but that one line is exposed.

A note on production

This whole problem is specific to environments behind basic auth. Production sites are usually public, so their warm requests are never challenged and none of this applies. It is a staging-only fix.

It is also a good example of why caching is never a set-and-forget switch. A cache that fails quietly looks exactly like one that works, until someone notices the site is slow. Catching that gap is the day-to-day of a website maintenance plan.