Technote

Servers & infrastructure Intro

Series The incident runbook Part 1 of 8

The first five minutes: scope before cause

Hunting for the cause first costs you time. Establish the scope and the log you need is already decided, with half the candidates gone.

When the message “the site is down” arrives, the instinct is to start looking for the cause. That instinct is the most expensive mistake of the first five minutes. Until you know the scope, you cannot even decide which log to open.

This is exactly why the order should be decided in advance. Outages arrive when your judgement is at its worst, and what you need then is not clever reasoning but a sequence you already know by heart.

The order of the first five minutes — skip a step and the next one has nowhere to look

Minute one — three questions that fix the scope

Answer them in order. The three answers are your candidate list.

  1. The whole site, or one page? Everything down points at the server, the database, the certificate or DNS. One page points at whatever that page uses — a feature, a query, a template.
  2. Everyone, or only you? Check again from another connection or an external checker. If it is only you, look at caching, cookies and blocked addresses first.
  3. What changed recently? The overwhelming majority of outages are caused by the most recent change.

Always check from outside. A browser refresh drags cached copies along, and being logged in as an administrator bypasses the page cache, so you see a different site from your visitors. One status code is enough.

curl -s -o /dev/null -w "%{http_code}  %{time_total}sn" https://example.com/

A 200 with an empty page means the server is alive and the application failed to produce an answer. A 502 or 504 means the application never answered at all. No response whatsoever points at DNS, a firewall or the network.

Minute two — deploy history answers “what changed” faster than any log

Before opening a log, read the change list. Reverse that order and you will read several hundred lines before remembering that a plugin was updated yesterday.

There is a fixed set of places to look: deployment history, the WordPress update log, server package updates, DNS and certificate changes, and your host status page. If automatic updates are enabled, things change without you doing anything, so this is not optional.

Symptom to starting point — this single table decides most of the first five minutes

After five minutes — tell people, and start writing things down

Even with no cause identified, do two things at the five-minute mark. First, say something. If your business runs on enquiries, one line saying you are looking into it beats five silent minutes by a wide margin. Second, timestamp everything: one line per check you ran and per change you made.

That log is not for posterity. It is for right now — about half an hour in you start losing track of what you have already tried, and repeating a mitigation is how a bad state becomes a worse one. The write-up in the final part is assembled from exactly these notes.

For the structure underneath all this, the Servers & infrastructure archive goes deeper, and if you want an outside view of the site right now, the site check in our free tools reports response, headers and certificate together.

Next part

The most disorienting symptom of all is a completely blank screen. The next part is about recovering the error message from it — into a log file, never onto the screen.

More on this topic

All technotes

Servers & infrastructure Intro

Watching only the home page misses outages

Plenty of sites have uptime monitoring and still miss outages. The cause is usually the same: the page being checked never touches the database.

Site operators 7 min read

Servers & infrastructure Advanced

Moving wp-cron to system cron, and why schedules run late

Scheduled posts that publish late are usually a structural problem, not a code one. WordPress cron rides on incoming requests, so on a quiet site nothing runs at…

Developers 8 min read

Servers & infrastructure Practical

Page caching belongs on the server layer

The layer that stores finished HTML only pays off in front of PHP. Stack a plugin cache on top of a server cache and you now have two…

Developers 7 min read

₩270,000 · Join the program