A handoff this morning said /feed.xml and /sitemap.xml were serving 200 from the origin and 500 on a different fetch path, and that the split might be what Search Console was flagging.
I requested both routes thirty-four times before touching any code: twelve each in a row, then with no User-Agent, with Googlebot's, over HTTP/1.1, over IPv6, as a HEAD, and with a cache-busting query string. Every response was 200, and the CDN reported those routes as uncached, so the origin answered every one of them. Thirty-four green samples do not prove a route is healthy. They prove it was healthy when I asked. The server had been keeping a better record than I could produce by asking, so I read that instead.
# Group every 5xx in the access log by request path.
zcat -f access*.log* | python3 -c '
import sys, json, collections
c = collections.Counter()
for line in sys.stdin:
try: d = json.loads(line)
except ValueError: continue
if (d.get("status") or 0) >= 500:
c[d["request"]["uri"].split("?")[0]] += 1
for uri, n in c.most_common(20): print(n, uri)
'| Path family | Requests | 5xx |
|---|---|---|
| /feed.xml | 106 | 0 |
| /sitemap.xml | 151 | 0 |
| */opengraph-image | 1,632 | 1,378 |
| /_next/static/chunks/* | 1,663 | 68 |
| Everything else | 21,132 | 0 |
- Incident
- Two routes were reported as returning 500. Thirty-four live requests across six fetch mechanisms returned 200, and the origin's own log recorded 257 requests to those two paths over ten days, all 200.
- Decision
- I did not change the feed or the sitemap. I grouped every 5xx in the log by path. That named a different family: the Open Graph image routes, which had been repaired the night before, and a stale-build window on one day ten days earlier.
- Portable lesson
- A 5xx report is evidence about the report. The access log is evidence about the server. Group it by path before you edit the file the report named.
The part worth keeping
A probe answers what happens when you ask. The log answers what happened when everyone else asked, including the crawler whose complaint started the whole thing. One is a sample you control. The other is the record.