Count real readers from a server log

A raw access log overcounts readers by a wide margin, and the easy filter (keep browser user agents, drop declared bots) does not fix it. On this site, six days of log said 4,958 pageviews from 529 visitors. Checked against asset loads and request timing, it was 48 pageviews from 21 addresses.

This guide is the method that gets you from the first number to the second: what to check, the commands, and how far you can trust the result.

#What you need

• An access log that records the real client address, the user agent, the method, the status and the URI for every request. The commands below assume Caddy's JSON format, one object per line; with another server, get the same fields out of its log format first.
• jq, sort and uniq.
• Enough retention to be worth counting. A few days answers "is anyone here"; it does not show a trend.

If you would rather not use a terminal, the Access Log Reader runs the same checks on a log file in your browser. The file never leaves your device.

#1. Put every log file in one stream

Rotated logs are usually compressed. Read them together with the live file:

sudo sh -c 'cat /var/log/caddy/*.access.log; \
            gzip -dc /var/log/caddy/*.access.log.gz 2>/dev/null' > /tmp/all.log
wc -l /tmp/all.log

#2. Count the client, not your CDN

Behind a CDN such as Cloudflare, the server's TCP peer is the edge node that proxied the request, not the reader. Caddy logs that peer as .request.remote_ip. Its .request.client_ip is the reader only when Caddy trusts the CDN's address ranges (trusted_proxies with client_ip_headers in the server options); without that, it is the edge address too. In the log I counted, client_ip still held edge addresses, so unique visitors counted on it were datacenters. Compare the three shapes:

jq -r '.request.remote_ip' /tmp/all.log | sort | uniq -c | sort -rn | head -5
jq -r '.request.client_ip' /tmp/all.log | sort | uniq -c | sort -rn | head -5
jq -r '.request.headers["Cf-Connecting-Ip"][0] // "none"' /tmp/all.log \
  | sort | uniq -c | sort -rn | head -10

Edge addresses collapse everyone into a handful of rows. If client_ip already matches the header column, your server resolves the reader and either works. The commands below read the CDN header first and fall back to client_ip, so they give the same identity either way. If your log has neither a resolved client_ip nor the header, you have a proxy inventory, not a visitor count.

#3. Remove your own tooling

Your monitors, audits and deploy checks hit the site more than most readers do, and some of them send a browser-shaped user agent. List every agent and read all of it:

jq -r '.request.headers["User-Agent"][0]' /tmp/all.log \
  | sort | uniq -c | sort -rn | head -40

Mark every agent you recognise as yours. On this site my own tooling was 36% of all requests, curl alone was 5,578, and six of my agents had slipped past my exclusion list, including one that starts with Mozilla/5.0 (compatible; ...).

Tuning this list should move the unverified pile, not your headline number. If the confirmed count changes every time you add an exclusion, the method is not measuring anything yet.

#4. Prove that assets reach your origin

The next test only works if static files actually arrive at your server. If your CDN serves every asset from cache, a missing asset request tells you nothing. Check first:

jq -r 'select(.request.uri | startswith("/_next/static"))
       | .request.headers["Cf-Connecting-Ip"][0] // .request.client_ip' /tmp/all.log > /tmp/assets.txt
wc -l < /tmp/assets.txt            # static requests
sort -u /tmp/assets.txt | wc -l    # distinct addresses that fetched them

Use your own framework's asset path. I saw 1,740 static requests from 278 distinct addresses, so assets were demonstrably arriving. If this comes back near zero, stop here: the rest of the method is unsupported on your setup.

#5. Split pages from assets per address

A browser that renders a page also fetches its CSS, scripts and fonts. A scraper reads the HTML and leaves. Count both per address, using only successful GET requests:

jq -r 'select(.request.method == "GET") | select(.status >= 200 and .status < 300)
       | (.request.headers["Cf-Connecting-Ip"][0] // .request.client_ip) as $ip
       | .request.uri as $u
       | if ($u | test("^/_next/static|\\.(css|js|woff2?|png|svg|ico)($|\\?)"))
         then "\($ip)\tasset" else "\($ip)\tpage" end' /tmp/all.log \
  | sort | uniq -c | sort -k2

An address with page rows and no asset row is not a browser. Then look at timing: one address here requested five pages inside a single second with no CSS, no script and no font. In a dashboard that looks like an engaged visitor. In the log it is a machine.

Keep the GET and 2xx filters. Without them, a scanner that sent POST / and got a 404 can count as a pageview, and if the same address fetched an asset that day it gets promoted to confirmed. That was a real bug in my first version.

#6. Sort what is left into tiers

Only one tier is a number worth saying out loud.

Confirmed plus unverified is the naive count. The gap between them is the point.

#7. Check every failing status, not only 404

An audit that greps for 404 misses the other ways a URL can fail. Look at the whole histogram:

jq -r '.status' /tmp/all.log | sort -n | uniq -c | sort -rn

I had been ignoring 73 requests that got 410 Gone, because I served 410 on purpose. Two of them were real people arriving from Google on a page I had retired. Ask "what is failing", not "what is missing".

#What the number is not

The confirmed tier is a floor. A returning reader with a warm cache asks for the HTML and nothing else, so the method files them as a machine. The true count sits above the confirmed number and nowhere near the naive one.

Addresses are not people either: a household or a carrier NAT merges several readers into one address, and VPNs or rotating IPv6 prefixes split one reader into several. At low volume one person swings a daily figure by twenty percent, so a week of data is not a trend.

#What to do with it

Use the number to decide what to fix. Here, 56% of confirmed pageviews were app privacy and support pages, and the services page with 513 raw hits had one confirmed human. Reading the raw count, I would have rewritten the services page. The real problem was that almost nobody saw it, and more analytics would only have measured that at higher resolution.

Run step 4 on your own log before you quote a visitor count. It either confirms the number you have been telling people or it does not.