The 5 questions to ask any report you’re paying for

A report is a claim, and a claim can be checked

You get a PDF every month. You skim it, you file it, and by the time the next one turns up you couldn’t name one thing the last one changed. That isn’t you not paying attention.

A report can be written to describe a month or to be checked. The five questions below are the ones that separate the two. None of them are clever and none are traps. Each one either has an answer or it doesn’t, and the not-having is the finding.

1. What changed on my site?

Not what moved. What changed. A change is something with a before and an after that you could point at: a page that didn’t exist and now does, a title that read one thing and now reads another, a category added to your Google profile, 6 photos uploaded.

Impressions (times you were shown in Google) up 14% is not a change. It’s weather. Weather happens to you whether or not anybody was working, and it moves for reasons nobody in the report controls. A competitor shut down. School holidays. Google shipped an update on a Tuesday.

Read the report with a pen and underline every sentence that names a specific thing on your property. If you underline nothing, the month went somewhere other than your site.

2. On what date?

A number without a date is decoration. Every figure in a report should carry the day it was taken, because everything under it moves.

Ours do. A number measured in August was right in August: it described one day, and detached from that day it stops being evidence of anything.

What to look for in yours: an “as at” date on every table and chart. If the same chart appears month after month with no date on it, you can’t tell whether it was regenerated or copied across, and neither can the person who sent it.

3. Which file shows it?

Ask for the thing the summary was written from. Not a longer summary. The output.

Sometimes the honest answer is that there isn’t one. The report is the artefact, typed up from a dashboard and a memory of the month. That’s not fraud, it’s just a different product to the one you thought you were buying.

A real output file is boring to look at. A fixed list of checks, run page by page, with the thresholds printed on it and the failures counted. You can open it, count them yourself, and hold any line of it against your own site.

One thing to check when you get one: does it contain warnings? If a file you’re handed has never once carried a warning, either nothing is being checked, or something is being tidied before it reaches you. From the outside those look identical.

4. What would a failure have looked like?

This is the strongest of the five. If a report can only come out one way, it isn’t measuring anything.

So ask it straight: what result last month would have made this a fail? If the answer is a shrug, or “well, if the numbers had gone down”, you’re back to weather.

A real check has its failure defined before it runs. A fixed list, a threshold, and a number at the end. Ours runs before a page goes live and you get the results either way. A page passes clean or it doesn’t go up. The list is fixed before the page is written, which is the only thing stopping it from being bent to fit whatever the page turned out to be.

5. Who checked the checker?

Last, and least asked. Any check can be built to pass everything, and a check that passes everything is a rubber stamp in a nice font.

So ask whoever produced yours how the check itself gets tested, and what happened the last time somebody made it less fussy. You would not be able to tell from the outside.

The other half of this is whether faults get reported. Ask whether small faults get reported to you too. Ours go into your results with a plain explanation beside them, because a fault you disclose costs an awkward paragraph and a fault the client finds costs the client.

Run the five over the report on your desk

Take a typical monthly report, section by section, and mark which parts survive.

Section	How it holds up
Executive summary	Survives only if every sentence in it points at something further down the document. On its own it’s a cover letter.
Keyword ranking table	Fails question 1 outright. Nothing on your site changed. Usually fails 2 as well, because a local result differs by where the phone is standing and the table doesn’t say where it was measured from.
Traffic chart	Weather. Fails 1 and 4. Worth glancing at, worth nothing as evidence of work.
Work completed this month	The section that can survive. It passes question 1 if each line names a page. It fails question 3 unless a file sits behind it.
Backlinks acquired (other sites linking to yours)	Survives if each one is a URL you can open today and see your name on. Ask for the date each was acquired.
Recommendations for next month	Not a report at all. It’s next month’s invoice with a friendly heading. Neither passes nor fails, just don’t count it as work done.

If you’d rather ask than mark, one paragraph does the job. Send this:

Can you send me, for last month, the list of things that changed on my site with the date each one happened, and the output file the summary was written from. If there isn’t an output file, just say so and I’ll stop asking for it.

The last sentence matters. It gives an easy, honest answer that costs nobody anything. Whether you get that answer, a file, or three weeks of silence is the actual result of this exercise.

Two report lines that mean less than they look

“Keyword rankings improved.” Ask which keywords, and whether anybody types them. A rank of 3 for a phrase nobody searches is a rank of 3 for nothing. Ask for the search volume next to every keyword in the table, and watch what happens to the length of the table.

“Traffic is up.” Ask up against what, and whether the comparison period is the same length and the same season. A plumber comparing July to December is comparing a burst-pipe month to a holiday month and calling the difference performance. The honest version of that line names both periods and says which is which.

What a five-out-of-five report still won’t tell you

Even a report that answers all five won’t tell you where you rank or how many leads to expect. Nothing we run measures either of those, so nothing we hand you claims them. It’s in the retainer wording rather than in somebody’s head: nothing is judged on traffic for at least the first month.

And none of the five work at all without a starting point. A report is a comparison, and a comparison needs a number you wrote down before anyone started. That’s the day-zero file, and it takes an hour.