What a uniqueness check actually measures

One number, and what it belongs to

A uniqueness check takes every possible pairing of pages on a site and gives each pairing one number between 0 and 1. Zero means the 2 pages have no phrasing in common. One means they are the same page. Anything at 0.45 or above fails, and both pages in that pair come back.

Hold on to one thing from that sentence: the number belongs to the pair, not to the page. No page has a similarity score on its own. That fact explains most of what follows.

Two toolboxes tipped onto the floor

Here is what the machine does to a page before it compares anything.

First it throws away the filler words. The, and, your, we, for, of. Then it runs a 5-word window down what’s left: words 1 to 5, then 2 to 6, then 3 to 7, all the way to the end. Every one of those 5-word runs is a tool. The page becomes a toolbox full of them.

Now tip 2 pages’ toolboxes onto the floor. Count the tools that turn up in both piles. Divide that by the number of different tools on the floor altogether. That fraction is the score.

The analogy holds up when you push on it, which is why it’s worth using:

• Doubles don’t count twice. A phrase you used 4 times on one page is one entry, the same way owning 3 identical shifters is still one kind of shifter. Repeating yourself on a single page doesn’t move the score.
• Why 5 words and not 1. Every trade page in Australia shares single words. Hot. Water. Same-day. Five content words in the same order is a phrasing fingerprint, and nobody produces the same one twice by accident.
• Why the window overlaps. Change 1 word in a sentence and the sentence still shares most of its 5-word runs. Swapping a synonym in here and there doesn’t get a duplicate page past it.
• Why the bottom of the fraction is every tool on the floor. The more distinct phrasing 2 pages have between them, the lower the score goes. Two long pages that share one identical paragraph score low, and that’s correct — one shared paragraph is not a duplicate page. Repeated paragraphs get caught by a different check, which reports any sentence of 6 words or more appearing on more than one page and names the pages.

Why the line sits at 0.45

It’s a judgement, not a constant out of a textbook, and it’s worth knowing which measurements it was set against.

Pages stamped from a template, where only the suburb name changes, score well above 0.6 and carry no facts of their own. Pages written from different jobs score far lower.

So put the line at 0.7 and it waves through the exact pattern it exists to catch. A check that passes a page with 0 unique facts out of 9 isn’t a check, it’s decoration.

Put it at 0.2 and every honest website in the country fails. Two pages on the same site share a menu, a footer, a phone number, a licence line, a business name and the same list of suburbs. That isn’t duplication, that’s a website having a shape.

0.45 sits in the gap, with clear air on both sides. It is well below what templated pages score and well above what pages written from different jobs score. That matters more than the digits: templated pairs score well above the line and pages written from different jobs score well below it.

It is also a fail line, not a target. 0.44 isn’t a good score. It’s a page that got away with it.

What the score can’t see

It reads words. It does not read meaning. Two pages that say precisely the same thing in different words score low and sail through.

Two pages can say exactly the same thing in different words and pass, which is why there’s a second check.

So similarity is only half of it. The second check counts facts, and it is fussy about what a fact is:

• A number or a price. $1,450, or 40 minutes, or 1970s.
• A name of 2 or more words. An estate, a business park, a road, a supplier.
• A single capitalised word only counts if it appears mid-sentence. A capital at the front of a heading proves nothing.
• The page’s own suburb name is stripped out before counting, on purpose. The suburb is allowed to differ. Everything else has to earn its place.

Every page needs at least 3 facts that appear on no other page. Which is why “we service this suburb” scores nothing, and neither does “quality workmanship at competitive prices”.

Why anything under 80 words is ignored

A short page makes very few 5-word runs. With only a handful in each toolbox, one shared line swings the score from almost nothing to almost everything. The number stops describing anything real.

Below 80 words you are looking at a contact page, a thank-you page or a privacy stub. Whether 2 of those resemble each other tells you nothing about whether the site was stamped out of a template, and letting them into the arithmetic drags the honest scores around.

It is not a loophole for thin content. A real page that lands under 80 words has a different problem and a different check catches it: the structural check applied to every page before it ships has a 250-word floor, and a page under that floor fails there instead. Being skipped by the uniqueness check and being allowed to go live are not the same thing.

Why the whole site, every time

Back to the first section. The score belongs to the pair. A new page on its own has nothing to be compared against, so “check the new page” is an unfinished sentence until you say what you’re checking it against.

The fact count works the same way, and this is the part people find hardest. A page’s unique facts are the facts on that page minus the facts on every other page of the site. Add page 11 in month 6, and page 4’s count changes. Three facts that belonged to page 4 alone now appear twice, and page 4 can fail a check it passed on launch day without a single word of it being edited.

A page can go bad while nobody touches it. The only way to know is to score everything against everything, every time, old pages included.

The cost of doing that is close to nothing. There’s no account and no vendor in the middle. It compares every page with every other page and lists every pair that fails, with its score, so there’s nothing to argue about.

What to do with any of this

Ask whoever built your pages for the duplicate-check results, pair by pair, with the pass mark printed on them. Pass or fail, pair by pair. If nobody can produce one, that isn’t proof your pages are duplicates, but it does mean nobody has ever looked.

And you can do the eyeball version yourself in a minute. Cover the suburb name on 2 pages and read them again — the swap test is the same question asked without a machine.