Measure the noise floor before you set a visual diff threshold

A threshold answers one question — how much difference is meaningful — and that question cannot be answered until you know how much difference appears when nothing has changed. Most of the published advice skips that step entirely. Start under one per cent, it says, then tune it if the suite gets noisy.

Tuning against failures optimises for a green suite, which is the one thing a threshold must never be optimised for. This repository started where everyone starts, with numbers chosen by eye, and its config admitted as much in the file: the canvas numbers were marked PROVISIONAL, placeholders until a same-machine reference-versus-reference run established the true noise floor.

That run happened the same evening, in commit b5175ae, and it moved every number in the file. It moved them in the direction almost nobody expects — the measurement made the suite stricter, not looser, and it took one gate away completely.

The tool has one purpose and it is written in its first line: measure how much the instrument disagrees with itself. It serves a single build, captures the reference twice through the same capture script the real suite uses, and compares run A against run B. Whatever comes back is noise, because nothing changed between the two.

The comparison is deliberately thin — two numbers per image pair, no diff image, no interpretation. pixelmatch supplies a count of pixels that crossed a colour cutoff, and a first-party SSIM at an eight-pixel window with a stride of four supplies the perceptual view. Both the mean and the worst window are kept, because a shader regression is usually local and a mean of 0.99 can hide a broken corner.

Three populations come out and they are never merged: structural screenshots, raw canvas captures, and observation drift — every non-pixel field of the per-checkpoint digest that differed between two runs of the same page. Percentiles are reported as min, p50, p95 and max over each sorted set.

function comparePair(a, b) {
  if (a.width !== b.width || a.height !== b.height) return { ok: false, reason: 'size-mismatch' }
  const total = a.width * a.height
  // No diff image is wanted here — only the count. pixelmatch accepts an
  // undefined output buffer for exactly this case.
  const changed = pixelmatch(a.data, b.data, undefined, a.width, a.height, {
    threshold: 0.1,
    includeAA: false,
  })
  const score = ssim(a, b, { windowSize: 8, stride: 4 })
  return {
    ok: true,
    diffRatio: changed / total,
    changedPixels: changed,
    totalPixels: total,
    ssimMean: score.mean,
    ssimWorst: score.worst,
  }
}

It costs exactly twice a single-viewport capture, and a viewport here is six to nine minutes of wall clock, so the measurement is a quarter of an hour rather than free. Its output is committed to the repository on the grounds that it is evidence, and small.

Same machine, same browser, same URL, two runs of the Golden Master homepage at 1440 × 900, stamped 2026-08-19T19:33:49.368Z on Node v22.23.1. Nineteen structural screenshots and eighty-nine raw canvas captures came back from each side.

The structural population disagreed about nothing whatsoever: pixel diff ratio min 0, p50 0, p95 0, max 0, with eighteen of the nineteen pairs byte-identical and the nineteenth a marquee that is captured live on purpose, because continuous motion is the behaviour under test. The canvas population disagreed about almost everything: p95 0.025829475308641974, max 0.8112244897959183, an SSIM mean falling to 0.0181484156493548 and a worst window reaching -0.886465409531798.

That negative reading is a real value rather than a fault. SSIM runs from -1 to 1 as Wang, Bovik, Sheikh and Simoncelli defined it in 2004, and this repository's unit test pins the fact by asserting that an inverted image scores below zero. A metric that can legitimately go negative is telling you the two windows are structurally anti-correlated, which is what a canvas caught at two unrelated moments looks like.

That last one is the reason to run this over more than screenshots. listenerTotal read 447 on one run and 431 on the other at the settled hero, a drift of 16, while rootFontSizePx, documentHeight and globalsLeaked held exactly across both runs and across all 21 checkpoints of a later matrix. The assertion was demoted from exact to tolerant with a maxDelta of 24 — above the observed 16 and below anything that would read as a listener leak — and the others stayed exact, because the measurement said they could be.

The same discipline — measure the condition before you trust the number — produced this site's published accessibility figure, where an uncontrolled condition inflated the count about 140 times.

The guessed canvas gate demanded an SSIM mean of at least 0.98. The measured floor is 0.018148. There is no threshold between those two numbers, and no amount of tuning would have found one: a suite shipped at 0.98 would have failed on every run, been fixed by loosening, and ended up proving nothing about the highest-risk surface on the page.

So the gate was not relaxed, it was disqualified. enforced false, meanMin null, worstMin null, and a status string that names the condition rather than shrugging: BLOCKED, needs deterministic time control on the scene side. The comparator honours that in code and not only in prose — it computes the SSIM on every run, records the mean, the worst window and the coordinates of the worst window, and returns ok regardless.

The real cross-target run confirms the verdict from the other direction. Across the 306 canvas SSIM rows comparing the two builds, the mean has a median of 0.7284514638837428 and the worst window has a median of -0.20839572391449632. A gate whose median worst window is negative between two builds believed correct cannot separate a regression from noise.

BLOCKED is only defensible when it carries a named unblocking condition, and here that condition landed. The rebuild's own WebGL scene exposes a test hook that renders exactly one frame at an explicit time, and with it the same content became gateable byte for byte: a pinned frame is now asserted identical across two independent page loads. Parity against the legacy stays a human review against captured stills, by construction, because the legacy cannot be pinned to a frame.

Demanding determinism also found two defects that looking at the image would not have. A composite pass left a settled frame blended over an uncleared buffer, caught because frames pinned at 1.5 seconds and at 9 seconds came back byte-identical — a pinned time that changes nothing is a uniform that is not reaching a shader. That zero difference was the bug, and a comparator tuned only to detect too much change would have reported green.

Pinning a canvas to one frame is the same capability a canvas needs to answer a reduced-motion request honestly — what WebGL and Rive owe prefers-reduced-motion.

The structural row of that measurement reads 0 four times over, and the thresholds file quotes it as eighteen of nineteen checkpoints byte-identical across two runs. Both statements are exactly true. Decoding the committed screenshots shows what they are true about.

The reference homepage at initial-load is 1440 × 900, 5,851 bytes, and one colour: 1,296,000 pixels of #ff00ff. So is hero-settled, and hero-pointer-corner, and otot-section, and footer, and twelve more. Sixteen checkpoints, at sixteen different scroll positions from y 0 to y 17809, share a single sha256.

This is the documented mask policy working exactly as specified. The reference page carries two full-viewport canvases, and before each shot a fixed magenta overlay is pinned to each canvas's viewport rect at the top of the stacking order — never drawn into the canvas itself, because filling one through a 2D context destroys its contents and would corrupt a WebGL surface outright. Two full-viewport canvases produce a full-viewport overlay at every scroll position.

The finding is discoverable only because the instrument writes down what it refused to look at. Every screenshot record carries the rectangles it excluded, and the architecture doc names that field as one of the two things to read before trusting any pixel result. Nothing in the repository is false; the sample size behind one sentence is simply smaller than the sentence implies.

The two checkpoints that do carry page pixels carry them by accident of policy. menu-open and menu-closed-again hide their canvases instead of masking them, because the menu's canvas sits inside a container that is still transforming when the mask rect is computed, so a fixed overlay would drift off it. Both were byte-identical across the A and B runs, and both are still byte-identical in an independent full-matrix run five days later — which is real evidence, and it is why the tightened 0.0001 is right. It rests on two images, and the file says eighteen of nineteen.

Nor is it a one-off. Decoding every structural screenshot named in the two full-matrix manifests gives 108 of 126 on the Golden Master and 114 of 132 on the rebuild that are 100 per cent one colour, at all six viewports, on both targets. The cheapest check available here is a one-liner: decode a baseline and count its distinct RGB values before you trust a zero.

A noise floor tells you what your instrument can resolve. It cannot tell you whether the checkpoint was aimed at anything, and those two failures look identical from the summary line.

At three of the six viewports — breakpoint-991, tablet-768 and mobile-390 — the screenshot named menu-open and the screenshot named menu-closed-again have the same sha256, on both targets. The click that opens the menu is recorded successful, failedInteractions is 0 everywhere, and the pixel comparison passes at ratio 0 in both directions because both targets are equally blank. Parity between the two builds is genuinely preserved. The checkpoint whose stated purpose is to prove that the close path leaves no residue proves nothing at half the matrix.

The observation record contains the answer and it is split across two adjacent fields, one of which is far more inviting to read than the other.

The general rule has two halves and no threshold can merge them. Measure how much the instrument disagrees with itself, and separately assert that every checkpoint reached the state its name claims — a visible element, a non-zero box, a digest field that actually moved between the before and the after. The first half is arithmetic over images. The second is a different assertion entirely, and it is the one that stops a suite from grading its own blank paper.

The committed cross-target run reads 132 compared, 132 passed. It also reads 162 canvas comparisons that never took place, and the comparator prints that number next to the pass count deliberately: a run that says 132 of 132 while silently holding a hundred unmade comparisons is the shape of a manufactured pass.

The largest category is a crop artefact, which is the page agreeing with itself while the screenshot cropper does not. An element 13.328125 CSS pixels wide came back cropped to 15 by 14 on one side and 14 by 15 on the other, because the element screenshot scrolls the element into view first and the sub-pixel origin moves under it. The fix is not a tolerance: the CSS box is gated and enforced, 468 of 468 box checks passing, while the PNG size is recorded as informational — the box is what the page decides, the crop is what the cropper decides, and only the first is a statement about the build.

The other two categories are cheaper to explain and far more expensive to run. 192 of 792 canvas capture attempts found no box to photograph at all. 66 attempts timed out at 30 seconds each, every one of them on a single canvas — the reference's page transition, which never held still for one of its 66 attempts. That is at least 1,980 seconds of a 2,620.6-second matrix run: three quarters of the wall clock spent waiting to photograph something that could not be photographed.

The rebuild's own matrix, a slightly larger one at 132 checkpoints, finished in 4.82 minutes against the reference's 43.68. Nearly all of that difference is one canvas that cannot be pinned to a frame, which is a performance fact and an instrument fact at the same time: the thing your suite cannot photograph is also the thing it spends its afternoon on.

Wall clock is its own measurement problem, and the scroll layer on this site got the same treatment — Lenis against native scroll, measured rather than argued.

Half the confusion about thresholds is that the word names two different numbers. In pixelmatch — the library under this comparison, and the model Playwright's own screenshot assertion follows — threshold is a per-pixel colour-distance cutoff and not a per-image tolerance. In version 7.2.0, the one pinned here, threshold 0.1 becomes a maximum squared YIQ distance of 35215 × 0.1 × 0.1 = 352.15, against a maximum possible value of 35215. The function then returns a count of the pixels that crossed it.

The per-image allowance is a separate number, and it is the one a noise floor tells you. Playwright makes the split visible and leaves half of it empty: threshold defaults to 0.2, while maxDiffPixels and maxDiffPixelRatio have no default at all — so out of the box, a single sufficiently different pixel fails the assertion. Vitest ships no default allowance either and says so outright, recommending a ratio computed on the size of the screenshot rather than a fixed count.

One more thing the word hides: the metric underneath it is not stable across versions. pixelmatch on its main branch has replaced the YIQ distance with the OKLab HyAB metric, so threshold 0.1 is not one fixed quantity over time. And includeAA reads backwards from its name — false, which is the default and this repository's setting, means anti-aliasing detection is on and AA pixels are not counted as differences.

None of these defaults knows anything about your page, which is the entire point of measuring your own. Both Playwright and Vitest also tell you to regenerate baselines in the environment you will compare them in, and every number quoted above is one machine, one Chromium build, headless, with no determinism flags passed at capture time — no font-render-hinting, no forced device scale. The 0.0001 allowance is described in the file as headroom for font rasterisation across machines, which is headroom for a case this repository has not yet measured.

The procedure is short and most of it is bookkeeping. What makes it hold up is that the output is committed beside the numbers it justifies, so its age is visible in a git log rather than in somebody's memory.

Two caveats this repository demonstrates on itself, both of them visible in its own config. Two of its six viewports, both at deviceScaleFactor 2, have never had a floor measured at all, and their thresholds carry the admission where the next reader will meet it: not yet measured, provisional. The architecture doc states the consequence without softening it — a pass or a failure at those two viewports proves nothing about the instrument, and no candidate should be signed off on them.

And the floor itself has exactly one commit. It was taken on 2026-08-19 and has never been re-run, while the scenario it measured has changed twice since and grown from 19 checkpoints to 21. A noise floor is perishable, it decays as soon as the page, the browser or the machine moves, and the only reason anyone can tell how stale this one is comes down to a small dated file sitting in version control next to the numbers it produced.

The same argument settled a typographic number on this site, where a ratio everyone repeats was replaced by a line-height floor computed from the outlines.

Instrument work like this is part of how the sites here are built, which is described in the four disciplines.