Four accessibility checkers, one page, thirty known defects

The fixture is a single HTML page containing thirty deliberate defects, each mapped to a WCAG 2.2 success criterion, chosen to span the common failure classes (text alternatives, names and labels, contrast, structure, ARIA, keyboard, timing and media) and to include several that require judgement rather than pattern matching. It was served over HTTP from localhost and each tool was run on 2 September 2026 in headless Chrome 143 with default settings and the WCAG 2.2 AA policy where a tool offers one.

A defect counts as found when the tool reports a violation on the right element for the right reason. A report that only appears in a needs-review or potential category is recorded separately, because most continuous-integration setups fail a build on violations and never surface the rest. A report on the right element for the wrong reason is recorded as wrong.

Download the fixture page and run it yourself. It is marked noindex; each defect is labelled in a comment.

The core is shared. Twelve defects were found by all four tools, and they are the ones that matter most in litigation: missing alt text, unlabelled fields, empty buttons and links, untitled frames, low contrast, no page language. These are the failures that appeared in the complaints in Domino's, Winn-Dixie and Target, and any of the four would have caught them.

The edges differ, and IBM covers more of them. Only IBM Equal Access caught the div with a click handler and no role, the label pointing at nothing, and the two tables with no header cells. Only HTML_CodeSniffer caught the duplicate id, a check axe dropped in version 4. Only axe and Lighthouse caught the positive tabindex. Running two engines is cheap and finds more than either alone; IBM plus axe covered 22 of the 30 outright.

Review categories hide real failures. axe put the broken aria-labelledby reference and the uncaptioned video in its incomplete list; IBM put the untabbable widget, the filename alt text and the video in potential violations. A build that fails only on violations never sees these. Read the review lists.

Five defects are invisible to automation, and they are common. Meaningless link text, repeated link text with different targets, a skipped heading level, a page title placed after the content, and a focus outline removed in CSS were missed by every tool. Screen reader users rank nonsensical links and buttons third among their most serious problems, and GDS found lack of visible focus among the top issues in UK public sector monitoring. A clean automated run says nothing about them.

A score is not a measurement. Lighthouse gave this page 49. It found fifteen of thirty defects, and its weighting means the number would move if the same defects were rearranged. WebAIM's 2026 evaluation found a median Lighthouse score of 85 across the top million home pages, 95.9% of which had detectable failures. Use it to spot regressions, not to make a claim.

Noise is a cost. axe reported twenty-eight best-practice findings on a page with no landmarks, all correct and none a WCAG failure. HTML_CodeSniffer reported a captioned data table as a layout table with a caption. Both are the kind of result that trains developers to ignore the report.