Automated vs Manual Accessibility Testing: What Each Really Catches
Automated scanners are fast, cheap, and tireless — and they still miss most of what makes a site unusable for people with disabilities. Here is an honest breakdown of what each method catches, what it misses, and how to combine them.
Reviewed by Khushwant Parihar, CPACC
Updated
The Coverage Question: How Much Can Automation Find?
The most commonly cited industry estimate is that automated tools detect roughly 30-40% of WCAG issues. It is worth being honest about where that number comes from: it is an estimate range, not a single definitive study, and the answer depends heavily on how you count. Measured by distinct WCAG success criteria that can be fully verified by a machine, coverage sits near the low end. Measured by raw issue volume, it can look higher — Deque, the maker of axe, has reported that axe-core rules surfaced about 57% of issues by volume in its own dataset, largely because the failures automation catches best (missing alt text, low contrast, unlabeled fields) are also the most numerous.
Under any counting method, the conclusion is the same: a clean automated scan does not mean a page is accessible. It means the machine-checkable subset of problems is clean. Whether a human can actually perceive, navigate, and operate the page is a different question — one only manual testing answers.
What Automated Testing Catches Well
Automated engines excel at binary, machine-verifiable checks — and because they run in milliseconds, they can apply those checks to every page on every commit.
- Missing or empty attributes: images without
alt, form fields without labels, buttons and links without accessible names, missing page language and title. - Color contrast: text and background combinations below WCAG AA thresholds (for programmatically determinable colors).
- Invalid ARIA: nonexistent roles, disallowed attributes, broken
aria-labelledbyreferences, ARIA on elements that forbid it. - Structural red flags: skipped heading levels, duplicate IDs, missing landmarks, tables without headers, positive
tabindexvalues. - Regressions: the single biggest strength — once a rule passes, CI ensures it keeps passing on every future change, for free.
You can see this class of checks in action by running any URL through our free URL accessibility auditor, which is powered by the axe-core engine.
What Only Manual Testing Catches
The issues below are largely invisible to scanners — and they are the ones users with disabilities hit hardest, and the ones most often cited in legal complaints.
Keyboard Flows
Can a user Tab through the whole page in a logical order, operate every menu, modal, and widget, and never get trapped? A scanner can verify an element is focusable; it cannot verify the journey works. See our keyboard accessibility guide.
Screen Reader Experience
Announcements that make sense, live-region updates that are actually heard, reading order that matches meaning, and custom widgets that behave as their roles promise. Our screen reader testing guide covers how to test with NVDA, JAWS, and VoiceOver.
Quality and Accuracy
A scanner confirms alt text exists — not that it describes the image. It confirms a label exists — not that it makes sense. Meaningful link text, accurate captions, and sensible error messages all require human judgment.
Cognitive Load and Context
Is the flow understandable? Are instructions clear, is enough time given, do error states help users recover, does content on hover or focus behave predictably? These experience-level criteria are inherently human calls — best validated through user testing with disabled participants.
Side-by-Side Comparison
| Dimension | Automated testing | Manual testing |
|---|---|---|
| Speed | Seconds per page; scales to thousands of pages | Hours to days per audit scope |
| Cost per run | Near zero once integrated (most engines are free) | Expert time — the main cost of an audit |
| WCAG coverage | Roughly 30-40% of issues (common industry estimates; varies by study and counting method) | Full coverage, including all judgment-based criteria |
| Consistency | Perfectly repeatable; ideal for regression detection | Varies with tester skill and methodology |
| False positives | Low with good engines, but flagged items still need human triage | Rare — findings come with context and reproduction steps |
| Best used for | CI gates, monitoring, catching regressions early | Conformance claims, VPATs, real usability, legal defensibility |
Tool Categories at a Glance
- Browser extensions (axe DevTools, WAVE, Accessibility Insights): on-demand, single-page checks during development and design review. See our axe vs WAVE comparison for choosing between the two most popular options.
- CI/test-runner libraries (axe-core with Playwright, Cypress, or Jest; Pa11y; Lighthouse CI): fail builds when new violations are introduced.
- Site-wide scanners and monitors (crawler-based platforms and online checkers like our URL accessibility auditor): track issues across many pages over time.
- Linters (eslint-plugin-jsx-a11y, axe Linter): catch issues at author time, before the code even runs.
- Manual methods: keyboard-only passes, screen reader testing, zoom and reflow checks, expert WCAG audits, and moderated usability sessions with disabled participants.
The Hybrid Workflow We Recommend
This is not automated versus manual — mature teams run both, layered by cost. Each layer catches what the cheaper layer below it misses.
- 1
Automate in CI — every commit
Lint accessibility at author time and run axe-core checks in your test suite so machine-detectable violations fail the build. This keeps the noise floor at zero so human testers spend time on real problems.
- 2
Quick manual checks — every feature
Before a user-facing feature ships, someone does a keyboard-only pass and a brief screen reader check of the new flow. Fifteen minutes here prevents most of the serious barriers automation cannot see.
- 3
Expert audits — periodically
A full professional audit against WCAG 2.2 — typically annually or after major redesigns — provides complete coverage, prioritized findings, and the documentation you need for VPATs and legal defensibility. Our website accessibility audit guide walks through the methodology.
- 4
User testing — for critical journeys
For checkout, signup, and other revenue-critical flows, moderated sessions with screen reader users, keyboard-only users, and people with cognitive disabilities reveal friction no audit checklist captures.
Start With a Free Automated Baseline
Run any page through our axe-core-powered auditor to see the machine-detectable issues in seconds — then plan the manual testing that covers the rest.
Frequently Asked Questions
What percentage of accessibility issues can automated testing find?▾
Commonly cited industry estimates put automated detection at roughly 30-40% of WCAG issues, though the true figure depends on how you count. Studies that measure by distinct success criteria tend to land in that range, while Deque has reported that axe-core rules can surface a higher share (around 57%) when measured by issue volume, because the failures automation catches — like missing alt attributes and low contrast — occur very frequently. The consistent finding across studies is that a meaningful majority of issue types require human judgment, so no automated tool can verify full WCAG conformance on its own.
Can automated testing replace manual accessibility audits?▾
No. Automated tools verify machine-checkable facts: an attribute exists, a contrast ratio meets a threshold, an ARIA role is valid. They cannot judge quality or experience — whether alt text is accurate, whether focus order makes sense, whether a screen reader user can complete checkout, or whether error messages are understandable. A site can pass every automated scan and still be unusable with a keyboard or screen reader. Automation is a regression net; manual testing is how you establish actual conformance.
Is manual accessibility testing worth the cost?▾
For any organization with real compliance obligations or real users with disabilities, yes. Manual testing is where the highest-impact barriers are found: broken keyboard flows, focus traps, inaccessible custom widgets, and confusing screen reader experiences — exactly the failures cited in ADA lawsuits and flagged in procurement reviews. The cost-effective pattern is to let cheap automated checks handle the machine-detectable issues continuously, reserving paid expert time for the issues only humans can find.
How often should I run each type of testing?▾
Run automated checks continuously: in code review and CI on every pull request, plus scheduled scans of production. Run focused manual testing (keyboard pass, screen reader spot checks) whenever you ship a new user-facing flow or component. Commission a full expert audit on a recurring cadence — annually is a common baseline, or after major redesigns — and include usability testing with disabled participants for your most critical journeys.
What tools do I need to start automated accessibility testing?▾
A practical free stack: the axe DevTools or WAVE browser extension for on-demand page checks; axe-core integrated into your test runner (jest-axe, @axe-core/playwright, or cypress-axe) for CI; Lighthouse for periodic scoring; and an online scanner such as our URL accessibility auditor for quick full-page reports without any setup. Add a linter like eslint-plugin-jsx-a11y to catch issues at author time.
What is a hybrid accessibility testing workflow?▾
A hybrid workflow layers methods by cost and coverage: linting and automated tests in CI catch machine-detectable regressions on every commit; developers and QA run quick manual keyboard and screen reader checks on new features; periodic expert audits provide full WCAG 2.2 coverage and formal documentation; and usability testing with people with disabilities validates that key journeys genuinely work. Each layer catches what the cheaper layer below it misses.
Essential Accessibility Resources
Comprehensive tools, checklists, and guides to help you create inclusive digital experiences