Why You Can’t Trust Automated WCAG Testing Tools for Accessibility Compliance

We’re all getting more reliant on automation. I am too. There’s a reasonable assumption baked into that – that any test is at least capable of detecting everything it’s designed to test. In most software categories that’s roughly true.

Accessibility testing is different. Automated WCAG testing tools for websites, SaaS products, and especially mobile apps are extremely limited in ways that aren’t obvious from the outside – and almost nobody explains this upfront. Not the tool vendors, not the marketing materials, not the dev teams recommending these tools to their organizations.

Which is why I get calls every week from people who say some version of the same thing: “We just got hit with an ADA lawsuit. Google Lighthouse is showing 95% for accessibility. How is that possible?”

This is a long story. Let me give you the short version first.

TL;DR

• Automated tools check code syntax against a fixed ruleset. They can’t navigate a live interface, test real user flows, or evaluate anything that requires judgment.
• There are three distinct tiers of automated testing tools: free page checkers like Lighthouse and WAVE, paid site-wide scanners like axe DevTools and SortSite, and AI-vision platforms like Evinced. They’re not interchangeable – the price points and capabilities are very different – but they all hit the same structural ceiling.
• The most capable tool available right now is Evinced, which we use on every audit. In a 2026 study against real manual audits, it found 62.8% of issues. axe-core alone found 22.6%. The rest requires a human – trained and experienced tester.
• WCAG – the Web Content Accessibility Guidelines – was written as a guideline, not a compliance checklist. Roughly half its 87 criteria require human judgment to evaluate. That’s by design, not a gap waiting to be closed by better software.
• A clean scan report isn’t a legal defense. Courts care about whether users were blocked, not what the score was.
• Overlays make this worse, not better. 119 companies were sued in a single month in 2025 while actively running an accessibility widget. The FTC fined AccessiBe $1 million for claiming their tool could make any website WCAG-compliant.
• Experience is the variable the tools can’t replace.

Not All Automated Tools Are the Same

The “automated accessibility tool” category spans free browser extensions to six-figure enterprise platforms. Grouping them together is part of why teams end up with a false picture of where they stand. There are three distinct tiers, and knowing which one you’re using matters a lot.

Tier 1: Free Page Checkers

Google Lighthouse and WAVE (Web Accessibility Evaluation Tool, published by WebAIM) are where most teams start. Lighthouse is built into Chrome DevTools and runs its accessibility checks using the open-source axe-core rules library developed by Deque Systems. WAVE injects a visual overlay directly onto the page. Both are free, zero barrier to access, and genuinely useful for catching obvious coding errors during development.

The problem is the output. Lighthouse returns a number from 0 to 100. That number is where the confusion starts. A 100 means the page has no violations of the programmatic checks in Lighthouse’s static ruleset – nothing more. It says nothing about whether a keyboard user can get through a checkout flow, whether a screen reader user can complete a form, or whether a modal traps focus and locks out everyone who doesn’t use a mouse. A perfect Lighthouse score on a genuinely inaccessible page isn’t unusual. We see it on almost every new client engagement.

Tier 2: Site-Wide Rule-Engine Auditors

axe DevTools by Deque Systems, SortSite by PowerMapper, and Siteimprove operate at a different scale. These are paid platforms – axe DevTools Pro runs around $4,200 per user per year, SortSite starts around $49 a month, Siteimprove reaches $15,000 to $50,000-plus annually at the enterprise level. They crawl entire sitemaps, deduplicate template-level issues so a broken nav element across 10,000 pages registers as one fix rather than 10,000, and plug into CI/CD pipelines and issue trackers.

More thorough than a page checker, yes. Still bound by the same structural ceiling. They’re rule engines – they inspect DOM syntax against a list of machine-verifiable conditions. The moment a WCAG success criterion requires live interaction, contextual judgment, or human sensory evaluation, a Tier 2 tool hits the same wall as Lighthouse. The price goes up. The ceiling doesn’t move.

Tier 3: AI-Vision Platforms

Evinced is the tool in this category with the most documented real-world performance, and it’s what we use on every audit we run. Their engine doesn’t just read static code – it renders pages and analyzes them visually using computer vision and machine learning, building a structural semantic model of the interface. That means it can identify an unmapped icon functioning as a search button and check whether the code handles that accessibly. It also incorporates Deque’s axe-core ruleset under a formal licensing agreement, so it captures everything axe catches and layers its own visual-semantic analysis on top.

In a January 2026 study, Evinced ran their engine against 35 manual audits conducted on production sites by experienced accessibility consultants. axe-core alone found 22.6% of the manually reported issues. Evinced found 62.8%. That’s a real and meaningful improvement over standard scanning. It’s also still 37 points short of what a human auditor finds. That gap isn’t a technology problem that better software will eventually close. The ceiling is structural.

What WCAG Actually Is – and Why It Matters Here

This is the part most automated tool marketing skips entirely, and it’s the reason the ceiling exists regardless of how good the tools get.

The Web Content Accessibility Guidelines (WCAG) are published by the World Wide Web Consortium (W3C) as a technical framework for making web content work across an enormous range of disabilities and an equally enormous range of device, platform, browser, operating system, and assistive technology combinations. The name says exactly what it is: guidelines. The W3C explicitly states that determining conformance requires both automated testing and human evaluation – that language comes directly from their own documentation.

WCAG 2.2 has 87 success criteria. Roughly half are non-automatable – not because the tools haven’t caught up, but because applying them correctly requires human sensory evaluation, live interaction testing, and contextual judgment that deterministic software can’t replicate. The W3C defines a criterion as “reliably human testable” only when 80% or more of knowledgeable evaluators would reach the same conclusion. That threshold exists because for a lot of these criteria, interpretation is the actual work.

WCAG was never designed to be a binary legal test. It became one because civil rights law needs measurable benchmarks, and WCAG Level AA was the most credible technical framework available when courts and regulators needed something to point to. The DOJ’s 2024 ADA Title II final rule codifies WCAG 2.1 Level AA as the mandatory standard for state and local government websites. What started as authoring guidance is now in complaint filings – and that shift matters enormously when you’re trying to understand what “compliance” actually demands.

For a closer look at how WCAG 2.2 applies to your product specifically, our post on what’s new in WCAG 2.2 covers the current standard in practical terms.

What Automation Can’t Test

The UK Government Digital Service tested 13 automated accessibility checkers against a single page with 142 known, documented barriers. The best tool – SortSite – caught 40%. WAVE caught 30%. axe caught 29%. Every one of the 13 tools missed reading order issues, lack of visible focus indicators, and context-dependent use of color.

Deque’s own Automated Accessibility Coverage Report (March 2021) goes deeper. Analyzing over 2,000 first-time audits spanning 13,000+ pages and nearly 300,000 issues, the study found that 57% of total issues were caught by automated testing – but that figure is skewed heavily by color contrast, which automation detects at near-perfect accuracy and which accounts for a disproportionate share of all recorded violations. Per-criterion detection rates for the functional failures that actually stop someone from completing a task are a different story. Keyboard navigation and focus order failures – the issues that determine whether someone using only a keyboard can get anywhere on your site – come in close to zero. A high volume number can obscure a genuinely bad picture once you look at what’s actually in the gap.

Here’s a concrete example. A scanner checks a modal dialog and sees an aria-label. Green check. What it doesn’t test: does focus actually move into the modal when it opens? Can a keyboard user get back out, or are they stuck while the rest of the page sits behind it? Does pressing Escape close it? When it closes, does focus return to the element that triggered it? Those are four separate WCAG failures, every one requiring live keyboard testing with a screen reader, and they fail in production more often than teams expect. All four are invisible to automated scanning.

Single-page applications built in React or Vue are another consistent problem area. When the app changes views by manipulating the DOM, a scanner sees only the initial page state. It can’t follow a route change, confirm that focus moves correctly to the new content, or verify that a screen reader user doesn’t have to re-read the whole page to find their place.

Custom widgets – dropdowns built with <div> tags, drag-and-drop interfaces, carousels, dynamic data grids – don’t exist to a rule engine unless they carry explicit ARIA markup. Even then, whether that markup produces the right screen reader behavior only comes out when you actually test it with assistive technology.

WCAG 2.2 made the human-testing requirement more explicit. Focus Not Obscured (2.4.11) requires checking whether sticky headers or chat widgets visually cover focused elements – which changes by viewport and scroll position. Accessible Authentication (3.3.8) requires walking through a real login flow to confirm it works with password managers and doesn’t impose cognitive barriers. Dragging Movements (2.5.7) requires confirming that alternative pointer methods actually function. None of that shows up in a scan. For screen reader compatibility overall, automated detection covers roughly 23% of real issues. For cognitive accessibility, around 15%.

The Legal Reality

Automated scan reports don’t hold up as a legal defense in Americans with Disabilities Act (ADA) litigation. Plaintiffs’ attorneys use scanners to find targets – the obvious coding errors that are easy to document. Actual complaints document real users with disabilities hitting functional blocks that stopped them from completing something. Buying a product. Submitting a form. Logging in. Courts have consistently held that the standard is equal access, not a clean score. A 100 on Lighthouse is legally irrelevant if a plaintiff can show they were blocked.

There’s a newer dynamic making this worse. Pro se ADA Title III filings surged 40% in 2025. People with disabilities are using AI tools to paste page source code, identify WCAG failures, and generate legally structured complaints without an attorney. The cost to file has dropped to near zero. Any site with detectable automated failures is a viable target, regardless of size or industry.

Overlays make this worse still. The JavaScript widgets sold as one-line compliance solutions – AccessiBe, UserWay, and similar products – don’t fix underlying code. They layer a UI patch on top of structural failures while frequently interfering with the assistive technologies they claim to support. In January 2025, the Federal Trade Commission (FTC) filed a complaint against accessiBe and finalized a $1 million consent order in April 2025, finding that their tool did not make websites WCAG-compliant despite explicit claims that it would. In May 2025 alone, UsableNet reported that 119 defendants were sued while actively running a third-party accessibility widget. Overlays don’t reduce legal exposure. They signal that the problem was papered over rather than fixed. We’ve covered this in more depth in our post on why accessibility overlay widgets attract lawsuits.

Government and education procurement is a different kind of problem. Experienced reviewers at .gov and .edu institutions can spot an Accessibility Conformance Report (ACR) built on automated output immediately. An ACR that claims full WCAG support based on scan data will have “Supports” entries for criteria that are actually broken under manual evaluation – and reviewers who process these documents routinely know exactly which criteria those are. A defensible ACR requires manual verification, assistive technology testing with screen readers like NVDA and JAWS, and keyboard-only navigation testing across real user flows. For SaaS teams going through this for the first time, our VPAT guide for product teams covers what that process actually involves.

Why Experienced HUMAN Auditors Are Key

WCAG is interpretive. Applying it correctly across the full range of disabilities and the full matrix of assistive technologies, devices, and interaction patterns takes years of hands-on practice – not years reading about it, years doing it across hundreds of real products.

Our auditors each bring more than 14 years of direct accessibility testing. Our team lead has been doing this work for over two decades and contributes directly to the W3C working groups that write the standards. So when I talk about experience, that’s what I’m talking about.

The tools are part of our process. Evinced runs on every audit. So does manual keyboard testing, screen reader evaluation across multiple assistive technology and browser combinations, and judgment-based review of every WCAG criterion that can’t be evaluated from static code. The tools give us a starting point and a regression check. The audit is everything that comes after.

Wrap-Up

A high Lighthouse score means your site passed a specific set of automated checks against a static snapshot of the DOM. It doesn’t mean your site is accessible to people with disabilities. It doesn’t mean you’re protected under the ADA. And it says nothing about whether the people who rely on keyboard navigation or screen readers can actually get through your checkout, your application, or your contact form.

Automated tools are a necessary part of any accessibility program, and they’ve gotten meaningfully better. But the gap between what they catch and what actually blocks users is still exactly where the legal exposure lives.

If you’ve been relying on scan scores to assess your compliance posture, a manual WCAG accessibility audit is the right place to start.