The limits of automated auditing.

Automated accessibility tools can test only a portion of the WCAG criteria; the rest requires human judgment and never appears in the report.

The most dangerous thing about accessibility tools is not that they make mistakes — they usually do not. The danger is that the questions they do not ask never appear in the report. A tool runs thirty rules and you pass all thirty; the report is green. In that report there is no trace of the absence of the hundred rules that never ran.

The result is a widespread and costly fallacy: a high score is mistaken for an accessible site. Yet the score is only the answer to the questions that were asked — and the value of an audit hides less in the questions asked than in the ones that were not.

What the tools are genuinely good at

The purpose of this page is not to belittle the tools. Automated auditing does in seconds a job impossible by hand, and it should run on every project. Its strong areas are clear, and all are questions a machine can answer definitively.

1. Exists-or-not questions — is there an alternative text attribute, does the form field have a label, is the page language declared, is there a document title.
2. Numeric thresholds — is the text contrast above 4.5:1. With the colors known, the math is exact.
3. Structural consistency — is there more than one h1, does the hierarchy skip, is the same ID used twice.
4. Invalid declarations — is a nonexistent role assigned, does an attribute carry a wrong value, does an ID reference point at nothing.
5. Focusability — is an element clickable but not focusable.

Every item on this list represents real defects, and automated auditing finds them far better than a human. On a site with thousands of images, finding the ones missing alternatives by hand takes days; the tool lists them in seconds. That is why automated auditing goes IN FRONT of manual auditing, not in its place.

What the tools cannot see

The boundary lies in whether the question is machine-answerable. The items below produce real accessibility defects, yet none can be detected automatically with confidence.

1. The CORRECTNESS of alternative text — the text exists, but does it describe the image? A file name is text too, and the tool counts it as valid.
2. The meaningfulness of link text — “Read more” is valid link text; whether it makes sense out of context the tool cannot know.
3. Whether a heading is really a heading — the h2 exists, but does it describe the section beneath it?
4. Non-text contrast — which part is the component’s boundary is a question of visual judgment; most tools do not test this criterion.
5. Whether the focus order makes sense — the order can be technically valid and still meaningless to the user.
6. Whether the error message helps — the message exists and is bound, but does it tell the user what to do?
7. The correctness of keyboard contracts — the component works with the keyboard, but does it offer the expected key behavior?
8. The discomfort of motion — the animation exists; no rule measures whether it produces vestibular distress.

What this list shares is striking: everything passes the “does it exist” question and fails at “does it work.” Automated auditing measures presence; it cannot measure sufficiency. And most accessibility defects are born not of something missing but of something insufficient.

A concrete case

An example that happened on this site and is on record teaches more than abstraction. In one round the accessibility picture was: Lighthouse accessibility score 96, zero axe violations. A site in good shape by every indicator.

At the same time, the only visual boundary of the text fields in the contact form — the site’s single conversion path — measured 1.29:1 contrast against the ground. WCAG 1.4.11’s threshold is 3:1. Where a form field began and ended was unclear even for a fully sighted user — on the site’s most critical screen.

The root cause was not a tool error. Axe’s contrast rule measures TEXT against its own background, and in that measurement everything was flawless. 1.4.11 asks a different question — it concerns the component’s boundary — and the tool was not asking it. The gap came from not checking which question the tool asks.

The score was high because the score was not asking that question.

Layered auditing

The right approach is building layers instead of trusting one tool or one method. Each layer catches a different class of defect, and none replaces another.

1. Layer 1 — the automated rule engine. On every deployment, in seconds. Catches: missing attributes, invalid declarations, numeric threshold violations.
2. Layer 2 — custom guards. Project-specific tests freezing measured decisions. Catches: the questions the tool does not ask but you know.
3. Layer 3 — the keyboard tour. Manual, ten minutes. Catches: focus order, focus visibility, traps, broken components.
4. Layer 4 — the screen reader tour. Manual, on the critical screens. Catches: meaningfulness, announcements, navigability.
5. Layer 5 — the real user. The most valuable and the least done. Catches: everything the other four never thought of.

The second layer is the most skipped and the highest-yield. When a defect is found by hand, fixing it removes it once; writing a test that measures it prevents its return permanently. That is exactly what was done in the form case above — the border was fixed and a guard was added.

The general lesson fits one sentence: the most dangerous state in accessibility auditing is not a defect that cannot be found but a defect that is never SOUGHT. A green report is a beginning, not a conclusion.

When the score becomes a target

Automated audit scores carry a second danger, and it concerns not the measurement itself but how it is used. When a score becomes an internal target — “our accessibility score must stay above 95” — the team starts raising the score, not the accessibility.

The two usually move together, but where they diverge, the score wins. Writing a meaningless but present alternative text raises the score and helps no user. Hiding a component from the accessibility tree silences the warnings about it and makes things worse for the user.

The score should therefore be used as an indicator, not a target. The right target is that the human-judgment part of the checklist has been walked too — and that has no score, only a record.

Knowing which question the tool asks

The practical summary of this page condenses into one habit: when a tool passes a rule, look at what that rule measures EXACTLY. Does the contrast rule measure the text, or the boundary too? Does the alternative text rule check presence, or content too? Does the focus rule check focusability, or visibility too?

The answers are written in the tools’ documentation, and reading it takes half an hour. That half hour teaches permanently what the report proves and what it does not — and a green report is never again loaded with more meaning than it carries.

Then comes the second step: turning the questions the tool does not ask but you know into your own tests. Every measured project decision — a contrast value, a target size, a flow rule — can become a guard. Those guards close the gap no general tool will ever cover.