Accessibility Testing Automation
Automatically scan web pages for common accessibility violations (missing alt text, colour contrast failures, missing form labels). Fast and reliable for catching mechanical issues. Automate to catch low-hanging fruit; test manually for the rest.
1 The Hook
A bank ships a redesigned login page for its internet banking. Before release, the team runs axe in CI. Zero violations. Green tick. Everyone relaxes — the page is “accessible.”
A fortnight later a customer who uses a keyboard files a complaint: he cannot log in. The new “remember this device” control is a custom toggle that looks perfect and has flawless ARIA — role="switch", a proper label, the lot. But the developer forgot to make it focusable and never wired up Space to flip it. axe checked the markup and found nothing wrong, because the markup is correct. What is broken is the behaviour, and a rules engine that reads the DOM cannot press Tab.
This is the trap with automated accessibility testing: a green scan feels like a pass, but the tools only catch the mechanical, code-detectable issues — missing alt text, low contrast, duplicate IDs. They cannot tell you whether a keyboard can reach the control or whether a screen reader makes sense of it. Roughly 30–40% of accessibility problems are even visible to them. Treat the green tick as “the obvious stuff is clean,” not “this is accessible.”
The failure mode I keep seeing on government projects — Revenue NZ, MBIE, TransitNZ — is not that teams skip scanning. It is that they scan static HTML before JavaScript executes. CI spins up the app, axe runs against the server snapshot, reports zero violations, and everyone celebrates. Then the real browser loads, React mounts two hundred components, ARIA attributes get set dynamically, and the actual DOM is nothing like what the scanner saw. You have a clean report and a broken product. Always run your accessibility scan against a live Playwright or Puppeteer session after full JS execution. Static-HTML scanning is not accessibility testing — it is accessibility theatre.
2 The Rule
Automate the mechanical, code-detectable checks on every build to catch regressions fast — but a clean scan only proves the obvious issues are gone, never that the page is accessible, so always pair it with manual keyboard and screen-reader testing.
3 The Analogy
A spell-checker on a job application.
Run your cover letter through a spell-checker and it will catch every typo and misspelling — fast, reliable, and worth doing. But a clean spell-check does not mean the letter is good. The spell-checker cannot tell you the argument is weak, the tone is wrong for the role, or you have addressed it to the wrong company. It checks spelling, not sense.
Automated accessibility tools are the spell-checker. They catch the mechanical errors — missing alt text, bad contrast, malformed ARIA — brilliantly and on every build. But “zero violations” is “no typos,” not “this is a good letter.” Whether a keyboard user can actually log in to Harbour Bank internet banking is the part only a human reader catches.
What automated accessibility testing can catch
Automated accessibility testing uses rules engines to analyse page structure and DOM attributes, flagging violations against WCAG 2.1 criteria. Tools scan for mechanical issues that can be reliably detected by code:
- Missing or empty alt text on images
- Colour contrast failures (foreground/background ratio below 4.5:1 for normal text)
- Missing labels on form inputs
- Heading hierarchy violations (h1 jumps to h3, skipping h2)
- Duplicate ID attributes on the page
- ARIA misuse (invalid roles, incorrect attribute values)
- Empty buttons or links (no accessible name)
- Images used as buttons without alt text or ARIA labels
- Form fields without visible labels
- Page language not declared (
langattribute missing on<html>)
These are the low-hanging fruit: high-confidence violations that are straightforward to fix.
The automation limit: Automated tools catch about 30-40% of accessibility issues. The rest require human judgment. A page might have perfect ARIA structure but be unusable with a keyboard or screen reader. Automation finds the bugs code can detect; testing finds the bugs only humans can find.
Tools in the automated accessibility ecosystem
| Tool | Format | Key strength | Best for |
|---|---|---|---|
| axe DevTools | Browser extension + API | Widely used, excellent explanations and remediation guidance, integrates into test frameworks | First line of automated testing; CI/CD pipelines; starting point for teams new to accessibility |
| Pa11y | CLI + API, Node.js | Scriptable, reports to JSON, easy to integrate into CI/CD, supports multiple reporters | CI/CD pipelines, bulk scanning many URLs, teams that want full automation control |
| WAVE | Browser extension, also standalone API | Visual overlays directly on the page, colour-coded icons for errors/warnings/structure | Manual review by testers, visual scanning, design team collaboration |
| Lighthouse | Built into Chrome DevTools, CLI, API | Bundled with Chrome, runs as part of broader performance/quality audit, free | Integrated into test suites, part of overall quality gates, CI/CD that needs accessibility + performance |
Common issues automated tools find (and how to fix them)
Missing alt text
Issue: <img src="button.png"> with no alt attribute. Screen reader announces "image" with no context.
Fix: Add descriptive alt text: <img src="button.png" alt="Close dialog">. For decorative images, use empty alt: <img src="divider.png" alt="">.
Colour contrast failure
Issue: Grey text (#777) on light grey background (#EEEE) has a contrast ratio of 2.1:1. WCAG requires 4.5:1 for normal text.
Fix: Darken the text or lighten the background. The tool reports the current ratio and required ratio, so designers know exactly what to change.
Missing form label
Issue: <input type="email"> with no associated <label>. Screen reader doesn't announce what the field is for.
Fix: Add a label: <label for="email">Email</label><input id="email" type="email">. The for attribute must match the input's id.
Heading hierarchy broken
Issue: Page has h1, then h3, then h2. The outline is confusing to screen reader users navigating by heading.
Fix: Make headings descend in order: h1, h2, h3. If you need bigger text, use CSS, not heading tags.
ARIA misuse
Issue: <div role="button" aria-pressed="true"> but the element isn't actually focusable by keyboard or activatable by Enter/Space.
Fix: Use a real <button> tag. If you must use ARIA, ensure keyboard handling and focus management are implemented correctly. (Better: use native HTML.)
What automation can't catch
- Keyboard navigation: An element might have perfect ARIA markup but not be reachable by Tab. Automation can't tell; manual testing can.
- Screen reader behaviour: How does a screen reader actually announce this component? Automation doesn't run screen readers; only human testers using NVDA or JAWS can verify.
- User experience for disabled users: Is the experience actually usable? Is the page understandable? These require human judgment and preferably testing with real users who have disabilities.
- Context and meaning: An image of a chart with alt text "Chart" is technically labelled but not actually accessible. Only a human can judge if the alt text is descriptive enough.
- Dynamic content updates: A loading spinner appears, then results load. Did the page announce the update to screen readers? Automation can check for
aria-liveregions but can't verify they actually work as intended.
Integration in CI/CD pipelines
Automated accessibility testing fits naturally into CI/CD. On every build:
- Start a fresh instance of the application.
- Run the automated accessibility scanner on critical pages.
- Report violations (errors, warnings, or notices).
- Fail the build if errors are found (optional: allow warnings to pass).
- Archive the report as a build artifact.
Example with axe in Selenium/WebDriver tests
// JavaScript + WebDriver
const results = await runAccessibilityAudit(page);
if (results.violations.length > 0) {
const errors = results.violations
.filter(v => v.impact === 'critical' || v.impact === 'serious');
if (errors.length > 0) {
throw new Error(`Accessibility violations found:\n${errors.map(e => e.id).join(', ')}`);
}
}Example with Pa11y in CI/CD
// pa11y.json config
{
"runners": ["axe"],
"chromeLaunchConfig": {},
"timeout": 10000,
"wait": 1000
}
// In GitHub Actions
- name: Run Pa11y scan
run: |
pa11y http://localhost:3000 \
--runner axe \
--standard WCAG2AA \
--json > a11y-report.json
- name: Check for violations
run: |
if grep -q '"level":"error"' a11y-report.json; then
echo "Accessibility errors found"
exit 1
fiWorked example: axe integration in an automated test suite
Scenario: You want every test to check for accessibility violations. If a functional test passes but accessibility fails, the test fails overall.
// Jest + Puppeteer + axe-core
const { axe, toHaveNoViolations } = require('jest-axe');
describe('Checkout flow', () => {
it('submits order and is accessible', async () => {
await page.goto('http://localhost:3000/checkout');
// Perform the action
await page.click('input[name="email"]');
await page.type('input[name="email"]', 'test@example.com');
await page.click('button[type="submit"]');
// Wait for success state
await page.waitForNavigation();
// Check accessibility
const results = await axe(page);
expect(results).toHaveNoViolations();
});
});Manual testing as the complement
After automated tests pass, manual testing fills the gaps:
- Tab through the page. Can you reach every interactive element with Tab? Is the focus order logical?
- Use a screen reader. Open NVDA (Windows) or VoiceOver (macOS) and navigate the page. Does the structure make sense? Are buttons and form fields announced clearly?
- Test at 200% zoom. WCAG requires content to remain usable at 200% zoom. Does the layout reflow correctly?
- Test with browser zoom and Windows high contrast mode. Do colours remain readable? Do images still make sense?
- Verify error handling. When you trigger an error (invalid email, missing field), is the error message clear and linked to the field?
WCAG 2.1 coverage: what automation addresses
Automated tools address a subset of WCAG 2.1 criteria. The breakdown varies by tool, but roughly:
- Perceivable (images, colour, captions): ~60% coverage. Automation catches missing alt text and colour contrast. It doesn't verify if captions are accurate or if text is readable.
- Operable (keyboard, timing, seizures): ~25% coverage. Automation flags missing focus indicators in code, but can't verify keyboard navigation works in practice.
- Understandable (language, labels, errors): ~40% coverage. Automation checks for language tags and form labels, but can't judge if language is simple enough or if error messages are helpful.
- Robust (HTML validity, ARIA correctness): ~85% coverage. Automation is excellent at catching malformed ARIA and invalid HTML.
This is why automation is layer one, not the whole solution. You need human testing to cover the gaps.
Context guide
How the right level of accessibility automation effort changes based on project context.
| Context | Priority | Why |
|---|---|---|
| Public-sector portal (Benefits NZ, FamiliesNZ, NZ Police) subject to NZ Web Accessibility Standard 1.2 | Essential | Legal obligation under NZWAS 1.2 (WCAG 2.2 AA); organisations must publish an accessibility statement. Automation is the baseline regression net — without it, violations accumulate silently across releases. |
| CoverNZ or HealthNZ patient-facing digital service used by people with disabilities or recovering from injury | Essential | User base has elevated likelihood of relying on assistive technology. An inaccessible claims or appointment portal directly excludes people from services they are entitled to; this is both a legal and ethical obligation under the Human Rights Act 1993. |
| Revenue NZ myIR or TransitNZ licence renewal — high-traffic transactional web app with frequent UI changes | High | Frequent releases mean constant risk of regression. CI-integrated axe or Pa11y catches a newly introduced contrast failure or missing label the moment it is merged — before it reaches millions of New Zealanders who have no alternative channel. |
| Harbour Bank or Pacific Bank internet banking — private sector with a large, diverse customer base | High | Reputational and regulatory risk if blind or keyboard-only customers cannot access their accounts. Privacy Act 2020 obligations around online account management reinforce the expectation of full digital access. |
| Pacific Air or Spark internal staff tool — authenticated portal for employees, not publicly accessible | Medium | NZWAS 1.2 applies to public-sector sites, not private internal tools, but employment law requires reasonable workplace adjustments. Run automated scans to avoid discriminatory barriers for staff with disabilities; manual testing is warranted when roles specifically involve staff who use assistive tech. |
| Native iOS/Android app (e.g. Pacific Bank mobile, Pacific Air app) — no browser DOM layer | Low | DOM-based tools (axe, Pa11y, Lighthouse) do not apply to native apps. Use platform-specific methods: AccessibilityInspector on iOS, Accessibility Scanner on Android. Web-based accessibility automation is irrelevant here. |
Trade-offs
What you gain and what you give up when you choose accessibility automation.
| Advantage | Disadvantage | Use instead when… |
|---|---|---|
| Catches regressions instantly on every PR — a newly broken contrast ratio or missing label is flagged before code review, not after it ships to millions of Revenue NZ taxpayers. | Only covers ~30–40% of WCAG criteria. Keyboard navigation, focus management, meaningful screen reader announcements, and cognitive accessibility are invisible to any rules engine. | You need to confirm the experience is actually usable — run a keyboard-only pass and a screen reader walkthrough with NVDA or VoiceOver. |
| Produces objective, repeatable evidence — JSON reports with rule IDs, severity, and element selectors attach directly to defects and to an accessibility statement under NZWAS 1.2. | Report quality degrades when the scan runs against static HTML before JavaScript executes. Dynamic ARIA attributes, conditionally rendered fields, and aria-live regions are all invisible to a DOM snapshot. | You are producing a formal WCAG conformance statement for a regulatory submission — automated output alone is insufficient; a human evaluation is required. |
| Scales to large sites cheaply — a Pa11y script can audit all 300 pages of a TransitNZ site overnight, producing a systematic first-pass report far faster than any manual review. | Scanning too many pages in CI slows the pipeline and generates noise, which erodes team trust in the gate. Misconfigured severity thresholds (failing on notices) train teams to suppress or ignore alerts. | The component under test is a third-party iframe (e.g. an embedded payment widget on an Harbour Bank form) — the scanner cannot reach inside the embedded document and will give false confidence. |
| Low barrier to entry — axe DevTools is free, ships with Chrome, and requires no test-framework setup. Any tester can run it on a page in 30 seconds, making it a practical first step for teams new to accessibility. | A green scan creates a false sense of completion. Teams that equate "zero violations" with "accessible" routinely ship products that fail keyboard users — which is exactly what happened on an NZ bank's login page redesign (the Hook example). | You are testing a native iOS or Android app — use AccessibilityInspector or Accessibility Scanner; DOM-based tools do not apply. |
Enterprise reality
How accessibility automation changes at 200–300-developer scale in NZ
- At this scale you stop running axe ad hoc and centralise it: a shared Pa11y configuration pinned to WCAG 2.2 AA is maintained by the QA chapter and embedded in every squad’s pipeline template. Squads do not configure their own scanners — they inherit the standard, which means a contrast-token change in the design system instantly fails every downstream pipeline simultaneously, not just the one that introduced it.
- Governance matters at this scale: Harbour Bank and Pacific Bank both operate under the Privacy Act 2020 and the Responsible Lending Code, and any customer-facing channel that discriminates against users of assistive technology creates legal exposure under the Human Rights Act 1993. The practical outcome is a formal Accessibility Register — a tracked record of every known non-conformance, its severity, the squad that owns it, and the remediation ETA — reviewed quarterly by the CISO’s office alongside security findings, not buried in a testing backlog.
- Tooling at volume means automated scans alone cannot keep pace: a 300-page TeleNZ MyAccount portal or an Revenue NZ myIR redesign will surface thousands of violations on first run. Experienced enterprise QA teams triage by component rather than by page — a broken heading hierarchy in the shared
PageHeadercomponent appears on 200 pages but is one fix. Suppression files (“baselines”) track accepted exceptions for third-party widgets the team cannot modify, with a named owner and a review date so they are not silently inherited by the next engineer. - Across 10+ squads the coordination problem is social, not technical: four different squads may own overlapping UI components with conflicting ARIA patterns. Enterprise teams solve this with an Accessibility Guild — one representative per squad, meeting fortnightly to align on patterns, review the Accessibility Register, and decide which violations warrant a platform-level fix versus a squad-level one. Without this, squads fix the same violation in different (and sometimes contradictory) ways and the product experience is inconsistent for screen-reader users even when every individual pipeline is green.
◆ What I would do
Professional judgement — when to reach for accessibility automation, when to skip it, and what to watch for.
The bottom line: Automated scanning is the regression net for the mechanical 30–40% — wire it in and leave it running. But on any NZ site where a real person with a disability might use the service, treat the keyboard pass and screen reader check as non-negotiable. The tools tell you the markup is correct; only a human can tell you whether the product actually works.
Best practices and anti-patterns
Automate the mechanical checks; test the rest manually. Don't wait for perfect automation coverage. Run axe or Pa11y on every build to catch regressions, then add manual accessibility testing at key points (new feature launch, design change, before release).
- Fail the build on critical violations only. Many tools report errors, warnings, and notices. Set CI to fail on "error" or "serious" severity, but allow warnings to pass (and track them separately). This prevents false-positive build failures from minor issues.
- Review violations with a purpose. Don't just auto-approve every fix. Have a designer or accessibility expert review significant changes to ensure fixes are actually accessible, not just compliant.
- Test representative pages, not every page. Scanning 500 pages is expensive and noisy. Focus on critical pages (homepage, checkout, forms) and representative component instances (button, input, dropdown).
- Pair automation with manual keyboard testing. Keyboard navigation is the most impactful manual test. Spend 10 minutes tabbing through every page and checking focus order. This catches more real-world issues than any automation.
- Run automated checks early and often. Accessibility is cheaper to build in than to fix later. Run axe during development (IDE integration, local pre-commit hook) before code review.
- Include accessibility in your definition of done. No feature is done if it doesn't pass automated scans and manual keyboard checks. Make it a checklist item.
4 Industry Reality
- Most teams treating a green axe scan as a sign-off tick — no one runs NVDA, no one presses Tab. When a complaint arrives (often from a screen-reader user or disability advocate), the team is surprised, because “the CI was green.” You will need to push for a definition of done that includes manual checks, not just automation.
- Legacy codebases drowning in violations. Running axe for the first time on a 10-year-old government portal can produce hundreds of errors. Senior testers triage these by severity and business impact, not try to fix everything at once: focus first on WCAG 2.1 AA critical-path failures (login, forms, checkout) and treat the rest as a tracked backlog.
- Accessibility exceptions and documented non-conformances. Under the New Zealand Web Accessibility Standard 1.2 (which applies to public-sector sites), organisations must publish an accessibility statement. In practice, many teams carry documented exceptions for known issues they haven’t yet resourced. You will track these in a register, not just suppress them in the scanner config.
- Developers who argue a passing WAVE screenshot proves a third-party widget is accessible. Real-world widgets — date pickers, rich text editors, modal dialogs from component libraries — are notorious sources of keyboard and ARIA problems. Your job is to test those components manually, not accept the vendor’s accessibility statement at face value.
- Time pressure collapsing accessibility down to “run Lighthouse and see if the score is above 90.” The Lighthouse accessibility score bundles many checks into a number that looks reassuring but masks gaps. Push back: a score of 92 can coexist with a broken keyboard flow and a completely uninformative form error. The number is a conversation-starter, not a pass.
5 When to Use It — and When Not To
✓ Use it when
- You have a web or hybrid-mobile application where HTML/DOM is the rendering layer — automated scanners need the DOM to run against.
- You need to prevent regressions on a codebase that changes frequently; CI-integrated axe or Pa11y will catch a newly introduced contrast failure or missing label the moment it is merged.
- Your team is new to accessibility and needs a baseline of quick wins; starting with automated scanning surfaces the high-confidence, easily explained violations first, building the team’s understanding before manual testing is introduced.
- You are auditing a large site (50+ pages) for an initial accessibility statement under the NZ Web Accessibility Standard; a scripted crawler with Pa11y produces a systematic first-pass report far faster than manual review alone.
- You want an objective, repeatable measure in a test report — automated results log the rule ID, severity, element selector, and remediation guidance, making them straightforward to attach to a defect.
✗ Skip it when
- You are testing a native mobile app (iOS/Android), a desktop application, or a PDF document; DOM-based accessibility tools do not apply and you need platform-specific methods (XCTest/AccessibilityInspector on iOS, Accessibility Insights for Windows, tagged-PDF validation).
- You need to verify the actual user experience for someone using a screen reader or keyboard only — automation cannot tell you whether the content is understandable, the focus order is logical, or announcements make sense. Use manual testing for those questions.
- You are validating compliance with an accessibility standard for a legal or regulatory submission — automated results alone are not sufficient evidence; a conformance report (VPAT or similar) requires manual evaluation and human judgement.
- The component under test is a third-party iframe or widget your code cannot inject into; the scanner will report missing context from the host frame but cannot analyse the embedded document, giving you false confidence.
- You are treating it as a replacement for involving disabled users in usability testing; no tool catches whether the overall experience is actually usable.
6 Best Practices
- ✓ Fail the build on serious/critical only. Set your CI gate to error and serious/critical severity; track warnings and notices in a separate accessibility backlog ticket. This keeps the gate trustworthy and avoids training the team to ignore it.
- ✓ Scan representative pages, not every page. Pick the homepage, the authentication flow, the primary form, the checkout, and one representative of each reusable component. Scanning all 300 pages slows the pipeline and produces noise; scanning the right 10 gives you 90% of the signal.
- ✓ Run scans as early as the developer machine. Add the axe DevTools browser extension to your browser and the axe ESLint plugin to your editor. Catch contrast failures and missing labels before you even push — far cheaper than finding them in CI after a pull request review.
- ✓ Archive scan reports as build artifacts. Every CI run should save the JSON/HTML accessibility report so you can track whether the violation count is trending up or down over time, and have evidence for an accessibility statement.
- ✓ Triage violations by impact before logging defects. axe reports four impact levels: critical, serious, moderate, minor. File separate defects for each critical/serious violation with the element selector and remediation steps copied from the tool output — it halves the developer’s time to fix.
- ✓ Use baseline files for known accepted violations. Both axe and Pa11y support configuration files that suppress known, accepted violations (e.g., a third-party chat widget you cannot modify). Document the reason in the config comment so future testers know it is deliberate, not an oversight.
- ✓ Always follow up automation with a manual keyboard pass. After the automated scan passes, spend 10 minutes tabbing through the critical path. Keyboard navigation failures are the most common real-world accessibility blocker and the one automation cannot see.
- ✓ Test in the browser, not just against static HTML. JavaScript renders much of the DOM; run your scanner against a live running application (or Playwright-rendered page), not a static HTML file, or you will miss dynamically injected content and ARIA state changes.
- ✓ Include accessibility acceptance criteria in your user stories. If the story says “add a date-picker component,” an explicit AC of “operable by keyboard, announced correctly by NVDA” gives developers a clear target and gives you a testing anchor beyond running axe.
- ✓ Keep a NZ-specific compliance note in your accessibility statement. Public-sector New Zealand websites must meet the NZ Web Accessibility Standard 1.2 (based on WCAG 2.1 AA). Reference the standard explicitly when writing defect reports and test summaries so stakeholders understand the regulatory basis, not just the tool output.
7 Common Misconceptions
❌ Myth: A zero-violation axe scan means the page is accessible.
Reality: Automated tools catch roughly 30–40% of WCAG issues — the mechanical, code-detectable ones. A clean scan means the obvious structural errors are gone. It says nothing about whether a keyboard user can reach every control, whether a screen reader announces dynamic updates meaningfully, or whether alt text is genuinely descriptive versus technically present. A green scan is the floor, not the ceiling.
❌ Myth: A Lighthouse accessibility score above 90 means you can skip manual testing.
Reality: The Lighthouse score is a weighted aggregate of automated rule results. It can read 92 while the site has a broken Tab order, an uninformative ARIA live region, and a modal that traps keyboard focus. The score is a rough signal for stakeholder conversations — not a substitute for a keyboard pass or screen-reader test. Teams that gate on the score alone routinely ship inaccessible products.
❌ Myth: Running Pa11y once before release is sufficient accessibility testing.
Reality: Accessibility regressions are introduced constantly as the UI evolves — a contrast-ratio change in a design token, a new modal that breaks focus trapping, a refactored button that loses its ARIA label. Running a one-off scan before release catches violations that exist at that moment, but misses regressions introduced in the next sprint. Integrating scanning into every PR check turns it from a release gate into a regression net.
8 Now You Try
Three graded exercises — spot, fix, then build. Write your answer, run it for AI feedback, then compare to the model answer.
A RealMe-linked form has the six issues below. For each, say whether an automated tool (axe / Pa11y / Lighthouse) would reliably catch it or whether it needs manual testing, and why. (a) an image with no alt; (b) the submit button cannot be reached by Tab; (c) body text at 3:1 contrast; (d) alt text that just says “image”; (e) a duplicate id on two inputs; (f) a loading result that never gets announced to a screen reader.
Show model answer
(a) No alt — AUTO. A missing alt attribute is a high-confidence, code-detectable violation. (b) Button not reachable by Tab — MANUAL. Whether keyboard focus actually reaches an element is behaviour a DOM scan cannot determine; you have to Tab through it. (c) 3:1 contrast — AUTO. Contrast ratio is computed from the foreground/background colours; below 4.5:1 for normal text is flagged reliably. (d) alt="image" — MANUAL (mostly). The alt attribute is present, so the tool sees no violation; only a human can judge that "image" is meaningless and not descriptive. (e) Duplicate id — AUTO. Duplicate IDs are a mechanical HTML-validity issue tools catch every time. (f) Result not announced — MANUAL. A tool can check whether an aria-live region exists, but cannot verify it actually announces the update as intended; that needs a screen reader. The pattern: tools catch what is in the markup; humans catch what only happens at runtime or requires judgement.
A KiwiSaver provider added an accessibility gate to CI, but it is set up badly. Describe what is wrong and how you would fix it.
• Scans all 500 pages on every commit (~25 min, often times out).
• Fails the build on any issue, including notices and warnings.
• Team treats a green scan as “accessibility done” — no manual testing ever happens.
• The scan only runs nightly, long after code review.
What is wrong and how to fix it:
Show model answer
Problem 1 (scope): scanning all 500 pages every commit is slow, flaky, and noisy. Fix: scan representative pages (homepage, checkout, key forms) and representative component instances; reserve a full crawl for nightly runs. Problem 2 (severity): failing on any issue including notices/warnings causes false-positive build failures and trains people to ignore the gate. Fix: fail the build only on "error"/"serious"/"critical" severity; track warnings separately rather than blocking on them. Problem 3 (false confidence): a green scan is NOT "accessibility done" — tools catch only ~30-40% of issues. Fix: keep automation as layer one and add manual keyboard + screen-reader testing at key points (new feature, design change, before release); put both in the definition of done. Problem 4 (timing): nightly-only feedback arrives long after the code is written and reviewed, when it is expensive to fix. Fix: run the scan early and often — locally / pre-commit and in the PR check — so regressions are caught before review. The corrected pipeline: fast scan of critical pages on every PR, fail on serious+ only, manual checks gated into the definition of done, full crawl nightly.
A TransitNZ team is starting accessibility testing on a new licence-renewal app from scratch. Design a layered strategy that combines automation and manual testing. State which tool/check runs at each layer (developer machine, PR/CI, pre-release), what each catches, and how you stop the team mistaking a green scan for full accessibility.
Show model answer
A layered strategy for the licence-renewal app: Layer 1 (developer machine) — axe DevTools browser extension and/or an editor/pre-commit check. Catches mechanical issues (missing alt, contrast, missing labels, bad ARIA, duplicate IDs) before code is even pushed — cheapest place to fix. Layer 2 (PR / CI gate) — Pa11y or axe-core run on representative pages in the pipeline. Catches regressions on every change. Fails the build only on "error"/"serious"/"critical" severity; warnings tracked but not blocking. Archive the report as a build artifact. Layer 3 (pre-release manual) — keyboard-only pass (Tab through everything, check focus order and ring), a screen-reader pass with NVDA + Chrome on the main renewal flow, a 200% zoom check, and error-message verification. Catches the ~60-70% of issues automation cannot: keyboard reachability, focus management, meaningful announcements, whether alt text is actually descriptive. How you prevent "green = done": make explicit that a clean scan only means the mechanical issues are gone; require the manual keyboard + screen-reader checks as sign-off items, not optional extras. Definition of done items: automated scan passes (serious+), keyboard navigation works end to end, screen-reader pass on the core flow, usable at 200% zoom, te reo Maori tagged with lang="mi". The shape: automate early and often for speed and regression safety; gate manual testing into release for the judgement-based issues.
Senior engineer insight
The shift that changed how I approach this: stop thinking of axe as a test and start thinking of it as a linter. You wouldn't ship code that fails ESLint, and you wouldn't ship code that fails axe — but passing ESLint doesn't mean the code is correct, and passing axe doesn't mean the page is accessible. Once your team internalises that framing, the pressure to treat a green scan as sign-off evaporates. On NZ government contracts I've seen this distinction matter enormously: teams that treat automation as a quality floor consistently ship better products than those who treat it as a quality ceiling.
The most common mistake: running the accessibility scan against a server-rendered HTML snapshot before JavaScript executes. Half your ARIA attributes don't exist yet. The scan looks clean, the live product is broken, and nobody can explain the gap.
From the field
A Wellington team delivering a public-sector benefits portal under the NZ Web Accessibility Standard 1.2 had axe-core wired into their GitHub Actions pipeline from day one — zero violations on every PR for six months. During a pre-launch accessibility audit, the auditor opened NVDA, navigated to the dynamic income-declaration form, and within three minutes found that the conditional field set revealed after a "Yes" answer was never announced to the screen reader. The aria-live region existed in the DOM — axe had confirmed it. But the JavaScript injecting content into it was writing to a detached node, so the announcement never fired. The tool saw the correct markup; the screen reader experienced silence. We added a mandatory screen-reader smoke test to the definition of done for every feature touching dynamic content. The lesson that generalises: if your form has any conditional logic, assume automation has blind spots there and test it with an actual screen reader before every release.
Why teams fail here
- Treating a zero-violation axe scan as full sign-off — automation catches 30–40% of issues; the rest require a keyboard pass and screen-reader test that never gets scheduled.
- Scanning static HTML instead of a live Playwright/Puppeteer session — dynamically injected ARIA attributes and state changes are invisible to a DOM snapshot, producing false confidence on the exact patterns most likely to break.
- Failing the build on every severity level including notices — this trains the team to ignore the gate or suppress warnings wholesale, eroding the tool's value within weeks.
- Accepting a Lighthouse score above 90 as a compliance statement — the aggregate score masks individual failures; a 94 can coexist with a broken keyboard flow, which is legally significant under the NZ Web Accessibility Standard 1.2 for public-sector sites.
Key takeaway
Automated accessibility scanning is a regression net for the mechanical, code-detectable 30–40% of issues — wire it into every build, but never mistake a green scan for a green light.
How this has changed
The field moved. Here is how Accessibility Automation evolved from its origins to current practice.
Accessibility testing is entirely manual — screen reader walkthroughs, keyboard navigation checks, contrast ratio measurement. No automated equivalent exists. Specialist testers with disability experience are rare and expensive.
axe-core open-sourced by Deque Systems. The first programmatic WCAG checker that is accurate enough for CI use. Teams begin adding axe scans to their build pipelines — catching about 30-40% of WCAG violations automatically.
Google Lighthouse integrates accessibility scoring. Browser DevTools adds accessibility panels. Pa11y and axe-playwright enable accessibility assertions in end-to-end tests. CI accessibility gates become achievable for any team.
WCAG 2.1 and 2.2 drive new automated checks — reflow, focus appearance, target size, non-text contrast. Automated coverage improves to ~57% of WCAG criteria. The remaining 43% require human judgement: cognitive accessibility, plain language, task flow.
AI tools can describe images, suggest alt text, and flag missing captions — expanding coverage into content accessibility. But automated tools still cannot test with real assistive technology or judge cognitive accessibility. NZ Web Accessibility Standard 1.2 requires WCAG 2.2 AA; automation is a gate, not a replacement.
Self-Check
Click each question to reveal the answer.
Interview Questions
What NZ hiring managers ask about Accessibility Automation — and what strong answers look like.
What automated accessibility checks can you include in a CI pipeline, and what percentage of WCAG issues do they typically catch?
Strong answer: Tools like axe-core, Lighthouse, and Pa11y can be embedded in CI to catch around 30–57% of WCAG violations automatically — things like missing alt text, insufficient colour contrast ratios, missing form labels, and empty button text. The remaining 43% require human judgement: keyboard navigation flow, screen reader announcement quality, cognitive load, and task completion with assistive technology. A CI accessibility gate prevents regressions but does not replace manual and assistive-technology testing.
Junior/Mid
A developer says our 98% Lighthouse accessibility score proves the site is accessible. How do you respond?
Strong answer: I explain that automated scores measure what automation can measure — about a third of WCAG criteria. Lighthouse cannot test whether content is understandable when read by a screen reader, whether the focus order makes sense for keyboard-only users, or whether error messages are meaningful in context. A 98% score means "no automated violations found," not "accessible." I would complement it with manual keyboard testing, a screen reader walkthrough of key flows, and testing with actual users who rely on assistive technology.
Senior/Lead
What is the difference between aria-label and aria-labelledby, and when would you test each?
Strong answer: aria-label provides a text alternative directly in the attribute: useful when there is no visible text to reference. aria-labelledby points to another element's ID to use its text as the label: useful when visible text already exists and should serve as the accessible name. I test aria-label by checking that the screen reader announces the label text instead of the element's content or role. I test aria-labelledby by verifying the referenced element exists, has the correct text, and is announced correctly. I also check that programmatic labels match visible labels — screen reader users and sighted users should get equivalent information.
Junior/Mid
Walk me through how you would test a custom dropdown component built with divs for accessibility.
Strong answer: First I check the component has role="combobox" or role="listbox" with appropriate aria attributes — aria-expanded, aria-haspopup, aria-activedescendant. Then I test keyboard: Tab to focus it, Enter/Space to open, arrow keys to navigate options, Enter to select, Escape to close. I check that focus does not escape the dropdown when open and returns to the trigger on close. I test screen reader announcement: option count, current selection, and state changes. Finally I verify that colour contrast for selected state, focus indicator, and placeholder text all meet WCAG 2.2 AA minimums.
Mid/Senior
Q1: Roughly what share of accessibility issues can automated tools detect, and what does a clean scan actually prove?
About 30–40%. A clean scan proves the mechanical, code-detectable issues are gone — missing alt text, contrast, malformed ARIA, duplicate IDs. It does not prove the page is accessible, because the majority of issues need human judgement or runtime behaviour to surface.
Q2: Give an example of a defect with perfect markup that automation still cannot catch.
A custom toggle with flawless ARIA (role="switch", a proper label) that is not focusable and has no keyboard handler. The markup is correct, so a DOM scan finds nothing, yet a keyboard user cannot operate it. Only tabbing through it manually reveals the bug.
Q3: Why fail the CI build on serious/critical severity only, rather than on every issue?
Failing on notices and warnings produces false-positive build failures from minor issues, which trains the team to ignore the gate. Failing on error/serious/critical keeps the gate trustworthy while warnings are tracked separately and addressed without blocking delivery.
Q4: Why run automated accessibility checks early (locally / on PR) rather than only nightly?
Accessibility is far cheaper to fix while the code is being written than after it has shipped. Early, frequent feedback — in the IDE, pre-commit, and the PR check — catches regressions before code review, instead of surfacing them long after in a nightly run.
Q5: An image has alt="image". Why does automation pass it when it is still a problem?
The alt attribute is present, so the rule (“images must have alt text”) is technically satisfied and the tool reports no violation. But “image” conveys nothing; only a human can judge whether the alt text is actually descriptive of the content.
Q6: Your team is building the Benefits NZ (Benefits NZ) Jobseeker application form. It has a dynamic section that appears after a user answers a question — revealing extra fields without a page reload. Which accessibility checks do you prioritise for this pattern, and why can axe alone not cover them?
The critical checks for dynamically injected content are: (1) whether an aria-live region announces the new section to screen-reader users, (2) whether keyboard focus moves into the revealed fields logically rather than stranding the user on the triggering control, and (3) whether the new fields have correct labels and error messages. axe can confirm that an aria-live region exists in the DOM, but it cannot run a screen reader to verify the announcement fires correctly, nor can it simulate Tab navigation to check the focus sequence. Both require manual testing — a keyboard-only pass and an NVDA/VoiceOver check on the dynamic flow. For a government benefit portal serving vulnerable populations, these are high-priority defects under the NZ Web Accessibility Standard 1.2.
Q7: What is the key difference between accessibility automation (axe / Pa11y) and visual regression testing (Percy / Playwright screenshots), and when would you use both on the same feature?
Accessibility automation analyses the DOM and ARIA structure to find semantic violations — missing labels, contrast ratios, malformed roles — regardless of how the page looks. Visual regression testing compares pixel-level screenshots to detect unintended layout or style changes, but it knows nothing about semantics. They complement each other: a design token change might pass axe (the ARIA is untouched) yet fail a visual regression (the button colour shifted). Conversely, a component refactor might break heading hierarchy and lose an aria-label while the screenshots remain identical. Use both on the same feature when a UI change touches colour, spacing, or interactive components — axe catches the semantic regressions, visual regression catches the rendering regressions. A good example is an Revenue NZ tax summary page redesign: axe would catch a contrast failure in the new colour scheme; visual regression would catch the tax-year table reflowing incorrectly at mobile widths.
Q8: When should you NOT rely on automated accessibility testing, even though you have it wired into CI?
Automated scanning is insufficient in four situations: (1) testing a native iOS or Android app (DOM-based tools do not apply; use AccessibilityInspector or Accessibility Scanner instead); (2) producing a formal WCAG conformance report for a legal or regulatory submission — a VPAT or NZ Web Accessibility Standard statement requires human evaluation, not just tool output; (3) validating third-party iframes or embedded widgets your code cannot inject into (the scanner cannot reach inside the embedded document and will give false confidence); and (4) assessing whether the overall experience is usable for a person with a disability, which requires usability testing with real users and cannot be replaced by any automated rule engine. In each case, name automation’s limit explicitly in your test report so stakeholders do not mistake a green scan for full compliance.
Q9: A developer on an CoverNZ (CoverNZ) claims portal says: “We ran Lighthouse and scored 94 — we don’t need to do any more accessibility work before release.” What is wrong with this reasoning and how do you respond?
The Lighthouse accessibility score is a weighted aggregate of automated rule results. A score of 94 can coexist with a broken Tab order, an ARIA live region that never fires, a modal that traps keyboard focus, and form error messages that are not programmatically associated with the fields that caused them — none of which automated rules can detect. For an CoverNZ claims portal, where claimants may be recovering from injury and are likely to include users of assistive technology, those keyboard and screen-reader failures are high-severity. The right response is to explain that the score is a useful rough signal, not a pass/fail gate, and that it addresses only the ~30–40% of issues tools can see. Propose a short manual keyboard pass and a screen-reader walkthrough of the claim submission flow before release. Frame it in terms of the NZ Web Accessibility Standard 1.2 and the risk of excluding claimants from a service they are legally entitled to access.
Related techniques: Accessibility Testing (WCAG), Visual Regression Testing.