- AODA Training Guide
Manual vs Automated Accessibility Testing: Which Do You Need?
Every AODA compliance audit involves two types of testing: automated scanning and manual evaluation. They are not alternatives — they are complementary. Automated tools, manual testing, and AT testing each find different issues. A credible WCAG audit uses all three.
- 4.6
93% Student Satisfaction (2K ratings)
Quantity Discounts
Price per course
6 to 20 courses
$20.95
21 to 50 courses
$17.95
51 to 200 courses
$13.95
201+ courses
Custom offer
Every AODA compliance audit involves two types of testing: automated scanning and manual evaluation. They are not alternatives — they are complementary. Automated tools find different issues from manual testing, and manual testing finds different issues from assistive technology testing. A credible WCAG audit uses all three.
What automated testing does
Automated accessibility testing uses tools — browser extensions, command-line scanners, or CI/CD integrations — to analyse a page’s HTML, CSS, and ARIA against a set of predefined rules. When a rule is violated, the tool flags it. The process is fast, consistent, and scalable.
What automated tools are good at
- Detecting the absence of required attributes — missing alt attributes on images, missing label elements on form fields, missing document language declarations
- Measuring colour contrast ratios against WCAG minimum thresholds
- Identifying duplicate IDs and other HTML validity issues that affect assistive technology
- Flagging ARIA attribute errors — invalid ARIA roles, missing required ARIA properties, or ARIA used on elements where it has no effect
- Finding missing document titles and missing skip navigation links
- Running consistently and quickly across large numbers of pages without fatigue
What automated tools are good at
Automated tools are binary: a rule either fires or it does not. They cannot evaluate context, intent, or user experience. This creates a class of WCAG failures that are genuinely invisible to automated tools:
- Alt text that is technically present but meaningless ("image.jpg", "photo") or that describes the image without conveying its communicative purpose
- Form labels that are syntactically associated with their field but do not describe the expected input clearly enough
- Heading structures that are syntactically correct but do not reflect the logical outline of the page
- Interactive elements that are keyboard-reachable but whose behaviour makes no sense when operated by keyboard
- Colour used as the only way to convey meaning — automated tools check that text contrast meets the ratio, but cannot assess whether an error state relies on a red border alone
- Dynamic content — modals, dropdowns, live regions — that renders correctly in static HTML but fails in practice when user interaction triggers it
- Accessibility Audit Tools: Full Tool Comparison
What manual testing does
Manual testing is where a trained accessibility specialist evaluates a website against WCAG criteria that cannot be checked algorithmically. This means reading alt text and asking whether it conveys the same information as the image. Navigating a page by keyboard and evaluating whether the experience makes logical sense.
What manual testing covers that automated tools cannot
- Alt text quality — does it convey the meaning and purpose of the image, or just describe it literally?
- Link text meaningfulness in context — would a list of all links on the page help a screen reader user navigate?
- Heading structure logic — does the heading hierarchy reflect the page's actual content structure?
- Keyboard usability — is Tab order logical? Are focus indicators visible and helpful? Are there keyboard traps?
- Form usability without sight — can a user complete the form knowing only what is communicated through text and markup?
- Error message helpfulness — does the error message explain what went wrong and how to fix it?
- Colour and sensory use — is any information conveyed only through colour, shape, sound, or spatial position?
- Time limits — are users warned before a session expires? Can they extend it?
- Reading order in context — does the page make sense when encountered top-to-bottom, which is how screen readers experience it?
What assistive technology testing adds
Assistive technology (AT) testing means using a screen reader to navigate and interact with a website as a blind user would. It is the most revealing form of accessibility testing because it reproduces the actual experience of the users who face the most significant barriers.
AT testing frequently uncovers issues that both automated and manual testing miss:
- Custom interactive components — tabs, accordions, carousels, date pickers — that appear functionally correct in code but produce confusing or broken announcements in a screen reader
- ARIA live regions that are supposed to announce dynamic content updates but do not fire correctly in practice
- Heading structure logic — does the heading hierarchy reflect the page's actual content structure?
- Focus management in modals and dialogs — focus does not move into the modal, or does not return correctly when the modal closes
- The actual reading experience of a page — how the content sounds when read sequentially, which is significantly different from how it looks visually
- Platform-specific failures — a component that works correctly with NVDA may fail with VoiceOver, or vice versa
WCAG criterion coverage: automated vs manual vs AT testing
The table below maps key WCAG 2.0 Level AA criteria to the testing approach that detects them. ✓ = reliably detected, △ = partially detected or context-dependent, ✗ = not detectable by this method.
Automated vs manual: head-to-head
When to use automated testing, manual testing, and AT testing
Building a combined testing process
The most effective accessibility testing programmes use all three approaches in sequence, with each building on the last.
- How to Perform an Accessibility Audit: Full Step-by-Step Process
Frequently asked questions
Is automated testing sufficient for AODA compliance?
- No. Automated testing catches 30–40% of WCAG 2.0 Level AA failures. The remaining 60–70% require manual testing and screen reader evaluation. An AODA compliance audit that relies only on automated tools will miss the majority of real accessibility barriers — including the ones most likely to prevent disabled users from completing core tasks.
How do I know which WCAG failures my automated tool missed?
- There is no automated way to know what automated testing missed — that is the nature of the limitation. The only way to find the issues that automated tools cannot detect is to conduct manual testing and AT testing. A professional accessibility auditor will work through the full set of WCAG success criteria, evaluating each one using the appropriate testing method.
Can a developer with no accessibility training conduct manual testing?
- A developer with WCAG knowledge can conduct useful manual testing — particularly keyboard testing and code review. However, manual testing is most effective when conducted by someone with practical experience evaluating web content for accessibility, including experience using assistive technologies. The gap between a technically correct implementation and one that works in practice for AT users is often only visible to someone who uses AT regularly.
How much does manual accessibility testing cost compared to automated?
- Automated testing tools are typically low cost or free (axe DevTools free, WAVE free, Lighthouse built into Chrome). Manual testing and AT testing require specialist time. A full manual + AT audit of a small website typically costs between $1,500 and $4,000. Automated scanning alone costs a fraction of this but provides a fraction of the compliance picture.
Should I run automated tests before or after manual testing?
- Always run automated tests first. Automated scanning identifies high-volume technical failures quickly. Fixing these before manual testing ensures manual testing time is spent on issues that automated tools genuinely cannot assess, rather than on issues that tools should have caught.
Do I need to test with both NVDA and VoiceOver?
- For a professional AODA audit, yes. NVDA and VoiceOver handle ARIA and HTML differently in significant ways. A component that works correctly with NVDA may fail with VoiceOver, or vice versa. For most Ontario business websites, NVDA with Firefox (Windows) and VoiceOver with Safari (macOS and iOS) cover the most important AT combinations.
Get a full audit — automated, manual, and AT testing combined
Our AODA compliance audits use all three testing methods in sequence: automated scanning with axe and WAVE, manual WCAG testing by a trained specialist, and screen reader testing with NVDA and VoiceOver. The result is a complete picture — not the 30–40% that automated tools alone provide.
- Automated scan with axe DevTools and WAVE across all scoped pages
- Screen reader testing: NVDA + Firefox (Windows) and VoiceOver + Safari (macOS and iOS)
- Prioritized report with WCAG references, severity ratings, screenshots, and code-level fixes
- Manual WCAG 2.0 Level AA testing: keyboard nav, alt text, link text, headings, forms, colour use
- Complete WCAG success criterion coverage — not just what automated tools catch
- Optional post-remediation retesting to confirm issues are resolved correctly
