Do we need to test the accessibility of our application using real devices like screen readers, etc.?

Question

We are trying to be compliant with accessibility standard wcag21aa for which we are using axe. In order for us to be compliant do we need to test them on real devices or can we simply use tools like AXE which will make sure we will be compliant without needing to test it on real time devices?

Answer 1

Short Answer

Automated tools are not sufficient to catch all accessibility issues. You need to test with assistive tech.

Longer Answer

Yes you will need to use a screen reader to test, as well as a screen magnifier, checking captions etc.

Automated tools only pick up about 40% of errors, so this is why automated tools alone will not help you make the site accessible.

Ideally real user testing is the best way, giving them tasks to perform such as purchasing product XXX so they can give feedback for barriers they face. However I know this can be cost prohibitive / hard to organise.

So you should use automated tools to catch the easy problems, then use assistive tech to catch problems such as keyboard traps, unclear labelling etc.

It is worth having a checklist while you are starting out, and ideally you would want an accessibility consultant in the early stages to help guide you through WCAG and (if appropriate) ATAG etc.

A great tool for starting out is Microsoft's Accessibility Insights for Web as it will run an automated test for the things that can be caught automatically, and can then guide you through the manual testing process for nearly every other issue type...and it is free!

One final thing to note, WCAG 2.2 is close to release, so maybe have a look at the WCAG 2.1 to WCAG 2.2 changes so you are "ahead of the curve".