In my previous post, I showed a technique that used axe-core along with pytest and Playwright to make sure that pages in your web apps have no accessibility violations. That's a great approach if it works for you, but realistically, most webpages have a non-zero number of accessibility violations, due to limited engineering resources or dependence on third-party libraries. Do we just give up on being able to test them? No! Instead, we use a different approach: snapshot testing.
Snapshot testing
Snapshot testing is a way to test that the output of a function matches a previously saved snapshot.
Here's an example using pytest-snapshot:
def emojify(s):
return s.replace('love', '❤️').replace('python', '🐍')
def test_function_output_with_snapshot(snapshot):
snapshot.assert_match(emojify('I love python'), 'snapshot.txt')
The first time we run the test, it will save the output to a file. We check the generated snapshots into source control. Then, the next time anyone runs that test, it will compare the output to the saved snapshot.
Snapshot testing + axe-core
First, a big kudos to Michael Wheeler from UMich and his talk on Automated Web Accessibility Testing for the idea of using snapshot testing with axe-core.
Here's the approach: We save snapshots of the axe-core violations and check them into source control. That way, our tests will let us know when new violations come up, and our snapshot files keep track of which parts of the codebase need accessibility improvements.
To make it as easy as possible, I made a pytest plugin that combines Playwright, axe-core, and snapshot testing.
python3 -m pip install pytest-axe-playwright-snapshot
python3 -m playwright install --with-deps
Here's an example test from a Flask app:
from flask import url_for
from playwright.sync_api import Page
def test_index(page: Page, axe_pytest_snapshot):
page.goto(url_for("index", _external=True))
axe_pytest_snapshot(page)
Running the snapshot tests
First run: We specify the --snapshot-update argument to tell the plugin to save the snapshots to file.
python3 -m pytest --snapshot-update
That saves a file like this one to a directory named after the test and browser engine, like snapshots/test_violations/chromium/snapshot.txt:
color-contrast (serious) : 2
empty-heading (minor) : 1
link-name (serious) : 1
Subsequent runs: The plugin compares the new snapshot to the saved snapshot, and asserts if they differ.
python3 -m pytest
Let's look through some example outputs next.
Test results
New accessibility issue 😱
If there are violations in the new snapshot that weren't in the old, the test will fail with a message like this:
E AssertionError: New violations found: html-has-lang (serious)
E That's bad news! 😱 Either fix the issue or run `pytest --snapshot-update` to update the snapshots.
E html-has-lang - Ensures every HTML document has a lang attribute
E URL: https://dequeuniversity.com/rules/axe/4.4/html-has-lang?application=axeAPI
E Impact Level: serious
E Tags: ['cat.language', 'wcag2a', 'wcag311', 'ACT']
E Elements Affected:
E 1) Target: html
E Snippet: <html>
E Messages:
E * The <html> element does not have a lang attribute
Fixed accessibility issue 🎉
If there are less violations in the new snapshot than the old one, the test will also fail, but with a happy message like this:
E AssertionError: Old violations no longer found: html-has-lang (serious).
E That's good news! 🎉 Run `pytest --snapshot-update` to update the snapshots.
CI/CD integration
Once you've got snapshot testing setup, it's a great idea to run it on every potential change to your codebase.
Here's an example of a failing GitHub action due to an accessibility violation, using this workflow file:
Fixing accessibility issues
What should you do if you realize you've introduced an accessibility violation, or if you are tasked with reducing existing violations? You can read the reports from pytest to get a gist for the accessibility violations, but it's often easier to use a browser extension that uses the same Axe-core rules.
- Axe DevTools for Edge
- Axe DevTools for Chrome
Also consider an IDE extension like VS Code Axe Linter
Don't rely on automation to find all issues
I think it's really important for web apps to measure their accessibility violations, so that they can avoid introducing accessibility regressions and eventually resolve existing violations. However, it's really important to note that these automated tools can only go so far. According to the axe-core docs, it finds about 57% of WCAG issues automatically. There can still be many issues with your site, like with tab order or keyboard access.
In addition to automation, please consider other ways to discover issues, such as paying for an external accessibility audit, engaging with your disabled users, and hiring engineers with disabilities.
No comments:
Post a Comment