Accessibility checks for changed website pages in local CI
@axe-core/playwright gives the full axe rules in a test you control. Its lead is narrow; keep Cypress or Puppeteer if you already use them. Pa11y CI fits teams wanting configuration instead of test code, with a status check in front.
@axe-core/playwright
Deque’s free package runs the full axe-core rules inside Playwright Test, after any clicks or sign-in you script, and fails the test when a rule fails.
Its lead is narrow: 4.03 out of 5 against wick-a11y’s 3.98; you keep test code and add a response-status check.
Why @axe-core/playwright is our PickTeam behind @axe-core/playwright? Get your award badge
No Rising Star selected
The November 6 search found no newer tool with the required release, Windows/macOS, rendered-page gate and safety evidence established.
AllyCat checks source files; Includa is unfinished alpha software; other newer candidates lack required eligibility evidence.
Who is this article for? Small website teams checking named changed routes locally and in CI before publishing, while keeping manual accessibility review in the workflow.
About 31 min read
What Preferred Pick, Rising Star and Ready to Rock mean
The quick list
Start with @axe-core/playwright for a new suite. Keep Cypress or Puppeteer if you already use them; choose Pa11y CI for a configuration file, with a status check in front.
These eight packages have no software charge for local checks, checked November 6, 2026. Existing computer and CI capacity are assumed. Licenses: MPL-2.0 for Deque’s axe packages, Apache-2.0 for Playwright, Lighthouse CI and IBM’s checker, LGPL-3.0-only for Pa11y CI, MIT for cypress-axe and wick-a11y. Cypress Cloud is optional. We could not confirm current prices for every alternative. Your costs are CI time, browser downloads and the time to write and maintain the checks. IBM’s package sends install-time telemetry unless you opt out.
The comparison at a glance
Compare what fails the build, which rules run, what you can do before a scan and how current each project is.
Start with @axe-core/playwright; its lead over wick-a11y is narrow. Cypress and Puppeteer integrations suit existing suites. Other choices follow their fit for changed pages in local CI. All tools remain selectable.
Hides matching answers for the selected tools. Unknowns and unverified claims stay visible.
| Feature | @axe-core/playwright | wick-a11y | cypress-axe | @axe-core/puppeteer | Lighthouse CI | IBM accessibility-checker | Pa11y CI | axe-core CLI |
|---|---|---|---|---|---|---|---|---|
| Editorial score out of 5 | 4.03 · Preferred Pick, narrow lead | 3.98 | 3.88 | 3.83 | 3.80 | 3.75 | 3.68 | 3.38 |
| First year / three years, software charges | $0 / $0 | $0 / $0 | $0 / $0 | $0 / $0 | $0 / $0 | $0 / $0 | $0 / $0 | $0 / $0 |
| Best fit | Preferred Pick for a new suite; full axe rules after any interaction; narrow lead | Cypress teams wanting HTML reports and annotated screenshots; narrow second place | Teams already using Cypress; keep test code and explicit failure checks | Teams already using Puppeteer; add assertions and error handling | Teams already using Lighthouse CI for performance that want missing pages to fail | A second opinion from IBM’s own rules, with baselines and spreadsheet reports | Teams wanting a configuration file and readable output, plus a separate status check | A quick one-off axe check where a matching ChromeDriver is already managed |
| How you set it up | Test file in JavaScript or TypeScript; Playwright can start your local site first | Install plugin and Cypress peers; register tasks and support commands; write tests | Install plugin and peers; add test commands and a logging task | Install Puppeteer and the integration; write a test and report handling | Configuration file, then collect and assert; built-in server for a static build folder | Text file listing pages, optional YAML settings, one command | JSON or JavaScript configuration and one command; Chrome comes with Puppeteer | URLs on the command line; ChromeDriver must match your Chrome version |
| Page that returns an error status | Fails only if your test checks the response status; navigation does not stop on 404 or 500 | Check the navigation response and expected page content in your Cypress test | Check navigation and expected page content in the Cypress test | Check the navigation response in your script; logging alone is not a gate | Collection fails: Lighthouse stops on 4xx and 5xx responses. An earlier check on October 1, 2026 stopped on a 404 | Not documented | Scanned as if it were your page; add a status check first. An earlier check on October 1, 2026 passed a missing page | No status check; fails only if the error page itself breaks a rule |
| Rules | axe-core 4.14: 91 WCAG A and AA and best-practice rules by default | axe-core through cypress-axe; configure all required rules and severities | axe-core supplied as a peer dependency; choose a supported current version | axe-core 4.14, like the Pick | A subset of axe-core; 0.16.0 upgrades to Lighthouse 13.6.0; older audit counts do not describe it | IBM’s own engine; IBM_Accessibility policy by default | HTML_CodeSniffer by default; axe-core 4.13 instead or as well | axe-core 4.14, the same rules as @axe-core/playwright |
| Before the scan | Any scripted step: click, type, sign in; Chromium, Firefox or WebKit | Cypress interactions before scanning; one accessibility analysis per test | Scan after Cypress interactions; check the state visitors use | Any interactions you script in Puppeteer before analyze() | A Puppeteer script before collection, mainly for signing in | None from the command line; use its test-framework integrations | Per-page actions such as click, type and wait; viewport and headers | A fixed delay after load; no interaction |
| Release record (dates checked stated below) | 4.14.0, October 15, 2026; Playwright Test 1.64.1, October 14, 2026 | 3.0.2, August 16, 2026; Cypress 15.10+ handling; checked November 6, 2026 | 1.7.0, August 29, 2025; Cypress 15 support; checked November 6, 2026 | 4.14.0, October 15, 2026; checked November 6, 2026 | 0.16.0, November 2, 2026; Lighthouse 13.6.0; checked November 6, 2026 | 4.0.34, September 8, 2026 | 5.0.0, October 5, 2026, with Pa11y 10.0.0 from August 28, 2026 | 4.14.0, October 15, 2026 |
| Price and license | $0; MPL-2.0, with Playwright under Apache-2.0 | $0; MIT plugin and local Cypress App; Cloud optional | $0; MIT plugin, local Cypress App; Cloud is optional | $0; MPL-2.0 integration; no paid runner service required | $0; Apache-2.0 | $0; Apache-2.0 | $0; LGPL-3.0-only | $0; MPL-2.0 |
| Reader reviews | No reviews yet. Write a review | No reviews yet. Write a review | No reviews yet. Write a review | No reviews yet. Write a review | No reviews yet. Write a review | No reviews yet. Write a review | No reviews yet. Write a review | No reviews yet. Write a review |
Software charges checked November 6, 2026 in project documentation; existing computer and CI capacity assumed. cypress-axe, Deque’s Puppeteer integration and Lighthouse CI releases were checked that day. wick-a11y’s README and release listing were checked November 6. Other feature evidence and download counts remain dated October 19, 2026.
No automated check certifies accessibility. Deque says axe-core finds about 57% of WCAG issues, by its own count of issues in its audits; in the UK Government Digital Service’s audit of 142 planted barriers, no single tool found more than 40%. W3C WAI says tools cannot check every accessibility aspect and can mislead. A page that returns a normal status with “not found” text passes every status check here.
All features and sources 13 criteria
Open the groups that matter to you. They compare the same selected tools. “Planned” means future work, not an available feature. Sources include the published conditions.
Build gate and findings
Show 5 criteria
| Feature | @axe-core/playwright | wick-a11y | cypress-axe | @axe-core/puppeteer | Lighthouse CI | IBM accessibility-checker | Pa11y CI | axe-core CLI |
|---|---|---|---|---|---|---|---|---|
| Page that returns an error status | Fails only if your test checks the response status; navigation does not stop on 404 or 500 | Check the navigation response and expected page content in your Cypress test | Check navigation and expected page content in the Cypress test | Check the navigation response in your script; logging alone is not a gate | Collection fails: Lighthouse stops on 4xx and 5xx responses. An earlier check on October 1, 2026 stopped on a 404 | Not documented | Scanned as if it were your page; add a status check first. An earlier check on October 1, 2026 passed a missing page | No status check; fails only if the error page itself breaks a rule |
| Allowing known issues | Exclude elements, disable rules or snapshot a fingerprint of rule and elements | Rule, context and severity filters; warning-only settings do not fail the build | Scope scans, configure axe, filter impact; skipFailures must stay false | Exclude elements or disable rules; add your own baseline comparison | Per-audit assertions; maxLength counts items, not which elements fail | Baseline files; a scan fails when results differ from the baseline | Ignore issue codes or set a threshold; keep the threshold at 0 for now | Disable rules or exclude parts of the page |
| Open reports affecting the gate | 4.14.0 can hang on some sandboxed iframes; a development-branch fix is not yet released | onlyWarnImpacts and impact filters can leave issues as warnings; keep every required severity failing | skipFailures logs violations without failing; filters can omit issues | The example logs results and catches errors; add a failing assertion | A fixed error and a new one cancel out under count assertions (1145) | Not reviewed in depth; README notes a Jest parallel-run error and Content Security Policy blocking | Unloaded pages pass with a threshold of 1 or more (368); pages skipped after a browser crash (282) | Chrome tab crash about four minutes into a CI scan (1338) |
| Rules | axe-core 4.14: 91 WCAG A and AA and best-practice rules by default | axe-core through cypress-axe; configure all required rules and severities | axe-core supplied as a peer dependency; choose a supported current version | axe-core 4.14, like the Pick | A subset of axe-core; 0.16.0 upgrades to Lighthouse 13.6.0; older audit counts do not describe it | IBM’s own engine; IBM_Accessibility policy by default | HTML_CodeSniffer by default; axe-core 4.13 instead or as well | axe-core 4.14, the same rules as @axe-core/playwright |
| Where you see the problem | Failed test lists violations; full results can be attached to the HTML report | Cypress log, terminal table and HTML report with element guidance and annotated screenshots | Cypress log; terminal detail needs callback and cy.task | Results object; format and retain it yourself | Terminal names the audits you assert; elements are in the saved HTML or JSON report | JSON, CSV, HTML or Excel reports | Terminal: element, code snippet and fix advice; JSON and other reporters | Terminal: rule, link and elements; JSON file on request |
These selected tools have matching answers in this group. Turn off “Show differences only” to see them.
Sources for build gate and findings
Checked October 19, 2026 in Playwright’s accessibility guide and navigation reference, the Pa11y CI README and Pa11y navigation code, Lighthouse’s error definitions and audit configuration, IBM’s configuration guide, the axe-core CLI README and each project’s issue tracker. The October 1, 2026 observations come from an earlier hands-on check made before our research-only method.
Pages, setup and upkeep
Show 8 criteria
| Feature | @axe-core/playwright | wick-a11y | cypress-axe | @axe-core/puppeteer | Lighthouse CI | IBM accessibility-checker | Pa11y CI | axe-core CLI |
|---|---|---|---|---|---|---|---|---|
| First year / three years, software charges | $0 / $0 | $0 / $0 | $0 / $0 | $0 / $0 | $0 / $0 | $0 / $0 | $0 / $0 | $0 / $0 |
| Before the scan | Any scripted step: click, type, sign in; Chromium, Firefox or WebKit | Cypress interactions before scanning; one accessibility analysis per test | Scan after Cypress interactions; check the state visitors use | Any interactions you script in Puppeteer before analyze() | A Puppeteer script before collection, mainly for signing in | None from the command line; use its test-framework integrations | Per-page actions such as click, type and wait; viewport and headers | A fixed delay after load; no interaction |
| How you set it up | Test file in JavaScript or TypeScript; Playwright can start your local site first | Install plugin and Cypress peers; register tasks and support commands; write tests | Install plugin and peers; add test commands and a logging task | Install Puppeteer and the integration; write a test and report handling | Configuration file, then collect and assert; built-in server for a static build folder | Text file listing pages, optional YAML settings, one command | JSON or JavaScript configuration and one command; Chrome comes with Puppeteer | URLs on the command line; ChromeDriver must match your Chrome version |
| Release record (dates checked stated below) | 4.14.0, October 15, 2026; Playwright Test 1.64.1, October 14, 2026 | 3.0.2, August 16, 2026; Cypress 15.10+ handling; checked November 6, 2026 | 1.7.0, August 29, 2025; Cypress 15 support; checked November 6, 2026 | 4.14.0, October 15, 2026; checked November 6, 2026 | 0.16.0, November 2, 2026; Lighthouse 13.6.0; checked November 6, 2026 | 4.0.34, September 8, 2026 | 5.0.0, October 5, 2026, with Pa11y 10.0.0 from August 28, 2026 | 4.14.0, October 15, 2026 |
| Security advisory evidence | Playwright: CVE-2025-59288, fixed in 1.55.1; none for @axe-core/playwright | No dependency audit established; keep Cypress, axe and browser dependencies current | No published plugin advisory found November 6; dependencies not exhaustively checked | No published integration advisory found November 6; keep the browser current | None listed | None listed | None listed | None listed |
| Price and license | $0; MPL-2.0, with Playwright under Apache-2.0 | $0; MIT plugin and local Cypress App; Cloud optional | $0; MIT plugin, local Cypress App; Cloud is optional | $0; MPL-2.0 integration; no paid runner service required | $0; Apache-2.0 | $0; Apache-2.0 | $0; LGPL-3.0-only | $0; MPL-2.0 |
| What leaves your machine | No account or upload documented; browsers download during setup | Local report files include page screenshots and offending elements; protect retained reports | Keep the local runner workflow; cloud recording is optional | No hosted scan required; keep reports local | Nothing with the filesystem target; the public-storage target makes reports public for days to weeks | Install-time IBM Telemetry, with a documented opt-out; rules fetched during runs by default | No account or upload documented | No account or upload documented |
| Weekly npm downloads, October 12 to 18, 2026 | About 9.4 million | Not checked | About 357,000 | Not checked | About 1.2 million, including performance use | About 14,000 | About 63,000 | About 67,000 |
These selected tools have matching answers in this group. Turn off “Show differences only” to see them.
Sources for pages, setup and upkeep
Release dates and licenses from each project’s release page and npm record, incumbent advisories from OSV, download counts from the npm downloads API, all on October 19, 2026. The cypress-axe advisory cell uses the GitHub advisory search and project security page checked November 6, 2026; the Puppeteer cell uses the Deque integration security page checked November 6. These bounded searches do not establish dependency safety. No separate advisory or dependency audit was established for wick-a11y; its cell records that limit. Data handling from Lighthouse CI’s services disclaimer and IBM’s package documentation. Downloads are a rough signal of use, not a count of teams.
What matters when choosing
@axe-core/playwright, run inside Playwright Test, is our Practical Pick for a small website team that wants its changed pages checked automatically, on a laptop and in CI, before it publishes. Deque’s free package runs the full axe-core rule set on any page Playwright opens, and a failed check fails the test like any other assertion. Because the check is ordinary test code, you can open a menu, fill in a form or sign in before scanning, and Playwright can start your local site before the run. Version 4.14.0, released October 15, 2026, uses axe-core 4.14. Among the five packages whose downloads we checked on October 19, it was by far the most used: npm counted about 9.4 million downloads of the package in the week to October 18, 2026, against about 67,000 for axe-core CLI and 63,000 for Pa11y CI. Downloads are a rough signal of use, not a count of teams. Playwright accessibility testing, releases, npm weekly downloads
The most important compromise is that you write and keep a small test file rather than a configuration file. Playwright also does not fail a missing page by itself: its documentation says page.goto does not throw when the server returns a 404 or 500, so add one line that checks the response status before the scan. For known problems you cannot fix yet, Playwright’s guide suggests recording a fingerprint of each one, its rule and affected elements, so a new error on another element still fails the test. Page navigation, known issues
The lead is narrow: @axe-core/playwright scores 4.03 out of 5, ahead of wick-a11y at 3.98, cypress-axe at 3.88 and @axe-core/puppeteer at 3.83. wick-a11y adds HTML reports and screenshots to Cypress checks. Choose a Cypress or Puppeteer integration if your team already uses that runner; Playwright remains the clearest starting point for a new suite.
If you will not write test code, choose Pa11y CI and add a status check before it scans. It reads a list of pages or a sitemap from one configuration file, can click and type before scanning, and prints each problem with the element and a suggested fix. But Pa11y tests whatever page comes back, including a “not found” page, and two open bugs can let a page that never ran pass the build. If you already use Lighthouse CI for performance budgets, it is the one tool here that fails a missing page without extra work, but it runs a subset of axe’s rules. Version 0.16.0, released November 2, 2026, upgrades to Lighthouse 13.6.0. IBM’s accessibility-checker is the choice for a second opinion from a different rule engine; axe-core CLI is the quickest one-off check if you already manage a matching ChromeDriver. Pa11y CI, Lighthouse CI releases
Every tool here is limited by what automation can find. Deque says axe-core finds “on average 57% of WCAG issues automatically”. Its own March 2021 study gave that figure from more than 2,000 of its audits, counted by number of issues rather than by WCAG success criteria. When the UK Government Digital Service planted 142 barriers on one page, the best single tool found 40%, axe 29% and HTML_CodeSniffer, Pa11y’s default engine, 20% (results last updated April 2018). W3C’s accessibility initiative says tools “can not determine accessibility, they can only assist in doing so.” A passing run means no automated rule failed on that page in that state. Keyboard use, screen-reader output, wording and most problems in context still need a person. axe-core README, Deque study, March 10, 2021, GDS tool audit, W3C WAI, updated May 13, 2024
No Rising Star selected. The November 6, 2026 search found no newer tool with all the required release, Windows/macOS availability, rendered-page gate and safety evidence established. AllyCat checks source files rather than rendered routes, Includa is explicitly unfinished alpha software, and wick-a11y was already public before the two-year newcomer window. AllyCat, Includa, wick-a11y.
We could not confirm current prices for every alternative.
Disclosure: Practical Picks articles are researched and written with AI, currently Anthropic’s Claude. Playwright documents test agents that run with Claude Code, and Deque advertises its paid axe MCP Server for use with Claude Code. We have no relationship with Microsoft, Deque or Anthropic over this article, and these Claude links carried no weight in our judgment. Playwright test agents, axe MCP Server
@axe-core/playwright
Preferred Pick · changed pages in local CI
Choose @axe-core/playwright when someone on the team can keep a small test file. Check each page’s HTTP status in the test, then scan with the full axe rules in the state your visitors see.
Reasons to choose it
- Full axe-core 4.14 rules, current as of October 2026
- Scan after clicks, form entries or signing in
- Playwright can start your local site before the run
What to watch
- You write and maintain test code
- A 404 or 500 does not fail the test unless you check the status
- Published 4.14.0 can hang on some sandboxed iframes; the fix is on the development branch
Playwright’s own accessibility guide shows the pattern: open the page, run new AxeBuilder({ page }).analyze() and expect the list of violations to be empty. You can limit a scan to part of the page, exclude elements, restrict rules to WCAG tags such as wcag2aa, disable specific rules, and attach the full results to Playwright’s report so the person fixing the page sees every affected element. A shared fixture keeps those settings the same across test files. The guide itself says automated tests “can detect some common accessibility problems such as missing or invalid properties” and “cannot detect all types of WCAG violations”. Accessibility testing guide
What makes it the strongest fit is control over the page state. Many accessibility failures appear only after an interaction: an open menu, a dialog, a form showing errors. In a test you reach that state with ordinary Playwright steps and then scan, in Chromium, Firefox or WebKit. Playwright’s webServer option can “launch a local dev server before running your tests”, so the same command works on a laptop and in CI. Deque’s package is free under the MPL-2.0 license and Playwright under Apache-2.0; neither needs an account. Playwright Test 1.64.1 was released October 14, 2026. Web server, package record, Playwright Test release
Read the full @axe-core/playwright analysis and sources
Add the status check yourself. Playwright documents that navigation “will not throw an error when any valid HTTP status code is returned by the remote server”, including 404 and 500, and returns the response so you can read response.status(). A test that skips this check will scan a “not found” page and may pass. A page that returns 200 with “not found” text passes this check too, so assert something that only the real page contains. page.goto
Choose another tool if nobody on the team will write or review test code. Open reports in Deque’s tracker include one, from October 8, 2026, where a scan hangs on a sandboxed iframe that does not allow scripts until the test times out, which fails the test but hides the cause; a fix was merged into the development branch on October 27, but the November 6 release page still lists 4.14.0, which predates it. Release record, checked November 6, 2026 OSV lists one advisory for the playwright package, fixed in 1.55.1 in October 2025: macOS scripts that reinstall branded Chrome and Edge downloaded installers without checking the server’s certificate (CVE-2025-59288). The same axe rules are available in Cypress through the third-party cypress-axe package and in Deque’s Puppeteer and Selenium packages, if your team already uses one of those. Issue 1469, advisory
Used @axe-core/playwright? Share your experience to help other readers choose.
wick-a11y
Existing Cypress tests · detailed reports
Choose wick-a11y if your Cypress team wants HTML reports with element guidance and annotated screenshots. Keep every required severity failing.
Reasons to choose it
- HTML reports and annotated screenshots
- Scan after Cypress interactions
- Local runner; Cloud optional
What to watch
- One accessibility analysis per test
- Warning and impact settings can weaken the gate
- You maintain tasks, configuration and test code
wick-a11y runs axe through cypress-axe after the interactions in your Cypress test, then adds a terminal table and an HTML report with affected elements, fix guidance and annotated screenshots. The MIT plugin needs no paid Cypress Cloud subscription. It scores 3.98 out of 5 against the Pick’s 4.03: a narrow gap, with stronger ready-made reports than plain cypress-axe but the same need to maintain browser tests. Choose it when your team already uses Cypress and wants reports it can share with the people fixing the page. README, checked November 6, 2026
Register the plugin’s tasks and support commands, visit each changed route in a test, and scan after the relevant interaction. Keep all required severities in the failing set: onlyWarnImpacts produces warnings, and impact filters can omit problems. You still need to check that the intended page loaded. The plugin allows only one accessibility analysis per test, so use separate tests for different states. Its reports include screenshots and page elements; protect them if your pages contain private data. Configuration and limits
Read the full wick-a11y analysis and sources
Version 3.0.2 is listed on August 16, 2026; the current README documents Cypress 15.10+ configuration and compatibility differences for older versions. We did not establish a complete dependency security audit. Its additional report detail does not establish a higher detection rate than the underlying axe engine. It was already public before November 2024, so it is not a newcomer for this comparison. Release listing and README, checked November 6, 2026
Used wick-a11y? Share your experience to help other readers choose.
cypress-axe
Existing Cypress tests
Keep Cypress if your team already uses it. Run axe after the interactions that matter, keep skipFailures false and add terminal reporting.
Reasons to choose it
- Full axe checks in Cypress tests
- Scan after page interactions
- Local runner; paid Cloud is optional
What to watch
- You maintain test code
- Detailed terminal output needs a callback and logging task
- Filters and skipFailures can hide violations
Choose cypress-axe if your changed-page tests already run in Cypress. The MIT-licensed plugin injects axe into a page after cy.visit(), then cy.checkA11y() asserts that no violations were found. You can scan again after opening a menu or submitting a form, scope the check to elements and select impact levels. Cypress’s own guide documents this approach separately from its paid cloud accessibility service. The local plugin does not require that service. Plugin README, Cypress guide, pricing, checked November 6, 2026.
It scores 3.88 out of 5, close to the Pick. You still maintain test code, and terminal reports with detailed violations need a callback and a cy.task logging task. Leave skipFailures false: setting it true logs violations while allowing the test to pass. Review filters and navigation assertions so an omitted check cannot hide a regression. Version 1.7.0, released August 29, 2025, adds Cypress 15 support. No independent current detection comparison establishes an accuracy advantage over the Pick’s axe rules. Releases, checked November 6, 2026.
Used cypress-axe? Share your experience to help other readers choose.
@axe-core/puppeteer
Existing Puppeteer scripts
Keep Puppeteer if it already opens your changed routes. Assert that violations are empty, fail on scan errors and check navigation responses.
Reasons to choose it
- Same axe generation as the Pick
- Script page interactions before scanning
- Scope scans and configure rules
What to watch
- Logging results alone does not fail CI
- You add report formatting and retention
- Some sandboxed frames need the unreleased fix
Deque’s free Puppeteer integration runs axe on a Page or Frame you already control. Choose it when Puppeteer already opens the changed routes and reaches the states you need. It supports selectors, exclusions, tags and rule configuration; version 4.14.0 uses the same axe generation as the Pick. API guide, releases, checked November 6, 2026.
It scores 3.83 out of 5. The example logs results and catches errors; it is not by itself a failing CI test. Assert that violations are empty, make scan errors fail the process, check navigation responses and retain the report. A Content Security Policy can require bypassing it before navigation to inject axe. As with the Pick, the development branch contains an iframe fix that the latest published release does not yet include. The extra gate and reporting code is the main reason to start with Playwright Test for a new suite.
Used @axe-core/puppeteer? Share your experience to help other readers choose.
Lighthouse CI
Also consider · already using Lighthouse CI
Add accessibility assertions to Lighthouse CI if you already run it for performance. It fails a page that returns an error status with no extra step, but assert named audits and keep reports on your own disk.
Reasons to choose it
- Stops on 4xx and 5xx responses by itself
- Category and per-audit assertions
- Built-in server for a static build folder
What to watch
- Runs a subset of axe-core’s rules
- Version 0.16.0 upgrades to Lighthouse 13.6.0; older audit counts no longer apply
- The public-storage upload target makes reports public
Lighthouse CI collects Lighthouse reports for a list of URLs, or for HTML files it finds in a static build folder that it serves itself, and then checks them against assertions. Lighthouse treats a page that answers with a 4xx or 5xx status as a failed load, and its command line exits with an error, so lhci collect stops the build before any score is calculated. In an earlier hands-on check on October 1, 2026, before we adopted our research-only method, Lighthouse CI 0.15.1 stopped on a missing local page with “Status code: 404”. One exception: its static server’s single-page-app option serves index.html for unknown paths, so a missing route then loads normally. Lighthouse error definitions, CLI exit on runtime errors, configuration, earlier check record
Its accessibility checks are a subset of axe-core’s. In the older Lighthouse 12.6.1 used for our October research, the accessibility category had 63 automated audits, against 91 WCAG and best-practice rules that axe-core 4.14 runs by default. The category score is a weighted average with no partial credit, and audits with zero weight, such as empty-heading in that version, do not affect it. Assert those audits individually if they matter to you. A category assertion on its own prints only the score; in the October 1 check, the terminal gave the failing page’s score, 0.72, and nothing more until we added assertions for named audits, and the affected elements appeared only in the saved report. A 2019 demonstration built a deliberately inaccessible page with a perfect Lighthouse score. Lighthouse 12.6.1 configuration, axe-core rules, scoring, updated October 22, 2025, matuzo.at, May 31, 2019
Read the full Lighthouse CI analysis and sources
Lighthouse CI 0.16.0, released November 2, 2026, upgrades to Lighthouse 13.6.0, so the long gap between releases is over. That improves its case for teams already using it, but its narrower accessibility coverage and report workflow still make it a different choice from the Pick. The older 63-audit count above describes 12.6.1, not the new release. The August report about count assertions remains relevant: a fixed error and a new one can cancel out when only the number of failing items matters. Releases, checked November 6, 2026, issue 1145, checked October 19, 2026
It is free under the Apache-2.0 license. Keep reports on your own disk with the filesystem upload target. The quick-start temporary-public-storage target makes reports “available to anyone on the internet with the link”, is run by a third party, Eris Ventures LLC, and keeps them for three days to five weeks. Upload targets, services disclaimer
Used Lighthouse CI? Share your experience to help other readers choose.
IBM accessibility-checker
Second opinion · a different rule engine
Consider IBM’s accessibility-checker beside an axe-based tool when you want a second rule set, baselines and spreadsheet reports. Plan for its telemetry opt-out and network access during runs.
Reasons to choose it
- IBM’s own rules, independent of axe
- Baselines fail a scan only when results change
- JSON, CSV, HTML and Excel reports
What to watch
- Installing it agrees to IBM Telemetry unless you opt out
- Fetches its rules during runs by default
- Error-page handling not documented
IBM’s accessibility-checker package, part of its Equal Access Toolkit, uses IBM’s own rule engine rather than axe, so it is a genuinely different second opinion. Its achecker command scans a text file listing pages, files or folders and writes JSON, CSV, HTML or Excel reports. By default it fails on violations and on potential violations that need review, and its source sets exit code 1 when any page fails or errors. Baseline files record known results, so later scans fail only when the results change. The same package works inside Playwright, Puppeteer, Selenium and Jest tests. Version 4.0.34 was released September 8, 2026, and IBM published releases in every month from June to September 2026. It is free under the Apache-2.0 license. Package README, configuration, command-line source, releases
Know what it sends and fetches. The README says the package uses IBM Telemetry and that “by installing this package as a dependency you are agreeing to telemetry collection”, with an opt-out documented by IBM. By default it uses the latest rule archive, loaded during the run from a content delivery network or IBM’s rule server, so a CI job needs network access, results can change when IBM updates its rules, and a page’s Content Security Policy can block the engine. A “versioned” setting pins the rules to those current when your installed version was released. Its documentation does not say how it treats a page that returns an error status. Configuration and telemetry, known issues
Read the full IBM accessibility-checker analysis and sources
Sparkbox’s February 2024 review of IBM’s browser extension, which uses the same engine, found 12 of 23 planted problems, including 9 of the 13 it expected a tool to find, with “at least one false positive”. That is one page and an older version, not a current detection rate. Choose another tool if you want page interactions without writing test code, or if your CI cannot reach the internet. Sparkbox review
Used IBM accessibility-checker? Share your experience to help other readers choose.
Pa11y CI
Also consider · no test code
Choose Pa11y CI when you want a configuration file instead of test code. Request every page first and stop if any returns an error, keep the threshold at 0, and include one page with a known error.
Reasons to choose it
- URL list or sitemap, with per-page click, type and wait actions
- HTML_CodeSniffer and axe-core rules, separately or together
- Readable terminal output with element and fix advice
What to watch
- Scans a “not found” page as if it were yours
- Open bugs let unloaded pages pass with a threshold of 1 or more
- Pages after a browser crash can go unreported
Pa11y CI 5.0.0, released October 5, 2026, runs Pa11y 10 against a list of URLs or every address in a sitemap, with options to rewrite sitemap addresses to your local server and to exclude some. Each page can have its own actions, such as click element #tab-1 or wait for path to be /myaccount, its own viewport and headers, and a screenshot. It runs HTML_CodeSniffer by default and can run axe-core 4.13 instead or as well, which gives two independent rule sets in one pass. Exit codes are documented: 0 when pages pass, 1 for a technical fault such as a bad configuration, 2 when at least one page fails. It is free under the LGPL-3.0-only license, and Pa11y 10 needs Node.js 22.13 or later. Pa11y CI README, Pa11y 10 release, actions
Pa11y does not check whether the page loaded. Its code opens the address and tests whatever the browser shows, without reading the HTTP status, and a contributor answering a 2020 request wrote that “it sure doesn’t look like there is a facility for this in pa11y right now.” In an earlier hands-on check on October 1, 2026, before we adopted our research-only method, Pa11y CI 4.1.1 tested a local server’s 404 page as if it were the intended page, reported no errors and exited 0. Before the scan, request every target address and stop the build if any returns an error status. Navigation code, issue 538, earlier check record, October 1, 2026
Read the full Pa11y CI analysis and sources
Keep the threshold at its default of 0 for now. A maintainer reported on October 6, 2026 that with --threshold set to 1 or more, a run in which pages failed to load but had no accessibility issues exits 0; a fix was proposed but not merged when we checked. A separate open report, from September 2025, says that if the browser crashes between pages, the remaining pages are neither tested nor reported; turning off useIncognitoBrowserContext avoided it for the reporter. Add one page with a known error to your list, so a run that silently stops testing still fails. Issue 368, issue 228, issue 282
In the same October 1 check, Pa11y CI 4.1.1 named all three planted defects on a small test page, an unlabeled email field, an image without alternative text and low-contrast text, with the element, the code snippet and the measured contrast ratio. That matches what its documentation describes, but one small page says nothing about how much it finds on real sites. Pa11y CI and Pa11y have each had three releases in 2026. Earlier check record, releases
Used Pa11y CI? Share your experience to help other readers choose.
axe-core CLI
Quick one-off axe check
Use axe-core CLI for a quick check of a few URLs with the full axe rules when your machine or CI image already keeps ChromeDriver in step with Chrome. Add a status check before it.
Reasons to choose it
- Same axe-core 4.14 rules as the Pick
- Several URLs in one command; --exit fails the run
- Rule links and elements in the terminal
What to watch
- ChromeDriver must match your Chrome version
- No interaction before the scan; not a crawler
- Does not read the HTTP status
Deque’s command line takes one or more URLs, runs axe-core and prints each rule that failed with a link and the affected elements; --exit makes it return 1 when any rule fails, and results can be saved as JSON. Version 4.14.0, released October 15, 2026, runs axe-core 4.14, the same rules as the Pick. Its README says it “is not a crawler” and suggests a test-framework integration if you find yourself checking dozens of pages. It cannot click or type before scanning; it can only wait a fixed time after load. It is free under the MPL-2.0 license; Deque’s terms for its free browser extension do not apply to it. CLI README, releases
It drives Chrome through ChromeDriver, and the two must match. Version 4.14.0 depends on ChromeDriver 152, so if your Chrome is a different major version you need to install a matching driver, as the README describes, or point --chromedriver-path at one. In an earlier hands-on check on October 1, 2026, before we adopted our research-only method, axe-core CLI 4.13.0 would not start until a matching ChromeDriver was added; it then reported all three planted defects with selectors and rule links. CLI README, package record, earlier check record
Read the full axe-core CLI analysis and sources
It does not read the HTTP status either: it loads pages through WebDriver, which does not report one. In the October 1 check it failed a missing page only because the server’s error page itself broke two accessibility rules; a well-built error page could pass. Add a status check before it. An open report from May 2026 describes Chrome crashing about four minutes into a scan on GitHub Actions; the crash stopped the run rather than passing it. Page loading code, issue 1338
Used axe-core CLI? Share your experience to help other readers choose.
Which product are we missing?
Know a product that belongs in this article? Tell us what it does differently and why it deserves a closer look. Newcomers welcome.
Readers and product teams can suggest candidates. Suggestions inform our research; they do not guarantee inclusion or a recommendation.
Opens an email draft with this article attached as a link. Nothing is sent until you send it.
No email app? Send the product name, website, and this article’s title to admin@practicalpicks.guide.
What people are saying
No Practical Picks reader reviews are available yet for these tools. The most useful public evidence is in the projects’ issue trackers, and it concentrates on the same point: a build should fail when a page was not really checked. Pa11y CI’s maintainers have open bugs for pages that fail to load but still let the build pass when a threshold is set, and for pages skipped after a browser crash. A Lighthouse CI user showed in August 2026 that a count-based assertion can pass when one known error is fixed and a new one appears; Playwright’s guide suggests recording known issues by rule and affected elements instead. Pa11y CI issue 368, issue 282, Lighthouse CI issue 1145, Playwright guide
Reports about the engines themselves are narrower. A July 2026 report against Deque’s Selenium package describes a contrast failure missed at a narrow window width in Chrome 151 because the text was treated as partly covered; it is open and unconfirmed by Deque. Independent reviews by Sparkbox, a web development firm, found 10 of 13 tool-detectable problems with the axe extension and no false positives in December 2020, and 9 of 13 with IBM’s extension in February 2024. These are single-page results from different years, not current detection rates, and a reported failure mode does not show how often it happens. Issue 1403, Sparkbox axe review, Sparkbox IBM review
How we researched
This is public-source research. For this review we did not install or run any tool, scan any page, create accounts or contact a maker. No maker supplied access or payment, and our links carry no affiliate tracking. We first compared Lighthouse CI, Pa11y CI and axe-core CLI on October 1, 2026, including a small hands-on check described below. On October 19, 2026 we rechecked documentation, release records, source code, issue trackers, security advisories and licenses, added two tools, and made the decision again from public evidence.
We wrote this for a small website team with a local preview and a CI pipeline that wants a list of changed pages checked automatically before publishing, and wants the build to fail for the right reasons. The fixed weights are gate reliability, including pages that do not load (25%); actionable findings (20%); page states and routes (15%); setup and upkeep (15%); project health and security (15%); and software cost (10%). The final November 6, 2026 ordering starts with @axe-core/playwright (4.03), wick-a11y (3.98), cypress-axe (3.88) and @axe-core/puppeteer (3.83). Pa11y CI (3.68) remains the configuration-first alternative, while Lighthouse CI (3.80) suits an existing performance pipeline and stops on missing pages. The scores are editorial judgments from the documented configurations, not measured detection rates.
We compared the Cypress and Puppeteer integrations because setup, assertions and reports matter even with the same axe engine. Other scored options were Salesforce sa11y’s WebdriverIO integration (3.80), @axe-core/webdriverjs and @axe-core/webdriverio (both 3.75), A11yEco CLI (3.53), Lighthouse CLI with assertions (3.48), accessio-scan (3.40), Unlighthouse (3.38), and AccessLint CLI (3.28). An unknown first release prevents a newcomer award, but does not remove a product from this comparison. WebDriver tools require driver management; bare Lighthouse needs assertion code; Unlighthouse emphasizes whole-site category budgets. Salesforce sa11y, WebdriverIO integration, Lighthouse, Unlighthouse, A11yEco CLI, accessio-scan, AccessLint, checked November 6, 2026.
Component tools such as jest-axe, vitest-axe and Storybook’s accessibility add-on do not replace a list of changed website routes. Storybook can run real-browser checks; its component focus, rather than a simulated browser, is the reason it is outside this comparison. GitHub’s scanner is an issue-filing GitHub Action, and paid hosted monitoring is a different workflow from the account-free local checks considered here.
For newer tools, we revisited the October 19 candidates and checked project documentation, the GitHub accessibility-testing directory, repository search and W3C’s tools overview on November 6, 2026. npm search and version histories were inaccessible. AccessLint’s explicit macOS CLI support, accessio-scan’s platform and independence evidence, and A11yEco’s first release and CLI safety evidence were not established; no newcomer qualified. Omi A11Y and a11y-ai lacked individual product evidence. Happo needs hosted credentials; MagentaA11y is documentation. UUV, current QualWeb, sitespeed’s axe plugin, Accessibility Insights Action, be-a11y and a11ymcp remain unscored because their current documentation did not establish a complete rendered-page gate for this workflow. SortSite documents a paid Developer CI edition, outside the account-free, no-required-purchase setup considered here. UUV, QualWeb, sitespeed.io, Accessibility Insights, be-a11y, a11ymcp, Happo, MagentaA11y, SortSite Developer.
In an earlier hands-on check on October 1, 2026, before we adopted our research-only method, we served two original pages on our own machine: one with an unlabeled email field, an image without alternative text and low-contrast text, and a corrected copy. Pa11y CI 4.1.1, axe-core CLI 4.13.0 and Lighthouse CI 0.15.1, all using Chrome for Testing 152.0.7977.54 on Linux, reported the three defects and nothing on the corrected page. On a missing page, Lighthouse CI stopped with a 404 error, Pa11y CI passed, and axe-core CLI failed only because the error page had its own accessibility problems. These observations match what the tools’ source code and documentation say. They did not decide the Pick, which was not part of that check. Earlier check record, October 1, 2026
| Tool | Public sources | What remains uncertain |
|---|---|---|
| @axe-core/playwright | Playwright accessibility, navigation and web-server guides; npm release records; Deque’s issue tracker; OSV advisory; npm download counts. | Detection rate on real sites, behavior on pages with complex iframes, how much test code a given site needs. |
| Pa11y CI | README and exit codes, Pa11y release notes and navigation code, issue tracker. | When the open threshold and crash fixes ship, current detection rate of HTML_CodeSniffer. |
| Lighthouse CI | Configuration and services documentation, November 2 release notes, older Lighthouse source, scoring guide, issue tracker. | Behavior on your pages under Lighthouse 13.6.0; the versioned category configuration was checked November 6, 2026. |
| IBM accessibility-checker | Package README and configuration, command-line source, releases, Sparkbox review of the extension. | How it treats error pages, exactly what the telemetry collects, operation without network access. |
| axe-core CLI | README, package record, page-loading code, releases, issue tracker. | Stability on long scans in CI, ChromeDriver matching on your runner. |
| cypress-axe | Plugin README and releases; Cypress accessibility guide and pricing, checked November 6, 2026. | Site-specific test code, navigation failures and plugin compatibility on your runner. |
| @axe-core/puppeteer | Deque’s API guide and releases, checked November 6, 2026. | The assertions, error handling and report formatting your team adds. |
| Newcomers | AccessLint monorepo and October 19 package record, accessio-scan source/package and announcement, A11yEco CLI documentation, Includa alpha notice, November 6, 2026. | Both-OS support, same-market ownership for accessio-scan, A11yEco first release and CLI safety; no Rising Star qualifies on established evidence. |
| wick-a11y | README configuration, reports, limits and release listing, checked November 6, 2026. | Exact dependency safety and behavior on your pages; reports do not prove greater detection coverage. |
None of this establishes how many problems any tool finds on your pages, how often it reports false problems, or how long a run takes. Every result depends on the page state, the browser and the rule version. Next substantive review is due December 18, 2026, or sooner if new documentation changes these recommendations or Pa11y CI fixes its open gate bugs. Found a changed release or behavior? Contact the editors.
Sources and evidence
Sources were checked October 19, 2026 unless another date is given. Project material establishes documented options, source behavior and published releases, not results on your site. Detection figures are their publishers’ own measurements. Download counts are the npm registry’s weekly figures for October 12 to 18, 2026.
- @axe-core/playwright and Playwright: accessibility testing guide; page.goto; web server; test agents; axe-core-npm releases, 4.14.0 on October 15, 2026; package record; Playwright Test 1.64.1 record; issue 1469; CVE-2025-59288 advisory; weekly downloads.
- Pa11y CI: README and exit codes; releases, 5.0.0 on October 5, 2026; Pa11y releases, 10.0.0 on August 28, 2026; Pa11y README and actions; navigation code; issues 538, 228, 282 and 368; weekly downloads.
- Lighthouse CI: configuration; services disclaimer; releases, 0.16.0 on November 2, 2026, checked November 6; commits; issues 1136 and 1145; Lighthouse 12.6.1 error definitions, command line and audit configuration; Lighthouse releases, 13.5.0 on September 18, 2026; accessibility scoring, updated October 22, 2025.
- IBM accessibility-checker: package README; configuration and telemetry; command-line source; releases; weekly downloads.
- axe-core CLI and axe-core: CLI README; package record; page-loading code; issues 1338 and 1403; axe-core README; axe-core 4.14.0 rule list; axe MCP Server; weekly downloads.
- What automation finds: W3C WAI on selecting evaluation tools, updated May 13, 2024; Deque coverage study, March 10, 2021, Deque’s own audits; GDS accessibility tool audit, updated April 13, 2018 and GDS blog, February 24, 2017; Sparkbox reviews of axe, December 9, 2020 and IBM, February 14, 2024; matuzo.at, May 31, 2019.
- Earlier hands-on check: method and results, October 1, 2026, a historical record from before our research-only method.
This is a researched comparison, not a product review. Method: evidence standard, editorial picks.