Site compatibility testing for Chrome extensions

When an AI writing extension breaks on WhatsApp Web, a proprietary CMS, or a company intranet, the developer has almost no way to know until users start complaining. By then, the user has already reinstalled twice and filed a frustrated support ticket — or quietly stopped using the product. One user in this dataset explicitly says the extension 'doesn't work on some sites' and they just gave up and switched to a competitor.

The root cause is structural: Chrome extension behavior is deeply dependent on the Content Security Policy of the host page. Some sites block inline scripts, some override DOM events, some throttle or break service workers. Testing an extension against the real web is not something a QA team can do manually at scale, and existing automated browser testing tools like Playwright or Selenium don't model the specific failure modes of Chrome extension injection — they test the extension in isolation, not how it degrades on hostile third-party pages.

Extension developers currently have no way to run a pre-release check that says 'your new build breaks on these 12 domains your users visit most.' They ship, users break, support tickets spike, someone manually repros it, and a patch ships two weeks later. The cycle repeats with every extension update.

This is a business because the cost of a broken extension update is concrete: it triggers a wave of uninstalls, some of which are permanent, and support volume spikes immediately. For a company charging $40/month per seat, even 2% churn from a bad extension release is measurable. And it recurs with every Chrome update, every Manifest V3 change, every site redesign — the need doesn't go away after one fix.

What to build

Build a CI-integrated test runner that loads a Chrome extension build against a configurable list of real URLs in headless Chrome, detects injection failures, service worker crashes, and DOM conflict signatures, and returns a per-domain pass/fail report before the extension is pushed to the Chrome Web Store.

Where to start

Target teams already using GitHub Actions for their extension builds — offer a pre-built GitHub Action that runs the compatibility check on every pull request, so adoption requires a single config file change rather than a new workflow.

The hard part

Headless Chrome has known gaps in extension support — some failure modes only reproduce in a real browser session, which means the test environment has to run full Chrome instances, making the infrastructure expensive and slow to scale to hundreds of test URLs per run.

How it makes money

Usage-based pricing tied to the number of URLs tested per month and the number of extension builds run, with a free tier covering 50 URLs and 10 builds to drive adoption among smaller teams.