Compatibility Testing
Your application works on your machine. The question is whether it works on your user's machine — their browser, their OS, their screen size, their accessibility settings.
1 The Hook
An Auckland recruitment platform has 80% of applications completed on desktop Chrome. The remaining 20% use mobile — mostly Safari on iPhone — and older Android Chrome. The platform launches a new CV upload feature.
Desktop Chrome: works perfectly. iPhone Safari: the file picker doesn't open — a known WebKit API difference. Older Android: the drag-and-drop fallback doesn't trigger. Both issues are in production for 3 weeks before users flag them.
No cross-browser testing was run before launch. The 20% of users who can't submit applications don't call support — they apply elsewhere. The platform never knows what it lost.
2 The Rule
Define your browser/device matrix before writing test cases, not after. Every organisation has a set of 'must support' platforms — test those platforms, not just the developer's laptop.
3 The Analogy
Compatibility testing is like a power adaptor.
Your laptop charger works perfectly in New Zealand. Plug it into a UK wall socket without an adaptor and nothing happens. The charger isn't broken — it's incompatible with that environment. The application that works in Chrome on a MacBook Pro may be equally incompatible with Safari on an iPhone 12 — and the environment is just as real.
4 Watch Me Do It
Step 1: Build the browser matrix from analytics data. Pull 90-day browser/OS data from Google Analytics or Plausible. Sort by session count descending. Any platform above 5% of sessions is "must test". 1–5% is "should test". Below 1% is "document and risk-accept".
| Browser | OS | Device | Priority | Notes |
|---|---|---|---|---|
| Chrome (latest) | Windows 10/11 | Desktop | MUST | Largest NZ desktop share |
| Chrome (latest) | macOS | Desktop | MUST | |
| Edge (latest) | Windows 10/11 | Desktop | MUST | Default in many NZ agencies |
| Safari (latest) | iOS 16/17 | iPhone | MUST | WebKit; file input behaviour differs |
| Safari (latest) | macOS | Desktop | MUST | ~15% of NZ Mac users |
| Chrome Mobile | Android 12+ | Phone | MUST | Budget Android devices common |
| Firefox (latest) | Windows | Desktop | SHOULD | Privacy-conscious users |
| Chrome (latest) | Android 9–11 | Phone | SHOULD | Older devices in lower-income demographics |
Step 2: Categorise findings from BrowserStack.
- True defect — behaviour violates the spec regardless of browser. Fix in code.
- Environment-specific workaround — CSS prefix or polyfill needed. Document, implement, retest.
- Won't fix / risk-accept — platform below threshold, fix cost too high. Document decision with Product.
5 When to Use It
- Before any feature launch involving file interaction, drag-and-drop, camera access, or CSS animation
- When analytics show more than 10% of users are on non-Chrome browsers
- When targeting NZ government or enterprise users — Safari is mandated in many NZ agencies on managed Mac devices
- When the user base spans age groups — older demographics more likely on IE-era browsers or older Android
- Any time the design system introduces new CSS features (grid, custom properties, container queries)
6 Common Mistakes
❌ I used to think: Chrome is all I need to test.
Actually: 15–25% of NZ users are on Safari (iOS and macOS combined). Safari uses WebKit, which has distinct behaviour for file input, scroll events, CSS backdrop-filter, and form autofill. If you only test Chrome, you're skipping a quarter of your users.
❌ I used to think: we can test compatibility after launch.
Actually: compatibility defects are expensive to fix post-launch. CSS or JS changes needed for one browser can break others — you're in regression territory immediately. Run compatibility testing in parallel with development, not after go-live.
❌ I used to think: BrowserStack replaces real device testing.
Actually: BrowserStack catches most compatibility issues and is the right tool for broad coverage. But touch interactions, performance on low-spec hardware, and some iOS-specific behaviours (especially around permissions and background tabs) need real device validation for critical user journeys.
7 Now You Try
Build a compatibility test matrix for a NZ health app used by GPs and practice nurses. The app must work on: Windows 10 PCs (typically 5 years old), iPads (iOS 16+), and occasionally on personal Android phones. Define the browser/OS/device matrix and the 3 highest-risk compatibility scenarios to test.
8 Self-Check
Click each question to reveal the answer.
Q1: What is the primary reason Safari on iOS requires specific compatibility testing, separate from Chrome on Android?
Safari on iOS uses WebKit, which has different implementations of file input APIs, scroll event handling, form autofill, and CSS properties compared to Chromium-based browsers. Additionally, Apple requires all browsers on iOS to use WebKit — so Chrome on iPhone is also WebKit under the hood, meaning Safari testing covers all iOS browsers.
Q2: How do you determine which browser/OS combinations belong in the "must test" tier?
Pull 90-day analytics data and sort by session count. Any platform above 5% of sessions is "must test". 1–5% is "should test". Below 1% is risk-accept with documented decision. Always cross-check with the product's target audience — government procurement may mandate specific platforms regardless of analytics share.
Q3: A compatibility defect found post-launch affects 8% of users on a specific browser. What are the two possible classifications and how do you decide which applies?
It's either a true defect (the code violates the spec and the fix applies across all browsers) or a browser-specific workaround (the code is spec-compliant but the browser renders it differently, requiring a prefix, polyfill, or conditional CSS). Check whether other browsers behave correctly against the spec — if yes, it's a workaround; if all browsers behave the same wrong way, it's a true defect.
9 ISTQB Mapping
ISTQB CTAL-TA v3.1.2, Section 3.3.4 — Non-functional testing: compatibility testing techniques. Testers at Advanced level are expected to define compatibility matrices based on risk, select tools appropriately, and categorise defects correctly.
Also relevant: ISO/IEC 25010 portability characteristic (co-existence, adaptability, installability), which defines the quality model underpinning compatibility as a testable attribute.