Test Management Tools Compared
Xray, Zephyr Scale, TestRail, Azure Test Plans, and qTest side-by-side — covering Jira integration depth, BDD support, requirements traceability, CI/CD pipelines, and NZ government PIR evidence requirements.
1 Why Test Management Tools Matter
A test management tool (TMT) is more than a place to store test cases. It is the system of record that answers three critical questions when something goes wrong in production:
- Was this scenario tested? Requirement → test case traceability tells you instantly.
- What was the result? Execution history with timestamps, environments, and pass/fail outcomes.
- Who signed off? Approver names, sign-off dates, and evidence exports for audit.
Without a TMT these answers live in spreadsheets, Confluence pages, and people’s heads — none of which survive a regulator visit or a Post-Implementation Review (PIR).
Traceability
Traceability is the ability to follow a thread from a business requirement through test cases to execution results. A requirement with no linked tests is untested territory. A test with no requirement is possibly wasted effort. Good TMTs make this map explicit and reportable.
PIR Evidence
A Post-Implementation Review demands evidence that the change was tested adequately before go-live. The evidence package typically includes: a test plan, a list of executed test cases with results, defect counts by severity, sign-off by a named QA lead, and confirmation of environment and data used. Your TMT should export this in minutes, not hours.
Regulatory Compliance
New Zealand government agencies (and banks under RBNZ oversight) must demonstrate controls over change management. The New Zealand Government Security Classification System and NZISM require that test evidence is retained and auditable. Tools that lock historical execution data and enforce workflow approvals make compliance significantly easier.
2 Feature Comparison
The table below rates each tool on the features that matter most for NZ enterprise and government projects. Ratings reflect the tool as of mid-2026; all five are actively developed.
| Feature | Xray | Zephyr Scale | TestRail | Azure Test Plans | qTest |
|---|---|---|---|---|---|
| Jira integration | Native app | Native app | Plugin / REST | None (ADO only) | Plugin / REST |
| BDD / Gherkin | First-class | Basic support | Via BDD plugin | Via Azure Boards | Supported |
| Requirements traceability | Strong | Moderate | Strong | Native work items | Strong |
| CI/CD integration | REST + GitHub Actions / Jenkins | REST + marketplace | REST API + plugins | Native Azure Pipelines | REST + Jenkins / GitLab |
| Test reporting | Good; Jira dashboard-based | Good; folder-based | Excellent; built-in | Basic; Power BI needed | Excellent; enterprise |
| PIR evidence export | PDF / HTML / Confluence | PDF; limited detail | PDF / CSV; rich detail | Manual; Excel export | PDF; audit trail |
| Fix version scoping | First-class (Jira versions) | Via Jira versions | Milestones | Sprints / iterations | Releases module |
| Pricing model | Per user / Cloud or DC | Per user / Cloud or DC | Per user / Cloud or Server | Per user (Azure DevOps plan) | Per user / enterprise quote |
| NZ govt cloud eligibility | IRAP-assessed Jira Cloud | IRAP-assessed Jira Cloud | US-hosted; data residency query | Azure Australia hosted | US-hosted; data residency query |
3 Xray
Xray for Jira
Native Jira test management with first-class BDD and audit-ready traceability
Xray lives entirely inside Jira as a native app (not a plugin). Every test artefact — Test, Test Set, Test Plan, Test Execution — is a Jira issue type, which means it inherits Jira’s permissions, workflow, search (JQL), and dashboards automatically.
Core hierarchy
- Test — the individual test case (manual or automated). Can be written in plain steps or Gherkin.
- Test Set — a reusable collection of Tests (equivalent to a test suite). Tests can belong to multiple Sets.
- Test Plan — scopes a release or sprint. Links to a Jira fix version or sprint. Holds Test Executions.
- Test Execution — an execution run of Tests in an environment at a point in time. Captures pass/fail, evidence attachments, and assignee.
Fix version scoping
Xray’s Test Plan maps directly to Jira fix versions. You can generate a coverage report that shows every story in version 3.4 and its associated test execution status — a single screenshot serves as PIR gate evidence for “all stories tested before release”.
BDD / Gherkin
Xray stores Gherkin scenarios as Test issues. It can export .feature files to a repository (e.g., GitHub) and import results from Cucumber, Playwright-BDD, or Behave back into Jira. This creates a closed loop: business writes acceptance criteria in Gherkin, developers automate them, CI pushes results back to Xray, and the Test Execution issue captures the outcome against the original story.
NZ government PIR fit
Xray is the most commonly used TMT in NZ government agencies running Jira (including Inland Revenue and several Ministries). Its native fix version scoping and requirements linking mean a Test Manager can generate a full test evidence PDF — covering requirement coverage, execution results, defect links, and approver names — directly from the Test Plan. No spreadsheet assembly required.
Strengths
- Zero context switching: everything in Jira including permissions and notifications
- JQL works across test issues:
issueType = Test AND fixVersion = "3.4" AND status != Passed - BDD closed loop with CI is best-in-class
- Traceability matrix is one click from the Test Plan
- Data Centre option for agencies that cannot use cloud
Weaknesses
- Requires Jira; no standalone mode
- Large test libraries (10,000+ tests) can feel cluttered without disciplined Test Set organisation
- Built-in reporting is Jira dashboard-based; custom charts require third-party gadgets or Confluence
- Pricing stacks on top of Jira licence costs
4 Zephyr Scale
Zephyr Scale (SmartBear)
Native Jira with folder-based library organisation; better for large test catalogues
Zephyr Scale is SmartBear’s native Jira app, formerly known as TM4J. Like Xray it runs inside Jira, but its design philosophy centres on test library management using a folder tree rather than issue-type hierarchies. Teams with thousands of tests often find the folder model easier to navigate than Xray’s Set/Plan structure.
Library organisation
Zephyr Scale organises tests in a folder tree within a project. Folders map to feature areas, regression suites, or product modules. Tests sit in folders and can be included in Test Cycles (equivalent to Xray’s Test Execution). The folder model is intuitive for testers coming from a spreadsheet background.
Traceability
Requirement coverage links are available but require more manual upkeep than Xray. The coverage report shows which stories have linked tests and which Test Cycles executed them, but the link from story to execution evidence is one level more indirect. This is workable but requires discipline — in Xray, fix version scoping enforces it structurally.
Strengths
- Folder tree is intuitive for large test libraries
- Good test-case reuse with parameters and shared steps
- SmartBear ecosystem integrates with ReadyAPI and Zephyr Automated
- Reasonable BDD support for straightforward Gherkin scenarios
- Faster onboarding for teams new to test management tools
Weaknesses
- Traceability is weaker than Xray; requires manual diligence
- PIR evidence export is less detailed; may require supplementary documentation
- BDD closed-loop with CI is less mature than Xray
- Reporting is folder-centric; cross-project roll-up requires add-ons
5 TestRail
TestRail (Gurock / Idera)
Standalone tool with industry-leading reporting; preferred by NZ banks
TestRail is the longest-standing dedicated test management platform, used extensively in NZ banking (ANZ, ASB, Westpac NZ) and financial services. It operates independently of any issue tracker, integrating with Jira, Azure DevOps, and GitHub via REST API rather than running inside them.
Reporting
TestRail’s reporting engine is its strongest differentiator. Built-in report types include test plan summaries, milestone progress, activity reports, comparison across runs, and defect density charts. Every report is exportable to PDF or CSV in a format that satisfies typical audit requirements without post-processing.
Test case organisation
Tests live in sections and subsections within a project. Milestones group test runs for a release. The plan/run hierarchy is simple: a Test Plan contains Test Runs (one per environment or role), each containing a filtered subset of the test suite. Testers work through their assigned run and mark outcomes.
CI/CD integration
TestRail’s REST API is mature and well documented. Official integrations exist for Jenkins, GitHub Actions, and Selenium. Automated results can be pushed from any test runner via the API, with results appearing in the run in real time.
NZ banking context
Several NZ banks chose TestRail before Jira became dominant. It remains the tool of record for regression evidence in those organisations. If you join a bank QA team, expect TestRail, and expect to export a Milestone Progress Report as part of every release sign-off.
Strengths
- Best reporting of any tool in this comparison; audit-ready out of the box
- Works with any project management tool via REST; no lock-in
- Milestone + plan structure maps naturally to release planning
- Mature; highly stable; large community of NZ users
- Cloud and self-hosted (Server) options
Weaknesses
- Standalone means duplicate data entry if requirements live in Jira
- BDD / Gherkin requires a third-party plugin; not first-class
- No native Jira issue type; linking requires API configuration
- US-hosted cloud; NZ agencies must confirm data residency acceptability
- UI feels dated compared to native Jira tools
6 Azure Test Plans
Azure Test Plans (Microsoft)
Native Azure DevOps integration; the natural choice for full Microsoft shops
Azure Test Plans is the test management module within Azure DevOps (ADO). If your team already uses Azure Boards for work items, Azure Repos for code, and Azure Pipelines for CI/CD, then Azure Test Plans adds test management without a separate tool or licence negotiation.
Work item linking
Because everything in ADO shares the same work item model, traceability from User Story to Test Case to Pipeline run is native. A story automatically shows its linked test cases and their latest execution status. This is the tightest work-item traceability of any tool here, but only within the ADO ecosystem.
Test runner
Azure Test Plans includes a browser-based manual test runner that captures screenshots, screen recordings, and system information automatically during execution. This evidence attaches to the test result and is immediately available for PIR packages — no manual screenshot management required.
Pipeline integration
Automated test results from any Azure Pipeline task publish to Azure Test Plans automatically. The Test Results Dashboard in ADO shows pass rate trends, flaky test detection, and failure analysis without any additional configuration.
NZ government fit
Microsoft Azure Australia is the dominant cloud provider for NZ government agencies under the All-of-Government cloud agreement. Azure Test Plans inherits this eligibility, making it straightforward to justify for agencies already on the Microsoft stack. Several NZ agencies (including some in the education and justice sectors) use ADO end-to-end.
Strengths
- Zero additional tooling for teams already on Azure DevOps
- Tightest work-item traceability in the comparison
- Automatic evidence capture during manual execution
- Pipeline integration is seamless; no API configuration needed
- Azure Australia hosting satisfies NZ govt data residency
Weaknesses
- Only useful within the Azure DevOps ecosystem; no Jira integration
- Reporting is basic natively; meaningful dashboards require Power BI
- BDD / Gherkin support is indirect; relies on Azure Boards custom fields
- Test case organisation (suites within plans) can become unwieldy at scale
- Microsoft licensing model can be complex for agencies with mixed tool stacks
7 qTest (Tricentis)
qTest (Tricentis)
Enterprise-grade; purpose-built for regulated industries and scaled QA teams
qTest is Tricentis’s test management platform, positioned at enterprise teams in regulated industries: pharmaceuticals, financial services, and telecommunications. It is part of the Tricentis suite alongside Tosca (test automation) and Flood (performance testing).
Regulated industry features
qTest is designed with audit trails, electronic signatures (21 CFR Part 11 compliance), and role-based access controls that satisfy pharmaceutical and financial regulatory bodies. Every state change in qTest is logged with user, timestamp, and reason — a level of audit detail that the other tools here match only partially.
Requirements traceability
qTest Requirements module mirrors a requirements repository (Jira, Rally, or qTest itself) and maps requirements to test cases to executions to defects in a single traceability report. For regulated teams, this “golden thread” from business requirement to defect resolution is the primary audit artefact.
Scale
qTest is optimised for large QA teams (50+ testers), multiple products, and thousands of test cases. Its enterprise-tier features include cross-project reporting, role hierarchies, and integration with SAP Solution Manager and ServiceNow.
NZ context
qTest appears in NZ financial services (insurance and banking) organisations with large QA departments or where Tricentis Tosca is already in use for automation. It is uncommon in government agencies and smaller teams due to its enterprise pricing model.
Strengths
- Best audit trail and electronic signature support in the comparison
- Purpose-built for regulated industries with compliance reporting
- Integrates with Tricentis Tosca for end-to-end enterprise test management
- Cross-project reporting at enterprise scale
Weaknesses
- Enterprise pricing; not appropriate for small-to-medium teams
- Steep learning curve; onboarding requires dedicated training
- US-hosted by default; data residency requires Enterprise contract negotiation
- Overkill for teams that don’t have formal regulatory compliance requirements
8 Decision Guide
Use these heuristics to narrow your choice. Most NZ teams should reach a clear answer after two or three questions.
If your team uses…
Jira + regulatory or government traceability requirements
Choose
Xray
Native fix version scoping and BDD closed-loop make PIR evidence generation a one-click operation. Best fit for NZ government agencies and regulated Jira shops.
If your team uses…
Jira + a very large test library (5,000+ cases) where organisation is the main pain
Consider
Zephyr Scale
The folder tree scales better for massive test catalogues. Traceability is weaker, so you need disciplined processes to compensate.
If your team uses…
Any project tracker + strong standalone reporting requirements
Choose
TestRail
Best-in-class reporting that is independent of Jira or ADO. Preferred in NZ banking. Works with any issue tracker via REST.
If your team uses…
Azure DevOps end-to-end (Boards + Repos + Pipelines)
Choose
Azure Test Plans
No additional tooling needed. Native work item traceability, Azure Australia hosting, and pipeline integration out of the box.
If your team uses…
Regulated industry (pharma, financial services) with 21 CFR Part 11 or equivalent requirements
Choose
qTest
Electronic signatures and immutable audit trails are built-in, not bolted on. Worth the enterprise cost when compliance is non-negotiable.
If your team…
Has fewer than 10 testers and a limited budget
Start with
TestRail or Xray
TestRail’s free tier (up to 10 users) or Xray Cloud at low user counts are both cost-effective entry points. Avoid qTest and Zephyr Scale at small scale.
9 NZ Government Artefact Requirements
Crown agencies and local government organisations procuring or deploying digital systems are expected to produce a defined set of test artefacts. The table below maps each artefact to the tool features that produce it and notes which tools handle it natively.
| Artefact | Purpose | Xray | TestRail | Azure Test Plans | Notes |
|---|---|---|---|---|---|
| Test Strategy | Scope, approach, risk assessment | Confluence export | External document | Wiki page / Word | None of the tools generate this; it is authored separately |
| Test Plan | Release-scoped plan with environments and schedule | Native Test Plan | Milestone + Plan | Test Plan in ADO | All three produce release-scoped plans natively |
| Requirements traceability matrix | Story → test case → result mapping | One-click report | Coverage report | Work item query | Xray’s is most automatic; ADO is native but manual to format |
| Test execution results | Pass/fail per case with evidence | Test Execution issue | Run report | Test result with screenshots | ADO captures screenshots automatically during manual runs |
| Defect log | Defects found during test, with severity | Linked Jira issues | Via Jira link | Linked ADO bugs | Xray and ADO surface defects inline; TestRail requires cross-tool query |
| Sign-off / approval record | Named QA lead sign-off before go-live | Jira approval workflow | Milestone close with user | PR / gate approval | None have formal e-signature; use Jira or ADO workflow gates |
| PIR evidence package | Bundled export for post-implementation review | PDF from Test Plan | PDF from Milestone | Manual export; Excel | Xray and TestRail are fastest; ADO requires manual assembly |
| Data residency declaration | Confirm test data stays in NZ / AU | Atlassian AU data centre | US-hosted by default | Azure Australia | For Sensitive or RESTRICTED data, confirm with vendor in writing |
NZISM and GCDO expectations
The New Zealand Information Security Manual (NZISM) and the Government Chief Digital Officer (GCDO) framework do not mandate a specific test management tool, but they do require that:
- Test evidence is retained for the life of the system or a minimum of 7 years for high-risk changes
- Access to historical test results is role-controlled and auditable
- Test data containing personal information is classified and protected under the Privacy Act 2020
- Security testing (penetration testing, vulnerability scans) evidence is retained separately under the NZISM security controls
Industry Reality
- Most NZ government agencies run Jira, making Xray the default choice in practice — but many teams pay for Xray licences and then manage tests in Confluence pages instead. The tool doesn’t fix process.
- NZ banks typically run TestRail for regression suites and Jira for defects, then manually link evidence at sign-off. This works but creates reconciliation effort.
- The most common audit failure is not missing test cases but missing evidence — the test was run but the result was never recorded in the tool.
- qTest is rare in NZ outside large insurance and telecommunications companies. If you encounter it, assume the team has a formal compliance programme.
- Azure Test Plans usage in NZ is growing, driven by government cloud migrations to Azure. Expect more job descriptions to mention it from 2025 onward.