Jira Xray Test Management
Xray turns Jira into an audit-grade testing platform. This lesson covers the artefact hierarchy — test cases, test sets, test plans, test cycles — and how to build a traceability matrix that satisfies NZ government PIRs, bank audits, and enterprise release gates.
1 The Hook
A Wellington government agency (Services and Food Safety) completes a $4.2M licensing platform upgrade. The project passes UAT and goes live. Six weeks later, MBIE commissions a Post Implementation Review.
The PIR reviewer asks one question: "Show me which test cases covered requirement BR-047 — fee calculation for Category B permits." The test lead opens Jira and searches. There are 280 test cases across dozens of stories. Some were created as subtasks, some as checklists in story descriptions, some as comments in the UAT cycle. None are linked to the BR-047 story via Xray. Two developers spend eleven days reconstructing evidence from Slack messages and sprint notes. The PIR finding: "Testing was conducted but coverage of business rules cannot be demonstrated."
The agency gets a follow-up audit. The fix costs more than the original testing. The entire problem was structural, not technical — the testing happened but the evidence was never wired together.
2 The Rule
"In Xray, a test case without a linked requirement is invisible evidence. A test cycle without a fixed version is a report that cannot be reproduced. Structure artefacts before execution — not after."
Xray evidence is not retroactive. You cannot link a completed test execution to a requirement after the fact and have it count in a traceability report with integrity. The structure — requirement → test case → test plan → test cycle → execution — must exist before you run a single test.
3 The Analogy
Xray is a legal filing system, not a shoebox.
A barrister fighting a case in court can win with evidence kept in a shoebox — if they are lucky and have time to reconstruct it. But if the case goes to appeal, or a regulatory body demands a full disclosure within 48 hours, the shoebox fails. Xray is the filing system: every document filed, cross-referenced, dated, and retrievable by case number (requirement). The content is the same either way. The difference shows up the moment someone external demands to see the paper trail.
4 The Four Xray Artefacts
Xray extends Jira with four new issue types. Understanding the hierarchy is the foundation of everything else.
| Artefact | What it is | Jira analogy | Created by |
|---|---|---|---|
| Test | A single test case — steps, expected results, and preconditions. Reusable across multiple cycles. | Like a Story, but for testing | QA engineer |
| Test Set | A grouping of related test cases (e.g., "All login tests"). No execution — organisational only. | Like an Epic for tests | QA engineer |
| Test Plan | Defines scope for a release or sprint: which tests, which fix version, which environments. The governance container. | Like a Release in Jira | Test Lead |
| Test Execution | An instance of running tests from a Test Plan in a specific environment. Captures pass/fail/blocked per test. | Like a Sprint, but for test runs | QA engineer / tester |
The relationship flows one way: Requirement → Test → Test Plan → Test Execution → Results. The traceability matrix only works when that chain is intact.
5 Building Test Cases in Xray
Xray supports three test types. Choose based on your team's process.
Manual (Step-based)
The most common type. Each test has preconditions, numbered steps, and expected results. Xray renders these as a table during execution so the tester can mark each step pass/fail.
NZ example: IRD myIR — Test Case: "Submit personal income return with overseas income". Precondition: User is logged in, income year 2023–24 is open. Step 1: Navigate to "Income tax return". Step 1 expected: Page loads, year shows "2023–24". Step 2: Enter overseas income $8,400 NZD. Step 2 expected: Field accepts value, no validation error. And so on.
BDD (Gherkin)
For teams using Cucumber or Playwright + Cucumber, Xray can import or link BDD scenarios directly. The .feature file becomes the test case; the Xray execution records pass/fail per scenario after CI runs.
Feature: KiwiSaver voluntary contribution
Scenario: Contributor submits a voluntary top-up
Given I am logged in as a KiwiSaver member
When I navigate to "Make a payment" and enter $500
And I confirm the payment
Then I see a confirmation with payment reference
And the contribution appears in my transaction historyGeneric (Automated)
For CI-run tests where Xray receives JUnit XML results. The test case in Xray is a shell — it holds the requirement link and metadata. The actual result is imported from your test runner. This is the pattern for Playwright, Selenium, or Jest suites.
Linking tests to requirements
After creating a test, open it and add a link using the type "Tests" pointing to the Story or Requirement issue. This one step creates the traceability chain. In the requirement's Xray panel you can immediately see "Coverage: 1 test linked, 0 executed."
Tip: Link requirements to tests during sprint planning, not during execution. That way your Test Plan coverage report shows gaps before the sprint starts, not after.
6 Test Plans and Fix Versions
A Test Plan answers the question: "What are we testing in release X?" It scopes which tests are in scope and which Jira fix version the results apply to.
Why Fix Version matters: Without a fix version, Xray cannot generate a "Test Coverage by Version" report. This is the report regulators and PIR reviewers actually ask for: "What was covered in v3.2.0?" Versions are set in Jira Releases — create them before your test plan.
7 Test Cycles and Execution
A Test Execution (also called a Test Cycle in some Xray versions) is the actual run instance. Think of it as "round one of execution against the UAT environment on 14 June."
Creating and running a Test Execution
Handling defect retests
When a bug is fixed and retested, create a new Test Execution — do not re-run against the original one. This preserves the history: "Round 1: 62% pass. Round 2 (after patch): 100% pass." Auditors need the full history, not an overwritten result.
Importing automated results
CI pipelines (GitHub Actions, Azure DevOps) can POST JUnit XML to Xray's REST API. Each test class/method maps to a Test issue in Xray. The execution is created automatically after every CI run.
# Example: import Playwright results to Xray Cloud
curl -H "Authorization: Bearer $XRAY_TOKEN" \
-F "results=@playwright-results.xml;type=text/xml" \
"https://xray.cloud.getxray.app/api/v2/import/execution/junit?projectKey=ACCPAY"8 Traceability Matrix and Coverage Reports
This is why Xray exists. Three reports answer the three questions every auditor, PIR reviewer, or release manager asks.
| Report | Question it answers | When to use |
|---|---|---|
| Requirement Coverage | "Which requirements have been tested, and what was the result?" | PIR reviews, sprint retrospectives, go-no-go gates |
| Test Coverage (by version) | "For release v3.2.0, what % of tests passed across all executions?" | Release sign-off, RBNZ/APRA reporting, stakeholder dashboards |
| Traceability Matrix | "Show me the complete chain: Requirement → Test → Execution → Result → Defect" | Regulatory audits, SOC 2 evidence, ISO 27001, NZ government PIRs |
What "untested" looks like in a traceability report
Xray colour-codes requirements in the coverage view. Grey means no test exists. Blue means tests exist but no execution has run. Green means executed and passed. Red means executed and failed. A PIR reviewer looking at a grey or blue requirement at go-live is a serious finding — even if informal testing happened, it is invisible.
NZ compliance note: Under the NZ Government Enterprise Architecture Framework, major IT projects require documented test evidence for acceptance. Xray traceability reports are the de facto standard for DIA and MBIE project audits. IRD and ACC programmes now mandate Xray or equivalent tooling in their QA governance frameworks.
9 Common Mistakes
✗ Creating test cases as Jira subtasks instead of Xray Test issues
Subtasks do not appear in Xray's traceability matrix. They are invisible to coverage reports. Always create tests as the Xray "Test" issue type, even if it means a workflow change for the team.
✗ Running executions without a linked Test Plan
Ad-hoc executions that are not connected to a Test Plan cannot be associated with a fix version or release. They exist in isolation. If a PIR asks "what was in scope for v3.2.0?" the answer is unknown.
✗ Marking a FAILED test as PASS to hit a coverage target
Xray's audit log records every status change with timestamp and user. Regulators and audit tools can surface this. Document the actual result, raise the defect, and get a formal waiver if the release must proceed with a known defect.
✗ Setting up Xray after UAT has finished
Retroactively creating test structure and linking back-dated executions is high-effort and produces low-integrity evidence. Any auditor who checks timestamps will find the gap. Xray setup belongs in sprint 0, before the first test is run.
✗ One massive Test Plan for the entire project
A single Test Plan containing 800 tests is unmanageable and produces a meaningless coverage report. Create one Test Plan per release or sprint, scoped to a fix version. This makes the timeline of coverage legible across the project history.
10 Now You Try
Three AI-graded exercises. Apply the Xray concepts to NZ scenarios.
Write a complete Xray manual test case (preconditions, numbered steps, expected results) for the following scenario:
Requirement BR-112: When an IRD taxpayer submits a Working for Families (WfF) application with a household income above $180,000 NZD, the system must display an ineligibility notice and prevent submission.
For each scenario below, name the correct Xray report type and explain why.
A: An ACC programme manager wants to know if all 34 ACC levy calculation business rules were tested before the 1 April rate-change release.
B: A bank CISO needs evidence that 100% of security requirements were covered in the PCI-DSS audit cycle.
C: A sprint retrospective needs to know the pass rate for round 1 vs round 2 of regression testing.
D: A regulator asks for the complete chain from requirement to test to defect for a specific failed control.
You are the test lead for a 3-sprint MSD (Work and Income NZ) benefits portal upgrade. Sprint 1 covers identity verification, Sprint 2 covers benefit calculation rules, Sprint 3 covers notifications and document upload. Design the Xray Test Plan structure for this project, including:
- How many Test Plans, and what each one covers
- How fix versions map to plans
- What Test Sets you would create
- When Test Executions are created relative to the sprint timeline
11 Self-Check
Click each question to reveal the answer.
What is the difference between a Test Set and a Test Plan in Xray?
A Test Set is a reusable organisational grouping of test cases (like a folder). A Test Plan is a governance container that scopes which tests are in a release, links to a fix version, and produces traceability reports. Test Sets are used to organise; Test Plans are used to control and report.
Why should you create a new Test Execution for a retest rather than re-running against the original execution?
Creating a new execution preserves the full history: round 1 results (pre-fix) and round 2 results (post-fix) remain visible separately. Overwriting the original execution destroys evidence of what failed and when. Auditors and regulators need the full timeline, not the final state only.
A requirement appears grey in Xray's coverage view. What does that mean, and what should you do?
Grey means no test case is linked to this requirement. The requirement is completely uncovered — no test exists for it. You should create a test case, link it to the requirement, add it to the current Test Plan, and execute it before sign-off. Leaving grey requirements at go-live is a finding in any PIR or audit.
What must exist in Jira before you can associate a Test Plan with a specific software release?
A Fix Version must exist in the Jira project's Releases section. Without a fix version, the Test Plan cannot be scoped to a release, and the "Test Coverage by Version" report cannot be generated. Fix versions are typically created in sprint 0 or project initiation.
How does Xray handle automated test results from a CI pipeline?
The CI pipeline runs tests and produces a JUnit XML report. After the run, the pipeline calls Xray's REST API to import the XML. Xray creates a Test Execution automatically, maps each test class/method to a Test issue (by name or issue key), and records pass/fail. This makes automated results traceable to requirements just like manual ones.
12 ISTQB Mapping
ISTQB CTAL-TM (Certified Tester Advanced Level — Test Manager)
- Chapter 2 — Test Management in Context: Test management tools, test artefact organisation, traceability to requirements
- Chapter 4 — Test Monitoring, Control, and Completion: Test execution metrics, pass/fail rates, defect tracking from test runs, completion criteria
- Chapter 6 — Test Tools and Automation: Test management tool selection, CI integration, test result import
ISTQB CTAL-TA (Advanced Test Analyst)
- Chapter 3 — Structure-Based Testing: Requirement-to-test linkage, coverage measurement
ISTQB CT-TAS (Test Automation Strategies)
- Automated test result reporting, CI/CD integration with test management tools