Multi-Vendor Programme Testing
Most large NZ technology programmes involve at least three vendors. Nobody owns the end-to-end. The Test Manager who understands how to govern quality across vendor seams is the one the Programme Director calls first when things go wrong.
1 The Hook
A large NZ Crown entity runs a programme to migrate to Oracle Fusion Financials. Three vendors: Oracle (the software vendor), a system integrator handling configuration and deployment, and a managed services provider owning infrastructure and middleware. The integration between Oracle Fusion and the legacy HR system runs through a custom middleware layer the MSP built.
Seven weeks from go-live, integration testing reveals that employee records from the legacy HR system are duplicating in Oracle. Oracle says it's the middleware. The MSP says it's Oracle's API. The SI says it's the data extract. Six weeks of defect disputes follow — three vendors, three lawyers, and a Programme Director who's running out of patience.
The Test Manager had no integration test ownership matrix — no document defining which vendor tests which seam. The go-live slips 10 weeks. The duplicate-record issue was present in the test environments for four weeks before anyone owned it. That's a process failure, and it was avoidable.
2 The Rule
"In a multi-vendor programme, the test manager owns the seams. Each vendor tests their own component. Nobody owns the joins — unless you make someone. That someone is you."
Senior engineer insight
The moment I started treating the integration test ownership matrix as a contractual instrument — not just a planning spreadsheet — everything changed. I once handed a matrix to a Programme Director mid-dispute and watched three weeks of vendor blame-shifting collapse into a 48-hour resolution, because the matrix had been signed off by all vendor leads at programme start. If it is not signed, it is opinion; if it is signed, it is obligation.
The most common mistake: treating vendor test completion reports as integration evidence when they only confirm the component works in isolation — the seam is always someone else's problem until you make it yours in writing.
From the field
On a Tier 1 NZ central government SI programme — three vendors, an Oracle ERP core, a cloud MSP, and a local SI handling customisations — the team assumed the SI's SoW covered end-to-end integration testing. Six weeks into SIT, a payroll file from the legacy HR system was arriving at Oracle HCM with a field delimiter mismatch that caused silent data truncation on employee names longer than 20 characters. No vendor owned the interface: the MSP owned the file transfer, the SI owned the Oracle ingestion mapping, and the agency's IT team owned the HR extract. Each had passed their own test. The issue sat open for four weeks while the vendors traded incident logs. What resolved it was a forced triage session where we walked the data flow end-to-end in a shared environment with all three vendor leads present simultaneously. The lesson: when an interface crosses more than two vendor boundaries, you cannot triage it asynchronously — you need everyone in the room at the same time with live data flowing, or you will never find the owner.
3 The Analogy
A multi-vendor programme is like building a house where the architect, builder, plumber, and electrician all work to different plans.
Each tradesperson does their piece correctly. But if no one tests whether the pipes fit where the builder left holes, and whether the electrician's conduit clears the plumber's runs, you get a disaster at handover. The Test Manager is the building inspector who tests the joins, not the individual work. Without you, every trade assumes someone else checked the interface.
4 Watch Me Do It
Here's how to build an integration test ownership matrix. This is the first artefact you produce when you join a multi-vendor programme.
The matrix structure
| Interface | Source | Target | Data Owner | Test Owner | Approach | Environment |
|---|---|---|---|---|---|---|
| GL → Reporting | Oracle Fusion | Power BI | Oracle | SI | API mock + E2E | SIT3 |
| Employee sync | HR Legacy | Oracle HCM | MSP | QA team (agency) | File-based SFTP | SIT2 |
| Orders | Salesforce | Oracle Financials | Oracle | SI | REST API contract test | SIT3 |
Running the vendor call to establish the matrix
Call all vendors into a single session. Start with a blank version of the matrix on screen. Go interface by interface and ask: "Who owns the data coming in? Who writes the test? What environment do we use?" Disagreements surface immediately — which is exactly the point. Document every objection. Any interface where two vendors both say "not us" is your highest-risk seam.
What to do when vendors dispute defects
Document a defect triage protocol at programme start. When a defect is raised on a seam, the protocol specifies: who owns the investigation (usually the team that controls the interface that is failing), the investigation timeline (48 hours), and what evidence is required to transfer ownership (logs, API traces, schema exports). Without this, every dispute becomes a time-consuming negotiation instead of a process.
5 When to Use It
- Any programme with more than two technology vendors
- When integration defects are being disputed across teams with no resolution process
- When the programme has a system integrator who is different from the software vendor
- When environment pipelines cross vendor boundaries — especially where environments are not jointly managed
- When joining a programme mid-flight and needing to understand who owns what before taking on risk
6 Common Mistakes
⚠ "The SI owns all integration testing because they integrated everything."
I used to think: The SI owns all integration testing because they integrated everything.
Actually: The SI owns the configuration; they don't necessarily own the interfaces. Clarify scope in the contract before testing starts. A system integrator who configured Oracle Fusion has no obligation to test the middleware layer owned by the MSP unless that is written into their SOW.
⚠ "I can trust each vendor's test results and aggregate them."
I used to think: I can trust each vendor's test results and aggregate them into a programme view.
Actually: Vendor test results cover their component in isolation. The seams between components are not in any vendor's scope unless you explicitly put them there. A component that passes unit tests and passes system tests can still fail integration tests at the join.
⚠ "Defect disputes between vendors are the Programme Manager's problem."
I used to think: Defect disputes between vendors are the Programme Manager's problem to sort out.
Actually: Defect ownership disputes block go/no-go decisions. The Test Manager needs a defect triage protocol that assigns resolution ownership with a timeline — not just blame. Without it, disputes sit open for weeks while go-live pressure mounts.
7 Now You Try
Send the prompt below to a real AI. Evaluate the response: does it produce a complete matrix with all required columns? Does it account for ownership disputes?
Why teams fail here
- The integration test ownership matrix is created but never signed off by vendor leads, so vendors treat it as advisory and dispute ownership at the first defect
- Defect triage has no defined investigation SLA — defects raised on seams sit in "under investigation" for weeks while go-live pressure mounts and Programme Directors push for workarounds instead of fixes
- Test environments are not jointly owned or synchronised across vendors, so SIT results are invalidated when one vendor updates their environment without notifying others — a common failure pattern in NZ SI programmes where the MSP controls the infrastructure layer independently
- Coverage metrics are reported per vendor rather than per interface, hiding the fact that the highest-risk seams have no test coverage while individual component pass rates look healthy
Key takeaway
In a multi-vendor programme, the Test Manager's job is not to test the systems — it is to own the joins that every vendor assumed someone else was covering.
Enterprise reality
SI programmes with 5+ vendors, shared environments, and contractual test obligations
- Test ownership boundaries are defined in contracts before the programme starts — at scale, verbal agreements at kickoff meetings are worthless; every seam must be assigned in the SoW or the master test plan before vendors mobilise
- Integration testing across 5+ vendors requires a neutral test coordinator — typically the client QA team or a separate assurance partner — because no individual vendor can be trusted to objectively report defects at seams they share with a commercial competitor
- Shared test environments must be governed by a joint change freeze protocol — one vendor refreshing their environment without notifying others invalidates in-flight SIT cycles and is the single most common cause of test delays in large NZ SI programmes
- Vendor test evidence is reviewed and formally accepted by the client QA team before it counts toward go/no-go — at enterprise scale, vendor self-certification is not sufficient; evidence review is a gate, not a formality
8 Self-Check
Click each question to reveal the answer.
Q1: What is the first artefact you create when you join a multi-vendor programme as Test Manager?
An integration test ownership matrix. This document maps every interface between systems to a named test owner, test approach, and environment. It is the foundation for everything else — without it, you have no visibility into who is responsible for testing the joins, and no basis for defect triage when disputes arise.
Q2: A vendor says "we've passed our unit tests" and won't participate in integration testing. How do you respond?
Refer to their contract. Most vendor SOWs include integration test participation as an obligation — "unit tested" does not satisfy "integration tested." If the contract is silent, escalate to the Programme Manager and document the risk: the vendor's component is untested at the seam level. That risk sits with the programme, not with you personally. Get the gap in writing.
Q3: How do you decide which integration interfaces are highest risk and should be tested first?
Use three factors: (1) data criticality — interfaces carrying financial, health, or identity data fail with higher consequence; (2) ownership ambiguity — interfaces where two vendors both say "not us" are highest risk because disputes slow resolution; (3) volume and frequency — high-frequency interfaces with large data payloads have more failure surface area. Test ambiguous ownership interfaces first — they take the longest to fix.
9 ISTQB Mapping
CTAL-TM v3.0 — Section 3.2: Test Planning — Integration Test Scope
Defines how Test Managers scope integration testing across system boundaries. The integration test ownership matrix is the practical artefact that implements this section in a multi-vendor context. CTAL-TM requires the Test Manager to explicitly plan for seam testing, not assume it is covered by vendor component testing.
CTAL-TM v3.0 — Section 3.4: Test Monitoring and Control
Applies directly to multi-vendor defect triage. CTAL-TM v3.0 Section 3.4 covers how to monitor progress and control quality when testing spans multiple teams. The defect triage protocol — assigning investigation ownership with a timeline — is the control mechanism.
CTAL-TM v3.0 — Section 5.1: Metrics for Test Progress Reporting
In multi-vendor programmes, standard coverage metrics are insufficient — you need per-vendor and per-seam metrics. Section 5.1 covers how to construct metrics that give the Programme Director visibility across all vendor streams, not just an aggregate that hides which seam is failing.