Synthetic Data Engineering
Replace brittle production dumps with AI-generated datasets that preserve referential integrity, pass privacy laws, and expose the edge cases real data misses.
1 The Hook — Why This Matters
An Auckland health-tech startup spent weeks trying to anonymize production patient records to test a new ML algorithm. They missed one secondary table containing home addresses. A junior developer accidentally committed a subset of the test database to a public GitHub repo. The resulting NZ Privacy Act breach cost them a major enterprise client and significant legal fees.
Relying on anonymized production data is a legal minefield and an engineering bottleneck. In 2026, Senior QA Engineers don't wait for the DBA to "scrub a dump." They use AI models and declarative tools to generate structurally identical, high-volume synthetic datasets on demand.
2 The Rule — The One-Sentence Version
Treat test data as infrastructure: generate it synthetically, version control the schemas, ensure referential integrity, and never use production PII (Personally Identifiable Information) in lower environments.
Synthetic data is not "fake data." It is engineered data that maintains the exact statistical distributions and relational constraints of real data, but with zero privacy risk.
3 The Analogy — Think Of It Like...
Crash test dummies vs. human stunt doubles.
You wouldn't use a real person to test a new airbag just because they are "more realistic." You engineer a crash test dummy that has the exact weight, joint flexibility, and sensors required to measure the impact, without any of the ethical or legal risks. Synthetic data is your crash test dummy for software. It behaves exactly like the real thing under stress, but nobody gets hurt if it leaks.
Senior engineer insight
The moment that changed how I think about synthetic data was watching a load test collapse because our generator had silently capped transaction amounts at $9,999 — a hard-coded Faker default nobody noticed. The app handled six-figure transactions in production daily, and we'd never exercised that code path once. After that, I started treating the generation schema as a first-class test artefact: peer-reviewed, committed alongside migrations, and explicitly asserting distribution ranges before a test run begins.
Most common mistake: teams write a generator once, never update it when the production schema evolves, and quietly end up testing a data shape that no longer matches reality.
From the field
A Wellington-based team building a payments platform for a NZ bank assumed their anonymised production slice would cover all meaningful test scenarios — after all, it held six months of real transactions. What they discovered mid-integration was that their synthetic supplement (hastily bolted on when the Privacy Act concern was raised) generated Revenue NZ numbers using a simple random-9-digit function, not the actual Revenue NZ check-digit algorithm. Every test involving Revenue NZ validation was silently passing against invalid numbers. The fix was straightforward once found, but it had masked a real validation bug that reached UAT. The generalised lesson: if your domain has structured identifiers — Revenue NZ numbers, CoverNZ claim codes, NHI numbers, bank account suffixes — your generator must encode the same validation rules the application uses, or you are testing with data the app would never see in production.
4 Watch Me Do It — Step by Step
Moving from hardcoded fixtures to synthetic data engineering requires a pipeline approach:
- analyse the Production Schema (Metadata Profiling)
You don't need the actual data; you need the shape of the data. Use profiling tools to extract constraints, foreign keys, and statistical distributions (e.g., 80% of users are active, 20% are suspended).
- Define the Generation Rules (Declarative Logic)
Use a framework (like Synthea for healthcare, or Faker/Mimesis mapped through an LLM for custom business logic) to declare how rows relate.
# Example: Generating linked relational data in Python from mimesis import Generic from mimesis.schema import Field, Schema fake = Generic('en') _ = Field('en') def user_with_transactions(): user_id = fake.uuid() return { "user_id": user_id, "name": _('full_name'), # Ensure age distribution matches production (e.g., normal distribution) "age": _('integer_number', start=18, end=85), "transactions": [ {"txn_id": fake.uuid(), "user_id": user_id, "amount": _('price')} for i in range(fake.random.randint(0, 5)) ] } - Amplify Edge Cases
Production data rarely contains the exact edge cases you need for a new feature (e.g., a user with a negative balance on a leap year). AI-driven synthetic generation allows you to deliberately skew the dataset to over-represent edge cases for rigorous testing.
- Pipeline Integration
The generation script runs as a step in the CI/CD pipeline, seeding the ephemeral database (created by Terraform) before the test suite executes.
5 When to Use It / When NOT to Use It
✅ Use Synthetic Data when...
- Testing systems that handle PII, financial, or medical records.
- You need to load-test a database with millions of records.
- You need to test machine learning models for bias or drift.
❌ Skip Synthetic Data when...
- Running fast, isolated unit tests (use simple mocks).
- Testing purely static configurations or public, non-sensitive reference tables.
Before building a synthetic generation pipeline, ask:
- Record Volume: How many rows do you need? If fewer than 1,000, handwritten fixtures may be faster. If millions, synthetic generation pays for itself through automation.
- Constraint Preservation: Are there foreign key relationships, uniqueness constraints, or date ranges that must be preserved? If yes, the generator must encode these rules explicitly.
- Ownership & Maintenance: Who owns the generation pipeline if production schema changes? Can your team maintain custom generation rules, or do you need a commercial tool like dbt or Great Expectations?
6 Common Mistakes — Don't Do This
🚫 Breaking Referential Integrity
I used to think: I can just generate a CSV of users and a CSV of orders.
Actually: If the user_id in the orders table doesn't map to a valid user_id in the users table, your database will throw foreign key constraint errors. Synthetic data pipelines must generate relational graphs, not just isolated tables.
🚫 Ignoring Data Distribution
I used to think: Random data is fine for testing.
Actually: If your real app has 90% free users and 10% premium users, but your test data is a 50/50 split, performance tests will yield inaccurate queries, and ML models will overfit. Synthetic data must map to the statistical reality of production.
🚫 Losing Edge Cases When You Scale
I used to think: Just run my generator and multiply the record count by 10.
Actually: A generator that works for 100 rows often breaks at scale. You'll discover missing cardinality constraints, incomplete enum lists, or date range boundaries that don't hold once you hit a million records. Test your generator at multiple scale levels (100, 10K, 1M) before relying on it in CI/CD.
⚠️ When It Fails
Synthetic data engineering assumes backend constraints (foreign keys, NOT NULL, CHECK clauses) are fully defined in your schema. If the backend database lacks integrity enforcement, your generator will produce data that looks correct but breaks at insert time. Worse, if the backend is deployed before the schema—a common pattern in rapid releases—your synthetic generator becomes obsolete. Always version the schema and the generator together. Also, if you're generating millions of rows but only retaining data for 30 days, ensure your pipeline cleans up old synthetic data so it doesn't pile up in snapshots or backup chains.
7 Now You Try — Interview Warm-Up
Scenario: You are tasked with populating a staging database for a new HR application. The production database contains highly sensitive salary and medical leave data. A developer suggests writing a script to copy production data, but replacing the "Name" column with random strings.
Why is this dangerous, and what is the engineering alternative?
The risk & solution:
Simply masking the name does not anonymize data. A user could easily be identified by cross-referencing their unique salary, job title, and zip code (Inference Attack), breaching the NZ Privacy Act. The alternative is Synthetic Data Generation. You profile the statistical distribution of salaries and titles in production, and use an AI model or script to generate entirely fictional HR records that match the math but belong to nobody.
Why teams fail here
- Generator drift: The generation schema is written once and never updated. When production adds a NOT NULL column or changes an enum, the generator keeps producing stale data that bypasses the new constraint entirely — tests pass, the bug ships.
- Faker defaults masquerading as domain data: Faker's built-in
phone_number(),ssn(), oriban()helpers produce internationally generic formats. NZ-specific identifiers (NHI, Revenue NZ, account number suffixes, TransitNZ licence formats) need custom providers — skipping this produces structurally invalid data the application rejects or, worse, silently accepts. - Scale assumptions baked in at 100 rows: Cardinality rules that look fine at 1,000 records break catastrophically at 1 million — duplicate primary keys from UUID collisions in naively seeded generators, enum values exhausted, date ranges overflowing. Generators must be tested at production-representative volumes before being trusted in CI/CD.
- Treating synthesis as a one-time data dump: Teams generate a dataset once, commit the output file, and then reuse it for months. The dataset becomes stale, stops covering new code paths, and the generation pipeline atrophies. Synthetic data should be regenerated fresh on every pipeline run from a versioned schema — the output is ephemeral, the generator is the artefact.
Key takeaway
Synthetic data engineering is not about hiding real data — it is about designing test data with the same rigour you would bring to designing the system itself: versioned, constraint-aware, domain-accurate, and continuously maintained alongside the schema it models.
8 Self-Check — Can You Actually Do This?
Click each question to reveal the answer. If you got all three, you're ready to practice.
Q1. What is the difference between Data Masking and Data Synthesis?
Data masking alters existing real data (e.g., scrambling a name), which is often reversible. Data synthesis creates brand new data based on statistical models, guaranteeing zero privacy leakage.
Q2. Why is referential integrity difficult in synthetic data generation?
Because generating tables in isolation breaks foreign keys. Generators must be stateful—remembering the IDs generated in a parent table (Users) to use them when generating child tables (Orders).
Q3. How does synthetic data assist in testing edge cases?
Production data may not contain the specific rare edge case you need. Synthetic generation allows you to explicitly write rules to over-represent rare scenarios (e.g., leap year birthdays, negative balances) to ensure the system handles them.
9 Interview Prep — Real Conversations
Senior engineers discuss synthetic data decisions in interviews. Here's what they ask and how to answer:
Q: When would you choose data masking over synthesis, and when the opposite?
Masking (e.g., replacing the name "John Smith" with "XXXXX") is faster for test environments where you have working data and just need to hide PII for speed. Synthesis is superior for compliance (NZ Privacy Act, GDPR) because masking is often reversible through inference attacks. Real answer: I'd use synthesis for production-related testing, masking only for dev/exploratory work where speed matters more than legal safety.
Q: You're generating 10 million synthetic customer records with 20 related order records per customer. How do you ensure referential integrity at that scale?
I'd build a stateful generator that remembers customer IDs as it generates them, then uses those IDs to generate orders. In Python, I'd use a class that accumulates IDs in memory (or writes them to a temporary index) and reads from that index during child table generation. For massive scale (100M+ records), I'd partition the work: generate customers 0-10M in shard A, generate their orders. Then repeat for shard B. This avoids memory overload and allows parallel generation.
Q: Your ML model training pipeline uses synthetic data. How do you know if the synthetic distribution is close enough to production?
I'd compute statistical measures like Kolmogorov-Smirnov tests on continuous columns and chi-square tests on categorical columns, comparing synthetic vs. real distributions. I'd also train two models side-by-side—one on real production data, one on synthetic—and compare their performance on the same validation set. If they diverge significantly, the synthetic distribution is wrong. For NZ fintech specifically, I'd ensure salary ranges, user segments, and transaction patterns match real customer cohorts.
Q: A teammate says: "Let's just snapshot the production database monthly and use that for testing instead of maintaining a generator." How do you push back?
I'd highlight three problems: (1) You're storing PII in version control or backup systems—legal liability under NZ Privacy Act. (2) If a schema changes, your old snapshot breaks. (3) You can't test edge cases that production data doesn't naturally contain (e.g., a user with a negative balance or a transaction on a leap second). Synthesis scales these problems away. I'd propose a middle ground: snapshot schemas, not data. Use the schema snapshot to drive a generator, not raw data.