Using AI to Expand a Small Test Dataset

A small, hand‑crafted test dataset is often the first thing a QA team builds. It works for smoke runs, but it quickly becomes a bottleneck when you need to exercise edge cases, performance limits, or data‑driven business rules. Generating more data by hand is slow, error‑prone, and rarely covers the combinatorial space that real production data occupies.

AI‑assisted data generation can turn a handful of seed records into thousands of realistic rows—if you treat the output as a candidate set that still requires validation. This post walks through a practical, evidence‑mindful workflow: decide when AI helps, pick the right tool, run a worked example, verify the results, and avoid the most common traps.

---

1. Why Small Datasets Hurt Test Effectiveness

Symptom	Root cause	Impact on QA
Flaky UI tests	Only a few happy‑path records exist	Tests pass on CI but fail on staging where data shapes differ
Missed boundary defects	No records near limits (max length, negative values, nulls)	Production bugs surface only after release
Performance blind spots	Load tests run on 10‑row tables	Index usage, locking, and pagination logic never exercised
Compliance gaps	No synthetic PII, GDPR‑style masks, or locale variations	Audits flag missing test coverage for data‑privacy rules

A seed set of 20‑50 rows is typical for a new feature. That size is enough to prove the happy path, but it rarely represents the cardinality, distribution, or correlation structure of production data.

---

2. Decision Criteria – When Does AI‑Assisted Expansion Make Sense?

Situation	AI‑assisted expansion recommended?	Reason
Domain‑rich, structured data (e.g., orders, claims, medical records)	✅	LLMs and tabular generators learn column relationships quickly
Highly unstructured or binary payloads (images, protobuf blobs)	❌	Current text‑to‑tabular models cannot reconstruct binary formats reliably
Strict regulatory schema (PCI‑DSS, HIPAA)	✅ with guardrails	You can embed validation rules in the prompt or post‑process
One‑off exploratory testing	✅	Fast turnaround, low cost
Continuous data‑pipeline testing	✅ if you automate the generation step	Integrate into CI/CD as a reproducible step
Team lacks any scripting skills	✅ use a no‑code UI	Tools like QA3’s free test data generator at /tools/test-data-generator expose a prompt‑box and schema upload

Rule of thumb: If you can describe the target schema in JSON/CSV and articulate a handful of business rules in plain English, an AI generator will likely produce a useful first draft.

---

3. End‑to‑End Workflow

1️⃣  Collect seed data + schema + business rules
2️⃣  Choose generation mode (LLM prompt, tabular GAN, rule‑based synthesizer)
3️⃣  Run generation → candidate dataset
4️⃣  Validate (schema, constraints, distribution, privacy)
5️⃣  Curate / augment (remove duplicates, inject edge cases)
6️⃣  Store versioned artifact (git‑LFS, DVC, or artifact repo)
7️⃣  Wire into test harness (parameterized tests, data‑driven frameworks)
8️⃣  Monitor drift (periodic re‑generation, schema change alerts)

Each step has a gate—if validation fails, you loop back to step 2 with a refined prompt or additional seed rows.

---

4. Worked Example – Expanding an E‑Commerce Checkout Dataset

4.1 Seed Data (5 rows)

order_id	customer_id	email	items_json	total_usd	currency	created_at	promo_code
1001	42	alice@example.com	[{"sku":"A1","qty":2},{"sku":"B3","qty":1}]	79.97	USD	2024‑01‑12 09:15:00	SAVE10
1002	17	bob@shop.io	[{"sku":"C9","qty":1}]	49.99	USD	2024‑01‑12 10:03:00
1003	42	alice@example.com	[{"sku":"A1","qty":1},{"sku":"D4","qty":3}]	124.50	USD	2024‑01‑13 14:22:00
1004	88	carol@domain.org	[{"sku":"E2","qty":5}]	199.95	EUR	2024‑01‑13 16:45:00	EURO20
1005	17	bob@shop.io	[{"sku":"F7","qty":2},{"sku":"G1","qty":1}]	89.00	USD	2024‑01‑14 08:30:00

Business rules (written in plain English for the prompt):

1. order_id is a monotonically increasing integer.
2. customer_id must exist in the customers table (we have 3 known IDs).
3. email must match the customer’s registered email.
4. items_json is an array of objects with sku (from the products catalog) and qty (1‑10).
5. total_usd = sum of line‑item prices × qty, after promo discount.
6. currency ∈ {USD, EUR, GBP}.
7. promo_code optional; valid codes are SAVE10 (10 % off), EURO20 (20 % off EUR orders), FREESHIP (free shipping, no price effect).
8. created_at spans the last 90 days, with realistic peak‑hour clustering.

4.2 Prompt for an LLM‑Based Generator

You are a test‑data engineer. Produce 500 rows of synthetic checkout orders in CSV format.
Schema (PostgreSQL):
order_id BIGSERIAL PRIMARY KEY,
customer_id INT REFERENCES customers(id),
email VARCHAR(255),
items_json JSONB NOT NULL,
total_usd NUMERIC(10,2),
currency CHAR(3) CHECK (currency IN ('USD','EUR','GBP')),
created_at TIMESTAMPTZ,
promo_code VARCHAR(20) NULL


Seed rows (first 5):
[PASTE THE 5 ROWS ABOVE AS CSV]


Business rules:
1. order_id increments by 1.
2. customer_id ∈ {17,42,88}. Email must match the customer table:
   17 → bob@shop.io, 42 → alice@example.com, 88 → carol@domain.org.
3. items_json: 1‑5 line items. sku ∈ {A1,B3,C9,D4,E2,F7,G1,H5,J2}. qty 1‑10.
4. Prices (USD): A1=19.99, B3=39.99, C9=49.99, D4=31.50, E2=39.99, F7=29.50, G1=19.00, H5=44.00, J2=24.99.
   EUR prices = USD * 0.92, GBP prices = USD * 0.78.
5. Apply promo discount before total_usd.
6. created_at: random timestamp between 2024‑01‑01 and 2024‑03‑31, weighted 70 % between 09:00‑18:00 UTC.
7. No duplicate order_id. Keep CSV header.
Output only the CSV.

Tip: Paste the prompt into the free QA3 test data generator at /tools/test-data-generator. The UI lets you upload the seed CSV, paste the prompt, and download the generated file.

4.3 Generated Sample (first 3 rows)

order_id	customer_id	email	items_json	total_usd	currency	created_at	promo_code
1006	42	alice@example.com	[{"sku":"H5","qty":3},{"sku":"A1","qty":1}]	151.97	USD	2024‑01‑15 11:07:12	SAVE10
1007	17	bob@shop.io	[{"sku":"C9","qty":2}]	89.98	USD	2024‑01‑15 14:23:45
1008	88	carol@domain.org	[{"sku":"E2","qty":4},{"sku":"J2","qty":2}]	215.96	EUR	2024‑01‑16 10:11:03	EURO20

The generator respected the schema, foreign‑key‑like email mapping, and promo logic.

4.4 Validation Checklist (run automatically)

Check	Tool / Query	Pass Criteria
Schema conformity	csvlint --schema checkout.json	Zero errors
Referential integrity (email ↔ customer_id)	SELECT COUNT(*) FROM generated WHERE email NOT IN (SELECT email FROM customers WHERE id = customer_id);	0
Promo discount correctness	Custom Python script recomputes total_usd from items_json + promo	≤ 0.01 % mismatch
Currency conversion	Verify total_usd ≈ Σ(line_price_usd × qty) × fx_rate	≤ 0.5 % variance
Date range & distribution	SELECT MIN(created_at), MAX(created_at), COUNT(*) FROM generated; + histogram	Within 2024‑01‑01 – 2024‑03‑31, peak‑hour weight ≈ 70 %
Uniqueness	SELECT COUNT(DISTINCT order_id) = COUNT(*)	True
PII leakage	Scan for real‑world emails not in seed list	None found
Statistical similarity	Kolmogorov‑Smirnov test on total_usd vs. seed distribution	p‑value > 0.05 (optional)

Automate the checklist in a CI job; fail the pipeline if any gate breaks.

4.5 Curating Edge Cases

AI output is statistically correct but rarely injects deliberate boundary values. Add them manually or with a second, targeted prompt:

Add 20 rows that test limits:
- qty = 10 (max)
- qty = 1 (min)
- total_usd = 0.01 (minimum charge)
- promo_code = 'FREESHIP' on EUR order
- created_at = 2024‑03‑31 23:59:59 (end of window)
- duplicate sku within same order

Append these rows to the generated file, re‑run the validation checklist, and you now have a dataset that covers both typical and extreme scenarios.

---

5. Tool Landscape – What to Use When

Category	Representative Tools	Strengths	Weaknesses	When to Choose
LLM‑prompt generators (OpenAI, Anthropic, local Llama)	QA3 free test data generator (/tools/test-data-generator), custom scripts	Natural‑language rules, fast iteration, no training data needed	Hallucinations on numeric constraints, limited to text‑serializable formats	Schema + business rules expressible in < 500 words
Tabular GAN / VAE (CTGAN, TVAE, SDV)	sdv Python library, ydata-synthetic	Learns joint distributions, good for high‑cardinality categorical columns	Requires ≥ 1 k rows to train well; opaque; harder to inject explicit rules	You already have a moderate‑size production dump (≥ 5 k rows)
Rule‑based synthesizers (Mockaroo, Faker, Tonic)	Mockaroo UI, Faker library, Tonic.ai	Deterministic, easy to encode constraints, good for CI	Manual rule authoring; limited learning of correlations	Strict regulatory schemas, need repeatable exact values
Hybrid platforms (Gretel, Mostly AI, Synthesized)	Commercial SaaS, some free tiers	End‑to‑end privacy guarantees, schema inference, drift monitoring	Cost, vendor lock‑in, less prompt flexibility	Enterprise‑scale, compliance‑first programs

Practical tip: Start with the free QA3 generator for rapid prototyping. If you hit a ceiling (e.g., need to model complex multi‑table referential integrity), graduate to a tabular GAN trained on a masked production snapshot.

---

6. Common Pitfalls & Mitigations

Pitfall	Why It Happens	Mitigation
Hallucinated foreign keys	LLM treats customer_id as free text	Enforce a lookup table in the prompt; post‑validate with a SQL join
Numeric drift (totals not matching line items)	Model computes totals loosely	Add a re‑compute step in validation; reject rows that diverge > 0.01
Privacy leakage (real emails, credit‑card numbers)	Seed data contains PII; model memorizes	Strip PII from seed; use synthetic PII libraries (Faker) for replacement
Schema drift after model upgrade	New LLM version changes output format	Pin model version; store generated schema hash; run schema diff in CI
Over‑generation (millions of rows, storage blow‑up)	“More is better” mindset	Define a target cardinality per test suite (e.g., 10 k for integration, 100 k for load)
Ignoring correlation (e.g., high‑value orders only in EUR)	Prompt didn’t specify cross‑column logic	Explicitly state correlation rules; or train a tabular model on real data
Single‑source bias (all generated rows look like seed)	Small seed → low diversity	Inject diversity prompts (“vary shipping address formats”, “use 5‑different promo codes”)
No version control	Files land in shared drive	Store CSV/Parquet in Git‑LFS or DVC; tag with test-data/v1.3

---

7. Embedding the Generated Data in Your Test Suite

# pytest fixture – parameterized by generated CSV


import pandas as pd
import pytest


@pytest.fixture(scope="session")
def checkout_orders():
    df = pd.read_parquet("testdata/checkout_orders.parquet")
    return df.to_dict(orient="records")


def test_checkout_total_matches_items(checkout_orders, api_client):
    for order in checkout_orders:
        resp = api_client.post("/checkout", json=order)
        assert resp.status_code == 200
        assert abs(resp.json()["total_usd"] - order["total_usd"]) < 0.01

Benefits