AI Test Data Generation Metrics: Validity, Diversity, and Coverage

Generating test data with large language models (LLMs) or specialised synthetic‑data engines has moved from “experiment” to “pipeline step” for many QA teams. The promise is simple: feed a schema or a few natural‑language rules and get thousands of rows that look like production data. The reality is messier. A data set that looks right can still hide schema violations, miss edge‑case combinations, or over‑represent a narrow slice of the domain.

To turn AI‑generated data into a reliable testing asset you need observable, repeatable metrics. This post walks through the three pillars—Validity, Diversity, and Coverage—shows how to measure them, and gives a concrete workflow you can adopt today.

---

Why Metrics Matter

Situation	What goes wrong without metrics	What a metric tells you
Schema drift	Generated rows silently violate a new NOT‑NULL column added last sprint.	Validity drops → CI fails fast.
Blind spots	All generated users are “premium” tier; free‑tier logic never exercised.	Diversity reveals class imbalance.
Requirement gaps	No rows exercise the “discount‑code + gift‑card” combo required by a new promotion.	Coverage flags missing combinatorial paths.

Metrics turn “looks good” into a gate that can be automated, versioned, and reviewed.

---

The Three Pillars

1. Validity – Does the data obey the contract?

Definition: The proportion of generated records that satisfy all hard constraints (type, range, foreign‑key, CHECK, NOT‑NULL, enum, custom business rules).

Typical checks

Constraint type	Example	Automated check
Data type / format	email matches RFC 5322 regex	Regex validation
Range / bounds	age ∈ [0, 120]	Simple numeric compare
Referential integrity	order.customer_id exists in customers	Join against reference table
Business rule	discount_pct ≤ 100 when is_employee = false	Custom predicate function
Uniqueness	sku unique across products	COUNT(*) = COUNT(DISTINCT sku)

Metric formula

Validity = (valid_rows / total_rows) × 100%

A target of ≥ 99.5 % is common for CI‑gate data; anything lower usually indicates a prompt or schema mismatch.

---

2. Diversity – Does the data explore the input space?

Definition: A set of statistical distances that capture how broadly the generated rows populate the domain of each column and the joint distribution across columns.

Key sub‑metrics

Sub‑metric	What it measures	Typical calculation
Category balance	Frequency of each enum / categorical value	max(count) / min(count) (ideal ≈ 1)
Numerical spread	Coverage of min‑max, quantiles, outliers	Kolmogorov‑Smirnov distance vs. reference distribution
Pairwise correlation drift	Whether known correlations (e.g., high_value_order ↔ premium_customer) are preserved	Pearson / Spearman diff
Rare‑event presence	Appearance of low‑probability combos (e.g., payment_method = crypto + currency = JPY)	Count of rows matching rare‑event predicate

Metric formula (example)

Diversity_score = 1 - ( Σ |p_gen(c) - p_ref(c)| ) / 2

where p_gen and p_ref are the empirical categorical distributions. The score ranges 0–1; > 0.85 usually signals a healthy spread for most test suites.

---

3. Coverage – Does the data hit the test‑case space?

Definition: The fraction of required logical paths, state transitions, or combinatorial tuples exercised by the generated data set.

Coverage models

Model	When to use	How to compute
Requirement‑traceability matrix	Regulated domains (medical, finance)	Map each requirement → data predicate; count satisfied predicates
t‑wise combinatorial coverage	Complex UI forms, API parameter matrices	Generate all t‑wise tuples (t=2 or 3) and check presence
State‑machine coverage	Workflow engines, order lifecycles	Simulate transitions; verify each transition appears at least once
Mutation‑kill coverage	Mutation testing pipelines	Run mutants against generated data; compute kill ratio

Metric formula (t‑wise example)

Coverage_t = (covered_tuples / total_tuples) × 100%

A pragmatic goal: ≥ 90 % pairwise (t=2) coverage for the most critical modules; higher t‑wise levels are optional and costlier.

---

Measuring the Metrics in Practice

1. Define a Reference Profile

Before you can score generated data you need a baseline. The reference can be:

• A production snapshot (masked / sampled)
• A hand‑crafted “golden” data set used for regression
• A statistical model derived from historic logs

Store the reference as Parquet or CSV plus a JSON schema that enumerates constraints, enums, and known correlations.

2. Build a Metric Harness

A lightweight Python (or Kotlin/Go) script that:

1. Loads generated data + reference + schema.
2. Runs the validity checks (SQL‑style CHECK statements, foreign‑key joins).
3. Computes diversity statistics (categorical balance, KS distance, correlation drift).
4. Evaluates coverage predicates (requirement matrix, t‑wise tuples).
5. Emits a JSON report and a Markdown summary for CI artifacts.

# pseudo‑code sketch


def run_harness(gen_path, ref_path, schema_path):
    gen = pd.read_parquet(gen_path)
    ref = pd.read_parquet(ref_path)
    schema = load_schema(schema_path)


validity = check_validity(gen, schema)
    diversity = compute_diversity(gen, ref, schema)
    coverage  = compute_coverage(gen, schema)


report = {
        "validity": validity,
        "diversity": diversity,
        "coverage": coverage,
        "timestamp": datetime.utcnow().isoformat()
    }
    write_json(report, "metrics_report.json")
    write_md_summary(report, "metrics_summary.md")

Run this harness as a post‑generation step in your CI pipeline. Fail the build if any metric falls below its threshold.

3. Version the Prompt / Config

Treat the prompt (or the synthetic‑engine config) as code. Store it in Git, tag each release, and include the metric report as an artifact. This gives you a traceable lineage:

prompt@v3.2 → generated_data_2024-03-15 → metrics_report.json (validity 99.8%, diversity 0.88, coverage 92%)

When a metric regresses, you can bisect the prompt history instead of guessing.

---

Worked Example: E‑Commerce Checkout Flow

Domain snapshot

Table	Key columns	Business rules
customers	id, tier (free/premium/enterprise), email, country	tier=enterprise ⇒ credit_limit ≥ 100 000
orders	id, customer_id, total, currency, discount_code, payment_method	discount_code only valid for tier ≠ free
payments	order_id, method, status, amount	amount = orders.total after discount

Step‑by‑step generation & measurement

Step	Action	Tool / Artefact
1	Export current schema + 5 k masked production rows as reference.	pg_dump --schema-only, pg_dump --data-only --where="ctid IN (SELECT ctid FROM customers TABLESAMPLE SYSTEM (0.5))"
2	Write a prompt for the AI generator: “Create 20 k checkout scenarios covering all tier‑discount‑payment combos, respecting FK and business rules.”	Prompt stored as prompts/checkout_v1.txt
3	Run the free QA3 test data generator at /tools/test-data-generator with the prompt and schema.	qa3 generate --prompt prompts/checkout_v1.txt --schema schema.json --rows 20000 --out data/checkout_gen.parquet
4	Execute the metric harness (see pseudo‑code).	python harness.py data/checkout_gen.parquet data/reference.parquet schema.json
5	Review the Markdown summary.	cat metrics_summary.md

Sample metric output

## Metrics Summary (2024‑03‑15 14:22 UTC)


| Metric      | Value | Threshold | Pass |
|-------------|-------|-----------|------|
| Validity    | 99.73% | ≥ 99.5%   | ✅ |
| Diversity   | 0.86  | ≥ 0.85    | ✅ |
| Pairwise Coverage (t=2) | 91.4% | ≥ 90% | ✅ |
| Rare‑event (crypto+JPY) | 12 rows | ≥ 5 | ✅ |

Interpretation

• Validity is solid – only 54 rows violated a CHECK on credit_limit. Those rows were traced to a prompt ambiguity (“enterprise customers may have high credit”).
• Diversity is just above the line; the tier distribution is 48 % free, 45 % premium, 7 % enterprise – close to production (50/45/5).
• Pairwise coverage hits 91 % because the generator missed the discount_code=SUMMER21 + payment_method=apple_pay tuple. Adding a single line to the prompt (“include SUMMER21 with apple_pay”) lifts coverage to 96 %.

Prompt iteration

- Create 20k checkout scenarios covering all tier‑discount‑payment combos…
+ Create 20k checkout scenarios covering all tier‑discount‑payment combos,
+   especially ensure SUMMER21 appears with apple_pay and crypto appears with JPY.

Re‑run steps 3‑5 → all three metrics improve, and the CI gate stays green.

---

Validation Checklist (run after every generation)

• Schema conformance – All columns present, correct types, NOT‑NULL satisfied.
• Foreign‑key integrity – Every customer_id in orders exists in customers.
• Business‑rule predicates – Custom functions return True for ≥ 99.5 % rows.
• Categorical balance – No enum value < 1 % of rows unless domain‑driven.
• Numerical distribution – KS distance < 0.07 vs. reference for each numeric column.
• Correlation drift – Absolute Pearson diff < 0.1 for known correlated pairs.
• Rare‑event presence – Each flagged rare combo ≥ minimum count (configurable).
• t‑wise coverage – Pairwise (t=2) ≥ 90 %; higher t if budget allows.
• Requirement traceability – Every mapped requirement predicate satisfied at least once.
• Artifact archiving – Prompt, generated data, metric report, and summary stored with build ID.

---

Common Pitfalls & Mitigations

Pitfall	Symptom	Root cause	Mitigation
Prompt drift	Validity drops after a model upgrade.	Implicit assumptions in prompt no longer match model behaviour.	Pin model version; treat prompt as code; run harness on every model change.
Over‑fitting to reference	Diversity ≈ 1.0 but coverage low.	Generator simply mirrors reference distribution, missing unseen combos.	Explicitly request combinatorial coverage in prompt; add synthetic “edge‑case” seeds.
Ignoring soft constraints	Business‑rule violations appear only in integration tests.	Rules expressed only in application code, not in schema.	Encode rules as CHECK constraints or a shared rule library used by both app and generator.
Metric noise	Validity fluctuates 99.3 % → 99.8 % run‑to‑run.	Small sample size for rare categories.	Increase generated row count for low‑frequency classes; use stratified sampling.
Coverage blind spots	Pairwise coverage high, but a critical 3‑wise combo missing.	t‑wise level too low for the risk profile.	Raise t for high‑risk modules; complement with model‑based test generation.
Tool lock‑in	Metrics only runnable inside vendor UI.	Proprietary export format.	Export to open formats (Parquet, CSV, JSON) and run harness offline.

---

Tool Considerations

Category	What to look for	Example options
AI data generators	Prompt‑or‑schema driven, supports FK & CHECK, exports Parquet/CSV, versionable config	QA3 free test data generator (/tools/test-data-generator), Gretel, Mostly AI, Synthea
Metric harness	Pluggable validators, statistical libraries, CI‑friendly CLI	Custom Python harness (shown above), Great Expectations, Soda Core
Coverage engines	t‑wise tuple generation, requirement‑to‑predicate mapping	ACTS, PICT, custom combinatorial script
Data versioning	Immutable snapshots, lineage tracking	DVC, LakeFS, Delta Lake
CI integration	Artifact upload, gating on thresholds, markdown reports	GitHub Actions, GitLab CI, Azure Pipelines

Decision matrix – If you already use Great Expectations for data contracts, extend its expectation suite with the diversity and coverage checks rather than building a second framework.

---

Next Steps for Your Team

1. Pick a pilot – Choose a single micro‑service or bounded context (e.g., the checkout flow above).
2. Create the reference profile – Export a masked production sample + schema + business‑rule catalogue.
3. Write the first prompt – Keep it short, explicit about required combos, and store it in Git.
4. Run the free QA3 test data generator at /tools/test-data-generator with the prompt and schema.
5. Execute the metric harness (copy the pseudo‑code, adapt to your stack).
6. Inspect the Markdown summary – If any metric fails, adjust the prompt or add a seed row, then re‑run.
7. Gate the pipeline – Add a CI step that fails when validity < 99.5 %, diversity < 0.85, or pairwise coverage < 90 %.
8. Document the workflow – Add a README.md in the data‑gen repo describing prompt versioning, metric thresholds, and how to interpret the report.

Actionable first task (today):

Clone the QA3 test data generator, feed it your checkout schema and a one‑sentence prompt (“generate 5 k orders covering all tier‑discount‑payment combos”), run the harness, and capture the three metric numbers. Use those numbers as your baseline for the next sprint.

---

*Metrics turn synthetic data from a “nice‑to‑have” into a first‑class testing artifact. By measuring validity, diversity, and coverage on every generation run you get fast feedback