10 Questions to Ask in a Test Data Generator Demo

When you sit down with a vendor (or an internal team) to evaluate a test‑data generator, the demo is the only moment you can see the tool behave on your data model, your constraints, and your pipelines. A flashy UI and a few happy‑path screenshots are not enough. You need to know whether the tool will survive the messy reality of production‑like schemas, evolving requirements, and CI/CD integration.

Below are ten concrete questions to ask during a demo, each paired with the reasoning behind it, a worked scenario you can run live, and a checklist you can hand to the presenter. Use them to turn a sales pitch into a technical evaluation.

---

1. How Does the Tool Discover and Model My Schema?

Why it matters
A generator that requires you to hand‑write JSON/YAML for every table, column, foreign‑key, and check constraint will become a maintenance burden the moment the schema changes.

What to look for

• Automatic reverse‑engineering from a live database, SQL DDL files, or an ORM metadata file.
• Support for composite primary keys, deferred constraints, and database‑specific types (e.g., JSONB, ARRAY, GEOGRAPHY).
• Ability to annotate discovered metadata (e.g., “this column is a PII field”) without editing source files.

Live scenario

1. Point the tool at a staging copy of your PostgreSQL schema (≈120 tables).
2. Ask the presenter to generate a schema model artifact.
3. Verify that a table with a composite PK (order_id, line_no) and a CHECK (quantity > 0) appears correctly.

Checklist

✅	Capability
☐	Connects to live DB (JDBC/ODBC)
☐	Imports DDL / migration scripts
☐	Reads ORM metadata (Hibernate, Entity Framework, Prisma)
☐	Exposes discovered model as editable JSON/YAML
☐	Handles vendor‑specific types

---

2. What Built‑In Data‑Generation Strategies Exist, and Can I Extend Them?

Why it matters
Realistic data isn’t just “random strings”. You need distributions that mimic production: zip‑code formats, credit‑card Luhn‑valid numbers, correlated columns (e.g., state ↔ zip), and domain‑specific enumerations.

What to look for

• Library of generators: format, range, weighted choice, regex, date‑time series, sequential, foreign‑key aware.
• Plug‑in architecture (Java SPI, .NET MEF, Node modules, Python entry points) so you can ship a custom generator as a JAR/DLL/whl.
• Versioned generator catalog so you can pin a generator version per project.

Live scenario
Ask the demo to generate 10 000 rows for a customers table where:

Column	Requirement
email	RFC‑5322 valid, unique
phone	E.164 format, country‑code weighted by country column
signup_date	Normal distribution centered on 2023‑06‑01, σ = 30 days
tier	Enum (bronze, silver, gold) with 70/20/10 split

Confirm the output matches the spec without post‑processing.

Checklist

✅	Generator Feature
☐	Format / regex generators
☐	Statistical distributions (Gaussian, Pareto, etc.)
☐	Referential integrity aware (FK lookup)
☐	Conditional / correlated generators
☐	Extensible plug‑in model
☐	Generator version pinning

---

3. How Does the Tool Enforce Referential Integrity Across Multiple Tables?

Why it matters
Generating a child row before its parent exists breaks foreign‑key constraints and forces you to run a second “fix‑up” pass.

What to look for

• Topological ordering based on FK graph.
• Option to defer constraints (e.g., generate all tables, then enable FK checks).
• Support for self‑referencing tables (employee → manager) and circular references (order ↔ shipment).

Live scenario
Create a minimal schema:

CREATE TABLE department (id SERIAL PK, name TEXT);
CREATE TABLE employee   (id SERIAL PK, dept_id INT REFERENCES department(id),
                         manager_id INT REFERENCES employee(id));

Ask the tool to generate 5 departments, 30 employees, and a random manager hierarchy. Verify that every dept_id and manager_id points to an existing row.

Checklist

✅	Referential Feature
☐	Automatic FK graph walk
☐	Deferred constraint mode
☐	Self‑reference handling
☐	Circular reference resolution
☐	Batch insert ordering control

---

4. What Data‑Masking / PII‑Protection Capabilities Are Built In?

Why it matters
If you clone production data for testing, you must strip or pseudonymize personally identifiable information before it leaves a secure environment.

What to look for

• Deterministic masking (same input → same output) for repeatable tests.
• Format‑preserving encryption for credit‑card, SSN, IBAN.
• Policy‑driven rules (e.g., “mask all columns tagged pii:true”).
• Ability to keep referential integrity after masking (e.g., masked email still unique).

Live scenario
Tag customers.email and customers.ssn as PII in the model. Run a generation job that produces 5 000 rows and simultaneously writes a masked CSV for a downstream analytics team. Spot‑check that emails remain unique and SSNs pass Luhn‑style checksum if applicable.

Checklist

✅	Masking Feature
☐	Deterministic / seedable masking
☐	Format‑preserving encryption
☐	Policy/tag driven rule engine
☐	Post‑mask uniqueness guarantees
☐	Audit log of masking actions

---

5. How Does the Tool Integrate With CI/CD Pipelines?

Why it matters
Test data must be fresh for every pipeline run (nightly, PR, release). Manual steps defeat the purpose.

What to look for

• CLI or container image with a deterministic exit code.
• Native plugins for GitHub Actions, GitLab CI, Azure Pipelines, Jenkins, CircleCI.
• Ability to output artifacts (SQL scripts, CSV, Parquet) as pipeline artifacts.
• Support for ephemeral databases (Testcontainers, tmpfs‑backed Postgres) so the generator can spin up a clean instance, load data, and tear down.

Live scenario
Add a step to a sample GitHub Actions workflow:

- name: Generate test data
  uses: qa3/test-data-generator@v1
  with:
    schema: ./db/schema.sql
    rows: 2000
    output: ./artifacts/testdata.sql

Run the workflow on a PR and confirm the artifact appears and the subsequent integration tests pass.

Checklist

✅	CI/CD Integration
☐	CLI with non‑zero exit on failure
☐	Official GitHub Action / GitLab CI component
☐	Docker image published to a public registry
☐	Artifact upload (SQL, CSV, Parquet)
☐	Ephemeral DB support (Testcontainers, tmpfs)

---

6. What Performance Characteristics Can I Expect at Scale?

Why it matters
A tool that generates 10 000 rows in 2 seconds may choke at 10 million rows or when generating wide tables (200+ columns).

What to look for

• Parallelism model (thread‑pool, fork‑join, distributed workers).
• Streaming vs. in‑memory generation (important for > GB output).
• Back‑pressure handling when writing to a DB (batch size, transaction boundaries).
• Published benchmarks on comparable hardware (not marketing numbers).

Live scenario
Ask the presenter to run a “stress” generation: 5 M rows for a 150‑column fact table, writing to a local PostgreSQL instance. Capture:

Metric	Target
Wall‑clock time	≤ 5 min
Peak RSS	≤ 2 GB
DB CPU utilization	≤ 70 %
Error rate	0 %

If they cannot run it live, request a recent benchmark report with the same hardware spec.

Checklist

✅	Performance Indicator
☐	Configurable parallelism
☐	Streaming writer (no full‑in‑memory buffer)
☐	Batch‑size / transaction tuning
☐	Published benchmark methodology
☐	Resource‑usage telemetry (CPU, RSS, GC)

---

7. How Does the Tool Handle Schema Evolution?

Why it matters
Columns are added, dropped, or change type every sprint. Regenerating the entire model from scratch each time is wasteful; you want incremental updates.

What to look for

• Diffing between current model and live DB (add/drop column, type change).
• Migration‑aware generators that can patch existing data sets (e.g., backfill a new created_at column with a realistic timestamp).
• Version‑controlled model files (Git‑friendly) with merge‑conflict resolution guidance.

Live scenario

1. Start from the schema used in Question 1.
2. Add a nullable preferred_locale VARCHAR(5) column to customers.
3. Run the generator in incremental mode and verify that existing 10 000 rows get a locale value drawn from a weighted list, while new rows follow the same distribution.

Checklist

✅	Evolution Feature
☐	Model diff detection
☐	Incremental data patching
☐	Git‑compatible model format
☐	Automated backfill strategies
☐	Rollback / snapshot support

---

8. What Output Formats and Target Systems Are Supported?

Why it matters
Your test harness may consume SQL scripts, CSV files, Parquet for Spark, Avro for Kafka, or direct DB inserts. The generator should not force a single format.

What to look for

• Pluggable writers (SQL INSERT, COPY, LOAD DATA, CSV, JSON Lines, Parquet, Avro, Protobuf).
• Ability to write to multiple targets in a single run (e.g., SQL for integration tests + Parquet for analytics).
• Partitioning / sharding options for large outputs (by date, tenant, hash).

Live scenario
Generate a dataset and simultaneously emit:

Target	Format	Partition
PostgreSQL	COPY binary	none
S3 bucket	Parquet (snappy)	dt=YYYY-MM-DD
Kafka topic	Avro (schema registry)	key = customer_id

Confirm each artifact is consumable by the downstream system without transformation.

Checklist

✅	Output Capability
☐	SQL (dialect‑specific)
☐	CSV / TSV / JSONL
☐	Parquet / ORC / Avro
☐	Direct DB bulk load (COPY, BULK INSERT)
☐	Multi‑target single run
☐	Partition / shard configuration

---

9. How Is Determinism and Reproducibility Guarantees Expressed?

Why it matters
Flaky tests often stem from non‑deterministic test data. You need a seed or snapshot that lets you recreate the exact same dataset months later.

What to look for

• Global seed that drives all pseudo‑random generators.
• Per‑table / per‑column seed overrides for localized reproducibility.
• Exportable generation manifest (seed, generator versions, schema hash) that can be stored alongside test artifacts.
• Ability to replay a manifest on a different machine / CI runner.

Live scenario
Run a generation with --seed 0xC0FFEE. Capture the manifest file. Delete the output, change the CI runner OS (Linux → macOS), re‑run with the same manifest, and diff the resulting CSV files (they must be byte‑identical).

Checklist

✅	Determinism Feature
☐	Global seed parameter
☐	Per‑generator seed overrides
☐	Manifest export (JSON/YAML)
☐	Manifest replay CLI flag
☐	Cross‑platform byte‑identical output

---

10. What Is the Licensing, Support, and Extensibility Model?

Why it matters
A tool that looks perfect technically can become a blocker if the license forbids CI usage, the vendor charges per‑core for parallelism, or the extension API is closed.

What to look for

• Clear license text (Apache‑2.0, MIT, BSD, or commercial with explicit CI/CD rights).
• Support tiers: community forum, SLA‑backed enterprise, response‑time guarantees.
• Public roadmap and contribution guidelines (if open‑source).
• Extension SDK documentation, example plugins, and a marketplace or registry.

Live scenario
Ask the presenter to show the license file, the pricing page (if commercial), and a sample custom generator written in the SDK language of your choice. Verify that the custom generator can be packaged and loaded without vendor‑only tooling.

Checklist

✅	Licensing / Support
☐	OSI‑approved license or transparent commercial terms
☐	CI/CD usage explicitly permitted
☐	Defined support SLA (if paid)
☐	Public roadmap / issue tracker
☐	Documented extension SDK with examples
☐	Plugin registry / marketplace

---

Putting It All Together: A Decision Matrix

#	Question	Must‑Have?	Weight (1‑5)	Demo Score (1‑5)	Weighted
1	Schema discovery	✅	5
2	Generator library & extensibility	✅	4
3	Referential integrity	✅	5
4	PII masking	☐ (depends)	3
5	CI/CD integration	✅	5
6	Performance at scale	✅	4
7	Schema evolution	✅	4
8	Output formats	✅	3
9	Determinism / reproducibility	✅	5
10	Licensing & extensibility	✅	4
Total			42		/210

Score each demo run, multiply by weight, and compare totals across vendors. A threshold of ≥ 150 (≈ 70 %) usually indicates a tool that will survive production‑grade usage.

---

Common Pitfalls to Watch During the Demo

Pitfall	Symptom	Mitigation
Hidden manual‑step “just run this script”	Presenter asks you to copy‑paste a shell snippet that isn’t part of the product.	Require the step to be reproducible via the CLI / container.
“We’ll add that in the next release”	Feature you need is on the roadmap but not shipped.