Test Data Tool Proof of Concept: A 14‑Day Evaluation Plan

Choosing a test‑data generation tool is rarely a one‑click decision. The tool must fit the data‑model complexity of your system, the automation stack you already run, the compliance constraints you operate under, and the skill set of the team that will maintain it. A structured proof‑of‑concept (PoC) lets you surface those fit‑gaps before you sign a contract or commit engineering cycles to a custom build.

Below is a ready‑to‑run 14‑day PoC framework. It breaks the evaluation into concrete daily goals, decision checkpoints, and review artefacts so that every stakeholder—QA engineers, test‑automation leads, developers, and engineering managers—can see progress, raise concerns early, and reach a data‑driven go/no‑go decision.

---

1. Set the Evaluation Scope (Day 0)

Before the clock starts, agree on what “good enough” looks like. Capture the following in a shared document (Confluence, Notion, or a simple markdown file in the repo).

Dimension	Questions to Answer	Success Threshold
Data domains	Which bounded contexts (e.g., billing, user‑profile, inventory) need synthetic data?	≥ 80 % of high‑risk test scenarios covered
Volume & variety	Minimum rows per table, required cardinalities, edge‑case distributions (nulls, outliers, locale‑specific formats)	10 k–100 k rows per core table; ≥ 5 distinct distributions
Integration points	CI/CD pipeline, test‑framework (Playwright, Cypress, JUnit, pytest), DB migration tooling	One‑click data load in ≤ 2 min for a full suite
Compliance & masking	PII handling, GDPR/CCPA, data‑retention policies	No real PII leaves the PoC environment
Team ownership	Who writes generators, who maintains schemas, who reviews output	Clear RACI matrix (see §2)
Budget & licensing	Open‑source vs. commercial, per‑seat vs. per‑run pricing	Total cost ≤ $X per month for projected usage

Deliverable: Scope‑Doc (1‑page) signed off by QA lead, dev lead, and engineering manager.

---

2. Define Ownership & RACI (Day 0‑1)

Role	Responsible (R)	Accountable (A)	Consulted (C)	Informed (I)
QA Engineer	Write data‑generation scripts, execute test runs	—	Dev lead (schema changes)	Engineering manager
Test‑Automation Lead	Integrate generator into CI, define pipeline hooks	QA lead	DevOps (artifact storage)	All
Developer / DB Owner	Provide schema snapshots, review generated DDL	—	QA Engineer (data shape)	Engineering manager
Security / Compliance	Approve masking rules, audit data‑access logs	—	QA Engineer	All
Engineering Manager	Approve budget, resolve blockers	—	All	—

Publish the matrix in the Scope‑Doc. It prevents “who owns the generator?” debates later.

---

3. Choose Candidate Tools (Day 1‑2)

Create a short list (3‑5 tools) based on the scope. Typical categories:

Category	Example Tools (no endorsement)	Why It Might Fit
Schema‑driven generators	Tool A, Tool B	Directly read DB migration files (Flyway, Liquibase)
Model‑based / DSL	Tool C, Tool D	Declarative YAML/JSON for complex relationships
AI‑assisted synthetic data	Tool E	Learns distributions from production snapshots (requires data‑access approval)
Lightweight CLI	Tool F (open source)	Zero‑dependency, good for quick spikes
QA3 free test data generator	https://qa3.io/tools/test-data-generator	Browser‑based, no install, useful for rapid prototyping of flat‑file or JSON payloads

Action: Populate a Tool‑Comparison Sheet (Google Sheet or markdown table) with columns for: licensing, language support, CI integration, masking, extensibility, community activity, and a subjective fit score (1‑5) from each evaluator.

---

4. 14‑Day PoC Execution Plan

The plan is split into four sprints of 3‑4 days each, each ending with a gate review.

Sprint 1 – Baseline & Environment Setup (Days 1‑4)

Day	Goal	Tasks	Exit Criteria
1	Provision isolated PoC environment	• Spin up a dedicated DB instance (Docker compose or cloud sandbox) <br>• Clone the latest schema migration scripts <br>• Create a dedicated CI pipeline (e.g., GitHub Actions workflow poc-data-gen.yml)	DB reachable, migrations apply cleanly
2	Load a representative production snapshot (masked)	• Export a masked dump from staging (use pg_dump --column-inserts + custom masking script) <br>• Import into PoC DB <br>• Verify row counts match scope doc	≥ 90 % of core tables populated
3	Run each candidate tool in dry‑run mode	• Execute generator against the PoC schema <br>• Capture logs, generated DML, execution time	All tools produce output without errors
4	Gate 1 Review	• Compare generated row counts vs. scope thresholds <br>• Note any schema‑mismatch errors <br>• Update Tool‑Comparison Sheet with baseline fit scores	Decision: drop any tool that fails > 2 critical mismatches

Artefacts: baseline-report.md, updated comparison sheet.

---

Sprint 2 – Functional Coverage & Data Quality (Days 5‑8)

Day	Goal	Tasks	Exit Criteria
5	Define test‑scenario data contracts	• List 10‑15 high‑value test cases (e.g., “expired subscription”, “multi‑currency order”) <br>• For each, write the exact column/value expectations (including nullability, enum sets, date ranges)	Contracts stored in data-contracts/ as YAML
6	Extend each tool to satisfy contracts	• Add custom generators, foreign‑key wiring, conditional logic <br>• Use tool‑specific extension points (plugins, scripts, DSL)	Each tool can emit data for ≥ 80 % of contracts
7	Measure data realism	• Run statistical checks: distribution histograms, referential integrity, uniqueness constraints <br>• Use a lightweight script (Python pandas + great_expectations)	≥ 95 % of contracts pass Great Expectations suite
8	Gate 2 Review	• Score each tool on coverage, realism, maintenance effort (lines of custom code) <br>• Record any tool‑specific bugs or limitations	Decision: keep top 2‑3 tools for performance sprint

Artefacts: coverage-report.md, Great Expectations HTML report.

---

Sprint 3 – Performance, CI Integration & Security (Days 9‑12)

Day	Goal	Tasks	Exit Criteria
9	Benchmark generation speed	• Run each tool to produce the full data set (target volume from scope) <br>• Capture wall‑clock time, CPU, memory, DB load	Generation ≤ 2 min for full suite (adjustable per project)
10	Embed in CI pipeline	• Add a pipeline step that runs the generator, archives artefacts (SQL, CSV, JSON) <br>• Trigger downstream test job that consumes the artefacts	End‑to‑end pipeline green on a sample PR
11	Validate masking & compliance	• Run a PII scanner (e.g., aws-macie, pii-scanner) on generated output <br>• Verify no real identifiers appear <br>• Confirm audit logs capture generator runs	Zero findings; audit log entries present
12	Gate 3 Review	• Consolidate performance numbers, CI stability, compliance pass/fail <br>• Update comparison sheet with operational scores	Decision: select final candidate (or two for final bake‑off)

Artefacts: perf-report.md, CI pipeline YAML, compliance scan logs.

---

Sprint 4 – Final Bake‑Off & Decision (Days 13‑14)

Day	Goal	Tasks	Exit Criteria
13	Run full regression suite with generated data	• Execute the complete automated test suite (UI, API, DB) using the chosen tool’s output <br>• Capture flaky‑test rate, test‑execution time, defect detection count	Flaky rate ≤ 2 %; no new false‑positives
14	Final Review & Sign‑off	• Present a 15‑minute deck: scope, scores, risks, cost estimate, rollout plan <br>• Collect go/no‑go votes from RACI owners <br>• Document decision in Decision‑Log (ADR style)	Signed ADR; approved budget for license or OSS contribution

Artefacts: final-report.md, ADR 001-test-data-tool-selection.md.

---

5. Worked Example: Generating Order‑Domain Data

Below is a concrete walk‑through using a schema‑driven generator (Tool A) and the QA3 free test data generator for a quick JSON payload prototype.

5.1 Schema Snapshot (PostgreSQL)

CREATE TABLE customers (
    id          UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    email       TEXT NOT NULL UNIQUE,
    full_name   TEXT NOT NULL,
    created_at  TIMESTAMPTZ NOT NULL DEFAULT now()
);


CREATE TABLE orders (
    id          UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    customer_id UUID NOT NULL REFERENCES customers(id),
    status      TEXT NOT NULL CHECK (status IN ('new','paid','shipped','cancelled')),
    total_cents BIGINT NOT NULL CHECK (total_cents > 0),
    placed_at   TIMESTAMPTZ NOT NULL DEFAULT now()
);

5.2 Tool A – Declarative YAML (excerpt)

tables:
  customers:
    count: 5000
    columns:
      email:
        type: email
        unique: true
      full_name:
        type: name
      created_at:
        type: timestamp
        range: "-2y..now"
  orders:
    count: 20000
    columns:
      customer_id:
        type: ref
        table: customers
        column: id
      status:
        type: choice
        weights: {new: 10, paid: 60, shipped: 25, cancelled: 5}
      total_cents:
        type: integer
        distribution: lognormal
        params: {mean: 10.5, sigma: 0.8}
      placed_at:
        type: timestamp
        range: "-1y..now"

Run: tool-a generate --config orders.yaml --db postgres://user:pwd@localhost/poc

Result: 5 k customers, 20 k orders, referential integrity guaranteed, realistic status mix.

5.3 QA3 Free Test Data Generator – Quick JSON Payload

1. Open https://qa3.io/tools/test-data-generator.
2. Choose JSON output, Array of 10 objects.
3. Define fields:
   • orderId → UUID
   • customerEmail → Email
   • status → Custom list ["new","paid","shipped","cancelled"] with weights
   • totalCents → Number, log‑normal (mean = 10.5, sigma = 0.8)
   • placedAt → ISO‑8601 date, last 365 days
4. Click Generate → copy payload into a contract test file.

Why use both? Tool A builds the full relational dataset for integration tests; the QA3 generator gives a lightweight, no‑install way to stub API contract tests or feed a UI component storybook.

---

6. Common Pitfalls & Mitigations

Pitfall	Symptom	Mitigation
Schema drift	Generator fails after a migration adds a NOT NULL column	Hook generator run into the migration pipeline; fail fast on DDL mismatch
Over‑fitting to current data	Generated data looks perfect now but breaks when a new enum value appears	Enforce contract‑first definitions (YAML/JSON) that are version‑controlled alongside code
Hidden performance cost	Generation is fast locally but stalls on CI agents with limited CPU	Benchmark on the exact CI runner type; set a hard timeout in the pipeline step
Masking leakage	PII scanner flags a real email in generated output	Use deterministic masking functions (hash + salt) for any column sourced from production snapshots
Tool lock‑in	Custom plugins become a maintenance burden	Prefer tools with standard extension mechanisms (SQL, Python, JS) and avoid proprietary DSLs unless justified
Insufficient stakeholder buy‑in	Decision stalls after PoC because managers weren’t involved early	Include engineering manager in Gate 1 and Gate 3; capture their concerns in the ADR

---

7. Review Criteria Checklist (Use at Each Gate)

• Scope Alignment – Does the tool meet ≥ 80 % of the data‑domain coverage defined in Day 0?
• Data Quality – Great Expectations suite passes ≥ 95 % of contracts.
• Performance – Full data set generated within the agreed time budget on CI hardware.
• Integration – One‑click pipeline step; artefacts consumable by downstream test jobs without manual copy.
• Compliance – Zero PII findings; audit log entries for every generation run.
• Maintainability – Custom code ≤ 200 LOC per tool; clear upgrade path for schema changes.
• Cost – Licensing + operational cost ≤ budget threshold for projected monthly runs.
• Team Comfort – At least two engineers can author/maintain generators without vendor‑only training.

Score each criterion 1‑5; compute a weighted total (weights agreed in Scope‑Doc). The tool with the highest total that also passes all must‑have gates wins.

---

8. Next Action

1. Create the Scope‑Doc (template linked below) and circulate for sign‑off.
2. Populate the Tool‑Comparison Sheet with the 3‑5 candidates you’ve shortlisted.
3. Spin up the PoC DB (Docker compose file provided in the repo infra/poc-db.yml).
4. Schedule Gate 1 for Day 4 on the team calendar.

Template: docs/poc/scope-doc-template.md – copy, fill, and PR for review.

Completing those four steps today puts you on track to finish the 14‑day PoC with a documented, defensible decision.