How to Generate Test Data from a Schema

Schema-driven test data generation sounds straightforward: point a tool at your OpenAPI spec, Protobuf definition, or database DDL, and get realistic payloads back. In practice, the gap between "valid according to schema" and "useful for testing" is where most teams lose time.

This guide walks through the decisions you'll face, a repeatable workflow, a worked example, and the failure modes that show up in production test suites.

---

The Problem: Valid ≠ Useful

A schema defines structure and constraints. It does not define business validity, state dependencies, or edge-case coverage.

Consider a typical Order schema:

{
  "type": "object",
  "required": ["orderId", "customerId", "items", "total"],
  "properties": {
    "orderId": { "type": "string", "format": "uuid" },
    "customerId": { "type": "string", "format": "uuid" },
    "items": {
      "type": "array",
      "minItems": 1,
      "items": { "$ref": "#/components/schemas/OrderItem" }
    },
    "total": { "type": "number", "minimum": 0 }
  }
}

A naive generator produces:

{
  "orderId": "550e8400-e29b-41d4-a716-446655440000",
  "customerId": "550e8400-e29b-41d4-a716-446655440001",
  "items": [{ "productId": "abc", "quantity": 1, "unitPrice": 10 }],
  "total": 10
}

Problems:

• total doesn't equal sum(items[i].quantity * items[i].unitPrice)
• customerId references a customer that may not exist in the test database
• No expired credit card, no backordered item, no loyalty discount applied
• All strings are generic UUIDs—hard to grep in logs, hard to reason about in assertions

Schema-first generation gets you syntactic validity. The rest is up to you.

---

Prerequisites: What You Need Before Generating

Artifact	Why It Matters	Typical Location
Schema source (OpenAPI, JSON Schema, Protobuf, Avro, SQL DDL)	Single source of truth for structure	api/spec/, proto/, db/migrations/
Enum / reference data catalog	Valid values for status, currency, countryCode	docs/enums.md, config/reference-data/
Business rule documentation	Cross-field constraints, state machines	docs/business-rules/, ADRs
Test environment data contracts	What seed data exists, what IDs are reserved	test/infra/seed/, docker-compose.test.yml
CI/CD pipeline access	Automated regeneration on schema change	.github/workflows/, .gitlab-ci.yml

Checklist before you start:

• Schema files are version-controlled and linted (spectral lint, buf lint)
• Enum/reference data is externalized (not hardcoded in generators)
• At least one consumer (API test, contract test, load test) is identified
• Team agrees on data ownership: who updates generators when schema changes?

---

Decision Criteria: Choose Your Generation Strategy

Three main approaches exist. Most mature teams use a hybrid.

1. Pure Schema-Driven (Zero-Config)

Tools: openapi-generator, json-schema-faker, hypothesis-jsonschema, schemathesis

Best for: Contract testing, fuzzing, initial scaffolding

Limitations:

• No cross-field logic (total === sum(lineItems))
• No referential integrity (foreign keys)
• Enum values often random, not representative
• Hard to inject scenario-specific data (expired card, VIP customer)

2. Template / Factory Layer (Code-First)

Tools: Factory Bot (Ruby), Factory Boy (Python), go-faker + custom builders, TypeScript zod + faker.js

Best for: Unit/integration tests where you need named scenarios (build(:order, :with_expired_card))

Limitations:

• Drift risk: schema changes, factories don't
• Maintenance burden grows with schema surface area
• Hard to share across languages (frontend vs backend tests)

3. Hybrid: Schema + Overlay Rules

Tools: Custom generator wrapping a schema validator + rule engine (CEL, Rego, JSONLogic), or QA3's free test data generator which accepts schema + constraint overlays

Best for: End-to-end suites, contract + scenario coverage, multi-team alignment

How it works:

1. Generator reads schema → produces structurally valid object
2. Overlay rules apply semantic constraints:
   • total = sum(items[].quantity * items[].unitPrice)
   • customerId ∈ seededCustomers
   • status ∈ ['PENDING', 'CONFIRMED', 'SHIPPED'] (not random enum)
3. Scenario presets compose overlays: abandoned_cart, high_value_vip, cross_border_tax

Trade-off table:

Criterion	Pure Schema	Factory Layer	Hybrid (Schema + Overlay)
Setup time	Minutes	Hours–Days	Hours
Schema drift resilience	High	Low	Medium–High
Business logic coverage	None	High	High (via overlays)
Cross-language sharing	Easy	Hard	Medium (overlay DSL)
Scenario composition	Manual	Native	Native
Debuggability	Low (opaque)	High (code)	Medium (declarative rules)

Recommendation: Start with pure schema for contract/fuzz tests. Add overlay rules when you hit the first cross-field bug that schema validation missed. Migrate factories to overlays when maintenance cost exceeds generator extension cost.

---

Workflow: From Schema to Test-Ready Data

┌─────────────┐     ┌──────────────┐     ┌─────────────────┐     ┌──────────────────┐
│ 1. Ingest   │────▶│ 2. Validate  │────▶│ 3. Apply Rules  │────▶│ 4. Materialize   │
│  Schema     │     │  Structure   │     │  (Overlays)     │     │  & Persist       │
└─────────────┘     └──────────────┘     └─────────────────┘     └──────────────────┘
       │                   │                      │                        │
       ▼                   ▼                      ▼                        ▼
  - OpenAPI           - Required fields      - Cross-field           - Write to DB
  - Protobuf          - Type/format          - Referential           - Emit JSON/XML
  - JSON Schema       - Enum membership      - State machine         - Publish to Kafka
  - SQL DDL           - min/max/pattern      - Scenario presets      - Feed API client

Step 1: Ingest Schema

Normalize to a common intermediate representation (IR). Most tools target JSON Schema Draft 2020-12.

# OpenAPI → JSON Schema


npx @apidevtools/swagger-parser dereference api/openapi.yaml > api/schema.json


# Protobuf → JSON Schema (via protoschema)


buf build --output - | protoschema > api/schema.json


# SQL DDL → JSON Schema (via ddl2jsonschema)


ddl2jsonschema --input db/schema.sql --output api/schema.json

Tip: Commit the generated schema.json to version control. It becomes a diffable artifact for schema change reviews.

Step 2: Validate Structure

Run a structural validator against generated samples before applying rules. Catch format mismatches early.

# validate_structure.py


import json, jsonschema
from jsonschema import Draft202012Validator


with open("api/schema.json") as f:
    schema = json.load(f)


validator = Draft202012Validator(schema)


def validate_sample(sample_path):
    with open(sample_path) as f:
        sample = json.load(f)
    errors = list(validator.iter_errors(sample))
    if errors:
        for e in errors:
            print(f"❌ {e.json_path}: {e.message}")
        return False
    print("✅ Structurally valid")
    return True

Run this in CI on every generated artifact.

Step 3: Apply Overlay Rules

Overlay rules live in a separate file (YAML/JSON/CEL). Example using CEL (Common Expression Language):

# overlays/order.yaml


rules:
  - name: total_matches_line_items
    expression: |
      total == items.map(i, i.quantity * i.unitPrice).sum()
    message: "total must equal sum of line items"


- name: customer_exists
    expression: |
      customerId in seed.customers.map(c, c.id)
    message: "customerId must reference seeded customer"


- name: status_valid_transition
    expression: |
      (status == 'PENDING' && !has(previousStatus)) ||
      (status == 'CONFIRMED' && previousStatus == 'PENDING') ||
      (status == 'SHIPPED' && previousStatus == 'CONFIRMED')
    message: "invalid status transition"


presets:
  abandoned_cart:
    - status: PENDING
      items: [{ quantity: 1, unitPrice: 99.99 }]
      total: 99.99
      createdAt: "{{ now - 24h }}"


high_value_vip:
    - customerId: "{{ seed.customers.vip.id }}"
      items:
        - productId: "PREMIUM-001"
          quantity: 5
          unitPrice: 499
      total: 2495
      status: CONFIRMED

Rule categories to standardize:

Category	Examples	Implementation Hint
Cross-field arithmetic	total = sum(lineItems), tax = subtotal * rate	CEL expression, computed field
Referential integrity	customerId ∈ seededCustomers, productId ∈ catalog	Lookup against seed data manifest
State machine	status transitions, shipmentDate > orderDate	Transition table + temporal check
Business invariants	discount ≤ subtotal, quantity > 0 if status != CANCELLED	CEL / JSONLogic / custom evaluator
Data quality	email matches corp domain, phone E.164 format	Regex + format validators
Scenario composition	preset: abandoned_cart + expired_payment_method	Layer presets, last-write-wins

Step 4: Materialize & Persist

Output formats depend on consumer:

Consumer	Output	Delivery Mechanism
API contract test	JSON	File fixture / HTTP mock
Integration test (DB)	SQL INSERT / ORM objects	Testcontainers / migration script
Load test (k6, Locust)	JSON / CSV	stdin / file feed
Frontend Storybook	JSON	.stories.ts import
Chaos/fault injection	Malformed but schema-valid	Mutation operator on valid base

Persistence checklist:

• Deterministic IDs for debuggability (seeded UUIDs, not random)
• Timestamp strategy: fixed clock (2024-01-15T10:00:00Z) or relative (now - 2h)
• Seed data version stamped in output (_meta: { seedVersion: "v2024.03.1", generatorVersion: "1.4.0" })
• Cleanup hooks for DB-persisted data (TRUNCATE CASCADE in afterAll)

---

Worked Example: E-Commerce Order API

1. Schema Source (OpenAPI 3.1 fragment)

# api/openapi.yaml


components:
  schemas:
    Order:
      type: object
      required: [orderId, customerId, items, total, status, createdAt]
      properties:
        orderId:
          type: string
          format: uuid
        customerId:
          type: string
          format: uuid
        items:
          type: array
          minItems: 1
          maxItems: 50
          items:
            $ref: '#/components/schemas/OrderItem'
        total:
          type: number
          minimum: 0
          multipleOf: 0.01
        status:
          type: string
          enum: [PENDING, CONFIRMED, SHIPPED, DELIVERED, CANCELLED, RETURNED]
        createdAt:
          type: string
          format: date-time
        updatedAt:
          type: string
          format: date-time


OrderItem:
      type: object
      required: [productId, quantity, unitPrice]
      properties:
        productId:
          type: string
          pattern: '^SKU-[A-Z0-9]{6}$'
        quantity:
          type: integer
          minimum: 1
          maximum: 999
        unitPrice:
          type: number
          minimum: 0.01
          multipleOf: 0.01
        discount:
          type: number
          minimum: 0
          maximum: 1
          default: 0

2. Reference Data (committed alongside schema)

# test/data/reference.yaml


customers:
  - id: "cust-00000000-0000-4000-8000-000000000001"
    email: "alice@example.com"
    tier: "STANDARD"
    creditCard:
      token: "tok_visa_valid"
      expiry: "2026-12"
  - id: "cust-00000000-0000-4000-8000-000000000002"
    email: "bob.vip@example.com"
    tier: "VIP"
    creditCard:
      token: "tok_amex_valid"
      expiry: "2027-06"
  - id: "cust-00000000-0000-4000-8000-000000000003"
    email: "charlie.expired@example.com"
    tier: "STANDARD"
    creditCard:
      token: "tok_visa_expired"
      expiry: "2020-01"


products:
  - id: "SKU-ABC123"
    name: "Widget A"
    price: 29.99
    category: "WIDGETS"
    inventory: 100
  - id: "SKU-PREMIUM99"
    name: "Premium Gadget"
    price: 499.00
    category: "GADGETS"
    inventory: 5
  - id: "SKU-BACKORDER"
    name: "Backordered Item"
    price: 19.99
    category: "WIDGETS"
    inventory: 0
    backorderable: true

3. Overlay Rules (CEL)

# overlays/order.yaml


version: "1.0"
schemaRef: "api/schema.json#/components/schemas/Order"


rules:
  - id: total_calculation
    severity: error
    expression: |
      total == items.map(i, i.quantity * i.unitPrice * (1 - i.discount)).sum()
    message: "total must equal sum of discounted line items"


- id: customer_tier_consistency
    severity: warn
    expression: |
      customerId in seed.customers.map(c, c.id) &&
      (customerId == seed.customers.vip.id ? tier == 'VIP' : tier == 'STANDARD')
    message: "customer tier should match reference data"


- id: inventory_respected
    severity: error
    expression: |
      items.all(i, i.productId in seed.products.map(p, p.id) &&
        (seed.products.filter(p, p.id == i.productId)[0].inventory >= i.quantity ||
         seed.products.filter(p, p.id == i.productId)[0].backorderable == true))
    message: "quantity exceeds inventory for non-backorderable product"


- id: status_timestamps
    severity: error
    expression: |
      (status == 'PENDING' && createdAt == updatedAt) ||
      (status in ['CONFIRMED','SHIPPED','DELIVERED'] && timestamp(updatedAt) > timestamp