How to Generate Positive and Negative Test Data Together

When a test suite only exercises the “happy path,” bugs that surface on malformed input, boundary violations, or missing‑field scenarios stay hidden. Generating both positive (valid) and negative (invalid) data in a single workflow gives you coverage for functional correctness and robustness without maintaining two separate data‑generation pipelines.

Below is a practical, evidence‑driven process you can adopt today. It works whether you script the generation yourself, use a CI‑integrated tool, or rely on a free generator such as the one at /tools/test-data-generator.

---

1. Understand the Problem Space

Aspect	Positive Data	Negative Data
Goal	Verify that the system accepts well‑formed input and produces the expected output.	Verify that the system rejects, sanitizes, or gracefully handles malformed input.
Typical Sources	Production‑like records, schema‑conforming JSON, valid enum values.	Out‑of‑range numbers, missing required fields, wrong data types, SQL‑injection strings, oversized payloads.
Risk if Missing	False confidence – you ship code that works only for “clean” data.	Undetected crashes, security holes, data‑corruption bugs.
Generation Complexity	Straightforward: follow the schema.	Higher: you must deliberately break the schema in many ways.

Key insight: Positive and negative data are two sides of the same contract (the API schema, DB constraints, UI validation rules). If you model the contract once, you can derive both families automatically.

---

2. Prerequisites

Prerequisite	Why It Matters	Quick Check
Explicit contract (OpenAPI, JSON Schema, Protobuf, DB DDL)	Provides a single source of truth for field types, constraints, enums.	✅ Contract file exists and is version‑controlled.
Test‑data policy (what “valid” means for your domain)	Prevents accidental generation of data that looks valid but violates business rules (e.g., future dates for a “birth‑date” field).	✅ Documented in a TEST_DATA_POLICY.md.
Deterministic seed (optional but recommended)	Makes runs reproducible for debugging flaky tests.	✅ Seed stored in CI environment variable.
Isolation strategy (separate DB schema, in‑memory store, mock service)	Guarantees that generated data does not pollute shared environments.	✅ CI spins up a fresh DB per pipeline.
Tooling (scripting language, data‑generation library, or the free generator)	You need a way to turn the contract into concrete rows/objects.	✅ Chosen and installed.

If any of these are missing, address them first. The rest of the workflow assumes they are in place.

---

3. Choose an Implementation Strategy

Strategy	Description	When It Shines	Trade‑offs
Schema‑driven code generation (e.g., openapi-generator + custom templates)	Generates typed builders for each model; you write a thin wrapper that flips validation flags.	Large codebases with stable contracts; teams comfortable with code generation.	Requires maintenance of templates; adds build step.
Declarative data‑spec files (YAML/JSON describing “valid” and “invalid” variants per field)	Human‑readable spec that a generic runner interprets.	Teams that want non‑developers to tweak edge cases.	Spec can drift from actual schema if not validated.
Programmatic mutation engine (take a valid instance, apply mutators)	Start from a known‑good object, then apply a library of mutators (null‑ify, overflow, inject payload).	Quick to prototype; works even when only runtime samples exist.	Harder to guarantee exhaustive coverage; mutators must be curated.
Hybrid (schema for positives, mutation for negatives)	Use the contract to produce a baseline valid payload, then feed it to a mutation engine for negatives.	Most real‑world projects – you get schema fidelity and flexible edge‑case injection.	Slightly more moving parts.

Recommendation: Start with the hybrid approach. It gives you a trustworthy positive baseline (no manual spec drift) and a controllable negative surface (you decide which mutators are relevant).

---

4. Worked Example – REST API for “Create Order”

4.1 Contract (OpenAPI 3.1 snippet)

components:
  schemas:
    CreateOrderRequest:
      type: object
      required: [customerId, items, shippingAddress]
      properties:
        customerId:
          type: string
          format: uuid
        items:
          type: array
          minItems: 1
          maxItems: 50
          items:
            $ref: '#/components/schemas/OrderItem'
        shippingAddress:
          $ref: '#/components/schemas/Address'
        promoCode:
          type: string
          maxLength: 20
          pattern: '^[A-Z0-9]{4,20}$'
    OrderItem:
      type: object
      required: [sku, quantity]
      properties:
        sku:
          type: string
          pattern: '^SKU-[A-Z0-9]{6}$'
        quantity:
          type: integer
          minimum: 1
          maximum: 100
    Address:
      type: object
      required: [line1, city, postalCode, country]
      properties:
        line1:
          type: string
          maxLength: 100
        city:
          type: string
          maxLength: 50
        postalCode:
          type: string
          pattern: '^[0-9]{5}(-[0-9]{4})?$'
        country:
          type: string
          enum: [US, CA, MX]

4.2 Positive‑Data Generation (baseline)

# generate_positive.py


import uuid, random
from faker import Faker
fake = Faker()


def random_sku():
    return f"SKU-{''.join(random.choices('ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789', k=6))}"


def build_valid_order():
    return {
        "customerId": str(uuid.uuid4()),
        "items": [
            {
                "sku": random_sku(),
                "quantity": random.randint(1, 100)
            }
            for _ in range(random.randint(1, 5))
        ],
        "shippingAddress": {
            "line1": fake.street_address()[:100],
            "city": fake.city()[:50],
            "postalCode": fake.zipcode()[:10],   # Faker respects US pattern
            "country": random.choice(["US", "CA", "MX"])
        },
        "promoCode": "".join(random.choices("ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789", k=random.randint(4,20)))
    }

Result: Every call to build_valid_order() yields a payload that passes schema validation and respects business‑rule limits (e.g., maxItems: 50).

4.3 Negative‑Data Generation (mutation layer)

# mutate.py


import copy, random, string


MUTATORS = {
    "nullify": lambda v: None,
    "empty_string": lambda v: "",
    "oversize_string": lambda v: "x" * (len(v) + 100) if isinstance(v, str) else v,
    "out_of_range_int": lambda v: v + 1000 if isinstance(v, int) else v,
    "wrong_type": lambda v: 12345 if isinstance(v, str) else "not-a-" + type(v).__name__,
    "invalid_uuid": lambda v: "not-a-uuid",
    "invalid_enum": lambda v: "ZZ",
    "sql_injection": lambda v: "' OR 1=1 --",
    "xss_payload": lambda v: "<script>alert(1)</script>",
    "missing_required": lambda v: "__REMOVE__",   # sentinel for deletion
}


def apply_mutators(base_obj, mutator_names):
    """Return a list of mutated copies, one per mutator."""
    mutated = []
    for name in mutator_names:
        obj = copy.deepcopy(base_obj)
        _mutate_recursive(obj, name)
        mutated.append(obj)
    return mutated


def _mutate_recursive(node, mutator_name):
    mutator = MUTATORS[mutator_name]
    if isinstance(node, dict):
        keys = list(node.keys())
        for k in keys:
            if node[k] == "__REMOVE__":
                del node[k]
            elif isinstance(node[k], (dict, list)):
                _mutate_recursive(node[k], mutator_name)
            else:
                node[k] = mutator(node[k])
    elif isinstance(node, list):
        for i, item in enumerate(node):
            if isinstance(item, (dict, list)):
                _mutate_recursive(item, mutator_name)
            else:
                node[i] = mutator(item)

Usage

from generate_positive import build_valid_order
from mutate import apply_mutators, MUTATORS


baseline = build_valid_order()
negative_cases = apply_mutators(baseline, list(MUTATORS.keys()))


# negative_cases now holds 10 distinct invalid payloads, each exercising a different failure mode.

4.4 Validation Checklist (run after generation)

✅ Check	How to Verify
Schema conformance (positives)	Run a JSON‑Schema validator (ajv, jsonschema) on every generated positive payload.
Schema violation (negatives)	Ensure each negative payload fails validation at least once (use the same validator, expect errors).
Business‑rule compliance	Execute a lightweight “domain validator” (e.g., promoCode pattern, quantity limits) on positives; confirm negatives trigger the expected rule violation.
Determinism	With a fixed seed, the same script must emit identical payloads across runs.
Coverage matrix	Map each mutator to the field(s) it touches; confirm every required field has at least one nullify and one wrong_type case.
Size limits	Verify no generated payload exceeds transport limits (e.g., 1 MB HTTP body).
Security sanitization	Scan negatives for known injection patterns; they should be present only in negative set.

Automate the checklist in CI:

# .github/workflows/test-data-validation.yml


jobs:
  validate-test-data:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Generate data
        run: |
          python generate_positive.py --out positives.json
          python mutate.py --in positives.json --out negatives.json
      - name: Validate positives
        run: python -m jsonschema -i positives.json schema.json
      - name: Validate negatives (expect failures)
        run: |
          python -c "
import json, jsonschema, sys
schema = json.load(open('schema.json'))
for i, case in enumerate(json.load(open('negatives.json'))):
    try:
        jsonschema.validate(case, schema)
        print(f'Case {i} unexpectedly passed')
        sys.exit(1)
    except jsonschema.ValidationError:
        pass
"

---

5. Common Failure Modes & Mitigations

Failure Mode	Symptom	Root Cause	Mitigation
Schema drift	Positive payloads start failing validation after a contract change.	Contract updated but generation script not regenerated.	Add a CI step that regenerates positive builders from the contract (e.g., openapi-generator run on every PR).
Over‑mutating	Negative payloads become so broken they never reach the code under test (rejected at gateway).	Mutators applied to all fields, including transport‑level envelopes.	Scope mutators to business fields only; keep envelope fields (headers, auth tokens) valid.
Missing business rules	Positive data passes schema but violates a rule like “promoCode must exist in DB”.	Generation only respects structural constraints.	Encode rule‑level validators in a separate “domain policy” file and run them during the checklist.
Flaky tests due to randomness	Same test sometimes passes, sometimes fails.	No fixed seed; Faker or random produce different values each run.	Seed the RNG (random.seed(42); Faker.seed(42)) and store the seed in the CI environment.
Data‑size explosion	CI job OOMs because mutation creates thousands of huge payloads.	Mutators like oversize_string applied to large arrays.	Cap the number of generated negatives per mutator (e.g., 1 per field) and limit array lengths in the baseline.
Security‑scan false positives	Static analysis flags generated negatives as vulnerabilities in the repo.	Injection strings committed to source control.	Keep generated negatives out of version control; write them to a temporary artifact directory that is ignored by .gitignore.
Insufficient negative coverage	Production bug caused by an untested edge case (e.g., Unicode normalization).	Mutator library didn’t include that class.

| Periodically review production error logs, add missing mutators, and treat the mutator list as a living document. |.

---

6. Scaling the Approach

Scale Dimension	Tactics
Number of services	Centralize the contract repository; each service consumes a shared “test‑data library” package that exports generate_positive(service_name) and generate_negative(service_name).
Data volume	Use streaming generators (Python generators, Node streams) instead of materializing full JSON arrays in memory.
Parallel test execution	Partition the seed space: each worker gets a distinct seed range (seed = base_seed + worker_id * 10_000).
Legacy systems without contracts	Record real traffic (with PII scrubbing) → infer a provisional schema (tools like schemalint) → apply the same hybrid pipeline.
Compliance‑heavy domains	Extend the policy file with regulatory constraints (e.g., GDPR‑field masking) and add a “compliance validator” to the checklist.

---

7. Tooling Options (Quick Comparison)

Tool	Language	Schema Input	Positive Gen	Negative Gen	Extensibility	Cost
QA3 Test Data Generator	Web UI / CLI	OpenAPI, JSON Schema	✅ (schema‑driven)	✅ (built‑in mutators)	Plugin API for custom mutators	Free (hosted)
Faker + custom mutators	Python/JS	None (code‑only)	✅ (code)	✅ (code)	Full code control	Free
Hypothesis (property‑based)	Python	None	✅ (strategies)	✅ (strategies)	Powerful shrinking	Free
DataFactory (Java)	Java	JSON Schema	✅	✅ (via InvalidValueProvider)	Annotation‑driven	Free
Mockaroo	Web	CSV/JSON Schema	✅	Limited (manual formulas)	UI‑only	Freemium

Pick the one that matches your stack and governance model. The free generator at /tools/test-data-generator is a low‑friction entry point if you want a UI‑driven, schema‑first workflow without writing code.

---

8. End‑to‑End Checklist for a New Project

[ ] 1. Publish the authoritative contract (OpenAPI/JSON Schema) to a shared repo.
[ ] 2. Write a thin positive‑generator that consumes the contract.
[ ] 3. Define a mutator catalog covering:
      - Null / missing required fields
      - Type mismatches
      - Boundary violations (min/max, length, enum)
      - Injection payloads (SQL, XSS, command)
      - Encoding edge cases (Unicode, overlong UTF‑8)
[ ] 4. Implement the hybrid mutation engine (baseline → mutators).
[ ] 5. Add deterministic seeding and CI validation pipeline.
[ ] 6. Run the validation checklist on every PR.
[ ] 7. Review production error logs quarterly; add missing mutators.
[ ] 8. Document the policy (business rules, compliance) alongside the contract.
[ ] 9. Package the generator as a reusable