GraphQL Test Data Generation for Queries and Mutations
GraphQL Test Data Generation for Queries and Mutations
Generating realistic test data for GraphQL APIs is a recurring pain point for QA teams. Unlike REST, where each endpoint returns a fixed shape, a GraphQL schema can be queried in countless ways. A single query may request nested objects, fragments, variables, and directives, while a mutation can accept complex input types, enums, and custom scalars. If the test data does not respect the schema’s constraints—nullability, list sizes, enum values, custom validation rules—the test suite will flake or, worse, mask real defects.
This guide walks through a practical, evidence‑based approach to generating test data for both queries and mutations. It covers decision criteria, a repeatable workflow, a worked example, validation checks, common pitfalls, and tool considerations. The goal is to give you a process you can adopt today, not a sales pitch.
1. Why GraphQL Test Data Is Different
| Aspect | REST | GraphQL |
|---|---|---|
| Response shape | Fixed per endpoint | Determined by the client query |
| Input validation | Usually per‑endpoint | Centralised in the schema (type system, directives) |
| Nested data | Requires multiple calls | Single request can fetch deep graphs |
| Variable handling | Query strings / body | First‑class variables with type checking |
| Schema evolution | Versioned endpoints | Backward‑compatible type additions/deprecations |
Because the client decides the shape, a test data generator must understand the schema and honour the specific query or mutation being exercised. A generic JSON generator that only knows “string, number, boolean” will produce payloads that the server rejects or that exercise only a tiny slice of the schema.
2. Decision Criteria for a Test Data Strategy
Before picking a tool or writing scripts, answer the following questions. Your answers shape the workflow and the level of automation you need.
| Question | Why It Matters | Typical Answers |
|---|---|---|
| How many distinct queries/mutations are covered? | Determines whether you need a one‑off script or a reusable pipeline. | < 20 → ad‑hoc scripts; > 20 → schema‑driven generator |
| Do you need deterministic data for regression? | Flaky tests often stem from random values that trigger different code paths. | Yes → seedable generators; No → pure randomness acceptable |
Are there custom scalars / directives (e.g., DateTime, @constraint)? | Generators must know how to produce valid values. | Yes → need extensibility points |
| Is the schema stable or evolving rapidly? | Frequent schema changes demand a generator that can re‑sync automatically. | Stable → static codegen; Evolving → CI‑integrated introspection |
| What is the test execution environment? | Some generators run only in Node, others in JVM, others as CLI. | Node/Jest, Java/TestNG, Python/pytest, CI containers |
| Do you need to simulate error responses (validation errors, permission denials)? | Negative testing requires deliberately invalid data. | Yes → generator must support “invalid‑by‑design” modes |
If you answer “yes” to three or more of the “schema‑driven” rows, invest in a schema‑aware generator rather than hand‑crafted fixtures.
3. End‑to‑End Workflow
Below is a repeatable workflow that works for both queries and mutations. It can be implemented with a combination of open‑source libraries, a small amount of glue code, and (optionally) the free QA3 test data generator at /tools/test-data-generator for quick prototyping.
1. Introspect the GraphQL schema
2. Define the target operation (query/mutation) + variable shape
3. Generate a *valid* data set that satisfies the operation
4. Optionally generate *invalid* variants for negative testing
5. Persist the data (files, DB, in‑memory) for test consumption
6. Validate generated payloads against the schema (static + runtime)
7. Hook the data into your test runner (Jest, pytest, TestNG, …)
8. Automate regeneration on schema change (CI step)
3.1 Introspection
# Example using Apollo CLI (Node)
npx apollo service:download --endpoint=https://api.example.com/graphql schema.graphql
The resulting schema.graphql (or JSON introspection) is the single source of truth for all downstream steps.
3.2 Operation Selection
Create a manifest (YAML/JSON) that lists each operation you want to test, together with the variable types you intend to exercise.
# operations.yml
operations:
- name: GetUserProfile
type: query
document: |
query GetUserProfile($id: ID!) {
user(id: $id) {
id
email
profile { firstName lastName avatarUrl }
posts(limit: 5) { id title publishedAt }
}
}
variables:
id: "User:123"
- name: CreatePost
type: mutation
document: |
mutation CreatePost($input: CreatePostInput!) {
createPost(input: $input) {
id
title
content
author { id email }
}
}
variables:
input:
title: "Sample Title"
content: "Markdown body"
tags: ["graphql", "testing"]
3.3 Generation
A schema‑aware generator walks the selection set of the operation and produces values for every leaf field, respecting:
- Nullability – non‑null fields get a value; nullable fields may be omitted or set to
nullbased on a configurable “density” factor. - List sizes – configurable min/max (default 1‑3).
- Enums – one of the defined values, optionally weighted.
- Custom scalars – plug‑in functions (e.g., ISO‑8601 for
DateTime, UUID forID). - Directives –
@skip,@includeevaluated against variable values.
Example Generator Snippet (Node + graphql-code-generator)
// generate.ts
import { buildSchema, parse, validate } from 'graphql';
import { generateTestData } from 'graphql-test-data-generator'; // hypothetical lib
const schema = buildSchema(fs.readFileSync('schema.graphql', 'utf8'));
const ops = yaml.load(fs.readFileSync('operations.yml', 'utf8'));
ops.operations.forEach(op => {
const ast = parse(op.document);
const data = generateTestData({
schema,
operationAST: ast,
variables: op.variables,
options: {
seed: 42, // deterministic
listSize: { min: 1, max: 3 },
nullableDensity: 0.2, // 20% chance to include nullable fields
customScalars: {
DateTime: () => new Date().toISOString(),
Email: () => `user${Math.random()}@example.com`,
},
},
});
fs.writeFileSync(`test-data/${op.name}.json`, JSON.stringify(data, null, 2));
});
Running node generate.ts produces a JSON file per operation that can be fed directly to your test runner.
3.4 Negative Variants
For each operation, create a variant spec that flips a single rule:
| Variant | Rule Change |
|---|---|
missingRequiredField | Omit a non‑null field |
invalidEnum | Use a value not in the enum list |
oversizedList | Exceed max list size |
malformedScalar | Pass "not-a-date" to DateTime |
A small script can clone the valid payload and apply the mutation, giving you a ready‑made negative‑test corpus.
3.5 Persistence & Consumption
- File‑based – simplest; commit generated JSON to the repo (or generate in CI and store as artifacts).
- Database seed – if your tests run against a real DB, write a seed script that inserts the generated entities.
- In‑memory mock server – tools like MSW or Apollo MockedProvider can ingest the JSON and serve it without a backend.
3.6 Validation
Two layers of validation catch regressions early:
- Static validation – run
graphql --validateagainst the schema for every generated payload. - Runtime validation – execute the operation against a test instance of the API (or a contract‑test mock) and assert that the response shape matches the selection set.
# Static validation example
npx graphql-cli validate --schema schema.graphql --documents test-data/*.json
3.7 CI Integration
Add a pipeline step that:
- Pulls the latest schema (or uses a committed snapshot).
- Runs the generator.
- Executes static validation.
- Fails the build if any payload is invalid.
# .github/workflows/graphql-test-data.yml
jobs:
generate-test-data:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Download schema
run: npx apollo service:download --endpoint=${{ secrets.GRAPHQL_ENDPOINT }} schema.graphql
- name: Generate data
run: node generate.ts
- name: Validate payloads
run: npx graphql-cli validate --schema schema.graphql --documents test-data/*.json
4. Worked Example: GetUserProfile Query & CreatePost Mutation
Below is a concrete walk‑through using the manifest from §3.2. The schema excerpt (simplified) is:
type User {
id: ID!
email: String!
profile: Profile
posts(limit: Int = 10): [Post!]!
}
type Profile {
firstName: String!
lastName: String!
avatarUrl: String @deprecated(reason: "Use avatar")
avatar: Media
}
type Media {
url: String!
width: Int
height: Int
}
type Post {
id: ID!
title: String!
content: String!
publishedAt: DateTime
author: User!
tags: [String!]!
}
input CreatePostInput {
title: String!
content: String!
tags: [String!]!
}
scalar DateTime
4.1 Generated Valid Payload for GetUserProfile
{
"query": "query GetUserProfile($id: ID!) { user(id: $id) { id email profile { firstName lastName avatarUrl avatar { url width height } } posts(limit: 5) { id title content publishedAt author { id email } tags } } }",
"variables": { "id": "User:123" }
}
Generated response stub (what the mock server would return):
{
"data": {
"user": {
"id": "User:123",
"email": "user0.7341@example.com",
"profile": {
"firstName": "Avery",
"lastName": "Chen",
"avatarUrl": "https://cdn.example.com/avatars/0.png",
"avatar": {
"url": "https://cdn.example.com/avatars/0.png",
"width": 200,
"height": 200
}
},
"posts": [
{
"id": "Post:987",
"title": "First post",
"content": "Hello world",
"publishedAt": "2024-03-12T14:22:00Z",
"author": { "id": "User:123", "email": "user0.7341@example.com" },
"tags": ["graphql", "testing"]
},
{
"id": "Post:988",
"title": "Second post",
"content": "Another day",
"publishedAt": "2024-03-13T09:10:00Z",
"author": { "id": "User:123", "email": "user0.7341@example.com" },
"tags": ["automation"]
}
]
}
}
}
Notice:
avatarUrlis present even though it’s deprecated (generator includes deprecated fields by default; you can toggle this).publishedAtuses the customDateTimescalar formatter.- List
postscontains two items (within the 1‑3 default range).
4.2 Generated Valid Payload for CreatePost Mutation
{
"query": "mutation CreatePost($input: CreatePostInput!) { createPost(input: $input) { id title content author { id email } } }",
"variables": {
"input": {
"title": "Generated Title 42",
"content": "Generated markdown body",
"tags": ["graphql", "testing", "ci"]
}
}
}
Expected response stub:
{
"data": {
"createPost": {
"id": "Post:1001",
"title": "Generated Title 42",
"content": "Generated markdown body",
"author": { "id": "User:123", "email": "user0.7341@example.com" }
}
}
}
4.3 Negative Variant: Missing Required title
{
"query": "...",
"variables": {
"input": {
"content": "Body without title",
"tags": ["oops"]
}
}
}
Running this against the real API should return a validation error with a message like "Field 'title' is required". Your test asserts the presence of that error and the HTTP status 200 (GraphQL errors are delivered in the response body).
5. Validation Checklist
Use this checklist each time you add or modify an operation in the manifest.
- Schema sync – Introspection file is up‑to‑date (CI step passes).
- Operation coverage – Every field in the selection set appears in the generated payload (including nested fragments).
- Nullability respected – No
nullon non‑null fields; nullable fields appear with configured density. - Enum values – All enum fields use values from the schema definition.
- Custom scalars – Formatter functions exist for each custom scalar (
DateTime,Email,JSON, etc.). - List bounds – Generated list lengths fall within
min/maxconfig. - Directives evaluated –
@skip/@includelogic matches variable values. - Deprecation handling – Decision documented (include / exclude deprecated fields).
- Static validation –
graphql-cli validatepasses for every payload. - Runtime smoke test – At least one successful execution against a test environment.
- Negative variants – At least one invalid payload per required field / enum / scalar.
- Determinism – Same seed produces identical payloads (important for flakiness).
- Version control – Generated payloads (or the generator script) are committed / archived as CI artifacts.
6. Common Pitfalls & Mitigations
| Pitfall | Symptom | Mitigation |
|---|---|---|
| Generator ignores fragments | Missing fields in payload, tests pass but production fails. | Ensure the generator resolves fragments recursively (most libraries do this if you pass the full AST). |
| Random data breaks deterministic assertions | Tests flake on CI, hard to reproduce locally. | Use a fixed seed; store the seed in the manifest. |
| Custom scalar not implemented | TypeError: Cannot read property 'toISOString' of undefined or server returns SCALAR_PARSE_ERROR. | Register a formatter for every custom scalar; fallback to a safe default (e.g., ISO‑8601 for any DateTime‑like scalar). |
| Enum values change in schema | Generated payload uses stale enum, server rejects. | Regenerate on every schema change (CI step) and run static validation. |
| List size defaults too large | Payload exceeds server‑side pagination limits, causing 500. | Configure listSize.max to match known server limits (often documented in the schema via @maxLength directives). |
| Deprecated fields still generated | Tests rely on deprecated fields that will be removed. | Add a config flag includeDeprecated: false and audit tests for deprecated usage. |
| Mutation side‑effects not reset | Subsequent tests |
Read more
Postman Test Data Generation with CSV and JSON Files
A practical guide to “Postman Test Data Generation with CSV and JSON Files,” with worked scenarios, tool considerations, validation checks, and actionable advice for QA teams.
Generate API Test Data Directly from an OpenAPI Specification
A practical guide to “Generate API Test Data Directly from an OpenAPI Specification,” with worked scenarios, tool considerations, validation checks, and actionable advice for QA teams.
JSON Test Data Generator Guide for QA Engineers
A practical guide to “JSON Test Data Generator Guide for QA Engineers,” with worked scenarios, tool considerations, validation checks, and actionable advice for QA teams.