What it is: @seontechnologies/playwright-utils is the shared toolbox used across all TEA-managed test suites. It wraps common testing operations — HTTP calls, authentication, polling, file handling, network monitoring — into single-import utilities so teams stop re-solving the same problems project to project.

Design philosophy: Functional Core, Fixture Shell. Each utility is a pure TypeScript function at its core, then wrapped in a Playwright fixture for clean dependency injection. Full TypeScript support with typed return values throughout.

The 9 Core Utilities

Importing utilities — pattern

// Each utility has its own fixture subpath export
import { test as apiTest }  from '@seontechnologies/playwright-utils/api-request/fixtures';
import { test as authTest }  from '@seontechnologies/playwright-utils/auth-session/fixtures';
import { mergeTests }        from '@playwright/test';

// Merge all into one test object (see fixture-architecture.md)
export const test = mergeTests(apiTest, authTest);

Quick-reference: before vs after

What it is: A pattern for organizing shared test setup code (fixtures) so that common needs — login state, clean databases, seed data, test users — are written once and reused everywhere without duplication.

Why it matters: Without this pattern, every test file duplicates the same login, setup, and teardown code. When the API changes, you update 50 files instead of 1. Fixtures are the solution — but only if structured correctly.

The Three-Layer Pattern

Full code example

// Layer 1: pure function (tests/support/factories/user.factory.ts)
export async function createUserViaApi(data: Partial<User>) {
  const res = await fetch('/api/users', {
    method: 'POST', body: JSON.stringify({ ...defaults, ...data })
  });
  return res.json() as User;
}

// Layer 2: fixture wrapper (tests/support/fixtures/user.fixture.ts)
import { test as base } from '@playwright/test';
export const test = base.extend<{ createUser: (data?: Partial<User>) => Promise<User> }>({
  createUser: async ({}, use) => {
    const created: User[] = [];
    await use(async (data) => {
      const user = await createUserViaApi(data);
      created.push(user);     // track for cleanup
      return user;
    });
    // afterEach cleanup: delete all users created during the test
    for (const u of created) await deleteUser(u.id);
  }
});

// Layer 3: merged export (tests/support/merged-fixtures.ts)
import { mergeTests } from '@playwright/test';
import { test as userFixture }    from './fixtures/user.fixture';
import { test as apiFixture }     from '@seontechnologies/playwright-utils/api-request/fixtures';
export const test = mergeTests(userFixture, apiFixture);  // one import for all tests
export { expect } from '@playwright/test';

// In your test file (tests/users/create.spec.ts)
import { test, expect } from '../support/merged-fixtures';
test('create user', async ({ createUser, apiRequest }) => {
  const user = await createUser({ role: 'admin' });
  // user is automatically deleted after this test
});

When to create a fixture vs inline code

Recommended folder structure

tests/
├── support/
│   ├── merged-fixtures.ts     ← Single import for all test files
│   ├── fixtures/
│   │   ├── user.fixture.ts
│   │   └── auth.fixture.ts
│   └── factories/
│       ├── user.factory.ts    ← Pure functions
│       └── order.factory.ts
└── features/
    └── users/
        └── create.spec.ts     ← import { test } from '../../support/merged-fixtures'

What it is: A mandatory rule for UI test reliability — always register network interceptors before navigating to a page or clicking a trigger. This is the single most common cause of intermittent test failures in Playwright.

Why it matters: When a page loads, it immediately fires API requests. If your test navigates first and then registers a listener, those early requests are already gone — your listener will never fire, the Promise will never resolve, and the test will hang until timeout.

The correct sequence (always)

// ✅ CORRECT: register listener FIRST, then trigger the navigation
const usersCall = interceptNetworkCall({ url: '**/api/users' });  // step 1
await page.goto('/dashboard');                                   // step 2
const { responseJson, status } = await usersCall;               // step 3: resolve
expect(status).toBe(200);

// ❌ WRONG: navigating first means the request fires before the listener exists
await page.goto('/dashboard');                // request already fired!
const usersCall = interceptNetworkCall({ url: '**/api/users' });  // too late

Two modes: Spy vs Stub

Stub example — testing error handling

// Force a 500 error and verify the UI shows an error message
const failedCall = interceptNetworkCall({
  url: '**/api/orders',
  fulfillResponse: { status: 500, body: { error: 'Internal Server Error' } }
});
await page.goto('/orders');
await failedCall;
await expect(page.getByRole('alert')).toContainText('Something went wrong');

Waiting rules — mandatory

• NeverHard-coded timeouts: waitForTimeout(5000), sleep(3000), cy.wait(2000) — these make tests slow and still flaky on overloaded CI machines.
• Nevernetworkidle in SPAs: Single-Page Apps with background polling never reach "idle" — this wait will always time out.
• AlwaysWait for a specific API response: page.waitForResponse('**/api/resource') — resolves the moment the named endpoint responds.
• AlwaysWait for an element state: locator.waitFor({ state: 'visible' }) — resolves when the element reaches the expected state.
• AlwaysWait for navigation: page.waitForURL('/expected-path') — resolves when the URL matches after a navigation action.

HAR Recording — offline testing

HAR (HTTP Archive) recording captures every network request/response to a file. This enables two valuable workflows:

What it is: A pattern for generating test data automatically — unique, random values per test run — so tests can run in parallel without stepping on each other's data.

The problem it solves: If all tests use the static email test@example.com, two parallel tests will conflict: test A creates the user, test B fails because the user already exists. Factories produce unique data every run.

Basic factory pattern

import { faker } from '@faker-js/faker';

interface User { name: string; email: string; role: string; }

// Factory function — generates unique data, allows specific overrides
export function createUserData(overrides: Partial<User> = {}): User {
  return {
    name:  faker.person.fullName(),    // "Jane Smith" — unique every run
    email: faker.internet.email(),    // "abc123@example.net" — unique every run
    role:  'viewer',
    ...overrides                       // caller can pin specific values
  };
}

// Usage: all unique, no conflicts in parallel
const admin    = createUserData({ role: 'admin' });  // random name+email, role=admin
const viewer   = createUserData();                   // fully random
const specific = createUserData({ email: 'fixed@test.com' }); // pin email when needed

Cleanup pattern — always clean up after each test

// In your fixture (inside the fixture wrapper from file-02)
const createdUserIds: string[] = [];

await use(async (overrides) => {
  const data = createUserData(overrides);
  const user = await apiRequest.post('/api/users', { body: data });
  createdUserIds.push(user.id);  // track every created entity
  return user;
});

// afterEach: runs automatically, even if the test fails
for (const id of createdUserIds) {
  await apiRequest.delete(`/api/users/${id}`);
}

Performance: API setup vs UI setup

A test suite with 100 tests, each requiring a pre-existing user: API setup = ~10 s total. UI setup = ~500 s total. Always use API factories for test data setup.

Factory rules

• RuleEvery factory function must use faker for all string/email/name fields — never hardcode static values
• RuleEvery created entity ID must be tracked and deleted in afterEach — even if the test fails midway
• RuleUse API calls (not UI) to create test data — 10–50× faster and does not depend on UI being functional
• NeverShare a single user/entity between multiple parallel tests — parallel tests read and modify the same data, causing race conditions
• NeverRely on the order of tests to build state — each test must set up its own data independently

What it is: Defines three test levels (Unit, Integration, E2E), when each is appropriate, the expected speed and scope of each, and the anti-patterns that arise when levels are misapplied. This is the foundational mental model for all test planning in TEA.

The Test Pyramid

Decision guide — which level to write

Anti-patterns

• AvoidE2E tests for business logic: Using Playwright to test a discount calculation costs 100× more time than a unit test and fails for UI reasons unrelated to the logic.
• AvoidUnit tests for framework behavior: Testing "does Express route a POST request?" is testing the framework, not your code. No value.
• AvoidInverted pyramid: More E2E than unit tests means very slow CI, very high maintenance, and very poor failure diagnosis. A failing E2E test tells you "something broke" — a failing unit test tells you exactly what.
• NoteAPI tests are a subset of Integration: Tests using apiRequest without a browser sit at the integration level and are 10–100× faster than browser-based E2E tests for the same coverage.

What it is: Four priority levels (P0–P3) that map to minimum coverage requirements. Every feature/requirement in a project must be classified at one of these levels before the test plan is written. Priority is determined by business impact of failure — not technical complexity.

Priority Levels

Classification examples

How to apply priorities in a test plan

What it is: A structured method for scoring, categorizing, and tracking risks in a software system so that testing effort is concentrated where failure would hurt the most. Each risk item receives a numeric score that maps to a mandatory action tier.

Risk Score Formula

Risk Score = Probability (1–3)  ×  Impact (1–3)  =  range: 1–9

Action tiers

Risk categories and examples

How to run a risk assessment

Risk register format

# Risk Register — Sprint 12
| ID    | Category | Description                    |  P  |  I  | Score | Action       | Mitigation                        |
|-------|----------|-------------------------------|-----|-----|-------|--------------|-----------------------------------|
| R-001 | SEC      | JWT stored in localStorage     |  3  |  3  |   9   | BLOCK        | Move to httpOnly cookie            |
| R-002 | PERF     | N+1 on /api/orders list        |  2  |  2  |   4   | Monitor      | Add eager loading, add perf test   |
| R-003 | DATA     | No migration rollback script   |  2  |  3  |   6   | Mitigate     | Write down migration before ship   |

What it is: A 29-criterion checklist across 8 categories used to assess whether a system is ready to be approved for production release from a quality and testability perspective. This is the gate evaluation model used by the bmad-testarch-nfr skill.

How it works: TEA reads the architecture documents, ADRs, and available evidence (test results, monitoring dashboards, deployment runbooks), then scores each criterion as met / partial / missing. The final score determines PASS / CONCERNS / FAIL.

All 8 categories and their criteria

Scoring and gate outcomes

What it is: Detailed definitions for the Probability (P) and Impact (I) axes used in the Risk Score formula from risk-governance.md. Without explicit definitions, two team members scoring the same risk often choose different values. This file eliminates that ambiguity.

Probability Scale (P)

Impact Scale (I)

Full 3×3 Score Matrix

Common real-world score examples

What it is: Defines the specific criteria, thresholds, and tooling used to evaluate Non-Functional Requirements (NFRs) across four domains: Security, Performance, Reliability, and Maintainability. These criteria are what the bmad-testarch-nfr skill checks when performing an NFR evidence audit.

Important distinction: TEA does not run these tests. It evaluates the evidence that already exists (k6 reports, Playwright results, CI dashboards) against these criteria to produce a gate decision.

NFR gate outcomes

What it is: Defines which HTML element selection strategies to use in UI tests, ranked from most to least resilient. Poor selector choices are the #1 cause of tests breaking after a UI redesign even when the underlying functionality is unchanged.

Priority hierarchy (use #1 whenever possible)

1. page.getByTestId('submit-btn')

   Best: Uses data-testid attribute — added by developers specifically for tests. Does not change when the UI is redesigned, text is translated, or CSS classes are renamed. Requires team agreement to add data-testid to meaningful interactive elements.
2. page.getByRole('button', { name: 'Submit' })

   Best: ARIA semantic selector — also tests accessibility at the same time. Stable across visual redesigns. Reflects how assistive technology sees the page.
3. page.getByLabel('Email address')

   Good: Finds the input associated with a label. Stable until the label text changes. Also validates correct label/input association.
4. page.getByText('Submit')

   Acceptable: Text-based selection. Fragile if the text is internationalised or reworded. Use regex (/Submit|Enviar/) for multilingual apps.
5. .submit-btn · #mainSubmit · form button:nth-of-type(2)

   Avoid: CSS classes are implementation details that change with every refactor. Positional selectors (nth) break when the DOM order changes. Never acceptable for tests that need to remain stable.
6. //button[@class='btn btn-primary']

   Never: XPath is verbose, brittle, and incompatible with Playwright's recommended locator patterns. The Playwright team explicitly discourages XPath use.

Filtering from a list — filter() not nth()

// ❌ BAD: positional — breaks when list order changes
await page.locator('.user-row').nth(2).click();

// ✅ GOOD: semantic filter — finds by content, not position
await page.locator('.user-row')
  .filter({ hasText: 'Jane Smith' })
  .getByRole('button', { name: 'Edit' })
  .click();

Migration guide — before and after

What it is: Catalogs the most common timing mistakes in UI and API testing, provides the correct event-based replacements, and documents the debugging tools available when a test produces intermittent failures on CI.

Why it matters: Timing issues cause tests that "work on my machine" but fail 30% of the time on CI. The root cause is almost always a hard-coded wait that is too short on a slow machine, or a missing wait that relies on implicit browser timing.

Complete timing anti-patterns table

Event-based waiting patterns

// Pattern 1: Wait for a specific API response after an action
const savePromise = page.waitForResponse('**/api/users/**');
await page.getByRole('button', { name: 'Save' }).click();
const response = await savePromise;
expect(response.status()).toBe(200);

// Pattern 2: Wait for element to reach expected state
await page.getByText('Changes saved').waitFor({ state: 'visible' });

// Pattern 3: Wait for navigation after form submit
await Promise.all([
  page.waitForURL('/dashboard'),
  page.getByRole('button', { name: 'Login' }).click()
]);

// Pattern 4: Poll for async background job completion
await recurse(
  () => apiRequest({ method: 'GET', path: `/jobs/${jobId}` }),
  (res) => res.body.status === 'completed',
  { timeout: 60_000, interval: 2_000 }
);

Debugging workflow for flaky tests

What it is: An 8-point checklist that defines what a production-quality test looks like. Used by the bmad-testarch-review skill when auditing an existing test suite. A test must pass all 8 criteria to be considered acceptable for production.

The 8 quality criteria — with examples

• No hard-coded waits — Zero waitForTimeout, sleep, or cy.wait(N) calls. All waits are event-based.
  Bad: await page.waitForTimeout(3000) · Good: await page.waitForResponse('**/api/users')
• No conditional logic inside the test — No if/else, try/catch, or ternary expressions in the test body. Each test must have exactly one deterministic execution path.
  Bad: if (await button.isVisible()) { await button.click() }
• Test body under 300 lines — If a test is longer than 300 lines, shared logic must be extracted into fixtures or utility functions. Long tests are hard to read and impossible to maintain.
• Completes within 1.5 minutes — Including setup, execution, and teardown. If a test consistently takes longer, it should be split into focused smaller tests or moved to a nightly suite.
• Self-cleaning — The test deletes all data it creates, regardless of whether it passes or fails. Cleanup runs in afterEach (not afterAll) to prevent data leaking to the next test.
• Specific, unambiguous assertions — Assertions check exact values, not just truthy/falsy states. No toBeTruthy(), no toBeGreaterThan(0) when an exact value is known.
  Bad: expect(user).toBeTruthy() · Good: expect(user.id).toBe('usr_123')
• Unique test data every run — All strings, emails, names, and IDs used in the test are generated with faker or crypto.randomUUID(). Never hardcoded static values that would conflict with a parallel test run.
• Parallel-safe — The test does not depend on other tests running before or after it. Does not modify shared state (global users, shared feature flags, hardcoded IDs). Can run in any order, at any time, on any worker.

How to score a test against these criteria

What it is: Five structured repair patterns for the most common categories of broken or flaky tests. When a test starts failing, diagnose the failure type, then apply the matching pattern. Each pattern has a diagnostic symptom, root cause, and step-by-step fix.

How to use this file

Pattern 1 — Stale Selector

Pattern 2 — Race Condition

// Before: assertion fires before API response arrives
await page.getByRole('button', { name: 'Save' }).click();
await expect(page.getByText('Saved')).toBeVisible();  // ← flaky

// After: wait for the API call that triggers the success message
const saved = page.waitForResponse('**/api/profile');
await page.getByRole('button', { name: 'Save' }).click();
await saved;                                                  // ← deterministic
await expect(page.getByText('Saved')).toBeVisible();

Pattern 3 — Dynamic Data Mismatch

// Bad: hardcoded value that depends on database state
expect(response.body.name).toBe('John Smith');

// Good: assert against what your factory generated
const userData = createUserData();
const user = await createUser(userData);
expect(response.body.name).toBe(userData.name);  // always matches

Pattern 4 — Network Timeout

Pattern 5 — Hard Wait Replacement

What it is: A testing approach where the interface contract between a consumer (the service that calls an API) and a provider (the service that hosts the API) is formally captured and verified. The contract is a living document — generated by the consumer and verified by the provider, enabling both sides to evolve independently without breaking each other.

Primary tooling: Pact.js (contract generation and verification) + PactFlow (contract broker — stores contracts, tracks verification status). The can-i-deploy CLI checks whether both parties have verified the current contract before allowing a deployment to proceed.

How contract testing flows

Consumer-side test: defining the contract

import { PactV3, MatchersV3 } from '@pact-foundation/pact';
const { like, eachLike, regex } = MatchersV3;

const provider = new PactV3({
  consumer: 'user-management-ui',
  provider: 'user-api',
  dir:      './pacts',
});

it('returns a user by ID', () =>
  provider
    .given('a user with ID usr_123 exists')
    .uponReceiving('a GET request for usr_123')
    .withRequest({ method: 'GET', path: '/api/users/usr_123' })
    .willRespondWith({
      status: 200,
      body: {
        id:    like('usr_123'),       // any string is OK
        email: regex('.+@.+\\..+', 'u@e.com'),
        roles: eachLike('viewer'),    // array, at least 1 item
      }
    })
    .executeTest(async (mockServer) => {
      const user = await new UserApiClient(mockServer.url).getUser('usr_123');
      expect(user.id).toBeDefined();
    })
);

Provider-side verification

import { Verifier } from '@pact-foundation/pact';

it('validates the contract from PactFlow', () =>
  new Verifier({
    providerBaseUrl:           'http://localhost:8080',
    pactBrokerUrl:             process.env.PACTFLOW_URL,
    pactBrokerToken:           process.env.PACTFLOW_TOKEN,
    provider:                  'user-api',
    publishVerificationResult: true,
    providerVersion:           process.env.GIT_SHA,
    stateHandlers: {
      'a user with ID usr_123 exists': async () => {
        await db.createUser({ id: 'usr_123', email: 't@e.com', roles: ['viewer'] });
      }
    }
  }).verifyProvider()
);

can-i-deploy gate in CI

# In the consumer's deploy pipeline
pact-broker can-i-deploy \
  --pacticipant user-management-ui \
  --version $GIT_SHA \
  --to-environment production \
  --broker-base-url $PACTFLOW_URL
# Exit 0 = proceed · Exit 1 = blocked (verification missing or failed)

Postel's Law — the asymmetry rule

When to use contract testing

What it is: A utility that persists authentication tokens to disk and reuses them across test runs and parallel workers — no re-authentication before every test. Supports multiple simultaneous user identities, ephemeral (one-time) auth, and worker-specific accounts for fully parallel execution.

Why it matters: Playwright's built-in authentication re-authenticates on every run, only supports a single user, and does not handle token expiration. With auth-session, a token is acquired once and reused everywhere — login overhead drops from ~10 s per test to <1 ms.

Global setup lifecycle — 4 calls in order

Custom AuthProvider interface — 5 methods

// Implement all 5 methods to plug in any authentication system
interface AuthProvider {
  getEnvironment():           string;            // 'local' | 'staging' | 'production'
  getUserIdentifier():        string;            // unique cache key per user role
  extractToken(response):    string;            // parse token from login response
  isTokenExpired(token):     boolean;           // check if cached token is still valid
  manageAuthToken():         Promise<string>;   // acquire or refresh token
}

Authentication modes

API-only auth — no browser required

// Return the raw token string — auth-session handles caching automatically
async manageAuthToken(): Promise<string> {
  const res = await fetch(`${baseUrl}/api/auth/login`, {
    method: 'POST',
    body: JSON.stringify({ email, password })
  });
  const { token } = await res.json();
  return token;   // no browser, no cookies — pure API token flow
}

// Using a different user role in a specific test file
test.use({ authOptions: { userIdentifier: 'admin-user' } });

test('admin sees delete button', async ({ page }) => {
  await page.goto('/users');
  await expect(page.getByRole('button', { name: 'Delete' })).toBeVisible();
});

• AlwaysCall authStorageInit() first in globalSetup — if the storage directory does not exist, the other functions fail silently or throw obscure errors.
• AlwaysUse userIdentifier overrides for role-based tests — never share one token across tests that need different permission levels.
• NeverHard-code tokens in test files — they expire, rotate, and get committed to version control where they become security liabilities.
• NeverCall logout actions in tests that use cached tokens — a logout invalidates the shared token for all subsequent tests in the run.

What it is: A typed HTTP client that wraps Playwright's request context with automatic JSON response parsing, schema validation (Zod, JSON Schema, OpenAPI), automatic retry on 5xx errors with exponential backoff, and a 4-tier URL resolution strategy. No browser required — ideal for pure API and service-layer tests.

Why it matters: Vanilla Playwright requires boilerplate for every API call: response.json(), manual status checking, custom retry loops, zero type safety. apiRequest replaces all of that with a single typed call and catches contract drift through schema validation.

Basic usage — request and destructure

// Response body is always pre-parsed JSON — no await response.json() needed
const { status, body } = await apiRequest({
  method: 'POST',
  path:   '/api/users',
  body:   { name: 'Jane Smith', role: 'admin' }
});
expect(status).toBe(201);
expect(body.id).toBeDefined();

URL resolution — 4-tier priority (highest to lowest)

Schema validation — Zod, JSON Schema, OpenAPI

import { z } from 'zod';

const UserSchema = z.object({
  id:    z.string(),
  email: z.string().email(),
  roles: z.array(z.string())
});

// Validation throws immediately if response shape doesn't match the schema
// Catches contract drift the moment a field is renamed or removed
const { body } = await apiRequest({
  method:         'GET',
  path:           '/api/users/usr_001',
  validateSchema: UserSchema     // Zod schema, JSON Schema object, or YAML OpenAPI path
});
// body is now typed as { id: string; email: string; roles: string[] }

Retry behavior — automatic

GraphQL — always check body.errors

// GraphQL always returns HTTP 200, even for errors — check body.errors not status
const { body } = await apiRequest({
  method: 'POST',
  path:   '/graphql',
  body: {
    query:     `query GetUser($id: ID!) { user(id: $id) { id email roles } }`,
    variables: { id: 'usr_001' }
  }
});
expect(body.errors).toBeUndefined();     // ← mandatory GraphQL error check
expect(body.data.user.email).toBeDefined();

Combine with recurse for polling

// Poll an async job until it completes — apiRequest + recurse combo
await recurse(
  () => apiRequest({ method: 'GET', path: `/api/jobs/${jobId}` }),
  (res) => res.body.status === 'completed',
  { timeout: 60_000, interval: 2_000 }
);

• AlwaysAdd validateSchema for responses at P0/P1 API endpoints — catches breaking provider changes at the moment they happen, not when a UI test fails weeks later.
• AlwaysCheck body.errors for GraphQL responses — HTTP 200 does not mean success in GraphQL.
• NeverUse apiRequest for WebSocket or Server-Sent Events — it is HTTP-only. Use Playwright's page.on('websocket') for those.

What it is: A polling utility that repeatedly executes a command until a predicate returns true, with configurable timeout, interval, logging, and post-poll callbacks. Returns typed error shapes for precise diagnosis. Fills the gap in Playwright's native expect.poll: no built-in logging, generic timeout messages, no post-success hooks.

Primary use cases: Background job completion, webhook delivery, email arrival, database eventual consistency, cache propagation, state machine transitions — any scenario where you must wait for an asynchronous outcome without knowing when it will arrive.

Core signature

await recurse(
  command,    // () => Promise<T>   — executed on every retry
  predicate,  // (result: T) => boolean | void  — return truthy OR run assertions
  options?    // RecurseOptions
)

Options reference

Three predicate styles

// Style 1: return truthy value
await recurse(
  () => apiRequest({ method: 'GET', path: `/api/jobs/${jobId}` }),
  (res) => res.body.status === 'completed',
  { timeout: 60_000, interval: 2_000 }
);

// Style 2: run assertions (no return needed — passes if assertions don't throw)
await recurse(
  () => apiRequest({ method: 'GET', path: '/api/emails/latest' }),
  (res) => {
    expect(res.body.subject).toContain('Welcome');
    expect(res.body.to).toBe('user@example.com');
  }
);

// Style 3: with post-poll callback after success
await recurse(
  () => getOrderStatus(orderId),
  (status) => status === 'shipped',
  { post: (s) => log.success(`Order ${orderId} shipped`), timeout: 30_000 }
);

Three typed error shapes

What it is: A single declarative call to intercept browser network requests — either observing real requests as they pass through (spy mode) or replacing them with a fabricated response (stub mode). Automatically parses JSON and returns a structured result with responseJson, status, and requestBody.

Why it matters: Vanilla Playwright requires three separate steps: page.route() for setup, page.waitForResponse() for capture, and manual response.json(). interceptNetworkCall collapses all three into one call and eliminates 60–70% of the boilerplate.

Mandatory setup order — intercept BEFORE navigate

Spy mode vs Stub mode

// ✅ SPY — real request passes through; captured for assertion
const usersCall = interceptNetworkCall({ url: '**/api/users' });
await page.goto('/dashboard');
const { responseJson, status } = await usersCall;
expect(status).toBe(200);
expect(responseJson).toHaveLength(3);

// ✅ STUB — request intercepted; fake response returned; server never receives it
const errorCall = interceptNetworkCall({
  url: '**/api/orders',
  fulfillResponse: { status: 503, body: { error: 'Service Unavailable' } }
});
await page.goto('/orders');
await errorCall;
await expect(page.getByRole('alert')).toContainText('Something went wrong');

URL glob patterns

Multiple intercepts in one action

// Set up ALL intercepts BEFORE triggering the action that fires them
const [usersCall, rolesCall] = [
  interceptNetworkCall({ url: '**/api/users' }),
  interceptNetworkCall({ url: '**/api/roles' })
];
await page.goto('/admin');                   // fires both requests
const users = await usersCall;
const roles = await rolesCall;

What it is: A utility that records network traffic to HAR (HTTP Archive) files during test runs, then replays those files offline — no live backend required. Unlike native Playwright HAR, it supports stateful CRUD during playback: a POST that creates a resource will appear in subsequent GET responses within the same test.

Why it matters: Traditional E2E tests require a live backend (slow, flaky, expensive). HAR-based testing provides deterministic, fast, offline tests — the same server responses every run, independent of network conditions or server state.

Two modes — one env variable

Setup — always before page.goto()

// networkRecorder.setup(context) MUST be called before any navigation
test('create and list users', async ({ page, context, networkRecorder }) => {
  await networkRecorder.setup(context);   // ← must come first
  await page.goto('/users');
  // record mode: real requests fired and persisted to HAR
  // playback mode: requests served from HAR, server never contacted
});

Stateful CRUD during playback

The recorder analyzes the HAR file, detects CRUD patterns by HTTP method and URL shape, and maintains an in-memory state during playback. This means full CRUD flows work without a backend:

• NotePOST /api/users — creates a user and adds it to the in-memory list.
• NoteGET /api/users (after the POST) — returns the list including the newly created user.
• NoteDELETE /api/users/id — removes the item so future GETs reflect the deletion.
• DiffNative Playwright HAR returns static responses only — it does NOT track state changes between requests.

Cross-environment URL mapping

// HAR recorded on staging, replayed in CI against localhost
await networkRecorder.setup(context, {
  hostMapping: {
    'staging.api.example.com': 'localhost:3000'
  },
  patterns: [
    { regex: /staging\.api/, replace: 'localhost:3000' }
  ]
});

What it is: A fixture that automatically fails any test where a background HTTP request returns 4xx or 5xx — even when the UI appears to work correctly. Zero configuration: import the fixture and monitoring starts for every test in that file. Think of it as Sentry for your test suite.

Why it matters: Backend errors can occur silently while the UI renders a success state (cached data, optimistic updates, or incomplete error handling). Without this monitor, tests pass falsely and bugs reach production. With it, any unexpected API error surfaces immediately as a test failure.

Activation — import only

// Import the fixture — monitoring is ON for every test in this file
import { test } from '@seontechnologies/playwright-utils/network-error-monitor/fixtures';

test('dashboard loads without errors', async ({ page }) => {
  await page.goto('/dashboard');
  // If ANY background API call returns 4xx/5xx → test fails automatically
  await expect(page.getByText('Welcome')).toBeVisible();
});

Opt-out for intentional error scenarios

// Tests that deliberately trigger 4xx/5xx must opt out
test('shows 404 message for missing resource', {
  annotation: [{ type: 'skipNetworkMonitoring' }]
}, async ({ page }) => {
  await page.goto('/items/nonexistent-id');
  await expect(page.getByText('Item not found')).toBeVisible();
});

Configuration options

Artifacts — network-errors.json

On failure the monitor writes a network-errors.json artifact alongside the HTML report. Each entry contains: url, status, method, timestamp.

What it is: A 6-level structured logging utility that surfaces output inside Playwright's HTML report as collapsible test steps — not just buried in the terminal. Automatically formats objects and arrays as indented JSON. Supports optional file-based logging for persistent logs across CI runs.

Why it matters: console.log() is invisible in HTML reports, creates no navigable steps, and disappears in terminal noise. Structured logging makes debugging visible exactly where it matters — inside the report, adjacent to the failing assertion.

6 log levels

AAA pattern with structured logging

import { log } from '@seontechnologies/playwright-utils';

test('create user and verify', async ({ apiRequest }) => {
  await log.step('ARRANGE: Build test user data');
  const userData = createUserData({ role: 'admin' });
  await log.debug(userData);              // auto-formats object as JSON in report

  await log.step('ACT: POST /api/users');
  const { status, body } = await apiRequest({
    method: 'POST', path: '/api/users', body: userData
  });

  await log.step('ASSERT: Verify 201 and user ID');
  expect(status).toBe(201);
  await log.success(`User created: ${body.id}`);
});

Sync variants for globalSetup / globalTeardown

// Async test context is not available in globalSetup — use sync methods
log.infoSync('Global setup: seeding database...');
log.successSync('Seed completed — 5 users, 3 orders');

// File logging for persistent artifacts across test runs
log.configure({
  fileLogging: { enabled: true, outputDir: './test-results/logs' }
});

• AlwaysUse log.step() before each major phase (ARRANGE / ACT / ASSERT) to create navigable sections in the HTML report.
• AlwaysUse log.debug(object) to inspect API responses or large data structures — it auto-formats as indented JSON.
• NeverLog sensitive data — passwords, auth tokens, PII, credit card numbers. CI HTML reports may be publicly accessible or archived.

What it is: Utilities that simplify file download handling in Playwright tests — capturing the download event and reading the file's content from CSV, Excel/XLSX, PDF, or ZIP formats. Reduces ~80 lines of boilerplate (event orchestration, path resolution, parsing config, error handling) to 2–10 lines.

Why it matters: File download testing requires orchestrating the download event, waiting for the file to land on disk, then parsing it with the right library — all before writing a single assertion. These utilities automate the entire pipeline.

Capturing a download

import { handleDownload, readCSV } from '@seontechnologies/playwright-utils/file-utils';

// handleDownload orchestrates: register event → trigger click → wait → return path
const filePath = await handleDownload({
  page,
  downloadDir: './test-results/downloads',
  trigger:     () => page.getByRole('button', { name: 'Export CSV' }).click()
});

Supported formats

Complete CSV export test

import * as fs from 'fs-extra';
const DOWNLOAD_DIR = './test-results/downloads';

test.afterEach(async () => fs.remove(DOWNLOAD_DIR));  // always clean up

test('users CSV export contains correct headers and data', async ({ page }) => {
  await page.goto('/admin/users');

  const filePath = await handleDownload({
    page, downloadDir: DOWNLOAD_DIR,
    trigger: () => page.getByRole('button', { name: 'Export' }).click()
  });

  const { data, headers } = await readCSV({ filePath });
  expect(headers).toEqual(expect.arrayContaining(['email', 'role', 'createdAt']));
  expect(data).toHaveLength(5);
  expect(data[0].email).toMatch(/.+@.+\..+/);
});

What it is: A pattern for combining fixtures from multiple sources — playwright-utils utilities and custom project-specific fixtures — into a single unified test object using Playwright's mergeTests(). Every test file imports from one place, updating a fixture propagates everywhere automatically.

Why it matters: Without composition, each test file must import from 4–6 different fixture modules. When a fixture's source path or API changes, every import across the project must be updated. The merged export is the single source of truth — change one file, all tests adapt.

Building the merged export

// tests/support/merged-fixtures.ts — the single import for ALL test files
import { mergeTests }          from '@playwright/test';

// playwright-utils fixtures
import { test as apiFixture }       from '@seontechnologies/playwright-utils/api-request/fixtures';
import { test as authFixture }      from '@seontechnologies/playwright-utils/auth-session/fixtures';
import { test as logFixture }       from '@seontechnologies/playwright-utils/log/fixtures';
import { test as monitorFixture }   from '@seontechnologies/playwright-utils/network-error-monitor/fixtures';

// custom project fixtures
import { test as userFixture }   from './fixtures/user.fixture';
import { test as orderFixture }  from './fixtures/order.fixture';

export const test = mergeTests(
  apiFixture, authFixture, logFixture, monitorFixture,
  userFixture, orderFixture
);
export { expect } from '@playwright/test';

// In every test file — one clean import, all fixtures available
// import { test, expect } from '../../support/merged-fixtures';

Name conflict rules

Lazy evaluation — zero cost for unused fixtures

// A test that doesn't request networkRecorder pays NO setup cost for it
test('fast API-only test', async ({ apiRequest, log }) => {
  // Only apiRequest and log fixtures are initialized — the rest are skipped
  const { status } = await apiRequest({ method: 'GET', path: '/api/health' });
  expect(status).toBe(200);
});

What it is: Patterns for direct API testing without browser overhead — REST CRUD, GraphQL, schema validation, error shape assertion, and async operation polling. API tests using apiRequest run 100× faster than E2E tests covering the same logic and produce more actionable failure messages because there is no DOM noise obscuring the root cause.

When to use over E2E: CRUD operations, business logic validation, error handling responses (4xx shapes), authentication flows, background jobs, service-to-service communication — any scenario where the browser is not the subject under test.

CRUD test scaffold

import { test, expect } from '../support/merged-fixtures';
import { createUserData } from '../support/factories/user.factory';

test.describe('Users API', () => {
  let userId: string;

  test.beforeAll(async ({ apiRequest }) => {
    const { body } = await apiRequest({
      method: 'POST', path: '/api/users', body: createUserData()
    });
    userId = body.id;
  });

  test.afterAll(async ({ apiRequest }) => {
    await apiRequest({ method: 'DELETE', path: `/api/users/${userId}` });
  });

  test('GET returns created user', async ({ apiRequest }) => {
    const { status, body } = await apiRequest({
      method: 'GET', path: `/api/users/${userId}`,
      validateSchema: UserSchema      // Zod schema — fails if shape drifts
    });
    expect(status).toBe(200);
    expect(body.id).toBe(userId);
  });

  test('PATCH updates user role', async ({ apiRequest }) => {
    const { status, body } = await apiRequest({
      method: 'PATCH', path: `/api/users/${userId}`,
      body: { role: 'admin' }
    });
    expect(status).toBe(200);
    expect(body.role).toBe('admin');
  });
});

Asserting error shape — not just status code

// Test that invalid input returns a structured, parseable error body
test('POST rejects missing required fields', async ({ apiRequest }) => {
  const { status, body } = await apiRequest({
    method: 'POST', path: '/api/users',
    body: { email: 'not-an-email' }     // missing name, missing valid email
  });
  expect(status).toBe(422);
  expect(body.errors).toEqual(
    expect.arrayContaining([
      expect.objectContaining({ field: 'email' }),
      expect.objectContaining({ field: 'name' })
    ])
  );
});

Async job polling — apiRequest + recurse

// Trigger a long-running job, then poll until it completes
test('bulk import job completes successfully', async ({ apiRequest }) => {
  const { body: job } = await apiRequest({
    method: 'POST', path: '/api/jobs/bulk-import',
    body: { fileKey: 's3://bucket/users.csv' }
  });
  expect(job.status).toBe('queued');

  await recurse(
    () => apiRequest({ method: 'GET', path: `/api/jobs/${job.id}` }),
    (res) => res.body.status === 'completed',
    { timeout: 60_000, interval: 3_000 }
  );

  const { body: result } = await apiRequest({ method: 'GET', path: `/api/jobs/${job.id}/results` });
  expect(result.importedCount).toBe(100);
  expect(result.failedCount).toBe(0);
});

API vs E2E — decision guide

What it is: A Test-Driven Development discipline for UI components: write a failing test first, implement the minimum code to make it pass, then refactor with passing tests as a safety net. Covers React/Vue components with Playwright Component Tests, Cypress cy.mount, or React Testing Library. Includes accessibility and keyboard navigation assertions from the start.

Why it matters: TDD for components provides immediate feedback — failing tests clarify requirements before code is written, minimal implementations prevent over-engineering, and tests act as a regression guard during refactoring. Without TDD, components grow beyond their tested scope and become untestable monoliths.

The Red-Green-Refactor loop

Provider isolation — fresh context per test

// Wrap every component with fresh provider instances to prevent state bleed
function AllTheProviders({ children }: { children: ReactNode }) {
  // new QueryClient() per render — prevents cache contamination between parallel tests
  return (
    <QueryClientProvider client={new QueryClient()}>
      <MemoryRouter>
        <AuthProvider>{children}</AuthProvider>
      </MemoryRouter>
    </QueryClientProvider>
  );
}

// Pass as the wrapper option to RTL render
render(<UserCard userId="usr_001" />, { wrapper: AllTheProviders });

// In Playwright Component Tests
await mount(<AllTheProviders><UserCard userId="usr_001" /></AllTheProviders>);

Accessibility testing in the TDD loop

// Include a11y assertions from the very first test — not as an afterthought
import { AxeBuilder } from '@axe-core/playwright';

test('UserCard has no accessibility violations', async ({ page, mount }) => {
  await mount(<UserCard userId="usr_001" />);
  const results = await new AxeBuilder({ page }).analyze();
  expect(results.violations).toEqual([]);
});

Keyboard navigation testing

// Test Tab, Enter, Escape flows — not just mouse interactions
test('dropdown is fully keyboard navigable', async ({ page }) => {
  await page.getByRole('combobox', { name: 'Status' }).focus();
  await page.keyboard.press('Enter');           // open dropdown
  await expect(page.getByRole('listbox')).toBeVisible();
  await page.keyboard.press('ArrowDown');        // navigate to first option
  await page.keyboard.press('Enter');           // select it
  await expect(page.getByRole('combobox')).toHaveValue('Active');
  await page.keyboard.press('Escape');           // close — listbox must hide
  await expect(page.getByRole('listbox')).not.toBeVisible();
});

• RuleWrite the failing test before any implementation code. If you can't write the test first, the component API is unclear — clarify requirements before coding.
• RuleEach render must use a fresh provider instance. Shared provider state between tests creates ordering dependencies and non-deterministic failures in parallel runs.
• AvoidVisual regression (screenshot comparison) as primary TDD assertions — they are expensive to maintain and break on trivial styling changes. Use them selectively for visually critical components only.

What it is: Configuration and workflows for capturing rich debugging artifacts — traces, screenshots, videos, HAR files — only on failure, plus tooling for time-travel debugging through Playwright's Trace Viewer and interactive Inspector. Reduces CI failure triage time by 80–90% by making remote debugging possible from a single trace.zip.

Why it matters: CI failures with minimal context force developers to reproduce them locally — a process that can take hours. Trace files, screenshots, and videos captured at the exact moment of failure show what the test saw, when, and why it failed.

Recommended artifact config in playwright.config.ts

export default defineConfig({
  use: {
    trace:      'retain-on-failure-and-retries',  // keeps all retry attempts
    screenshot: 'only-on-failure',               // saved as PNG in test-results/
    video:      'retain-on-failure',              // only for failed tests
  }
});

Trace Viewer — 5 panels

Opening and analyzing traces

# Open a specific trace file (downloaded from CI artifacts)
npx playwright show-trace test-results/users-create-chromium/trace.zip

# CLI trace analysis — useful inside coding agents (no GUI needed)
npx playwright trace actions --grep="expect" trace.zip
npx playwright trace snapshot 42 trace.zip    # DOM snapshot at action #42

Interactive Inspector — local debugging

# Run in debug mode — Inspector opens automatically, pauses before first action
PWDEBUG=1 npx playwright test users/create.spec.ts

# Or pause at a specific point inside test code
await page.pause();   // execution stops; Inspector opens for interactive exploration

# Step through the test action by action, run locator queries live
npx playwright test --debug users/create.spec.ts

CI artifact upload — GitHub Actions

# Upload all test artifacts even when the job fails
- name: Upload test results
  uses: actions/upload-artifact@v4
  if: always()                     # critical — must upload on failure
  with:
    name: playwright-report-${{ matrix.shard }}
    path: |
      playwright-report/
      test-results/
    retention-days: 30

• NeverUse video: 'on' in CI — records video for every test regardless of outcome. Generates gigabytes of files and significantly slows execution. Use 'retain-on-failure'.
• NeverUse trace: 'on' in CI for the same reason — trace files are large. 'retain-on-failure-and-retries' captures what you need without the overhead.
• AlwaysUpload test artifacts with if: always() in CI — without this, artifacts are not saved when the job fails, which is exactly when you need them.

What it is: Opinionated patterns for playwright.config.ts: centralized multi-environment URL management via envConfigMap, standardized timeout values, reporter configuration, parallelization and sharding, and auth-state project structure. The config file is the single source of truth for all environment and execution behaviour — when it drifts, every test becomes suspect.

Multi-environment URL management — envConfigMap

import { defineConfig } from '@playwright/test';

const envConfigMap: Record<string, { baseURL: string }> = {
  local:      { baseURL: 'http://localhost:3000'        },
  staging:    { baseURL: 'https://staging.example.com'  },
  production: { baseURL: 'https://example.com'          },
};

const env = process.env.TEST_ENV;
if (!env || !envConfigMap[env]) {
  // Fail-fast: unknown environment is always a mistake, never a default
  throw new Error(
    `TEST_ENV must be one of: ${Object.keys(envConfigMap).join(', ')}. Got: "${env}"`
  );
}

export default defineConfig({
  use: { ...envConfigMap[env] },
});

Standardized timeout values

Parallelization and sharding

import os from 'os';

export default defineConfig({
  fullyParallel: true,
  workers:  process.env.CI ? 1 : os.cpus().length - 1,  // 1 worker in CI avoids contention
  retries:  process.env.CI ? 2 : 0,                        // retries only in CI, never locally
});

# Shard the suite across 4 CI machines — each runs 25% of tests
# npx playwright test --shard=1/4   (machine 1)
# npx playwright test --shard=2/4   (machine 2)  ... etc

Auth-state projects — setup once, reuse everywhere

export default defineConfig({
  projects: [
    // Step 1: run auth setup once; saves auth state to disk
    { name: 'setup', testMatch: '**/auth.setup.ts' },

    // Step 2: authenticated tests depend on setup project
    {
      name:         'authenticated',
      dependencies: ['setup'],
      use:          { storageState: 'playwright/.auth/user.json' }
    },

    // Step 3: unauthenticated tests run independently
    { name: 'unauthenticated' }
  ]
});

Reporter configuration

export default defineConfig({
  reporter: [
    ['html',  { outputFolder: 'playwright-report', open: 'never' }],  // visual HTML report
    ['junit', { outputFile: 'test-results/results.xml' }],            // CI/CD parsing
    ['list']                                                            // terminal output
  ]
});

What it is: A CI pipeline architecture that runs changed test files 10× in a loop (burn-in) before allowing merge, then shards the full suite across multiple machines with fail-fast: false. Burn-in stress-tests new or modified tests to flush out flakiness before it reaches the main branch.

Why it matters: A test that passes once locally can still be flaky — it may fail 1 in 10 runs due to a race condition or timing issue. Running it 10 times in CI catches this before the test is trusted. fail-fast: false ensures all shards complete even when one fails, preserving the full evidence set for post-mortem analysis.

Pipeline stages

Burn-in loop — detect changed specs and repeat

# Detect changed spec files relative to main branch
CHANGED_SPECS=$(git diff --name-only origin/main...HEAD \
  | grep -E '\.(spec|test)\.(ts|js)$' \
  | tr '\n' ' ')

# Run changed specs 10 times — exit 1 on first failure
for i in {1..10}; do
  echo "Burn-in iteration $i/10"
  npx playwright test $CHANGED_SPECS || exit 1
done

Sharded full suite — GitHub Actions matrix

# .github/workflows/playwright.yml
jobs:
  test:
    strategy:
      fail-fast: false          # all shards run even if one fails
      matrix:
        shard: [1, 2, 3, 4]
    steps:
      - run: npx playwright test --shard=${{ matrix.shard }}/4

  merge-reports:
    needs: test
    if: always()                # run even when tests fail
    steps:
      - run: npx playwright merge-reports ./all-blobs --reporter html

Dependency cache

# Cache node_modules — busts when package-lock.json changes
- uses: actions/cache@v4
  with:
    path: node_modules
    key: ${{ runner.os }}-node-${{ hashFiles('package-lock.json') }}

• RuleAlways set fail-fast: false in the sharding matrix. A single failing shard should not prevent the other shards from running — you need the full failure picture to diagnose systemic issues.
• NeverUse ${{ inputs.* }} directly inside run: blocks — this is a script injection vulnerability. Assign to an env var first and reference that instead.
• AlwaysUpload test artifacts with if: always() so that trace files are available even when the job fails — which is the only time they matter.

What it is: A strategy for running only the tests that matter at each stage of the development workflow, using tags (@smoke, @p0–@p3, @regression) and changed-file detection. Tests are promoted through stages: smoke (<5 min) → PR (<10 min) → full regression (<30 min). Running the full suite on every commit wastes developer time without adding safety.

Tag taxonomy

Tagging tests

// Tag with priority and type — multiple tags allowed
test('@smoke @p0 login flow works', async ({ page }) => { ... });
test('@p1 search returns relevant results', async ({ page }) => { ... });
test('@p2 CSV export downloads correctly', async ({ apiRequest }) => { ... });
test('@p3 button hover shows tooltip', async ({ page }) => { ... });

Executing by tag

# Run only smoke tests (every commit)
npx playwright test --grep "@smoke"

# Run P0 + P1 (PR check)
npx playwright test --grep "@p0|@p1"

# Run full regression (pre-merge)
npx playwright test --grep "@regression"

# Exclude a tag (skip nightly-only tests)
npx playwright test --grep-invert "@p3"

Changed-file detection → tag mapping

# Map changed files to test tags automatically
CHANGED=$(git diff --name-only origin/main...HEAD)

if echo "$CHANGED" | grep -q "src/components/"; then
  TAGS="@smoke|@p0|@p1"          # UI changed → run component + E2E tests
elif echo "$CHANGED" | grep -q "src/api/"; then
  TAGS="@p0|@p1"                  # API changed → run integration + E2E tests
else
  TAGS="@smoke"                    # other changes → smoke only
fi

npx playwright test --grep "$TAGS"

Test promotion — three stages

What it is: A structured approach to managing feature flags with TypeScript type safety, metadata tracking (owner, expiry date, dependencies), comprehensive testing of both enabled and disabled states, and automated cleanup detection. Without governance, flags accumulate as technical debt — untested variations ship broken code and forgotten flags clutter the codebase indefinitely.

Centralized flag registry with type safety

// src/feature-flags.ts — single source of truth for all flags
export const FLAGS = Object.freeze({
  NEW_CHECKOUT_FLOW:    'new-checkout-flow',
  DARK_MODE:            'dark-mode',
  AI_SEARCH_ENABLED:    'ai-search-enabled',
} as const);

// Type-safe key — typos caught at compile time, not at runtime
export type FlagKey = (typeof FLAGS)[keyof typeof FLAGS];

// Metadata registry — governance information per flag
export const FLAG_REGISTRY: Record<FlagKey, FlagMetadata> = {
  [FLAGS.NEW_CHECKOUT_FLOW]: {
    owner:           'payments-team',
    createdDate:     '2024-09-01',
    expiryDate:      '2024-11-01',    // drives automated audit
    defaultState:    false,
    requiresCleanup: true,
  },
};

Testing both flag states — mandatory for P0/P1 features

// Always test BOTH enabled and disabled — never assume one state
test.describe('new-checkout-flow flag', () => {

  test('@p0 enabled: shows new checkout UI', async ({ page, featureFlags }) => {
    await featureFlags.setVariation(FLAGS.NEW_CHECKOUT_FLOW, true);
    await page.goto('/checkout');
    await expect(page.getByTestId('new-checkout-form')).toBeVisible();
  });

  test('@p0 disabled: shows legacy checkout UI', async ({ page, featureFlags }) => {
    await featureFlags.setVariation(FLAGS.NEW_CHECKOUT_FLOW, false);
    await page.goto('/checkout');
    await expect(page.getByTestId('legacy-checkout-form')).toBeVisible();
  });

  test.afterEach(async ({ featureFlags }) => {
    await featureFlags.reset(FLAGS.NEW_CHECKOUT_FLOW); // restore default after each test
  });
});

Flag lifecycle checklist

Automated audit script

# Detect governance issues — run in CI on a schedule
npm run feature-flags:audit

# Reports:
# - Flags past their expiry date (need cleanup)
# - Flags with no owner assigned
# - Flags with requiresCleanup: true but no cleanup ticket linked
# - Flags present in code but missing from FLAG_REGISTRY

• RuleEvery flag must have an expiryDate in FLAG_REGISTRY. Flags without expiry dates are never cleaned up — they become permanent dead code.
• NeverTest only the enabled state. The disabled fallback is the default for most users until the flag is fully rolled out — a broken fallback is a broken product.
• NeverShare flag state between parallel tests. Use per-test featureFlags.setVariation() with afterEach cleanup — parallel tests will interfere with each other otherwise.

What it is: Patterns for writing explicit error handling in tests — scoped exception catching (only ignore documented errors), retry validation (prove the retry mechanism actually fires N times before succeeding), telemetry logging with secret redaction, and graceful degradation testing (cached fallbacks, non-critical service failures).

Why it matters: Tests fail for two reasons: genuine bugs or poor error handling in the test itself. Without explicit patterns, tests become noisy (false positives from expected errors) or silent (swallowing real errors). Resilience testing also ensures the app handles real-world failures gracefully rather than exposing stack traces to users.

Scoped exception catching — only ignore documented errors

// ❌ BAD: global pageerror listener silences all JS errors
page.on('pageerror', () => {});

// ✅ GOOD: scoped listener — only ignore the one known error in this test
const KNOWN_ERROR = 'ResizeObserver loop limit exceeded';
page.on('pageerror', (err) => {
  if (err.message.includes(KNOWN_ERROR)) return;  // known, ignore
  throw err;                                         // unexpected — re-throw
});

Retry validation — prove N failures then success

// Stub the API to fail twice, succeed on the 3rd attempt
let attempts = 0;
await page.route('**/api/orders', (route) => {
  attempts++;
  if (attempts <= 2) {
    route.fulfill({ status: 500, body: '{"error":"Server Error"}' });
  } else {
    route.fulfill({ status: 201, body: '{"id":"ord_001"}' });
  }
});

await page.getByRole('button', { name: 'Place Order' }).click();
// UI must show retry indicator during attempts 1-2
await expect(page.getByTestId('retry-indicator')).toBeVisible();
// UI must show success after attempt 3
await expect(page.getByText('Order placed!')).toBeVisible();
expect(attempts).toBe(3);  // confirm exactly 3 attempts were made

Telemetry logging with secret redaction

// Log error context — never log sensitive fields
const REDACT_KEYS = ['password', 'token', 'creditCard', 'ssn'];

function safeLog(context: Record<string, unknown>) {
  const safe = Object.fromEntries(
    Object.entries(context).map(([k, v]) =>
      REDACT_KEYS.some(r => k.toLowerCase().includes(r))
        ? [k, '[REDACTED]']
        : [k, v]
    )
  );
  await log.error(safe);
}

Graceful degradation — non-critical service failure

// Analytics service is down — app must continue working normally
await page.route('**/analytics/**', route => route.fulfill({ status: 503 }));

await page.goto('/dashboard');
// Critical features must still work despite analytics being down
await expect(page.getByTestId('dashboard-content')).toBeVisible();
await expect(page.getByRole('button', { name: 'Create' })).toBeEnabled();
// No crash, no blocking modal — user is not informed of non-critical failure

• RuleTest retry behavior with a counter — assert exactly N failures before success. Without the counter, you can't confirm the retry mechanism actually fired.
• RuleTest graceful degradation for every non-critical service the app calls (analytics, feature flags, recommendations). If any of these going down breaks a core flow, it's a P0 bug.
• NeverUse a global pageerror handler that swallows all errors. Scope it to the specific test that expects a known error.

What it is: Patterns for testing email-based authentication flows — magic links, registration confirmation codes, and password reset — using a real email capture service (Mailosaur recommended, Ethereal free, MailHog self-hosted). Covers session caching to minimize email consumption, link extraction, and negative test cases.

Why it matters: Email auth introduces unique testing challenges: asynchronous delivery (wait for email before clicking), quota limits (AWS Cognito: 50 emails/day in sandbox), and per-email costs in production-like environments. Session caching allows 500 tests to consume only 1 email instead of 500.

Email capture setup — Mailosaur

import { MailosaurClient } from 'mailosaur';
const mailosaur = new MailosaurClient(process.env.MAILOSAUR_API_KEY);

// Generate a unique inbox per test — never hardcode email addresses
const serverId = process.env.MAILOSAUR_SERVER_ID;
const testEmail = `user-${crypto.randomUUID()}@${serverId}.mailosaur.net`;

Magic link extraction flow

// Step 1: Trigger the email (registration, login, password reset)
await apiRequest({ method: 'POST', path: '/api/auth/magic-link', body: { email: testEmail } });

// Step 2: Wait for email and extract the link
const message = await mailosaur.messages.get(serverId, {
  sentTo: testEmail,
  timeout: 30_000               // wait up to 30s for delivery
});

// Method A: direct link from Mailosaur SDK
const magicLink = message.html.links[0].href;

// Method B: parse HTML with JSDOM for custom button selectors
const { JSDOM } = await import('jsdom');
const dom = new JSDOM(message.html.body);
const magicLink = dom.window.document.querySelector('#magic-link-button')?.href;

// Step 3: Click the link and assert authentication succeeded
await page.goto(magicLink);
await expect(page).toHaveURL('/dashboard');

Session caching — 1 email for 500 tests

// globalSetup: authenticate once, save state to disk
export default async function globalSetup() {
  // 1. Register + verify email once
  const email = `seed-user-${crypto.randomUUID()}@${serverId}.mailosaur.net`;
  await registerUserViaEmail(email);
  const link = await extractMagicLink(email);

  // 2. Navigate and save auth state (cookies + localStorage)
  const browser = await chromium.launch();
  const page = await browser.newPage();
  await page.goto(link);
  await page.context().storageState({ path: 'playwright/.auth/user.json' });
  await browser.close();
  // All 500 tests reuse playwright/.auth/user.json — 0 additional emails sent
}

Negative test cases — mandatory for auth flows

What it is: Lightweight, stateless shell commands that allow coding agents (like Claude Code) to interact with a browser — take screenshots, query element references, inspect network traffic — without loading the full DOM accessibility tree into the context window. Each command returns compact element references (e15, e21) rather than a full DOM dump, saving ~93% of tokens per interaction compared to Playwright MCP.

Why it matters: Playwright MCP is powerful but expensive: every interaction loads the complete accessibility tree. The CLI returns only what was asked for — a snapshot, a screenshot, a selector — keeping the agent's context window free for actual test code.

Core commands

Typical agent workflow

Session scoping — avoid state leakage

# Use the -s flag to scope CLI sessions — prevents state leaking between explorations
playwright-cli -s=tea-explore open https://example.com/login
playwright-cli -s=tea-explore snapshot
playwright-cli -s=tea-explore screenshot

# Trace analysis — useful for debugging failing tests without a GUI
npx playwright trace open trace.zip
npx playwright trace actions --grep="expect" trace.zip   # filter by keyword
npx playwright trace snapshot 15 trace.zip              # DOM state at action #15

What it is: A smarter version of Playwright's --only-changed flag. Analyzes git changes, filters out irrelevant file types via skipBurnInPatterns (config files, types, docs), controls test volume with burnInTestPercentage, and supports CI sharding. Prevents the problem where a change to a shared config file would otherwise trigger hundreds of unrelated tests.

Why it matters: Playwright's native --only-changed uses the import graph to determine affected tests. A change to playwright.config.ts or a shared type file marks every test as affected — triggering the entire suite. Burn-in adds an explicit skip-pattern filter and volume cap to keep feedback fast.

Three-phase filtering pipeline

Configuration

// playwright.config.ts — burnIn section
export default defineConfig({
  burnIn: {
    skipBurnInPatterns: [
      '**/playwright.config.*',   // config changes don't affect test logic
      '**/*types*',               // type-only files: no runtime impact
      '**/*.md',                  // docs: never trigger tests
      '**/mocks/**',              // mock data: not production code
    ],
    burnInTestPercentage: process.env.CI ? 0.2 : 0.3,  // 20% CI, 30% local
    repeatEach: 3,    // run each selected test 3× for stress testing
    retries:    1,    // 1 retry to distinguish flaky from broken
  }
});

Example output — 21 changed files

What it is: Patterns for testing webhook delivery in event-driven architectures. Webhook delivery is eventually consistent — the application fires HTTP callbacks asynchronously after events. Tests must poll until the webhook arrives or time out gracefully. The module provides deterministic polling via recurse, typed matchers, rich timeout diagnostics, and parallel-safe isolation via startedAt scoping.

Why it fails without these patterns: Webhook tests fail for 4 structural reasons — eventual inconsistency (can't await directly), parallel journal pollution (worker A captures worker B's webhook), opaque timeouts (no context about what arrived vs what was expected), and cleanup leakage (registered expectations persist between tests).

Core polling pattern

import { webhookRegistry } from '../support/merged-fixtures';

test('order.created webhook delivered after checkout', async ({ page, webhookRegistry }) => {
  // Scope to this test's start time — ignores webhooks from prior tests
  const registry = webhookRegistry.forTest();

  // Trigger the event
  await page.goto('/checkout');
  await page.getByRole('button', { name: 'Place Order' }).click();

  // Poll until the webhook arrives — or throw WebhookTimeoutError
  const webhook = await registry.waitFor(
    orderCreatedTemplate(orderId)    // scoped by orderId for parallel safety
  );

  expect(webhook.body.event).toBe('order.created');
  expect(webhook.body.data.id).toBe(orderId);
});

Three matcher types — AND semantics

AND semantics: All matchers on a template must pass — a webhook matches only if every matcher in the template succeeds simultaneously.

startedAt isolation — prevents cross-test pollution

// Each registry instance records the timestamp of creation
// waitFor() only considers webhooks received AFTER that timestamp
const registry = webhookRegistry.forTest();  // stamps current time
// → webhooks from previous tests are automatically excluded

// Template factories MUST be scoped by entity ID for parallel test safety
const orderCreatedTemplate = (orderId: string) =>
  webhookTemplate('order.created')
    .matchField('data.id', orderId)        // scoped by ID — worker A won't match worker B's webhook
    .matchPartial({ event: 'order.created' })
    .build();

WebhookTimeoutError — rich diagnostics

// When waitFor() times out, the error carries full diagnostic context
try {
  await registry.waitFor(orderCreatedTemplate(orderId));
} catch (err) {
  if (err instanceof WebhookTimeoutError) {
    console.log(err.totalReceived);       // how many webhooks arrived at all
    console.log(err.receivedWebhooks);    // last ≤10 webhooks (inspect actual payloads)
    console.log(err.matcherDetails);     // which matchers were configured
    // totalReceived: 0 → webhook never delivered — check app event publishing
    // totalReceived > 0 but no match → matcher misconfigured — inspect receivedWebhooks[0]
  }
}

Drain pattern — sequential events

// When testing delete, first drain the create webhook to avoid ambiguity
// Otherwise: the cleanup matcher might match the create webhook, not the delete
await registry.waitFor(movieCreatedTemplate(movieId));  // drain create first
await deleteMovie(movieId);                               // then trigger delete
await registry.waitFor(movieDeletedTemplate(movieId));  // now safe to assert delete

What it is: Risk classification and complete test coverage requirements for systems that publish webhooks. Webhook integration boundaries are high-risk by default because external consumers depend on payload shape, delivery timing, and event ordering — all of which are invisible to unit and component tests.

Default risk score: P2 × I3 = Score 6 (High — must mitigate). Probability 2 (Possible: webhook delivery failure occurs occasionally) × Impact 3 (Severe: external consumers miss transactions, integrations break silently). This means webhook delivery must have dedicated integration test coverage unless explicitly downgraded with documented justification.

Why score 6 is the baseline

Complete test coverage requirements — Score 6

• Happy path: Action triggers event → webhook is delivered → payload matches expected shape. Must cover every event type the system publishes.
• Sequential drain pattern: For systems that emit create then delete events, consume create webhook before asserting delete webhook. Prevents matcher ambiguity.
• Parallel isolation: Multiple tests run simultaneously — each must use entity-ID-scoped templates. Worker A must never match Worker B's webhooks.
• Timeout error shape: When webhook is not delivered (e.g. event publishing is disabled), assert the correct WebhookTimeoutError is thrown with meaningful totalReceived count.
• Cleanup verification: After test teardown, confirm no unprocessed webhooks remain in the mock server journal that could pollute the next test.

Downgrading from Score 6 — requires justification

Timeout values — match your pipeline latency

What it is: Production-ready utilities from @seontechnologies/pactjs-utils that eliminate boilerplate in consumer-driven contract testing. Wraps @pact-foundation/pact with type-safe helpers: provider state creation, PactV4 JSON interaction builder, verifier config assembly, and request filter auth injection. Works for both HTTP and message (async/Kafka) contracts.

Exposed Utilities

Decision Tree — Which Flow to Use

Why Use It Over Raw Pact

• Raw Pact requires manual JsonMap casting — complex params (Date, null, nested) need manual serialization
• Eliminates 30+ line verifier config scattered with env var logic
• Prevents double-prefix token bug (Authorization: Bearer Bearer token)
• Automated CI version tagging from GitHub Actions / GitLab CI env vars

What it is: zodToPactMatchers(schema, example) converts a Zod schema directly into Pact V3 matchers so you never maintain two representations of the same response shape. The schema is the single source of truth; the example value (or .openapi({ example }) metadata) provides concrete data.

Zod → Pact Mapping

Example precedence

// Priority: arg > .openapi({ example }) > type default
const schema = z.object({
  id: z.number().int().openapi({ example: 42 }),
  name: z.string(),
});

// explicit example wins for name; openapi example wins for id
const matchers = zodToPactMatchers(schema, { name: 'Alice' });
// → { id: integer(42), name: string('Alice') }

Critical Rules

• Consumer-curated schema only — include only the fields the consumer actually reads. Never import the full provider schema.
• Do NOT wrap the result in like() — each field is already a matcher; wrapping adds a redundant outer layer.
• For array responses: the entire body becomes eachLike(zodToPactMatchers(itemSchema, example))
• Silent drift risk with hand-written matchers: schema changes must be applied in two places. zodToPactMatchers removes this entirely.

What it is: createProviderState, toJsonMap, setJsonContent, and setJsonBody build type-safe provider state tuples and reusable PactV4 JSON callbacks — eliminating manual JsonMap casting and repetitive inline builder lambdas.

Core Helpers

Usage Pattern

const state = createProviderState({
  name: 'a movie exists',
  params: { id: 42, createdAt: new Date('2024-01-01') },
});
// state = ['a movie exists', { id: 42, createdAt: '2024-01-01T00:00:00.000Z' }]

pact.addInteraction(
  new PactV4.HttpInteraction()
    .given(...state)           // spread the tuple
    .uponReceiving('GET /movie/42')
    .withRequest('GET', '/movie/42')
    .willRespondWith(200, setJsonBody({ body: matchers }))
);

Mandatory Vitest Config

• Exactly one pact.addInteraction() per it() block — FFI non-deterministically drops interactions when multiple are added
• One .pacttest.ts file per consumer+provider pair — prevents FFI collision across files for the same pair

What it is: buildVerifierOptions, buildMessageVerifierOptions, handlePactBrokerUrlAndSelectors, and getProviderVersionTags assemble complete provider verification config in a single call — handling local/remote flow, broker URL resolution, consumer version selector strategy, and CI-aware version tagging.

Env Vars Read Automatically

Consumer Version Selector Strategy

Mandatory Vitest Config

• pool: 'forks' + poolOptions.forks.singleFork: true — Pact Rust FFI holds process-wide state; parallel verification corrupts results
• Use buildMessageVerifierOptions for Kafka / async message provider verification (separate entry point)

What it is: createRequestFilter and noOpRequestFilter inject an authentication header into every request during provider verification. A pluggable token-generator pattern prevents the double-Bearer bug (Authorization: Bearer Bearer token) and keeps auth concern separate from verifier config.

API

// Synchronous generator — pre-resolve async tokens before calling this
const requestFilter = createRequestFilter({
  tokenGenerator: () => myPreFetchedToken,   // returns raw value, NO "Bearer " prefix
});

// Provider that needs no auth
const requestFilter = noOpRequestFilter;     // (req, res, next) => next()

Bearer Prefix Contract

Async Token Pattern

// tokenGenerator must be synchronous — pre-fetch async tokens first
async function setupVerifierOptions() {
  const token = await fetchServiceAccountToken();   // async, done before filter
  return buildVerifierOptions({
    ...config,
    requestFilter: createRequestFilter({ tokenGenerator: () => token }),
  });
}

• tokenGenerator must be synchronous — resolve async tokens before creating the filter, not inside it
• Use noOpRequestFilter when the provider endpoint is unauthenticated — avoids conditional logic in verifier setup
• Never prefix the token manually in tokenGenerator — the filter adds "Bearer " automatically, adding it yourself causes the double-prefix bug

What it is: SmartBear's MCP server that lets AI agents interact with PactFlow / Pact Broker inside the contract testing workflow. Provides 8 tools covering test generation, provider state fetching, quality review, and deployment safety checks — all accessible via Model Context Protocol.

8 Available Tools

Installation (Claude Code)

# PactFlow (cloud)
claude mcp add-json -s user smartbear \
  '{"command":"npx","args":["-y","@smartbear/pact-mcp"]}'

# Required env vars (PactFlow)
PACT_BROKER_BASE_URL=https://<org>.pactflow.io
PACT_BROKER_TOKEN=<your-pactflow-token>

# Self-hosted broker (instead of token)
PACT_BROKER_USERNAME=<user>
PACT_BROKER_PASSWORD=<pass>

• Requires Node.js 20+
• Install per-project (-s user scopes to current user, not global) — different projects may use different PactFlow tenants
• Live broker queries mean the agent sees real provider states, not stale documentation

What it is: Opinionated scaffolding and battle-tested conventions for a Pact.js consumer CDC project — directory structure, Vitest config, package.json scripts, shell scripts, and CI workflow layout. Codifies the pactjs-utils reference implementation so new projects are CI-portable from day one.

Directory Structure

tests/
  contract/
    support/            ← shared helpers, NOT consumer tests
      pact-setup.ts
      test-context.ts
    movie-service/
      get-movie.pacttest.ts      ← one file per consumer+provider pair
      create-movie.pacttest.ts
pacts/                  ← generated pact files (gitignored in CI)
vitest.config.pact.ts   ← separate config, never share with unit tests

Required Vitest Config (vitest.config.pact.ts)

export default defineConfig({
  test: {
    include: ['tests/contract/**/*.pacttest.ts'],
    fileParallelism: false,   // ← mandatory
    pool: 'forks',            // ← mandatory
    poolOptions: { forks: { singleFork: true } },  // ← mandatory
  },
});

NPM Scripts

• Use .pacttest.ts extension — never .pact.spec.ts or .contract.ts; keeps pact tests excluded from unit test runs by default
• One .pacttest.ts per consumer+provider pair — splitting interactions for the same pair across files causes FFI collision
• Normalize interaction order before publishing: run jq -S 'sort_by(.)' on generated pact JSON for byte-stable broker publishes (avoids "Cannot change pact content for already published pact" errors)
• Support files live in tests/contract/support/ — never mix them with consumer test files

What it is: Configuring PactFlow webhooks to automatically trigger provider verification in GitHub Actions when a consumer publishes a new pact. Covers webhook URL and body setup, PAT management, secret rotation runbook, and staleness monitoring to detect silent webhook failures before they block deployments.

Webhook Setup Steps

Webhook Event Choice

What it is: Inject the Pact mock server URL into the real consumer HTTP client via an optional baseUrl field on the API context type. This ensures contract tests exercise the actual consumer code — including retry logic, header assembly, timeout config, and error handling — rather than hand-crafted fetch() calls.

The Problem with Raw fetch()

Implementation

// 1. Add optional baseUrl to your API context type
interface ApiContext {
  apiKey: string;
  baseUrl?: string;           // ← one-line change
}

// 2. Use nullish coalescing in the client
const client = axios.create({
  baseURL: context.baseUrl ?? API_BASE_URL,  // Pact URL wins in tests
});

// 3. Shared test context helper
function createTestContext(mockServerUrl: string): ApiContext {
  return { apiKey: 'test-key', baseUrl: mockServerUrl };
}

// 4. In the pact test executeTest() callback
const api = createApiClient(createTestContext(mockServer.url));
const result = await api.getMovie(42);
expect(result.id).toBe(42);   // assert on parsed objects, not HTTP status

• Assert on return values and parsed objects — not raw HTTP status codes. The contract already covers status codes; the consumer assertion should cover business logic.
• When the real client reveals a contract mismatch → fix the contract. The mismatch is a real bug that raw-fetch tests were silently hiding.
• Migrate existing consumer tests from raw fetch() to the DI pattern incrementally — each migrated test strengthens the contract's real-world coverage.