How a Playwright suite should be structured

Applies to every Qoliber extension with browser tests. The shape below is what makes a suite run on a colleague's machine, in CI, and on all ten furnace cells — rather than only where its author happened to create some content by hand.

Protocol suites

Under the test protocol, specs live in dev/tests/specs/<role>/, the seeder is named by e2e.seed in furnace.json, and furnace generates the Playwright config itself; a shipped playwright.config.* is a conformance problem. The layering below is the same whichever form a suite takes. See furnace.json.

The problem this solves

A suite that navigates to /faq/shipping/how-long-does-delivery-take and asserts on a specific question passes on one laptop and fails everywhere else. m2-faqs' suite failed 27 of 36 specs identically on all ten cells while the extension was entirely healthy — the GraphQL API answered correctly with an empty array, and the test wanted a question nobody had created.

Restoring a database snapshot into the cells is not the fix. They span Magento 2.4.6 to 2.4.9 and MageOS 1.3 to 3.4; one schema does not load into another, and a snapshot carries store IDs that mean different stores on different instances — loading cleanly while meaning something else.

The four layers

1. Seeder — preconditions. tests/furnace/e2e-data.php, run by furnace inside the cell before the production gate. Creates what specs cannot: store views, and the baseline entities the frontend asserts on. PHP, because store views cannot be created over REST. See writing an e2e seeder.

2. Admin specs — the admin surface, owning their data. Each creates what it needs and removes it. They are the only specs that touch admin forms.

3. Frontend and API specs — assert on seeded data. Never on another spec's residue.

4. Round-trip specs — the seam. One or two that create through admin, assert on the storefront, and clean up. This is the coverage nothing else provides, and the bug class that actually ships.

Ordering

Playwright's dependencies exists for setup projects, not for chaining assertions.

projects: [
  { name: 'admin-setup', testMatch: /admin-auth\.setup\.ts/ },
  { name: 'admin',    dependencies: ['admin-setup'], use: { storageState: './auth/admin.json' } },
  { name: 'frontend', /* no dependency on `admin` */ },
  { name: 'hyva' },
  { name: 'api' },
]

frontend must not depend on admin. If it does, a broken admin form fails every frontend spec too, and a version-specific setup failure is indistinguishable from a version-specific frontend bug. That ambiguity is what makes a suite expensive to debug.

Portability rules

Read the target from the environment. Furnace sets these:

variable
BASE_URL	the store view under test, e.g. https://m248.test/default/
ADMIN_USER, ADMIN_PASS	an admin furnace creates per run
ADMIN_PATH	absolute admin URL — not a path, since baseURL carries the store code

Never pin baseURL to a literal, in any project. The same config serves every store view, so a literal silently tests the wrong storefront. Furnace refuses a config that does this.

use: { baseURL: process.env.BASE_URL || 'https://local.test' },   // every project

Do not hardcode store codes. ?___store=de requires a store view that exists. Either the seeder creates it, or the spec reads it from the environment.

Assume Luma sample data. joust-duffle-bag is entity_id=1 on every cell, and the category tree exists. Reference it; do not recreate it.

Verifying a suite is portable

The test is whether it passes on a cell it has never seen:

cd tests/playwright
BASE_URL=https://m248.test/default/ ADMIN_USER=furnace_e2e \
  ADMIN_PASS='Furnace-e2e-123!' ADMIN_PATH=https://m248.test/admin \
  NODE_TLS_REJECT_UNAUTHORIZED=0 CI=1 npx playwright test --retries=0

Then, for real:

furnace run --gate=all --cell=m248

Run it twice. A suite that passes once and fails on repeat is leaving data behind.