Reading a result

Four states

Each gate ends in one of four states.

State	Meaning	Certification
ok	the gate ran and held	counts
fail	the gate ran and broke the ladder; the summary names the first failing step and its log	fails
skip	the gate should have run and could not: disabled by a flag, no runner, zero tests found	blocks
n/a	the extension does not ship that layer or tool, or a cell lacks a capability a role needs	does not count

The summary names the first failure

✗ m248  Magento 2.4.8-p4 / PHP 8.4  e2e[tests/playwright:default]: 9 passed, 27 failed

That cell passed install, cron, 93 integration tests and production mode. The line names the gate that broke the ladder, not everything that happened. The full picture is in the JSON:

furnace run --gate=all --json=/tmp/result.json --html=/tmp/result.html

The JSON carries every gate, every step, seconds, log paths, the install method and source, the extension's git state, the run's purpose, the matrix coverage and the list of gaps. The HTML report is a standalone certification matrix with no external assets, so it can be attached to a release.

"Passed", "certified" and "fresh install passed" are different claims

passed means nothing failed. certified means nothing failed, and every required gate actually ran, and the fleet covers every platform/PHP tuple the catalog requires. A gate that never ran has not passed, so a run reports its gaps:

Not certified. Nothing failed, but the run does not cover the claim:
  - m248: e2e gate did not run (disabled with --skip-e2e)
  - 13 of 22 required platform/PHP tuple(s) have no cell

Pass --allow-incomplete to accept them knowingly. The JSON and HTML carry certified, gaps and coverage, so an artifact cannot claim more than the run proved; the HTML banner reads CERTIFIED, INCOMPLETE or FAILED.

A fresh-install run records purpose: fresh-install. It demands the install and fresh-install gates per cell, nothing else, and its banner reads FRESH INSTALL PASSED. It is never called a certification.

A skip counts against certification; an n/a does not. A suite furnace cannot drive honestly (a shipped playwright.config or phpunit.xml) is recorded as unconfigurable, which sinks a certification but not a fresh install.

Coverage

furnace plan

22 required · 5 ready · 4 behind · 13 missing · 0 excluded

behind means a cell sits on the right release line but an older patch, 2.4.8-p4 against a catalog asking for p5. Such a cell is still tested, because matching on the release line is deliberate, but it does not satisfy the exact tuple and does not count as coverage. --require-ready is the CI form and exits non-zero on a gap.

Logs

Every gate streams to ~/.furnace/runs/<run>/<cell>/, readable during a run:

~/.furnace/runs/2026-10-28-162344-qoliber-62b46e71/
  run.json
  m248/install-composer-require.log
  m248/fresh-install-setup-install.log
  m248/integration.log
  m248/e2e-playwright-hyva.log

Each file records the command line, exit code and duration alongside the output. A failed step prints its log path inline, because a one-line summary like "deploy failed" is fine for watching a run and useless for debugging one. Secrets passed on a command line (a database password) are masked in the recorded command.

The e2e log's header records the target URL and admin path, worth checking when a browser result looks implausible, because a suite that ignores BASE_URL will happily test a completely different site.