proef — documentation index

proef (Dutch: test, trial — and tasting) is a declarative, modular, multi-engine end-to-end test runner, mostly Rust. Tests are Gherkin .feature files in business prose; macro packs bind the prose to executable steps; a pluggable engine runs each step batch — embedded Hurl for API testing (the seam admits future engines; none are scheduled).

This folder is the project corpus: the product requirements, the decision log, the normative technical spec, the milestone plan, and the testing strategy. Written 2026-07-28 from a validated research round (a working spike ran 5/5 scenarios green under both a prototype native runner and stock hurl 8.0.1 on identical generated artifacts); implementation has since delivered milestones M0–M5 and everything after them — through v0.19.0 (the 0.18 survey waves, then the gates that could not see what they covered). Only M6 (a second engine) is unscheduled. The repo-root CLAUDE.md carries the live status. This corpus is also published as a website: https://emrecdr.github.io/proef/.

Reading order

Decision log (ADR index)

Naming & identifiers

Project/binary proef · crates proef-core, proef-engine-hurl, proef-cli, proef-fixture, proef-harness, proef-lsp · run records .proef-runs/ · persistent World .proef-state.json · config proef.toml. The crates.io names proef, proef-core, proef-engine-hurl, and proef-lsp are published and owned.

Provenance

Grounded in two verified sources: (1) hurl master (Orange-OpenSource) — source-level verification of every library seam used, with file:line citations in TECH-SPEC §5; (2) a working spike proving the front end and artifact contract end to end (the spike predates this repo). Ecosystem practices are drawn from cargo-nextest, cucumber-rs, sqlx, rustls, probe-rs, and current (2026) Rust guidance.

Installing proef

Pick whichever fits your machine:

# Homebrew (macOS or Linuxbrew, arm64 or x86_64):
brew install emrecdr/proef/proef

# Prebuilt binary via cargo-binstall (any Rust dev environment):
cargo binstall proef

# From source via crates.io:
cargo install proef --locked

Or grab a prebuilt archive from a GitHub Release — five targets ship per tag (macOS arm64/x86_64, Linux arm64/x86_64-gnu, Windows x86_64-msvc), each with SLSA build provenance (gh attestation verify <archive> --owner emrecdr) and a .sha256 sidecar (sha256sum -c proef-<tag>-<target>.tar.gz.sha256 from the download directory). The Windows zip bundles the libcurl/libxml2 DLLs the binary needs; Linux binaries expect the distro’s libcurl4 and libxml2 (present on virtually every system, or apt install libcurl4 libxml2).

Prebuilt binaries need no Rust toolchain and no build prerequisites. Building from source (the cargo install/cargo binstall-fallback path) needs the native libraries the embedded hurl engine links: apt install build-essential pkg-config libssl-dev libcurl4-openssl-dev libxml2-dev libclang-dev on Linux; the Xcode command-line tools suffice on macOS.

Completions and the man page

Shell completions and a man page travel in every release archive (completions/, proef.1) — or generate them from any installed binary:

proef completions zsh > "${fpath[1]}/_proef"   # also bash, fish, powershell, elvish
proef man > /usr/local/share/man/man1/proef.1

In CI

cargo binstall proef resolves the release archives, so any workflow with a Rust toolchain installs in seconds; without one, download an archive and its .sha256 from the Release directly. A complete GitHub Actions workflow — secrets, JUnit, sharding, the regression gate — is in CI.

First run

proef doctor verifies the environment (embedded engine, native libraries, project layout, runs-dir writability). Then Getting started takes you from proef init to a green run.

A note on the dev fixture

The tutorial’s zero-network first run uses the dev fixture API (cargo run -p xtask -- fixture), which lives in this repository and is not part of the installed binary — it needs a checkout and a Rust toolchain. With an installed binary alone, point [url] base at any API you own instead; the proef init scaffold passes as-is against the fixture, and its routes (/health, /search, /version) are one file away from targeting yours.

Getting started — your first suite in ten minutes

proef runs end-to-end API tests written as plain Gherkin prose. The prose stays readable by anyone; a YAML pack binds each sentence to real HTTP work. This walkthrough builds a two-file suite from nothing and runs it.

Install first (Installing), then verify:

$ proef doctor
engine `hurl`:
  [ok  ] embedded hurl            hurl 8.0.1 (exact pin, ADR-0003)
  ...

0. Or scaffold it: proef init

$ proef init
  created ./proef.toml
  created ./suite/case.feature
  created ./suite/packs/api.yaml
  created ./hurl/api.hurl
  created ./.gitignore
  created ./suite/packs/proef-pack.schema.json
  ok ./suite/packs/api.yaml (modeline added)

created 6 file(s), skipped 0
next: proef test --dry-run  (then point ${url:base} at your API — the scaffold's routes are placeholders)

proef init writes the three files this walkthrough builds by hand below — proef.toml, suite/case.feature, suite/packs/api.yaml — plus a one-entry hurl/api.hurl, the pack JSON Schema and a .gitignore. The .hurl file is there because a step has two body forms: the scaffold’s pack shows an inline hurl: block and a ref: naming that file’s # @proef entry, so the form an adopter with an existing hurl corpus wants is visible from the first command rather than only in the docs. It is a deliberately smaller suite than the one this tutorial builds: no Then the first hit is record "r-1" step, no firstHit expect: macro, no ${secret:apiToken}, and search targets a plain /search route instead of the tutorial’s /api/v1/admin/search/records — while adding one step the tutorial does not, And the corpus reports its version, whose macro is the scaffold’s ref: example. Pasting the tutorial’s Then line into the scaffold’s feature file fails with bind::unbound_step — read on to build the fuller suite by hand, or extend the scaffold yourself once you understand the pieces.

1. A suite is three files

suite/
  case.feature        # the prose — what the test says
  packs/
    api.yaml          # the vocabulary — what each sentence does
proef.toml            # the configuration — where URLs and variables live (§3.5)

Layout is convention, not configuration. proef test suite takes one path and discovers everything under it: every *.feature file is a test file (at any depth), and every *.yaml/*.yml file directly inside a directory named packs is a macro pack — the packs directory itself may sit at any depth. All packs merge into one vocabulary shared by all feature files — grow the suite by adding files; nothing needs registering.

2. Write the prose

suite/case.feature:

Feature: Directory search
  Scenario: A known record is found
    Given the service is healthy
    When the operator searches for "Acme"
    Then the first hit is record "r-1"

The feature is pure prose — no URLs, no environment data. The target host and any variables live in proef.toml (§3.5) and reach the packs as ${url:…} / ${vars:…}.

3. Bind the prose

suite/packs/api.yaml:

macros:
  health:
    match: the service is healthy
    steps:
      - hurl: |
          GET ${url:base}/health
          HTTP 200

  search:
    params: [term]
    match: the operator searches for {term}
    steps:
      - name: search records for ${term}
        hurl: |
          GET ${url:base}/api/v1/admin/search/records
          Authorization: Bearer ${secret:apiToken}
          [Query]
          q: ${term}
          HTTP 200
          [Captures]
          recordId: jsonpath "$[0].id"

  firstHit:
    params: [id]
    match: the first hit is record {id}
    expect:
      - hurl: |
          jsonpath "$[0].id" == "${id}"

Three macro shapes are on display: a fixed sentence (health), a parameterized one ({term} captures the quoted word — quotes are shed), and an assert-only expect: macro whose lines merge into the previous request’s asserts. The hurl: blocks are raw Hurl — validated with the real parser the moment the pack loads.

3.5 Where URLs and variables live (proef.toml)

Variables are declared in proef.toml — never in the .feature files — and referenced as ${url:…} / ${vars:…}. The pack’s ${url:base} above resolves from here; proef finds the nearest proef.toml, searching up from the working directory (like cargo/git):

# proef.toml
[run]
suite = "suite"                    # `proef test` needs no path argument
# fragments = "hurl"               # uncomment to `ref:` entries of real .hurl files (§3.6)

[url]
base = "${env:PROEF_BASE_URL:-http://127.0.0.1:8787}"   # → ${url:base} (env override wins)

[vars]
apiVersion = "v1"                  # → ${vars:apiVersion}

[env.staging.url]                  # per-environment overrides (mirror the base tables)
base = "https://staging.example.com"

[env.prod.url]
base = "https://api.example.com"
[env.prod.http]                    # an env may override a runner setting too
timeout-ms = 60000

A macro then reads GET ${url:base}/api/${vars:apiVersion}/… with nothing declared in the feature. Pick an environment at run time:

$ proef test                       # base [url]/[vars]; runs `[run] suite` — "suite/" in the file above (`tests/` is only the fallback when the key is unset)
$ proef test --env prod            # [env.prod.*] deep-merged over the base (or set PROEF_ENV=prod)

The rule is uniform: under [env.<name>], url.* / vars.* override variables and http.* / run.* override runner settings — anything unlisted inherits the base, so an environment names only what changes (the Cloudflare-Wrangler / Cargo-profile model). Secrets never live here — they stay in the encrypted store (${secret:…}, §5).

3.6 A step’s body has two forms

Everything above uses an inline hurl: block, which is complete and permanent. The other form is ref: <name>, which points at one entry of a real .hurl file marked # @proef <name>, with values supplied by a bind: table:

Uncomment fragments = "hurl" in §3.5’s proef.toml (a second [run] table would be a TOML error — one table, both keys), put the annotated file under hurl/, and point a step at its entry:

    steps:
      - ref: admin.search          # one entry of hurl/admin.hurl
        bind: { q: "${term}" }

That file stays valid hurl, so the same bytes run under stock hurl and under proef — useful when a corpus already exists and you would rather annotate it once than transcribe it. Choose by capability, not taste: inline splices text (so ${docstring} can carry a multi-line body, which no binding can express), while ref: buys a name, reuse, standalone runnability, and a static check that every variable is supplied. AUTHORING.md §“hurl: or ref:” has the full comparison.

4. Validate without a network

$ proef test suite --dry-run

This binds every sentence, resolves every ${…}, emits the artifacts, and parse-validates them — no request is sent. Typos in prose, packs, or hurl blocks all fail here, with the file, line, and a “did you mean” where one exists. Wire your editor too: proef schema --add-to suite/packs/api.yaml gives packs autocomplete via the JSON Schema.

5. Provide the secret and a target, then run

Point PROEF_BASE_URL at any HTTP API you can reach — or start proef’s own dev fixture in a second terminal: from a checkout, cargo run -p xtask -- fixture binds the default base port (8787), so no PROEF_BASE_URL is needed (it prints a PROEF_BASE_URL line to export only when it ends up somewhere other than 8787 — the port was busy, or you passed a different ... -- fixture <port>). It also prints the fixture’s own token line: export PROEF_SECRET_APITOKEN=fixture-token — the value the next step needs, because every /api/v1/ route rejects anything else with a 401. Then:

$ proef secret set apiToken        # paste fixture-token; or: export PROEF_SECRET_APITOKEN=fixture-token
$ proef test suite
running 1 scenario(s) with 8 job(s) — run 019f…

  Scenario: A known record is found (suite/case.feature)
    ✓ suite/case.feature:3 — the service is healthy (2ms)
    ✓ suite/case.feature:4 — the operator searches for "Acme" (5ms)
    ✓ suite/case.feature:5 — the first hit is record "r-1" (0ms)
    ✓ scenario A known record is found

summary: 1 passed · 0 failed · 0 skipped

Secret values never appear anywhere — not in artifacts, events, logs, or reports.

6. When it fails

A failing assert names the feature line, the artifact line, and hands you a reproduce command (absolute paths; captures ride along in a .vars file):

  ✗ suite/case.feature:5 — assert failure (artifact case--a-known-record-is-found.hurl:12)
  reproduce: hurl --test /abs/path/.proef-runs/<run-id>/artifacts/case--a-known-record-is-found.hurl --variables-file /abs/path/….vars

Because this suite binds ${secret:apiToken}, the replay also needs the secret — proef never writes its value anywhere, so add it yourself: --secret apiToken=fixture-token (the artifact’s # replay: header says so).

Every run leaves a record under .proef-runs/<run-id>/: events.jsonl (the machine-readable event stream), run.log (the console mirror), and artifacts/ — the exact .hurl files that were executed, byte for byte. proef explain summarizes the latest run from the record, proef diff compares two runs — surfacing what regressed, what got fixed, and which steps turned flaky or slower — and proef report writes a self-contained HTML page of a run (scenario tree, timings, and deep-links to the executed artifacts).

Where next

• AUTHORING.md — the full pack reference: composition with use:, retries, optional steps, guards, captures across scenarios, fakes.
• TROUBLESHOOTING.md — exit codes, the glyph legend, and the frequent failures; DIAGNOSTICS.md indexes every error code you might hit.
• proef flows suite lists every scenario with tags; --format json feeds the nextest harness (one IDE test per scenario).
• proef test suite --watch reruns on every file change.
• Tag a scenario @skip / @skip:reason to park it — still counted, its reason in every report — or @quarantine so a flaky one stops gating CI while you fix it (AUTHORING.md · Reserved tags).
• Exit codes are a contract: 0 pass · 1 test failure · 2 your input · 3 environment — safe to wire straight into CI, with --junit for reports.

Writing scenarios

For the person who writes tests, not the person who wires them up.

You describe what should happen, in sentences. Somebody on your team maintains the vocabulary — the list of sentences proef understands — in files called packs. You never have to open one. This page is your whole loop.

If you also maintain the vocabulary, you want AUTHORING.md instead; this page deliberately stops at the boundary.

1. A test is sentences in a file

A test lives in a .feature file and looks like this:

Feature: Directory search
  Scenario: A known record is found
    Given the service is healthy
    When the operator searches for "Acme"

• Feature — what area you are testing. One per file.
• Scenario — one situation worth checking. A file can hold many.
• Given / When / Then — the steps, in order. proef strips the opening word and matches only the rest of the sentence, so all of them behave identically; pick whichever reads best. Given for setup, When for the action, Then for the check. And and But work too, and read better than repeating yourself:

    When the operator searches for "Acme"
    Then the response status is 200
    And the value at "$.results" is a non-empty list

Each step is one sentence, and every sentence must be one the vocabulary knows. That is the only rule you have to satisfy.

2. See the sentences you may write

$ proef macros

builtin:core.yaml
  expectPresent                0×  the value at {path} is present  (builtin, unused here)
  expectStatus                 0×  the response status is {status}  (builtin, unused here)
  …
suite/packs/api.yaml
  health                       1×  the service is healthy
  search                       1×  the operator searches for {term}

10 macro(s) · 0 unused

The right-hand column is what you can say. The left is the internal name — you do not need it. The 1× counts how many steps across the suite use that sentence.

{term} is a blank you fill in. the operator searches for {term} means you write:

    When the operator searches for "Acme"

Quotes are the house style and make the value easy to see.

This command works even when your file has a mistake in it — which is exactly when you need it. If some step does not bind, you still get the list; only the 1× counts are withheld (they would be misleading), and they show as —.

Editor tip. If someone has set up proef lsp for your editor, these sentences appear as autocomplete while you type, and mistakes underline as you go. It is worth asking for — but nothing on this page needs it.

3. Check before you run

$ proef test --dry-run

This reads your file and tells you whether every sentence binds. It sends no requests and touches nothing, so it is always safe and takes well under a second. Run it constantly.

  ok suite/case.feature — 1 scenario(s), 2 step(s), 1 batch(es)

dry-run OK: 1 feature(s), 1 scenario(s), 2 step(s) …
next: proef test

When it says next: proef test, your sentences are good.

4. Run it

$ proef test

  Scenario: A known record is found (suite/case.feature)
    ✓ suite/case.feature:3 — the service is healthy (6ms)
    ✗ suite/case.feature:4 — the operator searches for "Acme" (0ms)

✓ passed · ✗ failed · ∅ skipped, because an earlier step in the same scenario failed.

A ✗ here is a real result — proef reached the system and the answer was wrong. That is the test doing its job. Take it to whoever owns the API, with the line it names.

5. The two mistakes you will actually make

Everything else is someone else’s problem. These two are yours.

“no macro matches …”

proef::bind::unbound_step

  × no macro matches `the operator serches for "Acme"` — did you mean
    `the operator searches for {term}`?
   ╭─[suite/case.feature:4:5]
 4 │     When the operator serches for "Acme"
   ·     ────────────────────────────────────
   ╰────

You wrote a sentence the vocabulary does not know — usually a typo, a plural, or a word order that drifted. proef points at the exact line and, when it can, guesses what you meant.

Your fix: say a sentence that exists. proef macros lists them.

The message also shows a macros: block. That is instructions for the person who maintains the vocabulary — hand it to them if the sentence you need genuinely does not exist yet. It is not something you need to write.

“url variable … is not set”

proef::resolve::missing_config_var

  × in macro `health`: url variable `bse` is not set — define `[url]` `bse`
    in proef.toml (or in the active `[env.<name>.url]`) — did you mean `base`?

A sentence you used needs an address that nobody has filled in. This one lives in proef.toml, which is a short settings file, not a pack:

[url]
base = "https://api.your-company.com"

Your fix: if you recognise the name, correct the typo it suggests. Otherwise this is a setup question for whoever configured the project.

One more you may meet once

If a brand-new project fails on its very first run and says it is still the proef init scaffold, nothing is broken — the starter files point at a placeholder address and placeholder routes on purpose. Somebody needs to point [url] base at the real API and replace the example routes. After that, the loop above is all yours.

6. The loop

read the sentences   →   proef macros
write a scenario     →   your .feature file
check it binds       →   proef test --dry-run
fix what did not     →   the two errors above
run it               →   proef test

That is the whole job. When you need something the vocabulary cannot say, that is a conversation with whoever maintains the packs — and AUTHORING.md is the page for them.

Where to go next

Authoring reference — packs and features from the author’s seat

Everything here is validated at load or at --dry-run time; nothing fails only at execution that could have failed earlier. Start with GETTING-STARTED if this is your first suite.

Feature files

Standard Gherkin: Feature:, Scenario:, Background: (prepended to every scenario), Rule:, Scenario Outline: + Examples: (expanded, #N-deduped — note the #N is positional: inserting an Examples row above renames every instance below it, which re-keys their JUnit history and re-buckets them across --shard. A column placeholder in the outline’s name (Scenario Outline: search finds <q>) keeps each instance’s identity tied to its data instead of its row number), data tables (rows become step arguments), and docstrings (delivered to the macro as the docstring param). Keywords (Given/When/Then/And) don’t affect binding — only the sentence text does. Prose between the Feature: line and the first scenario is the feature’s description: never executed, but proef flows prints it (JSON: featureDescription) so a file’s intent travels with its inventory.

An outline’s <column> placeholders substitute into the docstring too, not just the scenario name, step text and table cells. That is how a request body gets data-driven without leaving the feature file:

  Scenario Outline: Posting <label>
    When a record is posted
      """
      {"label": "<label>", "priority": "<priority>"}
      """
    Then the response status is 201

    Examples:
      | label | priority |
      | alpha | high     |
      | beta  | low      |

A <name> that is not an Examples column is a parse-time error wherever it appears, docstrings included.

Variables are declared in proef.toml ([url] / [vars]), never in the feature file, and referenced from packs as ${url:key} / ${vars:key} — see CONFIG.md. Feature files stay free of URLs and environment data.

Tags (@smoke) accumulate feature→scenario; proef test --tags <expr> selects scenarios by a boolean expression over them — and, or, not, and parentheses, with the @ optional (e.g. --tags "@api and not @slow" or --tags "(smoke or nightly) and not wip"). A bare tag is a valid expression; a selection matching nothing is an error, not a silent green run. Atoms may glob: * matches any run of characters, ? exactly one — anchored to the whole tag and case-sensitive — so --tags "JIRA-*" selects every ticket-tagged scenario, while a metachar-free atom stays plain equality.

Reserved tags

Two tag names carry behavior; every other tag is yours (selection via --tags, grouping, traceability):

• @quarantine — the scenario runs and reports, but a test-failure does not gate the exit code, reaches JUnit as <skipped message="quarantined failure (non-gating): …">, and maps to # TODO in TAP. For flaky tests while they are being fixed — a User/System fault still fails the run.

  Quarantine is a holding pen, not a destination, and proef flaky is what keeps it one: over the retained records it separates a quarantined scenario that fails every run (DISABLED — switched off, and nobody is watching those failures because by design nothing reports them) from one that has been green throughout (recovered — the tag outlived the problem and is now suppressing the next real regression). Neither is visible any other way.

• @skip / @skip:<reason-token> — the scenario is parked: never prepared or run, counted as skipped, and the reason (the tag spelling itself) appears in the console, JUnit, TAP, the record, the report, explain and flows. --tags "not @skip*" unselects both spellings when you want them gone from the totals too. All-skipped exits 0; --dry-run still validates a skipped scenario (skip is not a validation waiver). In [run] setup/teardown features, reserved tags have no effect.

A tag that is almost reserved — @quarantined, @skipped, @Skip — is an ordinary tag with no effect, and since 0.18 it warns (tags::reserved_tag_typo) with the spelling it likely meant, because a scenario its author believed quarantined would otherwise gate the build in silence.

Conditional, data-dependent skipping is a step concern and stays in packs: when: guards a step, optional: soft-fails one, retry: bounds one. That split is deliberate and permanent — scenario prose stays declarative; there is no scenario-level IF/WHILE/TRY and none is planned (ADR-0019).

Macros (macros: in a pack)

macros:
  name:
    match: the record {name} is resolved   # sentence pattern (optional)
    params: [name]                         # declared parameters
    defaults: { index: records }           # defaults for optional params
    description: One line for humans.
    tags: [Admin]
    steps: [...]                           # OR expect: [...] — never both

• match: binds prose. Patterns need at least one literal word (no capture-only patterns), captures are {name}, quoted arguments shed their quotes, and matching is leftmost with ambiguity rejected — two macros that could claim the same sentence fail pack load.
• Every {capture} must be a declared param; defaults: keys must be declared params; adjacent captures ({a} {b} with nothing between) are rejected.
• A macro without match: is composition-only (reachable via use:).
• proef macros lists every macro with its match: prose — the sentence a feature file may say — and its call count across the corpus, flagging pattern macros no scenario binds (dead prose bindings); use:-only helpers and unused builtins are listed but not flagged. It also flags near-duplicate pattern macros — two that differ only in their {capture} names (the same literal skeleton), which are confusable to authors. Both are advisory only: they never change the exit code, and --format json carries pattern, unused and nearDuplicateOf fields for a CI hygiene gate.
• When a step does not bind, macros still lists the vocabulary (that is when you most need it) and keeps exit 2. Counts are withheld, not zeroed: calls/unused render as —/null, because an unbound feature contributes no calls and would make its own macros look dead. proef flows still refuses — it promises every scenario, and a silently partial list is a wrong answer.

Steps

steps:
  - name: human label (${…} resolves here too)
    optional: true                    # failure warns instead of failing
    retry: { count: 10, interval_ms: 300 }   # finite; 1..=10000
    delay: 250                        # ms before the request (capped at 1 hour)
    when: "${env:RUN_SLOW:-}"         # skips when empty or false/0 after resolution
    saveAs: { recordId: global }      # promote a capture to the global store
    hurl: |                           # the payload — raw hurl, one or more entries
      GET ${url:base}/api/v1/records/{{recordId}}
      HTTP 200
  - use: otherMacro                   # composition (cycle-checked, depth ≤ 32)
    with: { term: "${name}" }

retry:/delay: are baked into the entry’s [Options] so the emitted artifact replays with identical semantics under stock hurl. optional: steps run as their own batch so a failure cannot poison neighbours. Raw [Options] written inside a hurl: block are linted by the same rules — retry:/repeat: finite and at most 10 000, delay:/retry-interval:/max-time: at most one hour, and a key declared both in YAML and in the block is refused (pack::option_declared_twice) — and whatever the values, a batch’s watchdog budget never exceeds four hours (ADR-0007).

expect: macros carry no requests: status: 200 and/or raw hurl: assert lines merge into the previous request entry (a Then before any When is an error).

hurl: or ref: — two body forms, chosen by capability

A step’s body is an inline hurl: block or a ref: naming an entry in a real .hurl file (ADR-0018). They are not two spellings of one thing, so the choice is not a matter of taste:

Reach for inline for a request only this macro makes, and always when you need to splice something hurl cannot template — ${docstring} as a request body is the clearest case, since a bound value is a single-line scalar.

Reach for ref: when the same request serves several macros, when the hurl was written by somebody else, or when the file must stay runnable on its own.

bind:                              # pack scope — every macro in this file
  base:     ${url:base}
  apiToken: ${secret:apiToken}     # injected at run time, never into an artifact
macros:
  search:
    params: [q]
    match: "the operator searches for {q}"
    bind: { q: "${q}" }            # macro scope
    steps:
      - ref: admin.search
        bind: { index: records }   # step scope — the most specific wins

hurl: and ref: are chosen per step, not per suite. A step is one or the other, but a macro mixes them freely — which is what adopting an existing corpus looks like in practice: ref: the requests the corpus already has, write inline for the ones it doesn’t.

  archiveFirstResult:
    match: the operator archives the first result
    steps:
      - ref: admin.search        # the corpus already has this request
      - hurl: |                  # this one is new, and splices ${…}
          POST ${url:base}/api/v1/admin/records/{{recordId}}/archive
          HTTP 204

recordId is captured by the fragment and read by the inline step: the World threads captures across both forms, and contiguous same-engine steps batch together whichever form they were written in. Pick per step by capability — inline splices ${…} anywhere (including a multi-line ${docstring} body, which a single-line bound scalar cannot express); ref: keeps the file runnable on its own and checks its interface by name.

proef fragments lists the corpus: which entries exist, how many scenarios run each, and — the two questions a listing exists for — which are annotated but reached by nothing, and which carry no # @proef at all and so cannot be referenced. --check exits 1 on the first; add --require-annotated to include the second, which is opt-in because an unannotated entry is inert by design (pointing at a corpus you did not write costs nothing), and only a team mid-port means “not done yet” by it.

Set [run] fragments to the directory holding those files (see CONFIG.md). Every {{variable}} a fragment reads must be bound in one of the three scopes, captured by an earlier step, or supplied by the fragment itself; nothing is implicit, because hurl’s per-entry variable: assigns into one shared set rather than scoping, so an unbound name would quietly inherit an earlier entry’s value.

A fragment supplies its own value with an ordinary [Options] variable: line — which is how a corpus file stays runnable on its own, with fewer variables to pass in:

# @proef admin.search
GET {{base}}/api/v1/admin/search/{{index}}
[Options]
variable: index=records      # the file answers its own question
HTTP 200

Do not then also bind: that name. Both spellings reach the entry as variable: index=, hurl takes the last, and the fragment’s own line is last — so the bound value would never reach the request. Proef refuses the pair (pack::option_declared_twice) rather than picking one silently; delete whichever is not authoritative.

Bindings resolve once per scope instantiation — pack scope once per scenario, macro scope once per invocation, step scope per step — so one bind: entry is one value, and two are two. That is what makes a pack-scope ${fake:email} a single identity for the whole scenario.

A binding nothing can read is refused rather than dropped, at two levels:

• The table — a bind: with no ref: in scope to read it (proef::pack::bind_without_ref), checked at all three scopes.
• One key — a bind: entry no fragment in that scope reads (proef::pack::unread_bind_key), with did-you-mean over the names that are read. This is the one a typo produces: bind: { token: …, toekn: … } binds one real key and one that never arrives.

The key check is a union over the scope, never against a single fragment — a pack-scope table is the plumbing every macro in the file needs, so a key serving one macro and not its siblings is correct usage.

Note the one that surprises people: a macro-scope bind: does not reach a use: target — the target resolves its own pack and macro scopes — so the table belongs on the macro that actually carries the ref:.

You do not have to memorise a corpus you did not write: with proef lsp running, completing inside a bind: table offers the {{variables}} the fragments this pack ref:s actually read, each labelled with the fragment that wants it. The names are read off the .hurl file itself, so they cannot drift from it. If a name still goes unsupplied, proef::lower::unbound_placeholder names it at lower time — --dry-run is enough to surface that, no server needed.

The same check reads inside your bind values — with hurl’s own parser, so what hurl calls a function is never mistaken for a variable ({{newUuid}} and {{newDate}} need no supplier). A {{name}} that is a variable is templated when the entry runs, so it must be supplied by then: an earlier step’s capture (the usual shape: bind: { path: "records/{{recordId}}" }), the fragment’s own [Options] variable: (its lines evaluate before the injected ones), a secret in scope, or a sibling literal bind whose name sorts before this one — the injected lines are written and evaluated in name order, so base can feed q but not the other way around. And the reverse collision warns rather than fails: a literal bind: that re-uses a name an earlier step captured wins silently from that entry on (hurl’s variable: assigns into one shared set), so proef::lower::bind_shadows_capture names it — rename the binding if the capture was the point.

Where a file,…; asset lives

A file body — a request body, a multipart part, or a file,…; an assert compares against — sits beside the source that names it:

This is hurl’s own rule (its --file-root defaults to the .hurl file’s directory), and the one Karate, pytest and Jest use for fixtures. You never set a root: proef stages each asset from its own source into the run’s artifacts/assets/<slug>/, and the emitted artifact carries the matching --file-root in its replay line, so the hand-off runs unchanged under stock hurl.

Two consequences worth knowing. The path must be a plain relative one — no leading /, no .. — because it names a file inside your suite, not a location on the machine. And a missing file is refused at --dry-run, naming the directory it was looked for in (proef::run::asset_unstageable) — so the gate CI runs before standing an environment up answers it, rather than a failing request minutes later, or hurl reporting an unreadable body against the artifact. Staging resolves beside the file the parser actually read, so the directory you run from does not matter; a symlink already at a staging destination is replaced, never written through; and two references that name one file on a case-insensitive filesystem (Data.json and data.json on macOS or Windows) are refused (proef::run::asset_unstageable) rather than silently last-writer-won.

Recipes — the three shapes every real suite needs

Log in, then use the token

The most common real-world flow: the API mints a token at a login endpoint, and every later request carries it. A capture crosses steps through the World, so the shape is one login macro capturing the token and any number of authed macros reading it:

macros:
  logIn:
    match: the operator logs in
    steps:
      - hurl: |
          POST ${url:base}/auth/login
          {"user": "${vars:user}", "password": "${secret:password}"}
          HTTP 200
          [Captures]
          token: jsonpath "$.token"

  listRecords:
    match: the records are listed
    steps:
      - hurl: |
          GET ${url:base}/records
          Authorization: Bearer {{token}}
          HTTP 200

{{token}} is hurl run-time vocabulary (the second tier): the capture is assigned when the login entry runs, and every later entry in the scenario reads it — no bind:, no config. To reuse one login across scenarios, add saveAs: { token: global } to the login step and read ${global:token}; note the promotion is refused if the value carries a secret (ADR-0005).

Wait for an eventually-consistent result

A POST answers 202 and the resource appears a moment later. Don’t sleep — put a finite retry: on the step that polls, and let its asserts be the condition:

  awaitRecord:
    params: [id]
    match: record {id} is eventually visible
    steps:
      - retry: { count: 10, interval_ms: 300 }
        hurl: |
          GET ${url:base}/records/${id}
          HTTP 200

The retry re-runs the entry until its asserts pass or the count runs out — and the count must be finite: hurl cannot be interrupted mid-call, so an unbounded retry is a hang the watchdog would have to abandon (the same reason retry: -1 is refused at load). The retry also bakes into the emitted artifact’s [Options], so a replay under stock hurl polls the same way.

Seed data before, clean up after

Three scopes, three mechanisms — pick by lifetime:

• Per scenario: a Background: in the feature file runs its steps before every scenario in that file — provisioning prose, bound to macros like any other step.
• Per suite: [run] setup / [run] teardown in proef.toml name feature files that run once around the whole pool — seed a database, then delete the run’s residue. Teardown runs even when the suite fails or is interrupted; a teardown failure is exit 3, never silent. The keys are documented in Configuration.
• Across scenarios: a saveAs: { id: global } capture in setup (or any scenario) persists into later scenarios and runs as ${global:id} — the handle teardown needs to delete what setup created.

Asserting responses — the hurl vocabulary

Assertions live inside a step’s raw hurl: block (or an expect: macro), so the whole hurl 8.0 assert grammar is available untouched: proef resolves ${…} before the run and hands the rest to the embedded engine verbatim (ADR-0005). The block is parsed at pack-load time, so a grammar slip fails fast with a diagnostic — only JSONPath semantics (a query that lexes but never matches at run time) can slip through. An assert reads <query> [filters…] <predicate>; HTTP <status> is the implicit status assert. The authoritative list is hurl’s own asserting-response and filters docs; the common shape:

GET ${url:record}
Authorization: Bearer ${secret:apiToken}
HTTP 200
[Asserts]
jsonpath "$.id"        isUuid                 # type/shape checks — schema-lite
jsonpath "$.status"    == "active"
jsonpath "$.createdAt" isIsoDate
jsonpath "$.tags"      count == 3
header "Content-Type"  contains "application/json"
duration               < 1000                 # response-time budget (ms)

• Queries (what to read): status, header "<n>", cookie "<n>", body, bytes, jsonpath "<expr>", xpath "<expr>", regex "<pat>", sha256, md5, url, redirects, variable "<n>", duration, certificate "<f>".
• Predicates (the check): == != > >= < <=, startsWith, endsWith, contains, includes, matches "<regex>", exists / not exists, isEmpty, and the type family isBoolean isInteger isFloat isNumber isString isCollection isList isObject isDate isIsoDate isUuid isIpv4 isIpv6.
• Filters transform the value before the predicate, chained left→right: count, nth <n>, first, last, split "<sep>", replace/replaceRegex, toInt/toFloat/toString, toDate "<fmt>", format "<fmt>", base64Decode/base64Encode, urlDecode/urlEncode, daysAfterNow/ daysBeforeNow, jsonpath "<expr>", regex "<pat>", utf8Decode.

[Asserts]
jsonpath "$.items"     count > 0
header "Set-Cookie"    split ";" nth 0 startsWith "session="

RFC 9535 JSONPath (hurl 8.0): filter expressions and functions are in scope — jsonpath "$.books[?(@.price < 10)]", length(), count(), match(), search(). Use the bracket form for names with hyphens ($['x-custom-id']), and assert a missing path with not exists (a non-matching path yields no value, not count == 0). The same query+filter grammar drives [Captures], threading a value into later steps as {{name}}.

Built-in shape macros

For the most common single-value shape checks, the built-in Core pack ships a small, product-neutral set of expect: macros so you rarely hand-write the predicate. Each reads a JSONPath ({path}) into the previous response and merges one assert:

When the record is fetched
Then the value at "$.id" is a uuid
And  the value at "$.name" is a string
And  the value at "$.tags" is a non-empty list

Available: the value at {path} is a string / … a number / … a boolean / … a uuid / … an ISO date / … present / … a non-empty list. Quote the path in prose when it contains spaces (the quotes are optional and shed). These are a convenience layer over the predicates above — they deliberately do not cover whole-body structural matching; reach for a raw expect: hurl: block for anything they omit.

Negative cases — one macro per malformation, one shared expectation

A validation suite is naturally combinatorial, and its cases differ structurally rather than by value: one omits a key, one empties it, one adds a key the caller may not set. A single parameterised macro cannot express that — the bodies are different shapes, not one shape with a hole in it. Outline placeholders do substitute into docstrings, but an Examples cell cannot practically hold JSON, and a raw body in the feature file defeats the prose the design exists to protect.

Name each malformation. One macro per case, each sentence saying what is wrong in business terms:

macros:
  createWithEmptyTitle:
    match: creating a task with an empty title is refused
    steps:
      - hurl: |
          POST ${url:base}/tasks
          Content-Type: application/json
          {"title": "", "priority": "high"}

  createWithServerOwnedField:
    match: creating a task that sets its own id is refused
    steps:
      - hurl: |
          POST ${url:base}/tasks
          Content-Type: application/json
          {"title": "ok", "id": "caller-chosen"}

Then let one expect: macro serve the whole catalogue. Because an expect: merges its asserts into the previous request entry, a single parameterised expectation covers every case in the set — typically the largest de-duplicator in a validation pack:

  expectErrorCode:
    params: [code]
    match: "the error code is {code}"
    expect:
      - hurl: |
          jsonpath "$.error.code" == "${code}"

The scenarios then read as the specification they are, and each one’s asserts land on its own request:

  Scenario: An empty title is rejected
    When creating a task with an empty title is refused
    Then the response status is 422
    And the error code is TITLE_REQUIRED

# emitted
POST http://127.0.0.1:8787/tasks
Content-Type: application/json
{"title": "", "priority": "high"}
HTTP *
[Asserts]
status == 422
jsonpath "$.error.code" == "TITLE_REQUIRED"

The trade, stated plainly: the pack grows with the malformation catalogue — one macro per case, where a value-driven test would have used one row. That is the cost of feature files that read as prose, and it buys a suite a non-engineer can review. The expectation side does not grow with it.

Variables — the two tiers

${…} is recursive (captured arguments may themselves contain ${…}, depth ≤ 8) and $${ escapes a literal ${. ${secret:NAME} never inlines the value — it lowers to {{NAME}} and the engine injects it through hurl’s redaction, so artifacts carry placeholders only.

Config variables — ${url:key} and ${vars:key} come from proef.toml’s [url] / [vars] tables, deep-merged with the active --env profile. This is how you keep URLs and settings out of the feature files entirely; a referenced-but-undefined one is a lower-time error. See CONFIG.md (ADR-0012).

Captures ([Captures] in a hurl block) flow forward within the scenario as {{name}}. saveAs: { name: global } additionally promotes the captured value into the persistent global store (.proef-state.json), where later scenarios and later runs read it at lowering time as ${global:name}.

Fakes are deterministic synthetic data: firstName, lastName, name, fullName, email, username, phoneNL, postCode, city, street, int, number, digits4, digits8, bool, word, uuid (unknown generators fail at load). Each ${fake:kind} reference gets its own value — an occurrence counter advances every time a scenario resolves one, so independent ${fake:email} references in the same scenario never collide, however many a step ends up resolving. A step’s name: label is the one deliberate exception: it is not independent of its own payload, so it replays from the start of the step’s own occurrence window instead of minting new ones — the label’s Nth ${fake:…} reference reuses whichever occurrence the payload’s (and when:’s) Nth reference consumed, matched by position, not by generator kind. When the label’s ${fake:…} references mirror the payload’s in kind and order — the common case, e.g. a label that names the same field the payload sends — this reproduces the payload’s own value exactly. When they diverge in kind (say the label’s first reference is ${fake:fullName} but the payload’s first reference is ${fake:email}), the label instead shows whatever that occurrence generated for its own kind — a value the request did not send. That includes a label with more ${fake:…} references than its payload: each extra one still reserves its own place in the sequence, so a later step can never be handed a value the label already displayed. That whole sequence is a pure function of ${run:id}: the same --run-id reproduces the same fakes, byte for byte, across runs. The counter restarts at zero for every scenario — known limitation: two different scenarios that each resolve ${fake:email} at the same position in their own step order (typically each scenario’s first fake reference) get the same address, because both count from zero independently. If two scenarios must not collide, key the value yourself (fold in ${run:id} or a captured id) rather than relying on ${fake:*} alone.

Secrets

Reference with ${secret:NAME}. Values come from PROEF_SECRET_<NAME> environment variables first, then the encrypted store (proef secret set NAME, or … | proef secret set NAME --stdin for scripts (never argv — ps shows it); proef secret list names, proef secret rm NAME removes — XChaCha20-Poly1305, key auto-created 0600 under ~/.config/proef/). Values never appear in artifacts, events, logs, or reports; events carry capture names only. The store file (.proef-secrets.json, mode 0600) holds ciphertext only and is gitignored by default; the key (~/.config/proef/keys/default.key) must never leave your machine.

Artifacts — the executed input

Every scenario emits <feature-path>--<scenario>.hurl — the feature’s suite-relative path with its extension dropped, then the scenario, both slugified (so two same-named features in different directories cannot collide) and capped at 120 bytes with a hash tail — the exact bytes the engine executes — plus .map.json (artifact lines ↔ feature lines, batch and step indices) and .vars (referenced globals as values, secrets as names). The header’s # replay: line is a complete stock-hurl command, including --secret NAME=<value> placeholders for you to fill. proef artifacts <dir> -o out/ --run-id ci emits the same set deterministically for hand-off.

Runs, records, CI

• Exit codes: 0 pass · 1 test failure (including cancelled runs) · 2 input error · 3 system error.
• .proef-runs/<run-id>/events.jsonl is the record — one JSON event per line (scenario/step lifecycle, per-attempt progress, failure detail); proef explain summarizes it; --junit and the GitHub Actions summary derive from the same run.
• --format json prints one machine-readable summary object on stdout (the human report moves to stderr); --jobs N controls parallelism; proef.toml holds project defaults ([run] jobs, [http] timeout-ms).
• proef fmt suite --check keeps hurl blocks canonically formatted; proef flows lists scenarios.

IDE integration — one test per scenario

The harness crate bridges proef into any nextest/libtest UI (rust-analyzer, RustRover, cargo nextest run): it lists scenarios via proef flows --format json and runs each as its own test via proef test --scenario <name>.

PROEF_HARNESS_SUITE=suite cargo nextest run -p proef-harness

Set PROEF_BIN=/path/to/proef when the binary is not on PATH. With PROEF_HARNESS_SUITE unset the harness exposes nothing, so a plain cargo test stays green. Each scenario appears as a separate test in the IDE’s runner — click-to-run one scenario without touching the terminal.

Unset is not the same as unreadable. A variable set to bytes that are not valid UTF-8 means you asked for something the harness cannot read, so it exposes a single failing proef::config trial naming the variable — it will not fall back to proef on PATH, and it will not report green having listed no tests.

Secrets in CI

Two working setups:

• Values via env (simplest): set PROEF_SECRET_<NAME> from your CI’s secret storage — no store, no key, nothing on disk.
• Committed ciphertext store: commit .proef-secrets.json (it holds ciphertext only) and supply the project key as the PROEF_KEY CI secret (base64 < ~/.config/proef/keys/default.key). A set-but-invalid PROEF_KEY is always an error, never a silent fallthrough.

Note: .proef-state.json (the persistent World) is plaintext and therefore gitignored — captures promoted with saveAs: global land there. A capture whose value equals a known secret is refused with a warning; it never persists (proef doctor also reports store/key health).

Environment variables

Set-but-unreadable is an error, never silence. A variable whose value is not valid UTF-8 is something you asked for and proef cannot read, so it is reported rather than treated as unset: the commands above exit 2 naming the variable (proef doctor reports it as a failed check and exits 3, with its other environment findings), and the harness exposes a failing proef::config trial. Reading a malformed PROEF_KEY as absent used to mean decrypting with the wrong key and reporting tampering; a malformed PROEF_ENV meant running against the wrong environment. PROEF_CONFIG_DIR is a path and is read as raw bytes, so it has no such failure mode.

Suite-defined env vars (like PROEF_BASE_URL in the guides) are a convention of ${env:…} references in proef.toml, not built-ins — name yours freely.

Style

Keep prose at business level — no URLs, headers, or JSON in feature files; that’s what packs are for. Prefer several small macros composed with use: over one large one. Give steps name: labels — they anchor artifacts, events, and failure output.

Label a macro’s steps whenever it has more than one. A macro with three steps turns one feature sentence into three engine steps that share a step anchor exactly — same file, same line, same text — so on the console, in the HTML report, in JUnit, TAP and the job summary they arrive as the same sentence repeated, one row per step. The name: is the only thing that distinguishes them:

✓ tests/features/a.feature:9 — the workspace is provisioned › fixture warm-up probe (4ms)
✗ tests/features/a.feature:9 — the workspace is provisioned › provision the environment (7ms)

Without the labels those two lines differ only by their glyph.

proef in CI

Everything proef’s differentiators buy — typed exit codes, JUnit, sharding, the rerun overlay, the regression gate — pays off in CI, and each piece is documented on its own page. This page is the missing last mile: one paste-ready workflow, then the pieces it composes.

A complete GitHub Actions workflow

name: e2e
on: [push, pull_request]

jobs:
  e2e:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2, 3]
    steps:
      - uses: actions/checkout@v4
      - name: Install proef
        run: |
          curl -LsSf https://github.com/cargo-bins/cargo-binstall/releases/latest/download/cargo-binstall-x86_64-unknown-linux-musl.tgz | tar -xz -C /usr/local/bin
          cargo-binstall proef --no-confirm
      - name: Run the suite
        env:
          # Secrets reach proef only through the environment — never files,
          # never flags (values would land in the process listing).
          PROEF_SECRET_APITOKEN: ${{ secrets.API_TOKEN }}
        run: |
          proef test tests/features \
            --env staging \
            --shard ${{ matrix.shard }}/3 \
            --junit auto \
            --meta commit=${{ github.sha }}

The pieces:

• --junit auto writes report.junit.xml into the run dir — under GITHUB_ACTIONS only, so local runs stay clean. Failures also reach the job summary and the PR’s changed-files gutter as ::error annotations automatically; no extra step.
• --ctrf report.ctrf.json writes the same verdicts as CTRF JSON — the format that carries what JUnit XML cannot: retries and flakiness natively (a pass-after-retry lists its real failed attempts with messages), tags, and a file path per test. Point a CTRF consumer such as ctrf-io/github-test-reporter at it. The two files never disagree: a quarantined failure is skipped with a message in both (ADR-0019), because a dashboard reading “failed” beside exit 0 would contradict itself. Like --junit, the file is written even when [run] setup aborts the run — a job gating on it must never see it missing.
• --shard I/N partitions by a stable hash of (file, scenario): adding a scenario never re-buckets the others, so shard timings stay comparable across commits. Every matrix job runs the same expression and the shards partition exactly. To balance by time instead of by count, see below.
• --meta commit=… records provenance the run cannot harvest itself: proef never reads GITHUB_SHA or any CI variable (ADR-0020) — what the workflow hands over explicitly is what the record carries.
• PROEF_SECRET_<NAME> supplies ${secret:name} values. They never appear in artifacts, events, logs, or reports — including base64/hex/ percent-encoded reflections.

Balancing the matrix by duration

A hash split balances by count, and a matrix finishes when its slowest shard does — so one long scenario can leave three runners idle. Every run that reaches its suite writes a small timings.json into its run directory; archive it once and hand it to the next run’s matrix:

      - name: Run the suite
        run: |
          proef test tests/features \
            --shard ${{ matrix.shard }}/3 --shard-weights timings.json
      # one job publishes the file the next run's matrix reads
      - uses: actions/upload-artifact@v4
        if: matrix.shard == 1
        with:
          name: proef-timings
          path: .proef-runs/*/timings.json

Every job must read the same file. The tempting shortcut — letting each job weight by its own local run history — is silently wrong: matrix jobs run on different machines with different (usually empty) runs-dirs, so each would compute a different assignment and scenarios would run twice or not at all while the suite still reported green.

A scenario the file does not mention falls back to the frozen hash, so a test added after the timings were captured still runs exactly once. A missing or malformed file is exit 2 — never a silent fall back to the unbalanced split. Full detail in CONFIG.md.

Gating on regressions between runs

proef diff compares two run records; --fail-on-regression makes it a gate. Download the base branch’s record (uploaded as an artifact by its own run) and compare:

proef test tests/features --junit auto        # today's run
proef diff path/to/base-events.jsonl          # vs the downloaded baseline
# exit 1 on a regression; new flakiness and perf deltas print either way

Continuing a cancelled run

A run stopped by --max-fail, a runner timeout, or a docker stop records what it never reached. SIGTERM and SIGHUP take the same graceful path as Ctrl-C: in-flight batches finish, the rest record as skipped, [run] teardown runs, the reports are written, and the record closes with run_finished + cancelled (exit 1) — so a timed-out job still leaves a complete record and its JUnit. Only a second signal (exit 130) or a SIGKILL truncates it.

proef test --rerun re-runs the last run’s failures and the scenarios it never got to — and its JUnit and HTML report cover the whole suite via the rerun overlay (rerun_of in the record), so one report stands for the composed result, never a false green.

Flakiness over history

Keep a few records ([run] keep-runs in proef.toml sets the rotation) and proef flaky renders verdicts over them — flapping, passes-only-on-retry, always-failing — from the same records CI already produced.

It also audits @quarantine itself, which nothing else can: a quarantined scenario failing every run is DISABLED (switched off — its failures gate nothing, so no job ever reports them), and one green throughout is recovered (the tag can come off). --format json carries quarantined and the verdict key, so a scheduled job can gate on either without parsing the table.

--by <key> splits the same history per run context — --by env, or any [meta]/--meta key such as --by runner. A scenario that flaps in one environment and is solid in another is not flaky but context-dependent, and a pooled history cannot tell those apart: it reports the one conclusion the merged view can never reach, naming the scenarios whose verdict changes with where they ran. A run that never set the key is its own (unset) bucket rather than being folded in with the runs that did.

Verdicts are keyed by the run’s input fingerprint — inputs.json beside each record, a hash of the feature sources, the loaded macros and fragments, and the resolved ${url:…}/${vars:…} scope — so a pack or config edit starts a fresh window instead of mixing runs of different inputs; --by splits within it. Three guards, each a [flaky] key with a flag twin (CONFIG.md): a scenario seen in fewer than min-samples runs (default 10) reads insufficient-data rather than earning a verdict — a fresh matrix needs ten records before the table says anything, and --min-samples 2 restores the pre-0.18 floor; a flagged scenario stays flagged until recovery-runs (5) trailing clean runs, so it cannot flip between adjacent runs; and a run in which more than outage-rate (0.8) of the suite failed is an environment incident, excluded wholesale, so one staging outage cannot mark the suite broken.

Editor setup — proef lsp

proef lsp starts a Language Server Protocol server that speaks generic LSP over stdio. Point any LSP-capable editor at proef lsp and get, live as you type:

• Diagnostics — the same validation proef test --dry-run performs (unbound steps, unknown step kinds, malformed hurl blocks, unresolved ${…} references), published across the whole suite as you edit.
• Go-to-definition — jump from a Gherkin step to the macro that binds it, or from a pack’s ref: to the # @proef annotation in the .hurl file (ADR-0018).
• Completion — step completions offering the suite’s macro patterns; on a pack’s ref: line, the fragment names in scope; and inside a bind: table, the {{variables}} those fragments actually read, each labelled with the fragment that wants it. A variable a fragment supplies itself ([Options] variable:) is left out: it needs no bind:, and binding it would be refused as option_declared_twice.
• Find-references — every step across the suite that a given macro binds.
• Document symbols — a feature outlines to its scenarios (tagged ones showing their tags), a pack to its macros (detailed by the pattern each matches). Which vocabulary applies is decided by what discovery found in the file, not by its extension.
• Hover — the macro a step binds, or a use: targets, with its pack, pattern and params; on a ref:, the fragment’s file and the variables still needing a bind:. Every fact comes from the same analysis the diagnostics do, so a hover can never contradict the squiggle on the same line.
• Semantic tokens — the two variable tiers, told apart on screen. Inside a pack’s hurl: | block, ${…} (resolved at lower time, by proef, before any request exists) and {{…}} (resolved at run time, by hurl) look identical to every editor: a YAML highlighter sees a string, and a hurl highlighter never runs because the block is not a file. proef is the only party that knows which is which. ${…} is reported as a macro — a substitution performed before execution, which is what a macro is — and {{…}} as a variable; every mainstream theme colours those differently already, so nothing needs configuring. A $${ escape stays unhighlighted, because it is a literal ${ proef will not substitute.
• Quick fixes — a misspelled name that already earned a “did you mean” becomes an applicable edit: use: and ref: targets, with: and bind: keys, step kinds, Examples placeholders, and data-table columns. A fix is offered only when it is certain — the suggested name is near enough, and the misspelling occurs exactly once as a whole token in that same file — so applying one is never a guess. Reach it from the squiggle or from the token itself; a use: error underlines the macro’s name key, which is often several lines above the word you typed.

The server analyzes the configured suite — proef.toml’s [run] suite if set, else the tests/ convention — resolved under the directory it is launched in (its working directory), discovering every .feature file and every packs/*.yaml / packs/*.yml macro pack beneath that root — the same resolution proef test uses, so the two never diverge. Launch your editor from the project root (or configure the server’s root/working directory to it).

When analysis fails

A panic inside analysis or a feature never ends the server. proef reports it once through window/showMessage — the channel an editor actually surfaces — and keeps serving; the next edit retries, and reports again only if the new state also fails. A server that died would show nothing, which reads as “proef has no opinion about this file” rather than as the failure it is.

Running proef lsp by hand also prints the panic to stderr, which is where the detail lives.

Naming the config: --config

proef lsp --config <path/to/proef.toml> names the config to read instead of searching for one, exactly as it does for every other subcommand (CONFIG.md). Reach for it when discovery cannot find the right file — most often a config that sits beside the suite rather than above it, which an upward search launched from the repository root can never reach.

For proef lsp the flag also outranks the workspace root the client announces: the flag names a file, and a named file is not a guess to be improved on. Without it, an editor rooted somewhere else loads a different config than the runner, and the diagnostics stop being trustworthy in exactly the layout the flag exists for — proef test --config … runs green while every ref: reads as unknown in the editor.

cmd = { "proef", "lsp", "--config", "/abs/path/to/proef.toml" },

Unlike the runner, an unreadable or absent file does not stop the server: it starts on defaults, because an editor offering less is better than one that will not boot. A relative path works, but prefer an absolute one — an editor’s working directory is rarely the one you assume.

--env <name> travels the same way (proef lsp --env staging, or PROEF_ENV): it selects the [env.<name>] profile the analysis resolves ${url:…}/${vars:…} against, so the editor reports the missing_config_var a staging run would hit and not the one the default profile would. Before 0.18 the flag was accepted for lsp and silently dropped.

File types served

Neovim

Built-in LSP client (Neovim 0.8+), no plugin required. Add to your config and open a .feature file from inside the suite:

vim.api.nvim_create_autocmd("FileType", {
  pattern = { "cucumber", "gherkin", "yaml" },
  callback = function(args)
    vim.lsp.start({
      name = "proef",
      cmd = { "proef", "lsp" },
      -- The server scopes analysis to the configured suite under its launch
      -- directory; anchor root_dir at the nearest proef.toml (or the current
      -- file's directory) so the two agree.
      root_dir = vim.fs.dirname(
        vim.fs.find({ "proef.toml" }, { upward = true, path = args.file })[1]
      ) or vim.fs.dirname(args.file),
    })
  end,
})

With nvim-lspconfig you can instead register it as a custom server via vim.lsp.config/configs, using the same cmd = { "proef", "lsp" }.

Helix

Add a server and attach it to the languages you author. In ~/.config/helix/languages.toml:

[language-server.proef]
command = "proef"
args = ["lsp"]

[[language]]
name = "gherkin"
language-servers = ["proef"]

# Attach to pack YAML too, so pack diagnostics surface while editing macros.
[[language]]
name = "yaml"
language-servers = ["proef", "yaml-language-server"]

Run hx --health gherkin to confirm Helix found the proef binary.

Emacs (Eglot)

Eglot ships with Emacs 29+. Associate your feature-file major mode (for example feature-mode) with the server:

(with-eval-after-load 'eglot
  (add-to-list 'eglot-server-programs
               '(feature-mode . ("proef" "lsp"))))
;; M-x eglot in a .feature buffer opened from the suite root.

For lsp-mode, register a stdio client whose connection is (lsp-stdio-connection '("proef" "lsp")) for the same major mode.

v1 limitations

This is the first release of the language server. Known boundaries:

• proef.toml config is a startup snapshot. ${url:…} / ${vars:…} values are read once when the server starts. After editing proef.toml (or switching PROEF_ENV), restart the server to pick up the change; until then those references analyze against the old (or, if the file could not be loaded at startup, an empty) scope and may warn.

• Built-in macros have no jump target. The expect* family lives in a pack compiled into the binary, not a file on disk, so there is nothing for go-to-definition to open. Hover still answers — the macro is in the analysis like any other, and reports its pack as builtin:…, which is why the jump is unavailable. proef macros lists the whole family with the sentence each binds.

• Completion ranking is best-effort. All of the suite’s macros are offered; ranking is a lightweight edit-distance heuristic. Full context-aware ranking is a follow-up.

• External edits to closed files need a reopen. The server does not watch the filesystem; it re-reads a file’s bytes from its open editor buffer. If a .feature or pack file is changed outside the editor (or by another tool) while closed, reopen it so the server sees the new bytes.

• No VS Code extension yet. v1 is a server-only generic-LSP binary. It works with any editor that speaks generic LSP (Neovim, Helix, Emacs, Sublime LSP, …); a VS Code wrapper is a possible follow-up.

  If one is built, its documentSelector must match on path ({ scheme: "file", pattern: "**/*.feature" }), not on a language id. The ecosystem is split — the two established Gherkin extensions register cucumber and feature respectively — so a selector naming either id attaches for some users and silently does nothing for the rest. The table under File types served is that split; a path selector is the only thing all of it has in common.

• No Zed support. Zed binds language servers to languages it has a tree-sitter grammar for, and there is no Gherkin grammar in it. That grammar is a prerequisite, not a configuration step, so Zed waits on work outside this repository.

• Overlay lookup can still miss if the suite root is reached through a symlink. The root is deliberately left uncanonicalized (canonicalizing would resolve symlinks and desync source names from the client’s document URIs), so if an editor resolves a symlinked suite root differently than the server’s raw working directory, the overlay lookup can miss and the LSP analyzes the saved on-disk bytes instead of the unsaved buffer. An editor’s percent-encoding choice no longer matters here — the overlay matches open buffers by decoded source name, not the raw URI.

Troubleshooting

First stop, always: proef doctor (native libraries, environment, secret store/key health) and proef test <suite> --dry-run (full validation with located diagnostics, no network). Every diagnostic code is indexed in DIAGNOSTICS.md.

Reading the output

Exit codes are a contract:

Step glyphs:

Under --console dotted the tree collapses to one glyph per scenario — . passed, F failed, s skipped, w warned (lowercase = non-gating) — and failures still print in full after the pool; --console quiet keeps only the run line and the summary. On a terminal the status vocabulary is colored; NO_COLOR (or a non-terminal stream) turns it off, and run.log never carries the paint either way. The record and the exit code are identical in every mode. Every run’s last line names its run id and wall-clock — the id is the reproduction key --shard, --shuffle and ${fake:…} all hang off.

Frequent situations

“missing secret value(s)” — the suite references ${secret:NAME} with no value available. Fix: proef secret set NAME (or export PROEF_SECRET_<NAME>=…). In CI, see AUTHORING — Secrets in CI.

An assertion fails on values that look identical — they differ only in whitespace, and proef says so:

Assert failure (f--s.hurl:9: actual: string <ok> expected: string <ok >)
  [whitespace only — actual: string·<ok>, expected: string·<ok·>]

The bracketed note appears only when the two values differ solely in whitespace, and repeats them with every whitespace character drawn: · for a space, \t / \r / \n for the usual escapes, and \u{a0} for the exotic ones — a non-breaking space pasted out of a browser being the classic. Usual causes: a trailing space in the expectation, a CRLF fixture leaking \r into a body, or a tab where the author typed spaces.

“unknown environment <name>” — --env <name> (or PROEF_ENV) names an environment proef.toml doesn’t define; the error lists the known ones. Fix the name or add the [env.<name>] section. Exit 2.

“<url|vars> variable <key> is not set” (resolve::missing_config_var) — a pack references ${url:key} / ${vars:key} that neither the base [url]/[vars] nor the active [env.<name>] defines. Add it, or select the environment that has it with --env. See CONFIG.md.

“no path given and no default suite found” — proef test got no path and there is no [run] suite in proef.toml nor a tests/ directory. Pass a path, set [run] suite, or create tests/. Exit 2.

Exit 3 with connection errors — the target is unreachable. Check the URL your ${url:base} resolves to for the active --env (a --dry-run prints nothing wrong because no request is sent), then the network. The dev fixture (cargo run -p xtask -- fixture) binds the default ${url:base} port (8787), so with no PROEF_BASE_URL set it becomes your local target automatically (it falls back to an ephemeral port, printing PROEF_BASE_URL, only if 8787 is busy).

“batch budget exceeded — scenario thread abandoned” — the watchdog killed a batch that outran its computed budget (timeouts × attempts + delays + repeats + margin, ADR-0007). Usually a huge retry:/delay: (or a hurl-side [Options] repeat:/ retry-interval:/max-time:) value — the pack lint caps literals (counts at 10 000, durations at one hour), and the computed budget itself clamps at four hours, so a lint-clean product that would otherwise run for days is abandoned at the ceiling. A {{var}}-driven retry:/delay:/[Options] repeat:/max-time: cannot be estimated at all — it resolves inside hurl at run time — so the batch falls back to the default budget ([http] timeout × 4, at least 60s) rather than to an estimate that assumes no retries. If a legitimately long templated retry is being abandoned, raise [http] timeout.

“global state file .proef-state.json is not valid JSON” — the persistent World is derived data (only saveAs: global promotions live there). Deleting .proef-state.json is safe; the next run recreates it.

Corrupt .proef-secrets.json — proef doctor names it; the next proef secret set moves the wreck to .proef-secrets.json.corrupt and starts fresh. Values are re-enterable; nothing else references the file.

“no scenarios matched the filters” — --tags/--scenario selected nothing; exit 2 by design so a typo’d filter can never produce a silent green CI run.

A failure line carries (via tests/hurl/admin.hurl#admin.search) — not an error. The step ran a named fragment (ADR-0018) rather than an inline hurl: block, and that is the third file involved: the request lives there, not in the feature or the pack. The spelling is the one ref: accepts, so it pastes straight back into a pack. Every failure sink carries it — console, explain, TAP, JUnit, the GitHub summary, the HTML report.

“ref: names no loaded fragment” (pack::unknown_ref) — either the name is a typo (the message suggests the closest) or no fragment files were loaded at all, which the message says plainly. The usual cause is a missing [run] fragments in proef.toml: without it nothing is scanned, so every ref: is unknown. Note the root resolves against the config file’s directory, not your working directory.

“reads name, which nothing supplies” (lower::unbound_placeholder) — a fragment’s variables are not implicit: what the .hurl file reads must be bound at pack, macro or step scope, captured by an earlier step, supplied by the fragment’s own [Options] variable:, or carried by a secret of that name — and inside a bind: value, an earlier-sorting sibling literal counts too (injected lines evaluate in name order). --dry-run reports it without a network. With proef lsp running, completing inside a bind: table offers what the pack’s ref:ed fragments still need — the union over every fragment the pack refs (ranked by the nearest ref:, each labelled with its owner), minus the names a fragment supplies itself.

“index is supplied twice” (pack::option_declared_twice) — the fragment sets the name in its own [Options] variable: and a bind: supplies it. Both reach the entry as variable: index= and hurl takes the last, which is the fragment’s — so the bound value would silently never be sent. Delete whichever is not authoritative: the bind: if the file’s own default is right, the variable: line if the pack should decide. The same rule already applies to retry:/delay:.

Go-to-definition on a ref: does nothing — check that [run] fragments is set and that the editor’s workspace root matches where proef.toml lives; the server resolves the corpus relative to that file’s directory.

“payload does not parse: contains no hurl entries” — the hurl: block is comments/blank lines only (a temporarily commented-out request). proef refuses it at load: a step that executes nothing must not report green.

A Then step shows ∅ not run (its request entry did not run) — the request it asserts on was skipped (guard or earlier failure); the asserts had nothing to attach to.

Windows/| head pipelines — closed pipes are tolerated everywhere; if a pipeline misbehaves, check the consumer, not proef’s exit code.

Building from source

proef-engine-hurl links native libraries. Debian/Ubuntu: apt install build-essential pkg-config libssl-dev libcurl4-openssl-dev libxml2-dev libclang-dev. macOS: Xcode CLT suffices. Verify with proef doctor — it reports the embedded hurl, parser/libxml2 linkage, and libcurl. Prebuilt binaries (brew/binstall/GitHub Releases) need none of this.

Digging deeper

Every run leaves .proef-runs/<run-id>/: events.jsonl (the machine record — EVENTS.md), run.log (console mirror), and artifacts/ with the exact executed .hurl files. proef explain summarizes the latest run, proef diff [base] [new] compares two of them (regressions, fixes, flakiness, perf), and proef report [run] writes a self-contained HTML page of a run; the reproduce: hurl --test … line under a failure replays the artifact with stock hurl, taking proef out of the loop entirely.

A cancelled run (Ctrl-C) is a complete record, not a truncated one — proef explain/proef report never banner it as incomplete, because its nonzero skipped count already says what didn’t get to run. proef diff --fail-on-regression disagrees on purpose: it still refuses to certify “no regressions” against a cancelled run, since a regression could be hiding among the scenarios it never reached. The three commands’ differing treatment of cancelled is a deliberate choice, not an inconsistency.

proef diff takes each side as a run id, a record directory, or an events .jsonl file — the stream is the record, so a file means the same thing under any name. That third form is the CI baseline flow: upload .proef-runs/<id>/events.jsonl as an artifact on your main branch, download it in the PR job as (say) baseline.jsonl, then

$ proef test tests/features --run-id pr
$ proef diff baseline.jsonl pr --fail-on-regression   # exit 1 on passed → failed

and a scenario that regressed against main fails the PR without a run record store shared between jobs.

proef diff keys steps by (text, ordinal) so that line shifts don’t lie and repeated steps stay distinct. The trade-off is positional: if a scenario loses an earlier duplicate of a step, every later instance shifts down one ordinal, and the comparison lines up two different runs’ steps. Timing and attempt counts for the shifted steps can then be attributed to the wrong one. Renaming or reordering steps between the two runs being compared is worth a second look at the numbers; adding or removing steps at the end is not affected.

proef.toml — the project configuration reference

proef.toml lives in the project root — found by searching up from the working directory, so any subdirectory works (a project is where its proef.toml is, not where your shell is) and is committed project config — it describes the suite, not your machine. Every key is optional; an absent file means all defaults. Unknown keys are rejected (deny_unknown_fields), so typos fail loudly instead of being ignored.

The file holds a few kinds of thing, kept in distinct sections: runner config (how proef behaves — [run], [http], [sla]), suite variables (data your tests reference — [url], [vars]), run metadata ([meta], recorded in the run head) and report tag links ([tag-links]), plus per-environment overrides ([env.<name>]). Secrets never live here (see below).

Precedence (highest wins): built-in defaults < proef.toml base tables < active [env.<name>] < command-line flags. For a suite variable that means [url]/[vars] < [env.<active>]; for a runner setting like jobs, the --jobs flag still wins over both.

Reference

# ── runner config (how the tool behaves) ────────────────────────
[run]
suite    = "tests"          # default suite path — `proef test` needs no argument
fragments = "tests/hurl"    # root of the hurl files packs may `ref:` (optional)
jobs     = 8                # parallel scenario workers
runs-dir = ".proef-runs"    # where run records land
keep-runs = 200             # past records kept there (0 = only the run in flight)
setup    = "tests/setup.feature"      # run once before the pool (optional)
teardown = "tests/teardown.feature"   # run once after the pool (optional)
exclusive-tags = "@serial"  # scenarios matching this run with the pool to themselves

[http]
timeout-ms      = 30000     # per-request timeout (batch-level default)
follow-location = false     # follow redirects
max-redirs      = 10        # redirect ceiling (only meaningful with follow-location)
user-agent      = "proef"   # sent with every request — how a server spots test traffic
cookie-store    = true      # false = run cookie-less (hurl's --no-cookie-store):
                            # prove a stateless API by refusing to carry a session
# TLS and proxy: the settings that differ *between environments*, which is why
# they belong here and not in a pack. Paths resolve against this file.
insecure        = false     # skip certificate verification (warns loudly when on)
cacert          = "certs/ca.pem"       # trust this CA instead of the system store
client-cert     = "certs/client.crt"   # mTLS: the certificate to present
client-key      = "certs/client.key"   # mTLS: its private key
proxy           = "http://proxy.corp:3128"
no-proxy        = "localhost,.internal"

[sla]                       # opt-in run-level latency budget (omit = no gate)
p95-ms = 250                # 95th-percentile per-step duration ceiling
max-ms = 1000               # slowest single-step ceiling

# ── suite variables (data your tests reference) ─────────────────
[url]
base = "http://127.0.0.1:8787"   # → ${url:base}; the dev fixture's default port

[vars]
apiVersion = "v1"                 # → ${vars:apiVersion}
username   = "dev@example.com"    # → ${vars:username}

# ── per-environment overrides (mirror the base tables) ──────────
[env.staging.url]
base = "https://staging.example.com"

[env.prod.url]
base = "https://api.example.com"
[env.prod.vars]
username = "release@example.com"
[env.prod.http]                   # an environment may override a runner setting too
timeout-ms = 60000

A custom --run-id is kept, but never rotated

--run-id accepts any single path component, and a run named that way writes a perfectly good record that explain, diff, flaky, report and --rerun all find — a directory is a record because it holds an events.jsonl, not because of the shape of its name (ADR-0021).

“The latest run” is resolved by time rather than by spelling, and where that time comes from differs by name: a generated id is a uuid-v7, which carries the moment proef minted it, while a custom id carries no time at all and orders by the directory’s modification time instead. Both are wall-clock, so the two compare directly — but a custom-id record that is later copied or restored sorts by that moment, not by when it ran.

What a custom id changes is rotation: proef deletes only uuid-named directories, so [run] keep-runs will never reclaim .proef-runs/pr-1234/. A caller minting a fresh custom id per build accumulates records without bound. That is deliberate and the asymmetry is the point — runs-dir may be ., and guessing at user-named directories is the worse failure by a wide margin. Reading a directory is safe; deleting one is not.

Use --run-id when you want a stable, externally-known name — a CI job archiving .proef-runs/pr-1234/ by path — and clean those up the same way you created them.

Environments

An [env.<name>] profile mirrors the base tables and deep-merges over them, key by key — so an environment lists only what changes; everything else inherits from the base (the Cloudflare-Wrangler / Cargo-[profile.*] model). Select the active environment at run time:

$ proef test                 # base [url]/[vars]; discovers the default suite
$ proef test --env prod      # [env.prod.*] merged over the base
$ PROEF_ENV=staging proef test   # same, via the environment variable

The rule is almost uniform: under [env.<name>], url.* / vars.* override variables and http.* / sla.* override runner settings wholesale — but [env.<name>.run] accepts only jobs. Every other [run] key (suite, runs-dir, setup, teardown, exclusive-tags, …) names what the suite is, not how hard to run it, and letting an environment swap those made two environments run different tests while claiming to be the same suite — so listing one there is a hard parse error (exit 2), never a silent no-op. A named-but-undefined --env is a user error (exit 2) listing the known environments. A ${url:key} / ${vars:key} referenced in a pack but defined in neither the base nor the active environment fails at lower time (proef::resolve::missing_config_var).

TLS, proxies and mTLS ([http])

These are the settings that describe an environment rather than a test, which is why they live here and not in a macro pack. Staging presents a self-signed certificate, CI traverses a corporate proxy, production speaks mTLS — and the same suite has to run against all three without a single pack changing. Put the differences in [env.<name>.http] and the shared parts in [http]:

[http]
user-agent = "proef-suite"          # every environment identifies test traffic

[env.staging.http]
insecure = true                     # staging's certificate is self-signed
proxy    = "http://proxy.corp:3128"

[env.prod.http]
cacert      = "certs/prod-ca.pem"   # a private CA, not the system store
client-cert = "certs/client.crt"    # mTLS
client-key  = "certs/client.key"

Three things worth knowing before you rely on this:

• Paths resolve against proef.toml, not the working directory — the same one-path rule every other config path follows, so proef test from a subdirectory finds the same CA bundle it finds from the project root.
• insecure = true warns on every run, naming the profile that set it. A suite that goes green without verifying a certificate has not proved what a green suite normally proves, and the run record deliberately carries no config, so the warning is the whole audit trail.
• A per-entry [Options] block still wins — for all of these but one. These are batch-level defaults, so one request that must skip verification, or present a different certificate, says so in its own hurl block and overrides the table for itself alone. The exception is user-agent: hurl has no per-entry option for it, so [http] user-agent is run-wide and a single entry cannot opt out. Set a User-Agent: request header in that entry instead.

Credentials stay out of this table on purpose. There is no [http] user or netrc key: a password belongs in the secret store (proef secret set), where it is encrypted at rest and masked out of artifacts, events, logs and reports. A proxy URL is the one edge — if yours embeds credentials, it is plaintext in a file you probably commit, and it belongs in an environment variable your shell expands into the proxy’s own configuration instead.

Balancing a shard matrix by duration (--shard-weights)

--shard I/N assigns scenarios by a frozen hash of (file, scenario), so adding one test never re-buckets the others. What it cannot do is balance by time — and a CI matrix finishes when its slowest shard finishes, so a count-balanced split routinely leaves runners idle.

Every run that reaches its suite writes timings.json into its run directory (a run whose [run] setup aborted has no suite to weigh, and writes none). Archive that one file and hand it to the next run’s matrix:

# each matrix job, all reading the same archived file
- run: proef test --shard ${{ matrix.shard }}/4 --shard-weights timings.json

Every job must read the same file. The tempting alternative — “let each job weight by its own newest local record” — is silently wrong: matrix jobs run on different machines with different (usually empty) runs-dirs, so each would compute a different assignment and scenarios would run twice or not at all while the suite still reported green.

Two rules place scenarios, and they partition rather than compete:

• a scenario the file mentions is placed by the balanced split (longest-processing-time-first: heaviest to the lightest shard, repeatedly);
• a scenario it does not mention falls back to the frozen hash.

So a test added after the timings were captured still runs exactly once. That property is pinned by a test that runs a whole matrix and asserts set equality both ways.

What you give up is the thing hash mode was chosen for: a balanced split is not stable under insertion, so adding a slow scenario can move others. That is what balancing means, and it is why the flag is opt-in rather than the default. A missing or malformed weights file is exit 2, never a silent fall back to the unbalanced split you passed the flag to avoid.

SLA gate ([sla])

The optional [sla] table is a run-level latency budget: after a run, proef folds every step’s wall-clock duration into two aggregates and fails the run if either exceeds its ceiling. p95-ms caps the 95th-percentile step duration (the typical slow request); max-ms caps the single slowest step. Skipped steps (which never hit the network) are excluded from the population.

The gate is opt-in and off by default — with no [sla] table a run behaves exactly as before. A breach prints the offending metrics and the slowest steps on stderr and maps to exit 1 (a test failure), the same code as a failed assertion. It never introduces a new exit code and never downgrades a User/System fault (exit 2/3), so a breach can only turn an otherwise-green run red. Tighten or loosen per environment the usual way:

[sla]                       # baseline budget
p95-ms = 250
max-ms = 1000

[env.staging.sla]           # staging tolerates slower responses
p95-ms = 800

This is distinct from hurl’s per-request duration < <ms> assert (which fails one step inside a hurl: block): the [sla] gate is an aggregate budget over the whole run, while duration < is a targeted per-request check. Use the assert for a hard per-endpoint SLA and [sla] for a suite-wide budget.

Hurl fragments ([run] fragments)

Names the root directory holding the .hurl files a pack may ref: (ADR-0018), scanned recursively. Those files stay valid hurl — the # @proef <name> annotation is an ordinary comment — so the same file runs under hurl and under proef, and a corpus you already own is annotated once instead of transcribed into YAML where it drifts.

[run]
fragments = "tests/hurl"

# tests/hurl/admin.hurl — runs as-is under `hurl --variables-file …`
# @proef admin.search
GET {{base}}/api/v1/admin/search/{{index}}
Authorization: Bearer {{apiToken}}
[Query]
q: {{q}}
HTTP 200

# the pack supplies the fragment's variables; nothing is implicit
bind:
  base:     ${url:base}          # pack scope — every macro in the file
  apiToken: ${secret:apiToken}   # injected at run time, never into an artifact
macros:
  search:
    params: [q, index]
    defaults: { index: records }
    match: "the operator searches for {q}"
    bind: { q: "${q}", index: "${index}" }   # macro scope
    steps:
      - ref: admin.search

Relative paths resolve against this file’s directory, not the working directory, so the root may sit outside the suite — even outside the repo — and the config still works from any subdirectory. There is no convention fallback: with the key unset a ref: reports proef::pack::unknown_ref and says no fragment files were loaded, which beats guessing at a directory. Entries with no # @proef annotation are inert, and the files are not scanned at all until some pack names a fragment — so pointing at a corpus you did not write neither changes what runs nor costs anything to parse. When a pack does name one, the corpus is scanned once per command, not once per pack load: a run with [run] setup/teardown loads packs four times against the same files, and on a 200-file corpus rescanning each time was most of the run.

proef fmt never touches these files: it normalizes hurl blocks inside packs by locating them in YAML, and a corpus you do not own is not proef’s to rewrite.

The read is bounded. A single corpus file over 8 MiB is skipped with proef::pack::oversized_fragment_file, and the reader stops once the corpus as a whole passes 64 MiB (proef::pack::fragment_corpus_too_large). Both name the file, both leave every other file loading, and a ref: into a skipped file then reports unknown_ref beneath them.

These are not tuning knobs — they are the line past which the input is not a test corpus. A hurl file is human-authored text, so 8 MiB is on the order of 200,000 lines; the largest corpus anyone has reported is 15 files and 112 fragments. The bound exists because the root is one config line: pointed one directory too high it reads whatever is underneath, and a 279 MB file cost 601 MB of resident memory on proef flows — a command that never looks at a fragment. If you hit either limit, the fix is almost always to narrow [run] fragments to the directory that actually holds your hurl files.

Suite setup & teardown ([run] setup / [run] teardown)

[run] setup and [run] teardown each name a feature file run once around the whole suite (the model Playwright/Jest globalSetup use, ADR-0014). setup runs before the parallel pool and merges its saveAs: global promotions into the shared store before any scenario lowers, so it is the place to seed a fixture or provision shared state that every scenario then reads via ${global:…}. teardown runs once after the pool for cleanup.

[run]
setup    = "tests/setup.feature"
teardown = "tests/teardown.feature"

Failure semantics are deliberate: a setup failure aborts the run before the pool and maps to a user (2) or system (3) fault — a broken fixture is not a failing test, so it never becomes exit 1. Teardown runs only if setup succeeded (it still runs after ordinary scenario failures, for reliable cleanup), and a teardown failure is a distinct exit 3 cleanup fault — the suite’s own verdict stands, but the failure is never silently swallowed. Both features are excluded from the pool, so a setup/teardown feature inside the suite directory never also runs as an ordinary scenario. Auth is already covered by pre-set secrets, so setup is for seeding/provisioning, not obtaining a runtime token.

Run metadata ([meta], [env.<name>.meta], --meta)

Key/value pairs recorded in the run head and shown by the HTML report, the GitHub summary, explain and diff — never interpreted, never harvested. Static facts live in config; invocation facts ride the flag; the active env profile’s table deep-merges over the base, and flags win over both — the same base < env < flags chain as every other setting:

[meta]
team = "payments"

[env.staging.meta]
tier = "staging"

proef test --meta commit=$(git rev-parse HEAD) --meta build=$CI_JOB_URL

proef itself never reads git, the hostname, or CI variables — if you want the SHA recorded, your shell harvests it and proef records it (ADR-0020). The active --env profile name is recorded automatically as its own field: it is user-chosen input, and without it two records of one suite against different [env.<name>] merges read as regressions in diff.

Tag links ([tag-links])

Tag glob → URL template; a matching tag becomes a link in the HTML report’s by-tag table and the GitHub summary ({tag} is substituted). The pattern language is the same anchored glob --tags atoms use — one matcher for selection, exclusivity and links:

[tag-links]
"JIRA-*" = "https://jira.example/browse/{tag}"

Everything else ignores the table; a tag matching no pattern stays plain.

Scenarios that cannot share the pool ([run] exclusive-tags)

Some scenarios cannot run beside anything: one asserting absolute positions (items[0]) needs a store no concurrent scenario writes to. exclusive-tags is a tag expression — the same language --tags takes — selecting those. Atoms may glob (* any run, ? one character, anchored — @serial-* is the whole family; case-sensitive, unlike Robot Framework):

[run]
jobs = 8
exclusive-tags = "@serial"

  @serial
  Scenario: The report lists every record in order

A matching scenario waits for the pool to drain, runs alone, and the pool refills after it. Ordering is unchanged: scenarios still start in discovery order, so an exclusive one never loses its place — and that is also what the cost is made of. Queueing is strict FIFO, so while an exclusive scenario sits at the head no new scenario starts: the pool drains to empty, the exclusive one runs by itself, and only then does parallelism return to jobs width. Expect a throughput dip around each exclusive scenario roughly as long as the slowest scenario already running, plus the exclusive one’s own duration. Put another way, the cost is bounded and predictable rather than free.

Exclusivity is enforced against the dispatcher’s own bookkeeping. A scenario the watchdog abandons for blowing its batch budget (ADR-0007) leaves the active set immediately, but its detached thread keeps going until the request in flight returns — hurl cannot be cancelled mid-entry. In that window an exclusive scenario can start while an abandoned neighbour is still issuing requests. It takes a budget blowout in the same moment to happen, and it is the one caveat the isolation guarantee carries; a run that never trips the watchdog never meets it.

This is exclusion, not ordering. A scenario that must run before the others — installing a fixture the rest depend on — belongs in [run] setup, which already runs once before the pool exists.

A tag expression in config, not a reserved tag name, deliberately: with a bare convention, a scenario added months later lands untagged in the parallel pool and breaks isolation intermittently — which reads as flakiness rather than as a missing declaration. The expression keeps the rule in one reviewable place. A malformed one is a user error (exit 2), never a silently-ignored key.

Editor support (JSON Schema)

TOML language servers validate against JSON Schema, so one file buys completion, hover documentation and typo detection in proef.toml itself:

proef schema config > proef-config.schema.json

Then point your editor at it. Taplo and tombi (the engines behind VS Code’s Even Better TOML, and the usual Neovim/Helix TOML setups) both read a first-line directive:

#:schema ./proef-config.schema.json

[run]
suite = "tests"

The schema is generated from the same Rust model that parses the file, so it describes the keys as they are written — kebab-case (runs-dir, keep-runs, exclusive-tags), not the struct spelling — and inherits deny_unknown_fields, so an editor flags a typo’d key before you run anything. Each key’s hover text is its documentation from this page’s source.

Regenerate it after upgrading proef; there is no --add-to here (that installs the pack schema beside packs), because a project has one proef.toml and the #:schema line is a one-time edit to a file proef otherwise only ever reads.

What does not live here

• Secrets — proef secret set / PROEF_SECRET_<NAME> env, on their own encrypted channel (${secret:…}); never in proef.toml (see AUTHORING — Secrets).
• Per-machine values — proef.toml is committed and shared; anything that differs between teammates belongs in an environment variable (${env:NAME:-default}) or an environment profile you don’t commit.

Feature files reference ${url:…} / ${vars:…} / ${secret:…} and declare none of them — variables have exactly one home, this file, and test files stay pure prose. proef discovers the nearest proef.toml by searching up from the working directory (like cargo/git), so it is found from any subdirectory.

The search only goes up, so a discovered file must sit at or above the directory you run from — that is a requirement, not a convention. Putting proef.toml beside the suite (tests/proef/proef.toml) and running from the repository root does not work by discovery: nothing searches downward, so the config is simply not found and every ${url:…} reads as unset.

--config <path> names the file instead and removes the constraint entirely:

$ proef test --dry-run --config tests/proef/proef.toml

It is global — every subcommand accepts it, including proef lsp and proef test --watch. The editor loading a different config than the runner is the drift that makes diagnostics untrustworthy, and --watch watches the file the run actually resolved through, so editing it retriggers.

For proef lsp the flag also outranks the workspace root the client announces: the flag names a file, and a named file is not a guess to be improved on.

A named file that does not exist is a user error (exit 2) for the runner, not a fall back to defaults: discovery finding nothing means “no project here”, but a named path that is not there is a typo, and answering it with a silently unconfigured run is the thing worth refusing. proef lsp makes the opposite call and starts anyway — an editor offering less is better than one that will not boot, and unlike a run it produces no results to be wrong.

It applies to every subcommand in the same way, including the ones that read nothing from the file: proef fmt --config missing.toml exits 2 rather than formatting and reporting success. The flag names a file, and a named file that is not there is a typo whatever the command was going to do with it.

Which directory a path is relative to

Paths written in proef.toml resolve against the directory holding proef.toml. Paths typed on the command line resolve against the working directory. That is the whole rule, and it has no exceptions: suite, fragments, setup, teardown and runs-dir all follow it, as do the two files proef keeps beside them — .proef-state.json (the persistent World) and .proef-secrets.json (the secret store). An absolute value is taken as written, so a corpus or a record store may sit outside the project entirely.

The consequence worth stating plainly: a project is where its proef.toml is, not where your shell is. Running from a subdirectory finds the same config by walking up, and now resolves the same suite, writes the same run records, and reads the same World and the same secrets as running from the root. It is the convention Cargo (path/members are manifest-relative), tsconfig.json and pytest’s rootdir all follow.

With no proef.toml in scope there is nothing to be relative to, so paths stay relative to the working directory, unchanged.

A config that does not sit at the project root spells its paths from where it sits — a proef.toml in tests/proef/ beside a suite in tests/features/ writes suite = "../features".

How a path is spelled in what proef writes

The same rule, read backwards. Nothing proef records names the machine it ran on: a path that reaches an artifact, a sidecar, an event, a report or a diagnostic is spelled relative to the directory holding proef.toml.

That matters because ADR-0010 makes the emitted .hurl a contract — the same inputs must give the same bytes — and a run record is meant to travel from a laptop to CI and back. So all four of these produce the same record:

$ proef test                              # path derived from [run] suite
$ proef test suite                        # typed
$ proef test /home/you/proj/suite         # typed absolutely
$ cd sub && proef test                    # from a subdirectory

Two cases keep an absolute path, because no project-relative spelling of them exists: a suite or fragment corpus that genuinely sits outside the project (fragments = "/opt/shared-corpus"), and a run with no proef.toml in scope. A path typed relative is recorded exactly as typed — it is machine-independent already, and it is the spelling your terminal can open.

How long run records live

Each run writes one directory under runs-dir holding its artifacts, its events.jsonl and its run.log. [run] keep-runs bounds how many are kept: the default of 200 suits an archive, and a suite re-run on every save wants far fewer. The artifacts are the largest part and are byte-identical between runs of an unchanged suite — but they are what that run executed, and once the corpus moves on proef artifacts no longer reproduces them, so the cost is bounded rather than dropped.

Rotation only ever deletes directories named by a generated run id, because runs-dir may be . or otherwise shared with your own files. A record written under --run-id <name> is therefore never rotated: if CI mints a fresh name per build into a persistent directory, prune it yourself.

A starter file ships as proef.toml.example in the repository root.

Diagnostic codes — the greppable index

Every proef diagnostic carries a stable code (proef::<area>::<name>), printed above the rendered message. Codes are a contract: they never change meaning, and searching this file (or tests/errors/) for a code you hit is the fastest route to the cause. The corpus column marks codes with a seeded broken example under tests/errors/<area>__<name>/ — dry-running one shows the exact rendered output.

Severity is error (fails validation/exit 2) unless marked warning.

proef::feature::* — feature-file parsing and expansion

proef::pack::* — macro-pack loading and validation

proef::bind::* — matching prose to macros

proef::lower::* — lowering to engine batches

proef::resolve::* — ${…} variable resolution

proef::emit::* — artifact emission

proef::run::* — preparing a scenario to execute

proef::config::* — proef.toml loading

proef::source::* — source access (LSP whole-suite analysis)

proef::tags::* — reserved-tag recognition

Coverage note

The fragment-file codes (pack::duplicate_fragment, pack::bad_annotation, pack::unreadable_fragment_file, pack::unread_bind_key, lower::unbound_placeholder, lower::multiline_bind, lower::secret_in_composite_bind, run::asset_unstageable) are covered in crates/proef-cli/tests/fragments.rs rather than tests/errors/: they need a [run] fragments root, and the seeded corpus is deliberately config-independent.

The config::* codes are covered in crates/proef-cli/tests/cli.rs for the same reason: a broken proef.toml cannot live in the config-independent seeded corpus.

30 of the 77 codes carry a seeded corpus case today; the corpus guard asserts a minimum, not parity. When you add a diagnostic, add its code here and prefer seeding a tests/errors/<area>__<name>/ case alongside it. tags::reserved_tag_typo is a warning (a near-miss is not an error), so it cannot live in tests/errors/ — that corpus fails dry-run by design — and is covered by a unit test instead.

Every row here is a code some code path actually emits. That was not free: this file used to carry a pack::load row for a defensive case that never had a code (a non-diagnostic core failure while loading packs renders as a bare error:), so a reader who grepped for it found nothing and had no way to tell the index was wrong. A row nothing emits is worse than a missing row.

Event schema — the run record for machine consumers

.proef-runs/<run-id>/events.jsonl is the run record (ADR-0008): one JSON object per line, in emission order, no second format. proef explain, diff, flaky and the console tree derive from this stream; --junit and the GitHub summary are built from the same run’s in-memory outcome (identical content, different plumbing — the stream stays the only persisted format; the one exception is a --rerun’s JUnit, whose carried base scenarios are reconstructed from the base record — see rerun_of below). Consume it with jq, a log shipper, or anything line-oriented.

Two derived sidecars sit beside the record and are not records: timings.json (per-scenario durations, re-derivable from the stream, read back by --shard-weights) and inputs.json (the run’s input fingerprint — a hash of proef’s own inputs, proef flaky’s equivalence class, ADR-0020 amendment; absent on pre-0.18 records). Neither adds a variant or a field to the stream.

Wire shape: serde-tagged with "event", snake_case names. The first line is always run_started and declares schema (currently 1); the last is run_finished. The schema is additive-only: new variants and new optional fields may appear, existing fields never change meaning or vanish. Consumers must ignore unknown variants and unknown fields.

Truncated records. A record whose last line is not run_finished is a run that did not finish — a hard interrupt (second signal, exit 130), a SIGKILL, or a crash. (A single SIGTERM/SIGHUP is a graceful cancellation: the record closes normally with run_finished + cancelled.) Treat it as partial: the scenarios present did happen, but the totals were never written and no verdict was reached. explain, report and diff classify these and say “run incomplete” rather than reporting a total they cannot know. Consumers should do the same rather than inferring zero.

One exception to “fields never change meaning”, recorded rather than hidden. run_finished’s passed/failed/skipped counted every phase before 0.6.0; from 0.6.0 they are the main-suite verdict and exclude [run] setup/teardown (ADR-0014), so those totals agree with the exit code, --format json/TAP, and the summary line. One deliberate divergence: a failed phase additionally appears in JUnit as its own suite (a gated pipeline must see the failure), so JUnit’s per-case counts can exceed these totals on a run whose setup or teardown broke. A pre-0.6.0 record read by a current consumer reports the older meaning.

Variants

run_started — head of every stream. schema (u32) · run_id (string — uuid-v7 by default, but --run-id passes any name through verbatim, so consumers must treat it as opaque).

scenario_started — scenario (string) · file (string) · timestamp_ms (u64, run-relative ms — only present with injected timing) · worker (u64, 0-based worker index — only present with injected timing). The timing pair is stamped at the CLI sink on the worker thread (the sans-IO core leaves it absent), and powers the HTML report timeline (ADR-0015).

batch_started — a contiguous same-engine step batch was dispatched. scenario · engine (e.g. "hurl") · steps (count).

entry_running — live progress: one event per execution attempt of an artifact entry, retries included. scenario · engine · entry (0-based ordinal within the scenario’s artifact) · retry (0 = first attempt).

step_finished — scenario · engine · step ({file, line, text}, the authored feature anchor) · status (passed | failed | skipped | warned) · attempts (u32) · duration_ms (u64) · captures (capture names only — never values) · fragment (file.hurl#name of the named fragment the step ran, ADR-0018 — only present for a ref: step, so an inline hurl: block and every pre-fragment record omit the key entirely) · label (the pack step’s authored name:, resolved — only present when the step has one. One feature sentence commonly lowers to several engine steps, and they share a step anchor exactly; the label is what tells them apart, and it is the same text the emitter writes into the artifact’s entry comment) · detail (string, only present on failures/warnings/skips-with-reason) · attempt_details (array of strings — the messages from earlier, failed attempts of a step that ultimately passed; only present for a flaky pass, feeds JUnit <flakyFailure>) · reproduce_hint (string — the redacted curl of the failing request, the same line the console prints as reproduce:; only present on failed steps, absent in every pre-field stream, masked at the sink boundary like detail).

scenario_finished — scenario · file (feature path — with scenario, the run-wide identity: names are unique only within one file; absent in records that predate the field) · status · timestamp_ms (u64, run-relative end ms — only present with injected timing) · worker (u64, in the schema but never populated by proef’s own writer today — it is emitted from the main dispatcher thread, not the scenario’s worker, so consumers must accept the field without expecting it — ADR-0015).

env / metadata / shuffled — on run_started, all additive and absent when unset. env is the active --env/PROEF_ENV profile name; metadata is the explicit user-supplied map (--meta k=v, [meta], [env.<name>.meta] — proef never harvests: no git, no hostname, no CI env sniffing; ADR-0020); shuffled says the order was re-dealt (the permutation is seeded by run_id, so the pair reproduces it). Keys and values pass the sink-boundary mask like every text field.

rerun_of — on run_started, optional (additive; absent = not a rerun). The base record’s run_id a --rerun re-ran failures from. The rerun’s JUnit carries the base’s not-re-run suite scenarios as ordinary testcases, and report overlays the base for a whole-suite page — composition over records, the record files themselves never merge (ADR-0008). Totals and the exit code stay the rerun’s own (ADR-0014).

tags — on scenario_finished, optional (additive; absent when the scenario carries none and in every pre-field stream). The accumulated tags (feature → rule → scenario → examples, deduped, authored order, @ stripped). On the finished event only: the cancel-skip path emits no scenario_started, and per-tag skip counts need every scenario.

exclusive — on scenario_started, optional bool (absent = false, which is what every pre-field record meant). The scenario ran with the pool to itself ([run] exclusive-tags) — the bool the scheduler itself read, recorded for timeline post-mortems (R11-6).

reason — on scenario_finished, optional (additive; absent in pre-field records and on every non-skipped scenario). Why the scenario is skipped: an authored skip carries the pasteable tag spelling (@skip / @skip:reason), which always begins with @; a mechanical skip (cancellation) carries proef-fixed prose, which never does. --rerun keys its re-queue decision on exactly that split (ADR-0019).

phase — on scenario_started/scenario_finished, optional. "setup" or "teardown" when the scenario belongs to a [run] lifecycle phase, absent for an ordinary suite scenario. Consumers should use it rather than inferring phase membership from the feature path: run_finished’s totals exclude phases (ADR-0014), so a consumer that counts every scenario_finished will not match them. Added additively — records without it have no phases, which is what they had.

run_finished — tail. passed · failed · skipped — the main-suite verdict: scenario counts for the primary suite only (ADR-0014). [run] setup/teardown scenarios still appear as their own scenario_started/ scenario_finished events earlier in the stream, but are excluded from these totals, so they agree with the console summary: line, proef explain, proef report’s HTML headline, --format json, TAP, the SLA gate, and the exit code — JUnit agrees too except that a failed phase rides in as its own suite (see above) · cancelled (bool, only present when true).

Example stream

{"event":"run_started","schema":1,"run_id":"019f…"}
{"event":"scenario_started","scenario":"reference","file":"suite/case.feature","timestamp_ms":0,"worker":0}
{"event":"batch_started","scenario":"reference","engine":"hurl","steps":2}
{"event":"entry_running","scenario":"reference","engine":"hurl","entry":0,"retry":0}
{"event":"step_finished","scenario":"reference","engine":"hurl","step":{"file":"suite/case.feature","line":4,"text":"the cookie session is exercised"},"status":"passed","attempts":1,"duration_ms":12,"captures":[]}
{"event":"step_finished","scenario":"reference","engine":"hurl","step":{"file":"suite/case.feature","line":5,"text":"the response status is 200"},"status":"passed","attempts":1,"duration_ms":0,"captures":[]}
{"event":"scenario_finished","scenario":"reference","file":"suite/case.feature","status":"passed","timestamp_ms":12}
{"event":"run_finished","passed":1,"failed":0,"skipped":0}

Guarantees

• Secrets never appear. Redaction applies once at the sink boundary before any reporter (property-tested); captures carries names only.
• Order: events of one scenario are internally ordered; scenarios running in parallel interleave. Group by (scenario, file) before assuming sequence across the stream — scenario alone collides when two feature files reuse a name.
• Flake-safe assertions: assert on attempts counts and normalized event order, never wall-clock (duration_ms is engine-measured and varies).

Recipes

jq -r 'select(.event=="step_finished" and .status=="failed") | "\(.step.file):\(.step.line) \(.detail)"' events.jsonl
jq -r 'select(.event=="run_finished")' events.jsonl          # the suite verdict (setup/teardown excluded)
jq -r 'select(.event=="entry_running") | .retry' events.jsonl | sort | uniq -c   # retry pressure
# which fragment files a failing run actually exercised (ADR-0018); `// empty`
# drops the inline steps, which carry no `fragment` key at all
jq -r 'select(.event=="step_finished") | .fragment // empty' events.jsonl | sort | uniq -c

proef — Product Requirements Document

Status: approved; US-1…US-12 all in service (multi-engine architectural only), plus post-M5 work: external config/environments (ADR-0012), the v0.6–v0.8 correctness series, v0.9.0, named hurl fragments (ADR-0018 — see the §3 amendment), the adoption response, the 0.11–0.14 hardening and CI-scale series, the RF capability audit and reserved tags (ADR-0019/0020), and the deep-improvement waves plus round 19. CLAUDE.md’s Status block is the running ledger; the goals and non-goals below are what has not changed. · Date: 2026-07-28, status refreshed 2026-08-31 · Owner: Emre Companion docs: ADRs for the why, TECH-SPEC for the how, IMPLEMENTATION-PLAN for the when.

1. Problem & context

End-to-end tests written as Gherkin business prose — bound to a compiled-in vocabulary of YAML macro packs — let non-programmers author real tests. Meanwhile the backend team maintains a corpus of hand-written Hurl files as its API-testing lingua franca. There is no tool that joins the two: Gherkin prose on top, Hurl-grade API execution underneath — and none whose engine seam can later admit further engines behind the same prose (architectural readiness only).

proef is that tool: a declarative, modular, multi-engine e2e runner (mostly Rust). One core parses/binds/lowers Gherkin; pluggable engines execute. The API engine embeds hurl itself (ADR-0001), so API semantics are hurl’s by construction, and every scenario also produces real .hurl artifacts the backend team can run and read (ADR-0010).

2. Goals

G1. Author API e2e tests as pure Gherkin prose in the 500-series style — no code, no URLs in prose, data tables and Scenario Outlines supported. G2. Execute with hurl’s exact semantics (asserts, captures, retries, templating) via the embedded engine; identical results to the stock hurl CLI on the same artifacts. G3. Emit canonical .hurl artifacts + sidecar maps for every scenario — debuggable with the tools the backend team already uses, replayable via hurl --variables-file. G4. Be modular: adding an engine is a new crate + a registry line, with zero changes to proef-core (the structural acceptance test, ADR-0002). G5. Track upstream hurl safely over time: exact pins, a zero-diff fork as patch vehicle, and an upgrade-canary CI job so new hurl releases are absorbed deliberately (ADR-0003). G6. First-class failure UX: every failure maps to the .feature line (and the artifact span), rendered with miette-style labeled diagnostics; stable exit codes 0/1/2/3. G7. CI-native: JUnit XML, GitHub job summary, JSONL run records, tag filtering, parallel scenarios, --dry-run validation gate, and (M5) a libtest-mimic harness so cargo nextest run and IDEs can drive scenarios.

3. Non-goals (v1)

Further engines (designed-for, not built — M6+); a desktop dashboard or server mode (precedent exists when needed); generating Gherkin, macros, or prose from hand-written hurl files (see the amendment below); OpenTelemetry export (semconv still immature — JSONL is the source of truth); dynamic plugin loading; Windows-static or musl-static binaries (dynamically linked like hurl’s own — consequence of ADR-0001); API mocking/contract testing; load testing.

Amendment (2026-08-11): the hurl non-goal is about generation, not direction

This non-goal previously read “importing/round-tripping hand-written hurl files into Gherkin (artifacts flow outward only)”. The clause and its parenthetical said two different things, and the parenthetical was the broader of the two: read literally it forbids hurl text being an input at all, which ADR-0018 (named hurl fragments) needs it not to.

What the non-goal protects is that proef never authors a test for you. A suite’s prose and its binding vocabulary are written by people, deliberately; a tool that derives them from an existing corpus produces scenarios nobody chose the words for, and the review that makes a feature file worth having never happens. That reasoning is untouched, and ADR-0016 (OpenAPI generation) stays declined on the same ground.

It does not extend to hurl text being an input source. A macro pack is already an input written in a non-Gherkin language; a .hurl file naming reusable fragments is another, and §1’s own framing — “the backend team maintains a corpus of hand-written Hurl files as its API-testing lingua franca. There is no tool that joins the two” — describes joining that corpus as the product’s purpose, not as a boundary. Nothing is generated: features and macros stay hand-authored, and a fragment is inert until a macro names it. The worklist reached the same reading independently for a different item (OPEN-FINDINGS M2: “the non-goal forecloses a direction of data flow, not the ability to check your own work”).

Recorded honestly: M3 asked that this charter be re-examined with a measured port cost, and that measurement does not exist. The narrowing is therefore argued from the non-goal’s own rationale rather than from evidence that pasting is expensive. If the measurement later shows pasting is cheap, that argues about priority — it does not restore a prohibition this amendment finds was never the point.

4. Users & personas

P1 — Test author (QA, PM, support engineer; not necessarily a programmer). Writes .feature files against the existing step vocabulary. Cares about: prose that reads naturally, --dry-run telling them exactly which step is wrong, failures pointing at their line, not internals. P2 — Pack maintainer (developer). Owns the macro packs: adds steps, params, asserts. Cares about: pack readability (raw hurl blocks, ADR-0004), schema autocomplete, load-time validation with did-you-mean hints, safe refactors (duplicate/cycle detection). P3 — Backend engineer (owns the hurl corpus). Consumes emitted artifacts; pastes between corpus and packs, or — since ADR-0018 — annotates a corpus file once and lets packs name its entries, so the same file stays runnable under stock hurl. Cares about: artifacts being idiomatic hurl, never containing secrets, runnable standalone; and that proef reading their corpus never edits or reformats it. P4 — CI pipeline (machine). Cares about: stable exit codes, JUnit/JSONL outputs, deterministic behavior, bounded runtime (cancellation/budgets, ADR-0007), the canary job.

5. User stories & acceptance criteria

US-1 (P1) As a test author I write a scenario in prose and run it against a live environment. AC: the four 500-series .feature files run via proef packs with prose unchanged except agreed wording fixes; proef test runs them green against the fixture; failures name feature file + line + step text. US-2 (P1) I validate without executing. AC: proef test --dry-run binds every step, expands every macro/outline, resolves ${…}, parses every generated artifact with hurl’s parser, checks every file,…; asset those artifacts read is present, and exits 2 with labeled diagnostics on any failure — no network I/O. US-3 (P1) I pass data per step. AC: inline {captures}, | key | value | data tables, and Scenario Outline <placeholders> all fill macro params; conflicts and missing required params are parse-time errors naming the line. US-4 (P1) Steps chain state. AC: a capture in one step (clientId) is usable in later steps of the scenario as {{clientId}}; saveAs: global persists across scenarios and runs (World, ADR-0005). US-5 (P1) Slow backends don’t flake. AC: a step with retry: polls until its asserts pass or the finite budget ends (maps to hurl [Options] retry); optional: steps warn instead of failing. US-6 (P2) I extend the vocabulary. AC: adding a macro with a match: pattern + raw hurl block to a pack makes the new prose step available; proef schema reflects it; load rejects ambiguous names, adjacent captures, literal-free patterns, infinite retries. US-7 (P3) I get artifacts. AC: proef artifacts writes per-scenario .hurl + .map.json (+ .vars when Worlds are referenced); stock hurl --test yields the same verdicts (spike-proven); no secret values appear in any artifact. US-8 (P4) CI consumes results. AC: exit codes 0/1/2/3 stable and integration-tested; --junit auto under GITHUB_ACTIONS; JSONL event log written per run; --tags filters. US-9 (P1/P4) Runs are observable. AC: console BDD tree with per-step timing/attempts; proef explain [run] summarizes the latest failures from run records; proef diff [base] [new] compares two runs for regressions, fixes, flakiness, and perf deltas; proef report [run] writes a self-contained HTML report of a run. US-10 (P2) Secrets stay secret. AC: proef secret set stores encrypted values; secret values never appear in artifacts, logs, reports, or events (property-tested). US-11 (P4) hurl upgrades are safe. AC: the canary job builds against the next hurl release and replays the suite; pins move only after it is green (runbook in IMPLEMENTATION-PLAN §7). US-12 (P1, M5) IDE/nextest integration. AC: the libtest-mimic harness lists one test per scenario and cargo nextest run executes and reports them. US-13 (P1) I can start from something that works. AC: proef init writes a minimal suite (proef.toml, one .feature, one matching pack) that passes --dry-run unchanged, installs the pack JSON Schema for editor completion, and never overwrites an existing file.

6. Functional requirements (condensed; TECH-SPEC is normative)

Authoring: full gherkin-crate grammar (Feature/Rule/Background/Scenario/Outline/ Examples/tables/docstrings/tags/i18n); tags filter runs; variables come from proef.toml (${url:}/${vars:}, ADR-0012), never the feature files. Packs: YAML skeleton with match: patterns ({name} captures), params/defaults/tags/description; step bodies as raw hurl: blocks (primary) or structured form (reserved for future engines); assert-only expect: macros merge into the previous request (Then-steps); use:/with: nesting with cycle/depth limits; schemars-derived JSON Schema; lint pass at load. Execution: scenario = unit of isolation and parallelism (--jobs); contiguous same-engine batching; engine-hurl via embedded run_entries with buffered I/O; per-entry [Options] override batch defaults (verified); World seeding/merge-back; cooperative cancellation + budgets. Artifacts: canonical emit, sidecars, vars files, # optional markers. Reporting: event spine → console/JUnit/JSONL/GitHub-summary reporters; run-record rotation. CLI: test ([path] --env --dry-run --tags --jobs --junit --format json|tap --watch --run-id --sarif --rerun --scenario[-file]; path optional — [run] suite then the tests/ convention), flows, macros (call counts + dead-macro report), artifacts, schema [--add-to], secret set|list, explain, diff [base] [new] --fail-on-regression, report [run] -o <file>, doctor. Config (proef.toml, ADR-0012): runner settings ([run] incl. setup/teardown suite lifecycle features, ADR-0014; [http]/[sla]) + suite variables ([url]/[vars], referenced ${url:key}/${vars:key}) + per-environment overrides ([env.<name>]); precedence defaults < base tables < active [env.<name>] (via --env/PROEF_ENV) < flags. Secrets via PROEF_SECRET_<NAME> env override → the encrypted store (proef secret set — hidden prompt, or --stdin for scripts), never in proef.toml.

7. Non-functional requirements

Correctness/fidelity: API semantics are hurl’s own binary-identical engine — no reimplementation drift is possible (ADR-0001/0010). Portability: Linux (glibc), macOS, Windows; documented build prereqs (libcurl/libxml2/libclang), doctor-checked; dynamically-linked dist binaries (hurl’s own model). Performance: startup overhead (parse+bind+lower for a 30-scenario suite) under ~1 s; execution dominated by the network; scenario-level parallelism with worker threads. Reliability: deterministic lowering (injected clock/run-id, sans-IO-lite core); bounded runtime under cancellation budgets; state writes atomic (temp+rename). Security: secret redaction invariants property-tested; artifacts guaranteed secret-free; encrypted-at-rest secret store; 0600 on sensitive outputs. Maintainability: exact pins + canary + thin-fork policy; CI gates: fmt, clippy -D warnings, nextest, doc -D warnings, deny, machete, zizmor, docs-check, public-api (audit nightly). Extensibility: new engine = new crate implementing EngineFactory/EngineSession + one registry line; step-kind schema contributed by the engine; zero core changes (the M6 acceptance test).

8. Success metrics

M-1: the four 50x API features run under proef with prose unchanged (US-1) — the port-fidelity bar. M-2: artifact parity — stock hurl CLI verdicts match proef verdicts on every artifact in the integration suite (already spike-proven; kept as a CI invariant until M4, then by construction). M-3: --dry-run catches 100% of the seeded pack/feature error corpus with line-accurate diagnostics. M-4: one hurl upstream release absorbed via the canary runbook with zero suite regressions. M-5: a future non-hurl engine lands (M6) with git diff --stat proef-core empty.

9. Release phasing

v0.1 = M0–M3 (authoring, validation, embedded execution, artifacts, console+JSONL); v0.2 = M4 (upstream tracking hardened, JUnit/GitHub reporters); v0.3 = M5 (breadth: multipart/form/docstring bodies, watch, explain, libtest-mimic harness); v1.0 = stability declaration of pack schema + CLI + event schema; M6 engines version independently. Detailed task breakdown: IMPLEMENTATION-PLAN.md.

proef — Technical Specification

Status: normative · Date: 2026-07-28, current through ADR-0020 (run metadata) · decisions referenced as ADR-XXXX. Post-M5 work — external config and environments (ADR-0012), the v0.6–v0.8 correctness series, v0.9.0, fragments, and the RF-audit series (reserved tags ADR-0019, run metadata ADR-0020) — is specified here too; “M0–M5” described this file’s scope only until those landed. Verified upstream facts cite hurl master @ 03fcb84c (2026-07-27) as file:line.

1. System overview

 .feature files          macro packs (YAML + raw hurl blocks)
      │                        │
      ▼                        ▼
 ┌─────────────────────────────────────────┐
 │ proef-core (pure: no IO, no clock, no   │
 │ rand — values injected)                 │
 │  parse ─ bind ─ lower ─┬─ emit          │
 │  (gherkin) (matcher)   │  (.hurl +      │
 │                        │   sidecars)    │
 │  dispatch: contiguous  │                │
 │  same-engine batches ──┼── events ──────┼──► reporter stack
 └────────────┬───────────┴────────────────┘    (console/JUnit/JSONL/GH)
              ▼ Box<dyn EngineSession>
   ┌──────────────────┐  ┌──────────────────┐
   │ proef-engine-hurl│  │ future non-hurl  │
   │ parse_hurl_file +│  │ engine (seam-    │
   │ run_entries      │  │ ready, none      │
   └──────────────────┘  │ scheduled)       │
                         └──────────────────┘
        World (typed vars + persistent global store) threads through every batch

Scenario = unit of isolation, parallelism, retry, and artifact emission. The orchestrator (in core, driven by cli) owns threads, the token, the World, and the event stream.

2. Workspace

proef/
  rust-toolchain.toml      # channel "1.97.1" (policy: latest stable at its x.y.1), rustfmt+clippy
  Cargo.toml               # virtual workspace, resolver="3", workspace.{package,dependencies,lints}
  deny.toml  .config/nextest.toml  proef.toml.example
  .github/workflows/ci.yml           # fmt, clippy -D warnings, nextest, doc -D warnings,
                                     # deny, audit, canary (ADR-0003)
  xtask/                   # automation as Rust (fixture, canary, docs-check, public-api); just aliases
  crates/
    proef-core/            # engine-agnostic: gherkin parse, packs, binding, lowering, IR,
      helpers/             #   emit, dispatch, World/state, events, errors, reporters
    proef-engine-hurl/     # EngineFactory/EngineSession impl over embedded hurl
    proef-cli/             # bin `proef`: clap, registry assembly, miette rendering
    proef-fixture/         # dev-only: in-process synchronous fixture API server (ADR-0011)
    proef-harness/         # libtest-mimic bridge: one Trial per scenario (US-12)
    proef-lsp/             # language server: SourceProvider + collect-all analyze_suite over core
  tests/                   # .feature corpus + fixtures
  docs/                    # this corpus

Dependency rules: engines depend on core; core depends on no engine; cli depends on both and is the only miette user (ADR-0009). Engines sit behind cargo features in cli (engine-hurl default-on; any future non-hurl engine would be added the same way — none scheduled). Only proef-engine-hurl carries native build prereqs; proef-core is pure Rust. Lints/conventions: a strict workspace lints table verbatim (clippy all=warn + curated pedantic slice), publish = false at the workspace root — overridden to true by the four crates that publish (proef, proef-core, proef-engine-hurl, proef-lsp), MIT OR Apache-2.0.

3. Core domain types (sketches; signatures normative, field lists indicative)

#![allow(unused)]
fn main() {
// world.rs — typed variable scope (ADR-0005)
pub enum Value { String(String), Number(f64|i64…), Bool(bool), Null }   // mirrors hurl Value subset
pub struct World { scenario: BTreeMap<String, Value>,
                   global: GlobalStore /* .proef-state.json, atomic temp+rename save */ }

// step.rs — lowered, engine-agnostic
pub struct StepRef { pub file: Arc<str>, pub line: usize, pub text: Arc<str> }  // feature anchor
pub struct LoweredStep { pub step: StepRef, pub kind: StepKindId, pub payload: StepPayload,
                         pub optional: bool, pub when: Option<Guard>,
                         // `file.hurl#name` for a `ref:` step, None for an inline block
                         // (ADR-0018) — qualified at lowering so a record stands alone
                         pub fragment: Option<String> }   // retry travels as baked [Options]
pub enum StepPayload { HurlEntries(String /* lowered hurl text */),
                       MergedAsserts { lines: usize /* expect: rows own the appended assert lines */ },
                       Structured(serde_json::Value) }
pub struct StepBatch { pub index: usize /* scenario-wide ordinal */, pub engine: EngineId,
                       pub steps: Vec<LoweredStep> }

// engine.rs — the seam (ADR-0002); see ADR text for EngineFactory/EngineSession
pub struct StepKindSpec { pub prefix: &'static str, pub schema: &'static str /* JSON-Schema frag */,
                          pub validate: Option<fn(&str) -> Result<(), PayloadProbeError>>,
                          // fragment files (ADR-0018): one Option, so extension and reader
                          // cannot disagree; discovery asks for the extension, never names one
                          pub fragments: Option<FragmentSupport>,
                          // raw [Options] keys → the core's budget policy, so option spellings
                          // live only in the engine that owns them (ADR-0007)
                          pub options: Option<fn(&str) -> Option<RawOption>>,
                          // which files a lowered payload sends, so the emitter records what an
                          // artifact reads without knowing the engine's body grammar (ADR-0002
                          // amendment, 2026-09-10 correction)
                          pub assets: Option<fn(&str) -> Vec<String>> }
pub struct FragmentSupport { pub ext: &'static str /* "hurl" */, pub scan: FragmentScanner,
                             // "which variables does one template value read", answered by the
                             // engine's parser — a hurl function ({{newUuid}}) is not a variable
                             pub template_reads: fn(&str) -> Vec<String> }
pub type FragmentScanner = fn(&str) -> Result<ScannedFile, FragmentScanError>;
// ScannedFile = { fragments: Vec<ScannedFragment>, unannotated: Vec<usize> } — the
// unannotated entry lines feed `proef fragments`' listing, not the pack loader
// Everything here is *read* from the entry — nothing is declared twice, so nothing can drift.
// A scanner reports ONLY annotated entries: an unannotated one is not a fragment, and a
// foreign corpus is mostly those, so building them only to be discarded is the bulk of a scan
pub struct ScannedFragment { pub name: String /* from `# @proef <name>` */, pub text: String,
                             pub line: usize, pub placeholders: Vec<String> /* reads */,
                             pub declared_options: Vec<String> /* ⊆ OPTION_FAMILIES */,
                             // `[Options] variable:` — what the entry supplies to itself.
                             // Answers its own placeholders (so the file runs standalone)
                             // and clashes name-to-name with a `bind:` of that name; kept
                             // apart from declared_options, which clashes family-to-family
                             pub supplied_variables: Vec<String> }
pub struct FragmentScanError { pub line: usize, pub column: usize, pub message: String }
// The vocabulary `declared_options` must use: matched by string equality against the pack's
// own option keys, so an engine-only spelling silences `option_declared_twice` rather than
// firing it. `MacroStep::declared_options()` derives the other half of that comparison.
pub const OPTION_FAMILIES: &[&str] = &["retry", "delay"];
// The one place `secret_bindings` (variable → secret) is joined with `secrets` (name → value).
// Yields borrows: an owned map would copy every secret value per scenario (ADR-0005)
pub fn secret_variables<'a>(bindings: &'a BTreeMap<String, String>,
                            secrets: &'a BTreeMap<String, String>)
                            -> impl Iterator<Item = (&'a str, &'a str)>;
pub struct DoctorCheck { pub name: &'static str, pub run: fn() -> DoctorResult }
pub struct BatchResult { pub steps: Vec<StepOutcome>, pub error: Option<EngineError> }
// `fragment` is carried here as well as on the event: JUnit, the job summary, the
// annotations, TAP and the console are built from RunSummary after the event stream has
// been written out, so they cannot read it back. Both are copies of one lowering-time
// source, so they cannot drift from each other.
pub struct StepOutcome { pub step: StepRef, pub status: Status, pub attempts: u32,
                         pub duration: Duration, pub detail: Option<String>,
                         pub attempt_details: Vec<String>, pub reproduce_hint: Option<String>,
                         pub fragment: Option<String> }

// events.rs — the spine (ADR-0008); serde, versioned
#[serde(tag = "event", rename_all = "snake_case")]
pub enum Event { RunStarted{..}, ScenarioStarted{..}, BatchStarted{..}, EntryRunning{..},
                 StepFinished{..}, ScenarioFinished{..}, RunFinished{.., cancelled} }
pub struct EventSink(Arc<dyn Fn(&Event) + Send + Sync>);   // borrowed events
}

4. Pipeline (all in proef-core; pure — inputs include injected run_id, now, env snapshot)

4.1 Load packs. The fragment corpus ([run] fragments) is read once per command into a pack::FragmentCorpus and scanned at most once, lazily — only when some pack actually carries a ref:, which is what makes “pointing at a corpus you did not write costs nothing” true of the scan (CONFIG.md). One proef test loads packs up to four times (the suite, then [run] setup/teardown, each validated and then run) against the same corpus, so the memo belongs with the corpus rather than the caller — a caller that scanned eagerly to share the result would trade the promise for the speed. Discover embedded helpers/ + project packs/; serde_norway with deny_unknown_fields; validation passes: (1) match: guard rails — must contain literal text, no adjacent captures, unclosed braces rejected; (2) params/defaults coverage; (3) duplicate macro names across packs → error (qualify pack.yaml#name); (4) use: cycle + depth ≤ 32; (5) unknown with: keys → “did you mean” (edit distance); (6) finite-retry lint — retry: requires a finite count (ADR-0007); (7) hurl blocks: lower a probe instantiation with placeholder params and parse_hurl_file it — syntax errors reported with block-relative spans mapped to pack file/line; (8) engine kinds: every step kind must be claimed by a registered engine’s StepKindSpec.

Fragments (ADR-0018) add five: (9) a ref: must name a loaded fragment → “did you mean” over the scanned names, or a pointer at [run] fragments when none loaded; (10) duplicate fragment name across files → error (qualify file.hurl#name, the same suffix matching pack.yaml#name uses); (11) a step is ref: xor a payload/use:; (12) an option family declared both in the fragment’s own [Options] and as the step’s YAML key → error (pass 6’s twinned-option rule, applied across the file boundary), and the same rule name-to-name for a variable both supplied by the fragment’s [Options] variable: and given by a bind: — the pair reaches one entry as two variable: k= lines, where hurl’s last-wins would drop the bound value into every later entry too; (13) a fragment file the engine’s FragmentSupport::scan could not read, or an annotation it could not attach, positioned in the .hurl file itself.

Fragments skip pass 7: they parse as authored, so the probe instantiation has nothing to guess. Whether every {{…}} a fragment reads is actually supplied — by a bind: in scope, by an earlier step’s [Captures], or by the fragment’s own [Options] variable: — is checked at lower time (§4.4, proef::lower::unbound_placeholder) — only lowering knows what the preceding steps captured, and a load-time half-check would be worse than one complete one. The self-supplied source is scoped to the fragment that declares it: a name another entry left in hurl’s shared set is the implicit inheritance this check exists to refuse, so only [Captures] carries a value forward.

4.2 Parse features. gherkin 0.16 (Feature::parse); tags from Feature/Rule/Scenario accumulate. Localized (# language:) features are supported and test-covered: the crate strips the dialect keywords, proef consumes the stripped step text, and a localized outline with Examples expands like any other (outline detection keys on Examples presence, which is dialect-independent). Caveat: a localized outline whose Examples block is omitted cannot be told apart from a plain scenario (the crate’s dialect keywords are private), so it surfaces as an unbound-step error rather than the crisp no_examples — a worse message, never a silent pass. # comment lines are plain gherkin comments (no # key: directive mechanism — variables live in proef.toml, ADR-0012).

4.3 Bind. For each step (keyword stripped): first macro whose match: pattern matches wins; ambiguity (2+ matches) is an error listing candidates. Captured {name} values: trimmed, surrounding quotes shed (quotes preserve inner spaces/commas). Data table rows | key | value | merge into args; key set by both capture and table → error. Defaults fill; missing required params → error. Unbound step → exit-2 error with closest-pattern suggestion (edit distance over pattern literals).

4.4 Lower. Outline expansion (parser does not do it): per Examples row, substitute <col> in scenario name, step text, docstrings, table cells; ragged rows / unknown placeholders → parse-time error with line. Background steps prepend to every scenario. Macro expansion: params bound, ${…} resolved recursively, depth ≤ 8 (captured args may contain ${…} — spike-verified necessity); $${ escapes; {{…}} passes through untouched. Assert-only macros (expect:) merge into the previous request entry — error if none (Then-before-When). Product: Vec<StepBatch> per scenario (contiguous same-engine runs; batch maximally — split only at optional: boundaries and engine changes, ADR-0010).

4.5 Emit. Canonical .hurl per scenario (stable formatting, snapshot-tested): header comment per entry # <file>:<line> — <step text>; # optional markers; sidecar <slug>.map.json (schema: { entries: [{hurl_lines: [a,b], feature: {file,line,text}, optional, captures: [names], batch: n, step: n}], schema: 1 }); <slug>.vars when ${global:} or ${secret:} referenced (secrets as names only). Artifact dirs: .proef-runs/<id>/ artifacts/ (per-run) and proef artifacts -o <dir> (stable hand-off).

4.6 Dispatch. Per scenario thread: check token → factory.open(ctx) lazily per engine on first batch → session.run_batch(batch, world, events, token) in order → merge outcomes/World → finish() all sessions (reverse open order) → emit events. optional: batch failure → warnings + continue; else fail-fast within the scenario.

5. proef-engine-hurl internals (verified seam facts inline)

Adapter. Per batch: seed VariableSet from World (insert; secrets via insert_secret) → parse_hurl_file(&batch_text) → run_entries(&file.entries, &batch_text, Some(&input), &runner_options, &variables, &mut stdout_buf, Some(&listener), &mut logger) with WriteMode::Buffered terms (upstream’s own threading mode: parallel/worker.rs:76,124-133) → map EntryResults to StepOutcomes via the sidecar (SourceInfo spans → feature lines) → merge HurlResult.variables back into the World (typed).

RunnerOptions mapping. Batch-level RunnerOptionsBuilder from config; per-entry [Options] override batch defaults by clone-then-override (runner/options.rs:43-58), variable= inserts persist for the rest of the call — verified semantics, relied upon.

HttpDefaults carries what [http]/[env.<name>.http] express: timeout-ms, follow-location, max-redirs, insecure, proxy/no-proxy, cacert, client-cert/client-key, user-agent, cookie-store. Each reaches the builder only when the project set it, so an unset key leaves hurl’s own default rather than this engine restating a constant that could drift from upstream on the next pin bump. Path-valued fields arrive already resolved — the CLI applies the one-path rule and core touches no filesystem (ADR-0012). Two keys are exceptions to the per-entry override rule above, verified against OptionKind rather than inferred: it has no UserAgent variant (run-wide; an entry opts out with a User-Agent: request header instead) and no cookie variant at all, so cookie-store is run-wide with no per-entry spelling whatsoever.

cookie-store = false maps to use_cookie_store(false) (hurl’s --no-cookie-store, 8.0.0). Verified: hurl enables curl’s cookie engine only under use_cookie_store (http/client.rs:390), which also means a cookie_input_file handed over with the store off is silently ignored — so the engine skips both halves of the §5 round-trip when the store is off. hurl’s own FIXME there (a handle once given cookie storage cannot have it removed) never reaches proef: the client is per-call (below), so a handle never transitions on → off.

Client lifetime (verified). run_entries constructs http::Client::new() internally per call (runner/hurl_file.rs:169) — fresh libcurl handle: connection cache and cookie jar do not survive across calls. Consequences implemented: batch maximally (§4.4); on forced splits, chain variables via HurlResult.variables (lossless) and, when cookies are in play, round-trip HurlResult.cookie_store → CookieStore::to_netscape() → temp file → next batch’s RunnerOptionsBuilder::cookie_input_file (http/cookie_store.rs:66-72, runner_options.rs:242-244) behind a SessionState struct. Upstream patch #1 (ADR-0003): run_entries(&mut Client) — two internal call sites (hurl_file.rs:124, worker.rs:124); adopt when accepted, delete SessionState cookie path.

Thread-safety (verified). No global mutable state in hurl (static mut/ lazy_static/OnceLock-mutable: zero hits); libcurl init is Once-guarded pre-main by the curl crate; sole FFI global write is libxml2’s error handler set idempotently per XPath eval — exercised concurrently by upstream itself. Scenario-per-thread is safe.

Cancellation & budgets (ADR-0007). No interrupt support exists upstream (verified); engine computes batch budget = Σ(timeout × (retry+1)) + intervals + margin, clamped to a four-hour ceiling (MAX_BATCH_BUDGET — lint-clean values still compose into an unbounded product, ADR-0007 amendment); watchdog abandons over-budget scenario threads; token checked between batches only.

Failure detail. Engine errors surface through hurl’s own DisplaySourceError::description into StepFinished.detail (additive event field), the console, JUnit, and the GitHub summary.

6. Pack schema v1 (normative field reference)

bind:                         # pack-scope fragment bindings (ADR-0018); macro and
  <var>: "${…}"               # step scope override, most specific winning
macros:
  <macroName>:                # unique across packs; qualify as pack.yaml#name on clash
    params: [a, b]            # required unless defaulted
    defaults: { b: "x" }      # optional params
    match: "…{a}…"            # step-definition pattern; omit → not Gherkin-reachable
    description: "…"          # docs + desktop palette later
    tags: [Domain]
    steps:                    # request steps (each lowers to ≥1 hurl entry)
      - name: "…"             # entry label (events/console)
        optional: true|false  # failure → warning (segments the batch)
        when: "${expr}"       # skip guard: skips when empty or literal false/0 after resolution
        retry: { count: N, interval_ms: M }   # finite only (lint); → [Options] retry
        saveAs: { captureName: global }        # promote capture(s) into the World
                              # (refused with a warning if the value equals a secret)
        bind: { <var>: "…" }  # step-scope fragment bindings (only with `ref:`)
        hurl: |               # PRIMARY form (ADR-0004): raw hurl, ${…} lowered first,
          …                   # {{…}} left for run time; validated by parse_hurl_file
        # OR structured payload (reserved for future non-hurl engines):
        # <kind>: { … }  (a future engine's structured payload)
      - ref: file.hurl#name   # ALTERNATE form (ADR-0018): one `# @proef <name>` entry
                              # of a scanned fragment file; `{{…}}` supplied by bind:,
                              # every one bound, captured by an earlier step, or set
                              # by the fragment's own [Options] variable: (which then
                              # may not also be bound — option_declared_twice)
      - use: pack.yaml#other  # composition, with:/inline args; cycle+depth checked
        with: { a: "${a}" }
    expect:                   # assert-only macro (Then-steps): merges into previous entry
      - status: "${status}"   # or raw hurl assert lines: hurl: |‐style fragment (M5)

JSON Schema is schemars-derived from these serde types plus engine-contributed StepKindSpec fragments; proef schema --add-to injects the editor modeline (a proven mechanism).

7. Gherkin mapping reference

One scenario = one flow/run-record/artifact set. Background: prepends. Rule: groups pass through (tags accumulate). Outline/Examples per §4.4. Data tables per §4.3. Docstrings: reserved for raw request bodies in generic steps (M5). Tags: @tag → flow tags, --tags filters by a boolean expression (and/or/not/parens, proef_core::tags); atoms may glob — * any run, ? one char, anchored, a metachar-free atom staying literal equality — and the one matcher serves --tags, [run] exclusive-tags and [tag-links]. @skip/@skip:reason and @quarantine are reserved tags carrying behavior, recognized at the CLI edge — core never reads tags (ADR-0019, ADR-0014). Step keywords: And/But resolve to the previous primary keyword (gherkin crate StepType); keyword itself is not matched against patterns.

8. Variables reference (ADR-0005)

Author-time (${…}, resolved in §4.4, recursive ≤ 8): ${param} · ${env:NAME} / ${env:NAME:-default} · ${url:key} / ${vars:key} (proef.toml [url]/[vars], base + active [env.<name>] deep-merged; injected — ADR-0012) · ${run:id} (uuid-v7-derived, injected) · ${global:key} (World read at lower time of the scenario) · ${secret:NAME} (encrypted store; emits {{secret_name}} + insert_secret) · ${fake:kind} (deterministic from run id and an occurrence index; the index is an incrementing counter scoped to one scenario — shared across every ${fake:…} resolve in it, so independent references never collide regardless of how many a step resolves; a step’s name: label resolves from a rewound copy of the counter so its Nth ${fake:…} reference reuses the payload’s/when:’s Nth occurrence by position, not generator kind — reproducing the payload’s own value exactly when the label’s references mirror the payload’s in kind and order, otherwise surfacing that occurrence’s own-kind value instead — then restores the real counter to the high-water mark the replay reached (never below it), so an extra fake the label alone introduces still reserves its slot and is never reissued; the counter resets to zero at the next scenario, so the same generator at the same position in two different scenarios currently coincides; port deterministic NL generators) · $${…} literal escape. Run-time ({{…}}): hurl captures and secret placeholders — resolved by the engine. Resolution order within a scope: step args

macro defaults.

9. Diagnostics (ADR-0009)

gherkin Span = 0-based byte offsets, end-exclusive → SourceSpan::new(start, end-start); parser appends a trailing newline when missing — attach the normalized source text to diagnostics (or clamp); never use LineCol.column (char-counted) in byte math. Pack YAML: serde_norway error locations; schema-path → YAML-location pass for lint findings. Engine failures render: feature line + step text + assert detail + artifact path:span (from sidecar). Every diagnostic carries a stable code (proef::pack::adjacent_captures, …) for greppability.

10. CLI reference (v1)

proef [--config PATH] [--env NAME] <command>   # global: the proef.toml to read, the [env.<name>] profile
proef init [dir]
proef test [file|dir] [--env NAME] [--dry-run] [--tags EXPR] [--jobs N] [--junit path|auto]
                      [--format json|tap] [--watch] [--scenario NAME] [--scenario-file FILE]
                      [--run-id ID] [--rerun] [--sarif PATH (with --dry-run)] [--max-fail N] [--shard I/N] [--shuffle] [--meta KEY=VALUE]... [--console MODE]
proef flows [file|dir] [--env NAME] [--format json]
proef macros [file|dir] [--env NAME] [--format json]
proef fragments [file|dir] [--env NAME] [--format json] [--check [--require-annotated]]
proef artifacts [file|dir] -o DIR [--env NAME] [--run-id ID]
proef schema [--add-to FILE…]  proef secret set|list|rm
proef explain [run-id] [--format json]          proef doctor [--format json]
proef diff [base] [new] [--fail-on-regression] [--format json]   # each side: run id, record dir, or events .jsonl
proef flaky [--format json] [--by KEY] [--min-samples N] [--recovery-runs N] [--outage-rate RATE]
                      # verdicts over the retained run history, keyed by the input fingerprint
proef report [run-id] [-o FILE]
proef fmt <file|dir> [--check]
proef lsp

A path-less test/flows/artifacts resolves [run] suite then the tests/ convention (else exit 2). Exit codes: 0 ok · 1 test failure · 2 user error · 3 system error (typed enum, assert_cmd-pinned). Ctrl-C, SIGTERM and SIGHUP all take the graceful cancel (ctrlc’s termination feature — a CI job timeout is a cancellation, not a kill); a second signal while a test/watch run is cancelling forces an immediate hard exit with code 130 (128+SIGINT, the shell convention; the handler carries no signal identity, so every second signal shares the code) — deliberately outside the graceful 0/1/2/3 taxonomy, so it is not an ExitCode variant. Config precedence: built-in defaults < proef.toml base tables < active [env.<name>] (selected by --env/PROEF_ENV) < flags; suite variables ${url:key}/${vars:key} resolve from [url]/[vars] deep-merged with the active env (ADR-0012). Secrets additionally resolve PROEF_SECRET_<NAME> env overrides before the store, and PROEF_KEY (base64) overrides the key file — CI decrypts a committed ciphertext store without the key ever touching disk. --dry-run = §4.1–4.5 including artifact parse-validation; no engine sessions, no network.

11. State & files

.proef-runs/<run-id>/ → events.jsonl (the record, ADR-0008), run.log (console tee), artifacts/*.hurl|.map.json|.vars (+ artifacts/assets/<slug>/, the staged file bodies), timings.json (per-scenario durations, read back by --shard-weights) and inputs.json (the input fingerprint, proef flaky’s equivalence class) — both derived sidecars, never a second record — report.html, report.junit.xml (when requested); [run] keep-runs-bounded rotation, default 200 (only uuid-named run records rotate; the in-flight run never does). .proef-state.json — persistent World: atomic temp+rename, 0600. proef.toml — project config: runner settings ([run] jobs/runs-dir/keep-runs/suite/fragments/setup/teardown/exclusive-tags, [http] timeouts, [sla] ceilings, [flaky] verdict thresholds) + suite variables ([url]/[vars]) + run metadata ([meta], ADR-0020) + report tag links ([tag-links]) + per-environment overrides ([env.<name>]); see docs/CONFIG.md, ADR-0012.

One path rule: a path written in proef.toml resolves against the directory holding proef.toml; a path typed on the command line resolves against the working directory. Absolute values are taken as written, and with no config in scope written paths stay relative to the working directory. This covers suite, fragments, setup, teardown, runs-dir, .proef-state.json and .proef-secrets.json — so a project is where its config is, not where the shell is, and running from a subdirectory reaches the same suite, records, World and secrets as running from the root.

And its naming dual: a path that reaches an artifact, sidecar, event, report or diagnostic is spelled relative to that same directory (front::SourceNaming). Resolution makes a written path absolute; naming makes it project-relative again, so nothing proef records names the machine that produced it — the four ways to point at one suite (derived, typed, typed absolute, from a subdirectory) emit one artifact byte-for-byte, which is what ADR-0010’s same-bytes contract means across machines. A path that arrives relative is recorded exactly as it arrived; one that lies outside the project keeps its absolute name, there being no project-relative spelling of it. DiskSourceProvider (proef lsp) is deliberately outside this: it keys document identity on absolute names.

12. Parallelism & cancellation

Scenario-per-OS-thread, --jobs bounded (default: available_parallelism, capped by scenario count); events funneled through the sink (the console reporter buffers per scenario and replays contiguously); one CancellationToken per run, child per scenario; Ctrl-C graceful / second Ctrl-C hard (ADR-0007). Global-World writes serialize through the store lock; scenario ordering within a file is preserved for artifact naming, not execution order — except that a scenario matching [run] exclusive-tags waits for the pool to drain and runs alone, which constrains concurrency rather than reordering the queue.

13. Security

Secrets: encrypted at rest (chacha20poly1305), surfaced only via insert_secret; redaction invariants (never in artifacts/events/reports/logs) property-tested; a saveAs: global capture whose value equals a known secret is refused (warned) — .proef-state.json is plaintext and never receives secret-derived material; sensitive files 0600; proef doctor reports store/key health. context_dir confines file bodies (hurl’s own sandbox option) to the scenario’s staged asset root — a directory holding only what proef put there, so the sandbox is narrower than the suite it used to be. Assets are staged from beside the source that referenced them (feature for inline, fragment for ref:), which is hurl’s own per-file rule; references must be plain relative paths, and one name may come from only one source per scenario. No telemetry.

14. Dependencies (exact at M0; policy: latest stable at adoption, Renovate-managed)

Engine: hurl =8.0.1, hurl_core =8.0.1 (--locked; ADR-0003). Core: gherkin 0.16, serde 1, serde_json 1, serde_norway 0.9, schemars 1, thiserror 2, tokio-util 0.7 (default-features = false; CancellationToken only). CLI: clap 4 (derive, env), miette 7 (fancy), uuid 1 (v7), notify =8.2.0, ctrlc, chacha20poly1305 rpassword base64, quick-junit, toml. LSP: lsp-server 0.7, gen-lsp-types 0.11 aliased as lsp-types with its url feature (proef-lsp’s stdio transport, wired into proef lsp; ADR-0017 amendment). Fixture/harness (dev): tiny_http (ADR-0011 — axum conflicts with the tokio-runtime ban), libtest-mimic. Engine runtime: tempfile (Netscape cookie round-trip between batches, §5). Dev: insta assert_cmd predicates proptest tempfile quick-xml + cargo-fuzz targets; openssl-sys rides as the engine’s vendored-openssl feature carrier. Synthetic data (${fake:*}) is a dependency-free SplitMix64/FNV implementation in-core — the fake crate was not needed. Datetime uses jiff, never chrono, in our own code (hurl’s internal chrono is its business) — currently jiff is a dev-only dependency of the fixture’s /health identity; the sans-IO core still reads no clock (injected timestamps only). Banned: serde_yaml/serde_yml, chrono (ours), reqwest (superseded), maybe-async, async-trait (v1). Build prereqs (doctor-checked): Debian build-essential pkg-config libssl-dev libcurl4-openssl-dev libxml2-dev libclang-dev; macOS: Xcode CLT.

15. Conventions

Toolchain pinned to latest stable adopted at its x.y.1 point release — ~3-4 weeks after x.y.0, so a new minor waits out its first patch (1.97.1 at writing; tools and third-party crates track latest immediately, and exact pins like hurl outrank everything). rust-toolchain.toml cites this section as its authority, so the policy is stated here rather than only in RELEASING.md, where the correction first landed (R18-2): an unwritten policy contradicting the written one is a docs defect, and fixing it in two files while the normative spec still said “always latest stable” left the contradiction in the source of truth. Edition 2024; resolver 3; workspace lints (the legacy suite table); CI gates (PR): cargo fmt --check, clippy --all-targets --all-features -D warnings, cargo nextest run, cargo test --doc, RUSTDOCFLAGS="-D warnings" cargo doc, cargo deny check, cargo machete, zizmor, xtask docs-check, proef doctor, fuzz smoke + xtask public-api (nightly rustdoc); cargo audit runs on the nightly schedule. Automation in xtask (+ just aliases); no shell scripts for logic.

proef — Implementation Plan

Status: M0–M5 delivered, and everything after them — external config and environments (ADR-0012), the v0.6–v0.8 correctness series, v0.9.0, and named hurl fragments (ADR-0018). M6 (future engines) remains unscheduled. CLAUDE.md’s Status block is the running ledger; the milestone sections below describe the plan as it was executed, not the current frontier. · Date: 2026-07-28, status refreshed 2026-08-12. Normative design: TECH-SPEC; decisions: ADRs. Sizes are t-shirt (S ≈ days, M ≈ small weeks, L ≈ multi-week) — deliberately not fake-precise.

0. Guiding principles

The porting rule: when the prior spike proved a feature, understand the feature and implement it the cleanest way in this architecture — no copy-paste, no compatibility with spike code. Every milestone lands green on all CI gates. Core stays pure (no IO/clock/rand — TECH-SPEC §4). Nothing merges without its tests (TESTING-STRATEGY). The spike (research/) is evidence, not a starting codebase.

Global definition of done (every milestone)

fmt, clippy -D warnings, nextest, doctests, rustdoc -D warnings, deny, audit all green · new behavior has unit + (where applicable) snapshot/property tests · no unwrap/ expect in library code paths (CLI main excepted) · public items documented · CHANGELOG entry · exit codes unbroken (assert_cmd suite).

M0 — Foundations (size M)

Objective: compilable, gated, empty-but-shaped workspace with the seam in place.

Tasks:

1. Init repo proef/; virtual workspace (resolver 3), workspace.package + workspace.dependencies + workspace.lints; rust-toolchain.toml pinned to current stable (1.97.1 at writing); deny.toml, nextest config, README.
2. Crates: proef-core, proef-engine-hurl (empty adapter), proef-cli (clap skeleton), xtask (+ justfile aliases). Reserve crates.io names (0.0.0 placeholders, publish = false locally thereafter).
3. Core scaffolding: ExitCode enum + CoreError/EngineError taxonomy (ADR-0009); Event enum v1 + EventSink (ADR-0008); World/Value types + GlobalStore (atomic temp+rename) (ADR-0005); EngineFactory/EngineSession/StepKindSpec/ DoctorCheck traits (ADR-0002); CancellationToken plumbing (ADR-0007).
4. CI: gates workflow + a stub canary job (builds proef-engine-hurl against hurl =8.0.1 — becomes the real canary in M4); Renovate config (pins grouped; hurl excluded from auto-bump).
5. proef-cli: doctor (native-lib checks from engine doctor() — first proof the capability hook works), --version, exit-code integration tests.

Acceptance: workspace builds on Linux+macOS CI; proef doctor reports libcurl/ libxml2 status via the engine-contributed check; assert_cmd pins exit codes; all gates green. Proves: ADR-0002 seam compiles and registry assembly works.

M1 — Front end: packs, binding, Gherkin, lowering (size L)

Objective: .feature + packs → validated, lowered scenarios; --dry-run without artifacts.

Tasks:

1. Pack model (serde_norway, deny_unknown_fields) incl. hurl: raw blocks, expect:, use:/with:, when:, retry:, saveAs:; schemars derivation + engine StepKindSpec fragment merge; proef schema [--add-to].
2. Validation passes 1–8 (TECH-SPEC §4.1) with miette diagnostics + stable codes; finite-retry lint; probe-instantiation parse of hurl blocks via hurl_core.
3. Matcher: {name} tokenizer + leftmost matcher + guard rails (cucumber-expression semantics; property tests: no-panic, quote round-trip, adjacent-capture rejection).
4. Gherkin: parse, tags, Background, Rule pass-through, outline expansion, data-table merge; binding with ambiguity detection + closest-pattern suggestions.
5. Lowering: macro expansion (cycle/depth), recursive ${…} resolver (depth 8, $${ escape; property + fuzz targets), Then-merge (expect: → previous entry), batch segmentation (maximal; splits at optional:/engine change).
6. proef flows, proef test --dry-run (no emit yet), corpus test over tests/.

Acceptance: the four 500-series features (+ seeded error corpus) dry-run with line-accurate diagnostics; property/fuzz targets in CI (fuzz smoke = N seconds, full = nightly). Proves: PRD US-2/3/6 front-end half.

M2 — IR, emitter, artifacts (size M)

Objective: lowered scenarios → canonical .hurl + sidecars; --dry-run complete.

Tasks:

1. Canonical emitter (stable formatting rules) + # optional markers + per-entry feature-ref comments; line-map construction.
2. Sidecar <slug>.map.json (schema v1) + <slug>.vars; proef artifacts -o DIR; per-run layout under .proef-runs/<id>/artifacts/.
3. --dry-run gains artifact parse-validation (parse_hurl_file on every emitted file — the real parser as the validator).
4. insta snapshot suite: features+packs → artifacts + sidecars (golden corpus).

Acceptance: spike parity — the 500-series features emit artifacts that stock hurl parses (checked in CI via the canary toolchain image); snapshots reviewed. Proves: ADR-0010 emit half; US-7 static half.

M3 — engine-hurl: embedded execution (size L)

Objective: proef test runs scenarios end to end via embedded hurl.

Tasks:

1. Adapter: VariableSet seeding (World + secrets via insert_secret), parse_hurl_file + run_entries with Buffered terms + EventListener → events; EntryResult → StepOutcome mapping via sidecar spans.
2. RunnerOptions mapping (config → builder; per-entry [Options] override relied on as verified); HurlResult.variables merge-back; saveAs: global promotion; typed Value bridging.
3. Segmentation runtime: optional: warn-and-continue; SessionState cookie round-trip (Netscape temp file) for split scenarios; variables chaining.
4. Parallelism: scenario threads + --jobs; budgets + watchdog + token checks; Ctrl-C graceful/hard paths.
5. Reporters v1: console BDD tree (attempts/timings), JSONL event record, run-record rotation; --output json.
6. Fixture server (tiny_http, ADR-0011) + integration suite: success/4xx/retry-delay/auth/ malformed-JSON/optional/World-chaining/cancellation-budget cases.

Acceptance: the 500-series features run green against the fixture with prose unchanged (US-1); failure demo maps to feature line + artifact span; exit codes correct under pass/fail/user-error/system-error; cancellation bounded-time test passes. Proves: ADR-0001/0005/0007 runtime; US-1/4/5/9.

M4 — Upstream tracking hardened + CI reporters (size S–M)

Objective: riding upstream is a runbook, not a risk; CI outputs complete.

Tasks:

1. Canary job real: build+test against next hurl release (scheduled + on release); failure opens an issue with the diff of HurlResult behavior.
2. Thin-fork rehearsal: apply a scratch one-commit patch via [patch."crates-io"] from the fork tag, build, revert — documents the mechanics; draft upstream PR #1 (run_entries(&mut Client)) from the verified two-call-site change.
3. JUnit (quick-junit) + GitHub job summary reporters (--junit auto under GITHUB_ACTIONS); reporter-stack decorators (Normalize/Summarize) formalized.

Acceptance: canary catches an artificially-pinned older/newer hurl mismatch in rehearsal; JUnit consumed by CI UI. Proves: ADR-0003; US-8/11; M-4 metric path.

M5 — Breadth + integrations (size M)

Objective: the conveniences that make proef the daily tool.

Tasks: multipart/form/docstring bodies through packs; expect: raw-hurl assert fragments; more [Options] exposure (delays, location); --watch (notify 8.2.0); proef explain; proef secret set|list (encrypted store port); ${fake:*} NL generators; libtest-mimic harness (one Trial per scenario; nextest contract: --list --format terse, --exact --nocapture) + docs for IDE use; proef fmt for pack hurl blocks.

Acceptance: US-10/12 green; nextest runs the suite; watch-mode demo. Proves: ADR-0008 harness leg; PRD v0.3 scope.

M6 — More engines (future; sized when scheduled)

A future non-hurl engine — its step vocabulary carried as structured step payloads rather than raw hurl. Structural acceptance test: git diff --stat proef-core is empty. Mixed-engine 500-series suites become runnable. (Note 2026-07-29: a driver bringing its own async runtime would conflict with the tokio-runtime ban — pick a sync driver at sizing time. The core’s structured-payload paths are already exercised by tests; no core work is expected.)

5. Sequencing & parallelization

M0 → M1 → M2 → M3 strictly ordered (each consumes the previous stage’s types). Within M1, tasks 1–3 parallelize with 4; M2.4 can start as soon as M2.1 emits. M4.3 reporters can start during M3.5. Docs (GETTING-STARTED + AUTHORING, delivered 2026-07-29) draft during M2–3 and finalize at M5 (deliberately after the schema stabilizes).

6. Risk register

7. Runbook — absorbing a new hurl release

1. Canary red/green report arrives (scheduled job). 2. Read upstream CHANGELOG diff.
2. Bump pins on a branch (=X.Y.Z, --locked), run full suite + snapshots. 4. If breakage: fix adapter; if upstream regression or removed seam: add minimal patch on the fork branch, consume via [patch], open upstream PR, note in ADR-0003 log. 5. Merge; tag; update TECH-SPEC §14 versions. 6. If the fork carries patches: rebase them onto the new release tag; drop any that merged upstream.

8. Day-one checklist

git init proef && cd proef → commit toolchain+workspace manifests (M0.1) → cargo new the four crates → copy workspace lints/deny/nextest configs → wire CI → first green pipeline → open M0 tracking issue with this plan’s task list. Suggested first PR sequence: M0.1+M0.2 together, then M0.3 split by module, then M0.4/M0.5.

proef — Testing Strategy

Status: normative · Date: 2026-07-28 · tools: cargo-nextest, insta, proptest, cargo-fuzz, assert_cmd, tiny_http fixture. Everything below is device-free and CI-green with no external network.

1. The layers

Unit (every crate): matcher tokenization/matching edge cases; resolver escapes and depth cap; Then-merge rules; batching/segmentation boundaries; sidecar math; World error→exit-code mapping.

Property (proptest): matcher — arbitrary patterns/text never panic, valid pattern+generated text round-trips captures; resolver — $${…} escape round-trip, resolution is idempotent once fully resolved, depth cap always terminates; secret-mask invariant — for arbitrary events/reports containing a known secret value, rendered output never contains it, nor any of its derived encoded forms (base64 both alphabets ± padding, hex both cases, percent-encoding, JSON-string escape — ADR-0005 as amended), with a companion property pinning that text free of the secret and its forms passes through untouched; World — snapshot/restore is an involution; fragment scanner (proef-engine-hurl) — over generated hurl files, every reported line lies inside the file, entries are accounted for exactly once, starts are ordered and distinct, and no fragment’s text runs into the entry after it. That last one is the entry-boundary arithmetic’s whole job, and it is asserted because a draft without it passed while the boundary was deliberately broken. The scanner is proptested rather than fuzzed on purpose: it needs hurl_core, and cargo dependencies are package-level, so putting it in fuzz/ would compile hurl for every target there and drag native libraries into a job that has none.

Fuzz (cargo-fuzz, nightly job + PR smoke): fuzz_match_pattern (pattern×text), fuzz_resolve (template strings), fuzz_tag_expr (tag expressions), fuzz_pack_load (YAML bytes → loader must error, never panic), and fuzz_fragment_binding (a pack against a real corpus: ref: resolution, unread bind: keys, a bind: colliding with a variable the fragment supplies itself). Parser-adjacent hand-written code is exactly where fuzzing pays.

Both loops take their target list from cargo fuzz list, never a list written into a workflow. The names used to be spelled out in ci.yml and nightly.yml, so a target ran nowhere until both were edited and nothing failed to say so.

fuzz_fragment_binding is structure-aware — it builds a well-formed pack and corpus from the input rather than hoping the fuzzer discovers one. That is a measured choice: a byte-oriented version never resolved a single ref: in 1.45 million runs, because reaching those rules means finding valid YAML and a matching corpus name at once. When adding a target that needs structure, verify it reaches the code by probe — panic on the condition under test, run briefly, confirm it fires — because a target that compiles and finds nothing reads exactly like a target that compiles and finds no bugs.

fuzz/ is its own workspace (the root Cargo.toml excludes it, since fuzzing needs nightly), so no root-workspace command compiles it: a changed proef-core signature breaks the targets while every root-workspace gate stays green, leaving the fuzz jobs as the only signal. cargo check --manifest-path fuzz/Cargo.toml --all-targets runs on the pinned stable toolchain in seconds, so the gates job carries it — earlier than the fuzz smoke, and on both gate platforms.

Snapshot (insta): emitter — golden corpus of (features + packs) → artifacts + sidecars, byte-stable (the canonical-format compatibility surface, ADR-0010); diagnostics — rendered miette output for the seeded error corpus (every validation pass in TECH-SPEC §4.1 has at least one golden failure); proef schema output; event-stream JSONL for a reference run (with injected clock/run-id — core purity makes this deterministic).

Integration (fixture server): synchronous tiny_http dev crate (proef-fixture) modeled on the spike’s fixture — not axum as originally written: axum’s tokio requirement conflicts with the workspace’s no-async-runtime ban (ADR-0006/0007 + deny.toml), and a sync fixture keeps that invariant binary-wide (errata 2026-07-28, M3). Endpoints, extended: bearer-auth endpoints, search, create (201/422 paths), delayed push-visibility (exercises retry for real), cookie-setting endpoints (exercises SessionState round-trip), slow endpoint (exercises budgets/watchdog), malformed-JSON endpoint. Suite covers: green path (the four 500-series features), capture chaining, World/global across scenarios, optional: warn-and-continue, cancellation (token cancel mid-run completes within budget, reports written), parallel --jobs determinism (event Normalize), artifact↔execution same-bytes assertion (hash the emitted file and the text handed to parse_hurl_file).

Fragments (ADR-0018): crates/proef-cli/tests/fragments.rs builds a self-contained project per test — its own proef.toml, corpus and pack in a temp dir — because the reference corpus under tests/ is config-independent by design (several tests run it from a temp cwd with settings passed by environment variable and no proef.toml in scope), so anything needing [run] fragments cannot live there. That is also why four diagnostic codes are covered here rather than in tests/errors/ (DIAGNOSTICS.md says which).

The headline case runs one file under both runners: proef test against the fixture, then stock hurl invoked on the same bytes with an equivalent variables file, asserting the corpus comes back byte-identical. The engine is embedded, so a hurl binary is not a build requirement — that half skips with a printed note when none is on PATH rather than being faked. Provenance is asserted at both ends: the JSONL record for the event-driven readers, and --junit for the RunSummary-driven ones, since those are fed by a second copy that a green suite would not otherwise exercise.

LSP over stdio (crates/proef-cli/tests/lsp_stdio.rs): the real binary, spoken to as an editor does. This is the only place proef.toml → DiskSourceProvider → document URI is exercised end to end: the proef-lsp unit tests inject absolute source names through a fake provider, so a config-layer change can break every go-to-definition while they stay green — which has happened. These tests canonicalize their temp root, because on macOS a tempdir is /var/… whose real path is /private/var/…, and without that any cwd-relative path logic silently no-ops and the test passes without reaching the behaviour.

Corpus: --dry-run over every .feature in tests/ — the suite’s own features are the regression corpus.

Documentation (xtask docs-check + crates/proef-cli/tests/docs.rs): the docs make claims a machine can settle, so they are settled mechanically rather than by review. docs-check reads files, and does six things: every workspace crate appears in TECH-SPEC §2 and CLAUDE.md; every ADR file appears in the decision log; every diagnostic code the workspace emits has a DIAGNOSTICS.md row and vice versa; every release names each kind of change once (repeats accumulate by appending, which is how a changelog gets written); every relative link resolves; and every fenced toml/yaml example parses with the product’s own parsers, so the check means “proef would accept this”, not “some parser would”. tests/docs.rs needs a built binary and therefore lives with assert_cmd: it asks clap whether every documented command and long flag exists.

The split is a rule, not an accident, and each half states it in its own header — a check that reads files belongs in docs-check even when a test would be easier to write, because the doc-only CI step is the fast one and a file-reading check placed in the test suite silently stops running there.

Both were written against defects that had already shipped — an ADR whose first example could not load, and a row marked shipped that named a --html flag which never existed. In each the surrounding prose was correct, which is precisely what a careful reader does not catch. Two scoping rules keep them honest rather than noisy: only the indexed corpus is linted (docs/superpowers/ is a dated archive, and editing history to satisfy a checker is the wrong direction), and command detection is restricted to code spans and fenced blocks — prose says “proef discovers packs”, and treating that as an invocation produced sixty false positives against four real ones. Names the docs discuss as proposals are listed explicitly in tests/docs.rs, so adding one is a decision rather than the check going quietly soft.

CLI (assert_cmd): exit codes 0/1/2/3 pinned per command and failure class; --format json schema-checked; --junit well-formed (quick-junit round-parse).

Canary (M4): scheduled + on-release job builds against the next hurl version and replays the integration suite; red = issue with behavior diff, pins never auto-move (runbook: IMPLEMENTATION-PLAN §7).

2. What is deliberately NOT tested here

Hurl’s own HTTP semantics (asserts, filters, templating execution) — that is upstream’s test surface; proef tests the adapter contract (options mapping, variable bridging, span mapping, segmentation) against the fixture instead of re-verifying hurl. This is a direct consequence of ADR-0001 and the reason the differential-oracle harness from the research phase was retired.

3. CI matrix & gates

Linux (ubuntu-latest, prereqs pre-baked) + macOS on every PR; Windows weekly (vcpkg libs) while the port stabilizes, then per-PR (port green 2026-07-28: VCPKG_ROOT export, hurl’s crates.io-missing icon supplied in CI, /-normalized path identifiers). Gates: fmt, clippy -D warnings, nextest (all crates), doctests, rustdoc -D warnings, deny, cargo-machete, zizmor (workflow static analysis), xtask docs-check, proef doctor smoke, public-api snapshot, fuzz smoke (30 s/target), corpus dry-run, CLI suite, and — on a pull request — the changelog-entry check (§8). The complexity ratios run as their own step, alone, for the reason §7 gives. Snapshot tests (insta) run inside nextest — a drifted snapshot fails there, no separate step. Nightly: full fuzz (10 min/target), canary, cargo-audit (advisories against unchanged code — deny covers PRs). Coverage: measurable on demand, not gated in CI (P13’s local half). just cover runs cargo llvm-cov nextest over the workspace (just cover-html for a browsable report, just cover-lcov for a CI service’s lcov); the number today is ~90% line coverage of the unit + integration suites (doctests excluded — nextest does not run them). When a CI coverage job lands it must be a ratchet, not a fixed threshold — the 2026 norm and the only kind that suits a pre-1.0 codebase: fail a PR only if coverage drops, never on an arbitrary floor, and keep it informational (a PR comment) rather than a hard merge gate. A fixed percentage gate is explicitly the wrong shape here; it punishes honest additions of hard-to-cover error paths and invites coverage theater. The xtask binary’s low number is expected — it is automation exercised by running it, not by unit tests.

4. Test data management

tests/features/ — the real suite (also corpus input). tests/errors/ — seeded broken features/packs, one file per diagnostic code, name = expected code (golden snapshots). Insta snapshots live next to their suites (crates/proef-cli/tests/snapshots/), reviewed via cargo insta review. Fixture data is generated in-process (no committed binary blobs beyond one JPEG for multipart, M5).

5. Determinism rules (make flakes structural, not cultural)

Core purity (no IO/clock/rand — TECH-SPEC §4) means every non-integration layer is bit-deterministic by construction. Integration layer: fixture delays are token-driven (visibility timestamps), not sleep-raced; retry tests assert attempt counts, wall time only as generous upper bounds; parallel tests assert on Normalized event order, never raw interleaving. Any test needing “now” receives it as a parameter.

Retry-until-green is the anti-pattern, and that is why proef ships no scenario @retry. A scenario-level retry is the headline feature of several runners and is deliberately absent here: re-running a test until it passes hides precisely the defects worth finding. A bug that fails one run in four survives three retries 99.6% of the time (1 − 0.25⁴), so the suite reports green while the product is broken for a quarter of its users. proef’s shape is detect-then-quarantine: proef flaky returns a verdict over run history, @quarantine stops a known flapper gating the build while keeping it visible in every sink (ADR-0019), and per-step retry: covers the case that is genuinely polling — a resource that becomes visible on the Nth attempt — rather than rerolling a verdict. proef’s own suite is held to the same rule: a red test here is reproduced and filed, never re-run until it cooperates and then forgotten.

6. Every diagnostic code is named by a test

DIAGNOSTICS.md calls codes “a contract: they never change meaning”. A contract with nothing holding it to it is a wish — 23 of 75 were in that state when the rule was written: reachable in production, documented, and exercised by nothing at all, not even an assertion on their message text.

source_guards.rs enforces it. A code counts as covered when either a seeded tests/errors/<area>__<name>/ directory exists (the corpus driver dry-runs it, so the rendered diagnostic is exercised end to end) or the literal code string appears in a test. Naming the code, not matching the prose — the wording is expected to improve, while the code is the part that promises not to change.

Two codes are exempted by name with recorded reasons (source::unreadable, config::unreadable need a file the process may stat but not read, which CI runners do not reproduce because they run as root). The guard checks its own exemption list too: an exemption that outlives its code silently excuses nothing.

Reaching a defensive guard is worth the effort rather than a reason to skip it. lower::kind_unrouted fires only when the engine registry and pack validation disagree, so its test makes them disagree; lower::expansion_too_deep sits behind pack validation’s identical limit, so its test bypasses validation with load_collecting — the only way to hand lowering a graph validation would have stopped, and therefore the only way to prove the second line of defence still works.

7. Complexity claims are asserted as ratios, never as benchmarks

A published performance claim is a claim like any other, and this project has now watched four separate ones decay in prose. The guard for a shape claim — “linear in the macro count”, “~2× per doubling” — is a ratio between two input sizes, not a stopwatch against a threshold:

• A ratio tests what was actually promised. The claim is a shape; a shape is a ratio.
• The separation is wide enough to be safe. validation_cost_stays_linear_in_the_macro_count observes ~2.05× against a bound of 3.0; restoring the pre-#138 quadratic shape measures 4.01×. Take the minimum of several interleaved samples — scheduler noise only ever adds, so the fastest observation is the closest to the work actually done — and assert the smaller load was slow enough to time at all, or the ratio is meaningless.

A timing test runs alone, or it does not run. These are #[ignore]d and have their own CI step and just perf; nothing else shares the machine. The first version of this section claimed the opposite — that a ratio “survives a shared runner” because load inflates both sides and cancels — and shipped a test that failed on its second full-suite run. Measurement: 2.05× alone, 3.09× under nextest’s full parallelism. The larger input has the larger working set, so memory-bandwidth contention penalises it more; the ratio drifts rather than cancelling, and interleaving cannot fix a systematic effect. nextest’s test-groups bound concurrency within a group, which does not isolate one from the rest of the suite — so #[ignore] plus a dedicated invocation is the only mechanism that actually delivers isolation.

Benchmark frameworks were considered and are deliberately absent. iai-callgrind is the right tool for gating in CI, because instruction counts ignore runner noise entirely — but it needs valgrind, making it a gate the maintainer cannot reproduce on macOS. criterion and divan measure wall time, which is the same noise regime as the ratio test while also adding a dependency tree to a workspace that audits every edge.

8. A change that lands records itself

RELEASING.md states that every landed change adds an [Unreleased] line in the commit series that lands it. Nothing enforced that, and the rule was broken exactly once — by the series that added the guard for the changelog’s shape. A rule whose only enforcement is a sentence in another document is a rule with a known decay rate, which is the same finding this suite keeps re-deriving.

So a pull-request job asks one question: did any crates/** or xtask/** .rs file change without docs/CHANGELOG.md changing too? If so it fails, naming the files and quoting the rule. [no changelog] in the PR title waives it.

It was sized before it was written, because a gate with a high false-positive rate trains people to reach for the waiver and is then worse than nothing. Across the 21 source-touching merges preceding it the rule would have fired once — on the one commit that actually broke it. Pure-test and pure-performance changes all carried an entry already, so “source changed” tracks “worth recording” closely here. That is a measurement of this repository’s habits rather than a general law, and the waiver exists for where it stops holding.

The check reads a diff rather than files, so it is neither a docs-check task nor a tests/docs.rs test — it lives in the workflow, which is the only place the base commit is known. The PR title reaches it through env, never interpolated into the shell body: a title is attacker-controlled text, and ${{ … }} inside run: is a template injection that zizmor flags.

Runbook — thin-fork patching (ADR-0003 tier 2)

Rehearsed 2026-07-28 (M4). The steady state carries zero diff; this runbook is the mechanics for the moment a release breaks the seam or a small change is needed before upstream accepts it.

The drill (as rehearsed, local-path variant)

1. Obtain the pinned source (fork tag in the real flow; the vendored registry copy suffices for a drill):

   cp -R ~/.cargo/registry/src/index.crates.io-*/hurl-8.0.1 /tmp/hurl-fork
   # apply the minimal patch (one commit on the fork branch in the real flow)
   

2. Wire the override at the workspace root (Cargo.toml):

   [patch.crates-io]
   hurl = { path = "/tmp/hurl-fork" }               # drill
   # hurl = { git = "https://github.com/<org>/hurl", tag = "8.0.1-proef.1" }  # real
   

3. Verify cargo resolves the fork, then build and run the full suite:

   cargo tree -p proef-engine-hurl | grep "hurl v"   # must show the path/git source
   cargo build -p proef-engine-hurl && cargo nextest run
   

4. Revert: remove the [patch.crates-io] block, git checkout Cargo.lock, confirm cargo tree shows the registry source again.

Rehearsal result: override resolved, engine + suite built green against the patched path, revert restored registry resolution. Elapsed ≈ one hurl rebuild.

The real flow (when a patch is actually needed)

1. Fork Orange-OpenSource/hurl; branch proef-patches-<version> off the release tag; apply the minimal diff (one commit per logical patch).
2. Tag X.Y.Z-proef.N; consume via the git [patch.crates-io] form above.
3. Open the corresponding upstream PR immediately (tier 3 — the fork’s diff must trend back to zero); note the patch in ADR-0003’s log.
4. On each upstream release: rebase the branch onto the new tag, drop merged patches, re-run the canary, move the pins per IMPLEMENTATION-PLAN §7.

Pin-bump checklist — hurl 8.1 watch items (recorded 2026-08-16)

Behavior changes already on hurl master that the canary cannot see — compile-and-test stays green while a guarantee shifts. Work each of these when the pins move to 8.1 (IMPLEMENTATION-PLAN §7); each names the fixture to add.

The structural reason these need a list at all: the fragment scanner matches OptionKind narrowly (if let OptionKind::Variable — one arm, by design, so a semver-allowed variant addition is not a compile break). New [Options] in a fragment file are therefore silently accepted, not flagged. That is the right default for options with local effect, and exactly wrong for the first item below.

• variables-file: (upstream #2021) — check the sandbox before accepting it. Upstream opens the named file with a raw File::open against process CWD, no ContextDir confinement. A fragment corpus is foreign by design, so a corpus file saying variables-file: ../../secrets.env would read a file outside the project on proef’s behalf. On bump: decide refuse-or-flag (option_declared_twice’s family machinery fits), and add a fixture — a .hurl with an escaping variables-file: must not silently read the target.
• Cross-host cookie strip (upstream #5118, landed) — a redirect across hosts stops forwarding cookies. Fixture: a fixture-server redirect pair asserting which cookies arrive, so the behavior flip shows up as a diff in our suite rather than as a user’s broken auth flow.
• --file-root resolution change (upstream #2830, watch) — multipart asset paths may move from CWD-relative to hurl-file-relative. This is proef’s multipart seam: artifacts are emitted to a different directory than the pack they came from, so relative asset paths are exactly the bytes that would change meaning. Fixture: a multipart scenario whose asset path only resolves under one of the two rules.
• Value::Duration (upstream #3519, watch) — a captured duration re-rendered into a template may change its string form. Fixture: capture a duration-typed value, splice it into a later request, snapshot the bytes.
• New [Options] variants generally (no-header, http2-prior-knowledge, fail-with-body, no-jsonpath-coercion, …): the scanner accepts them silently (above). On bump, sweep the new variants once and sort each into “local effect, fine” or “needs the variables-file: treatment”.

Currently drafted patches

• docs/upstream/0001-run-entries-reusable-client.patch — run_entries accepts &mut Client (verified two-call-site change; erases per-segment connection + cookie costs, deletes proef’s SessionState cookie round-trip once adopted). Applies cleanly to 8.0.1 and compiles (verified in the M4 drill). PR text: docs/upstream/0001-PR-DESCRIPTION.md.

ADR-0001 — Embed hurl’s crates in-process as the API engine

Status: Accepted · Date: 2026-07-28

Context

The API engine must execute HTTP tests with Hurl semantics. Constraint history: the original “entirely in Rust” rule (which forbade C-linked deps and favored a pure-Rust reimplementation) was relaxed to “mostly Rust — C system libraries acceptable within margin”, and the product intent was fixed as “a wrapper / Gherkin adapter over hurl, riding upstream hurl as it improves”. Verified facts: hurl’s runner::run_entries is the seam hurl’s own parallel workers use (buffered stdio, per-entry results with spans, captures, libcurl timings); hurl links libcurl + OpenSSL + libxml2 and needs libclang at build; hurl_core alone also links libxml2; the crates break API in minor releases (issue #3846); the format itself is spec’d and stable. A working spike cross-ran generated artifacts under a prototype pure-Rust runner and the stock hurl CLI: the first cross-check caught a real semantic divergence (contains = element equality on collections, not substring) — demonstrating the standing cost of reimplementation.

Decision

proef-engine-hurl embeds the hurl crates, pinned exactly: generate .hurl text from the IR, hurl_core::parser::parse_hurl_file, then hurl::runner::run_entries with WriteMode::Buffered terms, an EventListener for progress, and VariableSet in/out. No pure-Rust HTTP reimplementation ships; no hurl subprocess is required at runtime.

Consequences

Positive: hurl semantics by construction (zero drift class); full assert/filter/XPath/ cookie/redirect/HTTP-2 surface available to packs on day one; rich in-process results with source spans mapped back to .feature lines; ~a third less engine code than the reimplementation plan. Negative: build prereqs on every machine (libcurl-dev libxml2-dev libclang pkg-config; macOS ships the libs) — mitigated by docs + proef doctor; dynamically-linked release binaries (no musl static); crate API instability — mitigated by ADR-0003; run_entries is #[doc(hidden)] (semi-blessed seam) — covered by a pinned integration test.

Alternatives considered

Pure-Rust native engine + hurl-CLI differential oracle (the original recommendation under the zero-C rule): zero C deps and static binaries, but permanent semantic-drift liability and a differential harness to maintain; recorded in research/ as the path back if static distribution ever becomes a requirement. Transpile + hurl CLI subprocess: full fidelity, least code, but an external binary dependency, no in-process events, and clunky cross-scenario variable threading; its remnant lives on as the upgrade-canary and proef artifacts hand-off. Divergent fork of hurl: rejected — upstream keeps the format low-level by philosophy (issue #2090) and a divergent fork defeats riding upstream improvements (see ADR-0003 for the thin-fork nuance).

ADR-0002 — Multi-engine core: factory/session seam, step-kind routing, batching

Status: Accepted · Date: 2026-07-28 (amended 2026-09-01 — the core’s entry grammar is a named closed set; see the Amendment below, and its 2026-09-10 correction)

Context

Requirement: all tests are Gherkin; the parser dispatches to pluggable engines — API (hurl) now, a future non-hurl engine possible later behind the same seam — a factory/session seam (multiple engine implementations behind one trait, a step’s kind-prefix routes it to its engine, one shared variable scope). Balanced-architecture stance: deliberate seams for a future engine, no gold-plating. Ecosystem survey: probe-rs’s ProbeFactory/DebugProbe split is the closest production analog; sqlx registers compiled-in drivers explicitly; dispatch cost at batch granularity is noise, so dyn vs enum is decided on coupling, not performance (enum_dispatch would couple core to every engine crate — wrong direction).

Decision

Two traits in proef-core; engines implement both; the CLI assembles the registry.

#![allow(unused)]
fn main() {
pub trait EngineFactory: Send + Sync {
    fn id(&self) -> &'static str;
    fn step_kinds(&self) -> &'static [StepKindSpec];  // pack namespace + schema fragment
    fn doctor(&self) -> Vec<DoctorCheck>;
    fn open(&self, ctx: &ScenarioCtx) -> Result<Box<dyn EngineSession>, EngineError>;
}
pub trait EngineSession: Send {
    fn run_batch(&mut self, batch: &StepBatch, world: &mut World,
                 events: &EventSink, cancel: &CancellationToken) -> BatchResult;
    fn finish(&mut self) -> Result<(), EngineError>;
}
}

Routing: a macro step’s kind names its engine (http: → engine-hurl; other kind prefixes reserved for a future non-hurl engine). A lowered scenario is an ordered heterogeneous step list; the core dispatches contiguous same-engine batches in order. The World is the interop bus between batches and engines. Sessions are per-scenario, opened lazily, torn down in finish (+ Drop backstop); engines may hold sessions concurrently within a scenario. Registry: Vec<Box<dyn EngineFactory>> in proef-cli, engines optionally behind cargo features (one feature per engine). Lifecycle is enforced by ownership shape (only a session runs batches), not typestate generics (which would break dyn).

Consequences

Adding an engine = one crate + one registry line; pack schema and doctor extend via step_kinds()/doctor() without core edits — the acceptance test: a future non-hurl engine lands with zero proef-core diff. Core stays free of engine-specific types. Costs accepted: Box<dyn> indirection (irrelevant at batch granularity); two traits instead of one (justified: lifecycle safety + capability discovery). Engines own their artifacts (hurl files / screenshots / HAR).

Alternatives considered

Single Engine trait with runtime lifecycle state (v3 draft) — weaker lifecycle guarantees; enum dispatch — inverts the dependency direction; dynamic loading (dlopen/ WASM) — rejected as over-architecture, compiled-in covers every stated future; typestate generics — fights dyn, ownership shape gives most of the safety.

Errata

2026-07-28 (M1/M5): The routing example above names the API step kind http:; ADR-0004’s examples and TECH-SPEC §6’s normative pack schema use hurl: (the raw-block key doubles as the routing kind). The implementation follows the tech spec: the step kind and the engine id are hurl, so “a step’s kind names its engine” holds verbatim. Read http: in the Decision above as hurl:. Other kind prefixes remain reserved for a future non-hurl engine as written.

Amendment — the core’s entry grammar is a named closed set

2026-09-01 · Accepted. “Core stays free of engine-specific types” is true and stays true. “Core stays free of engine-specific syntax” was never true, and the worklist carried the gap for two rounds without resolving it. This amendment states the real boundary and makes it enforceable.

Why the core knows any hurl at all

The core performs text surgery on entries: bake_entry_options splices an [Options] block into each entry after its header block, and an expect: macro merges asserts into the previous request entry (ADR-0004). Both operations have to find an entry boundary in text the engine will later parse. That is structural, not incidental — the surgery is what the pack format is built on — so a boundary recogniser has to live somewhere, and pushing it behind the seam would move the literals without making the algorithm engine-independent.

The measurement

Not the “~290 lines, all in lower.rs” the worklist recorded — that figure counted #[cfg(test)] fixtures, where a core test exercising the pipeline necessarily writes some engine’s payload. The vocabulary is thirteen distinct literals across four files:

The four boundary recognisers (is_method_line, is_section_header, is_response_line, is_header_line) are already one canonical pub(crate) set shared by three of those files. That half is done.

The third group is the one this measurement nearly missed, and it is worth naming why. bind.rs renders a did-you-mean help string for an author whose sentence bound no macro, and that string contains a small hurl example. It generates nothing and parses nothing, but it is engine syntax living in the core, and it drifts like any other copy. The guard’s first version could not see it — the literal spans lines, and a per-line scan discards a run that never closes — while this amendment claimed the set was closed. Multi-line literals are where a larger piece of engine syntax would naturally be written, so the blind spot sat exactly where the risk is highest. The guard now lexes whole files.

The same row cost a second correction (2026-09-02). Lexing whole files surfaced HTTP 200; the GET ${url:base}/PATH line directly above it in the same literal stayed invisible for another round, because the guard classified four shapes — fence, response line, section header, option line — and a method line was not among them, though this section names it as one of the four recognisers. A guard is closed only over the shapes it can classify, so the two claims have to be checked against each other rather than assumed to agree. The classifier now knows method lines, which is what added the row above. In the same pass the scan stopped truncating at a file’s first #[cfg(test)] mod and began excising every test module instead: production code placed after one was silently unscanned, and html.rs and pack/validate.rs already carry a second test module.

proef’s own pack keys (macros:, match:, secret:, steps:, use:) are shaped like option lines and are excluded by name rather than listed as sanctioned rows: an inventory that is a third exceptions stops reading as a closed set.

The asymmetry this exposes

StepKindSpec::options exists, in its own words, as “the seam that keeps option spellings out of proef-core” — added because matching "retry-interval:" as a literal meant “one rule lived at two altitudes.” It covers recognising options. The core still writes retry:, retry-interval:, delay: and variable: as literals, so the same rule still lives at two altitudes, in the other direction.

Correction (2026-09-10) — the set was fourteen, and the fourteenth was unclassifiable

The measurement above says thirteen literals across four files. It was fourteen. The one it missed is "file," in emit.rs, where file_refs_in found the assets an artifact reads by scanning for that literal and a closing ; — hurl’s body grammar, in proef-core, for the entire life of asset staging.

It went unrecorded for the same reason the method line did, and this section had already written the rule that predicts it: a guard is closed only over the shapes it can classify. engine_grammar_kind knew fences, HTTP, [Section] headers, method lines and key: value options. A body constructor is none of those — no colon, no brackets, no uppercase — so the literal was never classified, never entered the inventory, and was never reported missing from it. The set was not measured and found closed; it was measured through a classifier that could not see this member.

That is the third decay of this section’s own claim: once by an order of magnitude in the count, once by a multi-line literal, and now by a shape. Each time the count was wrong in the direction of the guard’s blind spot, which is the only direction it can be wrong in.

Resolved by the second remedy, not the first. Decision 2 below sends the author to one of two options: widen the sanctioned set on the record, or put the syntax behind the seam. This is the first time the second was taken. The scan is now StepKindSpec::assets, a fourth engine-contributed hook beside validate, fragments and options — so the recognised group loses its body-reference member entirely rather than gaining a sanctioned row.

Moving it also fixed the reading. A text scan cannot tell a real file,…; body from the same six characters inside a JSON or assertion body; the engine reads its own AST and can. It also has to avoid hurl’s shared visit_filename hook, which carries the [Options] file paths (output, cacert, client-cert, client-key, netrc-file, unix-socket) alongside real bodies — output: names a file the run writes, and staging it would demand a source that cannot exist. Only the two body positions are read. None of that distinction is expressible in core, which is the argument for the seam stated as a capability rather than as a rule.

The classifier gained a body arm in the same change, so the blind spot is closed independently of the literal that exposed it: a bare lowercase keyword followed by a comma (file,, hex,, base64,) is now classified, and reintroducing one into core fails the guard with body "file," in emit.rs.

Decision

1. The set above is the sanctioned core entry grammar. It is closed: a token outside it, or an existing token appearing in another core module, is a defect against this ADR.
2. It is pinned by crates/proef-cli/tests/source_guards.rs (hurl_grammar_in_core_is_the_closed_set_the_adr_names), which lexes every production literal in proef-core and fails on growth, on relocation to another core module, and on shrinkage — then sends the author back here. A claim of this shape decays the moment it is only prose. This one already had, twice: once by an order of magnitude in the count, and once in this very section, which asserted a closed set while the guard behind it could not read a multi-line literal.
3. Migrating the written group behind the seam (an emitter beside StepKindSpec::options) is deferred, not rejected. It buys nothing today: hurl is the only engine and no other is scheduled, so the migration would add a fn pointer, a trait obligation and a public-API break to relocate seven literals that exactly one implementation will ever supply. Trigger: a second engine being scheduled. That is also when ADR-0002’s acceptance test — a new engine lands with zero proef-core diff — first has anything to say about them; until then it is unfalsifiable here either way.

Consequences

The acceptance test is narrowed on the record: a second engine lands with zero proef-core diff except the written group, which is a known, enumerated, guarded debt with a named trigger rather than an open question. Anyone reaching for new hurl syntax in the core hits a failing test that names both remedies.

ADR-0003 — Upstream tracking: exact pins, thin zero-diff fork, upgrade canary

Status: Accepted · Date: 2026-07-28

Context

Requirement (stated): “use a hurl fork with as few changes as possible — I want to keep using hurl as it improves over time; a wrapper/Gherkin adapter over hurl.” Verified: hurl ships breaking crate-API changes in minor releases (#3846; maintainers advise cargo install --locked); the release cadence is multiple per year; run_entries is #[doc(hidden)]. One concrete patch need is already identified (ADR-0010 / TECH-SPEC §5): run_entries creates its HTTP client internally, so accepting &mut Client would erase per-segment connection costs — a verified two-call-site change.

Decision

Three-tier policy, in order of preference. (1) Steady state: depend on published crates with exact pins (hurl = "=8.0.1", hurl_core = "=8.0.1"), build --locked; the GitHub fork exists but carries zero diff. (2) Patch vehicle: when a release breaks the seam or a small change is needed, carry a minimal-diff branch on the fork, consumed via Cargo [patch."crates-io"] (or a git-tag dep), rebased onto each upstream release. (3) Upstream everything: every patch is PR’d upstream so the fork’s diff trends back to zero. An upgrade-canary CI job (weekly + on upstream release) builds against the next hurl version and replays the full suite; pins move only after it is green, via the runbook in IMPLEMENTATION-PLAN §7.

Consequences

“Keep using hurl as it improves” becomes a scheduled chore, not a gamble; the wrapper never becomes a divergent fork; breakage is discovered pre-pin-bump. Costs: upgrade PRs are deliberate work per release; the fork must be rebased when (and only when) it carries a patch; MSRV follows upstream (hurl master already 1.97.1 — neutralized by the project’s always-latest-stable toolchain rule).

Alternatives considered

Track master via git dependency — unvetted breakage flows in continuously. Vendor the hurl source into the repo — a divergent fork in disguise; loses provenance and cadence. Caret/tilde version ranges — semver is demonstrably not honored for the library surface; exact pins are the only safe mode.

ADR-0004 — Pack format: YAML skeleton + embedded raw Hurl blocks

Status: Accepted · Date: 2026-07-28

Context

Requirement (stated): macro packs must be human-readable — “is there a better alternative than macro YAML?” Analysis (architecture review §8) evaluated YAML+schema, KDL, TOML, Pkl/CUE/Dhall, RON/JSON5, Rhai/Lua scripting, a custom DSL, and Karate-style Gherkin-native macros against readability/writability, comments, multiline bodies, templating interplay, editor tooling, serde support, and team familiarity (existing packs are YAML with schemars-driven autocomplete). Key insight: the unreadable part of packs was never YAML itself — it was HTTP-as-YAML-trees, while hurl’s own plaintext format is the human-readable HTTP DSL, and the backend team already reads/writes it fluently.

Decision

Packs stay YAML (serde_norway; schemars JSON Schema; comments; block scalars) but only as the thin binding skeleton: macro name, match: pattern, params, defaults, tags, description, composition (use:/with:), step modifiers (optional:, when:, retry:, saveAs:). The HTTP payload of a hurl step is a raw Hurl block:

steps:
  - name: Resolve the record name to its id
    hurl: |
      GET ${url:base}/api/v1/admin/search/records
      Authorization: Bearer ${secret:apiToken}
      [Query]
      q: ${name}
      HTTP 200
      [Captures]
      recordId: jsonpath "$[0].id"

Blocks are validated at pack load by parse_hurl_file after ${…} lowering — real hurl syntax errors with real spans. Structured step trees are reserved for a future non-hurl engine, which would have no native text DSL. Assert-only macros use expect: (merged into the previous request entry — the Then-step rule).

Consequences

Pack bodies are literally hurl: copy-paste flows both ways with the backend corpus; no bespoke assert/capture schema to maintain for the API engine; the emitter for hurl steps approaches the identity function. Costs: autocomplete inside the block is plain-text (mitigated: load-time parse errors are immediate; editors have hurl highlighting; a proef fmt pass can normalize blocks); one lowering pass must run before parse (already required for ${…}).

Alternatives considered

KDL — pleasant syntax but no schema/LSP story comparable to YAML, zero team familiarity; recorded as the fallback if YAML friction materializes. TOML — wrong shape for nested step lists (kept for proef.toml config). Pkl/CUE/Dhall — second language + toolchain, over-architecture at this size. Rhai/Lua — packs become programs; kills static validation and --dry-run guarantees. Custom DSL — a parser/LSP/formatter to own forever. Karate-style callable feature files — collapses the macro/test distinction; a typed params/defaults/validation model is strictly stronger.

Amendment (2026-07-30): the top-level key is macros:

The pack root key was renamed templates: → macros: to end a three-way naming split (the YAML key said templates, the docs and internal model said macro, the file/dir said pack). The entry is now uniformly a macro; a pack is a file of macros. Pure rename — format, schema, and semantics are unchanged (error-corpus snapshots regenerated, diff verified as templates:→macros: only). No templates: alias is kept — one canonical spelling (golden rule: one way to do one thing).

ADR-0005 — Two-tier variables (${…} / {{…}}), the World, and secrets

Status: Accepted · Date: 2026-07-28

Context

The author-time variable system (${var}, ${env:NAME:-default}, ${run:id}, ${global:key}, ${fake:*} seeded from the run id, $${} escape, recursive expansion) is proven with test authors. Hurl has its own runtime templating ({{name}}) fed by captures. The spike validated running both tiers side by side — including the bug it surfaced: step-captured args can themselves contain ${…} and need recursive (depth-capped) resolution.

Decision

Two explicit tiers. ${…} is author time: params, env (with defaults), run id, World reads (${global:key}), fake data, secrets references — resolved during lowering, recursively with a depth cap of 8, and baked into artifacts (except secrets). {{…}} is run time: hurl-native templates for captures, left verbatim in artifacts so the embedded engine and the stock CLI resolve them identically. World: one typed variable scope per scenario plus a persistent global store (.proef-state.json, atomic temp+rename). Engine bridging: seed hurl’s VariableSet from the World before each batch; merge HurlResult.variables back after; saveAs: global promotes a capture into the persistent store. Secrets: ${secret:NAME} resolves from the PROEF_SECRET_<NAME> environment override, else the encrypted store .proef-secrets.json (chacha20poly1305 + rpassword); values are injected via VariableSet::insert_secret (hurl redacts them in logs/reports); artifacts carry {{secret_name}} placeholders, never values; our reporters additionally redact by value (property-tested invariant).

Amendment (2026-08-16): “redact by value” includes each secret’s common encoded forms — base64 (both alphabets, with/without padding), hex (both cases), RFC 3986 percent-encoding, and the JSON-string escape — derived inside Redactions::new so every sink is covered by construction. Demonstrated live before the amendment: a server reflecting a bearer token base64-encoded put a trivially-decodable string into an assert-failure detail, the raw needle never fired, and the encoded credential reached the console and events.jsonl. The needle set covers the reversible transforms that occur at HTTP boundaries; a secret reflected hashed or re-encrypted matches no needle list, and the ADR does not claim otherwise. Over-redaction is the accepted failure direction.

Consequences

Artifacts are runnable by both toolchains with identical meaning; authors keep a familiar mental model unchanged; secrets are structurally absent from every persisted output. Cost: two syntaxes coexist in packs — mitigated by the strict rule of thumb (“$ = before the run, {{ = during the run”) documented in the pack authoring guide.

Alternatives considered

Single-tier (resolve everything at author time) — breaks capture chaining and makes artifacts non-parametric. Single-tier (everything hurl {{}}) — loses env defaults, fakes, and World reads. String-only World — kept typed here (hurl Value model) because captures cross engines; stringly-typed round-trips would lose numbers/bools at engine boundaries.

Errata

2026-09-07 (0.18): the invariant’s reach was found unenforced on one of its two paths. The event stream masks through Redactions::apply_event, an exhaustive destructure; the CI sinks that render from RunSummary (JUnit, CTRF, TAP, timings.json, the GitHub summary and annotations) masked failure detail, but five of them bypassed the masker for the identity fields (scenario, file, tags, the skip reason) — no live leak, since secrets lower to {{name}} and the engine pre-redacts details, but a boundary held by convention. Closed per sink (#171), then made structural (#178): Redactions::apply_outcome destructures ScenarioOutcome/StepOutcome without .., so a new text field fails to compile until it is masked, and each sink redacts one outcome after matching @quarantine on the raw identity. Per-sink rather than a wholesale apply_summary, because RunSummary also feeds exit_code_excluding, whose quarantine matching needs the unredacted identity.

2026-07-29 (v0.3.1): the “secrets reach no sink” invariant now explicitly covers the persistent World: a saveAs: global capture whose value equals a known secret is refused (the owning step warns) — .proef-state.json is plaintext at rest and must never receive secret-derived material. PROEF_KEY (base64) may supply the project key via the environment for CI use of a committed ciphertext store.

2026-07-28 (post-M5 hardening; API removed 2026-07-29 per YAGNI): “snapshot/restore across scenario retries” originally described a mechanism whose trigger was never specified anywhere in the corpus — no CLI flag, tag, or pack directive schedules a scenario-level retry (US-5’s step-level retry: is implemented and is the flake tool in practice). Decision: scenario-level retries are deferred indefinitely, and the unused GlobalStore::snapshot/restore API has been removed (YAGNI: no dead promised-behavior code). Whoever implements scenario retries specifies the trigger surface and its mechanism in a superseding ADR. Also note: the scenario merge-back is write-set-only (World tracks its saveAs promotions) — merging a whole snapshot back would lose concurrent scenarios’ updates.

ADR-0006 — Engine traits are sync + dyn; no async machinery in v1

Status: Accepted · Date: 2026-07-28

Context

All planned engines are blocking at the edge: hurl is synchronous libcurl, and a future non-hurl engine’s driver can be driven blocking — even one that also offers an async API. Verified ecosystem facts (mid-2026, stable 1.97): async fn in traits is stable for static dispatch but still not dyn-compatible (AFIDT is nightly-only, no timeline; RTN unstable); maybe-async’s sync/async toggle is a non-additive cargo feature with documented ecosystem breakage. Calls across the seam are coarse (batch-level), so async buys no throughput inside the runner itself.

Decision

EngineFactory and EngineSession are synchronous traits, used as Box<dyn …>. Parallelism is scenario-per-OS-thread. If a future host (server mode, desktop UI) is async, it wraps engine calls in spawn_blocking at its edge; the core never learns about executors (reinforced by the sans-IO-lite rule: core does no IO at all). maybe-async is explicitly banned. Traits meant to be implemented externally (Engine*) are not sealed; internal traits that must stay evolvable are sealed.

Consequences

Simple, dyn-compatible seam today; no async runtime in the dependency tree (tokio-util’s CancellationToken is runtime-independent — ADR-0007); a future async migration is additive (an AsyncEngineSession adapter or AFIDT adoption when stable), not a rewrite. Cost: a long-running async host pays one thread per in-flight scenario — acceptable at e2e-suite scale.

Alternatives considered

Async-first trait via async-trait — boxes every call, forces an executor decision on every consumer, and models nothing real while engines block. maybe-async dual API — non-additive feature hazard. Callback/actor-per-engine threading model — more machinery than batch dispatch needs; revisit only if an engine genuinely multiplexes (e.g. a driver with concurrent event streams), and then inside that engine crate, invisible to the seam.

ADR-0007 — Cancellation: cooperative at batch boundaries, with budgets

Status: Accepted · Date: 2026-07-28

Context

Source-verified: hurl has no cancellation mechanism anywhere — no signal handling in the workspace, no abort check in the entry loop; delay/retry-interval are uninterruptible thread::sleeps; retry/repeat accept Count::Infinite; the only bounds are libcurl’s per-request timeouts (default 300 s), which do not bound total entry time under retries. The standard here is structural cancellation (CancellationToken threaded everywhere). Verified: tokio_util::sync::CancellationToken is runtime-agnostic (tokio sync primitives are documented runtime-independent; default-features = false — no tokio runtime enters the tree); is_cancelled() polling and child_token() work from plain threads.

Decision

Cancellation is cooperative at batch boundaries: the orchestrator checks a per-run CancellationToken (child token per scenario) before opening sessions and before each batch; EngineSession::run_batch receives the token so engines may honor it at finer grain when they can (a future non-hurl engine might; engine-hurl cannot mid-run_entries). Stuck-batch policy, layered: (1) the pack lint rejects infinite retries (retry: must carry a finite count) and unbounded repeat; (2) engine-hurl clamps per-request timeouts and computes a batch budget = Σ(entry timeout × (retries+1)) + retry intervals + margin; (3) a watchdog marks a scenario thread abandoned when its budget expires — the runner records a System failure with full context and detaches the thread (process exit reaps it) rather than blocking the run on an unjoinable thread. Ctrl-C: first signal cancels the token (graceful: finish current batches, run teardowns, write reports); second signal hard-exits.

Consequences

Bounded, explainable runs; no dependency on hurl gaining cancellation; a clean seam for engines that can do better. Costs: a cancel can wait out one in-flight batch (bounded by its budget); abandoned threads leak until process exit (accepted: the process is short-lived by design). If interrupt support ever lands upstream, adopting it is an engine-internal change (candidate for the ADR-0003 patch pipeline).

One consequence reaches a later feature, so it is recorded here rather than discovered again. [run] exclusive-tags promises a matching scenario the pool to itself, and the dispatcher enforces that against its own active set. An abandoned scenario leaves that set the moment the watchdog fires, while its detached thread keeps issuing requests until the entry in flight returns — the whole reason abandonment exists. So an exclusive scenario can start while an abandoned neighbour is still talking to the target: exactly the interference the key promises away, in the one window this ADR knowingly leaves open. It takes a budget blowout in the same moment, and a run that never trips the watchdog never meets it, so this is documented rather than engineered away — closing it means a bounded grace on the exclusive fill gate while detached threads report alive, which buys a rare guarantee with a per-exclusive-scenario delay every run pays. Revisit if the isolation guarantee ever has to be absolute.

Amendment (2026-09-07) — the budget family is closed over its inputs, and bounded as a product

The 0.18 survey found three holes in the value-cap regime, one of them the exact shape this ADR exists to prevent:

• max-time: was read by the budget calculator and invisible to the lint. entry_timeout has always taken a literal max-time: as the entry’s timeout, while the option recogniser did not know the key — so [Options] max-time: 100000h was lint-clean and produced a multi-year batch budget the watchdog dutifully honoured. It now carries the duration cap like every budget input, and a test pins the rule the hole broke: every option the budget reads must be one the lint can see (every_budget_input_carries_a_value_rule).
• retry-interval: multiplied into the budget with no value cap. It was in the recogniser for double-declaration purposes only; it now carries the duration cap.
• Individually capped values compose into an unbounded product. retry: 10_000 (at the count cap) times a 30 s timeout is ~83 hours, lint-clean; saturated arithmetic reaches Duration::MAX, whose Instant + budget addition panics — contained by the dispatcher’s catch_unwind, but reported as a phantom “scenario thread panicked” system fault from a user-authored value. Two closures: the computed batch budget clamps to an absolute ceiling of four hours (MAX_BATCH_BUDGET — generous for any batch of API calls with finite retries, and a truthful watchdog abandonment for a runaway product), and the dispatcher’s deadline arithmetic uses checked_add with a far-future fallback, so an engine that ever hands core an unclamped budget degrades to “no deadline” rather than a panic.

In the same family: [http] timeout-ms = 0 was accepted and means no timeout to libcurl — the exact unbounded hang the default defends against, opted into by a value that reads like “immediately”. Refused as a user error now.

Alternatives considered

Killing scenario threads — unsound in Rust (no safe thread kill). Running each batch in a subprocess for killability — reintroduces the subprocess architecture ADR-0001 rejected, per-batch. Relying on timeouts alone — unbounded under retry loops (verified), and no graceful-report path on Ctrl-C.

ADR-0008 — Serde event spine, decorator reporters, libtest-mimic harness

Status: Accepted · Date: 2026-07-28

Context

a live-event seam (EventSink(Arc<dyn Fn(RunEvent)>), domain events decoupled from wire format) is proven; its run record is a separate reporter path. The two best in-domain designs both converge on “typed event stream + composable consumers”: cargo-nextest (runner emits structured events; reporter fans out to human/JUnit/machine outputs) and cucumber-rs (decorator Writer stack: Normalize → Summarize → leaves, with Tee, marker traits for ordering guarantees). Verified: nextest officially supports libtest-mimic custom harnesses (documented CLI contract; libtest-mimic 0.8.x); libtest’s JSON format itself is still unstable nightly territory; OTel test semconv is “development” maturity.

Decision

One serde-able event enum in proef-core is the spine: RunStarted, ScenarioStarted, BatchStarted, StepFinished (engine id, StepRef feature/line, status, attempts, duration, capture names), ScenarioFinished, RunFinished. The JSONL run record is the appended event stream — explain, history, and any future UI replay from disk; no second record format. Reporters are event consumers composed decorator-style: Normalize (repairs interleaving from parallel scenarios) → Summarize → leaves: console BDD tree, JUnit XML (via quick-junit), GitHub job summary, JSONL appender. Secret values never enter events (capture names only — redaction invariant, ADR-0005). M5: a libtest-mimic harness binary exposes one Trial per scenario, making cargo nextest run and IDE test UIs drive proef with zero custom protocol work; libtest JSON remains an output adapter, never the native schema. OTel export: deferred; if added, a thin optional reporter mapping to test.* semconv names.

Consequences

Single source of truth for live progress and persistence; reporters are ~a page each; new outputs are additive leaves. Replayability makes run records diffable and testable (insta snapshots over event streams). Cost: event schema becomes a compatibility surface — versioned with a schema field from day one.

Alternatives considered

a split design (live events + separate JSON record) — two sources of truth to keep consistent. tracing as the event bus — wrong tool: tracing is operator telemetry, not a typed result stream (kept for diagnostics). Cucumber’s writers verbatim — async trait

• World coupling we don’t need; the decorator shape is what’s adopted.

Errata

2026-07-29: the variant set has grown additively since acceptance: EntryRunning (live per-attempt engine progress) joined the six original variants, RunFinished gained a cancelled flag, and StepFinished gained a detail failure field — all serialized only when present, so pre-existing streams parse unchanged. The additive-only rule held; this note keeps the variant inventory honest.

2026-08-24 (RF wave 2): scenario_finished gained reason (why a scenario is skipped — authored spellings start with @, mechanical prose never does; ADR-0019) and tags (the accumulated tag set, finished-event only because the cancel-skip path emits no start); scenario_started gained exclusive (the scheduler’s own bool, R11-6); step_finished gained reproduce_hint (the failing request’s redacted curl). All serialized only when present; EVENT_SCHEMA_VERSION stays 1. run_started further gained env/metadata/shuffled (ADR-0020) and rerun_of (the E2 rerun overlay) — same additive discipline.

2026-09-06 (0.18): the record is now protected the way the console already was. events.jsonl was handed a bare File, so a disk filling mid-run truncated the record while the run exited by its verdict; the record’s writer now latches its first failure and the exit funnel turns it into a system error (exit 3) through the same fold as the JUnit/CTRF and GitHub-summary write failures (escalate_environment_failures). A single SIGTERM/SIGHUP is a cancellation — the record closes normally with run_finished + cancelled — so only a second signal, a SIGKILL, or a crash leaves a truncated record (EVENTS.md). The sidecars that sit beside the record (timings.json, inputs.json) are derived aids, never a second record: the stream stays the only persisted format, and EVENT_SCHEMA_VERSION stays 1.

ADR-0009 — Error taxonomy by fault, stable exit codes, miette at the edge

Status: Accepted · Date: 2026-07-28

Context

The error model categorizes by who is at fault — User / TestFailure / System(anyhow) — with a total mapping to its stable exit-code scheme (0 ok · 1 test failure · 2 user error · 3 system error), integration-tested via assert_cmd. 2026 ecosystem consensus (blessed.rs et al.): thiserror in libraries, anyhow at the application edge; miette’s Diagnostic adds codes/help/labeled source spans for user-facing errors. Survey of backend traits (tower/sqlx/rustls/probe-rs): behind dyn, a unified error enum with boxed sources beats associated type Error. gherkin 0.16 spans are byte offsets (verified), directly convertible to miette SourceSpan (with an EOF-trailing-newline clamp; LineCol.column is char-counted — never mixed into byte math). snafu/error-stack: adopted by some large codebases, unnecessary at this crate count.

Decision

A fault-category model, extended for engines: proef-core defines

#![allow(unused)]
fn main() {
pub enum CoreError { User(..), TestFailure(..), System(..) }        // → exit 2 / 1 / 3
pub struct EngineError { pub class: EngineErrorClass,               // Infra | AssertFailed | Setup
                         pub message: String,
                         pub source: Option<Box<dyn Error + Send + Sync>> }
}

AssertFailed folds into TestFailure; Infra/Setup into System. thiserror 2 everywhere; no anyhow in library crates (only inside System’s boxed source at the edge). miette lives only in proef-cli: parse/bind/validation errors wrap into Diagnostics with labeled spans into .feature files (gherkin byte spans) and pack YAML (serde_norway locations); engine failures render the feature line + artifact span from the sidecar. Exit codes are a typed enum, pinned by CLI integration tests.

Consequences

Every failure has a fault category, a stable exit code, and a source-located rendering; engine crates stay miette-free (usable headless); the explain command reuses the same classification. Cost: two error layers (core vs engine) — justified by the seam: engines can’t know exit codes, core can’t know engine internals.

Alternatives considered

Associated type Error on engine traits — erased behind dyn anyway. anyhow everywhere — loses matchable categories that exit codes require. snafu — per-crate context ergonomics we don’t yet need; revisit if the workspace grows past ~10 crates.

Amendment — 2026-08-04 (exit code 130 documented, not a variant)

test and watch hard-exit with code 130 (128+SIGINT) on a second Ctrl-C while a run is already cancelling (ADR-0007) — the shell’s own convention for a signal-terminated process. This is a sanctioned OS-signal escape hatch, not a graceful outcome the fault-category model classifies, so it is intentionally not an ExitCode variant; ExitCode stays the total 0/1/2/3 mapping above.

Amendment — 2026-09-06 (every signal, and undelivered output)

Ctrl-C, SIGTERM and SIGHUP all take the graceful cancel (ctrlc’s termination feature), so a CI job timeout or docker stop is a cancelled run — exit 1 with a complete record — not a kill; the 130 hard exit above fires on a second signal of any of the three, and the handler carries no signal identity, so one code covers them all. And output proef could not deliver never looks like success: a failed write of the run record, of a JUnit or CTRF file, or of the GitHub step summary re-classifies the exit to System (3) through one fold, escalate_environment_failures, beside the stdout latch this taxonomy already covered.

ADR-0010 — Artifacts as contract: emitted .hurl is the executed input

Status: Accepted · Date: 2026-07-28

Context

The backend team’s hurl corpus makes .hurl the interop format. The spike proved generated artifacts run identically under the prototype engine and the stock CLI — and that keeping two implementations honest requires differential testing. Embedding hurl (ADR-0001) enables something stronger: the artifact and the executed input can be the same bytes. Verified seam facts that shape the mechanics: run_entries creates its HTTP client per call (fresh connections; cookie jar seedable only via Netscape-format file; variables chain losslessly via HurlResult.variables); per-entry [Options] override batch-level RunnerOptions defaults (clone-then-override, verified).

Decision

For every scenario, the emitter produces canonical .hurl text; that exact text is what parse_hurl_file + run_entries execute — drift between artifact and execution is structurally impossible. Alongside each artifact: a sidecar map (<slug>.map.json: entry ↔ feature file/line/step text, optional flags, capture names, batch boundaries) and, when World/global values are referenced, a generated <slug>.vars file so the backend team can replay with hurl --variables-file. optional: entries carry an # optional marker comment (no hurl equivalent — the runner segments around them). Execution batches maximally: one run_entries call per scenario unless optional: boundaries or interleaved other-engine steps force a split; when a split occurs, variables chain via HurlResult.variables and cookies (if used) round-trip via a Netscape temp file behind a SessionState struct. Queued upstream patch #1 (per ADR-0003): run_entries accepting &mut http::Client — verified two-call-site change — which erases per-segment connection/cookie costs entirely. Artifacts never contain secret values (ADR-0005). Artifact output: per-run under .proef-runs/<id>/artifacts/ plus proef artifacts for a stable CI hand-off directory.

Consequences

Debugging = opening the artifact with tools the team already knows; the backend corpus and proef packs stay mutually copy-paste-able; --dry-run validation includes parsing the real artifact with the real parser. Cost: canonical formatting is a compatibility surface (snapshot-tested); segmented scenarios pay reconnection costs until patch #1 lands upstream.

Alternatives considered

Internal-only IR execution with optional export — loses the same-bytes guarantee and demotes artifacts to lossy exports. Differential-oracle architecture (v1 plan) — superseded by ADR-0001; preserved in research/ as the static-distribution fallback.

ADR-0011 — Fixture server is synchronous tiny_http, not axum

Status: Accepted · Date: 2026-07-28 (decision made at M3; recorded as an ADR during the post-M5 hardening pass — it had been noted only as inline errata in TECH-SPEC §14 and TESTING-STRATEGY §2)

Context

TESTING-STRATEGY as originally written named axum for the integration fixture server (proef-fixture). Axum requires a tokio runtime. The workspace bans the tokio runtime outright (ADR-0006/ADR-0007: engines and core are sync; only tokio-util with default-features = false enters the tree, for CancellationToken). A dev-dependency would not leak into shipped binaries, but it would put a full multithreaded async runtime into every cargo nextest process, contradict the “no async machinery” line we enforce with cargo deny bans, and normalize exactly the dependency the ADRs exclude.

The fixture’s needs are modest: a handful of JSON endpoints, per-env state isolation, deterministic token-driven delayed visibility, cookies, one deliberately slow route and one deliberately malformed one — all exercised by at most a dozen concurrent scenarios.

Decision

proef-fixture is built on tiny_http (synchronous, dependency-light) with a plain std::thread accept loop and an Arc<Mutex<_>> state map keyed by the X-Proef-Env header. It starts on an ephemeral port (Server::http("127.0.0.1:0")), reports its base URL, and shuts down via an AtomicBool + recv_timeout poll. Portability note: address introspection uses ListenAddr::to_ip() — the Unix variant of ListenAddr exists only on unix targets, so matching on it breaks the Windows build.

Consequences

• The workspace stays runtime-free end to end; the deny-list ban on tokio stands without a dev-dependency exception.
• The fixture is one file, debuggable with a thread dump, and starts in microseconds — each integration test spawns its own isolated instance.
• No HTTP/2, no TLS, no streaming in the fixture — acceptable: the engine’s HTTP behavior is hurl/libcurl’s concern, not the fixture’s; the fixture only scripts responses.

Alternatives considered

• axum (as originally specced): rejected — drags in the banned tokio runtime; every capability the fixture needs is available synchronously.
• hyper in blocking mode / raw std::net: more code for no additional fidelity.
• Out-of-process fixture binary: slower startup, port coordination, and a lifetime-management problem tests would have to solve; in-process tiny_http gives free isolation per test.

Amendment — 2026-07-31 (dev-loop CLI binds the advertised default port)

Fixture::start() stays ephemeral (127.0.0.1:0) — the integration suite spawns a dozen concurrent instances and each needs its own port. But the shipped proef.toml advertises base = http://127.0.0.1:8787, so a first-time cargo run -p xtask -- fixture on a random port left the default unreachable and forced a PROEF_BASE_URL export. The dev-loop CLI (xtask fixture, one instance at a time) now calls the new Fixture::start_on(port) to bind 8787 by default (override: ... -- fixture <port>), falling back to an ephemeral port — with the PROEF_BASE_URL line printed — only when 8787 is busy. The library API and the per-test isolation described above are unchanged; only the human entry point picks a stable, documented port.

ADR-0012 — Project configuration & environments in proef.toml

Status: Accepted · Date: 2026-07-30

Context

Test files must stay pure prose — no URLs, no environment data, no variable definitions (operator requirement: “test files for testing, not for variable definitions”). Before this, the only non-secret variable source was the in-feature # baseURL: directive plus ${env:…}, which put configuration inside the .feature files. Suites also had to be given an explicit path on every proef test invocation.

Decision

proef.toml (already the config file — ADR-0004 kept TOML for it) gains variable-bearing sections and per-environment override profiles, modeled on the Cloudflare-Wrangler wrangler.toml [env.<name>] pattern (and Cargo [profile.*]):

• [url] and [vars] — non-secret variables, referenced in packs as ${url:<key>} and ${vars:<key>} (new lower-time resolver namespaces, within the ADR-0005 ${…} tier).
• [env.<name>.<section>] — per-environment overrides that deep-merge over the base tables, key by key: [env.prod.url] / .vars override variables, .http / .run override runner settings. Unlisted keys inherit the base.
• --env <name> / PROEF_ENV selects the active environment.
• [run] suite — the default suite path, so proef test needs no path argument; the tests/ directory is the zero-config fallback convention.

Core stays sans-IO: the CLI loads proef.toml, validates the --env name, deep-merges, and injects the resolved scope as LowerCtx::config_vars (keyed "<namespace>:<key>"). The resolver reads that injected map — it never touches a file. Secrets remain on their own encrypted channel (${secret:…}) and never appear in proef.toml.

Consequences

• Feature files become pure prose; URLs / credentials / env data live in one external file — the single variable-definition mechanism (the legacy # key: directive was later removed; see the 2026-07-31 amendment).
• A referenced-but-undefined ${url:…} / ${vars:…} is a user error at lower time (proef::resolve::missing_config_var) — the same strictness as ${env:…}.
• Deep-merge (not Wrangler’s non-inheritable vars) means an environment lists only deltas; this is the deliberate divergence from Wrangler’s known footgun.
• proef test / flows / artifacts accept an optional path plus a --env flag.

Alternatives considered

• Per-environment files (proef.<env>.toml, Spring / dotenv style) — cleaner git diffs per env, but more files; the single-file [env.<name>] model keeps one mental model and reuses the existing loader. Recorded as the fallback if one env grows large.
• Wrangler-exact non-inheritable vars — rejected: forces re-listing every var per env (the documented Wrangler footgun); deep-merge is the more ergonomic default.
• A ${baseURL} magic bare name — rejected: namespaced ${url:base} is collision-free and consistent with ${secret:…} / ${env:…} (one way to do one thing).

Amendment (2026-07-31) — the # key: directive mechanism is removed

The original decision kept the in-feature # key: value directive (e.g. # baseURL:) working “for one-off per-file overrides.” That left two ways to define a variable — a directive inside a .feature file, and [url]/[vars] in proef.toml — violating one-way-to-do-one-thing (operator: “variables should be defined in 1 way, not multiple ways”). The directive mechanism is therefore removed:

• FeatureFile::directives, collect_directives, lower::resolve_directives, and the ResolveCtx::directives scope are deleted. ${…} plain-name resolution is now args > defaults only; feature files carry no variable definitions.
• Config becomes the single variable source. Config values may themselves embed ${env:NAME:-default} (resolved recursively), so the env-override + default that # baseURL: ${env:PROEF_BASE_URL:-…} provided is preserved as base = "${env:PROEF_BASE_URL:-…}" under [url].
• proef.toml is now discovered by walking up from the working directory (like cargo/git), so config is found from any subdirectory (needed once the value moved out of the self-contained feature file — e.g. the libtest-mimic harness invokes proef from its own crate dir).

# comment lines before Feature: remain valid gherkin comments; they are simply no longer parsed as directives.

ADR-0013 — Typed macro parameters

Status: Proposed — recommendation: defer (the shape below is recorded for when a real need appears) · Date: 2026-08-02

Context

A macro declares params and binds {capture} values from prose, data-table rows, defaults, and use:…with:. Every arg is an untyped string today: the matcher only checks that a capture names a declared param (matcher.rs), never that the value looks like what the step expects. A mistyped value (the record abc is fetched where a UUID was meant) is caught only at hurl run time — if at all — not by --dry-run.

Cucumber Expressions solve this with typed parameters ({int}, {uuid}, custom types) validated before execution. proef wants the same shift-left check, expressed within its architecture. Round-2 code validation (IMPROVEMENT-PLAN §12, item N1) established three hard constraints:

1. Placement must be the declaration site, not the pattern. Three of the four arg sources are not captures (data-table rows, defaults, with:), and use:-only macros have no pattern at all — so an inline {name:type} form could not type them, and a type buried in the match: string is invisible to proef schema (schemars).
2. The two-tier variable rule caps the value. An arg may legitimately be ${…} / {{…}} that only resolves at lower/run time (ADR-0005). Type-checking such an arg at bind time would false-positive, so the check must skip any raw arg containing ${ or {{ — it is a best-effort lint over literal args, not a runtime type system.
3. One-canonical-way forces a single params spelling. params is a YAML sequence today ([q, index]); adding a typed spelling alongside it would be two ways to declare params — the golden-rule violation that killed the templates: alias and the # key: directive.

Decision

• params becomes a name → type map: params: {q: uuid, index: int, note: any}. A missing/any type means “declared, unchecked” (today’s behaviour). Parsed with a custom Deserialize (not an untagged enum — CLAUDE.md bans those near the arbitrary_precision footgun). This is a breaking pack-shape change — every existing pack migrates params: [q, index] → params: {q: any, index: any}, with no alias (consistent with the templates:→macros: and # key: removals).
• The type set is a closed registry — any, int, number, uuid, email, iso8601, word — modelled on the fake::GENERATORS registry (fake.rs), validated at load so an unknown type is a pack error. No user-defined types in v1 (revisit if a real need appears).
• Validation lands at three sites: bound captures + data-table rows at bind time (proef::bind::param_type_mismatch), and static defaults / with: values at load time (next to the existing default_not_param / unknown_with_key checks). All checks are skipped when the raw arg contains ${ or {{ (constraint 2). Datetime types parse via jiff (sans-IO — literal parse reads no clock).
• proef schema reflects the typed params shape (constraint 1 satisfied).

Consequences

• Breaking: every pack’s params migrates to the map form. A one-time mechanical change, called out in CHANGELOG and GETTING-STARTED; the error-corpus gains a bind__param_type_mismatch case.
• Best-effort, not a guarantee: literal args are checked; ${…}/{{…}} args are not. This must be documented so authors do not read it as a type system — its value is catching typos in literal prose at --dry-run, not enforcing runtime types.
• --dry-run gains a real new class of caught mistake; proef schema autocomplete gets richer.
• The honest cost/benefit: a breaking migration for a best-effort literal-args lint. Recorded so the trade is deliberate, not incidental.

Best-practice basis & recommendation

Research into the field (2025–2026) is decisive on form and honest about worth:

• Form (if built): the industry norm is declaration-site typing referenced by name, never a full type spec inline. Cucumber Expressions put only a bareword type-name in the pattern (indexing a registry); SpecFlow/Reqnroll type by the binding method’s return type; Bruno (v4, 2024) added declaration-site typed variables with a string default. A name → type map is the canonical shape (JSON Schema properties, OpenAPI). The inline {name:type} form (behave/pytest-bdd) only works because their “declaration” is code, not a schema — with a schema it loses tooling. So the decision above (single-shape map, closed vocabulary) is the correct form, and the list-or-map union is the worst option on every axis (permanent double surface, weaker autocomplete, bifurcated examples — the exact ambiguity “one canonical way” forbids). ESLint’s flat-config hard break (v9→v10, bounded deprecation then cut) is the precedent for a single-owner format choosing a clean break over an indefinite dual-shape.
• Worth (candid): a best-effort lint over literal values is the mypy/TypeScript gradual-typing bargain — genuinely useful for shallow typos, provided it degrades to “unchecked,” never “pass,” on the parts it can’t see. But an API test runner’s args are deferred-heavy (base URLs, captured ids, run-time tokens — all ${…}/{{…}}), the exact population the lint cannot check, which shrinks the realized benefit. proef’s own corpus bears this out: today’s params are overwhelmingly string-ish, so the high-value types (uuid/int/iso8601) would rarely fire. Karate — the nearest-neighbour API tool — never types inputs at all; it types responses.

Recommendation: defer. The clean, best-practice-aligned move for proef right now is not to add a low-current-value lint behind a breaking format change; it is to record the correct shape (above) and adopt it if a pack corpus emerges where structured literal args are common. Building it earlier is defensible only in the disciplined form above — never as a list-or-map union. (If the operator wants it now regardless, implement the single-shape map with the honest-degradation discipline; the migration is mechanical.) (Cucumber Expressions, Reqnroll conversions, Bruno typed variables, OpenAPI 3.1, ESLint 10 removal, Karate schema validation)

Alternatives considered

• Inline {name:type} (Cucumber-Expression style) — rejected: invisible to proef schema, mis-binds in the tokenizer ({q:uuid} becomes a capture literally named q:uuid), and covers only 1 of the 4 arg sources.
• params accepts either a sequence (untyped, non-breaking) or a map (typed), via custom Deserialize — the non-breaking alternative. Rejected under one-canonical-way (two shapes for one field), but it is the fallback if a breaking migration is judged too costly for a best-effort lint. (This is the open fork for the operator.)
• Keep params untyped (do not do N1) — the status quo; the shift-left gap stays. A legitimate choice given the cost/benefit above.
• User-defined type registry — deferred; a closed set is simpler and covers the common shapes.

ADR-0014 — Suite-level setup & teardown

Status: Accepted · Date: 2026-08-02

Context

Gherkin Background runs steps once per scenario, and a persistent global store threads saveAs: global captures across scenarios and runs (ADR-0005). What is missing is a once-per-suite phase: authenticate once and share the token, seed a fixture before the run and tear it down after. Cucumber solves this with @BeforeAll/@AfterAll hooks; Karate with callSingle. Round-2 code validation (IMPROVEMENT-PLAN §12, item N3) fixed the constraints:

• Tags never reach the sans-IO core runner. ScenarioSpec/ScenarioOutcome carry no tags; @quarantine is computed at the CLI edge (exec.rs) as a non_gating set. So a lifecycle phase must be orchestrated at the CLI edge, not inside the runner.
• Only saveAs: global promotions cross scenarios, and each pooled scenario snapshots the global store at prepare time — so a setup phase must finish and merge its globals before the parallel pool starts, or early scenarios miss the seeded value.
• An assert-failed setup must not be masked. A setup step that fails an assertion classifies as a test failure (fault: None), which the worst-wins fold would let through as exit 1 without aborting the pool — every scenario would then run against un-seeded state and cascade confusing failures.

Decision

• Two new proef.toml keys — [run] setup and [run] teardown — each naming a feature file. setup runs (its scenarios, in order) once before the suite pool; teardown once after. State reaches the suite through the existing saveAs: global store; setup completes and merges before the pool is built.
• Orchestrated in execute() around the parallel pool (CLI edge), never in the sans-IO core. The setup/teardown feature is excluded from build_specs so it never also runs as an ordinary scenario, and it is invisible to --tags/--scenario/--rerun.
• Failure semantics (exit-code contract, ADR-0009) — validated against Playwright, Jest, k6, and pytest (see basis below):
  • A setup failure of any kind aborts the run before the pool launches and is never masked. It maps to a user (2) or system (3) fault, not a test failure (exit 1) — a broken fixture is not a failing test, the same distinction Playwright draws between a “clear setup error” and a “cryptic test failure.”
  • Teardown runs only when setup succeeded (gated on setup-success, not pool-success): k6 and pytest-yield both skip teardown when setup threw, because tearing down un-created state is itself a fault source. The setup feature is responsible for cleaning its own partial state on failure.
  • Teardown does run after the pool even when scenarios failed (Playwright/pytest/k6 all do), so cleanup is reliable.
  • A teardown failure is loudly reported and yields a distinct non-zero signal (a system/cleanup fault, exit 3) — never a silently-green suite (no mainstream tool masks a teardown failure). It stays distinct from a test failure (exit 1): a cleanup hiccup does not mean the API under test is broken, but it is not hidden.
• --dry-run is unaffected: it validates only (never calls the runner), and the setup/teardown features are validated like any other feature but never executed.

Consequences

• A genuine once-per-suite lifecycle, expressed declaratively (Hurl prose in a feature), with no code hooks — the glue-code path proef exists to avoid.
• Exactly one setup and one teardown mechanism — a [run] construct, not a second Background concept and not a tag. The setup feature is authored like any suite feature and reuses the whole pipeline (bind → lower → emit → execute).
• Touches the pinned exit-code tests: a failing setup gates the run; new assert_cmd cases cover the short-circuit and the “setup fault is never masked” invariant.
• Product-neutral: the setup steps are ordinary prose bound to macros; nothing about the mechanism assumes a particular backend.
• Auth-once boundary (explicit): the pattern that motivates suite-setup elsewhere — “authenticate once, share the token” (Playwright storageState, Karate callSingle) — is largely already covered in proef by a pre-set secret (proef secret set / PROEF_SECRET_*) resolved per scenario. Sharing a runtime-obtained secret through setup would collide with the invariant that saveAs: global refuses secret-valued captures (ADR-0005). Promoting a runtime capture into the secret channel is therefore out of scope for this ADR: the global store carries only non-secret setup state (seeded ids, fixture config). Recorded so the boundary is explicit rather than discovered later.

Amendment — 2026-08-10 (cancellation, and what --dry-run validates)

This ADR was specific about a failing setup and a failing teardown, and silent on the operator interrupting the run. That silence read as considered when it was not, and the behaviour it left was the opposite of this ADR’s premise: on Ctrl-C the teardown phase ran with the already-cancelled token, so every teardown scenario resolved Skipped, phase_failed ignored a phase that only skipped, and cleanup silently never happened.

Decision — cleanup outlives the interrupt. Teardown runs on its own, independent token, never the run’s. On Ctrl-C the pool stops at its batch boundary, the operator is told cleanup is running, and teardown completes — so an interrupted run does not strand whatever setup created.

Note the word: independent, not “child”. A child token cancels when its parent does, which is precisely the behaviour being fixed; child_token() would have re-implemented the bug.

ADR-0007’s responsive interrupt is preserved by the escape hatch that already existed: a second Ctrl-C hard-exits (130) out of teardown as out of anything else, and the announcement says so. A hung teardown is bounded by the same batch budgets and watchdog as any other phase, so this needs no timeout of its own.

This is the standard graceful-shutdown shape — first signal begins bounded cleanup, second forces exit — rather than the test-runner norm, which is worse: Jest does not call globalTeardown on Ctrl-C (#6029) and Go does not run t.Cleanup on SIGINT (#41891), both long-standing complaints rather than settled design.

Corollary — a phase that only skipped is a failure. A skipped phase carries no fault, so the worst-wins fold passed it silently; that is the shape that hid cancelled cleanup. A setup that completes no scenario now aborts the run (the suite would otherwise execute against state setup never created), and a teardown that completes no scenario is reported and fails the run. The setup abort is also what keeps teardown gated on setup-success as this ADR requires: the early return is the gate, so teardown never dismantles what was never built.

--dry-run now does what this ADR already claimed. The Decision above says the phase features are “validated like any other feature but never executed”. They were validated by nothing — --dry-run never read the keys — so a broken [run] teardown surfaced only after a full suite had run, while the identical mistake in [run] setup failed in milliseconds. Both phases are now validated by one loader shared with execute, which also pre-flights teardown before the pool, so the same mistake costs the same either way and a bad path is a user error (2) rather than a blanket system fault (3).

Best-practice basis

Config-key-names-a-file is the dominant model (Playwright globalSetup/globalTeardown, Jest globalSetup, Vitest) — the [run] table already supplies the “once per run” qualifier, so [run] setup reads like globalSetup without repeating global. Passing state out-of-band through a serialized shared store (not live memory) is universal — Playwright’s storageState file, Jest’s env-var workaround, Karate’s callSingle cache, k6’s returned data — and proef’s global store is the direct analog. Setup-completes-before- workers and setup-failure-aborts are unanimous. The refinements above (teardown skipped on setup failure; teardown failure is non-zero, not silent) come straight from k6/pytest and the Playwright/pytest “teardown is not silently swallowed” norm. (Playwright global setup, Jest config, k6 lifecycle, pytest #2508, Karate callSingle)

Alternatives considered

• @setup / @teardown tags — rejected. A tag means “filter/select”; overloading it with “change execution phase” is a second meaning, and tagged setup scenarios would entangle with --tags/--scenario/--rerun/flows/name-dedup with undefined ordering between two @setup scenarios. A [run] construct is single, explicit, and ordered.
• Code hooks (Before/After functions) — rejected. Arbitrary-code hooks are exactly the imperative glue the declarative-macro model avoids, and they break the sans-IO boundary. The legitimate need (setup/teardown IO) is met with Hurl steps instead.
• Per-scenario Background only (status quo) — does not cover once-per-suite work; authenticating in every scenario’s Background is wasteful and cannot seed shared state that must exist before the first scenario.
• A dedicated [setup]/[teardown] top-level table — rejected as heavier than needed; these are run-orchestration knobs, so they belong under [run] beside suite/jobs.

ADR-0015 — Injected observability timestamps (run-level timeline)

Status: Accepted · Date: 2026-08-02

Context

The HTML report has a per-scenario timing waterfall (IMPROVEMENT-PLAN §12, N6a) derived purely from each step’s duration_ms. It cannot show cross-worker occupancy — which scenarios ran concurrently, on which of the --jobs workers — because the sans-IO core reads no clock and the event stream carries no wall-clock timestamp or worker identity.

The core’s purity is deliberate (deterministic snapshots/properties). ADR-0012 established the escape valve: values the core must not compute (config) are injected at the CLI edge. run_id is already such a value — the core carries it (RunStarted.run_id) but the CLI generates it. The same pattern extends to timing.

Decision

• Add two optional, additive fields to the ScenarioStarted and ScenarioFinished events: timestamp_ms: Option<u64> (milliseconds since the run began) and worker: Option<u64> (0-based worker index). Both are #[serde(default, skip_serializing_if = "Option::is_none")], so old records parse unchanged and single-threaded/None cases serialize identically (ADR-0008 additive-only). They are kept off RunStarted, whose exact wire bytes are pinned.
• The core leaves them None — it carries fields it never fills, exactly as it carries a run_id it never generates (sans-IO preserved). The CLI wraps the event sink in a stamping closure: EventSink::new(move |ev| inner.emit(&stamp(ev))). stamp runs on the worker thread (emit is synchronous, called from the scenario worker), so it reads the run-start Instant and maps thread::current().id() → a stable 0-based index there, at the edge. The core never sees a clock or a thread id.
• Errata (2026-08-11): only ScenarioStarted carries a worker. As implemented, ScenarioFinished is emitted from the main dispatcher thread, not the worker that ran the scenario, so stamping a thread index there would name the wrong one; it carries the end timestamp and worker: None. The worker identity comes from ScenarioStarted, which is emitted on the worker thread, and the timeline pairs the two. EVENTS.md has always described it this way — the bullet above did not.
• The HTML report gains a run-level timeline: a lane per worker, each scenario a bar from its start to its finish timestamp — the Gantt/occupancy view. It renders only when the stamps are present; a record without them falls back to the N6a per-scenario waterfall alone. The view stays a pure function of the (now richer) event record.

Consequences

• Core stays sans-IO; the injection is the established run_id/config pattern.
• The JSONL record gains observability fields, additively (ADR-0008 — still the one record; no sidecar timing file).
• Snapshot honesty: old records parse unchanged, but every event in a new run now carries a (non-deterministic) timestamp_ms, so the reference_event_stream snapshot changes — it needs a new insta filter ("timestamp_ms":\d+ → 0) plus a deliberate cargo insta review. This is allowed (the same deliberate-acceptance policy as emitter changes), and is called out rather than glossed. The HTML snapshot likewise regenerates.
• Enables the cross-worker timeline; the derived HTML view remains pure over the record.

Alternatives considered

• Stamp inside the core — rejected: reading a clock in proef-core breaks the sans-IO invariant that makes snapshots and property tests deterministic.
• A second sidecar timing file — rejected: ADR-0008 makes the JSONL event stream the record; a parallel timing artifact is a second record format.
• Derive occupancy from duration_ms alone (no injected fields) — impossible: without an absolute clock there is no way to know which scenarios overlapped in wall-clock time; only the sequential per-scenario waterfall (N6a) is derivable, which is why it shipped first.
• Wall-clock (unix ms) instead of run-relative — rejected: run-relative starts the timeline at 0, is cleaner to render, and avoids putting absolute wall-clock in the record.

ADR-0016 — OpenAPI → suite generator (scope decision)

Status: Proposed — recommendation: defer; the oracle/drift mode is permanently rejected regardless · Date: 2026-08-02

Context

A recurring market expectation of a “serious” API-testing tool is to generate tests from an OpenAPI spec (Schemathesis, Dredd, Step CI). proef has none, and the Round-2 scope stress-test (IMPROVEMENT-PLAN §12.5-A) flagged it as the single biggest capability a reviewer would call missing — while also being the one that sits on proef’s permanent charter line.

PRD §3 names, as a permanent non-goal, “API mocking/contract testing”, and the operative expansion (IMPROVEMENT-PLAN §3) spells it: “contract testing (OpenAPI drift / Pact / Schemathesis).” Yet the validation established that a generate-then-freeze framing — read the spec once at the CLI edge, emit concrete editable .feature + macro packs, then execute them through the normal deterministic pipeline — technically clears the sans-IO and determinism objections (ADR-0012 is exact precedent for IO-at-the-edge). So the question is genuinely open and needs an ADR to settle the boundary rather than let it erode.

Decision

The bright line (normative, decided either way): an OpenAPI spec may be a one-shot seed — read once to emit prose + packs the author then owns, edits, and maintains — but it may never become a recurring oracle: re-read on every run, used to drift-check the live API against the spec, or gated on a generated-vs-committed diff. That is OpenAPI-drift contract testing (PRD §3), and it is permanently rejected. Concretely, a generator, if it ever exists, must have no --check/--verify/--diff mode and must never be consulted after the initial emission.

On the narrow scaffolder itself: defer (recommendation). A one-shot proef generate --openapi spec.yaml -o suite/ that scaffolds editable prose + packs (the shipped bind::unbound_step stub-gen at suite scale) is defensible in-charter under the bright line, but is not worth building now for the reasons below. If a concrete need emerges, it may be adopted only as: CLI-only (never proef-core), deterministic given a seed, output fully owned by the author after emission, and bound by the line above.

Consequences

• Deferring records the boundary so the omission is an intentional, documented call — and so the oracle mode is now explicitly foreclosed, not merely absent.
• Output-quality tension (the strongest argument against). OpenAPI describes an API in endpoint terms (POST /records/{id}/notes → 201); proef’s value is business prose (“a member posts a note to a record”). A generator produces the former, which the author must rewrite into the latter — so it saves little over hand-authoring and risks a corpus of mechanical prose that undercuts proef’s prose-first premise.
• Dependency + direction cost. A robust OpenAPI 3.0/3.1 parser (the dialect split, $ref, oneOf/allOf, discriminators) is a heavy, churny supply-chain surface to pin and clear through cargo-deny/cargo-audit. It also introduces a new inward-ingestion direction — proef has only ever flowed outward (ADR-0010: artifacts are emitted, never imported).
• One-canonical pressure. The first emission is cargo new-style and harmless; the risk is re-generation, which would give packs a second maintenance path (regenerate vs hand-edit). The one-shot-seed discipline is not machine-enforced, so it must be a documented rule.
• Already partly covered. The #9 stub-gen convenience emits a paste-ready match:+hurl: macro skeleton for an unbound step today — the same idea at step scale, without any spec dependency.

Alternatives considered

• Full OpenAPI-drift checker (Schemathesis/Dredd-style: spec re-consulted per run to catch divergence) — permanently rejected. It is verbatim the PRD §3 non-goal, non-deterministic against a live API, and the reason the bright line exists.
• Property/fuzz-case generation from the spec (Schemathesis-style negative cases) — out. Runtime fuzzing breaks the sans-IO/deterministic-artifact invariants; frozen generated fuzz cases are a variant of the one-shot scaffolder and share its deferral.
• Build the narrow scaffolder now — declined on cost/value (output quality, dependency weight, inward direction) despite being technically in-charter under the bright line. Buildable later without a new ADR, provided it obeys this one.
• Status quo (no generator) — the recommended near-term state; authors write prose against the macro vocabulary, aided by stub-gen for missing steps.

Best-practice basis

The generate-then-freeze vs re-consult-the-oracle distinction, the sans-IO-clearance via the ADR-0012 IO-at-the-edge precedent, and the “one --check flag from the non-goal” risk are from the Round-2 scope validation (IMPROVEMENT-PLAN §12.5-A). Schemathesis and Dredd are the reference generators; both re-consult the spec as an oracle, which is exactly what this ADR forecloses. (Schemathesis, Dredd, PRD §3, IMPROVEMENT-PLAN §12.5-A.)

ADR-0017 — proef lsp language server

Status: Accepted · Date: 2026-08-03 (implemented 2026-08-03) Design spec: docs/superpowers/specs/2026-08-03-proef-lsp-design.md

Context

Feature/pack authoring has no editor support — no live diagnostics, no jump-to-macro, no step completion. Weak IDE support is a documented gap in Karate and the BDD field (IMPROVEMENT-PLAN §10), and it is the one substantive unbuilt item on the Round-1 roadmap (§5 item #11, graded ✅ FITS). proef is unusually well-placed for it: its analysis is already headless and sans-IO — front::run yields the same Diag objects (stable code, byte span, severity, help) regardless of driver — so an LSP is a second front-end over existing analysis, not new analysis.

Decision

Build a proef-lsp crate (surfaced as the proef lsp subcommand) as a server-only, generic-LSP stdio binary delivering the full v1 feature set — diagnostics, go-to-definition, completion, and find-references. Key choices:

• Sync lsp-server + lsp-types (rust-analyzer-family), not an async stack — honouring ADR-0006’s tokio ban.
• Whole-suite model, recomputed wholesale on change (debounced), not an incremental (salsa-style) index. Mature LSPs index the whole project because cross-file features are core; they carry incremental machinery only because their scale is large. proef’s suite is tens of small files and the pipeline is milliseconds, so wholesale recompute buys the cross-file capabilities (live cross-file re-validation, find-references) at per-document simplicity. Incremental is YAGNI until a real perf ceiling.
• Second front-end over sans-IO core. The one enabling refactor: an injectable source provider so file discovery + reading go through a trait (disk for the CLI, overlay-then- disk for the LSP), and a collect-all mode (accumulate every diagnostic instead of fail-fast). Both keep proef-core sans-IO — the IO is injected, the ADR-0012 pattern.
• Server-only v1; a VS Code extension is deferred (off proef’s pure-Rust brand; a thin wrapper can follow).

Consequences

• A new crate + two deps, pinned as shipped: lsp-server 0.7.9 and lsp-types 0.97.0 (MIT/Apache — clean under cargo-deny); the provider trait moves the proef-core public-api snapshot (a deliberate, reviewed change).
• lsp-types 0.97 models document URIs as its own Uri type (RFC-3986), not url::Url. The 0.97 line dropped the url dependency, so the converter and every handler key documents on Uri (parsed/compared as an RFC-3986 string), never url::Url — a change from the pre-0.97 API that would silently fail to compile against the old assumption. Superseded — see the amendment below.
• The front-end refactor (injectable provider + collect-all) touches front.rs and its callers; the CLI path must stay behaviourally identical, guarded by the existing integration
  • snapshot suites.
• Highest bug-risk surface is the byte↔UTF-16 + source-normalization (BOM, trailing newline) converter; mitigated by property tests and by reusing tests/errors/ (real spans/ranges).
• A genuine competitive differentiator, and the natural depth move after the v0.4.0 breadth work — but a multi-week (L) effort; sequenced so each step (handshake → provider/collect-all/ converter → diagnostics → definition → completion → references) is independently testable.

Amendment — the types crate is gen-lsp-types, and Uri is url::Url again

lsp-types stopped receiving releases after 0.97; gen-lsp-types is the maintained successor, generated from the LSP metamodel. proef depends on it under the original name — lsp-types = { version = "0.11", package = "gen-lsp-types", features = ["url"] } — which is rust-analyzer’s own aliasing pattern and leaves every lsp_types:: path in the crate untouched. The decision above is unchanged: still sync, still the rust-analyzer family, still server-only.

Two consequences change:

• Uri is url::Url. The generated crate gates its URI type behind features (url, fluent-uri, or a bare String newtype). Choosing url costs nothing — the embedded hurl engine already pulls url into the workspace graph — and buys back from_file_path/to_file_path, the native-path bridge the 0.97 Uri had no equivalent for and that documents.rs therefore hand-rolled (drive-letter prefixes, segment joining, percent-encoding). That bridge is deleted; the wrapper that remains exists only to pin the pipeline’s source-name identity rule. The consequence the original ADR recorded is retired with it, and the swap removes three crates (lsp-types, fluent-uri, serde_repr) while adding none.
• Methods are enums, not string constants. Request::METHOD is now an LspRequestMethod<'static> whose From<&str> falls back to Custom, so dispatch compares enum values and an unrecognised method lands in a variant rather than matching nothing.

One behaviour moved: what counts as a malformed document URI. fluent-uri rejected a raw space; url percent-encodes it. The malformed-params test therefore asserts on a schemeless URI, which url genuinely rejects — the guarantee under test (a bad URI is answered with InvalidParams, never a dead server) is unchanged.

Alternatives considered

• Incremental (salsa) index — rejected for v1: it is the complexity mature LSPs accept for large scale, which a test suite does not have. Buys nothing at proef’s scale; can be added later behind the same interface.
• Per-document analysis (no cross-file model) — rejected: it would leave stale diagnostics when a pack changes and cannot do find-references; the wholesale-recompute model gives the cross-file behaviour without the index cost.
• Async LSP stack (tokio + tower-lsp) — rejected: violates ADR-0006; lsp-server is the sync, rust-analyzer-proven alternative.
• Ship a VS Code extension in v1 — deferred: a TypeScript/npm deliverable off the pure-Rust brand and a separate release surface; generic-LSP config covers Neovim/Helix/Emacs/Sublime today, and the extension is a thin follow-up.
• Diagnostics-only or diagnostics+go-to-def MVP — considered; the operator chose the full feature set for v1 (the complete authoring experience), sequenced internally so value still lands incrementally.

Amendment — 2026-08-04 (go-to-definition gaps closed)

v1 go-to-definition resolved only feature step → macro, landing on the macro’s name key; two narrower targets were cut and recorded only in a source comment. Both are now implemented: a use: reference inside a pack jumps to the macro it names, and either path lands on the macro’s match: line when one is locatable (falling back to the name key for use-only macros). Both are best-effort text-scan locators in proef-core::pack::locate, indexed at analyze time, following the existing sans-IO/text-scan idiom already used there — no parser change.

ADR-0018 — Named hurl fragments: a second macro body form

Status: Accepted · Date: 2026-08-11

Context

ADR-0004 made a macro step’s HTTP payload a raw hurl block embedded in the pack YAML. Field evidence says that was right: a real 844-line, 14-file hurl corpus was ported onto proef through the paste path with 100% coverage — all 844 lines used only [Asserts] (75) and [Captures] (15), with seven ordinary predicates, every one passing through untouched (OPEN-FINDINGS §“Positive evidence”). Nothing here weakens that.

Two things the embedded form cannot do, both structural rather than incidental:

1. The block is not valid hurl. It carries ${url:…} / ${secret:…}, so parse_hurl_file only ever sees it after a probe substitution that tries {{probe}}, then 1, and accepts whichever parses (pack/validate.rs, probe_lower). Editors, hurl’s own tooling, and hurl itself see a file they cannot read.
2. A block has no name, so it cannot be shared. use: composes macros, not payloads; two macros wanting the same request duplicate its text. For a corpus somebody else owns, the only route in is transcription — and a transcript drifts from its original the day after it is made.

The adoption question the worklist is now on (M1–M3) is not “can a suite be ported” — it can — but whether a team can keep both suites alive long enough to trust the new one. That needs one source of truth, not two copies.

Decision

A macro step’s body is hurl: | (as today) or ref: <fragment>, never both. A fragment is one hurl entry in a real .hurl file, named by a comment directly above it:

# tests/hurl/admin.hurl — runs under stock hurl, unmodified
# @proef admin.search
GET {{base}}/api/v1/admin/search/{{index}}
Authorization: Bearer {{apiToken}}
[Query]
q: {{q}}
HTTP 200
[Captures]
recordId: jsonpath "$[0].id"

bind:                                    # pack scope
  base:     ${url:base}
  apiToken: ${secret:apiToken}
macros:
  searchRecords:
    params: [q, index]
    defaults: { index: records }
    bind: { q: "${q}", index: "${index}" }   # macro scope (quote in flow style)
    steps:
      - ref: admin.search

• The annotation carries a name and nothing else, permanently. No retry=, no key/value growth. A comment holding one identifier and zero behaviour cannot become a second configuration language, needs no parser or schema of its own, and cannot drift from the YAML. All orchestration stays in the pack.

• Names are free-form dotted, globally unique (as macro names already are), with file.hurl#name as a disambiguator (as pack.yaml#name already is).

• One annotation claims exactly one entry. Forced, not preferred: a fragment reused by several macros must be composable into any position of any of them, and a multi-entry region is a fixed sequence only its original neighbours can reuse.

• bind: maps a fragment’s {{names}} to proef values, at pack, macro and step scope, most specific winning, nothing implicit. A foreign corpus names its own variables; convention-matching would only work on files we wrote. Values resolve once per scope instantiation — one binding is one value, two bindings are two values.

• Non-secret bindings emit as per-entry [Options] variable:; ${secret:…} never does, and routes to insert_secret as it always has, so no secret value enters an artifact (ADR-0005 intact).

• Every {{placeholder}} must be bound, produced as a [Captures] name by a preceding step, or supplied by the fragment’s own [Options] variable:. A fragment’s interface needs no declaration anywhere: what it reads is read off hurl’s own AST and crosses the seam as ScannedFragment::placeholders.

  The third source was missing from this list until it was found by audit, and its absence contradicted the decision above it: a file that answers its own question needs fewer variables passed in, which is precisely what makes it runnable on its own — so refusing those files refused the ones this ADR exists to accept. What a fragment supplies itself crosses the seam as ScannedFragment::supplied_variables.

  It is a supplier, so it also collides with a bind: of that name. Both reach the entry as variable: <name>=, hurl takes the last, and the fragment’s own line is last — so the bound value would never be sent, and hurl’s variable: assigning into the run-level set rather than scoping means the loss persists into every later entry. Refused as pack::option_declared_twice, the same rule a doubly-declared retry: gets, rather than resolved by an implicit precedence nobody wrote down.

  What earlier steps produce is derived differently, and deliberately: the core scans the emitted text of the steps it has already lowered (emit::capture_names). It has to, because a preceding step may be an inline hurl: block, which no scanner ever saw — there is no ScannedFragment for it. So the two halves of this check reach the core by different routes, and the produced half is the one that knows hurl’s [Captures] syntax inside proef-core. A second engine would need produced names on the seam instead; nothing is scheduled, and the note is here so the asymmetry is a recorded decision rather than a discovery.

Both body forms stay, because they are not two spellings

Inline does lower-time text splicing; ref: does run-time binding. Neither subsumes the other, and the boundary is demonstrable rather than stylistic: tests/features/packs/breadth.yaml’s postCustomNote splices ${docstring} — a multi-line Gherkin docstring — in as a request body. A hurl [Options] variable: value is a single-line scalar (VariableValue is Null/Bool/Number/String), so no binding can express it. Conversely no inline block can be named, shared, or run by stock hurl.

Splicing can substitute anything anywhere and is private to one macro. Binding is limited to what hurl can template, and buys a name, reuse, standalone runnability, and a static interface check. Choose by capability, not taste.

Consequences

The same file runs under proef test and under hurl file.hurl --variables-file … — one source of truth, no transcription, and ADR-0004’s “copy-paste flows both ways” becomes “no copy at all”. A fragment’s required and produced variables are machine-known, so an unbound placeholder is a load/lower-time error with a real file:line — a check the inline form structurally cannot perform. probe_lower’s two-candidate guessing does not apply to fragments: they parse as authored.

Costs, stated rather than discovered later:

• A test spans three files (.feature → pack → .hurl) instead of two. explain and LSP go-to-definition have to earn that back — and both now do. Go-to-definition on a ref: line lands on the annotation; a ref: step records the fragment it ran as file.hurl#name in step_finished (an additive event field — ADR-0008 — absent for inline steps, so no pre-existing record changes a byte), which explain prints under a failure as via …. The qualified spelling is the one ref: itself accepts, so a post-mortem line pastes straight back into a pack.

  The record carries the name because it must stand alone: by the time anyone reads it, the pack that named the fragment may say something else. That is also why the path is shortened to a project-relative spelling before it is stored — [run] fragments resolves against the config file’s directory, and an absolute root would put a machine-specific path in a durable artifact and stop two checkouts’ records from comparing equal.

• A bound value gets exactly one expansion pass, and no more. hurl’s eval_template is a single non-recursive pass, so a rendered variable’s value is never re-parsed as a template. But a [Options] variable: value is itself evaluated as a template before it is stored (hurl-8.0.1/src/runner/options.rs:508-511), which is one pass more than the entry body gets. So bind: { recordUrl: "${url:record}" }, where record is "${url:base}/api/v1/records/{{recordId}}", does work: {{recordId}} expands at option-eval time from a preceding capture, and GET {{recordUrl}} then renders the finished URL. [url]’s path table keeps working for fragments, and ADR-0012 is unchanged.

  The limit is the second level: a value that expands to text still containing {{…}} will emit those characters literally. In practice that is the same requirement the bound-or-captured rule already enforces — every placeholder must be in scope at the entry that reads it.

  (An earlier draft of this ADR concluded that fragment paths had to move into the .hurl file and that [url] would narrow to base. That was wrong: it read the non-recursive eval_template as applying to the binding path too. Recorded because the wrong version would have forced a needless proef.toml migration.)

• Two ways to write a request body — accepted deliberately above, and the reason is recorded so the 844-line evidence is not forgotten by someone later tempted to deprecate the inline form.

• proef reads files it does not own, so it must never write them: fmt refuses .hurl in directory discovery, and a foreign corpus stays byte-untouched.

Charter

PRD §3’s hurl non-goal is amended in the same change (PRD “Amendment (2026-08-11)”): it forbids generating Gherkin/macros/prose from hurl, not hurl text being an input source. Nothing here generates anything — features and macros stay hand-authored, and a fragment is inert until a macro names it. ADR-0016 stays declined on the untouched reasoning. The amendment records that OPEN-FINDINGS M3 asked for this re-examination to arrive with a measured port cost, and that it has not.

This is not M2 (mechanical equivalence between a hurl corpus and its proef port). The integration test that runs one fragment both ways proves the file is dual-runnable; it does not compare two suites’ results, and M2 stays open.

Alternatives considered

Region-claiming annotations (one comment claims every entry until the next) — fewer comments, but a claimed region is a fixed sequence, so the reuse requirement kills it; and pairing YAML modifiers positionally with claimed entries is exactly the coupling emit.rs’s MapEntry.step comment already warns against (“explicit, never positional”). One file per macro — trivial rule, but it dictates the layout of a corpus we do not own. Metadata in the annotation (# @proef.step retry=10x300ms) — maximum locality, at the price of a second configuration language with its own parser, schema and finite-retry lint, inside somebody else’s file. Replacing inline entirely — rejected: the 844-line corpus is evidence the paste path is sufficient for real work, and ${docstring} splicing has no binding equivalent. Implicit binding from config keys — shortest packs, but a name would bind from a file the pack never mentions, and a foreign corpus’s names rarely match anyway.

Prior art

The shape is well-established: -- name: magic comments naming queries inside valid .sql files (yesql, HugSQL, aiosql, sqlc); # @name naming requests in JetBrains’ .http client, which can import and run them by name across files; and the OpenAPI Initiative’s Arazzo specification, a separate declarative workflow document whose steps reference operations by operationId defined elsewhere — the same dependency direction chosen here, where the definition file knows nothing about its consumers. Reuse across hurl files is an acknowledged, unresolved gap upstream (Orange-OpenSource/hurl #317, #4574), so nothing here conflicts with a shipped hurl feature. The known failure mode of magic comments — invisible coupling — is answered by the name-only rule and by reading the annotation off hurl’s own AST, where comment-to-entry attachment is already modelled.

Amendment — 2026-08-12 (the scanner reports unannotated entries, by line)

FragmentScanner returns ScannedFile { fragments, unannotated } rather than Vec<ScannedFragment>. The named half is unchanged; the addition is the 1-based start line of every entry carrying no # @proef annotation.

The original contract dropped those entries at scan time, on the argument — still correct — that nothing downstream can use one, and that a corpus proef did not write is expected to be mostly unannotated, so building a ScannedFragment for each would be the bulk of a scan for nobody’s benefit. That reasoning covers building fragments. It does not cover counting, and the difference showed up in the field: a 97-entry corpus port found that missing an annotation on one entry produces a green dry-run and a silently absent test, with no signal at scan, bind, or run time — because the entry that would prove it was never built. Neither could any command state how many entries a corpus held, so there was no denominator against which the gap could be noticed.

A line number costs a push and is all a listing can point at, there being no name to print. The performance argument is therefore preserved intact: nothing extra is constructed, and proef fragments consumes what the scan already had to walk past.

Unannotated is not an error. proef fragments --check fails on annotated fragments no scenario runs; failing on unannotated entries requires --require-annotated. During a port “unannotated” means not done yet; in steady state it means deliberately not exposed, which is the premise that lets this ADR promise that pointing at a corpus you did not write costs nothing. Gating every adopter on the porting reading would have contradicted it, so the porting team asks for that check explicitly.

StepKindSpec also gained options, an engine-contributed recogniser mapping a raw option key to what the core’s ADR-0007 budget rules should make of it. The fragment half of that rule already crossed the seam (ScannedFragment::declared_options) while the inline half matched "retry-interval:" as a literal inside proef-core — one rule at two altitudes, and a second engine would have had its fragments linted and its inline blocks not. Option spellings now live only in the engine that owns them. Option baking (lower.rs) still writes hurl syntax directly; the emitter is hurl-shaped by ADR-0010 and is a separate question this amendment does not address.

Amendment — 2026-09-05 (a fragment’s file assets are its own, and staging is what delivers them)

“The same bytes run under stock hurl and under proef” was stated for the entry’s text. It was never true for the files that text reads. hurl resolves file,…; — a request body, a multipart part, a file-valued assert, and [Options] output: — against the directory of the file that wrote the reference (--file-root, defaulting to the .hurl file’s own parent). proef resolved every such reference against the feature, so a fragment’s asset, sitting where its own author put it, was unreachable: the same file passed under stock hurl and failed under proef as exit 2, blaming the author for a path that was correct.

Nothing worked around it. Moving the asset beside the feature breaks the standalone run this ADR exists to guarantee; a reaching ../ path is refused by hurl’s sandbox; and the advice that refusal prints — check –file-root option — names a flag proef does not expose and no proef.toml key supplies.

Per-source resolution, delivered by staging. Each asset is copied from beside the source that referenced it — the feature for an inline hurl: block, the fragment for a ref: — into that scenario’s own asset root, which is then the engine’s context dir.

Two roots at once was the obvious alternative and is not available: hurl exposes one context dir per run of entries and no per-entry override (its 42 OptionKind variants contain no file-root), while a single batch may mix both body forms — measured, not assumed: an inline step and a ref: step in one macro lower to one batch. Honouring two roots would therefore mean splitting batches on the authoring layout, which trades away the “batch maximally” rule TECH-SPEC §5 derives from run_entries building its client per call. Staging keeps batching intact, and copying fixtures into the build output is the standard answer to exactly this problem.

Three further consequences, each a correction rather than a cost:

1. The root is per scenario. Flat staging was keyed by the asset’s bare name, so two features that each kept a data.json staged to one file — last writer wins, silently — and the loser’s artifact replayed against the other’s bytes. artifact_slug already refuses that trade for the .hurl text; the files it reads now match it. Two sources claiming one name within a scenario, which no per-scenario root can separate, is refused (proef::run::asset_unstageable).
2. Staging is load-bearing, so its failure is fatal. A missing asset used to be skipped in silence because the copy only fed the record. It now feeds the run, and a file that did not arrive is a request reading nothing, not an incomplete record.
3. ADR-0010’s promise widens. “Artifacts are the executed input” held for text while assets were read from the suite. The artifact directory is now the whole executed input, and an artifact that reads a file says so in its replay line (--file-root assets/<slug>). One that reads none is byte-identical to before, which is why exactly one snapshot in the corpus moved.

The sandbox also narrows: the context dir is a directory holding only what proef staged, rather than the suite tree it used to be (TECH-SPEC §13).

ADR-0019 — Reserved tags and the authored skip

Status: Accepted · Date: 2026-08-24

Emerged from the Robot Framework capability audit (OPEN-FINDINGS, “RF wave 2”); every design fact below was verified against the tree or reproduced empirically before acceptance.

Context

proef had no way to park a scenario. A test that must not run — mid-migration, a known-broken dependency, a seasonal flow — could only be deleted or dodged with --tags, and both are invisible: nothing in any report says “this exists and was deliberately not run”. Robot Framework’s SKIP model (its 4.0 headline design, replacing criticality) is the industry convergence point: skip is a first-class visible status with a reason that survives into every report.

Mechanically, proef already had a scenario-level Skipped status — but it arose only from cancellation, its reason existed nowhere, and two consumers had baked “Skipped means never-ran” into their logic (--rerun re-queues Skipped-on-cancelled; diff reads Failed→Skipped as fixed — and --fail-on-regression certified it). An authored skip that ignored those two would have shipped a laundering bug, not a feature.

Decision

1. A reserved tag namespace, recognized at the CLI edge. @quarantine and @skip are the reserved tags; recognition lives in exactly one place (front::reserved), and core never reads tags — the front computes an instruction (ScenarioSpec.skip, like exclusive before it) per ADR-0014’s split. Reserved tags in [run] setup/teardown features have no effect: phases never pass through build_specs, and skipping your whole setup deliberately is spelled by deleting the config key.
2. The spelling is @skip or @skip:<reason-token>. The gherkin grammar accepts skip:migration-pending as one tag (verified empirically through the real pipeline). The recorded reason is the pasteable tag spelling itself — "@skip" / "@skip:migration-pending" — the same philosophy as the fragment field’s file.hurl#name.
3. Authored reasons start with @; mechanical reasons never do. That is the contract --rerun keys on: Skipped ∧ cancelled ∧ reason not authored re-queues as never-ran; an authored skip never re-queues. Pre-field records (no reason) read as mechanical, which they were.
4. No tag-list normalization. An earlier draft injected a canonical skip atom beside @skip:x so --tags "not @skip" excluded both. Tag globs shipped first, and not @skip* says the same thing without proef ever rewriting an authored tag list. Authored tags stay exactly authored.
5. A skipped scenario is selected, counted, and reasoned in every sink: console (∅ … — @skip:x), JUnit (<skipped message>), TAP (# SKIP @skip:x), the record (ScenarioFinished.reason, additive, schema stays 1), the HTML report, explain, flows --format json ("skip"), and the harness (libtest’s ignored flag). --tags remains the unselection mechanism — the two semantics stay distinct, as in RF.
6. All-selected-scenarios-skipped exits 0. Exit 2 is for faulty input; the empty-selection refusal exists for the typo’d filter whose silent green run nobody sees. An all-skipped run is neither silent (every surface prints the totals and reasons) nor accidental (each skip is authored, versioned, and visible in review). RF and pytest agree; pytest reserves its special code for empty collection, which is exactly the case that stays exit 2 here.
7. diff gives skip transitions their own bucket. Into-Skipped is neither fixed nor regressed (now skipped (was failing/passing)); out-of-Skipped has no meaningful baseline and takes the added shape.
8. A quarantined test-failure reaches JUnit as skipped-with-message. The exit code already said “non-gating”; the XML said <failure>, so Jenkins marked UNSTABLE and every dashboard contradicted the verdict. RF converts the status for the same reason. User/System faults stay failures — quarantine is for flaky tests, not broken input.
9. --dry-run still validates skipped scenarios. Skip is an execution-time decision, not a validation waiver — a broken-but-skipped scenario still fails --dry-run, deliberately.

Consequences

• Library-breaking (clean break, no shims): ScenarioSpec.skip, ScenarioOutcome.reason, Event::ScenarioFinished.reason, ScenarioRun.reason, write_junit/write_ci_reports gain the non-gating list. Wire-additive; EVENT_SCHEMA_VERSION stays 1.

• The sink wrappers that rebuild scenario events field-by-field (stamp_scenario_timing, phase_sink) must thread every new field — the exhaustive constructions turn forgetting into a compile error, and the e2e test pins the stamped stream.

• A related stance this ADR writes down because the audit found it held but unwritten: control flow lives in packs (when: conditional skip at step level, optional: soft-fail, finite retry:) — prose stays declarative; there is no scenario-level IF/WHILE/TRY and none is planned.

• 2026-09-06: a tag within a short edit distance of a reserved one (@quarantined, @skipped, @Skip) stays an ordinary, inert tag — but it now warns (tags::reserved_tag_typo) with the spelling it likely meant, since a scenario its author believed quarantined would otherwise gate the build in silence. Short reserved words get only a case-fold or a suffix match (ship/slip/step are one edit from skip); the long quarantine affords a distance-2 backstop.

ADR-0020 — Run metadata is explicit-injection-only

Status: Accepted · Date: 2026-08-24

The RF-audit wave-2 companion to ADR-0019; codifies the boundary R12-1 drew and the JUnit provenance decisions applied.

Context

A proef record could not say which commit, build, or environment produced it — run_started carried schema and run_id alone. Robot Framework’s --metadata name:value fills this in reports, and CI post-mortems genuinely need it: diff across records from different commits or environments has no context for what changed.

The hazard is on the other side. R12-1 removed harvested machine identity from every artifact because absolute paths broke the two-checkouts byte-equality ADR-0010 guarantees, and the JUnit sink deliberately omits timestamp/hostname for the same reason. Metadata must not reopen that door.

Decision

1. The axis is harvested vs. handed-over, not automatic vs. manual. proef never reads git, the hostname, wall-clock provenance, or CI environment variables (GITHUB_SHA, CI_COMMIT_SHA, …). If the user wants the SHA recorded, their shell harvests it: --meta commit=$(git rev-parse HEAD). What the user explicitly hands over, proef records verbatim.
2. One precedence chain, three scopes: [meta] < [env.<name>.meta] < --meta k=v — the same base < env < flags shape as jobs and [url]/[vars]. A duplicate key among the flags is exit 2 (loud over last-wins); a flag overriding a config key is the designed use. There is no PROEF_META_* — the values that motivate env vars are already in the shell where the flag is typed.
3. The active --env profile name is recorded automatically, as its own field (run_started.env). It is user-chosen input to the invocation, not an observed machine fact — and without it the record is uninterpretable: the same suite deep-merges different [url]/[vars] per profile, so diff warns loudly on a cross-env comparison.
4. shuffled rides the same head: with the permutation seeded by run_id, the bool plus the id reproduces an order exactly (deferred out of --shuffle’s own change so run_started moved once, not twice).
5. Metadata reaches the record, explain, diff, the HTML report, the GitHub summary and the --format json body — and nothing else. Never artifacts (.hurl bytes stay identical across checkouts and commits — ADR-0010, R12-2); not TAP (no slot a consumer reads); not JUnit <properties> (GitLab ignores them, Jenkins reads them only behind a non-default opt-in — same named-consumer method as R3-6, additive later if a consumer asks); not the console (the record and explain own it).
6. Everything passes the sink-boundary mask — keys and values both: a secret-bearing URL pasted into either position must not survive into the record or the body. The known limit stands recorded: a token proef was never told is a secret matches no needle, the same standing as any CLI argument.

Amendment (2026-09-07) — a computed input fingerprint is not harvested metadata

proef flaky’s equivalence-class fingerprint (the inputs.json sidecar) prompted the obvious question: does §1 forbid it? It does not, and the boundary is worth stating so the next reader does not re-litigate it.

§1 forbids proef from harvesting an environment fact — reading git state, the hostname, or CI variables and putting them in the record. The input fingerprint reads none of those. It is a hash of proef’s own inputs — the feature sources, the loaded macros and fragments, the resolved config scope — the same category as the artifact slug (emit::artifact_slug) or the shard hash: a derived identifier over data proef already holds, not a fact lifted from the surrounding machine. Derived identifiers have never been in scope here; ADR-0020 governs [meta]/--meta metadata, which this is not.

The git-commit case remains exactly as §1 requires: a user who wants commit-based grouping hands the commit over (--meta commit=$(git rev-parse HEAD), §1’s own worked example) and proef flaky --by commit groups on it. proef never runs git itself. So the survey’s “git tree SHA equivalence class” splits cleanly along this ADR’s own axis — a computed fingerprint proef may derive, plus a commit the user may hand over — and needs no new decision.

The sidecar is a derived aid like timings.json, not a second record (ADR-0008): the JSONL event stream remains the only record format, and the event schema is untouched (no run_started field was added).

Consequences

• run_started gains env, metadata, shuffled — additive, skip-serialized when unset, EVENT_SCHEMA_VERSION stays 1 (ADR-0008 erratum extended). The empty case is byte-identical to every existing record.
• Library-breaking: ProjectConfig/EnvProfile gain meta, exec::execute takes the merged map, RunRecord::open takes the head trio; clean break per policy.
• proef stays sans-IO in core: the CLI merges and injects; core never reads an environment.

ADR-0021 — Run discovery and rotation are separate questions

Status: Accepted · Date: 2026-09-02

Context

--run-id accepts any single path component, and TROUBLESHOOTING.md demonstrates --run-id pr. A run named that way writes a complete, valid record. It is then invisible to proef explain, diff, flaky, report and --rerun whenever they resolve the latest run, because every one of them enumerates through record::all_runs, which admits only 36-character uuid names.

That predicate is not an oversight. rotate_runs consumes the same function and says so:

record::all_runs is the one answer to “what is a run record here” — sorted, uuid-named directories only. Rotation adds a single further exclusion (the in-flight run), not a second enumeration rule.

and is_run_id states the reason:

rotation deletes the oldest run-shaped directories, so breadth here is a deletion hazard when the runs dir points somewhere shared.

Both are right. runs-dir may be ., so a broad predicate would let rotation delete directories proef never created. That risk is real and the narrow answer is the correct one — for rotation.

The defect is that one predicate serves two questions whose risks point in opposite directions:

Sharing the predicate meant the deletion-safety choice silently became a visibility choice. CONFIG.md documented the rotation consequence of a custom id (“this will never delete it”) and not the discovery one, so the surprising half was the undocumented half.

Decision

Split the predicate along the risk, not along the file.

1. Rotation keeps the uuid rule (is_rotatable, formerly is_run_id) — uuid-named directories only. Unchanged in behaviour, for the reason it was written; renamed for the question it answers. A custom-id record is still never deleted by [run] keep-runs, and that stays documented.

2. Discovery admits any directory containing an events.jsonl. A directory holding proef’s own record file is a proef record; the test cannot mistake target/ or node_modules/ for one, and — decisively — it authorises no deletion. Reading a directory that turns out not to be a record fails as a parse error naming the file, which is already how a corrupt record behaves.

3. Ordering stops relying on the name — but keeps relying on the uuid. all_runs documented that “uuid-v7 names sort chronologically, so lexical order is time order”, which a pr directory breaks. The fix takes the timestamp rather than the spelling: a uuid-v7 name carries 48 bits of unix milliseconds — the moment proef minted it — which is precisely why the lexical sort worked. Runs order by that where it exists, and by directory mtime where it does not (a custom --run-id carries no time). Both are wall-clock unix time, so the two sources compare directly.

   Not read from the record, though the first draft of this ADR said it would be. The head event carries event/run_id/schema and no timestamp at all — per-event times are injected observability on scenario_started (ADR-0015) — so “the record’s own run_started timestamp” does not exist. Reading it returned None for every real record and silently ordered everything by mtime; the test that covered it passed only because its fixtures fabricated a field no record has. An ordering that depended on the suite having run at least one scenario would not be an ordering anyway.

Consequences

• proef explain, diff, flaky, report and --rerun find custom-id runs. “Latest” means latest in time rather than latest in the alphabet, which is what every one of those commands already claimed to mean.
• Rotation’s blast radius is unchanged — the one property that could have made this dangerous.
• Two predicates now exist where the code deliberately had one. That is the cost, and it is why this is an ADR rather than a patch: the earlier single-predicate statement was a considered position, and superseding it needs to be on the record. The two are named for their questions (is_rotatable / holds_a_record) so a future reader cannot reach for the wrong one by picking the shorter name.
• Ordering costs a stat per custom-id run directory and nothing at all for a uuid-named one, whose time is read straight out of its name. Bounded by [run] keep-runs (200 by default) and paid only by commands that resolve “latest”.

Alternatives considered

• Leave it, document it. The status quo before this ADR. Rejected: the invisibility surprises exactly the user who chose a memorable id so they could find the run again, and --rerun silently operating on a different run than the one just produced is the worst shape of it.
• Make --run-id reject non-uuid names. Honest, and it would remove the trap — but it also removes the feature’s point. CI archiving .proef-runs/pr-1234/ by a known path is the use case --run-id exists for.
• Rotate custom-id directories too, and keep one predicate. Rejected outright: it makes runs-dir = "." a data-loss configuration, which is the hazard is_run_id was written to prevent.

Erratum — 2026-09-06

The JUnit report’s own uuid still had the trap this ADR removed elsewhere: a non-uuid --run-id parsed to the nil uuid, so every --run-id ci run reported 00000000-… and collided in any consumer keyed on it. A custom id now derives a stable UUIDv5 from its bytes; a uuid id passes through verbatim.

proef — Improvement Plan

Status: complete. Round 1 (§1–§11) and Round 2 (§12) are both shipped — every batch (N2·N6a·N9·N8·N7·N4·N6b·N3), with N1 deferred per ADR-0013, N5 rejected, and the OpenAPI generator settled in ADR-0016. This file is the historical feature roadmap and its file:line citations are from 2026-07/08; the live worklist is OPEN-FINDINGS and the running ledger is CLAUDE.md’s Status block. · Date: 2026-07-31, appended 2026-08-02, closed 2026-08-31 · Owner: Emre Companion docs: PRD (scope + the binding non-goals, §3), adr/ (the invariants every item must respect), TECH-SPEC (types/pipeline), IMPLEMENTATION-PLAN (milestones + definition of done).

0. What this is (and isn’t)

A competitive analysis of proef against the BDD and API-testing field, converted into a roadmap of candidate improvements — each validated against the current architecture at file:line, and filtered strictly through proef’s permanent non-goals (PRD §3).

Nothing here is committed work; it is the durable output of the post-M5 review round. Effort grades (S/M/L) and sequencing are advisory. Feature numbers (#1…#15) are stable identifiers used across the tables. File:line citations are as of 2026-07-31 and will drift — treat them as “start reading here”, not addresses.

Every item is in scope by construction: none proposes a second engine, mocking, contract testing, load testing, a dashboard/server, OpenTelemetry, or importing hand-written hurl — all permanent non-goals (§3 below). The work is almost entirely surfacing data the sans-IO core already computes, not new engine capability.

1. Headline finding

proef’s engine is already ahead of its cohort; the gaps are in reporting surfaces and authoring/maintenance DX, not in HTTP power. Because proef pins hurl 8.0.1, it already ships hurl 8.0’s full assertion arsenal (RFC 9535 JSONPath with filter functions, type predicates isUuid/isIsoDate/isString/isObject/isList, response-time duration < ms, the filter chain split/count/toDate/base64Decode/daysAfterNow). Pack authors can write Karate-grade assertions today, inside raw hurl: blocks — they are just undocumented and under-surfaced. The roadmap is therefore mostly exposure and tooling, which is cheap, rather than engine work, which is done.

2. Where proef already wins (positioning to defend)

3. Scope guardrails — what this plan will NOT propose

Permanent non-goals (PRD §3) — never revisit under “competitor parity”: further engines (browser/gRPC/etc.), API mocking, contract testing (OpenAPI drift / Pact / Schemathesis), load testing, a desktop dashboard or server mode, OpenTelemetry export, dynamic plugin loading, importing hand-written hurl into Gherkin (artifacts flow outward only), static musl/Windows binaries.

Named anti-patterns (from the 2025–2026 trend research) to avoid: silent retries / green-on-attempt-2; retry ceilings > 3 or unbounded (proef’s finite cap is already correct); permanent quarantine without owner+expiry; treating masking as a security boundary; LLM self-healing / non-deterministic test mutation; config sprawl.

A Karate-style marker DSL (#uuid, ##optional) is explicitly rejected — it would be a second assertion mechanism competing with raw hurl predicates. Achieve the same readability by surfacing hurl’s native predicates (item #2), not by inventing a layer.

The overriding design gate is one-canonical-way (see §6): four items must replace or augment an existing mechanism, never add a parallel knob.

4. What code validation changed about the roadmap

Three deep code-validation passes (reporting, CLI/tags, authoring) reshaped the outside-in list. Six meta-findings:

1. “Surface, don’t build” is confirmed at file:line. The sans-IO core already computes and records the data behind most items: attempts/duration_ms/detail on StepFinished (proef-core/src/event.rs), the winning macro_name per bound step (bind.rs:20), deterministic seeded fakes, and hurl’s own curl_cmd (already returned by EntryResult, currently discarded in engine-hurl session.rs).

2. Two items are already ~90% built — validation downgraded them.

   • #14 seeded fakes: fakes are already deterministic (hand-rolled SplitMix64, no rand crate, fake.rs) and already seeded by the injected run_id, which is already recorded in RunStarted. The feature collapses to “let test pin run_id the way artifacts --run-id already can.”
   • #9 stub-gen: the “did you mean” fuzzy suggestion already exists (bind.rs:214, levenshtein); only the paste-ready stub template is missing.

3. One item’s premise is broken — #13 impacted-only re-run. There is no stored content hash anywhere; ADR-0010 is enforced as a byte-identity assert_eq! on outputs (crates/proef-cli/tests/execute.rs:188), not a reusable digest. The only honest impact fingerprint — the emitted .hurl — is deliberately not stable run-to-run (run_id lives in runtime globals). #13 is gated behind a determinism prerequisite.

4. Two plumbing gaps gate a cluster. Scenario tags stop at the CLI edge — they never reach the runner (ScenarioSpec/ScenarioOutcome in runner.rs carry no tags) or the event stream — so #15 (quarantine) and part of #8 (rerun) need new plumbing. And #4 (boolean tags) is hard-blocked by value_delimiter=',' on --tags (main.rs:57); it is a contract-changing replace, not an add.

5. The recurring architectural rule is one-canonical-way. Four items (#4, #9, #14, #15) each risk a second mechanism; each must replace or augment an existing one.

6. The exit-code contract (ADR-0009) is a live wire for #15. Quarantine changes which scenarios feed RunSummary::exit_code (fine — the fold stays pure in core) but must extend the pinned assert_cmd tests and never let a quarantined system fault mask exit 3.

5. The validated roadmap (master table)

Status is what exists in the tree, re-verified against main on 2026-09-08 — --help for flags, source for the rest. Verdict is the 2026-07-31 judgement of whether the idea fits the architecture; it never meant “done”, and reading it that way is why this table looked like a backlog when, by 2026-08-10, 13 of its 16 items had shipped (14 today; one partial, one gated).

Status: shipped · partial · open · gated (premise rejected). Verdict legend: ✅ FITS · ⚠️ NEEDS-ADAPTATION · 🚫 premise broken. Effort: S ≤ ~1 day · M ~days · L ~weeks.

6. Prerequisites that unlock clusters

• P1 — carry scenario tags + a gating flag past the CLI edge into ScenarioSpec / ScenarioOutcome (runner.rs:30,121) and, additively, the event stream. Done (RF waves): tags ride ScenarioSpec/ScenarioOutcome and scenario_finished.tags; gating became the reserved-tag instruction plus the non-gating list (ADR-0019) rather than a bool. #15 and #8 both shipped on top of it.
• P2 — a shared record::failed_scenarios(run_id) + a multi-identity scenario predicate. Reused by explain, --rerun (#8), and proef diff (#12).
• P3 — a deterministic emitted-.hurl fingerprint (stable run-to-run despite run_id). Prerequisite for #13; do not attempt #13 without it. 2026-09-07: proef_core::fingerprint now hashes a run’s whole input set (feature sources, loaded macros and fragments, the resolved [url]/[vars] scope) into inputs.json — proef flaky’s equivalence class. Suite-level and deliberately coarse (any edit ends a window), so it is not the per-scenario, run_id-independent fingerprint #13 needs; P3 stands.

7. The one-canonical-way watch-list

Each of these must fold into an existing mechanism, never ship beside it:

8. Recommended sequencing

1. Batch A — free / small, all FITS, all reuse existing data. #2 cookbook (docs) → #1 annotations → #5 --curl → #3a attempt badge → #9 stub.
2. Batch B — small, high-leverage. #10 SARIF · #7 dead-macro · #14 --seed.
3. Prereqs → Batch C — medium, now unblocked. Build P1+P2, then #6 HTML · #12 diff · #8 rerun · #15 quarantine · #4 boolean tags · #3b flaky-detail.
4. Batch D — strategic. #11 LSP. And #13 only after committing to P3.

9. Per-item detail & competitor provenance

Each entry: what it borrows from whom → the validated architectural note. Numbers cross- reference §5.

#2 Assertion cookbook — from Karate’s fuzzy markers + hurl’s own docs. Confirmed a docs task: resolve() leaves everything but ${…} byte-for-byte (resolve.rs:163-208, test runtime_tier_passes_through), and pack load validates the full grammar via hurl_core::parser::parse_hurl_file (engine-hurl/src/lib.rs:95), re-checked on the emitted artifact (front.rs:159). Extension: ship a tests/features/ reference feature exercising each predicate so the cookbook is snapshot-locked against hurl upgrades (the canary catches drift).

#1 GitHub annotations — from the 2025–2026 CI-reporting shift (annotations displace log-diving). The failures loop already prints `{file}:{line}` — {detail} (ci_reports.rs:98). Emit ::error workflow commands as a sibling; title = scenario + step text. Risk: stdout is owned by --output json (exec.rs:114) — gate it.

#5 --curl export — from hurl’s loved --curl; Bruno/Postman “copy as curl”. hurl hands us EntryResult.curl_cmd already (session iterates result.entries at session.rs:346 but reads only captures/errors/duration). Must pass Redactions (session.rs:396) before any sink — the curl line contains resolved secrets. Cannot be derived pre-execution (needs runtime {{…}}).

#3a “passed on attempt N” — from the flaky-test-honesty consensus (never hide a retry). attempts is first-class (event.rs:72, step.rs:149) and already printed on the console (report.rs:220); JUnit simply drops it. Count-based badge is S.

#9 Stub-gen — from Cucumber/Behave snippet generation. The matcher already computes the nearest macro via closest_pattern/levenshtein (bind.rs:214, matcher.rs:248); add a match:+hurl: | skeleton to the help text for the zero-match arm only (an ambiguous step, bind.rs:99, must not get a stub).

#10 SARIF — from SARIF’s rise for static/validation findings inline in PRs. Dry-run diags are a structured Vec<Diag> before miette (diag.rs:123, front.rs:64). Diag maps ~1:1 to a SARIF result; the closed code set (one per tests/errors/ dir) pre-fills rules[]. cli-edge serializer, no core change.

#14 --seed — from seeded-faker reproducibility. Fakes already deterministic (fake.rs:12 SplitMix64/FNV, seeded fnv1a(run_id) ^ …), seed already recorded in RunStarted{run_id} (event.rs:26). Design fork: alias run_id (zero core change, but must stay uuid-parseable for JUnit) vs a dedicated recorded seed field (cleaner, but a second knob — resolve per one-canonical-way). Known limit: the occurrence counter is scoped per scenario (lower.rs:69-86, Refs::fakes, threaded through resolve()’s fakes: &mut usize parameter, not reset per call) → cross-step uniqueness within a scenario now holds, but cross-scenario uniqueness still does not: two different scenarios each resolving ${fake:X} at the same position in their own step order get the same value. Document before advertising “unique fakes” — it means per-scenario, not per-run or per-entity.

#7 Dead-macro report — from Cucumber’s usage formatter marking UNUSED. Binding records BoundStep.macro_name (bind.rs:192); iterate front.features[].scenarios[].bound.steps[].macro_name vs packs.macros.keys() (pack/mod.rs:116). use:-only macros (pattern:None) need reachability via the use: graph (pack/validate.rs:628) to avoid false “unused”. Report the whole corpus.

#6 HTML report — from Cucumber/Karate/hurl HTML reports; the industry’s convergence on the Cucumber-Messages/JSONL stream. Core render_html(&[Event]) -> String, cli writes; best as post-hoc over events.jsonl (explain.rs already replays it) so historical runs render. Events are pre-redacted at the sink (report.rs:124).

#12 proef diff — from Allure history / test-observability-without-OTel. Identity is (file, scenario) (report.rs:147 ScenarioKey; ADR-0008 added file for exactly this). Key step diffs on text, not the volatile line. attempts+duration_ms make it a flakiness/perf-regression detector. run_id is uuid-v7 → chronology recoverable.

#4 Boolean tags — from Cucumber tag expressions (and/or/not/()). Single filter fn tag_selected (front.rs:388), three callers. value_delimiter=',' (main.rs:57) blocks the operator syntax → drop it, take one expression string, replace the CSV contract. Grammar/evaluator → core (deterministic, fuzz-shaped). Preserve empty-match=exit-2.

#8 Rerun-only-failures — from Cucumber’s rerun formatter (@rerun.txt). explain already discovers + replays the latest record and extracts failed identities (explain.rs:63). Add a multi-identity predicate reusing build_specs (exec.rs:289). Empty failure set → reuse no_scenarios_matched (exit 2), never silent-pass.

#15 Quarantine — from the flaky-quarantine-with-owner+expiry consensus. Thread gating:bool from CLI (which sees scenario.lowered.tags, exec.rs:318) into ScenarioSpec/ScenarioOutcome; exit_code() (runner.rs:89) skips non-gating outcomes and stays pure in core. Events unchanged → scenario still reported. Extend the cli.rs/execute.rs exit-code assertions; a quarantined Fault::System still exits 3.

#3b Flaky-failure detail — from JUnit <flakyFailure> / Allure retries. Needs an additive attempt_details on StepFinished (ADR-0008 additive-only) and engine-hurl collecting per-retry messages (hurl retry is per-entry internal — verify the adapter isn’t already discarding earlier bodies).

#11 proef lsp — from Cucumber’s language server (unbound-step diagnostics, go-to-def, completion); a gap Karate never closed. Reuses feature::parse, bind, pack::load, resolve Probe mode, matcher — all headless, all with stable codes + byte-offset spans that already map to editor ranges. New work: sync lsp-server (tokio banned), byte→token API, and a “collect diags, don’t early-return” wrapper (bind/load are all-or-nothing today).

#13 Impacted-only re-run — from selective/affected-test re-run. Premise broken: no reusable input hash (ADR-0010 is a byte-identity assert_eq! on outputs, execute.rs:188), and the honest fingerprint (emitted .hurl) is not run-to-run stable because runtime globals include run_id (execute.rs:304). Gated on P3; a hash miss must never skip a scenario that would fail (needs --force/first-run fallback).

10. Karate feature ledger — considered / adopted / rejected

Karate (github.com/karatelabs/karate) was the closest competitor and the deepest research stream. This ledger makes the “considered → decision” trail explicit, so each Karate idea is an intentional call rather than an omission.

Adopted — drove a plan item or the positioning:

Rejected — with the reason (so it stays rejected):

Deferred — genuine candidates, not yet planned:

11. Sources (competitive research, 2026-07-31)

• Karate — match keyword / fuzzy markers / reuse: https://docs.karatelabs.io/assertions/match-keyword/, https://docs.karatelabs.io/reusability/calling-features/
• Cucumber-JS formatters (usage / rerun / snippets / html): https://github.com/cucumber/cucumber-js/blob/main/docs/formatters.md
• Reqnroll HTML report + Cucumber Messages; SpecFlow EOL: https://reqnroll.net/news/2025/06/roadmap-update-html-report/, https://reqnroll.net/news/2025/01/specflow-end-of-life-has-been-announced/
• Hurl 8.0 (RFC 9535 JSONPath, --curl, TAP, secrets redaction) + release history: https://hurl.dev/blog/2026/04/27/announcing-hurl-8.0.0.html, https://hurl.dev/docs/filters.html
• Bruno vs Postman (git-native trajectory): https://www.usebruno.com/compare/bruno-vs-postman
• GitHub Actions workflow commands (annotations + job summaries): https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-commands
• Flaky-test quarantine/hardening; GitHub Actions OIDC/masking limits; Allure history: https://pie.inc/blog/flaky-tests-cicd/, https://www.stepsecurity.io/blog/github-actions-security-best-practices, https://allurereport.org/docs/how-it-works-history-files/
• Cucumber Language Server: https://github.com/cucumber/language-server

12. Round 2 — post-execution competitive re-review (2026-08-02)

Round 1 (§1–§11) is largely shipped (across the releases that followed). Round 2 re-ran the Karate + Cucumber + adjacent-landscape survey against the post-execution codebase, then put every surviving candidate through a four-stream deep code-validation pass (matcher/binding · reporting/events · lifecycle/i18n/snapshot · scope-boundary), mirroring Round 1’s method. Identifiers N1–N9 are stable and doc-local — this registry is their only sanctioned home; they never appear in code comments (per the no-task-ids-in-source rule). File:line citations validated 2026-08-02 and will drift.

12.1 What the re-survey confirmed is already shipped (positioning to defend)

The external agents, blind to the just-landed work, flagged many “gaps” that Round 1 already closed — recording them so the omission-vs-decision trail stays explicit:

Convergent-evolution note (validates the architecture, nothing to adopt): Karate v2’s karate-events.jsonl (2025) is proef’s ADR-0008 event stream re-invented; Bruno’s plain-text git-native rise is proef’s text model; both confirm the design is industry-aligned.

12.2 The validated Round-2 roadmap (master table)

Verdict legend as §5 (✅ FITS · ⚠️ NEEDS-ADAPTATION · 🚫 rejected/premise-broken). Effort: S ≤ ~1 day · M ~days · L ~weeks.

12.3 Verdict-change ledger (validation overturned the first sketch)

The deep pass is on the record because it changed conclusions — the point of validating: