RFC-0013 · Test Execution Model
Versioned draft snapshot
RFCs, reference catalogs, and evidence reflect Spec 3b2e3e7a114fb826b1445d39a48756b8fff629cd. Counts and pending/status notes belong to that snapshot. Read the dated project update for later progress.
Source repositories are private during staging. Links marked “repository access required” need access to the linked repository.
| Status | Draft corpus. Reference catalogs and RDF projections have generation drift checks; Gnomon proofs and projection semantics remain unverified for release. |
|---|
View the generated reference catalog →
Read design questions and decisions →
Status: Draft Owners: DX, Compiler
1. Scope
Defines the execution model for test declarations in Gnomon source files. The test declaration kind is part of the language grammar (RFC-0008 §8.12) and uses the constraint expression language (RFC-0008 §8.22). This RFC specifies the semantics of the given, when, and then blocks; the in-memory object graph lifecycle during test execution; the gnomon test CLI harness; output format; CI enforcement requirements; and all diagnostic codes in the M013xx range.
Test declarations are exclusively a development-time construct. They are stripped from production bundles and MUST NOT appear in any artifact with ciState: ready (RFC-0008 §11.2, diagnostic M08786). This RFC does not define the language grammar — that is fixed in RFC-0008 §8.12 — nor does it define the constraint expression language (RFC-0008 §8.22). It defines what happens at runtime when a test is executed.
2. Goals
- Tests are first-class declarations in
.gnmsource files, authored with the same syntax as production code and subject to the same compiler passes. - The test execution environment is a deterministic, isolated, in-memory replica of the Gnomon object graph (RFC-0007). No external state, no network, no database.
- A single test exercises exactly one action invocation. Tests are not scenarios or end-to-end sequences.
- Test output is machine-readable (TAP-compatible) and human-readable simultaneously. No bespoke output format.
gnomon testis a required, blocking CI step. Projects without test enforcement are non-conformant.- Failure messages surface the specific assertion that failed, with a diff of expected versus actual state.
Linked References
The cross-RFC dependency graph is rendered in reference catalog § Linked References. Out-of-KG pointers to legacy code or external resources, when present, remain in this document under “Source-tree references” below.
4. Invariants
- A
testdeclaration MUST NOT be included in any compiled bundle artifact whereciState: ready. Including one is a compile error (M01300). - Each
testdeclaration MUST contain exactly onegivenblock, exactly onewhenblock, and exactly onethenblock, in that order. Any other structure is a parse error (M01301). - The
whenblock MUST reference exactly one declaredaction. Referencing an undeclared identifier, a non-action declaration kind, or more than one invocation is a compile error (M01302). - Every expression in the
thenblock MUST be a boolean-valued expression per the constraint expression language (RFC-0008 §8.22). A non-boolean expression inthenis a compile error (M01303). - The in-memory object graph seeded by
givenis isolated per test. No state leaks between test executions. - Test names MUST be unique within a module. Duplicate names within the same
.gnmfile are a compile error (M01304). - The
gnomon testharness MUST exit with code 0 only when every discovered test passes. Any failing test produces exit code 1. A compile error in any test file produces exit code 2 without running any tests. - TAP output MUST be emitted to stdout. Diagnostic messages (compile errors, harness warnings) MUST be emitted to stderr.
5. Test Declaration Anatomy
5.1 Syntax Recap
test "enrollment window is enforced" {
given {
Household { id: "h1", members: [...] }
Bundles { id: "b1", status: pending, household: "h1" }
}
when selectBundle { bundleId: "b1", actor: Client }
then Bundles.status == selected
}
The test name is a string literal. It is used as the label in TAP output and in failure messages. It SHOULD describe the invariant being verified, not the mechanism.
5.2 given Block — Object Graph Seeding
The given block seeds an isolated, in-memory instance of the Gnomon object graph (RFC-0007) with an initial set of entities. Each line in the block is an entity initializer:
<EntityKind> { <field>: <value>, ... }
<EntityKind>MUST refer to a declared entity kind visible in the module’s scope. An unknown kind is a compile error (M01305).- Every field listed MUST exist on the declared entity kind. An unknown field is a compile error (M01306).
- Fields not listed take their declared default values. If a field has no default and is not listed, that is a compile error (M01307).
- The
idfield is required for all entities. It MUST be a string literal. Duplicate IDs within the samegivenblock are a compile error (M01308). - Cross-entity references use the quoted string ID of the target entity, not a structural reference. The compiler resolves these IDs against the
givengraph at test compile time. A reference to an ID not present in thegivenblock is a compile error (M01309). - List fields accept inline array syntax (
[...]). Each element follows the same rules as standalone field values. - Enum fields use unquoted enum variant names (e.g.,
status: pending). An unknown variant is a compile error (M01310).
The given graph is built before the when invocation. It is strictly immutable from the perspective of the test harness setup phase — only the when action may mutate it.
5.3 when Block — Action Invocation
The when block specifies a single action invocation:
when <actionName> { <arg>: <value>, ..., actor: <PrincipalKind> }
<actionName>MUST resolve to anactiondeclaration (RFC-0008 §8.7) in scope. Any other declaration kind is a compile error (M01302).- The
actorfield is required and MUST match one of the principal kinds declared for that action. An unknown principal kind is a compile error (M01311). - All required arguments for the action MUST be supplied. Missing required arguments are a compile error (M01312). Extra arguments not declared on the action are a compile error (M01313).
- The action is executed against the object graph seeded by
given. The action’s declared preconditions, guards, and policy rules are evaluated as they would be at runtime. - If the action’s preconditions are not satisfied by the
givenstate, the test fails with diagnostic M01319 indicating which precondition was violated (not a compile error — this is a runtime test failure, reported in TAP output). - The harness does not mock the action. It runs the real compiled action logic against the in-memory graph.
- If the action under test contains an
execute {}block (RFC-0008 §6.10), the block is executed before mutations. In the test harness, external I/O calls withinexecute {}MUST be stubbed via the test declaration’smock {}block (if defined) or MUST be omitted — the test harness MUST NOT make real network calls.
5.4 then Block — Assertions
The then block contains one or more assertion expressions:
then
Bundles.status == selected
Household.members.length >= 1
- Each expression MUST be boolean-valued per RFC-0008 §8.22. A non-boolean expression is a compile error (M01303).
- Expressions are evaluated against the object graph state after the
whenaction has completed. - All assertions are evaluated independently. A failure in one assertion does not prevent evaluation of subsequent assertions. All failures for a single test are reported together.
- Dot-path access follows the same semantics as the constraint expression language (RFC-0008 §8.22): field access on an entity kind resolves against all entities of that kind in the graph. If the entity kind is ambiguous (multiple instances in the graph), the path MUST be qualified with an ID predicate (e.g.,
Bundles[id="b1"].status == selected). Ambiguous paths without qualification are a compile error (M01314). - Comparison operators, arithmetic, and logical connectives follow RFC-0008 §8.22 exactly. No test-specific operators are introduced.
6. Test Harness CLI
6.1 Command
gnomon test [path...]
pathis zero or more paths to.gnmfiles or directories. If no path is given, the harness discovers all.gnmfiles under the current working directory, excludingnode_modulesand any directory listed in.gnomonignore.- The harness compiles all discovered test declarations before running any of them. A compile error in any file causes the harness to exit with code 2 without running tests.
- Tests within a file are run in declaration order. Files are processed in lexicographic order by path.
6.2 Flags
| Flag | Description |
|---|---|
--filter <pattern> | Run only tests whose name contains pattern (substring match, case-insensitive). |
--fail-fast | Stop after the first failing test. Remaining tests are reported as skipped in TAP output (ok N - <name> # SKIP). |
--reporter tap | Default. Emit TAP-compatible output to stdout. |
--reporter pretty | Emit human-readable output with color and indented diffs. MUST NOT be used in CI (non-deterministic terminal codes break log parsers). |
--no-color | Suppress ANSI color codes in --reporter pretty output. |
6.3 Exit Codes
| Code | Meaning |
|---|---|
0 | All discovered tests passed. |
1 | One or more tests failed. |
2 | Compile error in one or more test files. No tests were run. |
6.4 Environment Variable
GNOMON_TEST_TIMEOUT_MS — per-test execution timeout in milliseconds. Defaults to 5000. A test that exceeds the timeout fails with a timeout diagnostic (M01315). This applies to action execution time only; graph seeding is not subject to the timeout.
7. Output Format
7.1 TAP Stream
The harness emits a TAP 14 stream to stdout. Each test produces exactly one result line:
TAP version 14
1..N
ok 1 - enrollment window is enforced
not ok 2 - bundle cannot be selected without household
---
message: "assertion failed"
assertion: "Bundles.status == selected"
expected: selected
actual: pending
...
ok 3 - actor Client cannot select a bundle in wrong region
- The plan line (
1..N) is emitted before any test result line.Nis the total number of discovered tests. - Skipped tests (from
--fail-fast) emitok N - <name> # SKIP. - Each failing test emits a YAML diagnostic block (lines between
---and...) containing:message: a human-readable summary of the failureassertion: the source text of the failing expressionexpected: the expected value or conditionactual: the actual value observed in the post-whenobject graph
- If the action’s precondition failed (rather than a
thenassertion), the TAP diagnostic block carries code M01319,messageis"precondition not satisfied", andassertionis the source text of the failing precondition guard. - Multiple failing assertions within a single test produce multiple YAML blocks, one per assertion, under the single
not okline.
7.2 --reporter pretty Output
The pretty reporter produces the same information in a human-readable form with indented diffs:
FAIL enrollment window is enforced
✗ Bundles.status == selected
expected: selected
actual: pending
PASS actor Client cannot select a bundle in wrong region
2 tests, 1 failed
The pretty reporter MUST NOT be used in CI environments. Its output is not machine-parseable and is explicitly not TAP-conformant.
8. CI Enforcement
gnomon test MUST be a required, blocking step in every project’s CI pipeline. A CI pipeline that does not include gnomon test is non-conformant with the Gnomon RFC suite.
The enforcement model follows the same pattern as gnomon fmt --check (RFC-0012 §6). Specifically:
- CI MUST run
gnomon testwith no--reporterflag (defaults to TAP) so output is machine-parseable. - The CI step MUST fail if
gnomon testexits with code 1 or 2. - Projects MAY use
--filterin additional optional CI steps (e.g., smoke tests on deploy), but MUST NOT use--filteras a substitute for the unfiltered required step. - TAP output SHOULD be captured and surfaced by the CI system’s test result parser.
- There is no waiver mechanism for skipping
gnomon testin CI. A project that disables this step is in governance violation and will not pass RFC suite conformance review.
Diagnostics
The diagnostic catalog for this RFC is authored in the structured Lean module and
rendered in reference catalog § Diagnostics. Inline mentions in
the prose above (e.g., “rejected with MNNxxx”) cross-link to the catalog
at render time.
10. Relation to the Install Plan
RFC-0005 defines the install plan that governs which declarations are included in a deployed module. Test declarations are excluded from the install plan at the compiler’s bundle-emission phase. Specifically:
- The compiler’s semantic analysis phase processes
testdeclarations in development and test build modes. In production build mode (ciState: ready), encountering atestdeclaration is a hard error (M01300). - The install plan (RFC-0005) never references test declarations. They have no install-plan entry, no version, and no migration lifecycle.
- The object graph seeded by
givenis entirely ephemeral. It is created by the test harness at test start and discarded at test completion. It shares no storage with the application’s runtime object graph.