gnomonlearn

RFC-0013 · Test Execution Model

Versioned draft snapshot

RFCs, reference catalogs, and evidence reflect Spec 3b2e3e7a114fb826b1445d39a48756b8fff629cd. Counts and pending/status notes belong to that snapshot. Read the dated project update for later progress.

Source repositories are private during staging. Links marked “repository access required” need access to the linked repository.

Status Draft corpus. Reference catalogs and RDF projections have generation drift checks; Gnomon proofs and projection semantics remain unverified for release.

View the generated reference catalog →

Read design questions and decisions →

Status: Draft Owners: DX, Compiler

1. Scope

Defines the execution model for test declarations in Gnomon source files. The test declaration kind is part of the language grammar (RFC-0008 §8.12) and uses the constraint expression language (RFC-0008 §8.22). This RFC specifies the semantics of the given, when, and then blocks; the in-memory object graph lifecycle during test execution; the gnomon test CLI harness; output format; CI enforcement requirements; and all diagnostic codes in the M013xx range.

Test declarations are exclusively a development-time construct. They are stripped from production bundles and MUST NOT appear in any artifact with ciState: ready (RFC-0008 §11.2, diagnostic M08786). This RFC does not define the language grammar — that is fixed in RFC-0008 §8.12 — nor does it define the constraint expression language (RFC-0008 §8.22). It defines what happens at runtime when a test is executed.

2. Goals

  1. Tests are first-class declarations in .gnm source files, authored with the same syntax as production code and subject to the same compiler passes.
  2. The test execution environment is a deterministic, isolated, in-memory replica of the Gnomon object graph (RFC-0007). No external state, no network, no database.
  3. A single test exercises exactly one action invocation. Tests are not scenarios or end-to-end sequences.
  4. Test output is machine-readable (TAP-compatible) and human-readable simultaneously. No bespoke output format.
  5. gnomon test is a required, blocking CI step. Projects without test enforcement are non-conformant.
  6. Failure messages surface the specific assertion that failed, with a diff of expected versus actual state.

Linked References

The cross-RFC dependency graph is rendered in reference catalog § Linked References. Out-of-KG pointers to legacy code or external resources, when present, remain in this document under “Source-tree references” below.

4. Invariants

  1. A test declaration MUST NOT be included in any compiled bundle artifact where ciState: ready. Including one is a compile error (M01300).
  2. Each test declaration MUST contain exactly one given block, exactly one when block, and exactly one then block, in that order. Any other structure is a parse error (M01301).
  3. The when block MUST reference exactly one declared action. Referencing an undeclared identifier, a non-action declaration kind, or more than one invocation is a compile error (M01302).
  4. Every expression in the then block MUST be a boolean-valued expression per the constraint expression language (RFC-0008 §8.22). A non-boolean expression in then is a compile error (M01303).
  5. The in-memory object graph seeded by given is isolated per test. No state leaks between test executions.
  6. Test names MUST be unique within a module. Duplicate names within the same .gnm file are a compile error (M01304).
  7. The gnomon test harness MUST exit with code 0 only when every discovered test passes. Any failing test produces exit code 1. A compile error in any test file produces exit code 2 without running any tests.
  8. TAP output MUST be emitted to stdout. Diagnostic messages (compile errors, harness warnings) MUST be emitted to stderr.

5. Test Declaration Anatomy

5.1 Syntax Recap

test "enrollment window is enforced" {
  given {
    Household { id: "h1", members: [...] }
    Bundles    { id: "b1", status: pending, household: "h1" }
  }
  when  selectBundle { bundleId: "b1", actor: Client }
  then  Bundles.status == selected
}

The test name is a string literal. It is used as the label in TAP output and in failure messages. It SHOULD describe the invariant being verified, not the mechanism.

5.2 given Block — Object Graph Seeding

The given block seeds an isolated, in-memory instance of the Gnomon object graph (RFC-0007) with an initial set of entities. Each line in the block is an entity initializer:

<EntityKind> { <field>: <value>, ... }
  • <EntityKind> MUST refer to a declared entity kind visible in the module’s scope. An unknown kind is a compile error (M01305).
  • Every field listed MUST exist on the declared entity kind. An unknown field is a compile error (M01306).
  • Fields not listed take their declared default values. If a field has no default and is not listed, that is a compile error (M01307).
  • The id field is required for all entities. It MUST be a string literal. Duplicate IDs within the same given block are a compile error (M01308).
  • Cross-entity references use the quoted string ID of the target entity, not a structural reference. The compiler resolves these IDs against the given graph at test compile time. A reference to an ID not present in the given block is a compile error (M01309).
  • List fields accept inline array syntax ([...]). Each element follows the same rules as standalone field values.
  • Enum fields use unquoted enum variant names (e.g., status: pending). An unknown variant is a compile error (M01310).

The given graph is built before the when invocation. It is strictly immutable from the perspective of the test harness setup phase — only the when action may mutate it.

5.3 when Block — Action Invocation

The when block specifies a single action invocation:

when  <actionName> { <arg>: <value>, ..., actor: <PrincipalKind> }
  • <actionName> MUST resolve to an action declaration (RFC-0008 §8.7) in scope. Any other declaration kind is a compile error (M01302).
  • The actor field is required and MUST match one of the principal kinds declared for that action. An unknown principal kind is a compile error (M01311).
  • All required arguments for the action MUST be supplied. Missing required arguments are a compile error (M01312). Extra arguments not declared on the action are a compile error (M01313).
  • The action is executed against the object graph seeded by given. The action’s declared preconditions, guards, and policy rules are evaluated as they would be at runtime.
  • If the action’s preconditions are not satisfied by the given state, the test fails with diagnostic M01319 indicating which precondition was violated (not a compile error — this is a runtime test failure, reported in TAP output).
  • The harness does not mock the action. It runs the real compiled action logic against the in-memory graph.
  • If the action under test contains an execute {} block (RFC-0008 §6.10), the block is executed before mutations. In the test harness, external I/O calls within execute {} MUST be stubbed via the test declaration’s mock {} block (if defined) or MUST be omitted — the test harness MUST NOT make real network calls.

5.4 then Block — Assertions

The then block contains one or more assertion expressions:

then
  Bundles.status == selected
  Household.members.length >= 1
  • Each expression MUST be boolean-valued per RFC-0008 §8.22. A non-boolean expression is a compile error (M01303).
  • Expressions are evaluated against the object graph state after the when action has completed.
  • All assertions are evaluated independently. A failure in one assertion does not prevent evaluation of subsequent assertions. All failures for a single test are reported together.
  • Dot-path access follows the same semantics as the constraint expression language (RFC-0008 §8.22): field access on an entity kind resolves against all entities of that kind in the graph. If the entity kind is ambiguous (multiple instances in the graph), the path MUST be qualified with an ID predicate (e.g., Bundles[id="b1"].status == selected). Ambiguous paths without qualification are a compile error (M01314).
  • Comparison operators, arithmetic, and logical connectives follow RFC-0008 §8.22 exactly. No test-specific operators are introduced.

6. Test Harness CLI

6.1 Command

gnomon test [path...]
  • path is zero or more paths to .gnm files or directories. If no path is given, the harness discovers all .gnm files under the current working directory, excluding node_modules and any directory listed in .gnomonignore.
  • The harness compiles all discovered test declarations before running any of them. A compile error in any file causes the harness to exit with code 2 without running tests.
  • Tests within a file are run in declaration order. Files are processed in lexicographic order by path.

6.2 Flags

FlagDescription
--filter <pattern>Run only tests whose name contains pattern (substring match, case-insensitive).
--fail-fastStop after the first failing test. Remaining tests are reported as skipped in TAP output (ok N - <name> # SKIP).
--reporter tapDefault. Emit TAP-compatible output to stdout.
--reporter prettyEmit human-readable output with color and indented diffs. MUST NOT be used in CI (non-deterministic terminal codes break log parsers).
--no-colorSuppress ANSI color codes in --reporter pretty output.

6.3 Exit Codes

CodeMeaning
0All discovered tests passed.
1One or more tests failed.
2Compile error in one or more test files. No tests were run.

6.4 Environment Variable

GNOMON_TEST_TIMEOUT_MS — per-test execution timeout in milliseconds. Defaults to 5000. A test that exceeds the timeout fails with a timeout diagnostic (M01315). This applies to action execution time only; graph seeding is not subject to the timeout.

7. Output Format

7.1 TAP Stream

The harness emits a TAP 14 stream to stdout. Each test produces exactly one result line:

TAP version 14
1..N
ok 1 - enrollment window is enforced
not ok 2 - bundle cannot be selected without household
  ---
  message: "assertion failed"
  assertion: "Bundles.status == selected"
  expected: selected
  actual:   pending
  ...
ok 3 - actor Client cannot select a bundle in wrong region
  • The plan line (1..N) is emitted before any test result line. N is the total number of discovered tests.
  • Skipped tests (from --fail-fast) emit ok N - <name> # SKIP.
  • Each failing test emits a YAML diagnostic block (lines between --- and ...) containing:
    • message: a human-readable summary of the failure
    • assertion: the source text of the failing expression
    • expected: the expected value or condition
    • actual: the actual value observed in the post-when object graph
  • If the action’s precondition failed (rather than a then assertion), the TAP diagnostic block carries code M01319, message is "precondition not satisfied", and assertion is the source text of the failing precondition guard.
  • Multiple failing assertions within a single test produce multiple YAML blocks, one per assertion, under the single not ok line.

7.2 --reporter pretty Output

The pretty reporter produces the same information in a human-readable form with indented diffs:

  FAIL  enrollment window is enforced
        ✗ Bundles.status == selected
            expected: selected
            actual:   pending

  PASS  actor Client cannot select a bundle in wrong region

  2 tests, 1 failed

The pretty reporter MUST NOT be used in CI environments. Its output is not machine-parseable and is explicitly not TAP-conformant.

8. CI Enforcement

gnomon test MUST be a required, blocking step in every project’s CI pipeline. A CI pipeline that does not include gnomon test is non-conformant with the Gnomon RFC suite.

The enforcement model follows the same pattern as gnomon fmt --check (RFC-0012 §6). Specifically:

  1. CI MUST run gnomon test with no --reporter flag (defaults to TAP) so output is machine-parseable.
  2. The CI step MUST fail if gnomon test exits with code 1 or 2.
  3. Projects MAY use --filter in additional optional CI steps (e.g., smoke tests on deploy), but MUST NOT use --filter as a substitute for the unfiltered required step.
  4. TAP output SHOULD be captured and surfaced by the CI system’s test result parser.
  5. There is no waiver mechanism for skipping gnomon test in CI. A project that disables this step is in governance violation and will not pass RFC suite conformance review.

Diagnostics

The diagnostic catalog for this RFC is authored in the structured Lean module and rendered in reference catalog § Diagnostics. Inline mentions in the prose above (e.g., “rejected with MNNxxx”) cross-link to the catalog at render time.

10. Relation to the Install Plan

RFC-0005 defines the install plan that governs which declarations are included in a deployed module. Test declarations are excluded from the install plan at the compiler’s bundle-emission phase. Specifically:

  • The compiler’s semantic analysis phase processes test declarations in development and test build modes. In production build mode (ciState: ready), encountering a test declaration is a hard error (M01300).
  • The install plan (RFC-0005) never references test declarations. They have no install-plan entry, no version, and no migration lifecycle.
  • The object graph seeded by given is entirely ephemeral. It is created by the test harness at test start and discarded at test completion. It shares no storage with the application’s runtime object graph.