gnomonlearn
Docs/Hosting/RFC-0047

RFC-0047 · Observability Contract

Versioned draft snapshot

RFCs, reference catalogs, and evidence reflect Spec 3b2e3e7a114fb826b1445d39a48756b8fff629cd. Counts and pending/status notes belong to that snapshot. Read the dated project update for later progress.

Source repositories are private during staging. Links marked “repository access required” need access to the linked repository.

Status Draft corpus. Reference catalogs and RDF projections have generation drift checks; Gnomon proofs and projection semantics remain unverified for release.

View the generated reference catalog →

Read design questions and decisions →


Reference catalog: the diagnostics, linked-RFC graph, defined terms, and normative-rule index for this RFC are in the reference catalog, generated from the structured Lean module. This document carries the prose: design rationale, entity-shape definitions in context, and migration notes.


1. Purpose

This RFC defines what every Gnomon runtime MUST emit for audit, tracing, and metrics. It is the contract that downstream observability tooling — log aggregators, trace collectors, metric stores, compliance auditors — relies on. It specifies the structured audit log schema, OpenTelemetry trace and span emission rules, metric naming conventions, PII redaction at boundaries, error reporting integration with core.errors.*, and correlation ID propagation across operation calls.

The design principle: observability is a runtime obligation, not a runtime feature. Modules cannot opt out, suppress, or alter the emissions defined here. Compliance and incident response depend on uniform, predictable, machine-readable signals.


2. Definitions

TermMeaning
Audit log entryA single structured record describing an operation invocation: who called what, when, with what arguments (hashed), with what result.
SpanAn OpenTelemetry tracing primitive: a typed, timed, attributed slice of work. Every operation invocation produces at least one span.
MetricA typed counter, gauge, or histogram emitted by the runtime, tagged by operation FQN and other dimensions.
Correlation IDAn opaque string that ties together related work across operation calls, sagas, event handlers.
RedactionThe replacement of a sensitive value with a placeholder before the value enters any audit, log, trace, or metric emission.
Required emissionAn audit/trace/metric event that the runtime MUST emit; suppression is a defect.

3. Audit Log

3.1 Schema

Every operation invocation produces exactly one audit log entry on completion (success, failure, or timeout). The entry has the shape:

AuditLogEntry {
  // Identity
  recordId:        UUID                     // unique per entry
  operationFqn:    Text                     // <module-namespace>.<operation-name>@<version>
  operationKind:   query | mutation         // (RFC-0043 §3.2)

  // Principal and tenant
  principalId:     Text | null              // null for anonymous (RFC-0048)
  principalKind:   Text                     // e.g. "user", "service", "system"
  tenantId:        Text | null
  authMethod:      Text                     // verified session method (RFC-0048)

  // Timing
  invokedAt:       Instant                  // request received
  startedAt:       Instant                  // first effect step (post-authorization)
  completedAt:     Instant                  // commit or rollback
  durationMs:      Number

  // Input
  argHash:         Text                     // SHA-256 of canonical-sorted JSON input (post-redaction)
  argSizeBytes:    Number

  // Result
  resultClass:     "success" | "client-error" | "server-error" | "timeout"
  errorType:       Text | null              // FQN of core.errors.* variant or domain error
  errorMessage:    Text | null              // human-readable; redacted

  // Effect pipeline
  effectsExecuted: Number                   // count of effect steps that completed
  effectsTotal:    Number

  // Idempotency
  idempotencyKey:  Text | null
  idempotencyHit:  Boolean                  // true if this was a replay

  // Correlation
  correlationId:        Text                // opaque
  parentCorrelationId:  Text | null         // null for root requests

  // Metadata
  apiSurface:      Text | null              // FQN of the api declaration that exposed this op (RFC-0033)
  protocol:        Text                     // "http", "grpc", "graphql", "trpc", "internal"
  callsite:        Text | null              // for internal calls: caller FQN

  // Context
  bundleVersion:   Text
  runtimeVersion:  Text
}

3.2 Mandatory emission

Audit emission is mandatory and MUST NOT be suppressible by the operation being invoked. Failures to emit (audit sink down, disk full, etc.) are escalated:

  • Audit-emission failure on a successful operation: the operation MUST be considered failed; the runtime MUST roll back (or emit a compensating effect) and return core.errors.InternalError. Emitting the audit successfully is a precondition for marking the operation complete.
  • Audit-emission failure on a failed operation: the failure is escalated to a higher-severity log (M47010); the operation result is unaffected (it already failed).

This invariant — “no completion without audit” — ensures every successful side-effect has a paper trail.

3.3 Argument hashing

argHash is a SHA-256 hex digest of the canonical-sorted-JSON representation of the operation’s input AFTER redaction (§6). The hash is intentionally not reversible; it lets auditors verify “the same input was used” without storing the input itself.

argSizeBytes is the size in bytes of the redacted, canonical-sorted JSON. It supports cost analysis and quota tracking.

3.4 Effect counting

effectsExecuted and effectsTotal (RFC-0043 §5) let auditors detect partial executions:

  • On success: effectsExecuted == effectsTotal.
  • On failure mid-pipeline: effectsExecuted < effectsTotal. The transaction was rolled back; no graph state changed; but the audit records exactly how far the operation got.

4. Tracing (OpenTelemetry)

4.1 Span per operation

Every operation invocation produces at least one OpenTelemetry span. The span has:

AttributeValue
name<operation-fqn>
kindSERVER for inbound (via API), INTERNAL for direct calls
attributes.gnomon.operation_fqn<operation-fqn>
attributes.gnomon.transportquery or mutation
attributes.gnomon.principal_id<principalId> (or <anonymous>)
attributes.gnomon.tenant_id<tenantId>
attributes.gnomon.correlation_id<correlationId>
attributes.gnomon.api_surface<api-fqn> (optional)
attributes.gnomon.protocol<protocol>
attributes.gnomon.idempotency_hittrue / false

4.2 Span events

The span MUST emit events at key pipeline boundaries (RFC-0043 §9.2):

  • gnomon.authentication.complete — after step 3
  • gnomon.authorization.complete — after step 4
  • gnomon.guards.complete — after step 7
  • gnomon.effect.<kind>.start / gnomon.effect.<kind>.end — for each effect step (§5.1 of RFC-0043)
  • gnomon.transaction.commit / gnomon.transaction.rollback — at step 11

Event timestamps allow trace consumers to reconstruct the operation’s per-phase latency.

4.3 Exception recording

On operation failure, the span MUST record the exception via OpenTelemetry’s recordException API:

  • exception.type = the FQN of the error variant (e.g., core.errors.Conflict)
  • exception.message = the redacted error message
  • exception.stacktrace = the stack trace, with redaction applied

The span’s status MUST be set to ERROR for failed operations.

4.4 Trace context propagation

Inbound requests carry an OpenTelemetry trace context (W3C Trace Context, B3, or similar). The runtime MUST extract this context and use it as the parent of the operation span.

Internal operation calls (via execute {}, saga steps, event handlers) MUST propagate the trace context through the runtime’s correlation ID mechanism (§7), so that traces correctly nest.

4.5 Sampling

Sampling is the operator’s choice and is configured via standard OpenTelemetry sampler configuration. The runtime MUST honor whatever sampler the OpenTelemetry SDK is configured with. The runtime MUST NOT internally drop spans before passing them to the SDK; sampling is the SDK’s responsibility.


5. Metrics

5.1 Required metrics

Every runtime MUST emit at least these metrics:

MetricTypeTagsDescription
gnomon.operation.calls_totalCounteroperation, transport, result_classTotal operation invocations
gnomon.operation.duration_msHistogramoperation, transport, result_classOperation end-to-end latency
gnomon.operation.in_flightGaugeoperation, transportIn-flight invocations
gnomon.transaction.commits_totalCounterkindTransactions committed (kind = mutation / install)
gnomon.transaction.rollbacks_totalCounterkind, reasonTransactions rolled back, by reason class
gnomon.idempotency.hits_totalCounteroperationIdempotency replays
gnomon.audit.emissions_totalCounterresultAudit log entries written; tagged success/failure
gnomon.audit.failures_totalCounter(none)Audit emission failures (M47010 events)

5.2 Histogram buckets

gnomon.operation.duration_ms SHOULD use exponential buckets covering 1ms to 60s: [1, 2, 5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000, 10000, 30000, 60000]. Operators MAY tune bucket boundaries via configuration.

5.3 Tag cardinality

The operation tag has high cardinality (one value per operation FQN). Implementations MUST NOT add additional tags that would multiply cardinality (e.g., principal_id MUST NOT be a metric tag). Per-principal data lives in audit logs, not metrics.

5.4 Custom metrics

Operations MAY emit custom metrics via the runtime’s metrics API:

services.metrics.counter("orders.placed", { region })
services.metrics.histogram("checkout.cart_size", value, { tier })

Custom metric names SHOULD use the format <module-namespace>.<metric-name>. The runtime MUST NOT enforce a naming convention but SHOULD lint custom metrics that conflict with the gnomon.* reserved prefix (M47020).


6. PII Redaction

6.1 Sources of redaction

The runtime tracks two sources of “sensitivity”:

  1. Configuration values marked sensitive: true (RFC-0046 §4.1) — values like signing keys, credentials.
  2. Field declarations marked sensitive (RFC-0008 §6.3a) — fields like email, ssn, dateOfBirth.

6.2 Redaction at emission boundaries

Sensitive values MUST be replaced with the placeholder <redacted> (or, for hash-based fields, with a hash digest like <redacted-sha256:abc123...>) at the following emission boundaries:

BoundaryBehavior
Audit log argHashSensitive fields are removed from the input before hashing; the hash is over redacted-and-canonical input
Audit log errorMessageSubstring match against known sensitive values; replace
Span attributesSensitive fields excluded from attribute serialization
Span eventsSensitive payload values redacted before event emission
Span exception messages and stacktracesSame substring-match redaction
Metric tagsTag values that match a sensitive field are dropped (the metric is emitted without that tag)
Log lines (any level)Same substring-match redaction

6.3 The redaction registry

The runtime maintains a registry of sensitive field paths derived from declarations. The registry is populated at boot from:

  • Every entity field declaration with the sensitive modifier
  • Every config declaration with sensitive: true
  • Every client operation’s body field with sensitive modifier (RFC-0024)

The registry is queryable for testing: services.observability.isFieldSensitive(path).

6.4 Hash-vs-omit choice

For fields whose presence is observable (e.g., audit needs to record “the user provided their SSN” without recording the SSN), the runtime emits a hash:

"ssn": "<redacted-sha256:abc123...>"

For fields whose presence is itself sensitive (e.g., medical-flag), the runtime omits the field entirely.

The choice is per-field, declared via the sensitive modifier’s optional sub-form:

ssn: Text required sensitive(hash)         // emit hash placeholder
medicalFlag: Boolean required sensitive(omit)  // omit entirely

sensitive without a sub-form defaults to hash.

6.5 Defect on leakage

Any leakage of a sensitive value into any of the boundaries in §6.2 is a defect (M47030). Implementations SHOULD have automated tests that scan emissions for known sensitive values and fail the build on detection.


7. Correlation IDs

7.1 Format

A correlation ID is an opaque, URL-safe string. Implementations SHOULD use a UUIDv7 or ULID-format value. The runtime MUST treat correlation IDs as opaque — no parsing, no semantics beyond equality.

7.2 Generation

A correlation ID is generated when:

  • A request enters the runtime via a protocol server (HTTP, gRPC, etc.) AND no correlation ID is provided in the request headers.
  • An internal operation is invoked with no caller correlation ID.
  • A scheduled job or cron-triggered operation starts.

The runtime MUST accept caller-supplied correlation IDs from a well-known header (X-Correlation-Id for HTTP, equivalent for other protocols) but MUST NOT trust them as authentication.

7.3 Propagation

Correlation IDs propagate through:

  • Internal operation calls (execute {}, saga steps, event handlers) — automatic via the runtime’s request context.
  • Outbound client operations (RFC-0024) — added as X-Correlation-Id header / equivalent.
  • Event emissions (event.emit effect, RFC-0043 §5.2) — included in the event envelope.
  • Background jobs derived from the operation — propagated as parentCorrelationId.

7.4 Parent linkage

When an operation is invoked from another operation, the new operation’s correlation ID is fresh; its parentCorrelationId (audit log §3.1) is set to the caller’s correlation ID. This forms a tree of correlated work that auditors can traverse.


8. Error Reporting

8.1 Integration with core.errors

Operation failures are reported as instances of core.errors.* variants (RFC-0035 §4) or domain error variants declared by the operation. The audit log’s errorType field carries the FQN of the variant; the span’s exception.type carries the same.

8.2 Domain errors

Operations MAY declare domain-specific error variants:

errors {
  InsufficientFunds:  variant
  PolicyViolation:    variant
}

Domain errors emit through the same audit/trace channel as core.errors.*. The audit errorType is the FQN of the variant; the audit errorMessage is the variant’s meaning field plus any structured payload, with redaction applied.

8.3 Stack traces

Stack traces SHOULD be included in span exception records and SHOULD be excluded from audit log entries (where they have no value and may leak internal paths).

The runtime MUST NOT emit stack traces in responses to external callers; that’s a separate concern (the API layer redacts, RFC-0033).


9. Invariants

  1. Every operation invocation produces exactly one audit log entry on completion (§3.2).
  2. Audit emission failure on a successful operation forces the operation to fail (§3.2).
  3. Every operation invocation produces at least one OpenTelemetry span (§4.1).
  4. Required metrics (§5.1) are emitted by every runtime; suppression is a defect.
  5. Sensitive values MUST be redacted at every emission boundary (§6.2).
  6. Leakage of sensitive values is a defect (§6.5, M47030).
  7. Correlation IDs propagate through internal calls, outbound clients, and events (§7.3).
  8. Tag cardinality on metrics is bounded; per-principal data is audit-only, not metric (§5.3).
  9. Stack traces are span-only; audit log entries do not include them (§8.3).
  10. The redaction registry is the authoritative source for “is this field sensitive” (§6.3).

10. Compatibility and Migration

This RFC formalizes contracts that legacy code partially observes. The migration path:

  • Existing audit-log emitters MUST be brought into compliance with the §3.1 schema. Tools that consume the legacy schema MAY be supported via a compatibility shim, but new emitters MUST use the schema defined here.
  • OpenTelemetry instrumentation MUST be added if not already present. The legacy system uses scattered logging; this RFC requires structured tracing.
  • The redaction registry is new. Modules that previously relied on opt-in redaction (or had no redaction at all) MUST opt into the new boundary-based redaction by tagging fields sensitive (RFC-0008 §6.3a).

Diagnostics

The diagnostic catalog for this RFC is authored in the structured Lean module and rendered in reference catalog § Diagnostics. Inline mentions in the prose above (e.g., “rejected with MNNxxx”) cross-link to the catalog at render time.

Linked References

The cross-RFC dependency graph is rendered in reference catalog § Linked References. Out-of-KG pointers to legacy code or external resources, when present, remain in this document under “Source-tree references” below.