RFC-0047 · Observability Contract
Versioned draft snapshot
RFCs, reference catalogs, and evidence reflect Spec 3b2e3e7a114fb826b1445d39a48756b8fff629cd. Counts and pending/status notes belong to that snapshot. Read the dated project update for later progress.
Source repositories are private during staging. Links marked “repository access required” need access to the linked repository.
| Status | Draft corpus. Reference catalogs and RDF projections have generation drift checks; Gnomon proofs and projection semantics remain unverified for release. |
|---|
View the generated reference catalog →
Read design questions and decisions →
Reference catalog: the diagnostics, linked-RFC graph, defined terms, and normative-rule index for this RFC are in the reference catalog, generated from the structured Lean module. This document carries the prose: design rationale, entity-shape definitions in context, and migration notes.
1. Purpose
This RFC defines what every Gnomon runtime MUST emit for audit, tracing, and metrics. It is the contract that downstream observability tooling — log aggregators, trace collectors, metric stores, compliance auditors — relies on. It specifies the structured audit log schema, OpenTelemetry trace and span emission rules, metric naming conventions, PII redaction at boundaries, error reporting integration with core.errors.*, and correlation ID propagation across operation calls.
The design principle: observability is a runtime obligation, not a runtime feature. Modules cannot opt out, suppress, or alter the emissions defined here. Compliance and incident response depend on uniform, predictable, machine-readable signals.
2. Definitions
| Term | Meaning |
|---|---|
| Audit log entry | A single structured record describing an operation invocation: who called what, when, with what arguments (hashed), with what result. |
| Span | An OpenTelemetry tracing primitive: a typed, timed, attributed slice of work. Every operation invocation produces at least one span. |
| Metric | A typed counter, gauge, or histogram emitted by the runtime, tagged by operation FQN and other dimensions. |
| Correlation ID | An opaque string that ties together related work across operation calls, sagas, event handlers. |
| Redaction | The replacement of a sensitive value with a placeholder before the value enters any audit, log, trace, or metric emission. |
| Required emission | An audit/trace/metric event that the runtime MUST emit; suppression is a defect. |
3. Audit Log
3.1 Schema
Every operation invocation produces exactly one audit log entry on completion (success, failure, or timeout). The entry has the shape:
AuditLogEntry {
// Identity
recordId: UUID // unique per entry
operationFqn: Text // <module-namespace>.<operation-name>@<version>
operationKind: query | mutation // (RFC-0043 §3.2)
// Principal and tenant
principalId: Text | null // null for anonymous (RFC-0048)
principalKind: Text // e.g. "user", "service", "system"
tenantId: Text | null
authMethod: Text // verified session method (RFC-0048)
// Timing
invokedAt: Instant // request received
startedAt: Instant // first effect step (post-authorization)
completedAt: Instant // commit or rollback
durationMs: Number
// Input
argHash: Text // SHA-256 of canonical-sorted JSON input (post-redaction)
argSizeBytes: Number
// Result
resultClass: "success" | "client-error" | "server-error" | "timeout"
errorType: Text | null // FQN of core.errors.* variant or domain error
errorMessage: Text | null // human-readable; redacted
// Effect pipeline
effectsExecuted: Number // count of effect steps that completed
effectsTotal: Number
// Idempotency
idempotencyKey: Text | null
idempotencyHit: Boolean // true if this was a replay
// Correlation
correlationId: Text // opaque
parentCorrelationId: Text | null // null for root requests
// Metadata
apiSurface: Text | null // FQN of the api declaration that exposed this op (RFC-0033)
protocol: Text // "http", "grpc", "graphql", "trpc", "internal"
callsite: Text | null // for internal calls: caller FQN
// Context
bundleVersion: Text
runtimeVersion: Text
}
3.2 Mandatory emission
Audit emission is mandatory and MUST NOT be suppressible by the operation being invoked. Failures to emit (audit sink down, disk full, etc.) are escalated:
- Audit-emission failure on a successful operation: the operation MUST be considered failed; the runtime MUST roll back (or emit a compensating effect) and return
core.errors.InternalError. Emitting the audit successfully is a precondition for marking the operation complete. - Audit-emission failure on a failed operation: the failure is escalated to a higher-severity log (M47010); the operation result is unaffected (it already failed).
This invariant — “no completion without audit” — ensures every successful side-effect has a paper trail.
3.3 Argument hashing
argHash is a SHA-256 hex digest of the canonical-sorted-JSON representation of the operation’s input AFTER redaction (§6). The hash is intentionally not reversible; it lets auditors verify “the same input was used” without storing the input itself.
argSizeBytes is the size in bytes of the redacted, canonical-sorted JSON. It supports cost analysis and quota tracking.
3.4 Effect counting
effectsExecuted and effectsTotal (RFC-0043 §5) let auditors detect partial executions:
- On success:
effectsExecuted == effectsTotal. - On failure mid-pipeline:
effectsExecuted < effectsTotal. The transaction was rolled back; no graph state changed; but the audit records exactly how far the operation got.
4. Tracing (OpenTelemetry)
4.1 Span per operation
Every operation invocation produces at least one OpenTelemetry span. The span has:
| Attribute | Value |
|---|---|
name | <operation-fqn> |
kind | SERVER for inbound (via API), INTERNAL for direct calls |
attributes.gnomon.operation_fqn | <operation-fqn> |
attributes.gnomon.transport | query or mutation |
attributes.gnomon.principal_id | <principalId> (or <anonymous>) |
attributes.gnomon.tenant_id | <tenantId> |
attributes.gnomon.correlation_id | <correlationId> |
attributes.gnomon.api_surface | <api-fqn> (optional) |
attributes.gnomon.protocol | <protocol> |
attributes.gnomon.idempotency_hit | true / false |
4.2 Span events
The span MUST emit events at key pipeline boundaries (RFC-0043 §9.2):
gnomon.authentication.complete— after step 3gnomon.authorization.complete— after step 4gnomon.guards.complete— after step 7gnomon.effect.<kind>.start/gnomon.effect.<kind>.end— for each effect step (§5.1 of RFC-0043)gnomon.transaction.commit/gnomon.transaction.rollback— at step 11
Event timestamps allow trace consumers to reconstruct the operation’s per-phase latency.
4.3 Exception recording
On operation failure, the span MUST record the exception via OpenTelemetry’s recordException API:
exception.type= the FQN of the error variant (e.g.,core.errors.Conflict)exception.message= the redacted error messageexception.stacktrace= the stack trace, with redaction applied
The span’s status MUST be set to ERROR for failed operations.
4.4 Trace context propagation
Inbound requests carry an OpenTelemetry trace context (W3C Trace Context, B3, or similar). The runtime MUST extract this context and use it as the parent of the operation span.
Internal operation calls (via execute {}, saga steps, event handlers) MUST propagate the trace context through the runtime’s correlation ID mechanism (§7), so that traces correctly nest.
4.5 Sampling
Sampling is the operator’s choice and is configured via standard OpenTelemetry sampler configuration. The runtime MUST honor whatever sampler the OpenTelemetry SDK is configured with. The runtime MUST NOT internally drop spans before passing them to the SDK; sampling is the SDK’s responsibility.
5. Metrics
5.1 Required metrics
Every runtime MUST emit at least these metrics:
| Metric | Type | Tags | Description |
|---|---|---|---|
gnomon.operation.calls_total | Counter | operation, transport, result_class | Total operation invocations |
gnomon.operation.duration_ms | Histogram | operation, transport, result_class | Operation end-to-end latency |
gnomon.operation.in_flight | Gauge | operation, transport | In-flight invocations |
gnomon.transaction.commits_total | Counter | kind | Transactions committed (kind = mutation / install) |
gnomon.transaction.rollbacks_total | Counter | kind, reason | Transactions rolled back, by reason class |
gnomon.idempotency.hits_total | Counter | operation | Idempotency replays |
gnomon.audit.emissions_total | Counter | result | Audit log entries written; tagged success/failure |
gnomon.audit.failures_total | Counter | (none) | Audit emission failures (M47010 events) |
5.2 Histogram buckets
gnomon.operation.duration_ms SHOULD use exponential buckets covering 1ms to 60s: [1, 2, 5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000, 10000, 30000, 60000]. Operators MAY tune bucket boundaries via configuration.
5.3 Tag cardinality
The operation tag has high cardinality (one value per operation FQN). Implementations MUST NOT add additional tags that would multiply cardinality (e.g., principal_id MUST NOT be a metric tag). Per-principal data lives in audit logs, not metrics.
5.4 Custom metrics
Operations MAY emit custom metrics via the runtime’s metrics API:
services.metrics.counter("orders.placed", { region })
services.metrics.histogram("checkout.cart_size", value, { tier })
Custom metric names SHOULD use the format <module-namespace>.<metric-name>. The runtime MUST NOT enforce a naming convention but SHOULD lint custom metrics that conflict with the gnomon.* reserved prefix (M47020).
6. PII Redaction
6.1 Sources of redaction
The runtime tracks two sources of “sensitivity”:
- Configuration values marked
sensitive: true(RFC-0046 §4.1) — values like signing keys, credentials. - Field declarations marked
sensitive(RFC-0008 §6.3a) — fields likeemail,ssn,dateOfBirth.
6.2 Redaction at emission boundaries
Sensitive values MUST be replaced with the placeholder <redacted> (or, for hash-based fields, with a hash digest like <redacted-sha256:abc123...>) at the following emission boundaries:
| Boundary | Behavior |
|---|---|
Audit log argHash | Sensitive fields are removed from the input before hashing; the hash is over redacted-and-canonical input |
Audit log errorMessage | Substring match against known sensitive values; replace |
| Span attributes | Sensitive fields excluded from attribute serialization |
| Span events | Sensitive payload values redacted before event emission |
| Span exception messages and stacktraces | Same substring-match redaction |
| Metric tags | Tag values that match a sensitive field are dropped (the metric is emitted without that tag) |
| Log lines (any level) | Same substring-match redaction |
6.3 The redaction registry
The runtime maintains a registry of sensitive field paths derived from declarations. The registry is populated at boot from:
- Every
entityfield declaration with thesensitivemodifier - Every
configdeclaration withsensitive: true - Every
clientoperation’sbodyfield withsensitivemodifier (RFC-0024)
The registry is queryable for testing: services.observability.isFieldSensitive(path).
6.4 Hash-vs-omit choice
For fields whose presence is observable (e.g., audit needs to record “the user provided their SSN” without recording the SSN), the runtime emits a hash:
"ssn": "<redacted-sha256:abc123...>"
For fields whose presence is itself sensitive (e.g., medical-flag), the runtime omits the field entirely.
The choice is per-field, declared via the sensitive modifier’s optional sub-form:
ssn: Text required sensitive(hash) // emit hash placeholder
medicalFlag: Boolean required sensitive(omit) // omit entirely
sensitive without a sub-form defaults to hash.
6.5 Defect on leakage
Any leakage of a sensitive value into any of the boundaries in §6.2 is a defect (M47030). Implementations SHOULD have automated tests that scan emissions for known sensitive values and fail the build on detection.
7. Correlation IDs
7.1 Format
A correlation ID is an opaque, URL-safe string. Implementations SHOULD use a UUIDv7 or ULID-format value. The runtime MUST treat correlation IDs as opaque — no parsing, no semantics beyond equality.
7.2 Generation
A correlation ID is generated when:
- A request enters the runtime via a protocol server (HTTP, gRPC, etc.) AND no correlation ID is provided in the request headers.
- An internal operation is invoked with no caller correlation ID.
- A scheduled job or cron-triggered operation starts.
The runtime MUST accept caller-supplied correlation IDs from a well-known header (X-Correlation-Id for HTTP, equivalent for other protocols) but MUST NOT trust them as authentication.
7.3 Propagation
Correlation IDs propagate through:
- Internal operation calls (
execute {}, saga steps, event handlers) — automatic via the runtime’s request context. - Outbound
clientoperations (RFC-0024) — added asX-Correlation-Idheader / equivalent. - Event emissions (
event.emiteffect, RFC-0043 §5.2) — included in the event envelope. - Background jobs derived from the operation — propagated as
parentCorrelationId.
7.4 Parent linkage
When an operation is invoked from another operation, the new operation’s correlation ID is fresh; its parentCorrelationId (audit log §3.1) is set to the caller’s correlation ID. This forms a tree of correlated work that auditors can traverse.
8. Error Reporting
8.1 Integration with core.errors
Operation failures are reported as instances of core.errors.* variants (RFC-0035 §4) or domain error variants declared by the operation. The audit log’s errorType field carries the FQN of the variant; the span’s exception.type carries the same.
8.2 Domain errors
Operations MAY declare domain-specific error variants:
errors {
InsufficientFunds: variant
PolicyViolation: variant
}
Domain errors emit through the same audit/trace channel as core.errors.*. The audit errorType is the FQN of the variant; the audit errorMessage is the variant’s meaning field plus any structured payload, with redaction applied.
8.3 Stack traces
Stack traces SHOULD be included in span exception records and SHOULD be excluded from audit log entries (where they have no value and may leak internal paths).
The runtime MUST NOT emit stack traces in responses to external callers; that’s a separate concern (the API layer redacts, RFC-0033).
9. Invariants
- Every operation invocation produces exactly one audit log entry on completion (§3.2).
- Audit emission failure on a successful operation forces the operation to fail (§3.2).
- Every operation invocation produces at least one OpenTelemetry span (§4.1).
- Required metrics (§5.1) are emitted by every runtime; suppression is a defect.
- Sensitive values MUST be redacted at every emission boundary (§6.2).
- Leakage of sensitive values is a defect (§6.5, M47030).
- Correlation IDs propagate through internal calls, outbound clients, and events (§7.3).
- Tag cardinality on metrics is bounded; per-principal data is audit-only, not metric (§5.3).
- Stack traces are span-only; audit log entries do not include them (§8.3).
- The redaction registry is the authoritative source for “is this field sensitive” (§6.3).
10. Compatibility and Migration
This RFC formalizes contracts that legacy code partially observes. The migration path:
- Existing audit-log emitters MUST be brought into compliance with the §3.1 schema. Tools that consume the legacy schema MAY be supported via a compatibility shim, but new emitters MUST use the schema defined here.
- OpenTelemetry instrumentation MUST be added if not already present. The legacy system uses scattered logging; this RFC requires structured tracing.
- The redaction registry is new. Modules that previously relied on opt-in redaction (or had no redaction at all) MUST opt into the new boundary-based redaction by tagging fields
sensitive(RFC-0008 §6.3a).
Diagnostics
The diagnostic catalog for this RFC is authored in the structured Lean module and
rendered in reference catalog § Diagnostics. Inline mentions in
the prose above (e.g., “rejected with MNNxxx”) cross-link to the catalog
at render time.
Linked References
The cross-RFC dependency graph is rendered in reference catalog § Linked References. Out-of-KG pointers to legacy code or external resources, when present, remain in this document under “Source-tree references” below.