Digest-bound source preservation proposal
Versioned draft snapshot
RFCs, reference catalogs, and evidence reflect Spec 3b2e3e7a114fb826b1445d39a48756b8fff629cd. Counts and pending/status notes belong to that snapshot. Read the dated project update for later progress.
Source repositories are private during staging. Links marked “repository access required” need access to the linked repository.
This is a trace and gap record from the pinned draft specification. It does not establish proof or implementation conformance.
Status: proposal, owned by Language, Compiler, and DX. This checkpoint implements a separate source artifact without selecting the pending canonical AST comparison stage, descriptor identity, or node identity contract.
Concrete implementation
source_trivia.proto (repository access required) defines a
gnomon.spec.proposal.source.v1.SourceArtifact. Its SHA-256 and byte length bind
it to the exact UTF-8 input. Comment ranges and declaration ranges are half-open
UTF-8 byte offsets. Comments retain exact delimiters, Unicode spelling, and
consumed LF or CRLF. Block ranges stop at */; line/doc ranges include their
required newline. No line ending, Unicode, or comment text is normalized.
Declaration source entries record the authored production and source-order ordinal. An ordinal is local to this digest-bound parse, not a stable or canonical node ID. Explicit optional scalar presence distinguishes ordinal zero from a missing owner. The map currently covers top-level declarations and language comments; it does not promise token, expression, or complete whitespace coverage for a lossless formatter. Comment-looking text inside strings and opaque foreign scripts is excluded. Script header comments remain Gnomon comments.
The artifact comes from retained grammar production locations, not a second
regular-expression lexer or fixture annotations. Syntax failure emits UNPARSED
with only the exact source binding: no partial or guessed comment map. Grammar
success emits RETAINED_PARSE even when the typed AST refuses an unsupported
documentation owner. A refusal contributes no successful AST coverage.
ProposalParseResult.source_artifact is separate from parsed_file. AST fixture
comparisons compare only the typed authored AST; source fixture comparisons
compare the separate artifact. The artifact can also be emitted independently:
node tools/parse_proposed_syntax.mjs --source-artifact input.gnm
python3 tools/check_proposed_source.py
python3 -m unittest discover -s tools/tests -p 'test_proposed_source.py'
Six handwritten source expectations
exercise Unicode and CRLF, comment/declaration order, leading documentation,
opaque script exclusion, an ownership refusal, BOM syntax refusal, and empty
input. Expected ranges and owner assignments were authored independently from
the lowering; digests were computed independently with Python hashlib.
Documentation ownership established by current sources
RFC-0008 §8.1 says consecutive /// lines belong to the immediately following
top-level declaration. RFC-0012 §4.3 places them directly above that declaration
with no blank line before its keyword. The proposal implements their
unambiguous intersection: consecutive leading /// lines, allowing indentation,
directly above an undecorated, unmodified top-level declaration. One attachment
records its declaration ordinal and the ordered comment ordinals. Empty doc
lines, Unicode, and CRLF are preserved. No prose normalization or synthesized
doc string is implied.
The artifact records this owner binding; compiled bundle doc metadata and its
representation remain compiler work. CommonMark validation, M08760, and rejection
of structured @tag semantics are not implemented by this syntax proposal.
Recording exact authored text does not authorize those semantics.
Attachment evidence and remaining acceptance
| Case | Current source evidence | Candidate behavior | Required acceptance before promotion |
|---|---|---|---|
Consecutive leading /// directly above bare declaration | RFC-0008 §8.1 and RFC-0012 §4.3 agree | Attach in separate source artifact | Both independent adapters reproduce owner/order/text; compiler emits reviewed doc metadata |
| Blank line or ordinary comment between docs and declaration | RFC-0012 requires direct adjacency | Refuse UNSUPPORTED_DOC_COMMENT; preserve exact source artifact | Owner decides rejection or recovery and normative diagnostic |
| Docs before/between decorators | RFC-0012 independently puts docs and decorators immediately above the declaration keyword | Refuse; preserve docs and complete declaration range | Specify one ordering and attachment rule; fixtures for before/between/after decorator groups |
| Docs above or between modifiers | RFC-0012 puts modifiers in the header, but does not define attachment across interrupted headers | Conservative refusal, including uninterrupted modified headers | Specify header anchor and allowed placement; verify modifier/order preservation |
| Body, inline, orphan, or trailing docs | RFC-0008 §8.1 assigns docs to following top-level declarations, not member nodes | Refuse rather than invent a member owner | Define permitted owners/recovery with independent fixtures and normative diagnostics |
/** ... */ | RFC-0014 calls it documentation; RFC-0008 lexical grammar and §12.1 classify it as an ordinary block comment | Preserve as BLOCK, without a doc owner | Reconcile RFCs and update lexical/attachment fixtures if this becomes a doc form |
//compiled: and //compiled-by: | RFC-0012 §4.3 requires verbatim preservation immediately after a rule | Preserve exact ordinary line comments/ranges; no rule owner ID | Define typed rule source references and adjacency; formatter tests retain exact text immediately after its rule |
CommonMark and forbidden @tag metadata | RFC-0008 §§8.1/12.1 specify prose, M08760, and no tag syntax | Preserve text without interpretation or semantic acceptance | Compiler fixtures test content diagnostics, warning behavior, and reviewed doc normalization |
Twenty source regressions test the exact bindings and explicit refusals, including malformed digests, missing ordinal presence, split Unicode ranges, altered text, invented declaration kinds, reordered ranges/owners, skipped or duplicate comment indices, and forged blank-line ownership. Location-free AST equality is tested separately from differing source artifacts. The fixtures do not exercise an independent compiler or compiled metadata emitter.
Remaining language conformance gates
The AST proposal now has 32 positive handwritten expectations and three
refusals, retaining 175/180 productions. Five retained catalog productions are
still unreachable from the generated start rule: WHITESPACE, query-shorthand,
nat-lang-stmt, nat-lang-block, and qualified-set. Their earlier evidence
and grammar-review requirements remain unchanged. No names are copied from
fixture annotations to manufacture coverage.
Canonical comparison-stage and namespace approval, production/message review,
documentation rules above, complete source mappings for formatter requirements,
known authored grammar ambiguities, semantic diagnostics, and an independent
compiler adapter remain open. The adapter must use the reviewed descriptor and
same independently authored canonical expectations; routing both sides through
this parser does not satisfy compiler conformance. grammar/fixtures/ still has
no normative corpus, and check_fixture_envelopes.py --require-corpus remains red.