gnomonlearn
Docs/Digest-bound source preservation proposal

Digest-bound source preservation proposal

Versioned draft snapshot

RFCs, reference catalogs, and evidence reflect Spec 3b2e3e7a114fb826b1445d39a48756b8fff629cd. Counts and pending/status notes belong to that snapshot. Read the dated project update for later progress.

Source repositories are private during staging. Links marked “repository access required” need access to the linked repository.

This is a trace and gap record from the pinned draft specification. It does not establish proof or implementation conformance.

Status: proposal, owned by Language, Compiler, and DX. This checkpoint implements a separate source artifact without selecting the pending canonical AST comparison stage, descriptor identity, or node identity contract.

Concrete implementation

source_trivia.proto (repository access required) defines a gnomon.spec.proposal.source.v1.SourceArtifact. Its SHA-256 and byte length bind it to the exact UTF-8 input. Comment ranges and declaration ranges are half-open UTF-8 byte offsets. Comments retain exact delimiters, Unicode spelling, and consumed LF or CRLF. Block ranges stop at */; line/doc ranges include their required newline. No line ending, Unicode, or comment text is normalized.

Declaration source entries record the authored production and source-order ordinal. An ordinal is local to this digest-bound parse, not a stable or canonical node ID. Explicit optional scalar presence distinguishes ordinal zero from a missing owner. The map currently covers top-level declarations and language comments; it does not promise token, expression, or complete whitespace coverage for a lossless formatter. Comment-looking text inside strings and opaque foreign scripts is excluded. Script header comments remain Gnomon comments.

The artifact comes from retained grammar production locations, not a second regular-expression lexer or fixture annotations. Syntax failure emits UNPARSED with only the exact source binding: no partial or guessed comment map. Grammar success emits RETAINED_PARSE even when the typed AST refuses an unsupported documentation owner. A refusal contributes no successful AST coverage.

ProposalParseResult.source_artifact is separate from parsed_file. AST fixture comparisons compare only the typed authored AST; source fixture comparisons compare the separate artifact. The artifact can also be emitted independently:

node tools/parse_proposed_syntax.mjs --source-artifact input.gnm
python3 tools/check_proposed_source.py
python3 -m unittest discover -s tools/tests -p 'test_proposed_source.py'

Six handwritten source expectations exercise Unicode and CRLF, comment/declaration order, leading documentation, opaque script exclusion, an ownership refusal, BOM syntax refusal, and empty input. Expected ranges and owner assignments were authored independently from the lowering; digests were computed independently with Python hashlib.

Documentation ownership established by current sources

RFC-0008 §8.1 says consecutive /// lines belong to the immediately following top-level declaration. RFC-0012 §4.3 places them directly above that declaration with no blank line before its keyword. The proposal implements their unambiguous intersection: consecutive leading /// lines, allowing indentation, directly above an undecorated, unmodified top-level declaration. One attachment records its declaration ordinal and the ordered comment ordinals. Empty doc lines, Unicode, and CRLF are preserved. No prose normalization or synthesized doc string is implied.

The artifact records this owner binding; compiled bundle doc metadata and its representation remain compiler work. CommonMark validation, M08760, and rejection of structured @tag semantics are not implemented by this syntax proposal. Recording exact authored text does not authorize those semantics.

Attachment evidence and remaining acceptance

CaseCurrent source evidenceCandidate behaviorRequired acceptance before promotion
Consecutive leading /// directly above bare declarationRFC-0008 §8.1 and RFC-0012 §4.3 agreeAttach in separate source artifactBoth independent adapters reproduce owner/order/text; compiler emits reviewed doc metadata
Blank line or ordinary comment between docs and declarationRFC-0012 requires direct adjacencyRefuse UNSUPPORTED_DOC_COMMENT; preserve exact source artifactOwner decides rejection or recovery and normative diagnostic
Docs before/between decoratorsRFC-0012 independently puts docs and decorators immediately above the declaration keywordRefuse; preserve docs and complete declaration rangeSpecify one ordering and attachment rule; fixtures for before/between/after decorator groups
Docs above or between modifiersRFC-0012 puts modifiers in the header, but does not define attachment across interrupted headersConservative refusal, including uninterrupted modified headersSpecify header anchor and allowed placement; verify modifier/order preservation
Body, inline, orphan, or trailing docsRFC-0008 §8.1 assigns docs to following top-level declarations, not member nodesRefuse rather than invent a member ownerDefine permitted owners/recovery with independent fixtures and normative diagnostics
/** ... */RFC-0014 calls it documentation; RFC-0008 lexical grammar and §12.1 classify it as an ordinary block commentPreserve as BLOCK, without a doc ownerReconcile RFCs and update lexical/attachment fixtures if this becomes a doc form
//compiled: and //compiled-by:RFC-0012 §4.3 requires verbatim preservation immediately after a rulePreserve exact ordinary line comments/ranges; no rule owner IDDefine typed rule source references and adjacency; formatter tests retain exact text immediately after its rule
CommonMark and forbidden @tag metadataRFC-0008 §§8.1/12.1 specify prose, M08760, and no tag syntaxPreserve text without interpretation or semantic acceptanceCompiler fixtures test content diagnostics, warning behavior, and reviewed doc normalization

Twenty source regressions test the exact bindings and explicit refusals, including malformed digests, missing ordinal presence, split Unicode ranges, altered text, invented declaration kinds, reordered ranges/owners, skipped or duplicate comment indices, and forged blank-line ownership. Location-free AST equality is tested separately from differing source artifacts. The fixtures do not exercise an independent compiler or compiled metadata emitter.

Remaining language conformance gates

The AST proposal now has 32 positive handwritten expectations and three refusals, retaining 175/180 productions. Five retained catalog productions are still unreachable from the generated start rule: WHITESPACE, query-shorthand, nat-lang-stmt, nat-lang-block, and qualified-set. Their earlier evidence and grammar-review requirements remain unchanged. No names are copied from fixture annotations to manufacture coverage.

Canonical comparison-stage and namespace approval, production/message review, documentation rules above, complete source mappings for formatter requirements, known authored grammar ambiguities, semantic diagnostics, and an independent compiler adapter remain open. The adapter must use the reviewed descriptor and same independently authored canonical expectations; routing both sides through this parser does not satisfy compiler conformance. grammar/fixtures/ still has no normative corpus, and check_fixture_envelopes.py --require-corpus remains red.