gnomonlearn

RFC-0016 · Schema Migration and Evolution

Versioned draft snapshot

RFCs, reference catalogs, and evidence reflect Spec 3b2e3e7a114fb826b1445d39a48756b8fff629cd. Counts and pending/status notes belong to that snapshot. Read the dated project update for later progress.

Source repositories are private during staging. Links marked “repository access required” need access to the linked repository.

Status Draft corpus. Reference catalogs and RDF projections have generation drift checks; Gnomon proofs and projection semantics remain unverified for release.

View the generated reference catalog →

Read design questions and decisions →

Summary

This RFC defines how Gnomon handles forward schema evolution of live production data. It introduces a first-class migration declaration kind, specifies the set of supported migration operations, establishes data safety rules and sequencing constraints, and extends the CLI with migration management commands. It preserves the bitemporal guarantees defined in RFC-0007 and integrates with the module versioning model defined in RFC-0014.

Motivation

RFC-0007 established the Gnomon storage model, including DDL fingerprints that detect schema drift. RFC-0014 introduced module versioning semantics. Neither RFC defines a mechanism for safely evolving the schema of live data once a module is deployed. Without such a mechanism, operators are left to write ad-hoc migration scripts that may violate bitemporal invariants, skip rollback paths, or bypass the approval gates defined in RFC-0005.

A first-class migration primitive closes this gap. It gives Gnomon the information needed to validate migrations statically, sequence them correctly, preserve historical queryability, and enforce human approval for destructive changes.

Scope

This RFC covers:

  • The migration declaration syntax and semantics
  • Supported migration operation kinds
  • Data safety rules and associated diagnostic codes
  • Migration sequencing and atomicity guarantees
  • Bitemporality preservation obligations
  • CLI commands for migration lifecycle management
  • Security and approval requirements

This RFC does not cover cross-module migrations (migrations that span more than one module namespace) or migration of data stored outside the Gnomon storage model.

Cross-References

  • RFC-0005 — Install plan and activation; the approval gate for destructive operations is defined in §4a of that RFC.
  • RFC-0006 §11 — Compatibility classification table; §10 of this RFC maps migration operations to those classes.
  • RFC-0007 — Storage model; defines the gnomon table structure, valid_time, system_time, and DDL fingerprints.
  • RFC-0008 §6.32 — transform declaration kind; user-defined pure data transforms referenced from core.migrations.ChangeType.
  • RFC-0014 — Module versioning; defines MAJOR, MINOR, and PATCH semantics and breaking-change classification.
  • core.migrations — Intrinsic schema operation library; canonical definitions are in RFC-0035.

1. The migration Declaration Kind

A migration is a named, version-scoped declaration that describes a schema change as a pair of operations: a forward block that applies the change, and a rollback block that undoes it. Migrations are defined in .gnm source files alongside the module they belong to.

Operations inside forward and rollback blocks are calls to core.migrations.* intrinsics — first-class Gnomon declarations (defined in RFC-0035) that the runtime implements. This design means migration operations are queryable through the Knowledge Index (RFC-0010 §9), statically analyzable by the compiler, and uniformly extensible without reserved DSL keywords.

migration AddHouseholdTier_v1_1 {
  meaning: "Add optional tier field to Household for premium tracking."
  from:    "com.example.benefits@1.0.x"
  to:      "com.example.benefits@1.1.0"

  forward {
    core.migrations.AddField {
      target:  Household
      field:   "tier"
      type:    String
      default: "standard"
    }
  }

  rollback {
    core.migrations.RemoveField {
      target: Household
      field:  "tier"
    }
  }
}

1.1 Grammar

migration-decl  ::= [ epistemic-modifier ] "migration" IDENT "{" migration-body "}"
migration-body  ::= meaning-slot
                    "from" ":" STRING NEWLINE
                    "to"   ":" STRING NEWLINE
                    forward-block
                    [ rollback-block ]
forward-block   ::= "forward"  "{" { migration-op } "}"
rollback-block  ::= "rollback" "{" { migration-op } "}"
migration-op    ::= QUALIFIED "{" { slot-assign } "}"
slot-assign     ::= IDENT ":" ( expr | IDENT ) NEWLINE

QUALIFIED in a migration-op MUST resolve to a core.migrations.* intrinsic (M01609) or a user-declared transform invocation (§2.7). No other declaration kinds are valid as migration operations.

1.1 Required Properties

PropertyTypeDescription
meaningstring literalHuman-readable rationale. Required. Surfaced in audit logs and gnomon migrate status.
fromversion rangeThe module version (or range using x wildcards) that this migration upgrades from. Must resolve to an installed version.
toexact versionThe target module version after this migration is applied. Must match a version declared in the module manifest (RFC-0014).
forwardoperation blockOne or more migration operations to apply. Required.
rollbackoperation blockOne or more migration operations to undo the forward block. Required for any migration that touches existing data (see §4).

1.2 Naming Convention

Migration names must be unique within a module namespace. The recommended convention is <Description>_v<major>_<minor> (e.g., AddHouseholdTier_v1_1). Names are used as stable identifiers in the migration DAG and in CLI output; they must not be renamed after a migration has been applied to any environment.


2. Migration Operation Kinds

All migration operations are core.migrations.* intrinsics. Each intrinsic is a named Gnomon declaration (RFC-0035) with a defined slot schema. The compiler validates slot types and required fields statically. The runtime implements each intrinsic natively.

2.1 core.migrations.AddField

Adds a new field to an existing entity.

core.migrations.AddField {
  target:  Household
  field:   "tier"
  type:    String
  default: "standard"
}
SlotRequiredDescription
targetYesThe entity to add the field to.
fieldYesThe new field name (string literal).
typeYesThe Gnomon type of the new field.
defaultConditionalRequired when live rows exist. Applied to all existing rows at migration time. valid_time of existing records is not altered.

If default is absent and live rows exist for the entity, the runtime raises M01601.

2.2 core.migrations.RemoveField

Removes a field from an existing entity.

core.migrations.RemoveField {
  target: Household
  field:  "tier"
  drop:   true
}
SlotRequiredDescription
targetYesThe entity to remove the field from.
fieldYesThe field name to remove.
dropConditionalRequired when live rows contain non-null values for the field. drop: true acknowledges permanent data discard.

If drop is absent and live rows contain non-null values, the runtime raises M01600. This operation is a MAJOR breaking change (RFC-0014) and requires human approval (RFC-0005 §4a).

2.3 core.migrations.RenameField

Renames a field. The old name is preserved as a deprecated alias until the next MAJOR version bump.

core.migrations.RenameField {
  target: Household
  from:   "tier"
  to:     "level"
}
SlotRequiredDescription
targetYesThe entity containing the field.
fromYesThe current field name.
toYesThe new field name.

Queries using the old name continue to resolve via the alias until the MAJOR bump removes it. The alias is surfaced in gnomon migrate status as (deprecated alias: tier).

2.4 core.migrations.ChangeType

Changes the storage type of a field. Requires a reference to a declared transform (RFC-0008 §6.32) that is total over the old type’s value domain.

// Declare the transform separately (RFC-0008 §6.32)
transform TierStringToEnum {
  meaning: "Coerce stored string tier values to the TierLevel enum."
  from: String
  to:   TierLevel
  apply {
    TierLevel.fromString(input) ?? TierLevel.STANDARD
  }
}

// Reference it in the migration
core.migrations.ChangeType {
  target:    Household
  field:     "tier"
  to:        TierLevel
  transform: TierStringToEnum
}
SlotRequiredDescription
targetYesThe entity containing the field.
fieldYesThe field whose type is changing.
toYesThe new Gnomon type.
transformYesA declared transform mapping old type → new type. MUST be total.
  • If transform is absent, the compiler raises M01602 at parse time.
  • If the referenced transform is not provably total over the old type’s value domain, the static checker raises M01603.
  • This operation is a MAJOR breaking change and requires human approval (RFC-0005 §4a).

The transform declaration (RFC-0008 §6.32) replaces the previous inline (v) => expr lambda syntax. Inline lambda expressions are no longer accepted in ChangeType slots.

2.5 core.migrations.AddEntity

Adds a new entity type to the module schema. MINOR change; no approval required.

core.migrations.AddEntity {
  target: TierAuditLog
  fields {
    householdId: String
    changedAt:   Timestamp
    newTier:     String
  }
}
SlotRequiredDescription
targetYesThe new entity declaration to add. Must be declared in the same module.
fieldsYesInline field block defining the new entity’s structure.

2.6 core.migrations.RemoveEntity

Removes an entity type from the module schema. MAJOR change; requires approval.

core.migrations.RemoveEntity {
  target: LegacyTierLog
  drop:   true
}
SlotRequiredDescription
targetYesThe entity to remove.
dropConditionalRequired when live instances exist. Acknowledges permanent data discard.

If live instances exist and drop is absent, the runtime raises M01604. Requires human approval (RFC-0005 §4a).

2.7 core.migrations.BackfillFrom

Populates a field in one entity by projecting values from another entity’s field. Useful when adding a field that can be derived from existing data rather than using a static default.

core.migrations.BackfillFrom {
  target:      Household
  field:       "regionCode"
  source:      Household.address.region
}
SlotRequiredDescription
targetYesThe entity and field to populate.
fieldYesThe field name to backfill.
sourceYesA field path expression on the same entity resolving to the backfill value.

BackfillFrom is non-destructive and is classified as a MINOR change. No approval required.

2.8 core.migrations.NoOp

Records a schema version bump with no structural change. Used when a MINOR or MAJOR version increment is required for documentation purposes but no data or schema mutation is needed.

core.migrations.NoOp {
  reason: "Version bump to align with RFC-0014 release tagging."
}

3. Data Safety Rules

The following rules are enforced at apply-time (and where statically determinable, at parse/compile time).

CodeConditionResolution
M01600core.migrations.RemoveField applied to an entity with live non-null data and drop: true is absent.Add drop: true explicitly, or drain the field before migrating.
M01601core.migrations.AddField with no default, applied to an entity with live rows.Provide a default value.
M01602core.migrations.ChangeType with no transform reference.Reference a declared transform (RFC-0008 §6.32).
M01603The transform referenced in core.migrations.ChangeType is not provably total over the old type’s value domain.Extend the transform to handle all possible values.
M01604core.migrations.RemoveEntity applied to an entity with live instances and drop: true is absent.Ensure zero live instances or set drop: true.
M01605Migration graph contains a cycle.Remove the cyclic dependency so the migration DAG is acyclic.
M01606A migration that touches existing data has no rollback block.Define a rollback block for every forward migration that modifies existing data.
M01607The from version range in a migration does not resolve to any installed module version.Correct the from range or ensure the prerequisite module version is installed.
M01609A migration operation references a QUALIFIED name that is not a core.migrations.* intrinsic or a user-declared transform.Use a valid intrinsic or declare a transform.

4. Migration Sequencing and Atomicity

4.1 Migration DAG

All migrations within a module form a directed acyclic graph (DAG). Each migration’s from version range establishes an edge from the source version(s) to the migration node, and the to version establishes an edge from the migration node to the target version. Cycles in this graph are a static error (M01605) and prevent module activation.

The install plan (RFC-0005) executes migrations in topological order. When multiple migrations are eligible at the same step (i.e., no ordering dependency between them), the runtime applies them in lexicographic order by migration name to ensure determinism across environments.

4.2 Atomicity

Each migration is atomic. The runtime wraps each migration’s forward (or rollback) block in a transaction. If any operation within the block fails, the entire migration is rolled back and the schema version is not advanced. Partial schema states are not observable by application queries.

4.3 Rollback Obligation

A rollback block is required for every migration whose forward block touches existing data (i.e., mutates rows, drops columns, or alters types). Migrations that only add new entities with no existing rows may omit rollback, but defining one is recommended. Absence of a required rollback block is a static error (M01606).

If an addEntity migration has been applied and rows have subsequently been inserted into the new entity’s table, rolling back the migration MUST archive (not delete) those rows using the system_time bitemporality column before dropping the table. The rollback block MUST include a preserveRows directive if any rows may exist at rollback time.


5. Bitemporality Preservation

Gnomon’s storage model (RFC-0007) records every fact with both valid_time (when the fact was true in the world) and system_time (when Gnomon learned about it). Migrations MUST NOT alter either dimension of existing records.

Specifically:

  • valid_time is immutable. No migration operation may rewrite the valid_time of a row that existed before the migration.
  • system_time is append-only. Structural changes (adding fields, changing types) produce new system_time entries for affected rows. They do not delete or overwrite prior system_time entries.
  • Historical queryability. After a migration, asOf(t) queries where t predates the migration continue to return data shaped as it was at time t. The runtime resolves the schema version in effect at t using the DDL fingerprints stored per RFC-0007, and projects rows accordingly.
  • removeField and removeEntity are exceptions. Data that has been explicitly dropped via migrate: drop or force: true is permanently removed and will not be returned by asOf() queries. This is the expected and documented consequence of those operations; the required human approval (RFC-0005 §4a) serves as the acknowledgment gate.

6. CLI Commands

6.1 gnomon migrate status

Displays the migration state for the current environment.

$ gnomon migrate status

Module: com.example.benefits
Environment: production

  NAME                        FROM         TO       STATUS    APPLIED AT
  AddHouseholdTier_v1_1       1.0.x        1.1.0    applied   2026-03-14T09:22:11Z
  AddTierAuditLog_v1_1        1.0.x        1.1.0    applied   2026-03-14T09:22:11Z
  DropLegacyTierLog_v1_2      1.1.x        1.2.0    pending   —
  RenameTierToLevel_v1_3      1.2.x        1.3.0    pending   —

Status values: applied, pending, failed, skipped.

A migration is failed if its transaction was attempted but did not commit. Failed migrations block subsequent pending migrations until resolved.

6.2 gnomon migrate apply

Applies all pending migrations in topological order.

$ gnomon migrate apply

Flags:

  • --dry-run — prints the operations that would be executed without applying them.
  • --module <id> — limits application to a specific module.
  • --to <name> — applies only up to (and including) the named migration.

Destructive migrations (those classified as MAJOR breaking changes per §2 and RFC-0014) require interactive confirmation unless --yes is passed. In CI environments without a TTY, --yes must be passed explicitly; omitting it causes the command to exit with a non-zero status and a message directing the operator to the approval workflow defined in RFC-0005 §4a.

6.3 gnomon migrate rollback <name>

Rolls back the named migration by executing its rollback block.

$ gnomon migrate rollback AddHouseholdTier_v1_1
  • The named migration must be in applied status.
  • If other applied migrations depend on the target migration (i.e., their from range is satisfied only by a version introduced by this migration), the command fails with a dependency error. Pass --cascade to roll back dependent migrations first, in reverse topological order.
  • Emits an audit log entry including the operator identity, timestamp, and migration name.

7. Security and Approval Requirements

Migrations that fall into any of the following categories are MAJOR breaking changes per RFC-0014 and require the same human approval workflow as activation operations defined in RFC-0005 §4a:

  • removeField (regardless of force or migrate: drop)
  • changeType
  • removeEntity

The approval record is stored in the Gnomon audit log with a reference to the migration name, the approving identity, and the RFC-0005 activation ID that authorized the operation. Migrations applied without a required approval record raise M01608 and are blocked.

Non-destructive migrations (addField, addEntity, renameField) do not require human approval and may be applied by CI pipelines with appropriate deployment credentials.


8. Invariants

The following invariants are enforced by the Gnomon runtime and static checker. Violation of any invariant blocks migration application.

  1. Acyclic migration graph. The migration DAG for a module must contain no cycles (M01605).
  2. Rollback coverage. Every migration that touches existing data must define a rollback block (M01606).
  3. Total transforms. Every transform referenced by core.migrations.ChangeType must be provably total over the old type’s value domain (M01603).
  4. Resolved from ranges. Every migration’s from version range must resolve to at least one installed version in the target environment (M01607).
  5. Bitemporality. No migration may alter valid_time or system_time of rows that predate the migration.
  6. Approval gate. MAJOR breaking migrations must carry an approval record before they are applied (M01608).
  7. Name stability. Migration names are immutable after first application. Renaming an applied migration is a static error.

9. Diagnostic Code Reference

All codes in this RFC fall in the M01600–M01699 range (M016xx).

CodeShort nameSeverityPhase
M01600REMOVE_FIELD_LIVE_DATAerrorapply-time
M01601ADD_FIELD_NO_DEFAULTerrorapply-time
M01602CHANGE_TYPE_NO_TRANSFORMerrorparse-time
M01603CHANGE_TYPE_NON_TOTALerrorstatic-check
M01604REMOVE_ENTITY_LIVE_DATAerrorapply-time
M01605MIGRATION_CYCLEerrorstatic-check
M01606MISSING_ROLLBACKerrorstatic-check
M01607UNRESOLVED_FROM_RANGEerrorapply-time
M01608MISSING_APPROVAL_RECORDerrorinstall-validation
M01609INVALID_MIGRATION_OPerrorparse-time

10. Backward-Compatibility Classification

This table maps each core.migrations.* intrinsic to its RFC-0014 version-bump obligation and its RFC-0006 §11 compatibility class. The compiler enforces that the migration’s declared from → to version bump is at least as large as the required bump column.

IntrinsicRFC-0014 BumpRFC-0006 §11 ClassApproval RequiredRollback Required
core.migrations.AddField (with default)MINORNon-breakingNoNo¹
core.migrations.AddField (no default, zero rows)MINORNon-breakingNoNo¹
core.migrations.RemoveFieldMAJORBreaking (field removal)YesYes
core.migrations.RenameFieldMINOR²DeprecatingNoYes
core.migrations.ChangeTypeMAJORBreaking (type change)YesYes
core.migrations.AddEntityMINORNon-breakingNoNo¹
core.migrations.RemoveEntityMAJORBreaking (entity removal)YesYes
core.migrations.BackfillFromMINORNon-breakingNoNo¹
core.migrations.NoOpAnyNon-breakingNoNo

¹ A rollback block is not required but is recommended. If rows have been inserted into an AddEntity-created table, the rollback must include core.migrations.RemoveEntity with drop: true.

² RenameField is MINOR because the old name remains as a deprecated alias until the next MAJOR bump. The MAJOR bump that removes the alias is what triggers the breaking-change classification and approval requirement.

10.1 Relationship to RFC-0006 §11 Compatibility Table

RFC-0006 §11 defines compatibility classes for type-model changes (field addition, field removal, type widening, type narrowing, etc.) from the perspective of schema evolution semantics. The table above maps those classes onto the operational level: given an intended schema evolution, which core.migrations.* intrinsic expresses it, and what obligations does the compiler enforce?

When a migration declaration’s from → to bump is smaller than the required bump for its operations, the compiler MUST produce error M01610 identifying the underdeclared version gap.

CodeShort nameSeverityPhase
M01610VERSION_BUMP_TOO_SMALLerrorstatic-check