RFC-0016 · Schema Migration and Evolution
Versioned draft snapshot
RFCs, reference catalogs, and evidence reflect Spec 3b2e3e7a114fb826b1445d39a48756b8fff629cd. Counts and pending/status notes belong to that snapshot. Read the dated project update for later progress.
Source repositories are private during staging. Links marked “repository access required” need access to the linked repository.
| Status | Draft corpus. Reference catalogs and RDF projections have generation drift checks; Gnomon proofs and projection semantics remain unverified for release. |
|---|
View the generated reference catalog →
Read design questions and decisions →
Summary
This RFC defines how Gnomon handles forward schema evolution of live production data. It introduces a first-class migration declaration kind, specifies the set of supported migration operations, establishes data safety rules and sequencing constraints, and extends the CLI with migration management commands. It preserves the bitemporal guarantees defined in RFC-0007 and integrates with the module versioning model defined in RFC-0014.
Motivation
RFC-0007 established the Gnomon storage model, including DDL fingerprints that detect schema drift. RFC-0014 introduced module versioning semantics. Neither RFC defines a mechanism for safely evolving the schema of live data once a module is deployed. Without such a mechanism, operators are left to write ad-hoc migration scripts that may violate bitemporal invariants, skip rollback paths, or bypass the approval gates defined in RFC-0005.
A first-class migration primitive closes this gap. It gives Gnomon the information needed to validate migrations statically, sequence them correctly, preserve historical queryability, and enforce human approval for destructive changes.
Scope
This RFC covers:
- The
migrationdeclaration syntax and semantics - Supported migration operation kinds
- Data safety rules and associated diagnostic codes
- Migration sequencing and atomicity guarantees
- Bitemporality preservation obligations
- CLI commands for migration lifecycle management
- Security and approval requirements
This RFC does not cover cross-module migrations (migrations that span more than one module namespace) or migration of data stored outside the Gnomon storage model.
Cross-References
- RFC-0005 — Install plan and activation; the approval gate for destructive operations is defined in §4a of that RFC.
- RFC-0006 §11 — Compatibility classification table; §10 of this RFC maps migration operations to those classes.
- RFC-0007 — Storage model; defines the
gnomontable structure,valid_time,system_time, and DDL fingerprints. - RFC-0008 §6.32 —
transformdeclaration kind; user-defined pure data transforms referenced fromcore.migrations.ChangeType. - RFC-0014 — Module versioning; defines MAJOR, MINOR, and PATCH semantics and breaking-change classification.
- core.migrations — Intrinsic schema operation library; canonical definitions are in RFC-0035.
1. The migration Declaration Kind
A migration is a named, version-scoped declaration that describes a schema change as a pair of operations: a forward block that applies the change, and a rollback block that undoes it. Migrations are defined in .gnm source files alongside the module they belong to.
Operations inside forward and rollback blocks are calls to core.migrations.* intrinsics — first-class Gnomon declarations (defined in RFC-0035) that the runtime implements. This design means migration operations are queryable through the Knowledge Index (RFC-0010 §9), statically analyzable by the compiler, and uniformly extensible without reserved DSL keywords.
migration AddHouseholdTier_v1_1 {
meaning: "Add optional tier field to Household for premium tracking."
from: "com.example.benefits@1.0.x"
to: "com.example.benefits@1.1.0"
forward {
core.migrations.AddField {
target: Household
field: "tier"
type: String
default: "standard"
}
}
rollback {
core.migrations.RemoveField {
target: Household
field: "tier"
}
}
}
1.1 Grammar
migration-decl ::= [ epistemic-modifier ] "migration" IDENT "{" migration-body "}"
migration-body ::= meaning-slot
"from" ":" STRING NEWLINE
"to" ":" STRING NEWLINE
forward-block
[ rollback-block ]
forward-block ::= "forward" "{" { migration-op } "}"
rollback-block ::= "rollback" "{" { migration-op } "}"
migration-op ::= QUALIFIED "{" { slot-assign } "}"
slot-assign ::= IDENT ":" ( expr | IDENT ) NEWLINE
QUALIFIED in a migration-op MUST resolve to a core.migrations.* intrinsic (M01609) or a user-declared transform invocation (§2.7). No other declaration kinds are valid as migration operations.
1.1 Required Properties
| Property | Type | Description |
|---|---|---|
meaning | string literal | Human-readable rationale. Required. Surfaced in audit logs and gnomon migrate status. |
from | version range | The module version (or range using x wildcards) that this migration upgrades from. Must resolve to an installed version. |
to | exact version | The target module version after this migration is applied. Must match a version declared in the module manifest (RFC-0014). |
forward | operation block | One or more migration operations to apply. Required. |
rollback | operation block | One or more migration operations to undo the forward block. Required for any migration that touches existing data (see §4). |
1.2 Naming Convention
Migration names must be unique within a module namespace. The recommended convention is <Description>_v<major>_<minor> (e.g., AddHouseholdTier_v1_1). Names are used as stable identifiers in the migration DAG and in CLI output; they must not be renamed after a migration has been applied to any environment.
2. Migration Operation Kinds
All migration operations are core.migrations.* intrinsics. Each intrinsic is a named Gnomon declaration (RFC-0035) with a defined slot schema. The compiler validates slot types and required fields statically. The runtime implements each intrinsic natively.
2.1 core.migrations.AddField
Adds a new field to an existing entity.
core.migrations.AddField {
target: Household
field: "tier"
type: String
default: "standard"
}
| Slot | Required | Description |
|---|---|---|
target | Yes | The entity to add the field to. |
field | Yes | The new field name (string literal). |
type | Yes | The Gnomon type of the new field. |
default | Conditional | Required when live rows exist. Applied to all existing rows at migration time. valid_time of existing records is not altered. |
If default is absent and live rows exist for the entity, the runtime raises M01601.
2.2 core.migrations.RemoveField
Removes a field from an existing entity.
core.migrations.RemoveField {
target: Household
field: "tier"
drop: true
}
| Slot | Required | Description |
|---|---|---|
target | Yes | The entity to remove the field from. |
field | Yes | The field name to remove. |
drop | Conditional | Required when live rows contain non-null values for the field. drop: true acknowledges permanent data discard. |
If drop is absent and live rows contain non-null values, the runtime raises M01600. This operation is a MAJOR breaking change (RFC-0014) and requires human approval (RFC-0005 §4a).
2.3 core.migrations.RenameField
Renames a field. The old name is preserved as a deprecated alias until the next MAJOR version bump.
core.migrations.RenameField {
target: Household
from: "tier"
to: "level"
}
| Slot | Required | Description |
|---|---|---|
target | Yes | The entity containing the field. |
from | Yes | The current field name. |
to | Yes | The new field name. |
Queries using the old name continue to resolve via the alias until the MAJOR bump removes it. The alias is surfaced in gnomon migrate status as (deprecated alias: tier).
2.4 core.migrations.ChangeType
Changes the storage type of a field. Requires a reference to a declared transform (RFC-0008 §6.32) that is total over the old type’s value domain.
// Declare the transform separately (RFC-0008 §6.32)
transform TierStringToEnum {
meaning: "Coerce stored string tier values to the TierLevel enum."
from: String
to: TierLevel
apply {
TierLevel.fromString(input) ?? TierLevel.STANDARD
}
}
// Reference it in the migration
core.migrations.ChangeType {
target: Household
field: "tier"
to: TierLevel
transform: TierStringToEnum
}
| Slot | Required | Description |
|---|---|---|
target | Yes | The entity containing the field. |
field | Yes | The field whose type is changing. |
to | Yes | The new Gnomon type. |
transform | Yes | A declared transform mapping old type → new type. MUST be total. |
- If
transformis absent, the compiler raises M01602 at parse time. - If the referenced
transformis not provably total over the old type’s value domain, the static checker raises M01603. - This operation is a MAJOR breaking change and requires human approval (RFC-0005 §4a).
The transform declaration (RFC-0008 §6.32) replaces the previous inline (v) => expr lambda syntax. Inline lambda expressions are no longer accepted in ChangeType slots.
2.5 core.migrations.AddEntity
Adds a new entity type to the module schema. MINOR change; no approval required.
core.migrations.AddEntity {
target: TierAuditLog
fields {
householdId: String
changedAt: Timestamp
newTier: String
}
}
| Slot | Required | Description |
|---|---|---|
target | Yes | The new entity declaration to add. Must be declared in the same module. |
fields | Yes | Inline field block defining the new entity’s structure. |
2.6 core.migrations.RemoveEntity
Removes an entity type from the module schema. MAJOR change; requires approval.
core.migrations.RemoveEntity {
target: LegacyTierLog
drop: true
}
| Slot | Required | Description |
|---|---|---|
target | Yes | The entity to remove. |
drop | Conditional | Required when live instances exist. Acknowledges permanent data discard. |
If live instances exist and drop is absent, the runtime raises M01604. Requires human approval (RFC-0005 §4a).
2.7 core.migrations.BackfillFrom
Populates a field in one entity by projecting values from another entity’s field. Useful when adding a field that can be derived from existing data rather than using a static default.
core.migrations.BackfillFrom {
target: Household
field: "regionCode"
source: Household.address.region
}
| Slot | Required | Description |
|---|---|---|
target | Yes | The entity and field to populate. |
field | Yes | The field name to backfill. |
source | Yes | A field path expression on the same entity resolving to the backfill value. |
BackfillFrom is non-destructive and is classified as a MINOR change. No approval required.
2.8 core.migrations.NoOp
Records a schema version bump with no structural change. Used when a MINOR or MAJOR version increment is required for documentation purposes but no data or schema mutation is needed.
core.migrations.NoOp {
reason: "Version bump to align with RFC-0014 release tagging."
}
3. Data Safety Rules
The following rules are enforced at apply-time (and where statically determinable, at parse/compile time).
| Code | Condition | Resolution |
|---|---|---|
| M01600 | core.migrations.RemoveField applied to an entity with live non-null data and drop: true is absent. | Add drop: true explicitly, or drain the field before migrating. |
| M01601 | core.migrations.AddField with no default, applied to an entity with live rows. | Provide a default value. |
| M01602 | core.migrations.ChangeType with no transform reference. | Reference a declared transform (RFC-0008 §6.32). |
| M01603 | The transform referenced in core.migrations.ChangeType is not provably total over the old type’s value domain. | Extend the transform to handle all possible values. |
| M01604 | core.migrations.RemoveEntity applied to an entity with live instances and drop: true is absent. | Ensure zero live instances or set drop: true. |
| M01605 | Migration graph contains a cycle. | Remove the cyclic dependency so the migration DAG is acyclic. |
| M01606 | A migration that touches existing data has no rollback block. | Define a rollback block for every forward migration that modifies existing data. |
| M01607 | The from version range in a migration does not resolve to any installed module version. | Correct the from range or ensure the prerequisite module version is installed. |
| M01609 | A migration operation references a QUALIFIED name that is not a core.migrations.* intrinsic or a user-declared transform. | Use a valid intrinsic or declare a transform. |
4. Migration Sequencing and Atomicity
4.1 Migration DAG
All migrations within a module form a directed acyclic graph (DAG). Each migration’s from version range establishes an edge from the source version(s) to the migration node, and the to version establishes an edge from the migration node to the target version. Cycles in this graph are a static error (M01605) and prevent module activation.
The install plan (RFC-0005) executes migrations in topological order. When multiple migrations are eligible at the same step (i.e., no ordering dependency between them), the runtime applies them in lexicographic order by migration name to ensure determinism across environments.
4.2 Atomicity
Each migration is atomic. The runtime wraps each migration’s forward (or rollback) block in a transaction. If any operation within the block fails, the entire migration is rolled back and the schema version is not advanced. Partial schema states are not observable by application queries.
4.3 Rollback Obligation
A rollback block is required for every migration whose forward block touches existing data (i.e., mutates rows, drops columns, or alters types). Migrations that only add new entities with no existing rows may omit rollback, but defining one is recommended. Absence of a required rollback block is a static error (M01606).
If an addEntity migration has been applied and rows have subsequently been inserted into the new entity’s table, rolling back the migration MUST archive (not delete) those rows using the system_time bitemporality column before dropping the table. The rollback block MUST include a preserveRows directive if any rows may exist at rollback time.
5. Bitemporality Preservation
Gnomon’s storage model (RFC-0007) records every fact with both valid_time (when the fact was true in the world) and system_time (when Gnomon learned about it). Migrations MUST NOT alter either dimension of existing records.
Specifically:
valid_timeis immutable. No migration operation may rewrite thevalid_timeof a row that existed before the migration.system_timeis append-only. Structural changes (adding fields, changing types) produce newsystem_timeentries for affected rows. They do not delete or overwrite priorsystem_timeentries.- Historical queryability. After a migration,
asOf(t)queries wheretpredates the migration continue to return data shaped as it was at timet. The runtime resolves the schema version in effect attusing the DDL fingerprints stored per RFC-0007, and projects rows accordingly. removeFieldandremoveEntityare exceptions. Data that has been explicitly dropped viamigrate: droporforce: trueis permanently removed and will not be returned byasOf()queries. This is the expected and documented consequence of those operations; the required human approval (RFC-0005 §4a) serves as the acknowledgment gate.
6. CLI Commands
6.1 gnomon migrate status
Displays the migration state for the current environment.
$ gnomon migrate status
Module: com.example.benefits
Environment: production
NAME FROM TO STATUS APPLIED AT
AddHouseholdTier_v1_1 1.0.x 1.1.0 applied 2026-03-14T09:22:11Z
AddTierAuditLog_v1_1 1.0.x 1.1.0 applied 2026-03-14T09:22:11Z
DropLegacyTierLog_v1_2 1.1.x 1.2.0 pending —
RenameTierToLevel_v1_3 1.2.x 1.3.0 pending —
Status values: applied, pending, failed, skipped.
A migration is failed if its transaction was attempted but did not commit. Failed migrations block subsequent pending migrations until resolved.
6.2 gnomon migrate apply
Applies all pending migrations in topological order.
$ gnomon migrate apply
Flags:
--dry-run— prints the operations that would be executed without applying them.--module <id>— limits application to a specific module.--to <name>— applies only up to (and including) the named migration.
Destructive migrations (those classified as MAJOR breaking changes per §2 and RFC-0014) require interactive confirmation unless --yes is passed. In CI environments without a TTY, --yes must be passed explicitly; omitting it causes the command to exit with a non-zero status and a message directing the operator to the approval workflow defined in RFC-0005 §4a.
6.3 gnomon migrate rollback <name>
Rolls back the named migration by executing its rollback block.
$ gnomon migrate rollback AddHouseholdTier_v1_1
- The named migration must be in
appliedstatus. - If other applied migrations depend on the target migration (i.e., their
fromrange is satisfied only by a version introduced by this migration), the command fails with a dependency error. Pass--cascadeto roll back dependent migrations first, in reverse topological order. - Emits an audit log entry including the operator identity, timestamp, and migration name.
7. Security and Approval Requirements
Migrations that fall into any of the following categories are MAJOR breaking changes per RFC-0014 and require the same human approval workflow as activation operations defined in RFC-0005 §4a:
removeField(regardless offorceormigrate: drop)changeTyperemoveEntity
The approval record is stored in the Gnomon audit log with a reference to the migration name, the approving identity, and the RFC-0005 activation ID that authorized the operation. Migrations applied without a required approval record raise M01608 and are blocked.
Non-destructive migrations (addField, addEntity, renameField) do not require human approval and may be applied by CI pipelines with appropriate deployment credentials.
8. Invariants
The following invariants are enforced by the Gnomon runtime and static checker. Violation of any invariant blocks migration application.
- Acyclic migration graph. The migration DAG for a module must contain no cycles (M01605).
- Rollback coverage. Every migration that touches existing data must define a
rollbackblock (M01606). - Total transforms. Every
transformreferenced bycore.migrations.ChangeTypemust be provably total over the old type’s value domain (M01603). - Resolved
fromranges. Every migration’sfromversion range must resolve to at least one installed version in the target environment (M01607). - Bitemporality. No migration may alter
valid_timeorsystem_timeof rows that predate the migration. - Approval gate. MAJOR breaking migrations must carry an approval record before they are applied (M01608).
- Name stability. Migration names are immutable after first application. Renaming an applied migration is a static error.
9. Diagnostic Code Reference
All codes in this RFC fall in the M01600–M01699 range (M016xx).
| Code | Short name | Severity | Phase |
|---|---|---|---|
| M01600 | REMOVE_FIELD_LIVE_DATA | error | apply-time |
| M01601 | ADD_FIELD_NO_DEFAULT | error | apply-time |
| M01602 | CHANGE_TYPE_NO_TRANSFORM | error | parse-time |
| M01603 | CHANGE_TYPE_NON_TOTAL | error | static-check |
| M01604 | REMOVE_ENTITY_LIVE_DATA | error | apply-time |
| M01605 | MIGRATION_CYCLE | error | static-check |
| M01606 | MISSING_ROLLBACK | error | static-check |
| M01607 | UNRESOLVED_FROM_RANGE | error | apply-time |
| M01608 | MISSING_APPROVAL_RECORD | error | install-validation |
| M01609 | INVALID_MIGRATION_OP | error | parse-time |
10. Backward-Compatibility Classification
This table maps each core.migrations.* intrinsic to its RFC-0014 version-bump obligation and its RFC-0006 §11 compatibility class. The compiler enforces that the migration’s declared from → to version bump is at least as large as the required bump column.
| Intrinsic | RFC-0014 Bump | RFC-0006 §11 Class | Approval Required | Rollback Required |
|---|---|---|---|---|
core.migrations.AddField (with default) | MINOR | Non-breaking | No | No¹ |
core.migrations.AddField (no default, zero rows) | MINOR | Non-breaking | No | No¹ |
core.migrations.RemoveField | MAJOR | Breaking (field removal) | Yes | Yes |
core.migrations.RenameField | MINOR² | Deprecating | No | Yes |
core.migrations.ChangeType | MAJOR | Breaking (type change) | Yes | Yes |
core.migrations.AddEntity | MINOR | Non-breaking | No | No¹ |
core.migrations.RemoveEntity | MAJOR | Breaking (entity removal) | Yes | Yes |
core.migrations.BackfillFrom | MINOR | Non-breaking | No | No¹ |
core.migrations.NoOp | Any | Non-breaking | No | No |
¹ A rollback block is not required but is recommended. If rows have been inserted into an AddEntity-created table, the rollback must include core.migrations.RemoveEntity with drop: true.
² RenameField is MINOR because the old name remains as a deprecated alias until the next MAJOR bump. The MAJOR bump that removes the alias is what triggers the breaking-change classification and approval requirement.
10.1 Relationship to RFC-0006 §11 Compatibility Table
RFC-0006 §11 defines compatibility classes for type-model changes (field addition, field removal, type widening, type narrowing, etc.) from the perspective of schema evolution semantics. The table above maps those classes onto the operational level: given an intended schema evolution, which core.migrations.* intrinsic expresses it, and what obligations does the compiler enforce?
When a migration declaration’s from → to bump is smaller than the required bump for its operations, the compiler MUST produce error M01610 identifying the underdeclared version gap.
| Code | Short name | Severity | Phase |
|---|---|---|---|
| M01610 | VERSION_BUMP_TOO_SMALL | error | static-check |