Migration

Migrate a flat (pre-four-layer) SOMA vault to the four-layer knowledge model.

What Migration Does

Before four layers, SOMA stored all entities in a flat vault — no layer field, no source_worker field. The migration command scans every entity and adds two fields:

Field	Value	Why
layer	archive (L1)	Safe default — L1 never expires, has no write restrictions, and does not influence agent behavior until explicitly promoted
source_worker	migration	Audit trail — you can always identify which entities came from the migration vs. which were created by workers

The migration is additive only. It adds fields to entities that lack them. It never modifies existing fields, never deletes entities, never changes entity types or statuses.

Why L1?

All migrated entities land in L1 (Archive) because:

1. L1 never decays. Migrated data will not silently expire on you.
2. L1 has no semantic weight enforcement. It does not influence agent behavior through the Policy Bridge unless explicitly queried with --intent route.
3. L1 is the starting point for the promotion pipeline. If any migrated entity contains valuable knowledge, the Synthesizer can detect it and propose it to L3. From there, a human reviewer promotes it to L4 canon.
4. L2 requires team_id and decay_at. Migrated entities have neither.
5. L3 requires confidence and evidence_links. Migrated entities have neither.
6. L4 requires ratified_by and origin_l3_id. Nobody has reviewed these.

Putting everything in L1 is the only choice that preserves data without making false claims about confidence or authority.

Run the Migration

npx soma migrate-layers

Expected output:

Migration: scanning vault at .soma/vault
  Found 142 entities
  Migrated 142 entities to L1 (archive)
    Added layer: 'archive'
    Added source_worker: 'migration'
  Skipped 0 (already have layer field)
Migration complete.

To migrate a vault at a non-default path:

npx soma migrate-layers --vault /path/to/custom/.soma/vault

Expected output:

Migration: scanning vault at /path/to/custom/.soma/vault
  Found 58 entities
  Migrated 58 entities to L1 (archive)
    Added layer: 'archive'
    Added source_worker: 'migration'
  Skipped 0 (already have layer field)
Migration complete.

Verify with Layer Status

After migration, confirm everything landed in L1:

npx soma layers status

Expected output:

Layer Status
────────────────────────────
  L4 Canon:              0
  L3 Emerging:           0
  L2 Working Memory:     0
  L1 Archive:          142
────────────────────────────
  Total:               142

All 142 entities are in L1. No entities in L2, L3, or L4 — that is correct. Knowledge must be earned through the pipeline (Synthesizer → Governance → Canon), not assumed.

Idempotency

The migration is idempotent. Running it again on an already-migrated vault changes nothing:

npx soma migrate-layers

Expected output:

Migration: scanning vault at .soma/vault
  Found 142 entities
  Migrated 0 entities to L1 (archive)
  Skipped 142 (already have layer field)
Migration complete.

Zero entities migrated on the second run. The migration checks for the presence of the layer field — if it exists (regardless of value), the entity is skipped. This means you can safely run migration as part of an upgrade script or CI pipeline without worrying about double-processing.

What About Existing Insights?

If your flat vault already contains insight, policy, or synthesis entities from the pre-four-layer Synthesizer, they start in L1 after migration. This is intentional:

1. They were generated without confidence scoring. The old Synthesizer did not produce confidence scores or evidence_links. They are not eligible for L3.
2. They were never governance-reviewed. Putting them in L4 would bypass the governance gate, which defeats the purpose of four layers.
3. The new Synthesizer will re-process them. After migration, run soma run with an LLM provider. The Synthesizer scans L1 candidates, detects patterns, and creates proper L3 proposals with confidence scores and evidence chains. The best of the old insights will surface again — this time with proper backing.

The workflow after migration:

# 1. Migrate flat vault to four layers
npx soma migrate-layers

# 2. Run the full pipeline — Synthesizer re-analyzes L1
npx soma run --provider openrouter --model anthropic/claude-sonnet-4

# 3. Check what the Synthesizer proposed
npx soma governance list

# 4. Review and promote the good ones
npx soma governance promote --id <proposal-id>

What the Migration Does NOT Do

• Does not create L2/L3/L4 entries. Everything goes to L1.
• Does not modify type, status, name, or any other existing field. Only layer and source_worker are added.
• Does not re-index the vault. The index is updated incrementally as each entity is written.
• Does not require an LLM. Migration is a pure metadata operation — no API calls.
• Does not touch the mutation log. Standard vault writes are logged to _mutations.jsonl as usual.

Troubleshooting

"Found 0 entities" — Check the --vault path. The default is .soma/vault. The migration scans for .md files in type subdirectories (agent/, execution/, insight/, etc.).

Entity has wrong layer after migration — This should not happen. The migration only adds layer: 'archive' to entities without a layer field. If an entity already has a layer, it is skipped. If you see an entity with an unexpected layer, it was set by a worker after migration, not by the migration itself.

Vault lock error — Another SOMA process is running. Wait for it to finish or check for stale locks. SOMA uses PID-based stale lock detection — if the lock holder process is no longer running, the lock is automatically removed on the next operation.