Skip to content

Validate native memory compression in a trained causal language model - #19

Merged
initial-d merged 3 commits into
mainfrom
codex/causal-lm-compression-20261024
Oct 24, 2026
Merged

initial-d merged 3 commits into
mainfrom
codex/causal-lm-compression-20261024

Conversation

@initial-d

Copy link
Copy Markdown
Owner

Row similarity and SGNS objectives leave full predictive distributions untested. Add a train-only, causal character n-gram + GRU language-model reference with full softmax, and put the native C embedding adapter in its actual inference path.

Exact suffix keys end at the current input character, before the next-character target. Vocabulary/key counts fit training text only; missing keys map to zero memory and unseen characters to an explicit unknown class. Evaluation streams recurrent state across chunks and scores every target after a common burn-in. Unit tests cover future perturbations, split boundaries, unknown targets, streaming state and scalar probability/KL oracles.

The benchmark trains maximum orders 4/8/12 with seeds 3/7/11, then compares lossless storage and direct cosine+norm approximate variants without retraining. Validation selects under fixed PPL/KL/top-1 budgets. A separate audit reports held-out budget exceedances without changing that choice; paired character-block intervals preserve local dependence.

Measured on a fixed 131072-character TinyShakespeare prefix, 64 dimensions and 1200x16x32 training targets per model:

  • Lossless native storage preserves validation/test logits and cross-entropy bit for bit in all nine models.
  • Mean selected native memory ratios: 1.107x / 1.385x / 1.489x; counting unchanged parameters and serialized registry: 1.099x / 1.359x / 1.453x.
  • Selected test PPL changes: -0.02696% to +0.04183%; top-1 changes at most 0.18401%.
  • Three models exceed the held-out mean-KL budget (order/seed 4/7, 8/3, 12/11). These remain explicit negative generalization evidence.
  • Disabling the trained model's memory at inference raises mean test character PPL from 5.005-5.741 to 13.578-14.312. This ablation confirms dependence on the table and is not a separately trained baseline.

Raw reports, hashes, exact commands, environment, audit and figure are in docs/causal_lm_evidence_20261024/. Tests: 52 passed; Ruff passed. Text, lookup keys and checkpoints remain local.

Scope: small character-GRU reference, finite training and one corpus. This does not establish trained Engram/Transformer quality, convergence, framework-level memory savings or universal threshold safety. Higher order compresses better here but also has worse original PPL under the fixed training budget. Relaxing cosine adds little compression while increasing validation KL.

@initial-d
initial-d merged commit ed52f93 into main Oct 24, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant