When Training Order Becomes the Algorithm: Endpoint-Induced Collapse and Memorization Without Rule Generalization

Author abstract

Abstract

Synthetic execution tasks are often used to test whether language models acquire algorithmic rules, yet aggregate accuracy can hide shortcuts induced by data order. EpiLang studies a register-language task in which a Qwen3.5-0.8B model must reduce an integer literal modulo 11, write the residue into a destination register, preserve the remaining state, and emit the same residue. Train and tune splits use disjoint raw literals. In an initial deterministic run, homogeneous residue blocks caused the model to generate residue 10 for every input. Reversing the blocks reproduced collapse to residue 0, while interleaving residues removed the single-class collapse and improved train fitting. Held-out execution did not improve, separating endpoint-controlled behavior from systematic rule generalization.

Research question

Does the model learn modulo execution or the order in which examples arrive?

The study uses a strict JSON register task so that an exact execution match requires the correct modulo residue, destination update, unchanged registers, and output format. The only intervention is byte-identical row order, allowing endpoint effects to be isolated from model, data, optimizer, and decoding changes.

Results

Key findings

Experimental design

Method at a glance

Model

Qwen3.5-0.8B adapted for a controlled register-language task

Operation

Compute an integer modulo 11, write the residue, preserve state, and emit strict JSON

Controls

Ascending blocks, reverse blocks, and interleaved residues with rotating epoch endpoints

Evaluation

Exact interpreter-based execution on train and held-out raw literals

Scope

Limitations

• The experiment studies one executable modulo-11 task and one adapted Qwen3.5-0.8B model.
• The eight-epoch budget does not establish or rule out later grokking-style generalization.
• The result is a causal case study of order and endpoint composition, not a universal claim about language-model training.

Open research

Preprint and reproducibility artifacts

Reuse and citation

Cite this preprint

Sezer, N. M. (2026). When Training Order Becomes the Algorithm: Endpoint-Induced Collapse and Memorization Without Rule Generalization. Zenodo.

@article{sezer2026trainingorder,
  title   = {When Training Order Becomes the Algorithm: Endpoint-Induced Collapse and Memorization Without Rule Generalization},
  author  = {Sezer, Nedim Mutlu},
  year    = {2026},
  doi     = {10.5281/zenodo.21718516},
  url     = {https://doi.org/10.5281/zenodo.21718516},
  note    = {Zenodo preprint}
}