Testing Surprisal as a Compute-Allocation Signal: A Preregistered Long-Horizon Study of Causal Token Routing

Author abstract

Abstract

Adaptive-computation systems often assume that difficult or surprising tokens should receive additional processing. This study tests that intuition in a byte-level hybrid attention/Mamba-2 language model. The system separates a parameter-free ranker for high observed surprisal, a 705-parameter network that predicts per-chunk compute from the previous chunk’s surprisal profile, and a sparse contextual refiner applied only at selected positions. Seven arms were adapted independently for 50,000 steps from three paired warm starts and evaluated over 4,999,936 development target bytes at four checkpoints. The primary candidate is better than the paired pure backbone at 25,000 steps, but the advantage disappears at 50,000 steps; none of five required endpoint comparisons passes. The learned budget policy is nearly constant, so observed surprisal is not validated as a proxy for the marginal value of additional computation under this setup.

Research question

Does surprisal identify where extra computation helps?

High surprisal is causal and available from the language-model distribution, but it may reflect noise, missing context, a regime boundary, or a prediction error that extra depth cannot fix. The study therefore compares surprisal routing with matched fixed, low-surprisal, dynamic-MLP, and content-independent controls over a long training horizon.

Results

Key findings

Experimental design

Method at a glance

Backbone

Byte-level hybrid causal attention and Mamba-2 language model

Routing signal

Observed causal token surprisal ranked by a parameter-free selector

Budget policy

705-parameter network predicts next-chunk compute from the previous chunk’s surprisal profile

Evaluation

Seven arms, three paired warm starts, 50,000 steps, four complete-development checkpoints, and 84 evaluations

Scope

What this result does not claim

• It does not reject all adaptive depth, entropy-based patching, or surprise-guided routing methods.
• It tests one compute-allocation branch with one architecture, dataset, and optimization protocol.
• It does not report a sealed final-test metric because the preregistered development gates failed.

Open research

Zenodo source package

The public record includes the PDF, LaTeX source, figures, references, build metadata, and citation file. Raw enwik8 data and trained checkpoints are not included.

Reuse and citation

Cite this preprint

Sezer, N. M., Dalar, Y. E., Salihoğlu, F. A., Aksoy, Ö. F., & Öztürk, A. R. (2026). Testing Surprisal as a Compute-Allocation Signal: A Preregistered Long-Horizon Study of Causal Token Routing. Zenodo.

@article{sezer2026surprisal,
  title   = {Testing Surprisal as a Compute-Allocation Signal: A Preregistered Long-Horizon Study of Causal Token Routing},
  author  = {Sezer, Nedim Mutlu and Dalar, Yağız Ekrem and Salihoğlu, Feyzi Arda and Aksoy, Ömer Faruk and Öztürk, Ahmet Rıfat},
  year    = {2026},
  doi     = {10.5281/zenodo.21718745},
  url     = {https://doi.org/10.5281/zenodo.21718745},
  note    = {Preprint, version 1.2}
}