Foresight Is Not Enough: Sentence-Level Future Signals, Self-Loop Hard Negatives, and the Calibration Gap in Small Language Models

Author abstract

Abstract

A language model can learn a representation that appears to anticipate future text, but such a representation may still fail to control generation behavior. We study this representation-to-behavior gap in ForesightLM-v2, a DistilGPT-2-scale language model trained with a sentence-level future-prediction head and self-loop hard negatives. Across three WikiText-103 seeds, the held-out hard-negative margin is consistently positive, indicating a stable training-side future signal; a one-seed WritingPrompts run provides a second-domain sanity check. However, fixed-pool generation controls show that most aggregate gains are explained by multi-candidate semantic reranking rather than the learned future term itself. Removing the future term changes only 2 out of 120 selected continuations, random and shuffled future-score controls closely match learned-future reranking, and a training-free base-model five-candidate reranker outperforms ForesightLM-v2 on several semantic diagnostics. The future score is nevertheless modestly aligned with prompt anchoring and progression at the candidate level. We therefore present ForesightLM-v2 not as a standalone coherence solution, but as a controlled case study in how learned semantic signals require calibrated behavioral interfaces and strong negative controls.

Research question

Does learning to predict the future change generation behavior?

ForesightLM-v2 asks whether a causal language model can use a learned sentence-level future representation to produce better continuations—not merely whether that representation can be measured during training. The study separates representation learning from behavioral control by evaluating every selector on the same fixed candidate pools.

Results

Key findings

Experimental design

Method at a glance

Model

Approximately 82M-parameter DistilGPT-2-scale causal language model

Training data

WikiText-103 and a one-seed WritingPrompts sanity check

Auxiliary objective

Sentence-level future prediction with self-loop hard negatives

Evaluation

120 prompts with fixed pools of five candidate continuations

Controls

Generic, random, shuffled, no-future, future-only, lexical, and training-free base-model selectors

Interpretation

A representation-to-behavior calibration gap

The future score carries semantic information at the population level, but it is too weakly calibrated to reliably rank the best continuation within each prompt. The result is a methodological warning for representation-based generation: showing that a model encodes a useful signal is not enough to show that the signal causally improves its behavior.

Learned representations should be evaluated not only by whether they exist, but by whether they change behavior beyond credible alternatives.

Scope

Limitations

• Generation quality is assessed with automatic diagnostics over 120 prompts rather than blinded human judgments.
• WritingPrompts is a one-seed sanity check, not broad evidence of domain generalization.
• The hard-negative pool and held-out validation set are deliberately small.
• The study does not establish that the future signal improves token-level decoding or search-space exploration.

Open research

Code and artifacts

Artifact status: The public GitHub repository and Hugging Face model currently reflect earlier phases of the project and may not reproduce every result reported in the v1 preprint.

Reuse and citation

Cite this preprint

Öztürk, A. R., Dalar, Y. E., Aksoy, Ö. F., Sezer, N. M., & Salihoğlu, F. A. (2026). Foresight Is Not Enough: Sentence-Level Future Signals, Self-Loop Hard Negatives, and the Calibration Gap in Small Language Models. Research Square. https://doi.org/10.21203/rs.3.rs-10331341/v1

Download the BibTeX citation

@article{ozturk2026foresight,
  title   = {Foresight Is Not Enough: Sentence-Level Future Signals,
             Self-Loop Hard Negatives, and the Calibration Gap in Small Language Models},
  author  = {Öztürk, Ahmet Rıfat and Dalar, Yağız Ekrem and Aksoy, Ömer Faruk
             and Sezer, Nedim Mutlu and Salihoğlu, Feyzi Arda},
  year    = {2026},
  doi     = {10.21203/rs.3.rs-10331341/v1},
  url     = {https://doi.org/10.21203/rs.3.rs-10331341/v1},
  note    = {Preprint}
}