Gurnee et al. (2025) have shown that the representations of token positions in a line are located on a multidimensional manifold. In this work, we reproduce a part of their paper on the open models. We show that, contrary to the demonstration of Anthropic,
- lala1
- lala2
- lala3
- Gurnee, W., Ameisen, E., Kauvar, I., Tarng , J., Pearce, A., Olah, C., & Batson, J. (2025). When Models Manipulate Manifolds: The Geometry of a Counting Task. Transformer Circuits Thread. https://transformer-circuits.pub/2025/linebreaks/index.html