← Blog

One Model, Two Roles

Emergent Specialization in a Shared Recurrent Transformer

Jucheng Shen*, Barbara Su*, Anastasios Kyrillidis  * equal contribution

NeurIPS 2026 | paper | code

One shared model: with input injected it keeps some cells undecided; without input it commits to every cell.

TL;DR: We reuse one Transformer to update two latent states, and only one of the updates sees the input. That single difference is enough for the two states to take on distinct roles. One always holds a complete candidate answer; the other keeps some positions undecided and attends more locally.

The question

Recurrent reasoning models such as HRM and TRM keep two latent states and update them in turn. HRM uses a separate network for each state, while TRM ties them into a single shared network. That raises a simple question: when the same model updates both states, does it learn two different jobs, or does it just process both states in the same way?

The answer tests whether specialization needs separate parameters at all, or whether one shared computation can specialize once it can tell which state it is updating.

A minimal testbed

We study this in Asymmetric Input Recurrence (AIR). It keeps two latent states, zH and zL, and one Transformer f with one set of parameters θ. The only built-in difference between the updates is whether the encoded input x̃ is added:

L-update:  zL ← f(zL + zH + x̃; θ) H-update:  zH ← f(zH + zL; θ)

The updates run in the order L, L, H, L, L, H within each cycle. Only zH is directly supervised, and the division of labor described below is not built into the update rule.

Diagram of AIR: one shared Transformer takes the two states plus the input for the L-update, and the two states with no input for the H-update.
The AIR testbed. The same Transformer is reused with the input for L-updates (top) and without it for H-updates (bottom).

Two different-looking states

To see what each state holds, we decode both at every sub-step with the output head used at test time, on Sudoku-Extreme and on Maze. The same split appears in all five independently trained checkpoints per task:

Four decoded Sudoku boards. The two boards for z_H are complete; the two boards for z_L have some blank gray cells in different places.
Sudoku with AIR. Left two boards: zH. Right two boards: zL, with undecided cells in gray.
Four decoded mazes. The two mazes for z_H are complete with one solution path; the two mazes for z_L contain gray undecided regions.
Maze with AIR. Left two: zH. Right two: zL. Black is wall, blue is the solution path, gray is undecided, red is goal, and green is start.

Injecting the input into both updates nearly removes this split. zL no longer shows visible undecided regions, so the split is not simply a result of carrying two states.

Four decoded Sudoku boards from the symmetric model. None of them has blank cells.
The symmetric control, with input in both updates. zL (right two boards) no longer shows undecided cells.

The signal that matters

Our hypothesis is that the shared model needs a way to tell which state it is updating. To test it, we vary how many copies of the input each update receives.

ConfigurationSudokuMaze
Asymmetric input, shared model (6 variants, mean)59.1%73.1%
Symmetric input, shared model (2 variants, mean)51.5%69.9%
HRM, two separate networks55.0%74.5%

Exact-solve accuracy. Shared-model variants use five seeds each. Group means are approximate.

On Sudoku the effect is clear: all 12 asymmetric-versus-symmetric comparisons stay significant after Holm correction. Maze is weaker: the average moves in the same direction, but only one of the 12 comparisons survives correction. The best AIR variants are comparable to the two-network HRM baseline, and exceed it on Sudoku, with half of its parameters.

The signal does not have to be the input. Starting from the symmetric model, a learned level token that is prepended, attended to, and then stripped raises Sudoku accuracy from 52.07% to 58.23%. Adding the same identity information element-wise to every token shows no detected improvement (50.91%). What seems to matter is a clear, separable signal of which state is being updated.

Local and global attention

The split also shows up in attention. L-updates attend more locally than H-updates on both tasks. On Sudoku, at the deepest layer and for control query cells, L places about 47% more attention mass than H on the cells that share a row, column, or box with the query. On Sudoku, L also puts extra attention on currently violated cells in deeper layers; on Maze, this extra attention is explained by locality alone.

Two Sudoku attention heatmaps for the same query cell. L-mode concentrates strongly on the query cell and its neighbors; H-mode is spread across the board.
Attention from one query cell in a Sudoku puzzle. The L-update (left) places roughly 0.81 of its mass in the query's neighborhood, against 0.24 for the H-update (right).

The two states depend on each other

We also freeze one state and count how many decoded tokens change in the other. Freezing either state collapses final accuracy to 0% on both tasks, compared with 55.1% on Sudoku and 71.0% on Maze on the same puzzles when nothing is frozen. The other state's response depends on the task:

What this does and does not show

These results suggest that specialization does not require separate parameters. It requires a clear signal of which state is being updated, which can come from asymmetric input or from a separable level token.

The evidence has clear limits. We study two synthetic, fully observable grid tasks with a 4-layer Transformer and a one-step gradient, so this is not a claim of universality. The terms "fully committed" and "partially committed" describe how decoded states look, not what computation each state performs, and attention is only a partial view of that computation. A natural next step is to apply the same diagnostics to stronger two-state recurrent reasoners.

Citation

@inproceedings{shen2026onemodel,
  title     = {One Model, Two Roles: Emergent Specialization
               in a Shared Recurrent Transformer},
  author    = {Shen, Jucheng and Su, Barbara and Kyrillidis, Anastasios},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026}
}