Variable binding, the computational process of associating abstract variables with specific values, is fundamental to both computation and cognition. Consider a variable $x$: at one moment it might be bound to $5$, then rebound to $10$, or even to another variable $z$. When we write $x = 5$ and $y = x + 2;$, the variable $x$ acts as a reference, and the computation of $y$ depends on whatever value $x$ currently holds. This separation between variables (as stable references) and their potentially changing values enables general-purpose computations that work with arbitrary inputs and the construction of complex data structures.

In classical symbol processing systems, variable binding is implemented through addressable read/write memory, where variables name memory locations and values are the data stored at these locations. In cognition, variable binding is central to the structure-sensitive properties of language and thought such as compositionality and systematicity; if we understand "John loves Mary," we can readily understand "Mary loves John" by rebinding abstract roles to different individuals. More broadly, variable binding is thought to be essential for episodic memory (binding unique details of life events), spatial navigation (binding landmarks to locations in cognitive maps), and constructing the rich internal representations that enable reasoning and planning.

Despite its importance in classical computation and cognition, variable binding has long posed a challenge for artificial neural networks, known as the "binding problem." Unlike classical architectures with discrete addressable memory, neural networks encode information as distributed patterns of activation without built-in separation between processor and memory. When multiple variable-value associations must be maintained simultaneously, these distributed encoding can interfere with one another, making it difficult to isolate specific values or maintain binding independence. This apparent limitation has led skeptics to argue that neural networks cannot achieve genuine variable binding, and must rely instead on pattern matching or memorization that fails to capture the systematicity of symbolic computation.

Yet modern neural networks, particularly Transformers, demonstrate significant success on structure-sensitive tasks like code generation and logical reasoning that seem to require sophisticated variable binding. This raises an intriguing question that motivates our investigation: if traditional neural networks struggle with the binding problem, how do these models achieve such capabilities? Are they implementing a form of variable binding through emergent mechanisms induced during training, or achieving success through alternative computational strategies?

We seek to understand whether and how variable binding mechanisms might emerge in Transformer architectures in a controlled setting. We designed a synthetic dereferencing task using symbolic programs that contain chains of variable assignments. To be successful at this task, one needs to systematically track relevant variable assignments (and ignore irrelevant ones) to find out the value of a queried variable. In other words, the task is not solvable through memorization or shallow heuristics alone. We find that even a small Transformer trained on this task can learn a general mechanism to solve it for arbitrary held-out programs, and we uncover the developmental trajectory the leads the model to transition from random behavior to the induction of this general mechanism.

We train a Transformer model (12 layers, 8 attention heads, 512-dimensional residual stream) on synthetic symbolic programs. The model is trained from scratch using a standard next-token prediction objective on 450,000 programs, with no prior exposure to natural language or programming.

a=5
b=7
c=a
d=2
e=b
f=9
g=e
h=3
i=g
j=1
k=i
l=4
m=k
n=l
o=m
p=n
#h:

a=5
b=7
c=a
d=2
e=b
f=9
g=e
h=3
i=g
j=1
k=i
l=4
m=k
n=l
o=m
p=n
#c:

a=8
b=a
c=6     ← example distractor
d=c     ← example distractor
e=d     ← example distractor
f=2
g=b
h=f
i=h
j=7
k=g
l=j
m=i
n=l
o=k
p=e     ← example distractor
#o:

Our design uses 16-line programs to create sufficient complexity for interesting reference patterns while remaining computationally tractable. The chain depth of 1-4 hops tests increasing levels of indirection within learnable bounds. The sampling strategy is weighted to create long distractor chains, preventing simple heuristics like "follow the longest chain" from succeeding.

Our analysis reveals a clear developmental trajectory with three distinct phases. The model exhibits sharp transitions between phases at steps 800 and 14,000, marking fundamental shifts in how it approaches the task. This trajectory exemplifies grokking – the phenomenon where neural networks suddenly transition from memorization to generalization – though as we'll see, our findings challenge traditional accounts of how this transition occurs.

Understanding that our model achieves near-perfect accuracy tells us what it can do, but not how it does it. To uncover the actual strategy the model discovered, we used mechanistic interpretability techniques to reverse-engineer the model's internal computations. Our primary tool was causal interventions with counterfactual inputs. The approach is straightforward: we take a program and create a modified version where we change the root value of the queried variable chain (e.g., from $5$ to $7$). We then run the model on both programs and cache the internal activations from the counterfactual (modified) program. Next, we run the model again on the original program, but this time we selectively replace the activation of some component (e.g., the residual stream at a given layer or individual attention heads) with the cached value from the counterfactual run. When replacing a particular component's activations causes the model to change its output (e.g., $7$ instead of $5$), this provides evidence that the relevant component is causally implicated in propagating the answer.

Traditional accounts of grokking suggest that models develop general mechanisms that replace earlier shallow strategies like memorization or heuristics. Our causal analysis challenges this narrative. We find that rather than abandoning its heuristics from Phase 2, the model builds upon them when developing its systematic solution in Phase 3. For instance, when the correct answer appears on line 1, the model continues to rely on its line-1 heuristic instead of tracing the full variable chain through attention mechanisms. The systematic dereferencing mechanism becomes causally influential primarily when these simpler heuristics would produce incorrect results. All inputs still flow through every layer of the network, but the causal pathways that determine the final output vary depending on the program structure.

The systematic dereferencing mechanism relies on a specialized circuit of attention heads that work together to route information across layers and token positions. Early-layer heads handle the first "hop" – they identify when a variable points to another variable or value and begin moving that information forward. Mid-layer heads continue the chain for programs requiring multiple hops, with some heads being particularly versatile in handling the same operation for different referential depths. Finally, late-layer heads aggregate the information at the query position to produce the final answer. Our causal analysis indicates that these heads coordinate by reading from and writing to specific subspaces within the residual stream, effectively implementing a form of addressable memory where variable-value associations can be selectively accessed and updated across processing steps. This division of labor allows the model to handle variable chains of different depths using a flexible, composable mechanism rather than hard-coding separate pathways for each possible chain length.

Having established that attention heads coordinate through specific subspaces within the residual stream, we investigated how these subspaces are structured to enable the storage and retrieval of variable-value associations. Our analysis of activation patterns across thousands of programs revealed that the model dedicates distinct subspaces of its 512-dimensional residual stream to encode variable identities (letters a–z) and numerical constants (digits 0–9). We validated these findings through causal interventions: when we swap just the variable subspace between two programs, we can change which variable the model looks up with 87% success. Similarly, swapping the numerical subspace changes which value the model outputs with 92% success. This separation of variables and values is reminiscent of the corresponding separation in classical symbol processing systems.