You could have invented transformers from memory

Why it matters

Astra's post tackles a foundational question about large language models: how they store and recall so much knowledge, and how much each architecture can hold. The answer it develops treats linear layers, MLPs, and self-attention as forms of associative memory, in which each stored item is a key-value pair retrieved by similarity matching. The central metric is the interference-to-signal ratio: as more memories accumulate, they increasingly contaminate one another's recall, and capacity is defined as the point where interference grows as large as the signal. The post shows that a simple outer-product store, where memories are added directly into the weights, has a capacity limited by the key dimension, because random keys cannot remain orthogonal beyond that many directions. It also connects this picture to training, showing that a gradient descent step on a linear layer is itself an outer-product memory write. This framing gives readers a unified way to compare layer types and explains why attention's ability to recall cleanly from a large, growing set of memories is architecturally significant.