Researchers from Stanford and UC Berkeley Introduce TokenGate: Token-Gated Attention Makes Long-Context Prefill Up to 11.3x Faster

Why it matters

Researchers from Stanford and UC Berkeley have released TokenGate, a drop-in module that adds learned token pruning to existing large language models, targeting the cost of long-context prefill. Because transformers process every token through every layer, attention expense grows steeply as input length increases, and prior approaches either lose accuracy or require retraining from scratch. TokenGate instead inserts a small MLP gate, trained with LoRA, that scores tokens after an early layer and lets only the highest-scoring ones continue through the rest of the network, while the question portion of a prompt is never pruned. To make the top-K selection trainable, the team borrowed a Mixture-of-Experts router trick, turning token scores into attention logit biases so gradients can reach the gate. Tested on Qwen3-1.7B and Qwen3.5-2B across benchmarks including needle-in-a-haystack and multi-hop question answering, the module reportedly speeds time-to-first-token and cuts peak prefill memory without accuracy loss, and works on hybrid-attention models where some baselines cannot prune. Code and checkpoints are available on GitHub under an Apache-2.0 license.