How Specializing the Vocabulary of SLMs Can Cut Inference Cost by 77% in Real Applications

Why it matters

The article, from the team behind the VocabX method, previews findings from a paper still under review. The authors run a production system that grades Portuguese essays, built on Qwen3-4B, and observed that fine-tuned models inherit tokenizers designed for many languages and domains. This creates two forms of waste: fragmentation, where the text a specialized model writes most often, such as repeated JSON structures and evaluation phrases, gets split into many tokens; and excess, where the model scores a huge vocabulary at every generated step, including entries the task almost never uses. The piece explains why generation is far more expensive than reading: during decode, each token requires a full pass through the network and the process becomes limited by memory bandwidth, so the output layer's size matters on every step. VocabX rebuilds the vocabulary around token cost rather than frequency, keeping the base model, data, and training recipe unchanged, and substantially shrinks the output layer. The authors note the figures may change before publication.