Token Pooling-Aware Training for Multi-Vector Embedding Models

Why it matters

Multi-vector retrieval models, which store a collection of contextual token vectors for each document, are expressive for search but expensive to index, since large corpora can hold enormous numbers of vectors. This article presents token pooling-aware training as a way to close the quality gap that compression introduces. Pooling works by merging similar token embeddings through Ward hierarchical clustering, so each group of related vectors is replaced by its normalized mean, shrinking a document's representation to fit a storage budget. The problem is that an encoder trained normally never learned to survive this merging, and distinctions that matter for retrieval can become unrecoverable. The author shows that training the encoder with the pooling objective in mind lets it retain far more of its unpooled retrieval quality at the same reduced vector budget. The piece situates the method within prior work, including Stefan Josef's Learn to Pool and LightOn's pooling regularization study, and offers a practical route: building a compressed retrieval model with Sentence Transformers. It also covers adaptive pooling budgets and threshold calibration against a sample corpus.