Co-LMLM: Continuous-Query Limited Memory Language Models

Co-LMLM, proposed by Yair Feldman and collaborators, pairs continuous keys with textual knowledge values in a limited memory language model architecture. The system generates flexible vector queries while integrating human-readable, attributable retrieved knowledge into generation. An accompanying annotation pipeline tags factual spans in arbitrary text, eliminating prior Wikipedia-only restrictions. Across Wikipedia and FineWeb-Edu pretraining at multiple scales, Co-LMLM outperforms previous LMLMs and standard LLMs in perplexity and factual precision, with 360M-scale models matching gpt-4o-mini on SimpleQA verification.