LLM analytics benchmark: the best model for your analytics agent

Why it matters

Cassis, which builds an analytics agent that answers business questions over a large schema and its accompanying context, has built a repeatable benchmark to decide whether to swap the model behind its agent on each new release. The article frames agent errors as having three possible causes: the needed information is missing or wrong in the context, the agent fails to find it, or the model misreads what it finds. The first two are Cassis's responsibility; the third changes with every model release. Its benchmark combines a public Formula 1 database drawn from an academic dataset with business questions inspired by real internal usage, including cases the available data cannot answer. An LLM judge evaluates SQL equivalence for business cases, and each test is repeated to measure consistency. The results prompted Cassis to switch its default agent model, and the piece also examines how its retrieval harness changes outcomes.