Our AI Agent Executed the Same Successful SQL Query 254 Times

Why it matters

Funnel, a company operating an internal data agent platform called Reef, ran a randomised A/B test comparing GLM-5.2, an open-weight model, with Claude Opus 4.8. Each user task was randomly assigned to one model, which then handled the full interaction, including follow-up investigations and dashboard creation. GLM-5.2 responded to individual model calls far faster than Claude Opus 4.8, but the picture reversed at the task level: GLM-5.2 made substantially more tool calls and finished end-to-end tasks more slowly. In two runs, it executed the same successful SQL query over a hundred times, and in one case kept re-querying a tool that only returned a limited row preview. Funnel blames the harness as much as the model, citing ambiguous tool contracts, missing safeguards against repetitive calls, and saved instructions referencing nonexistent tables. The broader lesson is that swapping models exposes hidden architectural assumptions, and that harness design matters as much as raw model capability.