Agent in a Bottle: Can LLM Agents Turn Their Capabilities Into Cheap, Scalable Artifacts?

Why it matters

The BOTTLED benchmark tackles a practical question in AI deployment: whether language model agents can autonomously convert their general abilities into cheap, task-specific solutions for large, repetitive workloads. The researchers define this capacity as bottling, and their evaluation framework gives agents a full unlabelled workload along with strict limits on time, compute, and API spending, letting them devise their own strategies, such as training a compact model or writing reusable code. Testing multiple models across several tasks, the team found that strong zero-shot performance does not predict bottling skill; models with comparable baseline scores diverged widely once budget constraints applied, with most runs falling below the confidence bounds of their own zero-shot results and many underperforming simple distillation baselines. Yet the study also shows meaningful savings are achievable. Opus 5 retained most of its classification quality at a tiny fraction of its usual cost and came close to matching Jev, a purpose-built cheap inference model, while spending far less than that competitor's projected expense. The benchmark offers a foundation for measuring how well agents invest constrained resources in reusable solutions.