Why harnesses matter for energy efficiency

Why it matters

Neuralwatt, a cloud provider that bills customers on measured energy use, analyzed a month of its own traffic and concluded that the agent harness calling a model — not just the model or hardware — plays a major role in inference energy consumption. Writing in a first-person report, co-founder and CEO Chad Gibson explains that nearly all of the company's traffic is agentic, dominated by long prompts and short outputs, making prompt caching the largest efficiency lever. When a harness edits text it has already sent, the server must recompute everything from the first difference onward, and a mostly uncached request can cost roughly twice the energy of a cached one, more on GLM-5.3. The problem compounds in long sessions, where a single turn that re-reads most of its prompt can equal hundreds of ordinary turns in prefill work and cause visible latency. Lab tests traced reproducible cache breaks to harness behavior, though server-side eviction may also contribute. The findings suggest much of this waste is fixable by harness developers.