Together AI Releases Together Compiler: Agents Make Inference Up to 2.6x Faster

Why it matters

Together AI has introduced Together Compiler, an agent-driven system that continually re-optimizes inference as production traffic shifts. Coding agents propose rewritten forward passes, a persistent engine measures each candidate, and the winning code ships into the serving stack. The company argues that engines tuned once go stale because real traffic patterns drift, and that kernels tuned on random data can mislead, since random inputs activated far more mixture-of-experts paths than genuine requests. The system was tested on models including Kimi K3, GLM 5.3, MiniMax M3.1 Flash, DeepSeek v4.1 Flash and Llama 1B, across NVIDIA, AMD and AWS hardware. Self-reported results show large gains on some settings and only marginal improvement on others, and the team also found that naive agents outperformed expert-curated optimization strategies. Readers should note that all figures come from Together AI itself and have not been independently verified.