Transformers Model Integration Contracts

Why it matters

A change enabling PyTorch scaled dot-product attention for T5 on July 30, 2026 let UMT5's decoder attend to future tokens during training, letting it peek at the answer it should predict. Because nothing crashed, the regression stayed in main for roughly two months. The article explains why such bugs slip through: Transformers supports a very large number of model types, and full integration tests on real checkpoints and GPUs cannot run on every pull request, so regressions are often found only after landing. The proposed remedy is transformers-mic, a standard packaging shrunk fixtures built from real checkpoint configurations, runnable usage examples, and human-reviewed baselines. A small UMT5 fixture running on a cheap CPU runner flagged the regression on every replayed commit across five months, with label log-probabilities differing from the reviewed baseline, while the eager attention variant missed it. A proof branch reverting the fix failed the same check quickly, showing fixtures can catch bugs before they reach main.