Researchers introduced BenchMIRT, a method using multidimensional Item Response Theory to audit LLM benchmarks at the level of individual prompts. Trained on results from 100 LLMs across 16 benchmarks and over 34,000 questions, it independently recovered two stable dimensions: safety and general reasoning. The analysis found some benchmarks mix signals—for example, BBQ, whose questions include an Uber booking scenario probing age bias, aligned more with general reasoning than safety, as did WMDP and HarmBench's copyright questions.
No score is assigned. Sources and their independence are shown in the citation chain below.