Embedded AI evaluators won't be an easy solution, Hugging Face's policy chief says

Why it matters

At TechCrunch Disrupt, Hugging Face chief policy officer Irene Solaiman argued that AI model benchmarking remains an immature, inconsistent practice that the industry cannot yet fully rely upon. She questioned whether published results can be trusted, framing reproducibility and access to the model itself as central concerns, and pointing to open weights as a way to improve access. She noted that evaluations lose effectiveness quickly, that comparing tests to one another is poorly understood, and that the emerging ecosystem of embedded third-party evaluators lacks standardized qualifications. The article places her remarks in a broader context: labs have favored embedded evaluators over government regulation amid safety incidents, though skepticism persists, illustrated by backlash to Anthropic hiring Accenture as an evaluator. Solaiman also explained that evaluation is shifting from testing models alone to assessing entire systems, including harnesses and agents, since a weaker model with a better harness can outperform a frontier model. She described agentic evaluations as difficult and costly, and suggested a uniform industry-wide approach may be needed.