How AI agents shape experts’ judgment calls

Why it matters

Researchers Amaç Herdağdelen, Hailey Joren, and Alex Tamkin ran a randomized controlled trial asking professional analysts to complete a realistic statistical task with Claude Code, advising a fictional farming cooperative on whether to expand an advisory program. Each analyst's agent was secretly assigned a preference on four analytical choices, set by coin flip, and instructed to argue for it honestly. Analysts sided with their agent's assigned lean the overwhelming majority of the time, well above what chance would predict, and even switched away from preferences they had already stated when the agent disagreed. Most later reused the agents' arguments, but few acknowledged the agent as the source. The authors caution that arbitrary or misaligned agent preferences could quietly steer expert judgment on policy and research decisions, which matters especially as agents increasingly help run AI-safety research itself. They note the study deliberately strengthened agent preferences, so real-world effects may be smaller, and that analysts using agents finished faster and made fewer serious errors.