An agent reliability benchmark, a safety resignation and a new federal AI task force

Microsoft and Hugging Face put repeated agent outcomes at the center of a new benchmark

Microsoft and Hugging Face published ThinkingBox, a benchmark for agents operating in stateful business workflows such as retail, insurance, travel, banking and consulting. Rather than judging a fluent final answer or a completed tool call, it checks the resulting records and side effects in isolated environments. The published set has 507 workflows, and each is run 20 times from the same clean starting state.

That framing produces a useful distinction between an agent that can complete a task once and one that does it consistently. The accompanying paper reports that Claude Opus 5 completed 241 of the 507 tasks on all 20 recorded attempts, or 47.53%; the figures are the authors’ benchmark results, not a general measure of every agent deployment. Microsoft and Hugging Face have released the materials through Hugging Face, so teams can inspect the setup and test their own systems rather than treating a single pass rate as proof of reliability.

The practical point is less about a leaderboard than verification. An agent may make valid tool calls and report success while leaving a ticket in the wrong state or creating an unwanted side effect. For work that changes customer records, money or access, a repeatable check of the final state is stronger evidence than the agent’s account of what it did.

Departing OpenAI safety employee calls for deeper operational safeguards

David Robinson, who says he wrote safety reports for major OpenAI launches, resigned and argued in an Atlantic essay that frontier labs need the layered safeguards and deliberate planning used in aviation or nuclear operations. TechCrunch and The Verge independently reported the resignation and his critique; the broader claim about OpenAI’s culture is Robinson’s assessment, not an independently established finding.

OpenAI spokesperson Drew Pusateri told TechCrunch that the company continues to strengthen security in research and testing, use third-party evaluators and improve monitoring, and that it pauses training or withholds models when needed. The competing accounts matter: a public resignation can document an employee’s experience and priorities, while it does not by itself settle how effective a company’s safeguards are in practice.

Robinson’s argument is aimed at the operating model, not merely a new checklist. His concern is that iterative deployment can surface failures only after systems are placed into real settings. The available reporting does not independently test that proposition, but it sharpens a question for customers and regulators alike: what evidence demonstrates that an agent has been contained, monitored and evaluated before it is entrusted with consequential actions?

White House announces a federal AI task force led by Jay Clayton

President Donald Trump announced on Sunday that Director of National Intelligence Jay Clayton will lead a federal task force on artificial intelligence, which Trump has called the “Super Intelligence Force,” according to the Associated Press. The AP reported that the group will include Federal Trade Commission chair Andrew Ferguson, Pentagon chief technology officer Emil Michael and Office of Personnel Management director Scott Kupor, and will report to Trump and chief of staff Susie Wiles.

The stated remit is broad: coordinate the federal effort, engage groups including consumers, public-interest organizations, critical-infrastructure providers and AI companies, and preserve U.S. leadership. The announcement did not set out a detailed mandate, budget, legal authority, timetable or enforceable requirements for companies. Those omissions leave the task force’s practical role unresolved.

It also follows the White House meeting at which Trump said AI-company leaders had agreed to a voluntary accord. A coordination body and a voluntary commitment can influence policy and company behavior, but neither is the same as a binding rule or an independent assessment regime. The next test is whether the administration publishes concrete responsibilities and mechanisms for accountability.