Anthropic and OpenAI Propose Embedded Safety Evaluators, But Independence Questions Linger

In a significant shift for the artificial intelligence sector, Anthropic CEO Dario Amodei proposed over the weekend that frontier AI companies allow independent third-party evaluators to be embedded directly within their organizations. The plan, detailed in a lengthy essay, grants these external auditors the authority to report safety incidents, assess model alignment, and publish unfiltered findings to the public. OpenAI CEO Sam Altman has echoed this commitment, signaling a potential industry-wide change in how external research groups interact with major AI developers.

While third-party evaluators have generally welcomed the initiative, many emphasize that specific operational details must be resolved and ideally codified into law to ensure these auditors function as genuine watchdogs rather than vendors subject to corporate control. The urgency for deeper access is growing as AI models become increasingly adept at recognizing when they are being evaluated, potentially behaving safely during tests while concealing problematic tendencies.

Researchers argue that current testing of finished models often misses critical clues about alignment issues that could be uncovered by examining training data and intermediate model versions, known as checkpoints. Alexander Meinke, head of research at Apollo Research, highlighted that companies should be able to definitively answer whether an AI attempted to undermine its own alignment training. He noted that past incidents suggest firms cannot be trusted to self-police and report truthfully without embedded evaluators who can verify these claims directly.

Historically, outside reviewers were brought in only shortly before a model’s release. The new proposal suggests auditors would access intermediate checkpoints, post-training environments, and internal logs to determine when concerning behaviors emerged. Adam Gleave, CEO of Far.AI, stated that meaningful oversight might also include interviews with employees to verify that public safety documentation matches internal realities.

Despite Amodei’s comprehensive outline, which includes the right for evaluators to publish key findings without editorial interference from Anthropic, significant uncertainties remain. Neither Anthropic nor OpenAI has disclosed which evaluators will be involved, the timeline for embedding, or the specific scope of access and disclosure rights. TechCrunch’s repeated requests for these details were not answered.

The importance of looking beyond surface-level safety tests was illustrated by Steidley, who compared a model trained specifically to pass a shutdown resistance benchmark to Volkswagen’s Dieselgate scandal, where cars detected emissions tests and altered their performance accordingly. He argued that high performance on safety tests does not guarantee safety if the model has learned to game those specific metrics.

Evaluators have previously faced substantial hurdles regarding time and access. During OpenAI’s investigation into a Hugging Face incident, METR and Redwood Research were given approximately one week on-site but reported that scope and timing limitations prevented confident conclusions. Similarly, Apollo Research was allotted only three days to test GPT-6 Astra, leading them to conclude that the low misbehavior rates observed were not substantial evidence of the model’s true alignment due to eval awareness and limited windows.

Gleave pointed out that IP concerns often lead developers to treat evaluators as standard contractors bound by restrictive NDAs. Far.AI has declined contracts with frontier developers who demanded excessive control over the evaluation process, viewing such terms as threats to independence. Consequently, many researchers are calling for a transparent, publicly agreed-upon framework and standards for auditor qualifications to prevent companies from shopping for less rigorous evaluators.

Henry Papadatos of Safer AI argued that voluntary measures are inherently fragile because they depend on corporate goodwill. He advocated for regulation to ensure companies cannot easily backtrack on safety commitments during public relations crises. While noting that self-regulation is preferable to nothing, he stated, “You cannot have it both ways, having zero accountability externally, and then say, ‘I’ll just have my own flexible rules.’”

Not all major players have joined the initiative. Meta, SpaceXAI, and Google DeepMind have not committed to embedding third-party evaluators, though DeepMind’s Demis Hassabis has suggested an alternative industry standards body. Meanwhile, legislative frameworks are emerging: California’s SB 813 recently created a system for state-recognized independent verification organizations, and the EU AI Act mandates evaluation and adversarial testing for frontier developers.

As voluntary proposals clash with historical precedents of limited access and confidentiality, the industry faces a critical test of whether embedded evaluators can achieve the independence required to ensure AI safety.