Anthropic's recent decision to bring on six external evaluators from the nonprofit Model Evaluation and Threat Research group to assess its AI models throughout development marks a notable step in the industry's efforts to self-regulate. But the program, as structured, falls short of genuine democratic accountability.
As mathematician and deep learning expert Yaim Cooper writes, the current setup is akin to a student picking both the subjects they'll be tested on and the friends who will administer the exam. The evaluators are selected by the company itself and are drawn exclusively from the AI safety research community—hardly a representative cross-section of the public that will bear the consequences of these technologies.
Cooper, who has long advocated for broader public input into AI design, argues that external evaluators should be embedded at AI companies but should represent the diverse communities affected by AI, not just safety researchers. They should be chosen by the public, not the corporations they're meant to oversee. Given the breakneck pace of AI advancement, traditional democratic processes are too slow to keep up. New infrastructures of democratic oversight are needed, and independent embedded evaluations should be a core component.
Two years ago, Cooper co-organized a workshop at NeurIPS, one of the largest AI conferences, to foster dialogue between AI developers and impacted communities. The most scandalous aspect of AI research, she contends, is not the catastrophic scenarios some fear—like human extinction—but the narrower visions of progress that dominate planning. A larger number of researchers worry about unprecedented concentration of power and capital, the erosion of worker dignity, and the expansion of surveillance networks that outpace constitutional protections. All of these risks demand mitigation, not just the most extreme ones.
Anthropic CEO Dario Amodei has endorsed the idea of "ongoing, employee-like access" for embedded third-party evaluators, with full visibility into training pipelines and processes. But Cooper insists these evaluators must be publicly funded and accountable to the public, not the companies they're monitoring. Moreover, the scope of evaluation must broaden beyond questions of mass casualties. AI's potential harms include psychological damage, disruption to creative industries, automated killing, unconstitutional surveillance, and the corrosion of democratic institutions themselves.
Cooper points to the tragic emergence of child suicides linked to chatbots as a grim reminder of the need for independent examiners with expertise in psychology and social science working alongside AI developers in real time. Post-hoc evaluations, she argues, are too late; interventions are most effective when integrated during the development process.
Jacob Coxon, an AI researcher who left Anthropic over what he calls the industry's recklessness, echoes the need for outside vigilance: "Keep your eyes open about the direction things are heading and not necessarily leave it all up to the people that are building it."
As public confidence in government and institutions wanes, and as Americans increasingly view tech leaders through a partisan lens, the demand for democratic oversight of AI is only likely to grow. The public deserves more than a "contact us" button that collects unread emails. It deserves a serious integration of democratically selected experts working side-by-side with AI researchers as they build a future we will all inhabit.
