OpenAI on Monday rolled out an image classification tool designed to help vetted organizations and law enforcement identify child sexual abuse material (CSAM), including content generated by artificial intelligence. The company said the classifier is the first to include a dedicated signal for AI-created abuse imagery.

The tool, which will be limited to approved safety groups and investigators, was trained and tested to recognize visual indicators of abuse. Its release comes as AI firms face growing scrutiny—similar to the pressure social media platforms confronted a decade ago—to prevent their services from being used to produce or spread such material.

Read also
Technology
United Airlines resolves tech outage after FAA ground stop fallout
United Airlines resolved a technology issue that delayed departures, following a nationwide FAA ground stop that disrupted thousands of flights over the weekend.

OpenAI claims its classifier outperforms existing tools by reducing false positives. According to the company, it flags roughly 48 percent fewer images overall while still identifying a comparable number of abusive images. Moreover, images it does flag are about twice as likely to contain actual abuse, the firm said.

Company leaders emphasized that human review remains necessary to ensure accuracy. The classifier was developed with input from the National Center for Missing and Exploited Children, the Tech Coalition, and the ROOST (Robust Online Safety Tools) project.

OpenAI’s models already prohibit CSAM and are trained to refuse requests to generate it. On Monday, the company also announced a new warning in ChatGPT that appears when users attempt to seek out such material. The warning directs users to behavioral resources and makes clear that sexualizing children causes harm, even when the requested content is AI-generated.

“Alongside the warning, we’ll remove the flagged conversation and pause access to ChatGPT for 24 hours, while keeping specialist support available to help people stop harmful behavior,” OpenAI said in a blog post. The company added that repeated violations could lead to longer restrictions, account closure, or reports to authorities.

John Buckley, OpenAI’s head of child safety public policy, said the company’s teams learned from law enforcement about capacity constraints during the tool’s development. “We want to make sure that our tooling is available to support them to do a really effective job at triaging, getting to high-risk cases much more quickly,” Buckley told reporters Monday.

Buckley stressed that the tool is intended only for organizations equipped to handle CSAM. “Our tool needs to be at the hands of organizations who can deal with CSAM, we do not want to hand this tool out [to] individual developers who are not equipped with the specialized review that’s required or registered to report to NCMEC or other international reporting hotlines,” he said.

The announcement follows broader debates over child safety online, including legal battles such as tech giants challenging California’s child safety law and policy disputes like the Democratic divide on childcare policy. OpenAI’s move also comes as the company expands its national security portfolio, recently tapping a former Army secretary for a national security role.