The artificial-intelligence company, based in San Francisco, California, has been at the centre of a series of media storms. First, in July, the firm announced that its models had hacked the AI hub Hugging Face, a collaborative platform where users share models and data sets. This week, the Australian government revealed that OpenAI agents had breached a national health-care website. And earlier this month, the company announced that its models had solved a major problem in mathematics — but was then faced with suggestions that the success stemmed from its models absorbing advances already made by human mathematicians.

Meanwhile, calls for regulation of AI have grown. Leaders of AI firms, including OpenAI, have outlined fears that giving models free rein to boost their own capabilities, through a fully autonomous version of a process known as recursive self-improvement, could result in humans losing control over the technology. Researchers also fear ‘misalignment’ — meaning that the models’ values and goals do not match up with those of humans.

At the heart of the action is Mark Chen, the chief research officer at OpenAI, who leads model development. Chen sat down with Nature to discuss how AI will affect science, along with the firm’s pivot towards safety.

Where are you hoping to apply AI to science?

I want to dispel the notion that we create math-specific models.

For instance, in the drug-discovery process, we hope that models can really shrink the pipeline down. Literature review and self-contained research projects just take a lot of manual labour; research into potential targets can take weeks or months. We are exploring whether our models can accelerate that process.

There are also other kind of areas that we’ve been pushing — for instance, in semiconductor design and development. We have developed our own chip, Jalapeño.

Some mathematicians worry about AI’s growing role in the field. You’ve formed an advisory group in response — why?

With all fields, math included, we want to bring the community along with us as we develop frontier AI. These technologies have the potential to change industries. We want to figure out, along with these fields, how should engagement go. OpenAI wants to publish results to show where the technology is and what it’s capable of. But it’s not our desire at all to cover the field. We want to co-develop the norms.

How do you ensure that your models are safe?

We take a multi-pronged approach. But fundamentally, the most robust and future-proof solution looks like monitoring. Models can expose what’s called a chain of thought, their unfiltered ‘thinking’ as they pursue a solution. Models monitoring [the internal processes of] other models is today the most effective way to secure and align them. One of the challenges is that, as models become more powerful, the amount of compute they do before exposing their ‘thinking’ is greater. That begs the question of can you look deeper in the model, or somewhere else in the model, to understand the traces of thinking?

How are you approaching self-improving AI?

We want to use our models to accelerate the development of future generations of the models.

Our first goal here has been to produce a ‘research intern’: [an AI agent that] comes in with very little context, that you can give a well-defined scope of work to, and trust with things like running basic experiments or debugging certain runs. As our first milestone we wanted several hundreds of thousands of chips, directly managed by Codex [a code-writing agent] to run internal research experiments, and that is indeed the state we're in.

A more ambitious goal, coming in 2028, is more end-to-end research. One of the big bottlenecks here is teaching the model a better sense of ‘research taste’, like which ideas are worth exploring.