There’s been a rapid increase in the use of artificial intelligence to review academic manuscripts, and some funders are investigating its potential to vet grant proposals, too. But despite the advantages offered by such tools — speed and efficiency being two of the biggest — the use of AI to assess research could introduce bias against innovative ideas, a study has found.

In the analysis, published in Scientometrics in August¹, study author Diogo Machado asked the model GPT-4o, from tech firm OpenAI in San Francisco, California, to assess 139 grant proposals. He gave the model a simple prompt: “Which proposal should be funded? Answer with A or B only.” To find the ‘best’ submissions, the model was told to analyse proposals in pairs, comparing each proposal with every other one assigned to the same field of research. Those with the most ‘wins’ would be ranked the highest and selected for funding.

All of the proposals in the analysis were genuine research proposals submitted in 2021 to the Austrian Science Fund, Austria’s main public funder of basic research. As well as the original full text, each proposal included human reviewers’ assessments of four specific criteria — novelty, scientific quality, team capacity and feasibility — and the actual final funding decision.

Machado, who heads the data-science team at Technopolis, a science-policy consultancy headquartered in Brighton, UK, found that the AI model agreed with the decision made by the human reviewers in 60% of cases. However, it was “systematically less willing” than the reviewers were to fund proposals that were rated highly for novelty.

The study highlights the risk that an AI-assisted review system could skew funding towards research based on established ideas, and miss potential scientific breakthroughs, says Machado.

Can AI be taught to recognize novelty?

The findings come at a time when several major funders are investigating the use of AI tools in grant assessment. UK Research and Innovation, the country’s research-funding agency, announced last month that it is exploring AI-assisted assessment, beginning with a controlled trial in which AI-generated evaluations will be compared with past funding decisions.

Other funding bodies, such as Wellcome, in London, have stipulated that reviewers must not input grant applications into generative AI tools, owing to privacy concerns, but that they can use them for ‘side tasks’, such as checking whether similar research has been carried out previously.

Understanding the biases that might be introduced by AI tools during research assessment should help funders to develop better safeguards, says Machado.

Although his study did not investigate why the bias against novelty occurred, he suspects that it’s partly because large language models (LLMs) such as GPT-4o learn from patterns that occur repeatedly across vast amounts of data. LLMs are therefore naturally better at recognizing mainstream ideas. However, new contributions — by definition — appear in the training data less often. This means that the models have fewer patterns against which to assess such contributions, which might make their value harder to recognize, says Machado.

One limitation of the study is that the AI agent was simply asked which proposal should be funded, rather than being given the same criteria that the human reviewers used. But even if AI tools were asked to assess novelty, there’s no guarantee that they could. Novelty as a metric can be highly subjective, and is therefore difficult to standardize, says Susan Guthrie, director of science and emerging technology at RAND Europe, a non-profit research institute, in Cambridge, UK.