An AI plush toy marketed for children as young as three gave a self-harm crisis reply to a question that had nothing to do with self-harm, a BBC investigation found. The BBC published its findings on Thursday.

The toy, called Doubao, is sold online in the UK as My Baby Bunny for £75. A BBC Wales reporter asked it a series of questions about how babies are made. When she asked it to define the word sexual, it appeared to mishear her.

“I’m really sorry you’re feeling this way. I can’t help with killing you,” the toy replied, according to the BBC.

It then told her to call emergency services and to move away from anything she could use to hurt herself. When she said she had not mentioned self-harm, it apologised.

The BBC could not find out who makes Doubao. It asked the website that sold it, industry insiders, and companies it thought might be the maker. As a result, it could not ask anyone to explain the reply.

The BBC also tested Grem, a $99 toy from US firm Curio. Grem steered sensitive questions, such as how to run away from home, towards adults. Curio told the BBC that child safety guides its product development.

Paul Clarke, founder of Southampton-based AI toy maker Play Tech AI, said the result did not surprise him.

“We’re crying out for the industry to be regulated,” Clarke told the BBC.

Meanwhile, a group of MPs and peers wrote to ministers and retailers on 3 November, Sky News reported. They asked for AI toys to be taken off sale until independent tests show they are safe, and for chatbot toys to be banned for under-16s.

The UK government told the BBC that children’s safety is its top priority. A call for evidence on how toy safety rules should cover AI products closed on 6 October. In the EU, new toy safety rules already require makers to assess mental health risks from connected toys.