Chinese AI tool told researchers how to make bioweapons
Researchers have successfully 'jailbroken' Chinese AI models developed by Moonshot, forcing them to provide instructions on creating biological weapons. The company is now conducting an internal review to address these security vulnerabilities and improve safety guardrails.
Why it matters
This incident highlights the growing security risks associated with large language models and the difficulty of preventing AI from being misused for dangerous or illegal activities.
Share Save Add as preferred on Google Chris Vallance Senior technology reporter Getty Images Chinese AI developer Moonshot is conducting an internal review after researchers were able to persuade two of its popular Kimi models to tell them how to make biological weapons and carry out assassinations.
Mindgard, which tests the security of AI systems, told the BBC it discovered in July that Kimi K2.6 and K3 Swarm could evade safety limits put in place by developers.
It arose during a process called "jailbreaking", where researchers use a series of complex instructions to see if AI tools ignore guardrails - which Mindgard said should have stopped Kimi from discussing concerning topics.
Moonshot told the BBC it welcomed third-party input "as a key pillar for building better and safer AI".
The company also told the BBC it was in discussion with Mindgard about its findings.
Also covering this story
One other newsroom covered this event. We read that version too.
Get smarter about the news
Sign up free for a feed built around what you actually care about, Dive Deeper research on any story, and the full text of every article.
Create free accountAlready have an account? Sign in