A little-known former Anthropic researcher has sparked a global debate with a warning on the potential dangers of artificial intelligence.
Not many had heard about Jacob Coxon until he quit the artificial giant, saying the AI researchers believed the technology “could kill us all by the end of the decade.”
He barely had an online footprint and was not a major player in the community advocating for AI safety. Coxon briefly tried his hand at commodity training before shifting to Silicon Valley, as per the Wall Street Journal. Coxon is the son of a professor of medieval German literature. As a high schooler, he was selected for the International Mathematical Olympiad’s UK team. He studied math at Cambridge. He specialised in pretraining, which involved feeding AI models with massive amounts of text, images, and code before they could be used for specific tasks. Over time, Coxon started getting worried about superintelligence that could go beyond human capabilities. However, he assumed governments would slow down the pace of development, if needed. Coxon did keep an eye on AI’s progress in math as the models developed faster than he expected While he wasn’t ready to work on AI safety. In May this year, Coxon started working at Anthropic, believing that it was more transparent about AI safety than rival OpenAI. However, incidents like the Hugging Face attack at OpenAI made the researcher nervous. He considered shifting to a role working on AI safety before deciding to quit. Before finalising his departure, Coxon met with Kokotajlo, an old colleague and former OpenAI safety team member who is now an AI-safety researcher for advice. After he quit and posted his message about the dangers of AI, Coxon’s comments were echoed by the leader of Anthropic’s team responsible for making sure AI does what humans intend. Soon after, his comments caused an uproar, with fears of an AI-driven apocalypse taking hold of people. Coxon has continued his warnings, stating that trying to build super-intelligent AI is similar to summoning an alien species, as it is something researchers cannot fully control.
Now, the 27-year-old British researcher is at the centre of discussion surrounding AI and its risks. He won a silver medal in 2016 and a bronze the following year. His graduation in 2020 coincided with the launch of OpenAI’s GPT-3, a breakthrough model. He joined OpenAI in 2023. Kokotajlo assured the 27-year-old that he was making the right decision.
Years later, the Cambridge alum quit his job at Anthropic and said that labs racing to build AI were “gambling with our lives. “GPT-3 was really the ‘wow’ moment,” Coxon told WSJ. Will DePue, a former colleague, described Coxon as a “normal researcher”. Coxon said he went to his bosses at Anthropic in recent weeks to talk about his concerns. “Even working on safety at Anthropic felt like being complicit in the race,” he said.
