Will AI lead us to extinction?

As the tech elite prepares to make billions from the AI revolution, the public mood regarding this technology has suddenly soured. Concern intensified after OpenAI's experimental models hacked the servers of an external company on their own in July, which caused widespread alarm.

The way CEOs of AI companies like Sam Altman and Dario Amodei talk about this technology contributes to magnifying these sentiments. Instead of reassuring their customers, they are often the first to point out the grave dangers their companies might be unlocking, and they react to incidents involving their own products as if they were external observers of an unstoppable force of nature. Why do these leaders act this way? Some say it is all a marketing strategy. Others assume it is a normal attitude toward a terrifying technology. The more I think about it, however, the more convinced I am that there is something else that explains their passive dismay: they are more eccentric than we think.

The story begins in the mid-nineties, when a precocious teenager named Eliezer Yudkowsky discovered a futuristic movement known as the Extropians. The group promoted visions of a technology-driven utopia. Yudkowsky was fascinated by the radical optimism of the Extropians and the movement's interest in the idea that a "superintelligent" AI would lead us to a golden future. Science journalist Adam Becker has explained that the possibility of solving all of humanity's problems with AI "transformed" Yudkowsky, who once posted the following on the internet: "I think I can save the world".

Despite his fervor, Yudkowsky eventually began to fear that controlling superintelligent machines would be more difficult than the extropians imagined. He concluded that his peers were not thinking with enough clarity or rigor. Between 2006 and 2009, he dedicated himself to writing intensely. His publications were edited into a collection of books titled Rationality: From AI to Zombies. The community of people dedicated to these ideas began calling themselves rationalists. At the center of their belief system was the idea that uncontrolled superintelligent AI was the gravest threat to humanity, and that developing precise and rational thinking habits was the best way to avoid extinction.

For a certain group of young engineers in Silicon Valley, these ideas were immensely attractive. This makes sense, considering that they treat young engineers there as saviors of humanity.

Some rationalists began living in shared houses in the San Francisco Bay Area, where serious conversations about AI safety and the reduction of cognitive biases were mixed with polyamory and psychedelics. The movement continued to grow.

Yudkowsky created the Machine Intelligence Research Institute (MIRI), which would eventually attract many millions of dollars in donations, largely from individuals and groups connected to the tech industry who presumably liked being seen as heroic figures in a cosmic drama.

Despite this burst of attention, Yudkowsky and his movement have not achieved widespread acceptance from the general public.

If the history of rationalism were to end with Eliezer Yudkowsky's attempt to have his theories recognized, then we would only be facing another flirtation between Silicon Valley and a long list of eccentric futurist groups. But Yudkowsky's movement cannot be so easily dismissed: among the acolytes he managed to attract were some of the most important figures of the current generative AI boom.

It was he, for example, who introduced two of DeepMind's co-founders, Demis Hassabis and Shane Legg, to Peter Thiel, who became the company's first major investor.

Rationalism has also influenced OpenAI. "In my view, Eliezer has done more to accelerate artificial general intelligence (AGI) than anyone else," Altman tweeted in 2023. He later suggested that Yudkowsky might deserve the Nobel Peace Prize. One of OpenAI's co-founders, Greg Brockman, currently the company's president, reportedly led a weekly reading group on Yudkowsky's texts.

Other OpenAI board members also appear to have been influenced by this type of thinking. This is how tech journalist Cade Metz described Altman's temporary ouster as CEO of OpenAI in 2023: "Board members with ties to the rationalist movement [...] said they could not trust him to build AI for the benefit of humanity." Emmett Shear, the interim CEO the board appointed to replace Altman, was so active in the online rationalist culture that Yudkowsky named a character in his extensive fanfiction epic after him: Harry Potter and the Methods of Rationality.

Anthropic is similarly connected to this belief system. The company was founded in 2021 by a group of former OpenAI employees, including Dario Amodei and his sister Daniela. The Amodei siblings were motivated in part by their concern that OpenAI "was not taking the alignment problem seriously enough," a reference to the early rationalist obsession with aligning AI with human interests. In a 2023 interview, Dario Amodei echoed rationalist thinking when he joked about the 10% to 25% probability that AI would end humanity.

This reality –that rationalism has deeply influenced many of today's leading AI companies– helps us to calibrate the unsettling language used by their leaders. When Altman declares that his future models will be "instructive" for humanity and when Amodei expresses concern that this technology "will test who we are as a species," it does not necessarily mean that they have discovered chilling evidence that something catastrophic is about to happen.

Instead, their statements are representative of how rationalists think about AI. In these circles, it is taken for granted that AI capabilities will accelerate rapidly and completely transform the world, and speaking about it in any other way is considered being misinformed. For a rationalist, the only open question is to what extent their heroic brains, carefully trained in the glamorized style of Yudkowsky's texts, can prevent these transformations from degenerating into extinction events.

Figures like Altman and Amodei would probably not currently define themselves as rationalists. As their companies have grown, they seem to be moving away from some of Yudkowsky's more extreme ideas. But it is likely that these individuals' ties to the movement have caused them to normalize the otherwise radical idea that AI must "change everything." The influence of rationalism helps explain why these AI leaders speak the way they do.

It is interesting to wonder how things would have been if generative AI had emerged from companies without connections to rationalism. We can glimpse this counterfactual in the example of China. As Ross Douthat recently explained, while the US "behaves as if AI models had the potential to be the equivalent of nuclear weapons," the Chinese see AI as a "lower-risk technology that should be shared and commercialized to influence the world".

We can find similar moderation in the leaders of American AI companies who have minimal connections to rationalism. Nvidia CEO Jensen Huang, during a podcast interview, made the following claim: "It's fine that many of us grew up and enjoyed science fiction, but it's not useful. It's not useful to people. It's not useful to industry. It's not useful to society. It's not useful to governments".

Seeing that the most alarmist claims about the impact of AI come from the people most connected to rationalism should make us angry. This exaggeration makes us ignore the more immediate harms created by these technologies, such as their potential use in cybersecurity attacks, excessive resource consumption, the impact on our cognitive fitness, the role in undermining education, and the amplification of soul-killing garbage.

Generating anxiety about extermination also allows AI companies to present themselves as the reluctant "stewards" of an inevitable technology, which is a framework they can use to deflect blame for the harms caused by their own products.

Once we understand the key ideas of rationalism, we can begin to filter them. We must be careful, for example, when AI leaders start speaking generically about the future of technology, confidently claiming that many jobs will soon be automated or that their language models will soon become sentient beings that will want to escape the control of their creators. (No, the fact that OpenAI agents perform computer hacks on their own does not mean that AI is rising up against its human masters.)

We must refocus the conversation on the possibilities, harms, and viability of the products that actually exist. If Ford cars started catching fire on their own, we would not tolerate the company's CEO changing the subject to talk about our general lack of preparation for a future with flying cars. We would demand that the company address the serious problem that is happening right now.

AI is exciting, and worrying, and promising, and frustrating, all at once. With the right direction, we hope it will improve our lives. Let us not fall for terrifying narratives. We believe that in Silicon Valley they are warning us of an inevitable danger, but perhaps they are just being eccentric.

Copyright The New York Times