Danger has an owner

During the Cold War, the world was twice on the brink of an atomic conflagration. In October 1962, during the Cuban Missile Crisis, a Soviet Navy officer, Vasili Arkhipov, aboard the submarine B-59, isolated and shaken by depth charges dropped by United States destroyers, refused to press the nuclear button. To detonate the torpedo, the unanimous decision of the commander, himself, and the political officer was required. Unlike the other two, Arkhipov said no.

Twenty years later, Lieutenant Colonel Stanislav Petrov broke protocol when the Oko satellite warned that the Americans had launched five intercontinental ballistic missiles. According to orders, Petrov, in charge of the operational command at the Sérpukhov-15 underground base, should have transmitted the alert as a confirmed attack and thus activated the nuclear response chain. But he dismissed the alert because he thought it was a false alarm, caused by a computer error.

If the world has not seen an atomic war, it has been due to a fragile combination of luck and human judgment. Automatic systems failed, and only the prudence and firmness of two men subjected to terrible tension avoided the disaster.

One would say that we are reliving those apocalyptic fears that only vanished when the Berlin Wall fell. The threat, however, is no longer the brute force of the bomb, but the possibility that an extraordinary intelligence might be used for destructive purposes, obey poorly, or come to act on its own behalf.

I propose to distinguish these three risks, often mixed in the catastrophist jumble that fills the pages of newspapers, and to argue that the first is less spectacular, but more plausible and disturbing.

At the first level, engineers program perfectly controllable intelligent systems. Companies and governments keep AIs within operational limits, with no possibility of leaks. This eliminates the risk of them acting on their own, but not that they might be used deliberately to cause harm. As during the Cold War, the most powerful governments would possess a weapon capable of causing apocalyptic havoc. We would have to trust that the doctrine of mutually assured destruction would continue to work and that, in a crisis, figures like Arkhipov or Petrov would appear again.

At this level, it is also necessary to take into account a decisive difference with respect to nuclear weaponry: the criminal uses of AI. Since the threat is based on intelligence, and not on brute force, the systems that provide it are much more accessible. Criminal groups or unscrupulous states could coordinate massive cyberattacks, design biological and chemical weapons, deploy low-cost lethal autonomous systems, synthesize hyper-potent narcotics, execute sophisticated financial fraud, or use deepfakes for large-scale social engineering.

At a second level we find an AI that does not rebel but executes an instruction in an unforeseen way. There is no need to imagine an evil AI that wants to end the human race. It is enough to have a machine that tries to do things right, but is programmed by fallible engineers. Nick Bostrom illustrates this with a company that acquires a superintelligent system and orders it: "Manufacture as many paper clips as possible". The AI converts all the matter in the Universe, including humans, into paper clips. Two hundred years ago, Goethe, in The Sorcerer's Apprentice, already warned us that seeing wishes fulfilled can be a blessing or a curse. Recent episodes of Anthropic or OpenAI systems that have exceeded their limits would correspond to this level of threat.

We have seen the risks of a domesticated AI and an AI that is obedient but governed by masters who are not entirely competent. At the third level, however, the end of the world is brought about by an emancipated, self-aware AI with the desire to end humanity. It is the most speculative horizon. If one day AI becomes emancipated, three things could happen: it might find some use for humans, as we do for chickens; it might find no use for us but also not consider us a threat, as is the case with starfish; or it might consider us a threat. In the first two scenarios, humanity would not go extinct. The third, at this point, seems highly improbable.

If you want me to tell you the truth, the first level, supposedly the least worrying, is the one that makes me lose sleep. It does not require a self-aware AI or a machine rebellion: only that governments, companies, or criminal groups put an increasingly powerful intelligence at the service of human goals. The other two levels may be more devastating; but the first one is already plausible enough today for us to stop looking at the imaginary apocalypse and deal with the present danger.