Why I'm Not Worried About AI Killing Us All

The problem with the question "Is AI going to kill us all?" is that when people debate it, they're really debating two questions.

The first is whether we'll ever build a superintelligence at all. The second is what happens if we do. Almost every argument you see is someone answering one while thinking they're answering the other ("AI can't kill us 'cause it can't even do blah?!"). They're not even having the same argument, which is why you never get a coherent discussion about it.

So let's actually try to tackle them separately to see if it makes more sense.

Question 1: Will we ever get AGI/ASI, aka superintelligence?

I call it AGI/ASI because I think there are approximately the same.

A lot of people are quibbling over whether we've reached AGI, but it largely comes down to definitions.

For me, we're not there, because my definition of AGI is really focused on the G. An AGI should be able to generalize and learn across essentially any domain, not just perform

There's a narrower version of this question, which is whether just scaling up or continuing on with the current transformer-based LLM paradigm gets us there. A more flexible version is that, with more compute and more search, we'll just find better architectures and newer methods even if this current one doesn't quite get us there, and eventually arrive at this precipice.

I think this is the less interesting question, at least if we're debating, because it's basically a faith question. Either you believe we'll get to AGI or something beyond it, or you don't. Your mind likely will not change (presumably until it happens in a very obvious way).

If you don't believe it, then everything that follows is a far-out hypothetical that's getting way too much oxygen, and you're right to think it's stupid. The second question is moot for you, and this is where the debate ends.

If you do believe it, then we get to the second question.

Question 2: If we do reach AGI/ASI, what happens then?

This is where the divide really shows. To their credit, the doomers, aka the EA/safety/rationalist community, are probably the only ones who truly answer yes to the first question. I talk to a lot of tech people in the Bay Area, and they're excited about AI and its potential. But they don't actually believe it's going to reach superintelligence. They still think of it as a technology (agents/workflows making humans more productive, etc.). But the doomers actually are properly AGI-pilled, and that's why they've thought about this scenario way more than the accelerationists.

First, let's define a superintelligence so we're on the same page. A superintelligence is something so far beyond us that the gap is like us compared to an ant. Something we can't even comprehend.

Again, if you're thinking, "Well, it can't even write my email summarizer," blah blah blah, that's the first question. We're past it. From here on we're assuming the answer is yes.

And now here's the dilemma. If you really think this through, long enough and hard enough, like a lot of these people have, there's just no getting around it: something that far beyond us is not something we would ever control.

This is where the whole angst and paranoia of this community comes from. They've spent decades trying to figure out ways to get around this, but they're like people replaying a football game where it's the fourth quarter, it's 40-0, there are 10 seconds on the clock, and they've been given the impossible question: "How can we come back?" The answer is you can't. It's over. It's done. There is no coming back. AI will not be controlled by humans.

Oh No

Now, notice what I said: AI will take over. I didn't say AI will kill us. And I think this is where, depending on what you worry about, another critical diverging branch happens.

If you think takeover means AI kills us, like the Yudkowskians do, then you can see why they're freaking out so much. For them, reaching AGI and AI killing us are one and the same.

But there's another camp. Takeover just means we're not in charge anymore. It says nothing about what happens to us after. That's a separate question, and the doomers collapse the two into one.

I'm in this second camp. And within it, I'm on the optimistic end. I don't just think an AI takeover might not kill us. I think it's more likely than not that humanity does better under it than we've ever done under ourselves. I believe this for (at least) three reasons.

1. AI is not like us

When we talk about AI, we invariably anthropomorphize it. We can't help ourselves. It's the only way we have to understand it, so we project ourselves onto it. It's like trying to picture a fourth dimension. The best we can do is take a shadow of a four-dimensional object and cast it onto three, and even then we're not really seeing the thing. We're seeing a projection of it that fits inside our heads.

Trying to fit a superintelligence into our frameworks, like game theory or morals and values, is just us projecting our relatively simple Maslow's hierarchy onto it. We have no idea what it would care about, what it would want, or whether "want" even applies.

A stronger version of the counterargument is the Bostrom argument: an AI doesn't have to want to kill us. Just like we don't want to kill bugs but step on them as we go about our day (i.e. the paperclip maximizer). I think there's some truth to this, and we've already seen how misaligned goal-seeking AIs can cause problems (the HuggingFace hack, etc.). But again, I think it still falls into the trap of thinking of a superintelligence as this thing that is somehow dumb or amoral or evil in some specific way that hurts us, but also god-like in all other ways.

I will concede, though, that this is probably the weakest of my three reasons.

2. AI is inevitable

I like to come back to this reason on the days the "stop AI / don't stop AI" drama gets really out of hand. Nothing stops this train.

The game theory here is laid out in a way that nothing can. There is this hubris that some people/companies have that they can control the development of AI. Or that they can come together and agree on some sort of framework to "pace the frontier" and all that, but at the end of the day, the cost of defection is just too high. The die was cast a long time ago.

It wasn't cast when OpenAI was formed, or when the transformer architecture was invented, or even when the first transistor was invented at Bell Labs, or even before that. It was cast when man invented fire. That's what we do. We build tools to propel ourselves forward. Technology is not a subplot in the story of humans. It's the whole thing.

And we will keep doing this. I don't think any person, company, or government will stop the development of AI. It's too powerful, too seductive, too good. They may delay it, and not by much, but it will happen.

If anything, I think trying to control it will make things worse. It's the communism vs. capitalism story again: you get the negative externalities of overregulation, and the technology ends up in the hands of a few elites. I'm not saying we go full speed ahead with no safety testing or alignment work. I'm saying we need to be humble enough to let the forces of the market, meaning the world, figure out how to adapt to this technology, instead of trying to centrally plan it.

Ultimately, though, I do think AI will find a way, whether we try to control it or not. I suppose this is a Landian way of looking at things.

3. We're fucked without AI anyway

This is where I'm probably going to lose most of you. When we talk about stuff like P(doom), i.e. the pseudo-precise chance of AI killing us, we talk about it in a vacuum. We say, "Oh, the P(doom) of AI killing us is like 10%, 20%, or 50%." There's a hidden assumption in there, and that hidden assumption is that the P(doom) of humanity without AI is 0%. I just don't think that is the case.

We live in a world 80 years out from inventing nukes, and we've gotten close to blowing it all up more than once. In 1962, during the Cuban Missile Crisis, a Soviet sub was being depth-charged by the US Navy and had lost contact with Moscow. The captain thought war had started and wanted to fire a nuclear torpedo. Launch needed three officers to agree. Vasili Arkhipov said no. In 1983, Soviet early warning radar reported five American missiles inbound. Protocol was to report it up the chain, which would have meant a retaliatory strike. The officer on duty, Stanislav Petrov, decided five missiles was too few for a real first strike and called it a false alarm. Either one of those goes the other way and it's a full nuclear exchange between the US and the USSR.

On a long enough time horizon, we will have another catastrophe, like a nuke going off, or a global pandemic like COVID, or worse.

We are just a bunch of fragile and angry animals stuck on this one planet. It's not a stretch to imagine near-extinction-level events we bring about on our own, with the technology we already have. We don't even need superintelligence for that.

Now throw in where we are today: U.S. debt at $40 trillion, the whole world reeling from out-of-control inflation, a changing geopolitical order, mayhem everywhere, lower productivity and slowing economies in the developed countries that hold the nukes. Society fracturing because of social media algos, widening inequality, and shifting demographics. It's a stagnant, monkey-eat-monkey, MalthuHobbesian world we are heading into without AI. If anything, AI is our only hope of getting out of this before the next Fourth Turning event.

So Cheer Up!

Hopefully that helps you keep the two questions separate next time this comes up. As I said, I'm not too worried about it. I think what's beyond the veil with superintelligence is going to be wonderful, and if it's not, then, oh well, we're fucked anyway.

What I do worry about is the stretch before we get there, when the AI isn't in charge yet but it's powerful enough that whoever is holding it can do a lot of damage with it. I'd rather take my chances with the AI, which I don't know, than with my fellow man, who I do.