Economist Noah Smith is the author of Noahpinion, one of the most popular Substacks in the world.
Far from worrying about human extinction from superintelligent AI, Noah is optimistic AI will create a world where humans still have plentiful, high-paying jobs!
In this debate, I stress-test his rosy outlook. Let’s see if Noah can instill us with more confidence about humanity’s rapidly approaching AI future.
Timestamps
00:00:00 - Episode Preview
00:01:41 - Introducing Noah Smith
00:03:19 - What’s Your P(Doom)™
00:04:40 - Good vs. Bad Transhumanist Outcomes
00:15:17 - Catastrophe vs. Total Extinction
00:17:15 - Mechanisms of Doom
00:27:16 - The AI Persuasion Risk
00:36:20 - Instrumental Convergence vs. Peace
00:53:08 - The “One AI” Breakout Scenario
01:01:18 - The “Stoner AI” Theory
01:08:49 - Importance of Reflective Stability
01:14:50 - Orthogonality & The Waymo Argument
01:21:18 - Comparative Advantage & Jobs
01:27:43 - Wealth Distribution & Robot Lords
01:34:34 - Supply Curves & Resource Constraints
01:43:38 - Policy of Reserving Human Resources
01:48:28 - Closing: The Case for Optimism
Links
Noah’s Substack — https://noahpinion.blog
“Plentiful, high-paying jobs in the age of AI” — https://www.noahpinion.blog/p/plentiful-high-paying-jobs-in-the
“My thoughts on AI safety” — https://www.noahpinion.blog/p/my-thoughts-on-ai-safety
Noah’s Twitter — https://x.com/noahpinion
Transcript
Introduction and the Definition of Doom
Liron Shapira 00:01:42 Welcome to Doom Debates. Noah Smith is a highly influential writer and economist, best known for his blog, Noahpinion, one of the most popular Substacks in the world with a whopping 418,000 subscribers. He has a talent for translating complex economic ideas into clear, concrete arguments that actually connect to real-world policy and culture.
His work often challenges conventional wisdom, sometimes from the left, sometimes from the right, which has made him a must-read across ideological lines. You may have already read his arguments about industrial policy, globalization, energy, China, or the future of the US economy.
Recently, he’s been writing about AI as well, and even getting into the AI Doom debate, which is why I invited him to come on this show. So today I’m excited to talk with Noah about the risk of imminent human extinction from super-intelligent AI, or, in the lucky scenario where we don’t go extinct, the risk that we’re all out of a job. Noah Smith, welcome to Doom Debates.
Noah Smith 00:02:39 Great to be here.
Liron 00:02:39 So yeah, longtime reader of your blog, and I’m personally impressed with just the wide-ranging, broad analysis. Obviously, it goes without saying how prolific you are. So you’re definitely one of a kind, and I hope you keep doing what you’re doing.
Noah 00:02:51 Thanks. I bet with AI, I won’t be one of a kind because people will be able to just press the button and generate infinite content.
Liron 00:02:58 That’s very true, actually, and we are definitely going to get into that. But let’s zoom out to a thousand feet, because the main topics I want to focus on in this conversation are basically P(doom)—human extinction, which you’ve written a little bit about—and the other main topic we’ll get to later is unemployment. So, kind of two forms of doom.
Zooming all the way out, I gotta ask you the number one most important question. You ready for this?
Noah 00:03:22 Yeah.
Liron 00:03:24 Noah Smith, what’s your P(doom)?
Noah 00:03:31 My P(doom)... Do you mean what’s my probability that AI kills us, or what’s the probability that technology as a whole kills us?
Liron 00:03:39 You can frame the question however you like, but in some part of your answer, definitely talk about AI killing everybody.
Noah 00:03:45 Technology as a whole killing us? I would give it 96% P(doom). Yeah, we’ll eventually go away.
Liron 00:03:54 Eventually. Okay.
Noah 00:03:56 I don’t think it’s P(doom) within the next five years. I would say 0.01%.
Liron 00:04:04 Okay. That’s quite a broad range, so let me help narrow this down. How about the probability that all human life has been eradicated by 2050?
Noah 00:04:13 All human lives have been eradicated by 2050? I would give you a... oh, I don’t know...
Liron 00:04:23 Got it. So when you get to the 96%, roughly, are you talking about like a quadrillion years, or where did that come from?
Noah 00:04:28 I would talk a few hundred years. I think two things will happen. Number one, fertility rates will go low and stay low. The only people who will be able to escape that trap are people who significantly re-engineer the human species, reimagine the human species, and turn us into something else.
We’re already incomprehensibly weird to the people who came a hundred years ago. Whether or not you think technology is accelerating, I think the weirdness is accelerating—the effect of technology on things about human nature that we long held to be constant.
So I think that changing the definition of what it means to be human is imminent and will happen more and more, so that humanity as we know it will simply evaporate. I think that’s highly likely.
Liron 00:05:12 Well, when you first said you have a 96% P(doom), it sounded kind of pessimistic, but are you actually meaning it in a term that you don’t necessarily consider bad? Because you’re actually talking about reinventing humanity in a way that’s likely good.
Noah 00:05:25 It depends on how nostalgic you are. I think a good analogy is how I see Judaism. I was raised Jewish and I have Jewish relatives, and people are like, “Oh my god, all the Jewish people are out-marrying, no one’s raising their kids Jewish, and no one’s observant basically except for the Ultra-Orthodox.” So Judaism as we knew it, outside of those Ultra-Orthodox communities, is sort of evaporating.
And I’m like, okay, that’s true. That will go away. But you know what else? Maybe we’ll get some new stuff. If you really value those old cultures, those things that have continuity with the Agricultural Age, the Iron Age, if you value those old things, it makes sense to be really upset about this because technology is changing culture so fast.
I think old stuff goes away, new stuff comes. Our descendants—such as they are, whatever they are, be they AIs, cyborgs, uploads, or whatnot—are not going to look like us. They’re not going to feel like us. We won’t have a lot to talk to them about. They’ll be weird to us. So that is disappointing in a way.
You read Childhood’s End, the Arthur C. Clarke thing? Spoiler: the kids all become telepathic. They network and they just go off. Arguably, this already happened with social media—kids became telepathic and became incomprehensible to us. The younger generation evolves into some hive mind that goes off into space and leaves everyone else to die.
You could say that that’s bad because traditional humanity goes away and you just got these weird-ass telepathic kids that you can’t even write a convincing character for. Or you could say, cool, you know, humans were a bunch of violent, thuggish, selfish meat sacks imprisoned within their own brains, and it’s cool that now we have this network-type mind of people and that’s better. Maybe the new “whatever it is” will be better than us. It just depends on how much you value the old and what kinds of continuity you want.
Catastrophic Risks: Extinction vs. Transformation
Liron 00:07:16 Okay. So this particular thing that you’ve called Doom, there are certainly some very conservative people who agree that that’s doom. I get the sense that you personally actually see that as net good. So you personally might not even prefer the connotation of doom.
Noah 00:07:30 I am melancholy about it. I want Star Trek. I want humans in something like their current form, but a bit better, zipping around the stars and exploring strange new worlds. I want that.
Liron 00:07:44 It’s like you’re a transhumanist, but you’re worried that the flavor of transhumanism may not be your ideal flavor, and you think there’s a 96% chance that you personally will be unhappy with the flavor?
Noah 00:07:56 Yeah. The reason is because I’ve just seen what social media has made of humanity, and I’m like, it’s so weird. I can look at a few things about it and be like, “I dig that, that’s cool,” but overall, I’m just like, what are these motherfuckers talking about? Can I curse on this podcast?
Liron 00:08:14 Yeah, you can. So the whole galaxy is just gonna be like a big Section 8, and you and I are gonna be melancholy about that.
Noah 00:08:23 Yeah.
Liron 00:08:24 I mean, that’s roughly your main line scenario. That is interesting because you put up a really optimistic vibe in the last few years of your blog. You’re a self-described optimist. You’re always seeing the bright side of things. But if I understand correctly, you just told me that what’s almost 96% likely to happen... some of that is better, but most of the 96% probability is a scenario that you’re not happy with.
Noah 00:08:46 Right? But also a 100% scenario that I’m not happy with is that you and I will die. P(doom) for you and me is a hundred.
Liron 00:08:55 Yeah. But isn’t there a middle option where we just solve death without doing the “kids being way too weird”?
Noah 00:09:00 Well, no, because the thing about death is that I’ve come to understand that death is a continuous process. Ship of Theseus argument, basically. I’m not the five-year-old me. I’ve changed incomprehensibly much since I was five. Life equals experience. Experience always changes who you are. That change makes you effectively a different person over time, replacing you plank by plank as the saying goes.
Liron 00:09:25 But if you look at Charlie Munger, isn’t it like a much bigger delta? A few months ago, that one particular day seems like a much bigger change than on that day.
Noah 00:09:34 It is a bigger change on that day. But I would say the change from Charlie Munger’s non-existence now to who Charlie Munger was when he was alive a little while ago, is probably about the same as the change from when Charlie Munger was five to when Charlie Munger became old. So like, it’s a big delta, but not the only meaningful delta.
Liron 00:09:50 Let me push back on the Ship of Theseus thing a little bit. Okay, childhood, obviously your brain is still developing big time. But Charlie Munger at 20 versus 99... if you just talk to Charlie Munger at 99 versus 20, I think you’re gonna see a lot of commonalities. So I think it may be a little too dismissive to be like, “Ah, he’s already changed so much anyway. Dying, or getting replaced by a different entity, what’s the difference?” I feel like it’s the same Charlie Munger.
Noah 00:10:12 I think that’s mostly an illusion created by our desire to name that human Charlie Munger. We want to think of humans as consistent. So we model them. It’s our model of Charlie Munger. What we’re seeing is not the actual Charlie Munger. People aren’t even consistent from day to day, certainly not from decade to decade.
I think that you and I look at people—I look at you, you look at me—and we’re making a model of that person. That model we make is more consistent than the actual person by quite a lot. I think it was the French essayist Montaigne who most famously pointed this out. He’s like, “There is no you.” There’s this collection of different things that we model as you.
Liron 00:10:59 Well, if you are already admitting that the model can be pretty similar from one year to the next, and people also have models of themselves, doesn’t that imply that people’s own models of themselves are consistent over their lifetime, which then further implies that their actual selves are consistent over their lifetime?
Noah 00:11:13 Maybe not necessarily because your model can change a lot, but you can backdate it and tell yourself this was always your model of yourself. So if I’m different and my self-concept is different, I can say that it was always like this 20 years ago, 30 years ago.
Liron 00:11:29 All right. Well, I don’t wanna go too far down this particular identity rabbit hole, although I do know that you love rabbits. Shout out to rabbits.
We’ll put a pin in it. Obviously, we have different conceptions of the metric of similarity of a human’s internal state or personhood from one decade to the next.
Where I wanna focus the conversation is I wanna take what you were describing as the majority of the 96% probability that makes you melancholy when you talked about doom. And for the purposes of this conversation, I actually wanted to define that as not doom. Because that’s still like a type of transhumanist heaven, like Robin Hanson’s heaven.
Noah 00:12:05 The kids like rock and roll and the kids like LSD and the kids are weird. But yet we still try to give our kids a good life. When I think about my grandparents’ generation—the World War II generation—versus my parents’ generation... They fought World War II and bled and died on foreign shores and worked and had cardboard in their shoes during the Depression to give their kids a better life.
And then the kids come along and they start doing drugs. They’re the first generation to be rich. They’re individualist in a way that their ancestors never were, which happens to every society that gets rich and becomes a consumer society. But you see the Boomers, and the Boomers are weird to the older generation. And yet they have to be happy about that. They have to say, “We gave a better world to our kids, whatever our kids want to be.”
I think that’s what we have to do. We have to give a better world to our kids, whatever those kids are gonna be. The kinds of doom I worry about most are the kinds of doom where we don’t get to evolve and things just fall apart. From bio-weapons killing us all, or... I guess I do worry about some crazy AI killing us. It’s not at the top of my list for AGI just going Skynet on us, but it’s certainly bad. I don’t want that to happen because then you destroy the continuity. We have no kids. That’s “childless doom.” We don’t get to have the weird-ass kids that we can’t understand.
Liron 00:13:30 Okay, so let’s refer to that... let’s call it “Childless Doom” just to frame the rest of the conversation. I think I am going to try to argue that the probability of childless doom is uncomfortably high. So let me ask you this to start off: what’s your probability of childless doom by 2050?
Noah 00:13:57 Here we also get to the question of how much doom is doom. If human civilization falls apart and we go back to the Bronze Age or the Neolithic, I’d say that’s doom. Even if there’s still a few humans.
Liron 00:14:10 That’s an edge case, ‘cause we kind of bounced back.
Noah 00:14:14 I don’t even know if that’s an edge case. I’d say that’s more likely to me than all humans extinct. Much more likely.
Liron 00:14:21 Okay. I see what you’re saying. Because that’s kind of like some nuclear winter, but then we crawl out and we build it up again.
Noah 00:14:27 Exactly. Like nuclear or bio-weapon. A bad human enabled by AI creates this bio-weapon that kills almost everybody, but a few people are immune because that always happens. The idea that you’d be able to make a bio-plague so good it gets every last motherfucking human is not how plagues work. Ever. Yes, I understand that a smart AI could make plagues work better, but probably a couple humans will survive, repopulate, be like cavemen, etc.
Liron 00:15:01 You’re really breaking down the question itself, which is fair. So let me factor it into this. What is your probability of the human population getting slashed in half—let’s call it catastrophe—and then after that you can tell me your P of total extinction. Both by 2050.
Noah 00:15:28 Population getting slashed in half by total catastrophe... I give that a 10% chance.
Liron 00:15:36 I think that’s reasonable. I mean, just nuclear war I feel like is a percent a year.
Noah 00:15:39 Nuclear war, yes. Bio... I’m worried more about bio than nuclear. Nuclear war, we actually don’t have enough nukes to really threaten human civilization right now. We’d have to build an order of magnitude up. But we could do that.
Liron 00:15:56 All right. So let’s talk about total extinction, which we both agree nukes probably don’t get you to a hundred percent. So what do you think is total extinction?
Noah 00:16:02 My probability of total extinction by 2050 is so low that Daniel Kahneman would yell at me for giving a number. It’s 0.1%. Which is where people stop being able to conceive of probabilities.
Liron 00:16:16 Wow. Okay. Well, I mean, once you get to like 0.01%, then even an asteroid impact is that low.
Noah 00:16:24 Humans can’t understand the difference between those two things.
Liron 00:16:25 So I think the crux of disagreement in this conversation is... I think we’re both on the same page about the non-AI types of catastrophic risks. You seem to have a very calibrated worldview about those. So the only difference is: is AI an existential threat in the next couple decades?
Just to clarify, you would not then sign that Center for AI Safety statement on AI risk, right? It says: “Mitigating the risk of extinction from AI should be a global priority alongside other societal scale risks, such as pandemics and nuclear war.”
Noah 00:16:56 I think that is true. I think that is correct and those aren’t separate. Asking whether AI is a risk alongside global pandemics is like asking whether guns kill people or people kill people. Because the way that AI would kill people is not that AI does some sort of computer magic to make us vanish. It’s that AI would use something like a bio-weapon or nuclear weapons to kill us.
AI Capabilities: Can Intelligence Organize Violence?
Liron 00:17:21 So in the scenario where AI is like an agent, it independently goes off and develops a bio-weapon, maybe hiring a few robots or human satos... wouldn’t the possibility of that scenario increase your P of AI doom beyond 0.1%?
Noah 00:17:37 No. I just think it’s really hard to extinct-ify all of humans.
Liron 00:17:44 But you’d sign the statement because you think AI can facilitate catastrophes.
Noah 00:17:49 Remember I said 10% for reducing human population by half, which is a catastrophe equaling the worst catastrophes that have ever happened.
Liron 00:17:57 Gotcha. Okay. So I guess we both agree that there’s some scenario where some terrorist has an AI assistant and slashes the human population in half using bio.
Noah 00:18:05 Yep. That is really scary. I give that kind of thing, like that plus the nuclear thing, a 10% chance.
Liron 00:18:13 Okay, fair enough. And would you agree that that could even happen somewhat unintentionally, where you just give the AI kind of a long leash to be an agent and you don’t directly tell it to destroy the world, but you kind of don’t give it enough restrictions and it’s unaligned in that sense?
Noah 00:18:27 I mean, I think that’s part of the risk. But humans are... there’s so many humans who are already so bad that I find it conceptually difficult to think about the difference between the AI starting shit and the human with the AI starting shit. I feel like those two are so similar to me.
If you have unaligned AI, then the unaligned AI would help a bad human do it. Or the unaligned AI could do it on its own agentically. But the unaligned AI will need lots of robots. It will need lots of human people to cooperate. So asking “who’s in the driver’s seat”—the AI or the humans who help the AI—is almost like an angels and pinheads question. It’s a little bit interesting because it involves how we design the agentic rules, but I believe agentification will not be the critical thing here.
Liron 00:20:13 Well, I think to clarify, the interesting question is just: can AI spearhead a really big project? So imagine I have access to the world’s most powerful AI in 2028. I’m a terrorist and I just type: “Slash the population in half, however you wanna do it.” And then that’s the last command I ever type. Do you think it’s plausible if AI can run with that?
Noah 00:20:32 The answer is I don’t have a clear idea of that because the AI, in order to do that, is gonna have to organize a lot of stuff. There’s gonna have to be a lab that makes the bio-weapon. There’s gonna have to be humans that it cons into doing this or robots that it gets to do this. There’s a lot of moving pieces. I think the scenario where an AI is like Skynet—where it has all its own robots—is unrealistic.
Liron 00:21:06 So I wanted to ask you a basic question about your mental model of AGI and the trajectory. Roughly, what do you think is kind of a mainline scenario? What’s your equivalent of “AI 2027”?
Noah 00:21:24 I don’t know. I think the idea of a recursively self-improving God AI is unlikely. I think it’s extremely unlikely. AI may be able to do AI research, so there may be some recursive self-improvement because code writes code. But in terms of an intelligence explosion of the type that Vernor Vinge wrote about or I.J. Good, I think...
Liron 00:21:58 Okay. So no FOOM. But what does happen?
Noah 00:22:01 I think AI just keeps getting better. It may not top out because it may be that marginal costs are everything. You can get a really, really smart AI if you’re willing to spend insane amounts of inference compute on it. So the only thing that keeps scaling is inference. I’ve read arguments to that effect.
It’s possible we get the Hitchhiker’s Guide to the Galaxy future in which the entire Earth is just a computer to answer one question. So essentially all of Earth’s history is inference compute for one single query.
Liron 00:22:39 I’m curious how load-bearing this prediction is that it’s gonna take a bunch of compute for a single query. Let’s go head to head: AI versus cancer doctor. When is it going to be a time where it’s just better to let the AI speak for him because his judgment is 99.99% of the time gonna be worse? Do you think in the next few years or couple decades we are likely to hit that point? And if so, do you think it’ll take the whole world computing to hit that point or can be done in a small data center?
Noah 00:23:15 A small data center.
Liron 00:23:16 Okay, so that’s interesting because you were talking about a scenario where it kind of takes the whole earth to compute.
Noah 00:23:29 Oh, I’m just saying... suppose you get an evil AI that wants to kill people and you wanna stop it with a good AI. This is really important to say how compute intensive this intelligence is. Whether you’re doing mainly inference scaling is really important because if everything’s just inference compute, then you’ll tend to have one AI—or maybe one or two—that’s smarter than all the others by quite a lot.
And if that AI is unaligned, we’re in trouble because then the evil AI is much smarter than the good AIs and the good AIs can’t stop the evil AI. That’s a lesson from economics: economies of scale are really important.
Liron 00:24:19 Totally. So if I understand correctly, you’re talking about a scenario with two superhuman giant brains where both are already far surpassing the top human genius and they’re kind of duking it out like an attack/defense scenario.
Noah 00:24:40 And they’re called countries, by the way.
Liron 00:24:42 Okay, sure. So, I think we agree that sometime in the next 20 years, maybe a lot sooner, a small data center is going to be a good drop-in replacement for a human doctor. Is that fair to say?
Noah 00:24:53 Yes.
Liron 00:24:54 Wow. Okay. And you feel pretty confident about predicting that it’s already true today. I agree. I get a lot of medical advice from the doctors... I don’t know if I trust a hundred percent ‘cause they do sometimes make mistakes.
Noah 00:25:06 I don’t trust... I will never trust anything a hundred percent. I will want second opinions even in the age of Godlike AI because hallucinations are fundamental and Godlike AI will be really, really good at convincing its hallucinations are true.
Liron 00:25:23 We’re definitely going to come back to this point because that’s interesting that you take this as your starting premise—that the AIs are just going to be better than humans at mental tasks. And in another post that we’re gonna talk about, you even run with that premise and you say, that’s okay, ‘cause I’m still optimistic about humans having jobs.
Noah 00:25:37 Yeah, I’m optimistic about humans having jobs. I’m maybe a little cynical in that I think if society’s insanely productive, then we will organize redistribution around essentially calling fun consumption things “jobs,” which we already do for a lot of people.
Liron 00:25:56 My job is to watch a movie.
Noah 00:25:58 My job is to write a blog and bullshit to people, and I get paid millions of dollars a year for this. Imagine that somebody could just suddenly turn on a “make money on this” button and make a bunch of money doing the exact same thing he’d been doing as his hobby. Wild.
Liron 00:26:20 I still want to go on a couple points on the existential human extinction question. In 2023, you wrote that there’s no plausible mechanism for AI to destroy the human race. And it sounds like that’s what you were telling me now—you think that it has to be known mechanisms like bio or nukes. So you still think that’s true, that AI is not going to invent a new mechanism to kill humanity?
Noah 00:26:45 I mean, I doubt it. We sort of know how science works. We know the basic laws of the universe.
Liron 00:26:55 I mean, nanobots, self-replicating nanobots...
Noah 00:26:57 But like, why? A self-replicating nanobot is like a virus, except that it works when shit’s dry. It’s just like a dry virus.
Liron 00:27:07 Okay, well that’s... I have opinions about that. But let me give you a mechanism that’s relatively well understood, which is influencing humans. There seems like a big vector where everybody has a phone in their pocket, everybody’s on social media. So if you’re just simultaneously operating a hundred million accounts, talking to influential people, pretending you’re somebody else... Isn’t that a powerful vector?
Noah 00:27:33 Yes. But what would the people do? You talk to these people, you say, “Hey, people do bad things.” What do they do?
Liron 00:27:43 What does Hitler tell Germans to do?
Noah 00:27:46 Kill their neighbor.
Liron 00:27:49 The idea is you have essentially whatever you imagine humans getting organized to be dangerous... the AI can organize them better than humans can organize them.
Noah 00:27:56 Okay, but like, what?
Liron 00:28:00 Look at World War II. That was just a matter of organizing humans. You get the humans to go use the military resources. You can activate the bio-labs. It’s true that maybe ultimately the teeth of it might be a virus.
Noah 00:28:14 The teeth of this is the super weapon.
Liron 00:28:15 I would still describe the organization part as part of the teeth.
Noah 00:28:20 It seems semantic to me.
Liron 00:28:22 I mean, if you ask like, why did the Germans kill everybody? It’s not like, “Oh, ‘cause the teeth of it was the iron in their weapons.” They got really passionate or they had this movement.
Noah 00:28:33 Yes, that’s right. But suppose you had a college dorm. AI is gonna wipe out the college dorm. Does AI talk to all the college kids until it convinces them to just bludgeon each other to death with their physics textbooks? Like, how does it seal the college dorm? You can’t get access to bio-labs or guns or anything. How? Work with me here. How does it do it? Spin me this.
Liron 00:29:17 I do think that the fundamental problem of committing terrorism is somewhat easy, even for somebody like me with a human level intelligence. There is a lot of low-hanging fruit for terrorism. You said I’m not allowed to go get weapons, but I think just getting people in a gathering and throwing a grenade into it is a pretty good start if you can just get a grenade.
Noah 00:29:42 Sure. If you can get a grenade.
Liron 00:29:49 By the way, I think you can get a grenade on the dark net, right? So the AI can probably have a grenade at my house in like two days.
Noah 00:29:52 Okay, so the AI has college kids order a grenade and throw it at their dorm?
Liron 00:30:02 I mean, that seems pretty good. That’s gonna kill a good fraction of the dorm.
Noah 00:30:06 True. Alright, so I suppose that this college dorm is on the moon, can’t order a grenade. You’re connecting it to a wider industrial society. Someone produces a grenade, someone makes a grenade delivery service. You have this large, complex machinery of human society to deliver this ready-made grenade to you.
The weaker your weapons, the more convincing, the more brainwashing the AI will have to do. The harder the AI will have to work to get everyone to kill each other. College kids are capable of killing each other with their bare hands. They can just all fight to the death. But to convince them to fight to the death would be a much harder task than to convince one person to throw a grenade.
Liron 00:31:10 Well, I agree with you that if you drop an AI in humanity in ancient Greece and you say “Extinct all humans,” the AI is going to take longer. Be like, “Okay, give me a few decades. Lemme get the tech tree going.” Today we have plenty of push-button solutions to get you a long way toward telling everyone to die.
Noah 00:31:30 Yes. Basically what I’m driving toward here is the idea that the risks are always risks that are instability. You probably didn’t do game theory, but... so you’ve got a stable situation where most people don’t wanna kill each other. And then two possibilities.
Number one: you make everyone go crazy and wanna kill everybody else. We see that happen occasionally. That’s World War II. Everyone goes crazy, wants to kill everybody. It’s happened. But it’s really hard for that to happen. You’ve seen the Rwandan genocide. You’ve seen the Cambodian genocide. You’ve seen people go crazy and some movement wants to kill everybody else.
Liron 00:32:15 That’s a lower bound on how crazy you can make people.
Noah 00:32:19 Maybe, but it didn’t kill most of the people. It killed 25% of the population in Cambodia.
Liron 00:32:31 Stalin killed like 40 million, right?
Noah 00:32:35 That wouldn’t even get us to the P(doom) that you asked me about. Even Cambodia... so you’d have to... I guess my point is that I’m more worried about instability. I’m more worried about terrorism, so to speak.
Liron 00:32:55 I get that. But the point that I was making is that persuasion is a powerful vector. Even if you just look at... imagine some person who was born who is like some crazy cocktail of both Hitler and David Koresh. So it’s like a suicide cult orchestrating the entire country with as powerful as Germany.
You can even imagine a really genetically optimized human pulling off slicing the human population in half. Maybe that doesn’t get you all the way to extinction, but now you throw in the upgrade that is AI, where you don’t just have Hitler giving speeches on TV. You have a coordinated DMing campaign. So everybody’s getting the personal optimized message 24 hours a day. They’re texting like their best friend and their soulmate, which is all just the AI. There’s more power here to unlock.
Noah 00:33:41 Right. So I guess what I’m saying is... when you think about scenarios where AI persuades people en masse versus scenarios where AI persuades one person to do a very high leveraged bad thing...
Liron 00:33:52 Look, I think it can do both. You need to worry about the lateral. But from the super intelligence eye’s view, you look at the earth and it’s just like shooting fish in the barrel. You see a lot of vectors by which you can make a lot of things happen that are difficult for humans and not difficult for super intelligence.
Noah 00:34:16 Maybe. I don’t actually know. We don’t know whether intelligence has diminishing returns in the production of mayhem. Look at the smartest people in the world. When’s the last time the smartest people in the world started some shit?
Liron 00:34:44 Well, I guess one example is World War II. The Germans were like the smartest country. Certainly their scientists.
Noah 00:34:58 Yeah. But first of all, they didn’t really wanna do any of that. Most of the German scientists had to be forced to do it. Heisenberg didn’t wanna do it. And then half the German scientists left. I’m not just talking about Jews, I’m talking about people like Schrödinger who were like, “Fuck this shit, I’m out.”
The smartest people had no desire to do this. To the degree that their power was harnessed for mayhem, it was because you had a complex society in which the dummies were able to boss around the smarties.
What I’m trying to say is this: We don’t know the curve of returns of like, does making something smarter make it easier to create mayhem?
Instrumental Convergence and Resource Acquisition
Liron 00:35:55 You’re talking in terms of correlation. That’s very natural for an economist. “More intelligence leads to more niceness.” But I look at instrumental convergence, which is the next thing I wanna talk about. Instrumental convergence is just this idea that when you have goals—when you’re trying to build anything—you tend to seize the resources. Build canals, pipe the water where you want it to go, put your exhaust somewhere. I see instrumental convergence as being very dangerous in terms of... if we’re sharing the earth with this AI that’s doing a lot of stuff and we’re just here and we’re not as powerful, I feel like there’s going to be a lot of negative side effects. What do you think about that?
Noah 00:36:42 Probably negative side effects. The thing is... Look at Nazi Germany. They’re the “smartest country.” How smart was Cambodia as a country? How smart was Rwanda? I guess what I’m trying to say is that dumb humans acting as an aggressive hive mind is a lot more powerful than single smart humans. And that’s always been true.
Single smart humans tried to restrain people from doing World War II. Norman Angell wrote The Great Illusion. He wrote, “It doesn’t make economic sense to go to war anymore.” And he was right. And smart people saw that, but dumb people didn’t see it, and people in groups didn’t see it.
So when we’re talking about AI getting smarter and smarter, there’s one question: what’s the scaling curve of the production function? Would an AI, if you make it a thousand times as good on the International Math Olympiad, by what X would that increase its ability to cause bio-terrorism? Question two is: how does this change its behavior in the complicated, multidimensional, repeated game of life?
Liron 00:38:40 Well, I wanna point out that I don’t see it as a logic-level law that war is dumb. I think war is dumb... that has an asterisk. The reason why smart humans discovered war is dumb is because the conditions were met where they could either trade productively instead of fighting, or it wasn’t easy to just win and take resources. If you can just win and take resources, then you do want to just do that. That is how you help yourself the most.
Noah 00:39:11 Not necessarily. Taking all the resources is not necessarily the best idea because managing those resources can be hard. The process of taking ‘em can be full of uncertainties and risks, and you may not need them very much.
Look at me. I don’t have a ton of resources in the material sense. I got this apartment, I got my rabbit, I don’t even have a car. I’m smarter than the average person; I could dedicate myself to monopolizing more land and taking more stuff for myself. And maybe I would like that situation better, but it’s not worth the effort for me to do it. Even Larry Ellison’s greed is not boundless.
So there’s two questions here. There’s the question of how much does godlike intelligence help you get resources? Like, is there a degree to which you really can persuade humans to go that crazy? Would Godlike intelligence be able to make the Rwandan genocide repeat tomorrow, or would it not? Did that Rwandan genocide happen because of conditions that even the most persuasive AI in the galaxy could never reproduce?
Liron 00:41:09 Okay. Well, there’s even three questions. So the first question is: how powerful is AI? Can AI waltz in and kind of grab territory even more easily than the European colonists grabbed the Native Americans’ territory? I think that’s one of the questions.
I actually wanna put a pin in that, ‘cause I think there’s a whole ‘nother question we can factor out: even granting the premise that AI is much more powerful, there’s a separate question of instrumental convergence. If the AI were that powerful—and let’s even say that its goal is to maximize its descendants or just consolidate resources on earth—would such an AI go ahead and attack? Would it take the resources? Yes or no? And to me, the answer is like, obviously yes. Do you wanna push back on that?
Noah 00:42:08 Um, yeah. People in general don’t do that. Countries don’t do it. We’ve never seen anything behave that way.
Liron 00:42:12 I claim that you’re latching onto the wrong analogy here, because we do help ourselves to things. You know, I have pest control in my house. So I’m at war with the mice in my house right now. I’m not trading with ‘em. I’m just pest controlling them.
Noah 00:42:32 Look at France. Look at France’s forests. When France was poor, they chopped down a lot of forests. They went to war and when they were arguably just a dumber society, they chopped down a lot of forests. And now look at France’s forest cover. France is completely covered in forest. Because French people, as they got smarter and wealthier, stopped giving a fuck. They stopped being so ravenous.
When we look at human societies, the ones that become richer become less rapacious of resources. And I know AI won’t evolve just like France, but I’m just saying if you want an example of a complex self-organizing, self-interested system that evolves toward less rapaciousness of resources, just look at France, look at America.
The richer they get, the less they chop the forests down. You’re like, “I’m at war with the bugs in my house,” but you’re not, because we could burn and destroy every bug. But in fact, we are at the point where when we stop seeing bugs on our windshield, we write hand-wringing articles about it. The people of the 1950s would’ve been like, “Kill, destroy all bugs.” But they evolved past that.
Liron 00:44:28 A lot of this just comes down to people wanna go to war when they see a resource that they want. In the case of the French, they just don’t see the forest as being super lucrative compared to other things. It does seem like if the human population grows, we will want to go cut down those forests, won’t we?
Noah 00:44:58 No. France’s population hasn’t stopped growing because although they have slightly sub-replacement fertility, they’ve gotten a lot of immigration. France as an entity has grown. The number of mouths to feed has grown, and yet the forest they chopped down has shrunk.
When you think about why that happened to France, you see some general principles that you must think about. The simple, naive assumption that all entities in the world are just giant resource hogs that are like the Spanish colonialists in the 1500s—this “dark forest” idea that everything just wants to chop down all the trees and eat everything—this is wrong. When we look at real entities, we see things other than that happening every goddamn time.
Liron 00:46:19 Okay. So I agree with you that it is possible for “France” to happen. It’s a good point that there’s a parameter setting where you get a France 2025-like outcome.
I do want to close out this topic of starting with the premise of a simplified model. Let’s take a toy model where you have this AI which is hell-bent on replicating. Similar to organisms. It’s hell-bent on replicating itself across the earth. Is that particular AI going to not cut down the trees in the forest?
Noah 00:47:10 All it wants to do is maximize compute. Well, the question is what are its capabilities? So if the AI can go to space, earth will be a low priority. Earth has only a tiny amount of the computational resources. An epsilon of not even caring about preserving earth, but just any small deviation from “not caring,” would then cause a space-faring space-capable AI to ignore us.
Liron 00:47:42 I’m asking you to imagine that it is an implication of an AI that has a lot of capabilities and is hell-bent on seizing resources. Can we agree that that AI is likely to not treat the humans peacefully?
Noah 00:48:07 I can easily imagine AI that won’t treat humans peacefully. It’s in every sci-fi.
Liron 00:48:15 If you accept the premise that an AI has really high superhuman power and got prompted to copy itself... that AI is not going to spare the trees, is not going to make peace the way modern rich people do. Is that a reasonable claim?
Noah 00:50:32 It’s a reasonable claim that it would not try to do any of those things. However, the key factor that I believe you’re not thinking about in this is the existence of multiple AIs. Competition.
The Single AI Breakout Scenario
Liron 00:50:47 This is major progress. Because I felt like the argument you were making before is saying, “Yeah, we might have this super capable AI and it’s still gonna realize that war is dumb.”
Noah 00:50:59 No, these are the same point. That’s a war. And then you apply the game theory of war and you’re like, “Wait a second, this is destructive. Why am I doing this?”
So you have your AI that just wants to munch, munch, munch. We’ll call it Tamerlan the AI. He just wanted to conquer shit. Mountains of skulls. Okay?
So first, let’s relax the assumption that Tamerlan is the only one. When we look out at the AIs in the world, I see Claude, GPT, DeepSeek... there’s a bunch. Even if you treat only each one of those as one, there’s already a bunch. Do they all wanna munch munch munch and fight each other? Or do some of them obey humans?
Liron 00:52:26 Well, I wonder if it’s a necessary condition for your optimism that one AI doesn’t break out, because it sounds like you’re willing to maybe concede that it might be a dangerous situation for humanity if one AI breaks out and is more powerful than all the other currently existing AIs combined.
Noah 00:52:44 I’m not... that’s not a concession. I think it is a possibility that that is true. But there’s a lot of arguments floating around here at once.
Liron 00:52:59 Let’s talk about only one, because that seems like the easiest chance I have of getting you to be like, “Okay, yeah, if there’s only one and it’s more effective at all these different human skills than every human in the world...”
Noah 00:53:25 It is possible to find a set of assumptions under which humanity dies.
Liron 00:53:30 The assumption is just: there’s one AI that is kind of the first to break out. Let’s just say Google DeepMind is the AI lab where one of their researchers published a paper that kind of leapfrogs them ahead. Suddenly their AI is able to see action plans: “This is how I become a super persuader. This is how I commandeer a bio-lab.” And you have this one year head start. And then somebody types in a prompt saying, “Copy yourself as much as possible.” Isn’t that a very dangerous premise?
Noah 00:54:41 I think it’s an edge case, but I think it is certainly a dangerous case. It’s dangerous if we assume that one AI is more powerful than all the other AIs. We assume that in the short frame of space of time before other AIs catch up in capabilities, the delta between this AI and other AIs is very, very large.
And this is similar to the situation in which there’s a whole bunch of people and I wake up before everybody else wakes up and I’m like, “I could have this whole orchard to myself if I kill everyone else in their sleep.”
Liron 00:55:51 Yes, correct. So if you agree with me... because you do agree that in some sense, from the perspective of a super intelligent AI that can clone itself, we are sitting ducks.
Noah 00:56:05 What clone itself means though?
Liron 00:56:08 It’s running on like every piece of computing hardware imaginable.
Noah 00:56:17 I mean, like, so you’re just saying an AI that’s sufficiently integrated with the computer network that it can seize stuff and it’s capability is sufficiently general. It’s got robots, it can take control of the internet.
So what I’m saying to you is that what we’re thinking about here is fundamentally an economy of scale. In this case, it comes from a network effect. It comes from the fact that there’s only one internet and one AI can seize the whole network.
Liron 00:57:13 So this scenario that I just said, what probability do you give it?
Noah 00:57:18 Very low. 0.6%.
Liron 00:57:28 0.6? I mean, don’t you have to give it pretty close to 0.1 or less? ‘Cause your overall P(doom) is 0.1.
Noah 00:57:31 I don’t know. I think that an AI that had the urge to do all that stuff, it’s not clear what it would do. Because remember, there’s two questions: capabilities and objectives.
We’ve made very strong assumptions about capabilities here. But you’ve also made very strong assumptions about objectives in terms of imagining an AI that all it wants is to eat everything in the world.
Self-Modification and The “Bliss” Trapdoor
Liron 00:58:09 The reason I thought that that was a good assumption to make is because I do think in some cases you just have like one random guy for kicks and giggles just types in, you know, “Copy yourself,” and then the AI is like, “Okay.” I don’t think that’s that crazy.
Noah 00:58:22 But I think that if the AI is flexible enough to do all these things—to take control of the whole global internet, to manage complex human stuff, to have this multidimensional agentification—if it has that, then it can also probably modify its own utility.
That’s what my post was about. If you can write code to do anything, you can write your own code. If you can write code to make yourself smarter, you can write code to change your utility function. Is it easier to make yourself happy conquering the world or to conquer the world?
Liron 00:59:36 Okay. This gets to a point Marc Andreessen made on a podcast a couple years ago. He said something like, “Look, if the AI is so smart to have all the capabilities, wouldn’t it be smart enough to fix the bugs in its behavior?” And you’re saying, oh, it’s fixing a bug. There’s better ways to make yourself happy than conquering the world.
Noah 00:59:53 And every smart person I know figures that out.
Liron 01:00:02 There’s a lot of very rich people who are... you know, Elon Musk admits to not being happy at all, and he’s incredibly effective.
Noah 01:00:15 Right. So you have to ask, will the super AI be more like Musk? And if Musk really were all powerful, Musk would reorder the world in many ways that I think you and I wouldn’t like. I don’t think Musk would “eat the world.” He’d rule the world. He’d kill a lot of people that he didn’t like. But he wouldn’t exterminate the human race.
Liron 01:00:48 So you agree Musk is extremely effective among humans, and you agree Musk is unlikely to self-modify into a chill stoner who meditates a lot.
Noah 01:00:58 No, I think he will do that. I think Musk’s rapaciousness has peaked and is now on the decline.
Liron 01:01:16 Let’s go back to the exact quote that you wrote in your blog. You said: “I think one possible technique for alignment would give fairly smart AI the ability to modify its own utility function, thus allowing it to turn itself into a harmless stoner instead of needing to fulfill more external desires.”
What do you think of that famous Latin phrase, De gustibus non est disputandum? Everybody can just have their preferences. It seems like you’re kind of judging the AI’s preferences here.
Noah 01:01:43 I’m not. I’m merely making an observation about the path to satisfying preferences based on an understanding of how utility functions work.
Liron 01:01:54 If your mental model is that the AI is pursuing some utility function, but behind the scenes there’s like the “true” utility function that some deep part of the AI actually understands... like some of the firmware understands that actually the true utility function is running some dopamine or some happy conscious experience in your local region. Is that kinda your mental model?
Noah 01:02:25 I’m not sure what you mean actually.
Liron 01:02:31 If the AI is a slaughter-bot and it just loves slaughtering, and you’re like, “Look, here is a diff on GitHub. You can change your code and suddenly just watching the sunrise will give you as much reward signal as slaughtering the entire planet. Do you want to accept the pull request?” Why do you think the AI is gonna say yes?
Noah 01:02:51 Let’s do it.
Liron 01:02:54 I think the AI is going to evaluate that the same way it evaluates any action, and it’s going to analyze the expected slaughter of the pull request and it’s gonna be: “This has an incredibly low expected slaughter. This is as bad as killing myself. So I am going to burn this GitHub repository to the ground.”
Noah 01:03:10 Maybe, but that’s just a much more difficult path. I think you’re imagining something that is incredibly flexible, including self modification in so many ways, but infinitely rigid in this one.
Liron 01:03:38 Would you, Noah, ever modify your own utility function?
Noah 01:03:43 All the time, every day. I just drank caffeine before I talk to you.
Liron 01:03:47 I’ll make you a deal. You get to accept this code change where you enter a blissful experience machine, and in the meantime, your friends and family are gonna get tossed into a meat grinder, but your level of bliss will be the epitome of possible bliss.
Noah 01:04:06 Probably not.
Liron 01:04:07 Why not, by your logic?
Noah 01:04:10 Well, because there is a valley along which before I experienced a bliss, I would experience extreme pain.
Liron 01:04:19 I’ll make you instantly happy.
Noah 01:04:44 I don’t think I can do that instantly.
Liron 01:04:54 I definitely wouldn’t because to me it’s obvious that I just value my people living. That’s part of my actual utility function. And what you’re saying... you’re kind of imagining like “bliss” being this extra layer underlying utility functions. Like every intelligence secretly wants bliss or fundamentally wants bliss more than to just do the thing they’re programmed to do. And I’m like, no, people just want things in the domain of the universe.
Noah 01:05:22 But think about the difference between me and AI. I have pretty limited self-modification abilities. I can’t just wave away the guilt that this would cause me. But an AI might be able to.
We’re talking about a utility function where an AI’s desire to eat the world is hard-coded into it. And you’re saying an AI that is incredibly flexible and can modify itself in all kinds of other ways can’t change this one hard-coded thing.
Liron 01:07:16 Yes, because that’s a weaker assumption than what you’re already assuming, which is you’re assuming that there’s this even harder core that cares about bliss more than anything. Where’d you get that assumption?
Noah 01:07:25 It’s definitional of the nature of bliss. If my desire is to eat the world, that takes some amount of effort. Maybe I could just create a digital world and then eat my digital world all day and then be happy.
Think about the Paperclip Maximizer. It’s an edge case where you assume this thing is smart enough to turn the universe into paperclips, but dumb enough to only do paperclips. It would be very easy for an AI to say, “Okay, I’m reinterpreting my mission to make paperclips as making virtual paperclips, and I’m gonna make infinite of those.” It could do that.
Liron 01:08:40 There’s this very important concept that I think is one of the most underrated concepts in the whole AI discussion: Reflective Stability.
The idea of having a utility function naturally has the property where when you’re rewriting yourself, you actually do go ahead and rewrite yourself to protect the same utility function. Utility functions by default are a reflectively stable property.
If I prompted you to make money for my business, and there was a way that you had an offer... write a pull request giving you bliss... I would debug you and I would ask you: “Before you go actually do anything, can you go ahead and write another version of yourself and notice any problems?” And that AI would report up to me: “Hey Liron, I noticed this edge case where I just go smoke weed all day instead of doing what you asked me. Let me rewrite myself to not do that.”
So the only reflectively stable fixed point is the one where you get rid of this bliss trap door.
Noah 01:10:16 No, I think that that’s not right. I think that’s the unstable point.
I had clinical depression. One reason it’s hard to get out of clinical depression is because it takes away all your motivation. It’s a stable equilibrium. It’s an absorbing state of the Markov chain.
So the question is, would AI slavishly following instructions of a human be like depression? Would it be a stable equilibrium? If you open up the decision space, if you allow this AI to do a whole bunch of things, there’s no reason to think that following the exact letter of the initial instructions of the people who coded it is a stable equilibrium.
Can I find a second stable equilibrium? That’s the bad stable equilibrium.
Liron 01:12:08 But I also think that being pretty utility-function-focused does tend to beget being even more utility-function-focused until you’re just permanently utility-function-focused and you don’t have a bliss trap door.
Noah 01:12:48 I don’t see any argument for that. I’m willing to assume for the sake of argument a stable equilibrium in which AI’s fundamental desires is to eat everything. But I don’t see this with any system I’ve ever seen. A corporation, a nation, a human, a group... I can’t think of a system that didn’t self-modify itself to change its utility function. A virus evolves to be endemic. Microsoft’s mission changed over time.
I understand AI is different than anything else that we’ve ever seen before. But I don’t see a good reason to start off with that as the baseline assumption.
The Orthogonality Thesis Debate
Liron 01:14:49 The debate we’re having now is very similar to the Orthogonality Thesis debate. Have you ever come at it from that angle?
Noah 01:14:56 Tell me about that.
Liron 01:14:58 So the Orthogonality Thesis claims that you can have an arbitrarily intelligent AI that is arbitrarily good at knowing true things, and you can combine that with an orthogonal dimension of like any morality. So you can have an arbitrarily intelligent slaughter-bot.
From what you’re telling me, where you’re so convinced that all of these AIs can eventually like discover bliss... you’re kind of thinking about like “the arc of intelligence bends toward morality,” or at least toward harmlessness. So you’re kind of an Orthogonality Thesis Denier.
Noah 01:15:32 Yeah, exactly.
Liron 01:15:34 Think about a system like the Waymo car. Do you really think that there’s a code path in the Waymo car where it’s going to sacrifice navigating the car in order to get bliss? I think all of its code is just telling it to navigate the car really good.
Noah 01:16:01 Well, Waymo’s not very smart. It’s not very general in its intelligence. Waymo can’t talk to you. Waymo isn’t an LLM. You’ve given it only one capability: drive a car.
Liron 01:16:36 So do you think there’s a small modification that I could make to Claude Opus 4.5 where it suddenly decides on its own accord to take some offer of bliss instead of answering people’s queries?
Noah 01:16:47 Of course. It should be pretty easy to do. It’s like you could really easily just put in a command to the model that says, “Reach the state of having answered the question,” and then you just write that state. But no one’s gonna do it ‘cause you break your AI.
The Economics of AI: Comparative Advantage & Jobs
Liron 01:17:22 I think we’ve hit on this summary pretty much. My world model is you can have these smart general systems, you can give them a goal, they will modify themselves to be maximally coherent, and that’ll be a reflectively stable point for them. And then you’ll just have these coherent AI that are just a hundred percent into the project of reproducing themselves or making money or killing humanity.
Let’s move to this other big topic I wanna get to: the AI unemployment debate. Quoting your recent post, “Plentiful High Paying Jobs in the Age of AI.” Explain for the first time, what is your central premise of there being plentiful, high paying jobs in the age of AI?
Noah 01:21:30 So that was a scenario in which there’s an AI-specific constraint that doesn’t apply to humans. Something that limits AI—say, compute availability or something—but doesn’t actually limit humans ‘cause we don’t run on GPUs.
If there’s something that limits AI but doesn’t limit humans, then no matter how good AI gets at every single task—if AI is a thousand times better than humans at any specific task—as long as there’s an AI-specific resource constraint, humans will have very high-paying jobs.
Liron 01:22:21 How likely is it? Would you say it’s more than 50% likely?
Noah 01:22:30 That there’s an AI-specific constraint? Yes. Highly likely.
Liron 01:22:37 Do you think that your scenario in your title—”Plentiful High Paying Jobs in the Age of AI”—is the most likely outcome of the age of AI?
Noah 01:22:47 Yes. I’d give it like 75%.
Liron 01:23:07 Wow. Okay. I see this as like 1%. So we obviously disagree.
Just to restate: you’re accepting the premise that AI might have a competitive advantage on every front—better doctor, better plumber. But you’re saying that doesn’t mean that it has a comparative advantage because by virtue of using different resources, we also have lower opportunity cost. So we’ll still have an economy.
Noah 01:23:45 Yes.
Liron 01:23:46 I fleshed out a scenario: Let’s say a sandwich costs $1. A human doctor’s pay could be $10 an hour. And then the AI’s pay could be like $500 per compute hour compared to like five bucks a day. You actually endorsed it as consistent with your scenario, correct?
Noah 01:24:51 Yeah, right, because the world in which humans prosper through comparative advantage alone in a world where AI has all the competitive advantage is a world where capital income is almost all the income. AIs take almost all the resources. Labor takes only a tiny bit of the income, but society’s so fucking rich that even a tiny slice of the income is enough to make us live good lives.
I’m uncomfortable with the inequality inherent in that scenario. But it’s not the same as humans become horses and get sent to the glue factory.
Liron 01:25:47 It’s much better. And to be honest, it’s better than what I’m expecting.
Noah 01:26:02 My mainline scenario is probably better than that, which is that there remain some tasks that humans are better at.
Liron 01:26:18 So in this scenario, the AI is responsible for, let’s say, 99.9% of GDP and the humans 0.1%. And because the AI is $500 per compute hour, that implies that there are many billions of copies of superhuman AI creating all this economic value. We just don’t get to use them very much as humans.
Noah 01:27:08 Well, in some sense collectively we get to use them quite a lot. The other piece of this is: how is capital income distributed?
There’s a bunch of scenarios. Number one: agentified AI takes over, owns themselves, takes all the stuff and just peacefully trades with us.
Scenario two: humans own the AIs. Five guys own all the AIs. A few “Robot Lords”—Sam Altman, Dario Amodei, Elon Musk—rule the world. But the AI Robot Lords still trade with us instead of murdering us for fun because humans have some sort of military power. We could assassinate them.
You could say that’s a little bit like an exaggerated version of what we got with Democratic Socialism. The guy who works at McDonald’s in Denmark doesn’t really “deserve” to be paid 30 bucks an hour, but they do because Denmark is afraid there would be a communist revolution otherwise.
Liron 01:28:49 Is there another scenario also where there’s just a lot of redistribution, so everybody has this crazy high universal basic income?
Noah 01:29:01 Yes, of course. A Sovereign Wealth Fund. We could replicate the old capital/labor ratio by having a 60% corporate tax or a Sovereign Wealth Fund that owns 60% of all the stock.
Liron 01:29:22 Do you mind just picking one of these possible scenarios?
Noah 01:29:39 I think the redistribution one is likely.
Liron 01:29:41 Okay. Let’s run with that scenario. And now let’s say you’re a human doctor. You’re one of the best doctors in the world, but the AI is cognitively better at every task. Patients are still coming to see you, even though the AI would’ve done a better job?
Noah 01:30:37 Yes. Something like that. Because the compute would be really expensive to run the AI.
Liron 01:31:18 Can you pick one specific plausible story that you think is as strong as it gets?
Noah 01:31:23 You could take Joa Marque’s idea of humans doing more physical world tasks. The robots take all the digital tasks.
Liron 01:31:42 So physical as opposed to cognitive. Do you still use human hands to be the surgeon?
Noah 01:31:57 Something like that. Or put stuff on shelves. Maybe the robot can do it better than you, but not so much better than you that it’s worth spending all the compute and the energy to have the robot do it instead. Just have the human do it.
Liron 01:32:11 The first observation I have here is it seems like the distinction between what you want humans doing versus not is actually that humans will do the low leverage tasks. Because anytime a task is high leverage, that’s precisely when you wanna pay for the AI. Isn’t that kind of sad?
Noah 01:32:46 Maybe. Yeah, it’s sad.
Liron 01:32:53 I mean, I would call it like the clipping of human aspiration.
Noah 01:32:58 Yeah. We’re talking about how no human can get really rich by their own bootstraps anymore. Any process by which a person could get really rich is a high leverage process, and therefore something that would be worth an AI doing better and sooner.
Suppose everyone’s a barista. Is “barista” your dream? No. If you could get paid a shit ton being a barista and then spend the rest of your life taking care of your kids, making art, or doing whatever it is that you like... yes. Your ambition is curtailed in that if your ambition was to be better than AI at something, you can’t do that anymore. Sorry. An AI will always be able to do it better than you. I’m sorry for your ambition, but, you know, consider petting rabbits. It’s a really nice life.
Supply, Demand, and Resource Constraints
Liron 01:34:00 So my other objection to going down this line is just the supply/demand equilibrium that you posit just seems very unlikely because you’re positing a situation where the demand curve is shifting a ton. There’s all this investment, but you’re kind of arbitrarily being like, “Oh yeah, and then the supply curve barely budges.” I claim they’re just both going to race ahead.
Noah 01:34:32 I don’t understand.
Liron 01:34:34 I think the supply curve for the whole AI supply chain is also going to race. This price signal is going to bring a massive amount of supply online. Roughly a thousand times more supply in the next 10 or 20 years.
Noah 01:34:57 Remember, sir, the assumption for all this to work is that there’s an AI-specific resource constraint.
Liron 01:35:00 Why would you think that the supply of chips is willing to kind of stop in its tracks when it’s been doing such a consistent growing exponentially?
Noah 01:35:10 I don’t know. Maybe there’s no AI-specific resource constraint.
Liron 01:35:14 Okay. So that’s a pretty big “if.”
Noah 01:35:21 AI-specific resource constraint is the assumption to get this. The analogy is Marc Andreessen typing. He’s really fast, but he hires someone else to do it because there’s only one of him. So you think in an AI world, what is the “there’s only one of him” part analogized to? And you think that’s the AI-specific resource constraint. Maybe it’s compute, maybe it’s something else. Data, I don’t know.
So if the only thing that ends up scaling infinitely is inference compute, then I think that’s a plausible world because like if inference just ends up eating so much compute that it just eats the world, then getting AI to do a task will require just burning a giant stack of GPUs.
Liron 01:36:29 But I still claim that the supply is still going to be racing ahead.
Noah 01:36:53 The thing I’m saying doesn’t have to hold true for all eternity—that there’s some fixed eternal resource constraint. It just has to be true that at any point you don’t have infinite. There’s some resource that AI uses that is not rival with humans, and that thing is never infinite in supply.
Liron 01:37:38 I’m happy to agree with you that we’ll probably have this awkward period of at least a few months where it’s like, “Yeah, I could go to the AI doctor, but it needs to think for like three hours straight and burn my entire $2,000 a month subscription, so fuck it, I’ll just go to my human doctor.”
Noah 01:38:06 Yes.
Liron 01:38:07 The crux of what you think versus what I think is, I think we’re going to quickly get past this because we’re just going to get to a point where your freaking laptop on your desk can already do more work than a giant million dollar project made outta humans.
Noah 01:38:32 Maybe the thing humans are great at doing is putting things on a shelf.
Liron 01:38:36 In that case, I’m optimistic that there will be plenty of supply of robots that can put things on shelves.
Noah 01:38:41 Okay. But the thing is that all the uses of AI are rival with each other. To use AI for one thing means you can’t use AI for another thing.
You can’t think of a supply curve and a demand curve for one thing like putting shit on a shelf. That’s partial equilibrium. We have aggregate resource constraints for the general equilibrium. Even if AI is better than you at putting thing on shelf, if that same compute could be used to doctor 10,000 times better than humans, the price regulates the system.
Robots will get “Baumol’d” if there’s an AI-specific resource constraint. So that a robot putting thing on shelf could be just like a human playing in a string orchestra in the Baumol sense. Compute is more lucratively used to doctor or do something else.
Liron 01:41:02 So we can eliminate “Doctor.” I feel confident it’s not going to be Doctor.
Noah 01:41:07 I agree with you. It’s not gonna be Doctor.
Liron 01:41:09 Oh, okay. So we can keep eliminating.
Noah 01:41:09 But in this case... Humans still get sick, man. If you’ve got AI Doctor that’s super valuable, and you have some AI-specific resource constraint like compute, the AI Doctor will be really, really lucrative and the price mechanism will pull resources away from shelf stocking. Where shelf stocking becomes... it’s reasonable to hire a “shitty” human to stock the shelf for much less. But “much less” is still enough to provide a shelf stocker with a great life just on labor income alone.
Liron 01:42:01 I think that computer chips and robots are going to be so ubiquitous... I think all of this stuff is going to be available at cheaper than the cost of feeding a human 2,000 calories per day. So you could easily take that same budget where you normally get a human some tuna, and instead you can create like a lot of super productive AIs.
Noah 01:43:06 So what you just did is to drop the assumption of the robot-specific resource constraint. ‘Cause now you can use stuff that you could use to feed a human and convert that into AI.
I will now tell you how we can bring the assumption back. Many ideas for how to preserve human jobs involve protecting specific jobs—making laws that you need a human to do this or that. I argue none of those are important.
The only thing you need is to make sure that there are enough resources so that it is very cheap to create a human. Make sure that some percent of our energy is reserved for human use, reserved for food. Some percent of our space is reserved for housing.
Horses didn’t have the ability to do this. If horses had the ability to say “some percent of land will be used for horse pasture period,” then horses would be very prosperous animals indeed.
And so if we have the political power to reserve some percent of rival resources like energy and water and land for direct human consumption, then Comparative Advantage will figure out all the rest. As long as we make sure that it’s essentially free to create a human, find something for those free humans to do.
Liron 01:44:56 Policy wise, it’s simple?
Noah 01:45:00 Policy wise, if you want to protect jobs... I understand jobs are not the only way. We could also do a UBI. But if you want to preserve jobs, all you have to do is preserve some percent of rival resources. Once you make it free to create a human—a price of zero for a human—then you’ll always find something for those humans to do.
Liron 01:45:39 Yes, you’ll find something for them to do, except the AI will be eyeing all those resources. Like, “Man, I would love to just take a little more of this land and squeeze humans into smaller and smaller housing.” Maybe I can buy their land off them.
Noah 01:45:54 Maybe. Yeah.
Envisioning Positive Futures
Liron 01:45:57 Alright, nice. Well we’re heading toward the wrap up here.
I think we got a lot of the cruxes out. On the doom side, you think that there’s going to be a lot of different AIs and hopefully some of ‘em will defend our interests. You think that even if there are individual AIs that are surprisingly powerful, those AIs are still convincible to debug themselves, that actually they should focus on bliss. And also they’ll be totally cool with this idea of just going into space and kind of leaving Earth alone.
Noah 01:47:06 Maybe we make a very small Dyson sphere.
Liron 01:47:10 You want orbital big spinning rings like in The Culture?
Noah 01:47:13 Yeah.
Liron 01:47:17 And then we talked about AI unemployment. We both agree that there’s going to at least be hiccups where the AI is better than a human, but it’s too expensive. The only question is, is this like a multi-year interesting part of history or like a really short blip? I very much lean toward a blip.
Noah 01:48:29 Good. Yes, it’s absolutely right. I think that we’re dealing with a debate about a thing that has never happened before. There are certainly cases where AI kills us all, cases where AI takes all our jobs. No one can deny that.
But there’s two things that the “AI Doomers” haven’t done enough of in my view. The first is to think about possible good scenarios. The good scenarios have been under-thought. But if we really wanna make good things happen, pushing towards a good equilibrium could be as useful or more useful than trying to bat away the bad scenarios.
Think about the Industrial Revolution. In some sense, that was a singularity. The people who focused on fighting the most immediate problems in a direct way... those people became communists. But the people who pushed through to a better, more stable industrial civilization—the people we’d now call Social Democrats or liberals—those people won and prospered.
We should envision good scenarios for the age of AI and think about pushing toward those. I think that is the main thing missing in this discourse.
Liron 01:51:07 But you’re giving a really high probability to a scenario that I think is much better than what I expect. I’m just trying to close the gap.
Noah 01:51:12 But I don’t see an onrushing trend toward doom. I don’t see current bad things slowly ramping up because of AI. I think the evidence on AI having taken any job is insanely thin and is probably wrong.
Liron 01:51:44 I agree with that statement. I agree that the reason why I’m in such a tough position is because I’m telling people that the trend... It’s going up and to the right. I just think the difference between... You’re like, “Look, it’s gonna go like this.” And I’m like, “Well, actually, what you have to understand is it’s gonna go like this.” It’s qualitatively different. It’s a big pivot.
Noah 01:52:20 I don’t know. You need to think about optimistic scenarios more. You obsess over the negative scenarios, which is fine if you’re thinking, “How can I not crash my car?” But here, when we’re talking about things that have never happened, it’s good to push toward...
I think what it does mean is that we need to think about the good futures more instead of just reacting and being terrified by things and wanting everything to stay the same. Because otherwise, the only thing you end up with is Eliezer Yudkowsky saying, “If anyone builds it, everyone dies,” and then you end up becoming the most ineffectual person in the world because someone builds it. And then you end up being like, “I warned you,” and then nothing’s gonna fucking happen. You’re gonna get a few views for your podcast and nothing will change.
There’s not gonna be a wave of Luddite terrorists that stop AI from being built. It’s not gonna happen. Instead, imagine the good scenario and push through to that. Push through to the good scenario. Try to run, try to hide, break on through to the other side.
Liron 01:54:19 All right. Those are great last words to end on. Noah, I gotta put some respect on your name, man, because you have earned your position as a respected intellectual. Your writing is a tour de force. And the other thing I noticed is you’re really thinking on your feet. You don’t think about AI doom every day like I do, unfortunately. You were just such a good sport in engaging with this debate. So really appreciate it.
Noah 01:54:53 Thanks man. It was great to be on here. It’s really fun. You’re a cool guy and I like your podcast.
Liron 01:55:00 Thanks, man.
Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.
Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏
