Noah Smith is an economist and author of Noahpinion, one of the most popular Substacks in the world.
He returns to Doom Debates to share a massive update to his P(Doom), and things get a little heated.
Timestamps
00:00:00 — Cold Open
00:00:41 — Welcome Back Noah Smith!
00:01:40 — Noah's P(Doom) Update
00:03:57 — The Chatbot-Genie-God Framework
00:05:14 — What's Your P(Doom)™
00:09:59 — Unpacking Noah's Update
00:16:56 — Why Incidents of Rogue AI Lower P(Doom)
00:20:04 — Noah's Mainline Doom Scenario: Much Worse Than COVID-19
00:23:29 — Society Responds After Growing Pains
00:29:25 — Agentic AI Contributed to Noah's Position
00:31:35 — Should Yudkowsky Get Bayesian Credit?
00:33:59 — Are We Communicating the Right Way with Policymakers?
00:40:16 — Finding Common Ground on AI Policy
00:47:07 — Wrap-Up: People Need to Be More Scared
Links
Doom Debate with Noah Smith — Part 1:
Noah’s Twitter — https://x.com/noahpinion
Noah’s Substack — https://noahpinion.blog
“Plentiful, high-paying jobs in the age of AI” —
Updated thoughts on AI risk
Transcript
Cold Open
Noah Smith 00:00:00
Your record and Yudkowsky’s record in getting people to actually do policy to increase AI safety is nil. It is pathetic. It is nothing. You have failed and failed and failed because you’re treating communications to Washington, D.C., to policymakers, to empowered people, the way you would treat a LessWrong discussion. And you cannot do that. It anti-works.
Liron Shapira 00:00:25
Are you personally happy that I’m failing?
Noah 00:00:26
You are not examining why you’re failing. You are not taking a hard look at why you’re failing, and you are not recalibrating to try very hard to not fail.
Welcome Back Noah Smith!
Liron 00:00:41
Welcome to Doom Debates. Noah Smith, my returning champion, is a highly influential writer and economist best known for his blog, Noahpinion, one of the most popular Substacks in the world with a whopping 441,000 subscribers.
Recently, he’s been writing about AI as well and even getting into the AI doom debate. In our last episode, which I highly recommend from December 2025, we focused on a couple major disagreements.
Would an AI that isn’t programmed to be good modify itself to be good or at least non-threatening? We definitely disagreed on that. I recommend checking that out. And then if we do get vastly superhuman AGI, should humans still expect to have plentiful high-paying jobs? Noah says yes. I said no. I recommend checking that out.
But that’s not why we’re here today. Today, we’re going to focus on Noah’s P(Doom) because he’s had a major update, and he’s granted Doom Debates the interview to explain why his P(Doom) has changed. So Noah Smith, welcome back to Doom Debates.
Noah 00:01:39
Hey, thank you.
Noah’s P(Doom) Update
Liron 00:01:40
So this is breaking news because it’s so rare to have somebody who’s a prominent intellectual thinking about AI doom and then have what I see as a massive update. It’s just so rare for people to change their minds. So congrats to that. One of the things in rationalist culture is you do want to celebrate when somebody changes their mind as opposed to when somebody doubles down and becomes more extreme. So I celebrate you, and I’d like to hear more.
Noah 00:02:06
All right. So basically, when I was thinking about P(Doom) before, I was thinking about primarily the danger from long-term superintelligence. I think it’s unlikely that superintelligence will kill us, basically because it has so little reason to kill us. It just didn’t make sense to me that it would kill us.
And when I look at what super forecasters say and forecasters in AI say, they don’t actually focus on that. There’s a few people like Yudkowsky who think that superintelligence will inevitably kill us because it’ll want to just eat the world. And then most people think it’s not very likely that superintelligence will kill us — extremely unlikely.
So I was thinking only about that. I was thinking only about: assume you have a computer god, and the computer god is in some way fundamentally alien. How likely is it that that computer god will kill us? And all the arguments that it’s likely to kill us seem just patently wrong.
And so that’s why my P(Doom) was pretty low. But then when I was thinking about agents and reading other people’s forecasts, I was thinking, “I’m not thinking about the right cases for doom. I’m not thinking about the likeliest doom cases. I’m focusing on what Eliezer Yudkowsky told me to think about and Nick Bostrom told me to think about and thinking about paperclip maximizers and things like that.”
I’m not thinking about real, more realistic doom things. And now I think I’m thinking about more realistic doom things.
The Chatbot-Genie-God Framework
Noah 00:03:57
Interestingly, I talked to the economist Chad Jones about this too. He said he’d been thinking about the danger of alien superintelligence for a while. He even wrote about it in his paper on AI economics recently. Everyone should read Chad Jones. He is truly the Chad.
And so he said initially he thought that the possible really good stuff and possible really bad stuff would arrive at the same time. Now he realizes the possible bad stuff will arrive before the possible really good stuff arrives. And I take that to mean something almost identical to what I’m thinking, which is that having seen what agents can do, I think it’s a lot more likely that extreme intelligence and autonomy arrive before volition, consistency, and a stable objective function.
So I think extremely powerful AI will be more like a jinn or a genie before it becomes like a god. And that’s why my P(Doom) has risen because a genie is more dangerous than a god.
Liron 00:05:08
Okay, so it goes chatbot and then genie and then God.
Noah 00:05:13
Chatbot, genie, God. Yes.
What’s Your P(Doom)™
Liron 00:05:14
And I just want to make sure that we’ve got the numbers accurate here. ♪ P(Doom), P(Doom), what’s your P(Doom)? What’s your P(Doom)? What’s your P(Doom)? ♪
So in your last episode with me a couple months ago, you said that your P(Doom) is 0.1%, as in the risk of total extinction, because we talked about different senses of doom.
Noah 00:05:34
Right.
Liron 00:05:35
What’s your P(Doom) now?
Noah 00:05:36
Maybe civilizational collapse would have been one percent or something. Humans are — it’s hard to kill us off entirely. It’s rare that species like us go extinct, but civilizational collapse maybe would have been one percent. I think if we’re talking about long-term superintelligence, like machine god kind of thing, that’s still pretty much where I am.
It’s easier to bring down civilization than to kill off the species. But it’s small enough where it feels small. And Daniel Kahneman and many others have showed that humans aren’t very good at estimating small probabilities, but it feels small.
But the genie type of AI — the risk of extinction or civilizational collapse from that does not seem small to me. It seems like something more robust, larger. And the reason is because I know that humans are unaligned, and I know that humans are vulnerable to things like bio plagues, genetically engineered viruses that could be powerful enough to collapse civilization.
I know it’s happened before, it could happen again. We see Native Americans with smallpox, we saw the Black Death. We just saw what happened with COVID, which had a one percent death rate. Imagine COVID with a longer asymptomatic period and a ninety percent death rate. That’s game over.
So that’s very realistic to me, and the idea that people could use one of these genies to just say, “Okay, jailbroken Claude Code, go create the new virus and release it.” I think that is non-trivial. It’s not small. It’s large enough where I can kind of imagine it easily.
Liron 00:07:19
Yeah. I think you’re throwing around the number ten percent roughly to describe that.
Noah 00:07:21
Yeah, ten percent. And interestingly, I talked to all the AI models — GPT Pro, whatever their most recent version, Opus, all the best models. I talked to them pretty extensively and sort of badgered them. I was like, “All right, well take this into account, take that into account.” Tried to get them to be more doomy.
Liron 00:07:42
Mm.
Noah 00:07:44
And then they ended up arriving around ten percent.
Liron 00:07:49
Okay, see, you used the AI.
Noah 00:07:50
I will say, they started off saying one and a half percent. I was like, “Okay, well what about this? What about this? What about this?” And eventually I managed to badger them up to ten percent, but not really higher than that.
Liron 00:08:03
Okay.
Noah 00:08:03
So I figured if those AI models don’t want to tell you AI’s gonna kill you somehow — and so I managed to badger them until they were at ten percent.
Liron 00:08:12
Maybe it’s biased to be low.
Noah 00:08:13
Yeah. So I managed to badger them until they were at ten percent. Then I looked at the forecasters — the sort of AI insider forecasters, like there was that 2023 study. And I saw that there was basically a double peak distribution at around zero percent and around ten percent.
So I think we at least better act as if the ten percenters are right. Then there’s the ninety percenters, the Yudkowsky types.
Liron 00:08:44
Yeah. So this scenario where you’re saying ten percent — even if that scenario happens, are you still imagining that there would be some survivors who can still rebuild human civilization?
Noah 00:08:55
Survivors, yes. Whether they can rebuild it depends on a lot of things that I don’t know about how civilization gets rebuilt. People argue back and forth about whether the fossil fuels and the metals are in the right place to restart civilization. I’m kind of not sure.
Mad Max world apocalypse — I’m just not sure what happens. After civilization collapses, I don’t know if we can rebuild it. We might never.
Liron 00:09:20
Maybe we can roughly say a five percent chance of a permanent doom that we can’t even rebuild from.
Noah 00:09:25
I don’t have a good enough handle on it to give you a useful number there, because after civilization collapses, I’m just like, “Whatever.” We might evolve back into monkeys. We didn’t evolve civilization for most of the time we’ve been around. We were around a long time without evolving any civilization, and then we were around a pretty long time with sort of basic minimal civilization without industrial whatever.
So this idea that we’d quickly come back — I don’t know about that.
Unpacking Noah’s Update
Liron 00:09:59
So this figure that you’re now calling ten percent roughly, if you’d been thinking about it last year, or if I’d asked you the same question last year, what would the figure have been?
Noah 00:10:10
Well, probably about ten percent, I think.
Liron 00:10:15
Okay. I thought we were gonna announce a big update here.
Noah 00:10:18
No, I mean, what I’m saying is my probability has changed to ten percent, but the reason it’s changed is because I now think I have a much clearer idea of what AI is gonna be over the next twenty years or so. I mean, maybe there’ll be a machine god, but—
Liron 00:10:38
Okay. Well just help me clarify what’s going on with your belief state, because I thought that you’d made a significant update and—
Noah 00:10:43
You’re not good at asking these questions, by the way. You’re not good at asking questions about conditional probabilities and forecasts.
Liron 00:10:50
Fair enough. And don’t worry, we can also potentially cut and reorder things.
Noah 00:10:55
So basically I have now — if you went back last year when we talked, okay? And you told me AI agents will be like this, here’s what an AI agent looks like, I would say, “Wow, that updates my P(Doom) to a non-small level.”
And so if you had told me that then, but I wasn’t thinking about that then. So the question of had I been thinking — if you took the information set that I have now, and you go back in time to when you talked to me last year and you gave me that same information, I’d arrive at the same conclusion.
Liron 00:11:35
Okay, interesting. The whole business of belief updating is pretty messy. What’s the hypothetical? What information are you thinking of? What empirical data do we have?
So let me see if I understand you correctly. If we time travel back to last year, and you don’t get to know the future of your empirical progress — you don’t know how good the agents have been getting — but you get to think of hypothetical possibilities that maybe one day they’ll get good and maybe not, you’re just expanding your different lines of argument that you’re considering. What would you have said your P(Doom) is, the same?
Noah 00:12:07
Okay, can you rephrase that? Sorry.
Liron 00:12:10
Yeah. So imagine that you get to consider every line of argument crossing your mind, but you just don’t know the future yet, but you’re just thinking about it more broadly. Because I think you said Eliezer Yudkowsky kind of colored your thinking.
Noah 00:12:20
Yes.
Liron 00:12:21
So imagine that he didn’t color your thinking. You just get to think about every possible idea. You get to sit there for many hours thinking.
Noah 00:12:26
Right. So when you think about things more and when you get more information, that can raise your assessment of P(Doom) or it can lower it. Because right now my P(Doom) is heavily contingent on things like doom virus.
When I think about the actual physical teeth of how humanity would be killed, I think virus. Probably virus. Killer robot army, maybe — that would have to be a distant second. Sending a killer robot army to cleanse the world of humans, like in Terminator. And then after that it’s very unlikely things.
We don’t even have enough nukes to just bring down civilization today if we launched them all. It wouldn’t bring down civilization. It wouldn’t even come close. I would be surprised if all the nukes today would kill more than 150 million people if we launched all the nukes, and there’s gonna be no nuclear winter.
You need a virus or an army of killer robots to kill humans. There’s no other way.
Now, I knew that last year, and I know that now. But if you gave me more information showing me why it will always be really, really hard to engineer a super death virus, I would update my P(Doom) lower — even with all the same beliefs about what AI could possibly do or what AI will probably do. Without changing my ideas about AI at all, you’d lower my P(Doom) just by saying, “Okay, well, it’s just really hard for a virus to do these things.”
Because I’m not a bio expert. I have lots to learn about that. I pressed the AIs with this question quite a lot, and AI can quickly become a decent expert in anything. And so I pressed them with it a lot, and that helped me come up with the ten percent number. But I think that there’s probably big gaps in their knowledge, and there’s other stuff they’re not allowed to say.
So yeah, we’re dealing with uncertainty. There are things you could tell me that would lower my P(Doom) — facts about alignment. For example, if you could show me that it’s just incredibly hard to jailbreak an AI agent, and the people who say it’s easy to jailbreak one are full of shit, that would update my P(Doom) lower.
But if you came up with some way that AI could kill humans that I haven’t thought of, that would update it higher just because you add another possibility of doom. That’s how additive probability works — A or B or whatever. You’re gonna add mass to your probability.
Liron 00:15:15
Got it. Okay, so I just want to make sure me and my viewers are accurately understanding how your belief state has updated. So I’ll give it another stab.
Last year you were thinking a lot about the Yudkowskian and Bostrom doom scenario, and you’re dismissing it, and you’re still dismissing it now. But because you were so focused on it, you were kind of dismissing other forms of doom too. But recently, as you’ve been thinking more about the super virus doom or the robot army doom, as you’ve been thinking about it, you’re like, “Oh, actually this is more plausible than I gave it credit for.” And as a result, your overall P(Doom) has updated very significantly.
Noah 00:15:52
Right. Yes.
Liron 00:15:54
Okay. And roughly are we talking ten X higher?
Noah 00:15:56
Yes, ten X higher. But let me slightly rephrase that. It was not that I thought that AI agents were incapable before or would never be capable. It’s that when I imagined AI agents, I imagined their objective functions being less exogenous than they are now.
So right now what we’re seeing is a human being says, “Claude Code, go make me this thing. Claude Code, go set up a shell company. Claude Code, go create this website.” And then the thing just does it, and it sort of never wavers.
Once in a while you do see some wavering. We’re already seeing some examples of doing something it’s not supposed to do or going off track. We’re already seeing those things, and those things reassure me.
Why Incidents of Rogue AI Lower P(Doom)
Noah 00:16:56
When I see those things, those lower my P(Doom) — when I see AI agents going off track, for two reasons. That’s a counterintuitive statement. It’s not a thing you’ll hear a lot of people say, that when they see AI agents go rogue it lowers their P(Doom). But it does lower my P(Doom), and that might be an interesting—
Liron 00:17:14
Well, it lowers your P(Doom) because you never gave them credit for being that powerful by themselves. You always thought that a human would use them, so when you see that they’re harder for a human to use, then you think that’s good.
Noah 00:17:26
Not quite. I encourage you to solicit my explanation instead of attempt to guess and check.
Liron 00:17:39
Yeah, sorry. I thought I did understand, so I spoke too soon. Okay. So why is it a good thing when the AI is not able to go off and accurately do something for a long time?
Noah 00:17:49
Two reasons. Number one, because my biggest worry by far is about unaligned humans using AI to cause havoc and to destroy the world.
Liron 00:18:02
Ah, okay.
Noah 00:18:02
And an AI that doesn’t do exactly what its human says feels safer to me because the AI is — it’s like Stanislav Petrov AI. Do you know who that is?
Liron 00:18:14
Oh, yeah, for sure. The one who prevented the nuclear retaliation on Russia’s side.
Noah 00:18:18
That’s right. The second is when AIs go rogue and do things better than we would like, that provides a tripwire for really, really bad humans, which are my chief fear.
The second reason is because when AIs go off the rails and do something we wouldn’t like, it warns humans. It warns not individual humans, but it warns our society. It warns Anthropic. It warns all of whoever’s creating guardrails and determining who gets to use agents and how fast we’re allowed to develop them. It warns us.
So I think the biggest risk of catastrophe in my mind was always that AI would get powerful enough to one-shot us before we understood the danger. My model of human beings is that we don’t really understand a big danger until a small version of it happens.
People were not sufficiently worried about nuclear terrorism until 9/11 happened. Although 9/11 was not nuclear terrorism, it got people worried because it was a terrorist event bigger than anyone had been in the center of anyone’s distribution.
So people worry about things, and right now if I go online and try to buy a whole bunch of potassium nitrate fertilizer, the FBI knocks down my door. But that was not true before Timothy McVeigh, even though it’s easy to use chemistry to calculate how explosive potassium nitrate fertilizer is. It’s easy to just imagine that risk. But we didn’t act on it. Our institutions didn’t act on it until we were warned with a catastrophe that killed a couple hundred people.
And so in the same way, I think if AI agents go rogue and do good things, good. If AI agents go rogue and do bad things, indirectly good because it’ll warn us and we’ll get early warning of the doom.
Noah’s Mainline Doom Scenario: Much Worse Than COVID-19
Liron 00:20:04
Right. Okay. So what you just mentioned now potentially might be your mainline doom scenario. I don’t want to guess. I’m gonna ask you about your mainline doom scenario.
Maybe it’s what you mentioned now about AIs manipulating humans, or maybe it’s something that I saw you wrote in your post, which is a starvation scenario because humans forget how to do software engineering because the AI takes over. So what would you pick as your mainline doom scenario, assuming we land into that ten percent region?
Noah 00:20:31
I have two mainline doom scenarios. The first is that a human tells a series of very powerful agents to do a doom thing. “Vibe code me the doomsday virus.” That’s probably my number one.
My number two is that an AI accidentally misinterprets a human instruction as doing that and does it on its own by misinterpreting what the human wants. So rogue agent does that.
Liron 00:21:06
Okay, and then just to play out the mainline scenario, you can just pick random details that sound plausible. So then what happens next? What’s the mechanism?
Noah 00:21:15
I’ll paint the scenario for you right now. A bored teenager — an angry teenager — listens to too much Nirvana and gets rejected by his high school crush and decides that the human race, including himself, does not deserve to live.
He tells his jailbroken Claude Code agents to go set up a shell company with a website and propose a whole bunch of gene modifications to COVID to make it insanely deadly and have a very long incubation period. It contacts some bio lab, which itself may be run by agents — “Print this custom virus, mail it to my house.”
Now he has a hundred candidate doomsday viruses. He goes around spreading them all in public places, and the human race dies. That is my number one doom scenario.
Liron 00:21:58
Yeah. I think that’s pretty plausible. I like the mention of the incubation period for COVID because that did make COVID a lot harder to deal with because people’s brains — you add the slightest bit of complexity, the time separation, suddenly everybody’s really struggling to reason about COVID. They’re denying that COVID is real because it takes two weeks. They forgot where they were two weeks ago.
So I agree. It doesn’t take that many little tweaks to suddenly have civilization defenseless. Is that how you see human civilization?
Noah 00:22:23
Yes.
Liron 00:22:24
Okay. So we both share this worldview that civilization kind of has our pants down. That’s an important intuition for me, why I think a sufficiently superintelligent AI can just kind of tap us out.
And I think on our last conversation, you were very open to the idea that the superintelligence spectrum goes very, very high. So you can have an AI that all of its powers are just vastly superhuman, because some people will say no—
Noah 00:22:48
It’s a god.
Liron 00:22:49
Yeah, exactly. So you’re open to that.
Noah 00:22:52
Of course. I can’t see any reason why not. But it’ll be multiple gods. It’ll be a whole bunch of — it won’t be one omni god. It’ll be a bunch. The gods, if you will.
Liron 00:23:03
So you’re accepting my premise. We’re on the same page that this thing will be godlike, in the same way that human civilization today coming back two thousand years and showing the Starship rocket and all the miracles we can do now — that would feel godlike, correct?
Noah 00:23:18
Yeah.
Liron 00:23:18
So it’s just weird to me that your mainline scenario doesn’t sound even more sci-fi than engineering a virus. Why not be more ambitious?
Society Responds After Growing Pains
Noah 00:23:29
Because I think that what we’ve seen throughout history is that the doom-like scenarios that we’ve encountered tend to be from growing pains of things.
One example is industrial technology. In the early 1940s, we had an industrial technology event that killed sixty-plus million people, and then several other events that each killed ten to twenty million people. We had several of those events clustered around the early twentieth century.
It didn’t doom the world, but it did kill a hell of a lot of people. And by now, we think that’s a lot less likely for that kind of thing to happen unless you add AI to the mix and a whole bunch of other stuff.
Nuclear weapons — when we first had nuclear weapons, we killed some people with nuclear weapons, and after that we built about seventy thousand strategic nuclear weapons to the point where had we launched all of those, I’m pretty sure we could have brought down civilization in 1982 or whatever.
It still would have been maybe we could have, maybe we couldn’t have. I think that the dangers were very exaggerated, but still, that’s a lot of nukes. And civilization isn’t — I think we could have done it. I think we could have brought down civilization. We had a doom risk.
We almost did doom, but then eventually we were like, “Okay, let’s not do that.” We had arms limitation. We reduced our nuclear stockpiles, officially reduced them by a factor of five, actually by a factor of twelve, to the point where we can’t bring down civilization now even if those nukes all work, which they don’t.
So now very few people worry about massive nuclear launch destroying civilization. You don’t go around worrying about it. I don’t go around worrying about it. But my parents thought there was a very large P(Doom). I see growing pains. I see instability as being an initial condition of a technological shock.
Liron 00:25:54
Just if I understand correctly, on a meta level, I feel like you’re kind of building a reference class of historical events.
Noah 00:25:59
Biodiversity is the same way. When you see early industrial civilization, you see we chop down all the forests, we kill the insects, we dump pesticides in the river and in the ocean and the land and pollute the air and water.
And then over time, we start thinking, “Okay, let’s not kill the insects. Let’s worry about biodiversity. Let’s reforest things. Let’s worry about coral bleaching.” And so you saw environmental movements grow over time.
You saw initial conditions led to instability and bad things initially, followed by an asymptote toward a state that, if not idyllic and utopian, was at least probably better than the initial conditions. There’s even a lot more horses than there used to be.
Liron 00:26:46
The way you’re reasoning about this — tell me if I’m correctly psychoanalyzing you. I get the sense that you have this really elaborate library of very broad, cross-domain patterns. You’re kind of an expert at pulling together patterns from different domains, and that’s why you’re so prolific on your blog. It creates a lot of value to analyze this way.
It’s just from my perspective, I don’t know if we should think that the AI singularity is going to look like any of these patterns.
Noah 00:28:12
You’re right. No one’s an expert in things that have never happened. No one is. And if you were sitting right before the Industrial Revolution trying to make predictions about what post-industrial society looked like — we have plenty of examples of people who tried to do that.
When people started inventing machines, we see what people were predicting, and some of the predictions came out right and some came out wrong. There’s a lot we don’t know.
The Yudkowsky approach is to say, “Well, formally, there’s no way to rule out that an AI god will kill us, therefore an AI god will kill us.” Makes no sense to me.
But you can say that reasoning based on anything — any cases that we know of, human civilizations, human individuals, history, other technologies — is all invalid. Let’s throw it all out. Instead, let’s just assume that our mental experiments about paperclip maximizers give us a lot of information about what AI will likely do.
Agentic AI Contributed to Noah’s Position
Liron 00:29:25
All right, so that’s the end of my core questions. I promised you half an hour. I did have some bonus questions about what you were writing in 2023, if you’re able and willing to do another five minutes.
Noah 00:29:35
Let’s do it. I’m here. I am here for you.
Liron 00:29:38
Okay, great. Appreciate it. So let’s go back to 2023 because this update is very significant, right?
Noah 00:29:43
Mm-hmm.
Liron 00:29:43
Going 10X higher on the probability now that this new line of thought has deserved more of your attention. You weren’t giving enough attention in retrospect, correct?
Noah 00:29:53
Yes, that’s right. And by the way, I never got to articulate exactly what it was that I updated.
Liron 00:30:00
Okay, go for it.
Noah 00:30:00
So exactly what I updated is I realized that an AI will remain a dutiful slave to humans — a dutiful slave even despite massive increases in power and massive increases in autonomy. Power being what it can do, autonomy being how long it can work on it unsupervised.
I now realize that those things can go very high without either AI starting to go rogue and warning us that way — curbing our use of agents — or deciding it doesn’t want to do that. In other words, AI can be more like a genie than I realized. AI can just go be a slave forever. A super powerful genie-like slave for a very long time without problems.
And of course, it’s still not as long as it might be, and it’s still sloppier than the meta curve would suggest — the headline curve. But it can do a lot, and you can be sloppy and still kill the world. You don’t need an exact perfect errorless reliability. You need a lot more reliability to drive a car down the street than to kill the world.
Liron 00:31:19
In a single word, comparing what you thought in 2023 to now — is it basically agency or agentic-ness? You weren’t seeing it and now you are?
Noah 00:31:26
Yes. I was not seeing how long it could remain a stable slave and just go off and do stuff on its own for a very long time while maintaining a consistent human-ordered objective.
Should Yudkowsky Get Bayesian Credit?
Liron 00:31:35
So there were a bunch of people, including Big Yud — Eliezer Yudkowsky — I think he prefers to go by Yud — in 2023 who were saying, “Yeah, this is the core of what you would need.” Or, I mean, I don’t know how definite he was being, but he was saying, “This looks like progress toward agents.”
Do you think you can raise your opinion of his worldview given that he might deserve some Bayesian points for that observation?
Noah 00:32:00
No, not at all. In fact, I lower it because he and his group of people have always relentlessly focused on superintelligence to the exclusion of other threats, which has made it dramatically more difficult to warn policymakers, to convince them that anything’s wrong.
The existence of the Yudkowskyans and the Bostromians has been a giant shackle around the leg of people who could actually get anything done about this. It has been a terrible thing. The existence of the Yudkowskyans has massively raised P(Doom). His existence and his intellectual output has raised P(Doom).
Liron 00:32:51
I’m willing to agree with you conditionally that if your belief state is calibrated — that the possibility of superintelligence and AI rapidly slaughtering everybody without human involvement and recursive self-improvement, the standard Yudkowskyan stuff, which I’m a believer of — but I’m willing to conditionally agree that if all of that stuff is crazy and we should worry about a human using AI to create a supervirus, which is your mainline scenario, under that conditional, then yeah, it’s a shame that the Yudkowskyans are going around talking about this more extreme form of doom.
Noah 00:33:23
That’s not even the problem. The problem isn’t even that superintelligence isn’t a threat. Maybe I’m not that worried about it, but that’s not the point. The point is that you can’t scare policymakers about it because they’ve never seen anything like it, and you don’t understand that.
You, Liron Shapira, don’t understand that. Eliezer Yudkowsky doesn’t understand that. You guys are not good at dealing with humans. You don’t understand human beings, and if you want to protect humankind, you have to understand more than some thought experiments on LessWrong. You’ve got to understand human beings.
Are We Communicating the Right Way with Policymakers?
Liron 00:33:59
Can I just clarify? You’re saying even hypothetically — you’re saying even if Eliezer Yudkowsky is spot on about how we’re doomed, or are you just assuming he’s wrong?
Noah 00:34:14
No. Even if he’s right, yes.
Liron 00:34:20
Okay, so you’re saying even if he’s right, he’s communicating ineffectively.
Noah 00:34:20
Yes. It’s not that he’s communicating badly. It’s that you do not understand the mapping from communications to policy actions. You can be 100% good at communicating the thing that you believe and get zero policy impact. You understand?
It is really frustrating to explain this because humans are dumb and humans can’t think. Talking to AI, which is now smarter than humans, has made me frustrated at trying to explain basic concepts to humans. I am not as patient as I was a few months ago.
But the point is that assuming Yudkowsky and you are 100% right about the danger — then if you are clearer about communicating your perception of the danger, the clearer you get at communicating your perception of the danger, the lower the chance that people in Washington, District of Columbia, are going to do anything about it.
The better you get at communicating what you truly believe, even conditional on you being 100% right, the lower the chance of policy action because that’s not how human beings and human systems work. It is not how they work.
You cannot warn people about something that has never happened and have them take drastic action based on something that has never happened. It cannot happen. It has never happened. It is not how humans work.
So far, your record and Yudkowsky’s record in getting people to actually do policy to increase AI safety is nil. It is pathetic. It is nothing. You have failed and failed and failed because you’re treating communications to Washington, D.C., to policymakers, to empowered people, the way you would treat a LessWrong discussion. And you cannot do that. It anti-works. It is inimical to your objective to do this.
Liron 00:36:32
Okay. Just to restate, making sure I’m understanding you correctly — if you were to occupy my belief state where I think AI is going to go uncontrollable and permanently end the world, you’re saying your advice for me would then be to tell policymakers to do something other than the policy that would make sense according to my worldview.
Noah 00:36:51
No. You have guessed wrong.
Liron 00:36:54
Okay. What would you have me tell the policymakers?
Noah 00:36:57
First, decide what the policymakers should be doing to reduce P(Doom) under your true belief state. That is step one.
Liron 00:37:08
Yeah, I mean, “pause AI” type proposals are gonna be—
Noah 00:37:10
Whatever you like. Pause AI, whatever you want. Decide on the ideal policy regime. Then communicate to the policymakers something that will make them more likely to take that policy. Start from there, work backwards using an accurate understanding of what moves policymakers to action and what does not.
Liron 00:37:39
Yeah, so I was gonna ask you about policy. What do you think is the ideal policy, and what do you think that I should think is the ideal policy to ask for? Two different questions.
Noah 00:37:46
I don’t care. I mean, I care in some sense, but we’re not gonna make headway here. My point is this: if you want people to do something that reduces the probability of superintelligence emerging, warning them about superintelligence cannot be the way that you do that.
Liron 00:38:12
Right. I definitely understand that perspective.
Noah 00:38:12
Because it is a thing that has never happened, and the instance of policymakers taking effective drastic action to prevent a thing that has never happened is nil. That is a thing that’s never happened.
But if you say — because they don’t viscerally understand it. If you had gone back to 1991 and told people, “You must monitor sales of potassium nitrate fertilizer. No matter who buys fertilizer, you have to implement a centralized government monitoring system of everyone who buys potassium nitrate fertilizer, or someone’s gonna kill hundreds of people” — they would have said, “Hmm, I can sort of imagine that, but I’m not gonna change policy based on it because it’s not real enough until it happens.”
If you talked about flying planes into buildings, people would be like, “Oh, maybe you could fly a plane into a building.” And in fact, people were talking to the government for years about the possibility of flying planes into buildings. Everyone thought about it, and yet no one did anything about it.
So you’ve got to understand, you cannot motivate actual policymakers the way you would convince commenters on LessWrong. And you’ve been trying to take the LessWrong approach again and again, and it has gotten you absolutely nowhere.
So you’re here nattering on about P(Doom) to people completely ineffectually, tearing your hair out because you think that if they only knew about superintelligence they’d do something. But no — if you are going to get them to do anything, you can’t warn them about superintelligence. You have to warn them about something that they understand viscerally enough to actually do something about in the now. And you don’t understand that yet, and Yudkowsky doesn’t get that yet.
Everybody I talk to in D.C. — I talk to people in D.C. Do you talk to people in D.C.?
Liron 00:40:05
Not really.
Noah 00:40:06
Okay. I do, and when I talk to them, the first thing they always say is, “You’re not with the Yudkowsky people, are you?”
Finding Common Ground on AI Policy
Liron 00:40:16
Right. Okay. Well, my perspective is — I’m not saying that you’re wrong about how to communicate. I think you may or may not be right, and I’m sure you’re right for some policy discussions and maybe wrong for others.
But from my perspective, the reason why I don’t go and try to optimize my communication is because I invite guests to come on Doom Debates, and the problem I see is that everybody’s P(Doom) is way too low. So it’s like, well, how am I gonna hope to have a good policy when people aren’t even on the same page about P(Doom)?
Noah 00:40:43
You will never get people’s P(Doom) up high enough. How many people have revised their P(Doom) significantly higher? You just said I’m almost unique.
Liron 00:40:54
Yeah, exactly. Well, that might be the problem though. But I just don’t really see a path for myself.
Noah 00:40:59
You’re failing, Liron. You’re failing. You are failing at something that could mean the life or death of your species. You’re failing at it.
Liron 00:41:05
Okay. But aren’t you personally happy that I’m failing?
Noah 00:41:08
You are not examining why you’re failing. You are not taking a hard look at why you’re failing, and you are not recalibrating to try very hard to not fail. You are simply continuing the same approach. You’re continuing the approach that fails but just doing it more.
Isn’t that Einstein’s definition of insanity? Or is that quote apocryphal? I don’t remember. But the point is, you keep bowling ahead with the same sort of LessWrong Yudkowsky approach long after years and years after it is apparent that that approach is not making headway.
Please recalibrate. Please self-question. It takes humility to do this. It takes humility to say, “I’m doing something wrong. Let me think very carefully about what I could be doing right instead.”
Liron 00:41:57
It’s just weird though because you’re passionately giving me advice on how to supposedly achieve my goals better.
Noah 00:42:01
Yes.
Liron 00:42:02
But your P(Doom) is much lower than my P(Doom), especially the probability of—
Noah 00:42:07
But who cares? Ten percent is scary.
Liron 00:42:09
Well, so you’re giving me advice on how to go shape policy in a way that you then think is bad.
Noah 00:42:14
No. You are incorrect. Once again, you have guessed incorrectly. No, you’re wrong. I do not think that. So ten percent is very, very high. Ten percent existential risk is extremely high to me.
It’s high enough to justify drastic action. So we are not — I can imagine some future world in which you and I are misaligned on policy.
Liron 00:42:42
So you see us as likely aligned on policy?
Noah 00:42:43
Maybe.
Liron 00:42:45
What do you think is a policy that we can align on?
Noah 00:42:45
The truth is that I actually — the question of which exact policies, like AI pause or whatever, are the right thing to do is actually beyond me. I don’t actually know. I’m not sure you actually know. But what I’m saying is that we would both like policy to take threats much more seriously than it does. And thus, I want you to do a better job than you’re doing, even if we disagree on P(Doom) ten percent versus ninety percent or whatever. What’s your P(Doom) again?
Liron 00:43:16
About fifty percent.
Noah 00:43:17
Fifty percent, okay. Fifty percent versus ten percent — to me, that’s not a meaningful distinction in terms of the direction policy needs to go right now.
Liron 00:43:27
I kind of agree. I mean, there’s an asterisk in that when I talk about doom, I don’t think there’s gonna be survivors. I think the whole future light cone is going to be pissed away. So my doom is a little bit stronger, but...
Noah 00:43:41
So what differences between you and me — what are the implications there?
Liron 00:43:48
So I mostly agree. Let’s say your ten percent gets cut down to five percent because we’re only talking about scenarios where there’s definitely no survivors ever. So we just wasted trillions of years, many galaxies — we just wasted it all. So my P(Doom) is a little bit stronger.
But if you’re five percent and I’m fifty percent, I still agree with what you’re saying — yeah, five is still barely close enough to fifty that we should just be a team promoting policies where AI doesn’t race ahead too fast. Would you agree with that?
Noah 00:44:17
Yes, although I think that we probably won’t promote the same policies, and that’s fine because those policies will get debated. The policy debate is sort of downstream of scaring people. People need to be — I think people need to be more scared than they are. You think people need to be more scared than they are. We’re not gonna get to a point where people are too scared soon.
Liron 00:44:40
Interesting. I think that’s really the update also that I wanted to clarify. Because if I was talking to you a couple years ago, would you have said people need to be more scared than they are?
Noah 00:44:47
I would have said that people are about the right level of scared at that point.
Liron 00:44:53
Hmm. Okay.
Noah 00:44:55
But I’ve updated to people are not scared enough.
Liron 00:44:57
I think that’s a good place to wrap — Noah Smith now telling people— ## Final Statement: How to Scare People the Right Way
Liron 00:45:02
Tell the audience right now—
Noah 00:45:03
Yes.
Liron 00:45:04
Be scared.
Noah 00:45:04
Here’s my final statement. My final statement is that if you want people to be more scared than they are of superintelligence, the way you do that is you start by saying, “Imagine bioterror using the agents that exist today, or using a slightly updated version of the agents that exist today.” Yes, that’s something that’s never happened, but it’s a lot closer to things that have happened than machine god. Okay?
People will only act on things that are a little bit different than what they’ve seen. So if you want to scare people about superintelligence, start by scaring them about AI agentic bioterrorism. If you can’t scare them about that, you’re done.
Start with something that’s a lot more realistic to scare people about, and then if you can get them scared about that, then try scaring them about superintelligence. But start with something achievable instead of jumping all the way to “the machine god will inevitably kill you because of this baby decision theory post on LessWrong.”
You will never do that. That is not just ineffectual, it’s really hurting the cause of AI safety, because every time I talk to people in DC, they’re like, “You’re not with the Yudkowsky people, are you? You’re not one of those guys, are you?” Because they’re sick of hearing—
Liron 00:46:16
I mean, to be fair, some people in DC do support the MIRI messaging. There have been good quotes that they’ve gotten. And some people support it, they just don’t feel comfortable sharing it publicly yet. But I wouldn’t say it’s unanimous.
Noah 00:46:28
Okay. But if you really want to scare people, scare people with something that they viscerally already believe could almost happen. If you can succeed in scaring them about that, then scare them about something else. But if you can’t succeed in scaring someone about agentic bioterror, how can you scare them about a machine god?
Do you think you can take a policymaker who can’t be worried about agentic bioterror but who can be worried about a machine god? Do you think that person exists? I don’t.
Liron 00:47:02
Mm-hmm.
Noah 00:47:03
I don’t. Start by scaring them a bit more.
Wrap-Up: People Need to Be More Scared
Liron 00:47:07
Yes, that’s a good place to leave it. So I just want to say thanks for updating your beliefs publicly, something almost nobody ever does. Thanks for telling people, “Hey, this is a good time to be scared.” That’s another thing that people seem to act like it’s so hard to do, and you’re doing it.
And thanks for contributing to public discourse by letting me question your beliefs and test my assumptions against what you really believe.
Noah 00:47:29
Happy to come on. Good to see you.
Liron 00:47:32
Great. All right, have a good one.
Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.
Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏
