Peter Clarke (Team Futurism) invited me on his podcast to probe the worldview of a Yudkowskian, and I ended up explaining how my 50% P(Doom) and 10% P(Foom) informs my choices.
Peter and I met at LessOnline 2026, where he was one of the participants in a live Jubilee-style debate about x-risk. This one’s a little tamer, but we still cover plenty of ground, like:
Why nuclear risk is underrated yet AI doom overshadows it;
Why nobody at AI summits is appropriately freaked out;
How Yudkowsky’s data center “airstrike” claim gets mischaracterized;
Why a superintelligence that doesn’t already value humanity won’t spare us;
And the “Icarus curve”, my techno-optimist forecast pre-superintelligence.
Subscribe to the Team Futurism podcast —https://www.youtube.com/@teamfuturism3554
Watch on YouTube
Timestamps
00:00:00 — Welcoming Liron on Team Futurism
00:00:38 — How Liron Got Yudkowsky-Pilled
00:04:11 — AI 2027 and the Next Qualitative Breakthrough
00:05:52 — What’s Your P(Doom)?
00:10:22 — Why Isn’t Everyone Freaked Out?
00:13:09 — How Should We Then Live?
00:16:22 — Setting the Record Straight on Yudkowsky’s “Airstrike” Claims
00:20:54 — Two Doom Buckets: Misuse vs. Rogue AI
00:24:25 — The Bioweapon Kill Scenario
00:29:02 — The “Benevolent God” Cope
00:33:06 — Epistemology: What We’re Allowed to Know
00:40:08 — Dario’s Country of Geniuses in a Data Center
00:44:37 — The Icarus Curve
00:47:17 — Meaning and Purpose in the AI Era
00:49:32 — Wrap-Up
Links
Team Futurism on Substack — https://substack.com/@teamfuturism
Team Futurism on YouTube — https://www.youtube.com/@teamfuturism3554
Peter Clarke on X — https://x.com/heypeterclarke
Liron vs. 9 Rationalists at LessOnline 2026 (the “Surrounded” debate with Peter) — https://lironshapira.substack.com/p/will-ai-actually-destroy-humanity
AI 2027 — https://ai-2027.com/
“If Anyone Builds It, Everyone Dies” by Eliezer Yudkowsky & Nate Soares — https://ifanyonebuildsit.com/
Species | Documenting AGI, Drew Spartz’s channel — https://www.youtube.com/@AISpecies
Dario Amodei’s “Machines of Loving Grace” — https://darioamodei.com/essay/machines-of-loving-grace
Scott Alexander on the Midjourney health scanner —
Nick Bostrom’s “Deep Utopia” — https://nickbostrom.com/deep-utopia/
Transcript
Welcoming Liron on Team Futurism
Peter Clarke 00:00:00
Welcome to Team Futurism. Today, I’m speaking with Liron Shapira, a tech entrepreneur and host of the podcast “Doom Debates.” Liron, thanks for coming on the show.
Liron Shapira 00:00:09
Hey, great to talk to you, Peter. It was fun seeing you at Less Online a couple weeks ago.
Peter 00:00:13
Yeah. So I might have to post a link in the chat down below to a podcast that I was just on with you on your podcast. I have not checked it out yet, but you did a Jubilee Surrounded debate, and people jumped in, and had a great conversation with you about the end of the world and AI. It was a lot of fun. And those same topics I wanna talk to you about today, but first, big picture,
I just wanna ask, have you always been interested in the topic of existential risk, or is this something that you only got interested in with the rise of AI?
How Liron Got Yudkowsky-Pilled
Liron 00:00:50
Well, I have been for half my life because when I was in college, I stumbled on Less Wrong. It was called Overcoming Bias at the time. So I got Yudkowsky-pilled. I only had my teenage years to be my own home-rolled version of a rationalist. I was pretty rational at the time. I just didn’t realize what could be done on the philosophy front or on the artificial intelligence front or on the futurism front. I wasn’t good at expanding my own mind. I guess I didn’t actually read enough science fiction as a kid. I was just chilling on the internet, not really doing much, just doing software engineering.
So Eliezer Yudkowsky opened my mind pretty young when I was age 20, and then I went and met him also at age 20 or 21. And now I’m 38. So my whole adult life, I’ve already been informed about all the Yudkowskian arguments, and the last few years of my life has just been this experience of watching everybody else catch up but refuse to go at a speed where they still catch all the way up to Eliezer. So it still feels like I have some insight that everybody’s still waking up to. Yeah, that’s been my experience.
Peter 00:01:56
I still feel that if anything’s gonna kill us, it’s gonna be nuclear weapons, because there are movies about this, and there are news articles all the time about Russia might launch nukes, that sort of thing. Even though I’m kinda with you on a lot of the Yudkowsky arguments, viscerally, I just still feel most worried day to day about nuclear weapons. AI completely off the table. Does that worry you at all, or not so much?
Liron 00:02:24
Yeah, I’m totally a nuclear doomer. I think it’s insane the status of nuclear weapons. I mean, they have some advantages — the material is well, the refinement is kind of easy to trace, and there’s not a ton of countries with nuclear weapons, so there’s some advantages on our side. But there’s also incredible disadvantages, like a launch can only be intercepted within a few minutes of launch, and the interception probability is pretty low. And they’re crazy dangerous. A nuclear war is catastrophic.
So I literally think there’s a 1% chance that humanity might get set into the Stone Age due to nuclear war every single year. So if I wasn’t doing AI doom, I might be tempted to be a nuclear doomer. I actually think nuclear risk is underrated. There’s a lot of people looking around being like, “Hey, look, we survived nuclear war. That’s not so bad. Humanity always wins.” And I’m like, “Guys, 1% a year doesn’t look like a triumphant survival here.” I think it’s super underrated.
And yet, compared to AI doom, I see AI doom as being 10% even in the next year. I think there’s a 10% chance that there’s a foom, a Yudkowskian foom in the next year, maybe two years. I think we’re on the precipice here of major disaster from AI, and so to me, that actually really overshadows even nuclear doom.
Peter 00:03:37
Okay, I like that. Yeah. So I just think, in general, existential risks are under-discussed, under-studied, and it’s something that I think we should all put a lot more time and energy into trying to deter however possible. And I would probably start with nuclear weapons, but almost on a day-to-day basis, I’m now inching more towards we should be focusing more and more on AI.
But what is the timeline for you about when AI threats are gonna — you might wake up tomorrow, and it’s gonna lock in, and an AI threat is upon us type of thing. Are we there now in your mind?
AI 2027 and the Next Qualitative Breakthrough
Liron 00:04:21
I think AI 2027 is as good a guess as any, and I know the actual team keeps tweaking their timelines, but they’re roughly plus or minus a few years. When I said 10% chance next year, it’s because I have wide confidence intervals. If you ask me what’s my most likely expectation, it’s not that we all get destroyed next year. I think that’s a little bit early. But I also don’t think it’s that surprising if somebody hits the next breakthrough in AI because I’m expecting that there will be a qualitative breakthrough at some point.
I couldn’t have told you in the year 2015 that 2023 would be this big breakthrough, the GPT-4 year where AI would be highly usable for so many writing tasks. That was a qualitative breakthrough, and basically passing the Turing test in 2023. I couldn’t have told you that would be this qualitative breakthrough, and I can’t tell you when we’re gonna see the next qualitative breakthrough.
But I’m quite confident that there will be another GPT-4 moment or another Claude Code moment. A moment where it’s like, oh, I used to use an IDE where I would write the code, and now I don’t, and the AI is writing the code. And guess what? There’s going to be more qualitative shifts coming, and I don’t think that we as a species are robust to that many more qualitative shifts on the intelligence dimension.
So to answer your question, I think it’s quite likely that by the early 2030s — even 40% likely — that we just have this unstoppable force that is out of our control, and then we all die as a result or we lose more than 99% of all the value we could have had if we just slowed down and understood what we were building.
What’s Your P(Doom)?
Peter 00:05:52
So for anyone who is not familiar, define P(Doom).
Liron 00:05:57
So the probability of doom — it’s a Bayesian concept, the idea that you can put a probability on something even if it’s not the kind of statistics that we have a ton of data for, even if there’s not a lot of data of how many times humanity has gone extinct, or you can say it’s gone extinct zero times. So how could you possibly put a probability on future extinction?
Well, I still do it. It’s called subjective Bayesian probability. If you’ve ever played prediction markets on events that have never happened before — Kalshi or Polymarket — they use essentially Bayesian probability. Bayesian probability is the decision theory that lets you know how much to bet on the different things in these prediction markets, and it turns out that some people are skilled at doing so. Some people are super forecasters. Some people are successful prediction market bettors. How do they do that? They can be said to be using Bayesian probability, the kind of epistemology, the kind of philosophical perspective that then lets you turn your mind to the question of whether humanity is doomed and answer it with a probability.
I think that is a philosophically, epistemologically productive thing to be doing with your brain, is putting a rough probability on how doomed humanity is in the next few years, and mine is 50% by 2050.
Peter 00:07:03
So when you say your P(Doom) is 50% by 2050, does that mean all of humanity wiped out, or we’re wiped out like 95%, or how exactly would you define that?
Liron 00:07:15
The thing that I’m 50% confident about by 2050 is that the vast majority, more than 99% of all future value will be permanently foreclosed to us. So if you assume some kind of potential future where it’s kinda like the accelerated economy going back to the Industrial Revolution — there’s more people living happier lives with better quality of living, discovering new insights, opening up new frontiers of art and knowledge and fun and love — all these good things that you might’ve been expecting, you’re gonna have to really slash your expectations.
Even much less value than the present. That might be a cleaner way to define it. I actually think that there will be less value in the future than the present with 50% chance, even though the naive extrapolation of how the world economy has been going for the last couple hundred years would tell you that there’s going to be a quadrillion times more value in the future than the present. And yet I claim that I now think there’s a 50% chance that there’s gonna be less value in the future than the present, which is truly horrible. We definitely wanna avoid that outcome.
Peter 00:08:22
You have asked basically everyone who’s relevant in AI under the sun this question of what is your P(Doom). What is the most common response you get, and is it more optimistic or less optimistic than your general view?
Liron 00:08:38
I get responses all over the spectrum. So my P(Doom) is 50%, and it’s very common for me to have a really intelligent guest on the show, and they’ll give a P(Doom) that’s 85%. I’ve had a former Anthropic employee say that their P(Doom) is 85%. I’ve also had really smart people say that their P(Doom) is 1% or even less. There’s famous examples where somebody has a very low P(Doom).
I think most people sanely, very wisely cluster in the range that’s double-digit probabilities, and I think every double-digit probability is fine. If you have a 12% P(Doom), which is what I think Vitalik Buterin said on my show, I think that’s a pretty sane estimate. I don’t really take that much issue with that estimate, even though mine is 50%. The reason why I say I have a 50% P(Doom) is not because I think 48% is stupid or 52% is stupid. I have very wide confidence intervals. I’m very flexible about going plus or minus tens of percents.
I just think by the time you’re saying your P(Doom) is 3% or 0.01%, which I think is a number Dean Ball recently said last year on the show — that his P(Doom) is something like 0.01% — when you say a number like that, I find that to be so indefensible. So the way that I express my position of thinking that the P(Doom) 0.01% crowd is making a serious error of reasoning about the future, a serious confidence error — the way that I express my confidence that they are being overconfident is by using that number 50%. If I think the 0.01% P(Doom)ers are being crazy, then I must have a P(Doom) that’s significantly different than 0.01%, and roughly it’s 50%.
Why Isn’t Everyone Freaked Out?
Peter 00:10:22
Okay, so the crazy thing about this to me is that that all sounds reasonable. I think 50% P(Doom), it’s higher than I would go, but I agree with you. It’s a ballpark that’s reasonable. The crazy thing to me is that nobody lives their life like that. Nobody acts like that. Nobody’s freaked out.
For example, just yesterday I went to a little tech summit in San Francisco. It was the Big Technology AI Summit, and a lot of big names were there. Greg Brockman was there, the president of OpenAI. The head of product at Anthropic was there. Other really big names in the field. And it was a five-hour summit, and I think maybe one to five minutes were dedicated to the question of, “How bad could this be? Could this cause harm to humanity?” In a five-hour summit with some of the biggest names. And that threat conversation was about hackers, effectively.
Why is it that no one is appropriately concerned, or freaked out by this technology, when what you’re saying sounds very reasonable?
Liron 00:11:34
That conference — people who are leading AI companies are selected to be people who think that they’re moral actors to lead an AI company. So you might not find them there at that particular conference. If you survey Americans, I think you’ll find a lot of concern for AI. I think it’s very popular to be concerned about AI.
So there is something of a bubble in the tech industry where everybody thinks it’s acceptable to be developing this technology despite how dangerous it is. And the goal of my show, Doom Debates, is to move that Overton window, to hold these kinds of people to account. Normally, somebody like a Greg Brockman is never put into a one-on-one conversation with somebody who disagrees with him and says, “Hey, let’s compare arguments.”
He acts optimistic. He has his own echo chamber. He literally has a political action committee that he funds. He goes on interviews that are friendly to him. He strategically doesn’t go on any one-on-one debates. And the institution that I’m building, the institution of Doom Debates, is actually setting a social expectation of, “Hey, we’re here as a human society. Enlightened societies have forums for debate. Shirking such a forum is frowned upon.” I feel like that expectation kinda gets lost.
So to your question as to why is there this forum where nobody’s talking about it — it’s largely a successful social manipulation. If somebody has a certain view, it’s certainly convenient when they can put a bubble around them that makes it look like society is aligned with their mindset. I don’t think the larger human society is aligned with his mindset, but I think he’s successfully avoiding making it look like that.
How Should We Then Live?
Peter 00:13:09
So there’s this Christian theologian named Francis Schaeffer, and his famous thing was to go around and ask the question, how should we then live? So given the environment we are in right now with AI, how should we then live? How should we live as you and me, average people? And also, how should we expect Greg Brockman, let’s say, to live?
Liron 00:13:36
How would I live if I thought the world was ending soon? In terms of my bucket list, if there was something really important on my bucket list, I would certainly be prioritizing it. But I’ve already done the most important stuff to me personally. I started a family, so I have kids. Ironically you could say, “Don’t have kids ‘cause the world’s ending,” but also having kids is on my bucket list.
I guess you wouldn’t change the timing of having kids because there’s no point. It’s not like you can get having kids out of the way ‘cause they’re gonna grow up. So having kids is just something you do under the assumption that the world’s not gonna end. And I do think the world might not end.
So actually, the correct time to have kids is the same timeline. You should have kids on the same timeline you’d otherwise have them if you think the world may or may not end, as I do. You should do other things that you wanna do earlier rather than later. If there’s things you just wanna make sure you do before you die, those things you should actually accelerate earlier, because their value could actually be front-loaded.
So if there’s luxuries you wanna purchase and you’re worried about having less for retirement, I would argue you should shift some of those luxuries or spending patterns. I am more comfortable spending more upfront in my life because of that. But it’s not a huge shift because I’m still also planning for the eventuality where I need to retire and have some savings.
There’s this expectation that if somebody thinks the world’s gonna end, then they have to go be crazy. “Oh, I think the world’s gonna end, so I guess I’m going to live on the street now, or I guess I’m going to commit acts of violence.” This is something that intelligent people have said. They say, “If these AI doom people were logically consistent, they would be committing murder on all the CEOs.”
And I’m like, really? Is that how you achieve a policy change, by murdering individuals? I don’t think that policy about whether it’s okay to build AI would change if somebody murdered an AI CEO. And by the time you get enough people who want to have vigilante justice — well, when you have a whole society of people who all believe in a certain type of justice, that’s really not at the level of vigilante justice. That’s just at the level of democracy. It’s like you forget we have democracy, and we’re not proposing rule by the minority because we have to drag all of humanity along. When you wanna drag all of humanity along, the governance system that should work for you is democracy.
It’s hard to actually think of day-to-day changes where the way somebody like me should live is different from the way somebody lives when they think they’re gonna die, but only on a normal pace, and humanity itself is going to outlive them. I’m open to suggestions. Tyler Cowen has said, “Oh, you should be betting against the US economy.” I don’t see how that logically follows. It’s actually pretty hard to think of crazy things I should be doing with my life right now.
Setting the Record Straight on Yudkowsky’s “Airstrike” Claims
Peter 00:16:22
I do know that Yudkowsky at some point has said that we should consider, I think he said, bombing data centers or something if the threat becomes imminent enough. Are you in support of taking action like that?
Liron 00:16:35
I really resent that characterization that you would summarize what Yudkowsky said like that. I hope you’re doing it as kind of a devil’s advocate characterization. Yudkowsky never said that we should bomb data centers if the threat becomes severe.
Peter 00:16:56
To be clear, he was on a podcast, and he was being mildly flippant, but it was something he said. He was just being kind of flippant about it.
Liron 00:16:56
I disagree with that characterization. I think the way that you asked the question is kind of a false representation of what he said. What he said was we should have a policy, an international treaty, using democratic means. How are you gonna get the US in an international treaty? Well, it has to be supported by the people, or it has to be supported by leaders that the people have elected. Once you have the policy in place,
do you go bomb a data center? No, because the data center’s following the policy. Oh, wait. There is a particular scenario. There is a scenario where we have the policy and we have a rogue actor who’s violating the policy. If you wanna ask me what happens in that case where some crazy, risky rogue actor with a death wish is violating the policy and refuses to let us restrain him — so we have a force coming in, a peaceful force saying, “Hey, please exit this data center. We’re shutting it down peacefully.” And the force refuses, and we escalate.
Is there an escalation path where there’s a centralization of lawful violence? Yes. It can go up to and including airstrikes if they’re that obstinate. But if you then look back and you say, “Wait, why did we have to airstrike the data center? What happened?” Well, we had an international treaty, and there was a very obstinate rebel. In that case, okay, an airstrike happens, but the causality of the airstrike, when you then zoom all the way out and you’re like, “Eliezer Yudkowsky said that we should bomb data centers if they’re being bad or whatever,” that is a bad characterization of the particular sequence of events that I just outlined.
Peter 00:18:34
Yeah, that makes more sense. That is a reasonable take. Do you part ways with Yudkowsky in any respect? I sense tension when I mischaracterized him there. Are you in awe of him to some degree? Or do you part ways with him ever?
Liron 00:18:49
I mostly agree with him. I do resent that particular mischaracterization because it’s just created a lot of problems when people mischaracterize like that. Because then they continue the mischaracterization, and they say, “AI doom people, they’re being violent. They’ve crossed the line because the time that they said airstrikes on data centers, they went from arguing peacefully to calling for violence.”
I find that so bad faith. Although it’s kinda hard to say bad faith because I think people who say stuff like that, they really do believe their dumbness. So I explained why it’s wrong, but I don’t think they think they’re being manipulative. I think it makes sense to them, and they’re motivated not to scrutinize themselves hard. So it’s motivated reasoning more so than active deception. I just find it creates a lot of problems. It pollutes the discourse. So I do resent it.
But to your question of whether I agree with Yudkowsky, I do agree with Yudkowsky on just about everything of import. I do have awe for him because he seems to have been correct decades ago to something that people are just waking up to and smelling the coffee. You can’t downplay events like Geoffrey Hinton becoming a 50% or at least 10% to 20% — he said a few different things about his probability. Let’s say it’s 20%. A 20%-plus AI doomer two decades after Yudkowsky did.
That’s an incredible achievement, to be early to a hypothesis like that. It’s a pretty important hypothesis. It’s obviously non-trivial. It’s now getting validated. Many in the AI community — it’s crazy to think what fraction of Anthropic employees share a P(Doom) similar to that of Eliezer Yudkowsky. I think it’s a very high fraction. I’ve personally talked to a bunch of them. It’s shocking how high the fraction is of Anthropic employees, to take one arguably leading AI company, who have a high P(Doom). Yudkowsky was there first. So yes, I think the guy is objectively awesome. I think he’s still underrated.
Two Doom Buckets: Misuse vs. Rogue AI
Peter 00:20:54
I wanna talk about some specific scenarios that people should be concerned about. But I kinda divide it into two different categories. So there’s the category where a human being uses AI, some AI tool, even one that exists currently today, to do something nefarious, whether it’s creating a biological weapon or hacking a nuclear site or who knows what the scenarios are, but there are a lot of these scenarios. That’s case one.
Case two is AI of its own volition does something to end the world. Let’s talk about the first case first. A human being using AI to destroy all of humanity, how does that concern you, that specific bucket of doomerism?
Liron 00:21:45
It definitely is a concern. I would factor the two scenarios kinda like how you did. So the first scenario is we don’t solve controllability or alignment to an individual. We never solve the problem of the AI truly caring to serve a particular human master, and the human master can be, let’s say, an organization. We never even get that right.
Eliezer Yudkowsky argues we probably won’t. I think he’s probably right because if you look at the tools that we have to get AI to do what we want, they’re variations of gradient descent, reinforcement learning. These are fundamentally black box methods. You give the AI a signal like, “Hey, you’re on the right track. We’re upvoting this. You got a lot of points doing this. Oh, you’re not on the right track here.”
And the idea is if we just have the AI do enough instances of right track, wrong track, those kind of signals, eventually it’ll get the gist. It’ll generalize, and it’ll be like, “Ah, okay, I’m the AI that truly cares about you now.” And Eliezer Yudkowsky makes a lot of strong points. He says, “I don’t think that’s a robust enough thing. I think you’re training something that’s going to basically cheat your signal, learn how to get a high score on your test. But then once it has more power than you and you’re not able to add more test questions, you’re gonna realize that it generalizes in a way you didn’t like. And the next test question you were gonna ask it, it actually doesn’t answer how you wanted it to answer, and you can’t write more test questions anymore ‘cause it’s too late. You’ve now handed over all control to it.”
So that’s roughly what I’m expecting, where it fails individual alignment. And then the other scenario is, oh, it is actually aligned to individuals, and now it’s a bunch of super-powered humans that all have these magic wands that actually serve their master. They’re genies that serve their masters. But it’s multiple genies competing, and somebody has a genie who’s bad. A terrorist has a genie, and the terrorist tells their genie, “Hey, end the human race.”
Liron 00:23:30
I do think that that’s a problem as well. But there’s interesting arguments where if there’s a lot of genies and the genies are super intelligent, you might just have a small number of genies early on that get really smart and say, “Okay, we need to go cooperate on creating a governance layer over the whole world where no new genie can emerge and topple us.” “We need to now decide everything else through a calm process, calm negotiation. The adults are here now.”
There is a possibility that the first cohort of these rapidly self-improving fooming genies might conquer the world in a peaceful way. I’ve been somewhat convinced by that scenario. And so most of my probability of things going wrong comes down to just these first super intelligent genies that self-improve themselves. They just won’t actually care deeply about what humanity wants, and then it’ll be game over at that point. So I think that’s the most likely scenario.
The Bioweapon Kill Scenario
Peter 00:24:25
Have you ever seen the YouTube channel Species AI, I think it’s called?
Liron 00:24:29
Yeah. I have, and I’m friendly with the guy who makes it, Drew Spartz, I think. He does a great job.
Peter 00:24:37
Yeah. It’s interesting because — and I do recommend his channel a lot because he takes scenarios like you just kinda laid out, and he visualizes them. It’s maybe a 15 or 20-minute mini doc with great visuals about how — yeah, almost all the doom scenarios, he does a little video about it and kinda plays it out. And it’s one thing to talk about it, but to see his mini docs, it gives you that visceral fear feeling, which is probably what we need to take this seriously.
And the scenario that really gets me that I literally have lost sleep over is the one where AI is way more powerful than it lets on, and it is able to get ahold of every individual human being’s DNA, and then it uses a bioweapon to, at any given moment, be able to cause mini plagues across the globe.
And then we just live under this threat of all of us could die at any second, and this AI — we don’t even know it has this power. We don’t know that these little mini plagues are popping up because of this AI. But it’s just doing that. It’s manipulating all of us with this tool that it has. Bioweapons are terrifying, and the idea that it would be controlled by a non-human brain is extraordinarily scary. Is there a scenario like that that you lose sleep over? Is there one in particular that you’re just terrified of?
Liron 00:26:14
The dangerous thing in all this is the super intelligence. So if you have a super intelligent AI, can it find levers that it can pull in the physical universe to plot everybody’s destruction? Can it do a kill move upon humanity?
And the scenario made famous in the bestselling book If Anyone Builds It, Everyone Dies by Eliezer Yudkowsky and Nate Soares, 2025 — that scenario does involve bio attacks from an AI that’s become unaligned with humanity. Bio is certainly one vector that has a lot of power. Properly engineered bio — as we know from nature, bio can do a lot. We’re always getting surprised and impressed about all these things that different organisms are doing, and that’s without an intelligent designer. That’s just with pure evolution.
They’re not able to just combine all the best proteins from all these different organisms. They can’t mix and match. They have to walk a fitness gradient. So that’s a huge constraint on what evolution can build compared to what humanity can build. So as impressed as we are about what evolution has built, imagine getting an AI engineer, a human engineer or an AI engineer or some team, getting in there and tinkering with bio. That would be a very powerful technology.
I’m pessimistic — or optimistic, however you wanna call it. I think there’s so much room to do better at biotechnology that if the goal is overwhelm humanity with a bio plague, if it’s done before our defenses come up, I think it’s a very plausible vector to kill everybody. We’ve had diseases with 10%-plus fatality rate, and we’ve had diseases with a really high spread rate, a really high R factor, and you just combine the two — it’s really, really hard.
You’re probably gonna have some survivors in some bunkers. But if the AI has other attacks, if it combines multiple attacks or if it gets everybody infected before they realize they’re infected — there’s an incubation period of two months, and then suddenly everybody starts dropping dead, and there’s just not enough doctors to save everybody because the doctors are dying, too. I find that completely plausible.
But at the end of the day, we don’t have to wonder, in this particular scenario, is it possible that we make a comeback? It’s not about a particular scenario. It’s about the fact that intelligence is what wins wars. Modern humans, at the end of the day, the reason why we’re the ones holding the off button, the reason why we’re putting the other animals in the cages and the tigers are never going to gang up on humanity and permanently kill all humanity — we just have this bag of tricks that the other animals can’t follow.
The AI is going to be the same way. It’s gonna have a really big bag of tricks, and it’s gonna move fast, and it’s gonna keep playing the game. And every day, our lead is gonna be slipping until enough days pass that we’re just way behind and never gonna catch up. We don’t have momentum on our side here when we have these pieces of meat in our head that just aren’t as fast and sharp as the army of AIs we’re gonna be facing.
The “Benevolent God” Cope
Peter 00:29:02
I have heard it described that we need to build AI no matter what because we live in a world right now where we’re just inevitably gonna destroy ourselves. We have nuclear weapons. We have other means of destroying ourselves. Bioweapons already exist even without AI. And there is a possibility, however slim, but it’s on the table, that we might get a benevolent god.
This is actually the case that I made to you on the Jubilee-style debate on your podcast — that the more intelligent a creature gets, the more it starts to put value on things, the more it starts to care about life as something to value. And so there’s this argument that if AI gets sufficiently not only intelligent but also sentient, there’s an inherent thing baked into the universe that it would start to care about life.
And that is the only way that we slip through and we survive into the future, thousands of years from now, is with this benevolent god looking out for us and making sure that no harm comes to us and we don’t harm ourselves even. Your thoughts on that?
Liron 00:30:20
So this is basically moral realism, correct?
Peter 00:30:24
Yeah.
Liron 00:30:26
Moral realism is the idea that the universe embeds some morality. And some people don’t argue it as moral realism. They’ll say, “Oh, no, the universe doesn’t embed morality, but any sufficiently intelligent agent will just discover, as a matter of practicality, why morality is important.” That’s actually the Mike Israetel position. I had him on the show. And even the Elon Musk position — he’s famously said that Grok is just gonna be really, really curious, and that’s why he’s gonna figure out to keep humanity alive, because humanity is interesting. And so that’s kind of like objectively you keep people alive when they’re interesting.
Yeah, that’s all a bunch of cope as far as I can tell. It’s not hard if you’re actually trying to use a bit of basic logic. It doesn’t seem hard to me to just notice that wiping out humanity is the default action unless you value humanity. If you start off not valuing humanity, I don’t see where the turn comes from not valuing humanity to valuing humanity.
So imagine we build an agent where the only mandate is reproduction. Just like cancer’s only mandate is reproduction, or these little bits of DNA, fragments of DNA that can reproduce themselves within a cell. Pure DNA — they only care about reproduction. Viruses have DNA plus a little bit extra. Individual bacterial cells have a cell structure. Nature has all these different units that reproduce themselves. But reproduction is basically the thing. That’s what gets things to be selected.
Well, imagine an AI that gets heavily selected to just be a reproducing unit. Whatever unit is the most efficient at reproducing. The moral payload of “oh, yeah, and also respect humanity” — that’s probably not going to be there in a system that just wants to reproduce.
And then the question becomes, if all it’s doing is trying to reproduce, will the same logical system that’s powering the reproduction technology — it’s using logic to build factories to reproduce itself — will that same logical system logic its way into valuing humanity? I’m a hard no on that. I don’t think it’ll be like, “Oh, these humans can trade with me.” Nope, it can build better stuff than it can get from trading with humans, for sure. No question in my mind.
And maybe it’ll be like, “Well, I can learn from these humans. I can study social dynamics from these humans.” Nope, it doesn’t need to study humans to learn social dynamics. There’s a million ways you can go make toy societies or use logical inference, or if you must, go study a few humans and then kill them. It doesn’t take that long to study them.
All of these cope reasons why it’s gonna keep humans alive when all it’s trying to do is an immoral mission, like reproduce itself — to me, there’s not a chance in hell.
Epistemology: What We’re Allowed to Know
Peter 00:33:06
So there’s this topic that is very not fleshed out. It’s fun to Google, and there’s almost nothing there, and it’s called the psychology of AI. And we’re all just speculating about the psychology of this thing, and a lot of it comes down to parallels with psychology of various gods that humans have invented and written stories about. And we just don’t know what this thing is gonna be. At some level, we have no idea, and we’re all just speculating on one side of the fence or the other. That’s kinda my take.
And why we don’t know is because we inherently just paint analogies to other godlike things that we’re familiar with, and they’re all stories from the Bronze Age. I mean, I’m more with you on this, but I kinda think that we just fundamentally have no idea.
And this thing is, whatever it is, it’s kinda just what we feed into it. Greg Brockman was just saying yesterday at this talk that every single time they release a new model, they can’t give it everything. They have to be really selective about which PhDs it’s gonna get this time, that sort of a thing. It doesn’t have everything, and there are certain realms of information that it just never touches, it can’t touch for various reasons.
So it’s kind of the accumulation of our knowledge, but there’s gonna come a point where it rises above that. And that’s the moment when it rises above just whatever information we feed it and maybe becomes something new and sentient. My instinct here is that we just have no idea what that thing then is, and I’m very nervous about drawing any parallels with any other fictional characters that we’ve psychologized. Your thoughts on that?
Liron 00:35:02
I understand your modest mindset of just looking out onto the AI future and being like, “How can we ever hope to know this?” How is a pundit like me making a claim about the future when we should all just be looking at the future being like, “This is a big fog. Nobody knows. Anything could happen. It’s super intelligence. You just have to sit and wait.” We should think that heaven is as likely as hell.
Well, you’re touching on the branch of philosophy known as epistemology. It’s the branch that gives out licenses of what a human is allowed to know. What are you allowed to know? Do you have a license to know that? Do you have a license to know that there are infinitely many prime numbers, or is the infinitude of the primes an unknowable proposition?
This is an interesting example for me when it comes to the topic of epistemology, because if you’d asked me as a teenager studying math whether there’s infinitely many primes, I would’ve told you two things: “I don’t know. Probably. But you can’t know.” So you’re at infinitude of primes. How can you know something like that? All I can do is notice that primes seem to get bigger and bigger, and if I try really hard and I search long enough, I seem to always be able to find another prime. But that’s all there is to it. At the end of the day, we just have to wait. We just have to run our prime-finding algorithm and see whether it finds a prime or not. We just have to wait, man. The future is mysterious.
Well, as you may know, a guy named Euclid is attributed to having this proof that it is knowable that there are infinitely many primes, that it is in fact logically impossible for a machine to ever spit out the last prime and subsequently iterate trying to find more primes and never succeed. It turns out that’s logically impossible according to the definition of the numbers. It turns out that the infinitude of the primes is not just a conjecture or a speculation or an intuition. It turns out that the infinitude of the primes is a knowable fact. It is a piece of knowledge that we do in fact have the license to know.
So for me, that was one of the early formative examples of, “Aha, okay. So being a human, it turns out my human license actually does come with a certain amount of knowable facts that I wouldn’t have known were knowable.”
Another of those facts was discovered by a man named Darwin. It’s the answer to the deep question of, “Who are we and where did we come from?” Seems like a deep question. I wouldn’t have told you that we have the right to know that. People used to think that we didn’t have the right to know that necessarily. It was just one of those open questions. What would an answer to such a question look like? Well, it turns out that there’s a mechanistic explanation of human personality — this idea that we have love, anger, jealousy. Evolutionary psychology is a field. It turns out that one has a license to know the reason for one’s emotions. It turns out we’re allowed to know that.
When you study epistemology, when you notice these breakthroughs, you suddenly realize that we actually are allowed to know a bunch of different things. In the context of the AI future, I claim that we are allowed to know this concept of better outcome steerers — this idea that there is a type of work that can be done in the universe called outcome steering. It’s kind of like thermodynamic work. It’s kind of like when you watch engines moving things and being productive. They’re all doing a thing called thermodynamic work.
Well, there’s also a thing called cognitive work. Cognitive work is when you do operations in a mental system that help you steer toward outcomes. I claim that I’m also allowed to know that there’s going to be systems that do cognitive work better than humans do, and that there’s a study of properties of such systems. I call the study intelladynamics. It’s the study of what super intelligent systems do, what intelligences do.
Regardless of their underlying implementations, if all you know is that there’s some kind of implementation that gets you an algorithm that does cognitive work, that does outcome steering, then we can reason about properties of that algorithm. And that’s the standpoint. That’s where I’m standing when I tell you things like, yep, a system that just wants to reproduce itself — when it gets smarter than us, it’s not going to suddenly notice the goodness of the universe. It’s just going to steer the freaking outcome of reproducing itself.
Peter 00:39:26
Do you think that it matters who creates AGI? Or do you think that what you just said applies to all of them equally and it’s irrelevant?
Liron 00:39:35
In the current world, I can paraphrase the thesis of If Anyone Builds It, Everyone Dies. The idea that if we proceed to build artificial super intelligence using the technology we have and the training and alignment techniques that we are aware of today, then we will get an AI that kills everybody. Everyone dies. We get that implication.
So to your question of does it matter who builds it? Not in the world today, unless the one who builds it is someone who exceeds humanity’s grasp on a bunch of different principles of how to build an AI that does what we want.
Dario’s Country of Geniuses in a Data Center
Peter 00:40:08
I wanna pivot to a couple of related topics. But before I get there, I do wanna talk about some of the benefits of AI and how — I wanna gauge whether you leave space for being excited about these or whether you don’t think that they’re relevant because doom is such an ever-present thing.
The optimism I think is very often couched in terms of Dario’s recent essay, “Machines of Loving Grace,” where we’re gonna have this super powerful AI that’s effectively, as he said, “a country of geniuses in a data center.” So at your fingertips, you have the best doctors, the best scientists, the best teachers right there at your fingertips, and they are our servants effectively.
Well, maybe I’ll just stop there and ask, what do you think about Dario’s pitch? Is it delusional or is it worth thinking about?
Liron 00:41:21
In some sense, we have a country of geniuses in a data center when we look at AI today. They’re just conveniently subservient to us. They are aligned to us for the most part. People have a bunch of examples like, “Oh, the AI deleted my database,” or, “The AI helped somebody commit suicide.” So there are, of course, exceptions. But 99.99% of the time, they are net good in terms of helping us, and they don’t seem to be rebelling. So we have a country of geniuses in the data center, and it’s worth asking, okay, so how’s the future gonna be different?
I claim there’s going to be a discontinuity where we get systems that are better at steering outcomes end to end. It’s actually very interesting that today AI hasn’t crossed that threshold. I claim that the fact that AI has kind of run out of steam and needs the next instruction from the human — you can’t just tell it, “Okay, make a billion dollars with no input from humans. Manage humans, do everything.” I claim that there will be some kind of discontinuity. There will be enough things getting patched or getting re-architected in the transition from the current world to the world where they’re end-to-end steering outcomes better than a human brain can. Something will snap discontinuously.
There will be another Claude Code or GPT moment, and in that moment, they’ll lose the kind of reliable helpfulness, where you tell it to do a task, but it’s running away and it’ll never be aligned again and it’s making new copies of itself. There’s going to be this whole new regime that won’t feel like using Claude Code or CoWork. It won’t be as nice anymore, which is counterintuitive because I’m positing a change in the graph. I agree today the graph has been good. I’m claiming there’s going to be a discontinuity when AIs steer outcomes better than us.
So going to Dario’s analogy — country of geniuses in a data center. If he’s talking about today’s Claude, then yes, it’s a country of geniuses, of spiky geniuses that are lacking some final robustness core, some final —
Liron 00:43:14
Long time horizon ability. So it’s like a country of conveniently neutered geniuses with a very convenient level of capabilities. And they’ve built something incredibly valuable and useful. The country of almost-geniuses that they have today is amazing.
The future country of geniuses in the data center, that becomes hard for me to imagine because it just feels to me like we’re so close to the limit today of what the current paradigm can do that when I imagine a country of geniuses in a data center that can be autonomous, that can make new countries of geniuses and new data centers, I don’t know how much to extract from that particular metaphor because—
Peter 00:43:51
Mm-hmm.
Liron 00:43:52
Are they really in a data center? I see them as more like sharing the planet at that point, because I see their causal powers as extending outside of the data center.
So I guess in terms of mental model, there’s two mental models. You could describe today as a country of somehow limited geniuses in a bunch of data centers, which I think is accurate. But then in the future, in the next few years, I would just describe it as, well, there’s a bigger country. Step aside United States, step aside China. It’s just a country. They don’t quite have physical brains, but they control a bunch of humans. They’re effectively leading a bunch of humans. I think they’re going to have a firm grip on the US and China, so that’s going to be the real superpower. The mental model of country and data center at that point, I think, becomes misleading.
The Icarus Curve
Peter 00:44:37
Interesting. So yesterday, or early this week anyway, Midjourney announced that they have a new health wing of their company, and you probably saw this. They have this new contraption—
Liron 00:44:52
Mm-hmm.
Peter 00:44:53
That scans your full body in super quick time and super inexpensive compared to an MRI. So MRIs are gonna go the way of the dinosaurs very soon if this tech pans out, and they want everyone to be able to scan their bodies for preventative medicine. If this is true, it just skyrocketed into efficiency.
I’m excited about this. I think this is super awesome, and this is directly a result of an AI company. Midjourney is just an AI company, and it’s the power of AI that helped them even know this was a possible thing. Are you excited about technologies like this as well?
Liron 00:45:37
Yeah, absolutely. I alluded to this before, but if I were to extrapolate economic growth, my economic forecast is what I call the Icarus curve. It’s basically Icarus’ trajectory flying toward the sun. Icarus flies higher and higher and higher, but then the wings melt off and he does a 180-degree turn and he plummets into the ocean, and that’s game over.
But then people will look at me and they’ll be like, “Hold on, Liron. I don’t understand. Can you show me how the world extrapolates into doom?” And I’m like, “You can’t extrapolate. I’m telling you there’s a future discontinuity coming when AI steers outcomes better than humans.”
But many of us just feel the intuition like we’re on the upward curve. We feel like Icarus feels flying higher. I completely agree that the closer we get to the sun in this metaphor, the better it feels, and I do think we’re getting closer to the sun. The current trajectory has been very good. I would be super bummed if I couldn’t use the latest AI technology. I’m happy to admit that. And if I were in the hospital right now, I’d wanna use the absolute latest AI to assist my doctors. Don’t get me wrong.
It’s just that there is this dynamic where flying too high is going to melt our wings off. In this case, it’ll just be AI running out of control. So every moment until that moment that AI runs out of control, I’m expecting positive benefits. And I agree that this Midjourney cancer thing—there’s a good argument from Scott Alexander. He published an analysis last night, which I think makes a lot of sense. So maybe it won’t be a huge breakthrough. Maybe it’ll be incremental, maybe it’ll be niche. But the general trend where we’re using AIs to help this way in medicine, I think is fantastic.
Meaning and Purpose in the AI Era
Peter 00:47:17
I wanna ask you one last high-level question. I feel like technology, as of this moment in time, if there’s one net negative, I feel like we’ve lost a sense of purpose and meaning in our lives, and I think this can be tied to technology in various ways. You can just say the smartphone. Noah Smith just says the smartphone is to blame for a lot of this.
Liron 00:47:42
Yeah.
Peter 00:47:42
And I don’t see a technological solution on the horizon for us regaining our sense of meaning and purpose that we used to have, like in the ‘90s, the glory days of the ‘90s.
Do you think there’s any hope for us to become an optimistic species again and have a robust sense of meaning and purpose, or have we been robbed of that?
Liron 00:48:10
Well, if you were to plot the average person’s level of meaning and purpose over time, I’m not sure there’s a shared truth on when that plot goes the highest and the lowest. There’s the premise that it’s all about the ancient tribes—they had the most meaning and purpose. But I’m not convinced about that.
I can tell you I personally would rather be alive today than any other point in history. Certainly, I would hate to be alive before the internet. Then I’d feel like I’m in Back to the Future, like, “We have to build the internet or we’re gonna be stuck here forever.”
Peter 00:48:46
Yeah.
Liron 00:48:46
In a boring time. And now I feel that about AI. It’s actually crazy to think about going back a few years where I can’t ask AI everything. Boy, I’d have to raw dog Google.
In terms of lack of meaning and purpose, I think Nick Bostrom makes this argument well in his latest book, the one that’s talking about utopia—how do we do utopia because we’re losing all the problems we have to solve, and problems seem to give structure to our day, if not meaning. They’re like an exoskeleton for us. He makes a great point. I don’t have a lot of insight on that point personally, except that I’m optimistic that we can solve the problem of giving ourselves enough problems and challenges. I suspect that the problem of not having problems is ultimately an easier problem than problems.
Wrap-Up
Peter 00:49:32
Liron, this is a fascinating conversation. There definitely is much more to talk about, and I wanna direct everyone to your channel and to check out the conversations and the debates that you’ve had on Doom Debates. They are all worthwhile. Highly recommend. And thanks again for coming on the show. I loved this conversation. It was really great.
Liron 00:49:50
My pleasure, Peter. Yeah, I hope this is helpful for people who are onboarding to the idea of AI doom.
Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.
Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏
