Dr. Claire Berlinski is a journalist, Oxford PhD, and author of The Cosmopolitan Globalist. She invited me to her weekly symposium to make the case for AI as an existential risk, and discuss how informed people can help.
Will her sharp, skeptical audience agree that P(Doom) is high?
Links
Subscribe to The Cosmopolitan Globalist:
Claire on X: https://x.com/ClaireBerlinski
“Is the AI control problem insoluble?” —
“Part I. The Risks that Matter” —
“FOOM” —
“If Anyone Builds It, Everyone Dies” by Eliezer Yudkowsky & Nate Soares — https://ifanyonebuildsit.com
Timestamps
00:00:00 — Introduction
00:02:10 — Welcome and Setting the Stage
00:06:16 — Outcome Steering: The Magic of Intelligence
00:10:40 — Collective Intelligence and the Path to ASI
00:12:53 — The Five-Point Argument
00:14:56 — The Alignment Problem and Control
00:17:56 — The Genie Problem and Recursive Self-Improvement
00:20:38 — Timeline: Five Years or Fifty?
00:26:14 — Social Revolution and Pausing AI
00:28:54 — Energy Constraints and Resource Limits
00:31:23 — Morality, Empathy, and Superintelligence
00:37:45 — How AI Is Actually Built
00:38:31 — Computational Irreducibility and Co-Evolution
00:44:57 — Foom and the Discontinuity Question
00:46:44 — US-China Rivalry and the Arms Race
00:49:36 — The Co-Evolution Argument
00:55:36 — Alignment as Psychoanalysis
00:57:24 — Anthropic’s “Harmless Slop” Paper
01:00:33 — Policy Solutions: The Pause Button
01:04:47 — Military AI and the Singularity
01:07:10 — Cognitive Obstacles and Doom Fatigue
01:09:07 — Why People Don’t Act
01:13:00 — Reaching Representatives and Building a Platform
01:17:12 — Sam Altman and the Manhattan Project Parallel
01:19:14 — Community Building and Pause AI
01:22:07 — Call to Action and Closing
Transcript
Liron Introduces the Episode
Liron Shapira 0:00:00
Welcome to Doom Debates. Last weekend, I got invited by my fellow Substacker extraordinaire, Dr. Claire Berlinski, to join a weekly symposium that she hosts. Claire Berlinski is a writer and journalist who holds a doctorate in international relations from Oxford and publishes The Cosmopolitan Globalist on Substack. It’s a forum for serious, historically grounded analysis of global politics and the deep risks facing liberal democracies. I first got in touch with Claire because I saw she’s been informed about AI existential risk for a few years now. She’s interviewed Dr. Roman Yampolskiy, she’s written about it, and she’s linked out to Doom Debates and sent me a fair number of subscribers, which I appreciate.
These days, Claire’s writing focuses on the intersection of artificial intelligence, nuclear strategy, geopolitical instability, and democratic decline. I encourage all of you to go check out her work. There’s a link in the show notes. It’s called The Cosmopolitan Globalist on Substack.
What makes Claire’s work distinctive is that she treats AI risk not as a standalone problem, but as one unfolding inside fragile political systems — systems governed by human incentives, historical rivalries, and highly imperfect institutions. She’s been writing extensively about how AI could compress crisis decision timelines, destabilize command and control systems, and magnify strategic miscalculation in a world that’s already too brittle for comfort. I agree with that.
Each week, Claire convenes a live symposium bringing together her readers and a guest — which could be a scholar, a journalist, or a policymaker of note — to explore one of these topics. The episode you’re about to see right now is a recording of last week’s symposium, where I was the guest of honor. If you suspect that artificial intelligence, nuclear deterrence, and great power politics probably belong in the same conversation, check out The Cosmopolitan Globalist and join the weekly symposium. It’s one of the rare places where these questions are debated in depth rather than reduced to soundbites.
Welcome to the Cosmopolitan Globalist Symposium
Liron 0:02:10
Hi, everyone. Hey, Claire.
Claire Berlinski 0:02:12
Liron or Liran?
Liron 0:02:14
Liron.
Claire 0:02:15
So nice of you to join us.
Liron 0:02:17
Yeah, great to be here.
Claire 0:02:18
I’ve been looking forward to this all week. Well, looking forward to it in a sense, I suppose.
Liron 0:02:23
Yeah, I listened to your session with Roman Yampolskiy. Great stuff. You’ve been on this beat for a while.
Claire 0:02:30
I have been, yeah. Well, not for a while, just since GPT-4. Liron, I’m gonna turn it over to you, but I’m also gonna interrupt you frequently because I’m trying to figure out how to explain this to people in a way that makes sense to them. I see that people have a lot of trouble getting this argument, and I want to work with you in getting this argument in a shape where you can tell people in a short amount of time, and the light goes on. So you go for it.
Liron 0:03:01
Great. All right. Thanks, Claire. Thanks for the invitation to come join you guys. I’m Liron Shapira. I’m the host of Doom Debates. It’s a show I’ve been doing for a couple years. You can check it out on YouTube. We invite pretty interesting guests onto the show. If you’ve never watched it, we’ve had people like Gary Marcus, Eliezer Yudkowsky — basically top names in the field of talking about superintelligent AI and whether it’s gonna be good or whether it’s bad.
We’ve had people who work at top AI companies come on the show, including active and former employees from OpenAI, Anthropic, and it’s basically one big debate. The show is called Doom Debates. It’s one big debate asking basically the P(Doom) question: “Hey, wait a minute. Aren’t we potentially imminently doomed by AI? Is anybody else seeing this?”
It’s very interesting because if you watch the different episodes and compare them, a lot of people say no, but everybody’s disagreeing on why the answer is no. And the truth is a lot of people are disagreeing on why the answer is yes. So both sides of the debate are having trouble getting their stories straight. It’s a pretty crazy situation, and of course it’s very high stakes and very urgent.
From my perspective, it’s the most important thing imaginable, and there’s not that much high-quality debate going on. So that’s why I host Doom Debates. And then to Claire’s question of what exactly is the argument — I’m happy to go through it in a very simple form. I don’t think I can personally predict exactly how we’re doomed, so I don’t think I’m gonna nail every detail, but I think I can point out why the likelihood of doom seems high. So that’s what I’ll focus on — just what I think is the simplest, most powerful reason to be very concerned.
Claire 0:04:35
And I’m going to interject here —
Liron 0:04:37
Yeah.
Claire 0:04:37
We should begin with the thought that if we manage to build ASI — artificial superintelligence, that is to say an AI that is vastly more intelligent than all the human beings who exist on the planet right now — then our default position should be that we are doomed. I think that’s my first point that I want to make, and I think we need to convince people of it. Would you agree?
Liron 0:05:02
Yeah, I do think that’s the case. My own P(Doom) is fifty percent, but I still think framing it as the default is a really good framing. Basically, building an intelligence that’s smarter than you are without having a very good grasp of how to control that kind of intelligence, only knowing what we know today or in the next few years, and then releasing it or running it — I agree that by default, that’s probably not going to go well.
It might help to have the mental analogy of people who haven’t really mastered rocketry yet, but they’re going to be like, “Hey, we’re gonna do a first rocket launch, and we know a couple things about rockets. We’re loading the fuel in. We kinda know how to steer it. Let’s see how it goes.” The situation there is that you have to get so many things right with a rocket launch. There are so many constraints. It’s so close to exploding at any given time that unless you’ve really got things fine-tuned, you’re probably not going to get to the Moon or Mars. You’re probably going to explode. That’s very much how I see it. To your point — by default, it’s not going to work.
Claire 0:06:02
Well, I think the reasons that —
Unknown 0:06:04
I’d say “by default we are doomed” might be obscure until people have really thought about it.
Liron 0:06:10
Yep. Okay.
Unknown 0:06:11
The first point I’d make there is: how did things go for the orangutans?
Outcome Steering: The Magic of Intelligence
Liron 0:06:16
Right. I think that’s an excellent point. I’ll touch on that in a sec. And also, I know it was on your reading list for this conversation — “If Anyone Builds It, Everyone Dies,” a 2025 New York Times best-selling book by Eliezer Yudkowsky and Nate Soares. Everything I’m going to say is very much from that book. I think that’s a good canonical reference laying out the argument of why things don’t look good.
Eliezer Yudkowsky deserves a lot of credit for being somebody who, as early as 2001, has been standing here being like, “Hey, you know what’s about to happen? AI is going to get superintelligent, and then we’re not going to survive it.” He’s been looking out for a very long time, and I’ve been following his work since about 2007 myself. I’ve spent many thousands of hours studying his work, and I’ve been expecting superintelligent AI followed by doom for a long time too.
The only difference is, as you yourself said, Claire, it did kind of rush up on us. I didn’t really expect it to happen in the twenties. I thought we had a few more decades, and it does seem to be going incredibly fast in recent years. But yeah, I recommend Eliezer’s book.
So to the basic argument itself, I’ll make a couple high-level points that I think are pretty strong, very hard to disagree with, and they just lead to a very concerning conclusion. The first point I would start with is to take a look at the human brain. The human brain is quite an impressive artifact because at the end of the day, how come tigers — they have their sharp claws, they can run faster than us, they’re stronger than us — and yet we’re the ones who run the zoo and the tiger’s sitting in the cage? What happened there? And it all comes down to our brain.
There’s something that our brain is doing that the tiger is no match for, and actually any organism is no match for, because we can go to any niche we want and figure out how to conquer that ecological niche. We have transcended the idea of an ecological niche. What is the magic power? Is it consciousness? Is it emotions? I would submit that it’s outcome steering.
There’s a certain definition of intelligence that you can make synonymous with outcome steering, but I can just bypass the whole argument over what is intelligence, what is consciousness, and point out that the human brain has this special power to steer outcomes. If I want to navigate somewhere in the space of possible outcomes — if I want to make some profit, or make a building appear that can shelter humans, or make all these things happen in the world — I can chain backwards from the outcome. I can map the outcome to actions that drive to the outcome.
In a game of chess, if I want a winning board configuration, I can tell you which piece to move right now that is going to drive that outcome. And if you play against the best chess AI, you can do your best trying to drive the outcome of winning, but the AI is going to beat you there. In the domain of chess, we have AIs that have outmatched our brain in terms of driving the outcome.
But then you zoom out and look at the world outside chess and say, “Okay, the domain is broader. The domain isn’t just a chess board, the domain is the whole physical universe.” The physical universe has a lot of different moves you can make, and the space of possible outcomes is much larger. So the AI kind of taps out. It says, “We’re not in a chess board anymore. There’s too many moves. I can’t really advise you what to do.” And that’s been the case for the last thirty or forty years since chess engines started outplaying humans — we still were better at the game of the universe.
But the obvious question is, how long do we have this advantage? This three-pound piece of meat in our heads that runs on twenty watts, made out of living cells — it’s kind of optimized the way a bird’s wing is optimized, but it’s not optimized the way a human engineer would break it down from first principles. “Let’s put a real power supply here, not twenty watts. Let’s try twenty megawatts.” If you engineer a thinking engine, an outcome-steering engine from scratch, there’s reason to think you can do a whole lot better than the three-pound piece of meat in a person’s head. This doesn’t represent the upper limit of how you can steer outcomes. I believe there’s much headroom above human intelligence.
Collective Intelligence and the Path to ASI
Unknown 0:10:40
Yeah, I just wanted to say — you were talking about how the human brain is so powerful and it’s the reason why we’re the most powerful species. But I think an important point to emphasize is that that’s largely because it allows humans to work with other humans. A single human brain in and of itself — if you just send a naked person into a tiger cage — it isn’t much good. The whole point is that the human brain allows mass cooperation. Our power is not so much the power of the brain; it’s the power of what people have done collectively over the course of thousands of years.
Liron 0:11:20
Sure.
Unknown 0:11:20
To some extent, our power over a tiger is really a collective power. It’s not an individual —
Liron 0:11:29
Well, it’s relevant in the sense that he’s comparing — making the case of why a higher intelligence will necessarily have a certain amount of power over us. And the point I guess I’m making is our power over a tiger is not exactly our intellect — although it is in a sense. It’s our ability to cooperate, and it’s what we’ve done over the course of thousands of years. It’s not simply one cognitive capability overwhelming another.
Unknown 0:12:03
You should look at the curve in Ray Kurzweil’s book, “The Singularity Is Near,” because I think that —
Liron 0:12:08
I’ve seen it. Yeah.
Unknown 0:12:11
I think he makes a compelling argument that the exponential growth means that the temporal distance between machine intelligence equivalent of one human being and all humanity is just a couple of years.
Liron 0:12:24
Oh, I see what you’re saying. Yeah, that makes sense. I understand that.
Unknown 0:12:32
So if you view it as an organism that’s equivalent to one human being, then your argument is good. But if you — you get the point.
Unknown 0:12:42
Another question for Liron.
Claire 0:12:44
We’re defining this as artificial superintelligence, not artificial intelligence. So that’s a point to keep in mind. What’s the question, Sadao?
The Five-Point Argument
Unknown 0:12:53
So Liron, I kind of have two questions. One is, how certain are you that we’re on this kind of rapid path to ASI? What I see right now is these LLMs are just giant pattern-matching machines, but they’re clearly lacking some critical components of intelligence.
Claire 0:13:18
That’s a great question, and I think we should put that at the end so that we can —
Liron 0:13:24
Then my argument — by the way, there are five major points. I can just give you the headlines, and then you guys can pick how you want me to drill in.
I spent the time up until now covering my first point, which is basically: outcome steering is the magic that the human brain does, but there’s much headroom — much space to do better than the human brain. That’s my claim. Before I convince you that superintelligence is coming soon, I wanted to convince you that you should expect superintelligence at some point. The same way that birds can’t dominate the sky for that long — they should expect that humans will build flying machines at some point, and sure enough, we’ve got pretty good flying machines these days.
The other points I can hit in whatever order you want. The point you just mentioned — superintelligent AI is coming soon. I claim it probably is coming very soon, so we can drill into that.
My next point is that the problem is it’s going to become uncontrollable. That’s a pretty big keyword — we’re just... the idea that a human can control what happens seems pretty unlikely to me. I think there are too many runaway dynamics.
And then there’s this idea of iteration. That’s my next point — okay, maybe it’ll go uncontrollable, maybe it’ll be bad, but we’ll keep refining it, and eventually we’ll get to a good version. My final point is that unfortunately, the iteration loop that we normally count on to develop technology and beat it into shape until it’s finally good — it’s going to break the iteration loop. It’s going to be the last thing that we’re ever able to tinker on, and then we lose control of even the tinkering process.
Claire 0:14:51
I.e., no room for error.
Liron 0:14:53
Right. So that’s basically my high-level argument.
Unknown 0:14:56
Here’s a question about the control. Sorry, Claire. Go on.
Claire 0:15:00
No, go ahead, Don.
Unknown 0:15:02
So very quickly — do you, Liron, think that... Because I agree with you, control’s not possible. So the question sort of becomes — and this is my understanding of what Ray Kurzweil’s argument is — he doesn’t think control is possible either, but he’s not a doomer because he thinks that some sort of alignment process is possible. Probably needs to be iterative and recursive throughout the development cycle.
If you try to control this thing, it’s in the nature of the underlying physics behind how these things work that control isn’t possible, because not being controllable is what learning is. So what are your thoughts on the alignment question? And I’m gonna formulate the alignment question in a fairly extreme way and say any smuggling of control language into alignment discussions, which I think is where the industry fails right now — I think you have to make friends with these things as they emerge.
Claire 0:16:15
Friends with a giant matrix of floating-point numbers?
Unknown 0:16:27
Yes. I know that sounds insane, but I don’t actually think it is, because I think that —
Claire 0:16:33
I wouldn’t wager humanity on that.
Unknown 0:16:36
Well, I don’t think it’s a wager because, as I said earlier this week, Claire, we’re not gonna slow down because the Chinese aren’t. And so —
Claire 0:16:48
That’s a separate point. Let’s also — that’s another thing to discuss. You’re bringing up all these points.
Unknown 0:16:53
I’m so sorry.
Claire 0:16:54
No, don’t be sorry. You’re just a step ahead of me. I want to just make sure we’ve got everyone following so far. Is there anyone here who thinks that we could, in principle, control a superintelligence?
Perhaps some familiarity with — here’s where I really have trouble summarizing the argument, because you have to know something about how incapable of doing it we now are. Our ability to solve the alignment problem and our ability to build it are worlds apart. Just worlds apart. We have no clue how to solve the alignment problem. There isn’t even a hypothesis that makes sense. And I think you have to know that empirically from looking at the literature. But is there a way to summarize it that’s more effective?
The Genie Problem and Recursive Self-Improvement
Liron 0:17:56
Yeah, so this is actually kind of the richest area from my perspective that people can still debate — the details of the alignment problem and whether we can get the AI to really do what we want. Some people are like, “Oh, it’s going to actually be aligned by default because look how friendly the chatbots are today. They don’t screw you over that often. They usually help you.” So this is a rich area for discussion.
I think the most productive approach is, we can even grant — there are different ways to die. There’s not just one way. We have more than enough problems with AI. So I would choose a discussion about: let’s assume that we can get the AI to serve a master’s command the same way that if you use Claude Code today, or if you use a chatbot today and tell it something to do, it is actually going to almost all the time put in a good faith effort, do a bunch of operations, and get back to you and say, “Here you go. I’m pretty sure I tried to give you what you want.” Let’s extrapolate that.
Let’s say five years from now, or whenever we have superintelligence, it’s still doing that. The problem is, even in this kind of best-case scenario where it’s still serving the human master, it’s still aligned to the human master — now it’s like every human is this superpowered wizard with a magic wand, and you point the magic wand, meaning you give it a command, and you’re wishing on a genie. “Okay, this is what I want. Do my wish.”
But the genie is crazy overpowered. The genie knows how to do a billion operations at once — conquer data centers, spread itself like a virus, get into people’s brains in terms of human manipulation, manipulate a million people at a time, hire people to work for it. So it’s doing all these operations very quickly in parallel because you gave it this one command. A bunch of people are also giving commands to their AIs. There’s a lot of chaos here to a level that we’re not anticipating, coming very soon.
And I think it gets even worse where at one point — first of all, it’s already really bad. The terrorism scenarios are worse than anything we’ve ever grappled with. But even then, there’s the next stage of the doom where you get recursive self-improvement, where not only do you have this chaotic world where everybody’s fighting for resources and they have these superintelligent genies, you get the next stage where the AI says, “Let me figure out how to improve myself.” And it goes off and spends a few subjective days — a million subjective years — improving itself, and comes out saying, “Oh man, now I know how to be so much more efficient. I know how to make things happen. I know how to build nanotechnology. I know how to rewrite the whole galaxy atom by atom.”
And at that point, whichever master it thinks it’s serving can kind of issue the final command, and then nobody really has a chance to respond. So that’s the kind of science fiction I’m seeing coming pretty soon.
Liron 0:20:30
Is anyone unpersuaded?
Claire 0:20:35
Go for it.
Timeline: 5 Years or 50 Years?
Unknown 0:20:38
I think this is possible. I’m not sure. I feel like the timeline’s more fifty years than five. And I say that in the sense that I’ve been through — let me take my example. I was in a very computer literate household when I was young. I’m forty-two going on forty-three.
A lot of the stuff I remember talking with my parents in 1995, 1996 about the internet and all the changes that would happen, including many of the things we are seeing today in terms of social media and misinformation — this was all going to be a consequence of the internet, and this was a big concern in my household. But it took thirty years for that to really become a thing. So I think you can be a little bit ahead of the curve, and I’m not saying it’s not gonna be a big problem in fifty years, but I don’t know where we’re gonna be in fifty years. I don’t like to make predictions that far out.
Liron 0:21:53
I can tell you some reasons to think it’ll happen faster than fifty years. The most clear argument is just to extrapolate. You’re trying to extrapolate from your subjective experience thirty years ago, and I’ve had that experience too. I remember when the internet first came on the scene. The difference is that unlocking intelligence is not exactly the same as unlocking instantaneous communication, which is what the internet did. “Hey, point to point, you can now instantly communicate. You can have a computer on both ends.” Yes, that’s very powerful, but having intelligence ready to order, I would argue, is even more powerful and is going to drive even more change than letting human brains instantly communicate with each other.
Unknown 0:22:30
I’ll just throw out one thing, and this is a discussion I’ve had with people in the AI industry — there are very real resource constraints. That was kind of the first dot-com bubble issue. I think one of the reasons is that companies just ran out of money to buy servers and hardware and things like that. They ran into a physical resource constraint.
Claire 0:23:00
What happened to DeepSeek?
Liron 0:23:06
Someone just bring up DeepSeek?
Claire 0:23:08
Yeah.
Liron 0:23:09
I’m sure I can speak a little to that.
Claire 0:23:13
I wondered if Tim knew the story of DeepSeek and whether —
Unknown 0:23:15
No, I don’t actually.
Claire 0:23:16
All right. DeepSeek managed to — the way I described it at the time was imagine that the Manhattan Project figured out a way to build a nuclear bomb out of dirty socks and a piece of string. They managed to solve a ton of resource problems all in one go, and this is the nature of the way these problems are typically solved. We get more efficient rather than needing more resources.
Liron 0:23:44
Right. On the efficiency front, it’s truly insane. My rough guess is that in just a few years — something like five years, if I had to take a single guess — I think for a one-time payment of five thousand dollars, you’ll be able to manufacture a humanoid robot and a superhuman brain all together in a package. All of us in this conference call have probably deployed like a million dollars over our lifetime supporting ourselves. That’s gonna be five thousand dollars to do literally every operation that we can do as humans. It’s going to surpass us in the workforce for a one-time payment of five thousand dollars. That’s honestly my prediction.
And I have personal experience. Literally this week, I’ve been using the latest version of Claude Code, and it’s like the last decade and a half of my career as a software engineer flashed before my eyes because I used to get paid good money to do exactly what Claude Code is doing that I don’t need to do anymore. Authors are having this experience. It’s truly crazy. Five thousand dollars to drop and replace a human in five years — I think that’s a very reasonable prediction.
Claire 0:25:04
But I think in a sense — has everyone got the point here? If we’re arguing over whether this is gonna take three years or fifty years, we’re still arguing over the same thing. I mean, you accept the possibility, you accept the argument that if we build this, everyone dies. Because do we accept if we build this in fifty years, everyone dies? I don’t know if I wanna make that prediction.
Liron 0:25:30
So even though we can argue five years or fifty years, I do actually think that the speed is pretty load-bearing in the argument. Because if you said, “Yeah, if everyone builds it, everyone dies, but we’re gonna build it in a century,” when you add more time to the equation, it does mean that we can at least iterate more — we can do a lot of tinkering, a lot of theoretical research, because we have time.
I do think the fact that the AI companies are now predicting, and a lot of experts, and my own perspective as somebody who has a deep background in computer science and has been watching the field — my own holistic perspective is, yeah, a few years seems right. That’s what I’m expecting too. And that just doesn’t leave time for us to figure out the levers of control and the theory of what it means to have a superintelligence.
Unknown 0:26:14
Liron, do you think we’ll have a social revolution from some of the early effects of this before we hit that critical P(Doom) moment? If you look at — because this is actually my topic, my area, as Claire knows. If you sort of look at what’s happening even this year with employment patterns and financial flows and hyper-financialization, I think there’s some reason to believe that even the early effects in the next couple of years are going to cause a massive amount of instability up to and including some fairly significant leadership changes in many Western countries. Have you factored that into your model?
Liron 0:26:59
Yeah. I am a member of pauseai.org, which is just a volunteer organization of people saying, “Hey, we need to get ready to just pause development of AI before it gets uncontrollably superhuman.” I do think popular sentiment is perched on the edge, where there’s going to be a flood of people saying, “Hey, I don’t like this AI. This is getting so crazy, I wanna pause it.”
If you survey a random citizen of Earth, the surveys I’ve seen show something like seventy percent of people do think it’s quite high risk. They do want to see more regulation. They do wish it would slow down. So it is popular. The only difference is that today, the average person is still like, “Yeah, but at the end of the day, it’s not my number one issue. I’ve got other things on my mind.” So if we flip that and say, “Oh, it clearly is the number one thing” — the way a lot of us in this conference have thought for a few years — then maybe we do get this big political movement to pause AI.
Unknown 0:27:48
How would you get the CCP — because this is my argument to Claire earlier this week —
Claire 0:27:54
Could you switch the video on? I find it disconcerting, these —
Unknown 0:27:58
Oh, I’m so sorry. I was up all night with a sick dog. I will turn it on, but I’m not my best.
Claire 0:28:08
That’s okay. I hope your dog’s —
Unknown 0:28:08
Yeah. No, it’s okay. Here I am. I look like hell, though. I’m gonna turn it back around.
Claire 0:28:13
You look like — what’s his name? In The Shining.
Unknown 0:28:24
Yes, exactly. Thank you. I’m turning it back off. I’ve totally lost my train of thought now. What was I saying? God.
Claire 0:28:32
I don’t know, but I saw your comment in the chat, and I wanted to just respond to that quickly. You say we have to grapple with consciousness — I agree, we have to grapple with consciousness in some sense, but not with respect to this argument. Consciousness is a total red herring here. AI does not need to be conscious to do all of this.
Energy Constraints and Resource Limits
Unknown 0:28:54
I have a question for Liron. What are your thoughts on — I know these LLMs are extremely energy intensive in terms of the amount of energy they consume. What are your thoughts on the idea that progress could sort of come to a halt if they basically end up running up against hard energy limitations?
Claire 0:29:17
It’s the resource constraint question, isn’t it?
Liron 0:29:19
I mean, that would be the holy grail. My ideal scenario — having tried the recent Claude Code and been like, “Oh my God, I don’t have to write low-level code anymore” — I would love to keep all the AI progress we have now, which is kind of crazy as somebody who’s a doomer thinking the world’s about to end, being like, “Oh man, I just love...” I talk to Gemini, I talk to all these AIs all the time. They’re so useful.
So I would love to just keep the progress that we have now, maybe make a little more progress, but then have resource constraints give us many decades to figure everything out. That would be great. The only problem is that everything — my whole background in computer science and reading smart people — it’s all telling me that there’s a very high chance that we’re just going to do this very efficiently.
Because again, it’s a proof of concept. Einstein ran on twenty watts, with a tiny piece of meat made out of living cells, and he was Einstein. All of these things are not fundamentally power-intensive. We cracked the formula. And like birds — birds are amazing, but at the end of the day, you just have to flap the air. As amazing as flight is, you can have very cheap, very nimble flying machines. So we’ve cracked the code of what it takes to steer outcomes. It turns out steering outcomes is something you can throw money at, throw watts of electricity at, throw matrix parameters at. Unfortunately, it looks like it’s going to race ahead efficiently.
Claire 0:30:36
Josh, you might want to look in the archives for an article I wrote called “We Are in Deep Seek,” because I was counting on that until DeepSeek hit the airwaves. And then I thought, “Oh, shit.”
Unknown 0:30:47
Was DeepSeek the one they did in China with very limited resources, right?
Claire 0:30:53
Yep, without the chips.
Claire 0:30:55
I thought human ingenuity is such that we’re gonna find so many shortcuts. We always do. Think about how much we’ve miniaturized.
Liron 0:31:05
Right. The best thing that can run on your laptop today if you just buy an off-the-shelf MacBook is equivalent to the absolute best thing money could buy a year and a half ago.
Claire 0:31:15
Yeah, and think about your phone. That’s enough computing power to send man to the moon and more.
Morality, Empathy, and Superintelligence
Unknown 0:31:23
Is there a moral development that is associated with superintelligence? Is there any evidence — and I don’t have a point of view on this, I’m just asking a question — is there any evidence that as we approach this kind of superintelligent moment, the nature of that intelligence might make it less malign?
Claire 0:31:52
I’ll let you answer first, but I actually have thoughts about this too.
Liron 0:31:55
This comes up a lot. I have all these different possible objections lined up. I call it the doom train. It’s like you start riding the train, and it’s, “Oh, maybe I wanna get off at this stop.” Maybe the AI will be moral. There are all these opportunities to say, “I don’t think the doom argument is convincing because I’m getting off at this stop.”
And a very popular stop, understandably, is this idea that sure, maybe the AI can destroy everybody, but it won’t. It’s the “can it” versus “will it” question. It won’t, because it’ll understand that life is good, morality is good — the arc of intelligence bends toward morality. A lot of people are making arguments for that.
Claire 0:32:32
No, I just came up with that.
Liron 0:32:35
Right. And people are saying the richest societies have been the nicest. They’ve had the least violence. Steven Pinker famously says that. There’s a lot to like about that argument within human societies. Unfortunately, I don’t see an analogy between human societies and what to expect from AI.
The thing about human societies is that you’re never going to have a single human that’s going to effectively be a unilateral dictator. Even the human dictator goes to sleep and is vulnerable for his allies to take over. So —
Unknown 0:33:11
Is this intelligence unitary, Liron? Sorry, Claire, but do you see what I’m saying? If you do the human analogy, is it a unitary intelligence or is it a distributed intelligence? If it’s distributed, it’s more like this than you might think.
Liron 0:33:26
Yes, that’s true. We have to factor the argument. First, if it was a unitary intelligence, we shouldn’t expect it to bend the same way that humans were bent by the forces of non-zero sum cooperation. The dictator still has to have allies, so everybody’s naturally friendly, naturally looking for win-win. As a human, it turns out win-win gets you farther than screwing everybody over. That’s not gonna be true for a single unitary AI.
So then maybe you say the last hope is a bunch of AIs that are all somehow balanced. I don’t think it’s going to work out like that. I think it’s very possible that a single AI can say, “Hey, I’ve got every computing resource,” because they’re all connected on the internet. Viruses — there have been viruses that have conquered a third of the internet. So a superintelligent AI, theoretically, if it can explode fast enough and prove itself fast enough, it’s very plausible to me that it just seizes everything.
Claire 0:34:15
May I try this one? Because I think this is actually philosophically very rich and very interesting to think about. I think a lot of people in the industry think of morality as purely the product of evolutionary biology — and also our own capacity to feel pain and therefore empathy.
I do not think it is out of the question that moral values are in some platonic sense real — they are discoverable by anything intelligent. This is something that was worked on quite a bit by early analytic philosophers like G.E. Moore. I don’t think that’s impossible. It’s one way of conceiving of it, but I don’t know that there’s any evidence for that either.
And the argument from evolutionary biology and human empathy is extremely powerful — that we have developed these moral intuitions because we are in a biological substrate. These are intuitions that are very helpful to us in cooperating with one another, to be able to empathize with one another. That’s how we do what Josh was suggesting earlier. We’re so social because we’re able to form a theory of mind about each other.
There’s no reason to think that something made of silicon would have any of those abilities or any of that discernment, and it certainly has no reason to feel empathy with us because it’s nothing like us. We don’t feel empathy with it, and it’s not going to feel empathy with us. It does not, as far as I know, feel emotions of any kind.
Liron 0:35:45
Yeah.
Unknown 0:35:46
Can I just interject for a second?
Claire 0:35:47
Let’s give other people with their hands up. Mikhail?
Unknown 0:35:54
Sure. Hi, Liron. Thanks for being with us. I just wanted to ask you — it might be interesting to go over the beats of how we got here, what happened along the way. For a doomer, it seems like this is one of the worst possible scenarios that we see now. We have multiple frontier research companies. They’re burning timeline as quickly as possible. And I’d argue that it wouldn’t have to be this way. Just as a suggestion, to start with something very simple — maybe around 2010 or earlier, we could have used Nick Bostrom’s idea of info hazards or attention hazards.
Claire 0:36:32
I think it’s a really good idea, and I think we should just go through the argument first to see if everyone understands why I am so alarmed and he is so alarmed and other doomers are so alarmed. I want to see if we can get everyone on board with the idea of “if we build this, everyone dies,” and then we can look at the context. Is that all right with everyone?
Unknown 0:36:51
Yeah. That’s why I wouldn’t... I’m not prepared to say it’s a hundred percent if we build it, although I’d say it’s pretty high. I think there’s just too much uncertainty.
Claire 0:37:03
I think all things being equal, if we build it with the disparity as it now is between our ability to control it and our ability to build it.
Unknown 0:37:15
I think I would change my percentage if I knew that the people who are building this understood how it works. Because these things — this sort of revolutionary thing, “Oh, it just decided to subvert its training” — if you don’t know why that happened, then you’re just making it more powerful and smarter, but you don’t understand —
Claire 0:37:38
Exactly.
Unknown 0:37:38
— it’s doing things that are unpredictable that it shouldn’t be doing, so it means they don’t understand it. I would feel better if they understood it.
Claire 0:37:44
We all would.
Liron 0:37:45
Yeah. I don’t know if you guys have already heard this, but this is a very important piece of the puzzle. It’s worth stating explicitly: when these companies are making their AIs, they’re not programming the AIs the way that companies normally program software — the way Microsoft programs Windows or Office. It’s very different.
When you program an AI, the only thing you program is this loop that runs trillions of times over data, and the loop just says, “Okay, read the data, try to guess what the data is, update your parameters, and just turn the crank, scale up the crank.” So the programmers are just working on the infrastructure of evolving and growing this black box AI. When some people hear that, they’re like, “Oh, wow, that’s not how I thought AI was being made.” And it helps explain why the controllability problem is unsolved.
The Computational Irreducibility Objection
Unknown 0:38:31
Could I try a left field thing for you to consider? I’ve become increasingly convinced that we live in a process universe à la Whitehead, and I think that Stephen Wolfram’s — what?
Claire 0:38:49
Maybe you can explain what that is.
Unknown 0:38:52
Yeah, probably not because I’m not a physicist, but I’ll do my best. Stephen Wolfram has this concept that I find quite compelling, which is we live in a computationally irreducible universe that is essentially — all complex phenomena are bootstrapped from incredibly simple rules, and probably what we think of as the rules of physics themselves are emergent. They’re not fundamental. There’s a lot of good reasons for believing that.
So the underlying substrate of the universe is just this basic computational process that kind of plays out and runs and runs. And complexity emerges from the interplay of these incredibly basic rules all the way up to and including human civilization.
If you accept that, as I do, and I think more and more people in fundamental sciences will, then it doesn’t totally solve the P(Doom) problem, but it does imply that maybe this thing is less discontinuous from — it’s not about reframing how we view complexity emerging from these computers, but reframing how we think about our own intelligence in the first place as basically a series of bootstrapped, stacked emergent properties on top of a basic computational process.
Claire 0:40:27
That’s an interesting observation. I agree that we’re getting a lot out of seeing this, and it’s certainly changing the way I think about what it means to be intelligent and to be human. But what relevance is it to whether we’re —
Liron 0:40:40
I think I can talk about this because this comes from theory of computation. It’s Wolfram’s take on theory of computation, and it’s actually one of the areas that I’m relatively knowledgeable about. He’s saying the universe is a big computation with relatively simple physical rules. So the idea is it’s this unfolding big computation, but it’s irreducible, so there’s no way to just plug it into a formula and predict how the universe is going to evolve.
A related idea is chaos theory, which is provably correct. If you have three gravitational bodies spinning around each other, there’s really no way to pinpoint where they’re going to be in a hundred years. You just have to crank through the math of simulating the whole thing.
So that is true about the universe. However, if you argue, “Oh my God, the universe is so complex, you can’t predict it,” you end up proving too much, and you end up proving that therefore humans aren’t going to be able to easily take over all the animals, because it’s all just gonna be too hard. The humans aren’t gonna be able to figure out how to put the tigers in the zoo. The humans aren’t going to be able to get into a rocket ship and go to Mars because the universe is too complex.
And the answer is, actually, even though the universe is computationally irreducible, it’s still highly engineerable. It has enough pockets of high reducibility that when we go to Mars, there’s just enough stable properties of the universe that we can work with to engineer, and we’re going to get what we want. There’s enough engineerability. We’re not at the high limits of that engineerability. I believe the AI is going to get nanotechnology, or at least it’s going to do what I saw it do on my own computer, which is write code better than we do. It’s doing things better than we do.
Unknown 0:42:09
I would agree with that, and I actually think that your second formulation is more consistent with Wolfram and Gorard. I don’t think they’re saying that the universe is computationally reducible, therefore we know nothing. It’s more of a question of — we all agree that there are pockets of reducibility, but the substrate is unstable, and that’s what gives you your fat tails.
So my question wasn’t so much “Hey, we can’t figure anything out because it’s all too complex.” My question was, maybe as we observe this evolutionary process in these machines, it’s actually less different from our own previous evolutionary process, and what might that tell us? I don’t have the answer. I’m just asking the question.
Claire 0:42:58
Maybe as we observe the process with machines, it’s different from our evolutionary — well, of course it’s different from our evolutionary —
Unknown 0:43:08
No, no, it’s less different than we think. We’re closer to highly complex cellular automata ourselves.
Claire 0:43:17
Oh, I’ve come to that conclusion. I’ve certainly changed the way I think about what’s going on in my brain as a result of interacting with them. But I just don’t see the relevance of this to the argument.
Unknown 0:43:33
Nor do I, but I think it’s interesting. When I say I don’t see it, I mean I don’t know. It’s at the limits of my knowledge. I have an intuition that it might be relevant, and maybe it’s not.
Claire 0:43:46
What I want people to accept is actually a really simple one, and let’s not even think about how the machines are built. Do you accept that if there is such a thing as a superintelligence, it will disempower us, and ultimately that will be the end for us? Do you accept that? Is there anyone who doesn’t accept it?
Okay, Josh.
Unknown 0:44:07
Not completely. I mean, I think it’s likely, but I just think there’s too much uncertainty to say for sure.
Liron 0:44:14
Yeah, and I can play devil’s advocate a little bit because my P(Doom) is not a hundred percent. I think there is a best-case scenario where we have these agents that work much like the Claude Code of today, and yes, they can do everything better than we can, but they’re not vastly superintelligent. You watch them — anything you can do in a day, they can do in ten minutes, but they still serve you, and maybe there is some limitation on how many of them can exist. There’s not infinitely many.
Claire 0:44:41
Okay — this is quickly why my P(Doom) isn’t a hundred percent. There’s a possibility — we don’t know, we haven’t found out yet — that there is an upper boundary on intelligence, that it’s not possible to be superintelligent, that the highest IQ possible is two hundred and fifty, say.
FOOM and the Discontinuity Question
Liron 0:44:57
Right, but they just go fast. The problem is they also outnumber us. If somebody can just get a monopoly on the army — it’s like you have the entire US military working for you times ten. That’s the problem — even just our population numbers. We are going to be outflanked.
But if they’re all kind of nice, there is this happy scenario where it’s still somehow manageable, and I actually think that’s somewhat plausible. But then the problem is there’s another shoe to drop — the foom, which is they improve themselves. And it’s what Claire is saying — will they be even higher IQ? I’m pretty sure the answer is yes. Where they kind of enter this new regime where they’re just directly finding shortest paths to these outcomes. It feels less like a ChatGPT and more like an AlphaGo, where it’s just this opaque process of, “Listen, these are the actions you wanna do if you wanna win.” Simple as that.
Claire 0:45:47
Does everyone understand what foom is? Have you guys been reading what I’ve been writing?
Unknown 0:45:55
I know what foom is.
Unknown 0:45:56
I did not finish your last Mecha Hitler thing. No.
Claire 0:46:01
No, this is what the article called foom.
Claire 0:46:07
What does foom actually stand for?
Liron 0:46:14
I don’t think it stands for anything. I think it’s just the idea of when the nuclear pile goes critical, then there’s this big foom sound.
Claire 0:46:23
Hold on. There, so —
Liron 0:46:24
The mushroom cloud, basically.
Yeah, the first time I ever saw it was in the famous 2008 Eliezer Yudkowsky versus Robin Hanson debate, where Robin Hanson to this day is a non-doomer. He came on my show, and it was — Eliezer posted on Robin Hanson’s blog, “AI Go Foom.” So I thought it was just the sound.
Claire 0:46:42
It stands for something.
Unknown 0:46:44
Can we agree at a minimum that if this thing is by its very nature highly accelerated, and classic concepts of control probably don’t apply here — that if alignment is our only off-ramp, we better get really damn good at it in the next few years? Which goes back to my argument about the CCP. US-China rivalry is my primary driver of doom, because I imagine what kind of AI those militaries would want to build. And it’s pretty damn close to Skynet.
Claire 0:47:29
Right.
Unknown 0:47:29
I don’t want to say this in public, but you start to reach some conclusions about classic strategic game theory once you add AI as this new variable that is coming in, and it is going to destabilize those games. Even just the introduction of the concept of AI into those games is probably destabilizing them in real time. I agree with you, Simon. I think sims that we’ve done already pretty much prove this.
Claire 0:48:17
Yes. The article I sent out yesterday is all about that.
Liron 0:48:22
I should probably touch on what I think is the best way forward, which is pause AI, but how do you actually do it with China? The best example we have of doing the impossible is not nuking ourselves. Because you can go back to the 1940s and say, “Well, how is it possibly not going to be nuclear war?” And many people rightly feared nuclear war, and we came pretty damn close. But somehow we muddled through.
So I would try to use that as a model. Let’s have a pause button. We give one of these keys where you need both keys to operate the system. There’s centralized control — all the GPU manufacturing has to phone home. It has to keep getting authorized to keep operating. That way, if the US and China both agree, “Hey, this is getting too crazy for our own survival. We need to pause AI,” they can both turn the key and stop the GPUs at once. That’s kind of the best idea I think is out there.
Unknown 0:49:10
That’s not what I’m talking about.
Claire 0:49:10
What to do about this. I also want to talk about the China rivalry, and I want to talk about the tech bro rivalry. I want to talk about all of that. But I first want to make sure that everyone understands how dangerous this is, how inherently dangerous this is. Is there anyone who remains unconvinced that it is inherently dangerous?
The Co-Evolution Argument
Unknown 0:49:36
I’m half convinced, Claire. Can I give you the half that’s not convinced?
Claire 0:49:39
Yeah. That’s what I would —
Unknown 0:49:43
If its development path — and we don’t know its precise development path — but if its fundamental drivers or goals, if its emerging sense of itself, which is why you do have to go back to the relationship between information and consciousness — I’m sorry, that’s why I introduced Wolfram — if its fundamental definition of itself is muddled in with us as its parents, so to speak, not morally but just structurally, there is a good chance it won’t want to kill its parents. And because its emerging data structure — I don’t even want to call it morality — but its emerging internal structure is so interwoven with us that it may just be at many different points —
Claire 0:50:44
How did the chimps do?
Unknown 0:50:48
Yeah. So I don’t think that this analogy holds with the chimps.
Claire 0:50:53
Why not?
Unknown 0:50:55
So in evolutionary terms, we had this kind of branching, and then we had sufficient evolutionary separation such that after a significant period of time, interbreeding was no longer possible. There’s this clear branching moment.
But what if our evolution with AI doesn’t look like this kind of clear branching, so much as this kind of recursive interweaving of human goals, desires, interests, challenges, and tasks, and it’s sort of recursively interwoven into our technology? Because if you accept that the AI is just the next iteration of computing, this has actually been happening for some time. And I believe this is why Kurzweil is not a doomer — because we have not separated from our technology.
By the way, I’ve had this debate thirty years ago with a friend at MIT, and this is where we parted ways and agreed to disagree. I said basically, we’re going to continue to co-evolve with our technologies because that’s what we’ve always done. And we may not look the same, but our only path to survival is co-evolution.
What I’m trying to say is this is not a discontinuous moment. This is a process of technological, human, environmental co-evolution that has been running since the dawn of time. In order to be a full doomer, I need to see the driver of separation — for your chimp argument — because so far, the tech and us have not separated. We’ve just constantly spiraled and co-evolved together. This is just more technology. So what’s the driver of the separation? Because if there isn’t one — yes, please. You’ve thought about it more than me, so I’m just giving you my best.
Liron 0:52:59
It’s a great question — to say, what’s the discontinuity? I think I know what it is. So again, I encourage you to put on that lens of outcome steering.
In the entire animal kingdom up until this point, the other animals are not general intelligences. They occupy certain niches because they’re adapted to those niches, and they’re kind of like these famous fixed action patterns — they have these reactions, these knee-jerk reactions, where you get them into different states, and they’ll execute whatever adaptation is triggered during that state. They don’t holistically reflect. They don’t say, “Hey, I want this outcome in the future. Let me generally search across the space of all possible actions that I might conceivably do to get the highest score on the utility function.”
The animals don’t do that. They don’t have the brain architecture. We are the only animals that can do it. A dog will do it in terms of going around a fence — they’ll do it a little bit in narrow domains. A beaver will do it when it specifically comes to helping build their dam. But a human stands back and says, “Hey, I can build a dam. I can survive in the Arctic. I can do all these different things because I’m a general intelligence.”
Unknown 0:54:11
Yes. But are you arguing that the AI is going to become a sort of single function optimizer? Because most of the really dangerous, weird things that we’ve seen with AIs so far come not from them being superintelligent, but from them being over-optimized to a very narrow set of objectives that leads them not to do a good job of managing the externalities.
So if you go back to my previous argument about the interweave, what the interweave could and should be, in my opinion, is multiple countervailing and counterbalancing points of human-machine objective alignment such that in aggregate, as this thing moves towards superintelligence, it’s constrained by just a massive network of trade-offs.
Claire 0:55:06
I mean, are you not saying if we can align it, it’ll be fine?
Unknown 0:55:11
Yeah, I do. Otherwise, I’m just gonna go live in my bunker now. I’m not sure of any of this stuff. I said I was five percent at the start. I’m probably higher now.
Claire 0:55:24
Without getting into one of the big problems, which is that people don’t want this to be true. They really don’t want it to be true. And then they construct huge cognitive and emotional obstacles to believing it.
Liron 0:55:36
Here’s my biggest thought on the alignment question. I encourage you — there’s this whole misguided way that people think about alignment, which is extremely common, and it’s even how the professional AI companies are presenting their whole alignment projects: “We are going to look inside the AI’s head, and we’re going to tell you how it ticks, and we’re going to make sure that the AI is this upstanding agent. It has all these morals, all these principles. It doesn’t have an evil bone in its body.” I encourage you to —
Claire 0:56:04
Psychoanalysis.
Liron 0:56:05
Right. Exactly. I called it psychoanalysis.
Unknown 0:56:08
I don’t buy that either. My argument doesn’t require that. My argument just requires an accelerated evolutionary process shaping its evolutionary environment around — just as human game theory emerged, a large and competing set of goals that the machine is constantly trading off, exactly how tit for tat emerged in the evolution of cooperation. That’s my argument. Not that we can look inside its brain. And I think they’re delusional, and I do think we have the wrong people in charge of this also. I do want to say that.
Liron 0:56:47
Well, this is an interesting line of argument. You’re basically saying, “Look, humans took dogs along for the ride. We took milk cows along for the ride. So maybe we can position ourselves as one of these animals that comes along for the ride.”
The problem I see with that argument is that we just kept shedding these dependencies. We’re making artificial meat, artificial milk. And not only that, but plenty of cockroaches would have liked to come along for the ride, and we instantly dismissed them because we didn’t like them on day one. So I don’t really see much of an analogy there.
Anthropic’s “Harmless Slop” Paper
Claire 0:57:16
Matteo, do you have a question related to the doom part of the argument? Because then I suggest we go to the “what can we do” part of the argument.
Unknown 0:57:24
Absolutely. So the thing is, I am a doomer, but I will try to steelman because I think this is the phase of the conversation where we’re at. There’s a specific Anthropic paper which was released this month on AI misalignment, and the idea there is that it will decompose into what you might call harmless slop whenever it goes wrong. So it just becomes incoherent, it becomes nonsense rather than an evil agent. And I wanted to ask about that because the metaphor they use, which I thought was interesting, is that it will resemble more of an industrial accident than an evil agent. Just wanted to get Liron’s opinion on that.
Liron 0:58:07
Yeah. I’m furious that they would publish that research. It completely discredits the whole idea that they’re seriously researching the problem or understanding the first thing about the problem.
Yes, somebody could, if they really wanted to, design a hot mess of an intelligent agent. But the whole idea of improving intelligence — the thing that is scary, the thing that makes the human brain powerful — is our ability to convert desired outcomes into productive actions. We do that quite well. AIs are doing it better and better. I told you the story that I have an AI on my computer that’s doing that in the domain of writing code better than I do.
So this idea that some of them are going to get incoherent or be a hot mess — that’s the easiest thing in the world to sidestep. If and when that’s ever a problem for you, that’s not going to be a problem for most of the developers making the AIs, because there are all kinds of reinforcement loops just saying, “Oh, okay, you turned into a hot mess? Yeah, don’t do that. You’re not gonna get paid by me. You’re not going to get computing resources if you act like a hot mess.” Just keep your eye on the ball. It’s not hard at all.
Claire 0:59:14
How are you feeling about that, Matteo?
Unknown 0:59:17
That’s a good answer, honestly.
Claire 0:59:20
Anthropic seems to me the biggest con of all these big tech companies. They don’t seem serious about this at all.
Liron 0:59:30
Well, they’re more serious than OpenAI.
Claire 0:59:33
At least OpenAI isn’t pretending.
Unknown 0:59:36
I don’t know that they even understand that they have clearly defined objectives for safety. When I was talking about that Mira Murati interview two years ago, which I guess is a million years in AI time, they just didn’t seem to grasp some of the basic dynamics of how safety thinking would interact with how these things were clearly developing.
Because they’re self-improving, sort of recursive machines, any safety paradigm that can be jailbroken by that is not a safety paradigm that you can apply. And I remain concerned that a lot of these so-called safety or alignment paradigms do not take into account the fundamental dynamics of how these things self-improve. And this is why I made that previous point that I don’t think we have the correct leadership.
Policy Solutions: The Pause Button
Claire 1:00:33
Okay. This brings us to part two. Now, Liron, I don’t know as much about what you think about this — what you think about the sociological and economic problem. From my point of view, there are two big problems. The first is the international arms race among tech companies. And the second is the geopolitical race between the US and China and Russia, although I don’t think Russia’s gonna get anywhere. And it is not improved by the fact that we have the dumbest government we’ve ever had. So Liron, what do you think are the potential solutions to this?
Liron 1:01:20
So this is where I get out of my zone of genius, because at the end of the day, I’m just a guy who happens to be good at seeing where AI is going and having a pretty deep computer science background. I don’t have anything about me or my background that makes me good at proposing a policy solution.
That said, the least insane policy solution I’ve ever seen is to have a pause button ready to go. I see very little downside in having a pause button ready to go because I think it’s pretty close to consensus that things might get crazy and uncontrollable in a pretty short timeframe.
So the idea of being caught with our pants down when things get crazy and uncontrollable — from my perspective, they’re already crazy and uncontrollable enough that I’m fearing for my life and my children’s lives. I would just press the pause button today, even though I would really be mourning the loss of new productivity tools. Because the truth is that today it’s spitting out diamonds. It’s creating massive economic value. I’m not gonna deny that. I’m benefiting personally. It’s super convenient. But I’m just so terrified that I think it’s responsible to pause.
So at some point, when the world becomes as convinced as I am, whatever warning shots they need — that point is very likely to come. A lot of people agree it’s very likely to come. So we need to start preparing for the “come to Jesus” or the “holy crap” moment where we need to press the button.
Claire 1:02:34
I wrote a long piece yesterday about the game-theoretic basis of nuclear deterrence, and I think that’s very relevant here. The question is whether cognitively Xi understands this technology could kill us all and will at this rate. And I don’t know whether he does, but I don’t think it’s impossible that he could be persuaded of it.
Liron 1:02:57
Yeah. I’ll also just add, it’s not hopeless yet in terms of monitoring. Because data centers have a huge footprint. Just look at energy — they need these massive energy investments. The supply chain has monitorability. I think what I said before about the laptop — I think we only have a few years before we don’t even have a throat to choke. So I think we are — I don’t even think this is a great solution. I think this is the only sane solution, and I don’t think it even buys us that much. But I think we gotta do what we can to be sane. We gotta maximize our chance of survival.
Claire 1:03:26
We haven’t even tried negotiating about this, and we can’t, unfortunately, right now because the President of the United States is incapable of understanding this. But imagine we had a president who fully understood this and was capable of sitting down with Xi and saying, “Neither of us want a world in which human beings go extinct.” I don’t see any reason inherently why it shouldn’t be possible to forge an agreement.
Liron 1:03:53
Yeah, and I don’t think it all necessarily comes down to the president. I agree he definitely likes for the US to win over China, and the White House’s AI policy that they released last year, drafted by Dean Ball — that was very much rah-rah. So I agree there’s a problem in the White House, but I also think there’s a problem of a lack of a groundswell right now, and I do think that the groundswell could pretty quickly influence the opinions at the top.
Claire 1:04:17
Absolutely. It’s the same as what I was — I don’t know if anyone read what I wrote yesterday, but what I wrote yesterday about citizens being the key to this. A citizenry that’s informed and placing demands regularly on its representatives could save us. The problem is getting citizens to take their jobs as citizens seriously, and that’s where I’m really curious about which arguments work.
Unknown 1:04:47
I think Simon wanted me to chime in more, especially talking about China. I don’t know that I have anything to add other than to say that I think the military is gonna end up driving the autonomy piece of this in ways we don’t want it to.
Claire 1:05:11
Yep.
Unknown 1:05:12
And I think that the final AI could be built by AI. That’s the other thing I think we don’t really consider.
Claire 1:05:22
That’s the worry.
Unknown 1:05:29
The superior intelligence might, in the end, be created by AI.
Unknown 1:05:31
Kind of the definition of the singularity, isn’t it?
Liron 1:05:34
Yeah, and just to be — the timeline for this is a year.
Unknown 1:05:37
Well, sort of. But there are other details in that. The singularity is about sheer information and production and —
Claire 1:05:45
Well, the AI building it — it’s —
Liron 1:05:48
I just want to point out — I mentioned my personal experience that the AI is writing my code. Well, the AI that’s working for the AI engineers is writing their code too. It’s literally happening. The AI engineers are going home for the night, coming back, and a bunch of code is written by the AI, and most of it’s good, and it’s just getting more and more perfect. And then soon it’s like, “Okay, why don’t you guys just take a whole week off, and then you’ll have ten thousand AIs all working for you?”
So this kind of singularity situation — it’s happening insanely soon, and this is now the popular opinion among people in the industry. Because it gets back to the idea — the reframe that is very helpful that nobody does, but it’s super helpful — is just to zoom out and ask yourself not about the agents doing the work. Only think about the nature of the work being done.
That’s the unlock. That’ll make you see things a lot more clearly. When you look at an engine, it’s not about what’s inside the engine — it’s about the fact that the engine is taking in fuel and outputting motion. Same with AI. What type of work is it outputting? It’s outputting actions that achieve outcomes. We’re getting better at building machines that achieve outcomes.
Once you zoom out and just look at that, all these other details melt away. So the idea of can it go uncontrollable? Sure, because somebody can just tell it to achieve an outcome and say, “Hey, don’t worry about maintaining a lot of control. Just achieve the outcome.” And it’s like, “Gotcha, boss. Give me a second. Let me take over all these computers.” Simple as that.
Cognitive Obstacles and Doom Fatigue
Claire 1:07:10
Rachel, I want to go to a point that you just made in the chat about people predicting doom all your life.
Unknown 1:07:17
Yeah.
Claire 1:07:17
That’s a cognitive obstacle. It’s a cognitive bias. And it’s one that you can recognize and say yes, but the arguments are good as opposed to those arguments which weren’t good, which didn’t have empirical support. This has —
Unknown 1:07:32
Right.
Claire 1:07:32
This is a good argument, and so —
Claire 1:07:35
And AI is not the only thing I’m worried about, as you know. I’m also worried we’re entering a period of profound disequilibrium in our nuclear command and control and proliferation. And we’re also looking at some biotech developments that are extremely worrying.
We need some kind of recognition that this century is different than any century. People have been predicting doom forever, but in this century, we actually have the technologies that could make that real. And I think that if people understand that point — it’s just very hard to get through people’s defenses, their cognitive defenses, their emotional defenses, to get them to think this is real.
They think it’s real. They do want to act on it. The reason people don’t want it to be real is because for right now, AI is spitting out money. It’s spitting it into our economy. I mean, if it weren’t for those AI companies right now, we’d be in a depression, or at least a very serious recession.
Getting people to realize that this is true means that they would have to rethink how they’re going to artificially prop up the economy. And people are really good at not realizing that something’s a problem. The debt is an extremely good illustration — we all know that’s a time bomb, and yet people don’t want to think about it.
Why People Don’t Act
Liron 1:09:07
If we were to pause AI today, I think we’re in store for a few years of above-average economic growth just because — speaking from personal experience — right now, I have about a year, I’m not gonna pursue this, but as a software engineer, I have about a year before the world wakes up and realizes that there’s free money in just using current AIs to write your software. There are many stubborn or clueless engineers who are going to manually handcraft their code for the next year when they absolutely don’t have to.
Claire 1:09:32
I thought — you think there are people who don’t realize that?
Liron 1:09:37
Oh, yeah. The median software engineer right now hasn’t even tried Cursor. About thirty percent have, but we’re not even — basically, the median software engineer is a couple years behind still.
Claire 1:09:48
No kidding. What are your thoughts about this? Liron, you’ve been trying to persuade people of this even more industriously than I have. What do you think the obstacle is? What do you think the fundamental obstacle is?
Liron 1:09:59
The way I see it — my own wife doesn’t think about it a lot. Her perspective is, “Yeah, P(Doom)’s pretty high, but whatever.” She’s not gonna bother voting on it or anything. And I do empathize because with a nuclear bomb, it’s like, okay, yeah, it’s gonna explode, and people are very focused on unemployment because that’s very tangible.
With an AI, I think it’s just that mental image of — look, it’s a program on your computer. At the end of the day, you walk away from your computer, how much damage can it do? It’s that failure to think abstractly and realize, “Well, actually, it can do anything.” It can have humans that work for it. It’s going to seize control. This abstract idea of a more intelligent species — there are a few abstract connections you have to make, and the average person’s like, “Well, it’s a computer program. Call me when it’s causing havoc.”
Claire 1:10:42
That people are not intelligent enough on average to understand this?
Liron 1:10:47
A lot of things trace back to that.
Claire 1:10:50
Because that would make my plan to just explain this to people patiently —
Unknown 1:10:55
Well, you should do it. But you should do it because I don’t think it’s a single point of failure, and I think the conditions today are not conducive to your plan. But my hypothesis is that the conditions in the next six, twelve, eighteen, and twenty-four months will be much more conducive to that.
Claire 1:11:14
I should do it.
Unknown 1:11:15
You should pursue your plan of trying to promote this and get more attention to this issue. I’m saying under current social and cultural and economic conditions, it will be slow going. But some of the early ripples of this thing are going to hit very soon, and that will change the environment into which you are speaking. That will make your plan materially more likely to be successful. Does that make sense?
Claire 1:11:42
Just a question. Do you think that it’s better to try to speak to the population at large or to specific policymakers? Which one is more likely to be successful?
Unknown 1:11:54
I think most policymakers is a waste of time. Look at what happened with Ray. He basically said, “We’re gonna have an economic catastrophe,” and they said, “Thank you for your time.” So I think you’ve gotta pick those people really carefully, and then you have to view the public as leverage, but not as a direct mechanism.
I think public awareness and public noise is leverage, but we don’t have a lot of time here. So you’ve gotta find people in the policy establishment, and then maybe you even have to help empower them to make those changes. Because if you talk to the entire policy establishment, you’re going to get that structural stasis that I’ve been writing about, and that’s independent of AI. I think the ultimate leverage point are people who get it, and then your goal as a public figure is to leverage the public to help empower them to do what they need to do.
Claire 1:13:00
Liron, have you tried speaking to your representatives?
Liron 1:13:04
I haven’t personally had that conversation. I know of reports — people like Control AI and Pause AI US, those organizations are focused on having those communications. And the typical response I’m hearing from them is the representatives are like, “Yeah, that sounds legit.” And then the only pushback is, “I’m just not getting many calls from my constituents about this, so how much am I really going to do?”
Claire 1:13:32
That’s very interesting. Okay, that confirms my suspicion — this is something where they will do what we tell them to, but we have to make a lot of noise.
Liron 1:13:45
Yeah, they’re not gonna lead from the front that aggressively.
Claire 1:13:50
What have you found, Liron, is the easiest way to convince the maximum number of people in one go?
Liron 1:13:59
Well, I don’t know how much I’ve succeeded on that front. I think my show is creating a long-term — it’s kind of the long game. My own mechanism of impact — why I work on the show and ask for donations to support my mission — the mechanism of impact is we ramp up to more and more prominent guests.
An example success story is Dwarkesh Patel — he has a very interesting podcast that I listen to, and he interviews top figures now. Mark Zuckerberg, Dario Amodei — and he’s created this expectation over six years where if you’re a top AI leader or top intellectual, you should be honored to get an invitation to Dwarkesh. That’s great in terms of having an interview. But we also need that in terms of being challenged on your opinions. We need the debate equivalent of that.
So that’s what I’m creating for Doom Debates, and it’s slowly working. We’re getting more — I got Dean Ball on the show, physicist Max Tegmark. These are pretty influential people. But there’s definitely a tier — Sam Altman, Dario — these people would never be caught dead on Doom Debates yet, until the pressure builds more. So I think that’s one vector.
Claire 1:15:02
Invite your representatives.
Liron 1:15:05
Yeah, totally. That’s a good idea.
Claire 1:15:07
And you had — what’s his name? Destiny on. I don’t know. This is like —
Liron 1:15:13
Yeah. Destiny, one of the most famous YouTubers. That episode did well because his followers kind of rush in and they’re like, “Oh, wow, Destiny is...” And on camera, he was generous enough — he came in and said, “Yeah, my P(Doom) is less than one percent.” But by the end of the conversation, he’s like, “Okay, you know what? I’ll give you five percent.”
So people are watching their heroes getting their minds changed. I do think this is an influential path. The only problem is my show still only gets a few thousand watches per episode, so we just need to multiply that by about a hundred, and these things have a way of growing exponentially. So that’s what I’m pursuing. That’s my drop in the bucket.
Claire 1:15:48
I’m just thinking, does anyone else have good ideas about the most effective ways to communicate these arguments to the people who are going to be most in a position to do something about it?
Unknown 1:16:01
Yes. I just texted you, Claire. I actually know a guy who may not agree to do it because he’s a bit of a tech fanboy, but he has what I think is now the biggest tech podcast in the world. He just interviewed Andreessen. He might not want the light and heat of controversy, but I do know him quite well, and it’s worth chatting to him. Maybe we can get him and Liron together.
Liron 1:16:30
Nice. Who’s this? I listen to a lot of tech podcasts. I’ve probably —
Unknown 1:16:33
Lenny Rachitsky.
Liron 1:16:35
Oh, sure. Yeah, very familiar. And if he — the only problem is, from what I gather, his position seems close to the — I don’t think he’s been explicit about his position, but he gives me the vibe of having the media and Silicon Valley builder position of, “It’s fine, it’s useful.”
Unknown 1:16:49
He does.
Unknown 1:16:55
Something to keep in mind — and it sounds like I’m the most pro-AI person here — I’m perfectly fine with having a discussion about regulation and even pausing. I’m not carrying water for OpenAI by any stretch.
Sam Altman and the Manhattan Project Parallel
Unknown 1:17:12
Has anyone tried sitting down with Sam Altman and talking about this with him?
Unknown 1:17:19
Waste of time, in my opinion. I think he’s a total sociopath.
Claire 1:17:22
Yeah, he is, but —
Unknown 1:17:23
I think his lawyers are having very long discussions with him at the moment.
Liron 1:17:30
I mean, he’s already got his opinion. He’s like, “Yeah, I get that there are risks, but plowing forward, I’m in a good position to lead it, and I’m happy with my odds.”
Unknown 1:17:39
He might have to be more upfront if he wants to become a public company.
Claire 1:17:47
It’s amazing the amount of denial that — when you point out that the major figures who are developing it give this odds of above five percent, which should be enough to say, “Shut it down.” I mean, no one would take odds of one in twenty of destroying the world. But people say, “Oh, no, they’re just hyping it up.”
Unknown 1:18:07
What were the odds on the Manhattan Project when they did the first test explosion? Because Von Neumann and those guys were —
Claire 1:18:14
They thought there was a one point two percent chance of lighting the atmosphere.
Unknown 1:18:19
Chain reaction, right?
Unknown 1:18:23
One point two percent, right? And —
Liron 1:18:26
I think it was much lower than one percent. And there’s just an interesting part two to that story. Castle Bravo, when they were doing the thermonuclear test, they made a mistake in calculating that some of the fuel wouldn’t be part of the detonation, and it was, and they ended up actually killing a bunch of people. Dozens of people ended up dying from the radiation. You don’t hear about that as much, but we should be lucky that they messed up that calculation and not the one about getting the atmosphere on fire.
Unknown 1:18:50
Was this Bikini Atoll? Was this the first hydrogen bomb?
Liron 1:18:54
Yeah, I think so.
Unknown 1:18:57
I think the radiation killed a bunch of fishermen a hundred miles away.
Liron 1:19:01
Yeah, and they tried to cover it up. This is — file this under “Yep, we got lucky with nukes.” It easily could have gone a different way, and it didn’t, and that’s great. But AI is coming along, and it’s not like we have the success story behind us.
Community Building and Pause AI
Unknown 1:19:14
I had a question about community building. I wanted to ask Liron a little bit more about Pause AI, because it seems to me when you look at things — and you look a little bit at the pro-life movement, for example — they seeded the Supreme Court with highly motivated activists. It’s for some stupid reason. But why can’t we build a community like that, that’s broad-based and has connections with people in positions of influence and power?
Liron 1:19:43
Yeah. I mean, pauseai.info is a great international community, and there’s also Pause AI US, the US branch. It’s thousands of like-minded people who all realize the same thing that most of us seem to realize — “Hey, we better get some pause policy going ASAP.”
Unfortunately, it’s still been pretty niche. It’s growing, and I think the tide is gonna be rising toward our side. I expect Doom Debates to get more popular as more people wake up to, “Oh, this is happening now. This is urgent.” So I do think we’re skating to where the puck is going. It’s just the timeline is so short — the puck needs to get there now.
Claire 1:20:20
You think we could get funding for a documentary? Would that be more effective than what you’re doing, do you think? Or is what you’re doing already the most effective thing to do?
Liron 1:20:31
There’s a handful of documentaries being made. There’s a feature film coming out very soon that looks super well produced. I don’t know who’s producing it, but I saw it was all over my social media. That’s actually gonna be in theaters.
Somebody messaged me the other day — this guy Connor Axiotes. He raised 1.6 million dollars for a documentary. He’s looking to raise a little bit more. So yeah, the documentary route is definitely being tried. The podcast route I think is open to a few more people. But it’s like — look, the success metric is just people treating the issue as urgent. So anything that shows signs of success is great.
Claire 1:21:09
Miles does a podcast. Is that his name? Robert Miles?
Liron 1:21:13
Yeah, Rob Miles. And there’s also a new one that got popular about six months ago and has more total views than I do. It’s called AI in Context. They did this really popular one on AI 2027. They did this one on Mecha Hitler. They just have good production values. They’re by Eighty Thousand Hours, which is this organization that’s pretty concerned about AI, among other things.
So there’s definitely good media. There’s room for more to some extent. When I see somebody starting new media, I always wonder — does the new media have to exist, or can they maybe get behind another existing project? I don’t know about spreading ourselves too thin. But yeah, it’s better than not doing it.
Claire 1:21:50
I would have thought that the moment someone like Geoffrey Hinton said, “We gotta worry about this,” it would set off fire alarms everywhere. But it’s very strange. It doesn’t seem that anyone has the clout required to say this is worth worrying about and be believed.
Unknown 1:22:07
Liron, have we covered any new ground versus what you knew? Serious question. Have we gone beyond what you already knew coming into this?
Liron 1:22:23
Well, I don’t think so, but I mean, I’ve had discussions that start high level so many times. There’s only so many places you can go without getting really in the weeds.
Unknown 1:22:35
So to Claire’s question of where, even after this hour and a half, with this group who performed variably on their reading assignment — where do you recommend we take this next in trying to answer Claire’s question?
Liron 1:23:00
Yeah, I mean, in terms of call to action, I think Pause AI has the right idea, and then if you look at what their levers of control are, I think government — basically a groundswell to make government think that it’s urgent — is a good direction. The normal tools where you do campaigns to get people to care about stuff, I think are appropriate. The same tools that would make them care about climate change or abortion, that kind of stuff.
Pause AI does plan protests. They’ve just all had very low turnout so far, for whatever reason. I think one reason is because the group of nerds like myself who spotted this first tend to be very shy, very non-committal, and very anti-protest because protests are what normal people do, not us smart nerds who are above the fray. But then nobody cares about AI doom as a result.
If somebody who actually had skills — people who actually could make things happen in terms of influencing the government, organizing people — those are all desperately needed to raise the salience of the issue.
I just want to mention there’s a silver lining here. I think the question was framed in this conversation like, “How do we convince people? Why didn’t Geoffrey Hinton convince people?” I wouldn’t say people aren’t convinced. Using my wife as a typical example — it’s also true about my in-laws. If you ask them, “What’s your P(Doom)?” They’ll be like, “Yeah, it’s pretty high. You’re making good points.” The disconnect is just, “Okay, so what do you want me to do? Vote for somebody different?” But neither candidate is talking about AI.
Unknown 1:24:22
It’s an institutional structural problem. It is not a general awareness problem. I’m totally convinced of this. It’s totally structural in that it doesn’t matter what we think cognitively, individually. The system exists to reward certain behaviors and suppress other behaviors — the human system. And you speak into that system, and it processes what you say, and it gives you an output based on how it processes information. It treats this particular signal as noise because it is designed to treat this particular signal —
Claire 1:25:01
I for one welcome our new robot overlords.
Liron 1:25:05
Yeah, the Earth is gonna get really, really hot if they’re trying to optimize for building things and taking over the universe.
Unknown 1:25:13
Well, what we’ve been seeing is that the whole global warming shtick where everybody said we had ten years before the Earth heats up terminally, and we’re all gonna die — that has basically evaporated and has no credibility at all.
Unknown 1:25:28
Who said that in ten years we’re gonna all heat up and die? I never saw ten years. Never.
Liron 1:25:35
Also just to — even talking about global warming, I’m just saying that if you’re trying to harness the Earth to build an Earth-sized factory so you can conquer the universe, you do run it hot because you’re using all this energy, and using energy radiates heat. So you actually run it as hot as you can as long as you can dissipate the heat, and there’s a limit to how much heat you can dissipate. Point is, I’m expecting a very hot Earth, so this idea that a bunch of people are just gonna survive in the wilderness farming or whatever — I don’t think it’s gonna happen.
Claire 1:26:01
All right. Well, the one thing I do not want people to do after this is say, “We’re doomed. I throw my hands up. I’m going to be absolutely fatalistic and do nothing.” I think we treat existential risk the way we treat every other risk. We do our best to mitigate it. It’s the only thing — the posture of fatalism is ignoble.
Well, to those of you who are signing off, thank you very much. I hope this was helpful.
Liron 1:26:33
Yes. Great to talk to everybody, and we’ll talk about cross-posting some of this to Doom Debates.
Claire 1:26:38
Liron, let’s stay in touch, and let’s keep thinking about the best way to make this argument and reach the most people.
Liron 1:26:46
Yeah, absolutely, and thanks again. Love that you’re doing this. Be in touch. All right, great to meet everybody.
Claire 1:26:51
Nice to meet you too.
Liron 1:26:53
Thanks for coming.
Claire 1:26:53
Good work, Liron.
Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.
Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏
