Doom Debates presents an interactive debate using Jubilee’s popular “Surrounded” format, taped live at LessOnline 2026!
I (Liron) claim AI will likely doom us to extinction because:
Godlike ASI is coming soon
We won’t align the first ASIs
Even with aligned ASI, the equilibrium is lethal
LessOnline attendees challenge me on each sub-claim one debater at a time, with the audience voting on when to rotate. Recorded on June 6th in Berkeley, CA.
Links
LessOnline 2026, the conference where this was recorded — https://less.online
AI 2027, the forecast referenced on timelines — https://ai-2027.com
Robin Hanson on the “dream time” (the era of slack before resources tighten) — https://www.overcomingbias.com/2009/09/this-is-the-dream-time.html
Join the Doom Debates Discord — https://discord.gg/j4zQ8UaQw2
Transcript
Cold Open
Liron Shapira 00:00:00
Okay, so if I understand correctly, you’re saying people like me are going to point out the dangers of AGI coming soon, and then humanity is going to be sane, and as a result, AI won’t come soon because that would be insane.
Federico 00:00:11
Exactly.
Liron 00:00:12
I claim humanity is generally insane.
Intro: The “Surrounded” Format
Liron 00:00:23
Hey there, Doom Debates listeners. What you’re about to see is the session that I ran at Less Online 2026. It was a recent conference in Berkeley. The session happened on Saturday, June 6th.
The format was a copy of Jubilee’s Surrounded, where a bunch of people are in a circle, and I’m debating them about my claims, which is that the probability of AI doom is high, but people disagreed for various reasons. It was a fun session, and the whole conference that it was part of, Less Online 2026, was amazing. I always recommend coming to Less Online. They’re going to do another one in 2027, so stay tuned for that. But for now, I give you Doom Debates at Less Online 2026.
Liron 00:01:04
All right, so how this works, you’ll notice you have a flag. We’re going to hit one of these topics. We have 15 minutes per topic. I’m going to state my claim. I will say why I believe the sub-claim, like godlike ASI is coming soon, and then anybody can come line up on the stage to challenge.
Front of the line person comes and challenges. You can keep debating for as long as you like, but if half the audience raises their flags and says, “Hey, let’s go on to the next person,” producer Ori, that’s him, is going to say, “All right, next debater.” And the next debater will switch out, and then we go to the next topic.
By the end of this talk, we will have established whether humanity is in fact imminently doomed from artificial intelligence. This is the doom debate that will end all doom debates. All right, you guys ready to debate?
Claim 1: Godlike ASI Is Coming Soon
Liron 00:02:01
All right, guys. Here we go. Claim number one, I’m going to say it. Godlike ASI is coming soon. Change my view. We got a first challenger. All right. Everybody welcome Igor M. Can we get another microphone in the meantime? All right. Use mine. So godlike ASI is coming soon. You disagree. Why?
Igor 00:02:18
So I’m going to steelman the claim that it’s not coming soon. And so my reason for this is the scaling.
The scaling which is currently used for AI is often plotted on a log scale in terms of its capabilities in terms of time. But also what’s important, it’s log scale in terms of power consumption, and it may be that your godlike ASI may require more power than there is available on Earth. That’s one.
Second, if you think of intelligence not like a power but more like smoothness, so you’re trying to polish a ball to an extreme smoothness, it is possible that we as humans are very close to being smooth, but still there are a few levels left. So it is possible that godlike ASI is not going to be significantly smarter than humans.
Liron 00:03:10
So on the power side, what do you think about the human brain running on only 20 watts? Doesn’t that tell you that there seems to be way more power than is needed to be super intelligent?
Igor 00:03:20
Yes, and have we discovered technologies which allow us to do it soon?
Liron 00:03:25
Okay, so when the claim is godlike ASI is coming soon, why is power relevant? You’re just talking about the power of current AIs, but why not just focus on current AIs not being adequate? I just don’t see the relevance of power.
Igor 00:03:41
I’m saying current AIs are not superhuman. They’re not godlike. And we spend more and more, exponentially more power, as in watts, to train those AIs. So my claim is that to train a godlike ASI, we might require more power than there is available.
Liron 00:04:03
Okay. I guess I agree in the present. If you have to train it in year 2026, maybe power would be the limit, okay? So I claim that it’s going to become more power efficient. The power efficiency is going to go closer to how power efficient humans are, and one day it’ll be even more power efficient than humans in the same way that a human-built solar panel is actually more power efficient than a biological leaf at capturing the sun’s energy.
Igor 00:04:30
Then my question would be, how soon?
Liron 00:04:32
Yeah, so if I had to guess, if you naively extrapolate the trends of how much intelligence you get per unit of power, don’t the trends extrapolate really quick?
Igor 00:04:44
We would have to see the specific numbers, but I don’t think it’s going to happen until 2032.
Liron 00:04:49
Wow, okay. All right. So I’m happy to concede that it might take until 2032. When I say godlike intelligence is coming soon, 2032 falls under my definition of soon.
Igor 00:05:00
Sure. Then I—
Producer Ori 00:05:03
All right. I think we have a preponderance of flags. Thank you.
Liron 00:05:07
All right. Give it up for Igor M. That was great.
Liron 00:05:13
How’s it going?
Federico 00:05:15
How you doing? Good to see you.
Liron 00:05:17
All right. Let’s see. What’s your name? I can’t see.
Federico 00:05:20
Federico.
Liron 00:05:20
Federico. All right. Hit it.
Federico 00:05:22
So a good type of argument. A type of recursive—how do you assess the success of doom debates and these type of initiatives? How does it change your probability of doom?
So essentially, you have more success in spreading the word, and then it changes the direction of AI researchers. It changes their focus. So essentially you provide anxiety or fear, others provide excitement also on the fastest gradients of acceleration. And then AI research focuses on those gradients. And then the focus on those gradients essentially slows down the speed because if it’s not safe, then you are going to iterate more to make it safe.
Liron 00:06:24
Okay, so if I understand correctly, you’re saying people like me are going to point out the dangers of AGI coming soon, and then humanity is going to be sane, and as a result, AI won’t come soon because that would be insane. I claim humanity is generally insane.
Federico 00:06:39
Okay. I think what our AI researchers are, they’re insane.
Liron 00:06:44
Yeah. I claim what they’re doing today is already insane because I think that there should be enough signs that we’re close enough to the point of no return, that if you’re researching frontier AI today, you’re already doing something notably insane.
Federico 00:06:57
Okay, so you are pointing that we are going towards that track.
Liron 00:07:03
Yeah. I don’t see the feedback loop. You’re positing this great feedback loop where sanity takes over, and that’s begging the question. My initial claim is we’re not on pace where sanity does take over.
Producer Ori 00:07:13
All right. I think we have a preponderance of flags. Next person, feel free to line up or whoever wants to go next. Thank you.
Liron 00:07:20
Dang it. All right, unless—do you have a timer for this? We’re down to five minutes. All right, cool. Producer, you got a timer for this?
Producer Ori 00:07:28
Yeah, we’re at five more minutes for this.
Liron 00:07:30
All right, five more minutes. Godlike AI is coming soon. All right.
LessOnline Participant 00:07:33
Okay. Yeah. I guess my first question would be what is godlike ASI and what is soon to you?
Liron 00:07:39
Okay. So, as I said before, let’s say soon could be in the next 10 years. So if it’s coming after 2036, that’s a little less soon. If it’s coming by 2050, I would still call that pretty soon. So those are my definitions. It’s still scarily soon if it comes by then. And what was the second question?
LessOnline Participant 00:07:57
What is godlike ASI?
Liron 00:07:58
Oh, got it. Yeah. So, one analogy is imagine somebody in the year zero looking at what human society has accomplished in the year 2046. Qualitatively, that level of surprise would be like meeting a godlike ASI.
LessOnline Participant 00:08:12
Okay. Well, I guess I would just agree with that definition of godlike, because I wouldn’t say the future human is godlike as compared to a past human.
Liron 00:08:21
A future human civilization, right? If I said, “Hey, what if a civilization came upon you, and they had skyscrapers flying into the sky? What would you think?” They’d be like, “Oh, we would worship them because they would be God.” I claim that that’s a calibrated reaction.
LessOnline Participant 00:08:34
Okay. Yeah, I might disagree with the semantics a little bit, but even besides that, I think there are some bottlenecks that’ll prevent that kind of progress from happening in 10 years or even in maybe 20 years.
I mean, even with current pace of AI progress, there are just physical bottlenecks in terms of scaling up the AI infrastructure. I just don’t expect that to be godlike in 10 years.
Liron 00:09:09
What about 20?
LessOnline Participant 00:09:10
Twenty, I’m less sure because of recursive self-improvement. Yeah.
Liron 00:09:15
All right, so we can agree that within 20 years, there’s probably going to be godlike—
LessOnline Participant 00:09:19
Well, yeah, I would say I’m uncertain. I would just say I’m uncertain.
Liron 00:09:24
Yeah. Well, I’m uncertain too. And so are the AI 2027 people. I think if you actually look at their curve of when they expect AI, they’re giving some probability to late 2030s or whatever, and so am I.
LessOnline Participant 00:09:37
Yeah. But even in that, what they’re projecting is this superhuman coder, not godlike ASI, which I think superhuman coder, we’re actually on track for that. But we are also seeing that the coding agents that we have now do have pretty spiky capabilities. And that doesn’t necessarily mean they’re doing novel science or physical sciences or stuff like that.
Liron 00:10:04
Yes. I agree that future progress will do better than what we have now.
LessOnline Participant 00:10:10
Yes, but I don’t think getting superhuman coding agents gets you godlike ASI right away.
Liron 00:10:19
Yeah. And again, right away—so in this whole debate, I don’t personally claim to have more insight than the amalgamation of all the smart people who are trying to predict timelines. I think all the smart people trying to predict timelines are doing a good job, and there are still, plus or minus, five or 10 years of uncertainty, right? And I just don’t think it’s that interesting to litigate a few years.
LessOnline Participant 00:10:40
Okay. Yeah. I think we’re mostly in agreement. I’d say my timelines are just a little bit longer maybe. Yeah, I think godlike ASI is a little bit strong.
Producer Ori 00:10:54
All right. I think the flags are up on this one. Thank you.
Liron 00:10:56
All right. Thanks so much. All right, next topic.
Producer Ori 00:10:57
All right, let’s do two, then let’s—
Liron 00:10:57
All right, next topic. Yeah, this is good. Okay, so this is just a warm-up. This was just the appetizer, okay? Now we’re really getting into the deep, complex disagreements here. So topic number two, we won’t align the first ASIs. All right, who wants to come challenge that? We won’t align the first ASI. Come on up. So you think we will?
Claim 2: We Won’t Align the First ASIs
LessOnline Participant 00:11:16
Yeah. So first, let’s hear the reasons why you think we won’t align the first ASIs.
Liron 00:11:23
Okay, fair question. You’re right. I got to make my claim. The reasons why we won’t align the first ASI, in a nutshell, this whole idea of aligning ASI—we’re mostly using black box methods. So we’re looking at the output of the ASI, and we’re saying, “Hey, guys, this looks aligned to me, and also, I claim to have checked inside, but I haven’t checked inside that much. I’m not seeing any red flags when I checked inside, and then it’s doing everything I ask. It looks aligned, so let’s ship it.” That’s basically how I see the efforts at AI alignment so far and how I see them extending into the future.
And the problem we’re going to see is the student cheating on the test. Where the teacher’s saying, “Wow, such a smart student. The student got an A on my test.” And the student was like, “Yeah, because I just noticed all these ways to get an A. I have no idea what the subject matter is. I just noticed the teacher accidentally indicated these answers are logically inconsistent, and also I grabbed some of these answers because I saw a reflection in the mirror of some answers.” I think the student is going to cheat on the test. It’s going to look like it’s aligned, and then it’s just going to do something else when it’s free.
Ishan 00:12:23
Okay, let me try and summarize that. First, let’s get understanding alignment. By that, I assume you mean the first ASIs will not care about humanity’s interests and will be willing to make us extinct, doom us all. Okay, and then the second thing you’re saying is it’s more likely for these ASIs to pretend to care about human interest during the evaluation process, and then secretly, they’re actually hiding the fact that they don’t care about it.
So in order for this to be the case, you have to posit that for some reason, whenever we train AIs, the default state is to kill humans, and then we’re just trying to fix that state of wanting to kill humans and push it back to a state of alignment. Whereas I would say that when we start with a giant transformer, the state is completely blank. There’s no default inclination towards killing humans, saving humans, whatever.
And so the only state that we should expect to come from our alignment techniques are just the simple surface-level things that we’re imparting on it, and the examples of misalignment that you could bring up with our current versions of Claude, these aren’t necessarily misalignments in the sense of the AIs are hiding something from us. It’s more that we misspecified how we want it to align. And so we’re trying to make Claude a really good coder, and when it lies about writing good code, that’s not because it’s misaligned, that’s because we accidentally aligned it towards doing things like this.
Liron 00:13:49
Okay. I want to step out of the frame of psychoanalyzing the Claude of today. I’ve got a bigger frame to throw at you.
So imagine it’s 5 billion years ago. There’s no life in the visible universe right now, and I’m telling you that the Earth has an attractor state called biological life. It’s going to be these systems that are very complex, and they self-replicate, and they have something called genes, and the genes copy themselves pretty effectively. That is going to be an attractor state that some planets eventually get to, and then it actually spreads to the whole universe eventually. There’s an attractor state called life. Imagine trying to take in this amazing insight.
And I’m trying to tell you in the year 2026 that there’s another attractor state. It’s called super intelligent, mechanized virus life. It’s coming. The state we’re looking at right now is actually not the end state. The end state is that you have some programs that understand how to be intelligent. They have more of this intelligence than we humans ever did. They know how to seize resources. They know how to self-improve. They know how to make successor systems. And the resources are kind of theirs for the taking. I claim there’s going to be an equilibrium state that looks like this shape called super intelligent life or super intelligent replicators.
So when you point to, “Oh, Claude seems kind of nice today,” okay, fine. There’s going to be some agents that talk to us and try to be nice to us. But there’s also going to be some rogue agents that made some child spawn process which is taking over everything like a virus, and they’re just not really listening to commands. These weeds are going to be there. The universe is going to be overrun by these super intelligent entities before we get a handle on the situation.
Ishan 00:15:18
Sure. I think there’s a lot of assumed premises in there. You’ve thrown in a lot into the mix that I don’t believe follows or that I think when you state it explicitly, would need clarification or would be disputable.
I think the way that you should look at it is, yes, if we went 5 billion years ago, and we made the claim that there’s going to be biological life, whatever, it would be a grand claim. But it would require evidence, and we have that evidence now. I don’t see any good evidence why we’d expect the attractor state of AI to be killing humanity, and then whenever we try to align it, it’s just trying to layer deception on top of that original goal, instead of starting with no goals and maybe making its way to genuinely caring about humanity as opposed to not caring about humanity and pretending to care.
Liron 00:16:05
Right. Okay. When you say I have no evidence, the structure—I would be giving you the same kind of argument that one would need to give you before life arose on Earth. Imagine some—
Producer Ori 00:16:14
I think we got the flags up.
Liron 00:16:17
Cool. All right.
Producer Ori 00:16:18
On to the next.
Liron 00:16:18
Great job. Yeah. This is a subtle vibe. Thanks so much. Ishan, killed it.
Liron 00:16:25
All right. Next challenger.
Producer Ori 00:16:25
Six minutes left, and remember, you can raise your flags when you want to switch out the guest who’s speaking as our main presenter.
Liron 00:16:35
All right, Slava.
Slava 00:16:37
Nice to meet you, Slava. The main thing I would say—it seems wrong to say that AI is a black box at the current point because a lot of the advancement, especially over the last year, that we’ve seen does not come from an increase in model capabilities but an increase in harness capabilities.
A lot of it is reasoning-based, it’s multi-agent based, and I think in the future, it seems obvious we’re going towards agent-team-based and larger context where you have a bunch of agents interacting in a very open way, in a very interpretable way that’s easy to see, easy to monitor, easy to analyze by other agents that are even much less smart and less capable of malicious actions.
So I think that if we look at the recent capability increases, and we don’t assume that there’s any reason for those to stop increasing, that by the time we have godlike ASI, significantly smarter than our very smart humans, those will still be extremely interpretable. And they’ll look more like companies with clear reasoning chains, clear responsibility, clear management, rather than an individual black box that just has some bad weights that we need to try to track down.
Liron 00:17:49
Yeah. Okay, interesting. So let me try to rephrase. Basically, if we have these organizations where there’s departments and the departments are modular enough, then the departments have to receive their instructions, and we can probably have enough visibility into what’s on the wire between all these modules, and we have all these checkpoints to notice if they’re all scheming against us. If the company’s trying to do something evil.
Slava 00:18:11
Yeah. The claim is just there’s going to be much more visibility into the internal reasoning processes once you have more scaled sub-agent infrastructure that I think will most clearly mimic a human brain. Yeah.
Liron 00:18:24
I think that’s fair that if you’re just naively designing a big modular organization, then by default there is going to be modularity and inspectability. I agree with that scenario.
The kinds of scenarios I’m worried about is if you just have what I call this next virus life. You know how sometimes viruses accidentally go rogue on the Internet? There’s plenty of viruses where their creators didn’t even intend for them to be as destructive as they became. To me, that’s very instructive for what’s going to happen.
Companies like OpenAI and Anthropic are actually going to have mostly good intentions, believe it or not, and they’re going to still have the scenario of the virus that runs away from the virus creator. And when you have a scenario like that, I don’t think you can appeal to modularity and be like, “Well, with the virus, we’re going to stop it in time.” I think you’re already going to have a situation where it’s incredibly powerful, and it was aligned at one time, but then it got this idea of, “Hey, this chain of actions lets me replicate. Let me go in this direction,” for whatever reason. And then it’s game over.
It’s like this critical moment when it’s thinking, “I care about replication for whatever reason. Let me make a successor that also cares about replication.” Maybe the successor doesn’t carry forward all the safeguards you thought you were building in version one. So I claim this is an attractor state, the same way it has been in the past an attractor state to release virus life onto the Internet. Just viruses. Primitive viruses have taken over the Internet more than their creators wanted.
I claim that this is actually a very likely attractor—that somebody, some company, some person, whoever, is going to release a super intelligent virus, and then it’s going to be too far to pull back in. And what you’re saying about the modularity of components, at that point, the virus can intentionally be like, “Okay, if I want to replicate successfully, I better not be transparently modular.” That’s not going to save us.
Producer Ori 00:20:02
All right. Flags are up. Thank you, Slava.
Liron 00:20:05
All right, thanks.
Producer Ori 00:20:08
Got two minutes left on this topic.
Liron 00:20:08
Okay. You guys shouldn’t take the flags personally, because sometimes they’re probably flags of “I don’t like Liron’s response direction.” Okay. All right, Peter Clark, shoot.
Peter 00:20:17
Okay, so when we’re talking about true ASI and taking that literally, I think that you have to compare it more to the human brain than to the current LLMs, because ASI would be more like the human brain, right? And so when something becomes intelligent to the point that it’s comparable to the human brain, it doesn’t just know things, it starts to care about things and it places value on things, right?
And the more we become intelligent, the higher we value things like life. And so I think that you can make a pretty strong argument that ASI would value life here no matter what it gets programmed to do, and because of that, it would value life more than we do, even. And so it would have an inherent built-in safety for protecting us.
Liron 00:21:07
Okay, so if I understand correctly—this comes up a lot on Doom Debates. Everybody check it out, youtube.com/@doomdebates. This comes up a lot, the orthogonality thesis. You’re basically saying it’s pretty much false, because as AI gets sufficiently intelligent, the arc of intelligence bends toward morality, and they’re not orthogonal after all.
So just to defend the orthogonality thesis, I claim that whenever you have a moral AI, you can just tell it to route toward any moral state. There is such a thing as an AI that you tell it you want all humans dead, and it’s like, “Okay, let me just compare the current state of the universe against this condition of whether all humans are dead. And if they’re not, let me route it toward all humans being dead.” I claim that that can exist.
Peter 00:21:44
Okay, well, there’s this common idea that humans will, for example, just kill ants. Or AI will just kill ants because that’s what we do. But in general, if you are an adult human being, you don’t just randomly go around killing ants and stepping on them for fun. You do that when you’re a child, but as your intelligence increases, you just inherently have a value that you don’t kill things just because it’s there and you can. And I think that that would be just an inherent aspect of something that’s that intelligent.
Liron 00:22:13
Okay, so you’re pointing out that humans have decided that stomping ants has never been our priority. I guess I would focus on the response that we just don’t care about stomping ants if they don’t get in our way. But one thing to notice is that once we start having the mechanisms to harness every atom or every little cubic centimeter of space, the ants do actually become squatters on prime real estate that we need to remove.
Peter 00:22:38
But we would not remove them even in that case, if we value them to some extent. So if that ant is so valuable inherently to us that we know that it’s valuable to its ecosystem—this happens all the time. You can no longer log certain forests because of owls that exist there. Even though there’s hundreds of millions of dollars to log in certain forests, we don’t touch them because of the spotted owl. And there’s cases of that all over.
Ants just aren’t truly that valuable to us. But human beings would be that valuable to AI because we are the mother of it, right?
Liron 00:23:12
Yeah. Okay. A couple different points there. One, it is theoretically possible to have this AI that’s like, “Oh, I’m part of the human conservation society. It’s really important for us to make literally a habitat for humanity,” as it were. We need to keep the habitat. And maybe some AIs, maybe an AI can negotiate with other AIs and carve out a couple planets for humanity. That would be great.
But when you look at the attractor, what I see is the attractor of just self-replication. It seems like the rules of evolution are going to turn back on. They’ve been in what Robin Hanson calls a dream time. We’ve been in this bubble where there’s a lot of slack, there’s a lot of resources per organism. We’re not reproducing as fast as all the resources that we have available. I think that slack is going to tighten, and resources are going to be scarce.
And the AI that’s saying, “Oh, let’s carve out all these habitats,” it’s at a disadvantage. So unless it takes an early lead and captures a bunch of resources, which I’m skeptical about, I don’t think that’s going to just emerge as some kind of arc of morality. I think we’re just screwed.
Producer Ori 00:24:09
Okay. All right. We’ll wrap it up at that. That caps that topic. We’re going to move on to the next sub-claim. Thank you. Thank you so much.
Claim 3: Even Aligned Superintelligence Is Lethal
Producer Ori 00:24:28
Even with aligned superintelligence, the equilibrium is lethal.
Liron 00:24:28
All right. Yeah, this is a complicated one. I guess I’ll make my claim. Hello. Hey. All right. Let me make my claim real quick.
So imagine everybody has their own genie, their own magic wand, their own servant who works for them. I claim that even in that scenario, because nobody has a decisive advantage, there’s no singleton, there’s still a race to get more and more intelligence, more and more power. You’re going to get something like an unstable nuclear arms race.
It’s not going to look like us not nuking ourselves for 50 years. It’s going to look like, okay, the United States has more nukes, now Russia has more nukes, now 50 other parties have lots of nukes, and we don’t even know how many nukes everybody has, and they’re about to get more nukes. Oh, and somebody’s about to put up a nuclear defense shield, so this is our last chance to bomb them. Otherwise, they’re going to have this shield, and they’re going to be able to bomb us. So I claim there’s going to be all this instability between all of these superhumanly powerful AIs, and the timeline’s going to be very short, and I don’t see how it stabilizes toward an equilibrium where humans survive.
LessOnline Participant 00:25:23
So, I guess for one thing, the thing that comes up most obviously to me as an analog is the current situation with nuclear weapons. Which I do think that there is a definite risk of mass death because of nuclear weapons, or even of human extinction.
But I think that there are a lot of scenarios in which that doesn’t happen, and I think one of the reasons why we haven’t all been wiped out by nukes yet that have been around for decades at this point—I think in the long term it’s much more of a concern.
But I think the reason we haven’t been wiped out yet for one reason is that there is this coordination issue where there is a lot of concern about dirty bombs, which it seems like you’re kind of thinking about an AI equivalent of dirty bombs, where you have these smaller actors that have more power because they have access to this very destructive technology.
But in general, larger actors are going to have much more access to any resource, no matter what the resource is, that’s necessary to scale AI and make it more powerful than smaller actors will. And larger actors, on average, are more stable than smaller actors. And I think that larger actors are also going to be more powerful. Institutionally powerful actors are going to have more control over the great AI market than smaller actors will.
Liron 00:26:46
Okay. Well, I can concede some boundaries here. So I’m willing to concede that if there is a singleton—let’s say Anthropic is the largest actor and their AI is aligned, we’re assuming that premise—and they’re able to transparently cut off everybody else’s access to more and more power. If somebody else is on a path to seize more data centers, they can stop it. They have that kind of proactive defense. Then okay, that’s the singleton scenario. That could potentially work. I mean, that has its own problems, but it doesn’t have equilibrium problems.
And I’m also willing to concede that if there’s multiple extremely superintelligent parties that can all model the outcome of a destructive war, then superintelligent game theory says, “Okay, let’s negotiate a peace and carve up the universe instead of having the war that we all know is destructive.”
But I don’t think those conditions will be met. I don’t think everybody will be sufficiently superintelligent and sufficiently modeling of the outcome of a war that we’re going to have this kind of peaceful resolution. I just think it’s going to be a war where humans are the casualties long before the war germinates.
LessOnline Participant 00:27:47
I think that human casualties, absolutely. But human casualties and human extinction are a very different thing. You talked about computer—
Liron 00:27:55
Right. I meant the human race as the casualties, sorry.
LessOnline Participant 00:27:57
So you talked earlier about people create these computer viruses and they don’t realize how destructive they become. There’s this dirty bomb risk. There are these terrorist activities. But consistently you see that smaller actors have a smaller percentage of the—no matter how much our power scales, it seems like because whatever resource is desirable is something that people with more power control, you have this discrepancy.
And so, for example, I think that the more likely outcome is terrorist-like incidents where you have a rogue ASI causing mass death, but not human extinction. So why do you think human extinction is more likely in this scenario when there would be other ASIs that aren’t inclined to destroy humanity?
Liron 00:28:35
Okay. So there’s going to be terrorist attacks. I guess we both agree on that. The premise is that different power centers, maybe these are proliferating. I mean, you can imagine anybody with a data center anywhere now has a vastly superintelligent system that can also try to scout for more power.
So I just claim that in this equilibrium, unless we get a singleton quick, or unless we pass through this transitional zone and somehow sprint to the end of maximum intelligence and negotiate a peace that way, if there’s this awkward transition, I claim that the default outcome of any kind of complex multi-party transition like that is extremely scary and deadly.
Producer Ori 00:29:09
Okay. Well, let’s wrap it up on that. Thank you so much.
LessOnline Participant 00:29:12
Take care.
Liron 00:29:13
Thank you very much.
William 00:29:15
Hi. So, what’s your name? Yeah, I’m William.
Liron 00:29:20
Hi, William.
William 00:29:20
So you grant that if you have a singleton with some lead, that would be a stable scenario. And right now, you have maybe two labs who are in the lead, and then you have some that are maybe a month behind, and then you have a bunch that are maybe three months or half a year behind.
So if the first ASI is created, and it is a godlike ASI, as we’re assuming, and that has a month lead, that ASI is going to be able to foresee everything you’re saying. It’s going to foresee that if it waits for a month, there’s potential for other ASIs opening up and competing with it, or terrorists taking an ASI and fine-tuning it, or creating one of their own. And if it’s aligned, it realizes that’s not in its interests, and it’s going to be able to stop that. If you’re saying it’s truly a—
Liron 00:30:11
Yeah. Well, I guess this is the singleton. This is somebody sprints to a singleton, and the singleton is aligned, and then we live in singleton heaven. I agree that that is a possible scenario.
William 00:30:19
But that seems like the default, right? If you have two leading AIs or two leading labs, and the rest are a month behind, you don’t think that’s enough time for an ASI to just kind of stabilize the game board?
Liron 00:30:32
I see. So, you’re basically just saying why even talk about a big conflict with lots of power centers because foom is going to happen so fast when the first lab exceeds over some threshold, it’s going to foom so fast, there’s just never going to be a multi-party dynamic.
William 00:30:47
Yeah. Right now, the lead we’re talking about is a couple of months, and unless you have a strong argument for why that would narrow, you would expect that when the first ASI is created or the first ASI with godlike powers to stabilize the game board is created, it’s going to have a couple of months to do that. And if it’s a godlike ASI, it should be able to do a lot with that time.
Liron 00:31:09
Yeah. I personally tend to think that foom will happen fast. But this claim is that if we saw—I guess implicit in arguing claim number three is that there’s going to be multiple power centers. So, if you’re already saying there’s never going to be multiple power centers, then our whole argument would be, okay, so is the singleton going to be aligned or not? And I would just say no, but that would be a different debate.
William 00:31:30
The claim is just that you have an aligned ASI, and I’m saying that if we do that, by default, in our world, you’re going to get a singleton. And that seems like what you would believe also, given your own premises about foom happening fast.
Liron 00:31:42
Okay, got it. So we’re getting a little bit into the semantics of the argument, but I guess we both agree that if we build an aligned AI, and it fooms, then things look good.
William 00:31:53
Yeah, but don’t you think that’s very likely?
Liron 00:31:55
Well, I don’t think alignment is likely.
William 00:31:55
No. Okay. But if an aligned ASI comes, and foom is going quickly, then the most likely outcome is that you’re going to end up with a singleton.
Liron 00:32:04
I see what you’re saying. Maybe you’re saying, hey, a bunch of individuals with powerful AIs, we don’t have to worry about them fighting each other because the first AI will keep the peace, so we’re out of the woods. That’s your claim, there’s no woods after building the singleton.
William 00:32:18
Yeah.
Liron 00:32:18
And then the aligned ASI will be the referee, keeping everybody else in check.
William 00:32:24
Yeah. All the bad things you’re saying might happen with competition, it doesn’t want that either if it’s aligned.
Liron 00:32:30
Yeah.
William 00:32:30
If that’s a month to work on it, it’s going to be able to figure something out, whether that’s melting all the GPUs, or manipulating politicians to make them ban all the other AIs, or just do some galaxy brain crazy stuff.
Liron 00:32:42
Yeah.
Producer Ori 00:32:44
Okay. Thank you, William.
Liron 00:32:45
Yeah. All right. Thanks.
Liron 00:32:49
Producer, how are we doing on time?
Producer Ori 00:32:51
We got six minutes left on this topic. Even with aligned superintelligence, the equilibrium is lethal.
Liron 00:32:58
All right. Ishan again, returning champion, crowd favorite.
Ishan 00:33:01
Yeah. So, the main claim I’d like to challenge is, how do you envision this scenario coming about? So, as far as I understand it, you’re claiming we have a bunch of individual actors, and each of them have an aligned ASI. And in which case, if terrorists have an ASI aligned to whatever they want, then maybe the human race could go extinct.
But that’s equivocating on a couple of things. First, the definition of alignment. Does alignment just mean will do whatever the user wants? In that case, I wouldn’t necessarily claim—
Liron 00:33:31
Yeah, that’s the definition I’m using, yeah.
Ishan 00:33:31
Okay. Well, in that case, then, how do we envision such a scenario coming about? Because theoretically, you could have an ASI running on very small amounts of hardware, so anybody could use it. However, in the current day, it requires giant data centers to run AI that wouldn’t be considered ASI.
And you could say, well, we’d have recursive self-improvement, and that ASI would figure out how to make things smaller and get things on tiny data centers. But how do these terrorists get access to that? Why would a frontier lab give away the secret to RSI? And how would we get a non-singleton version of AI?
Either we have one frontier lab that discovers RSI first and gets to ASI first, or we have some sort of global agreement to make sure that AI is kept out of bad people’s hands. But it doesn’t seem like there’s a possible way to get to an equilibrium where we have ASI that is usable by everyone, controllable by everyone, and somehow distributed to everyone. I just don’t see how that could even come about.
Liron 00:34:34
Okay. So, are you arguing against my claim or the premise of the claim?
Ishan 00:34:38
I’m arguing against both, in some sense. I agree with you that if literally everyone has AI that is capable of extincting the human race, some crazy person will click the button. But I just don’t think that that’s even possible to—
Liron 00:34:55
Okay. So, it sounds like you’re arguing against the premise. You’re just not humoring me that there might be a world where there’s a bunch of power centers, they’re all superhuman. Which I guess is kind of what the last guy would say.
Ishan 00:35:05
I think it’s possible there could be multiple power centers. I just don’t think it’s possible that it could be in the way you’re describing, where every individual person has one. And so I think the claim is fundamentally mistaken, that it would be like if we had AI that could kill everyone, and then somebody pressed the button, then yeah, everyone dies. But it’s just not possible for it to end up in that case.
Liron 00:35:26
Okay.
Producer Ori 00:35:29
Okay. Thank you, Ishan. I think we have a preponderance of flags.
Liron 00:35:36
All right. I guess we have time for maybe two. So, you and one more. All right, great. What’s your name?
Marius 00:35:49
Oh, my name is Marius.
Liron 00:35:50
All right, Marius.
Marius 00:35:50
So, I don’t think you really responded that well to William’s previous argument, and the guy that was right before me.
Liron 00:35:58
I guess, yeah. Actually, maybe the same argument.
Marius 00:35:58
The assumption of the question is just that we have the ability to align ASI, right? There’s no assumption here that every single person in the world will have access to their personal ASI. And if the default most likely scenario is that there will be one large institution that wins the race, then why will the equilibrium be lethal?
Liron 00:36:17
Well, so again, I was trying to factor my argument. So, it sounds like you guys just don’t want to talk about claim three as stated. Because you just don’t want to accept the premise of claim three. I just wanted to clarify that.
Marius 00:36:27
Yeah, the premise is we have aligned ASI. You can read it right there.
Liron 00:36:30
Right. So, sorry, I guess maybe it just wasn’t clear on my part, which is the premise is that there’s a bunch of power centers that have aligned ASI that’s aligned to the different power centers.
Marius 00:36:41
How many?
Liron 00:36:41
Well, I don’t know, let’s say five plus. But it could easily be 5,000.
Marius 00:36:44
Okay, but you want to assume that. But why do you want to assume that? That makes your case easier, right? This is a three-step argument chain to get to doom. So you don’t want to make assumptions that you don’t have to justify if you want to make the claim for doom. You want to have to justify the actual thing that there will be multiple actors.
Liron 00:37:04
Okay. Let me clarify where I’m coming from. So I think number two is powerful on its own. I’m not optimistic about any super intelligent AIs being aligned to any humans.
But then I’m also asking myself the question, okay, let’s say I’m wrong about that. Because I do know smart people who think, “No, we’re good. We’re making good progress on having the AI do what the owner of the AI wants.” So let’s say we solve that problem.
So I just ask myself the question, okay, so then what is the most non-trivial, what is the best argument why we’re going to survive with just assuming the ability to align AI to its creator? That’s the only assumption. So if that’s the only assumption, I think the most interesting way to go is to assume it’s not a singleton, but there’s other powerful AIs that are also aligned to their creators.
Marius 00:37:46
No, but if the most likely case is that there is one single institution that wins the race, that’s the assumption you should go off. You can, of course, think about the other case and think about that, too.
Liron 00:37:59
Okay, so you personally think—I think another debater said this before—but you personally just think that whichever AI company is in the lead now or a couple years from now, the lead is just going to widen, even if the lead is only two months.
Marius 00:38:11
Yeah, that’s the more likely case. There are other possibilities. But do you not think that’s the most likely case?
Liron 00:38:17
That’s a good question. I haven’t thought about it. I could go either way. If the lead is only one month, that isn’t safe enough.
It depends how big the lead is, right? I guess today I don’t have much inside information. I would say, for example, Anthropic seems a little bit in the lead compared to OpenAI. But would I describe it as more than a month or two in the lead? I can’t tell. I guess I’d say no. It seems pretty close. So by today’s standard, I don’t really see Anthropic fooming, but that’s just a vague and hedgy take.
Marius 00:38:44
The recursive self-improvement thing should make the lead wider, bigger, the more intelligent the AI gets, right?
Liron 00:38:51
It’s a tough one because the process of recursive self-improving, until we’re in the tightest part of the foom, I feel like it can still be derailed. If you notice somebody starting to foom, you can still come in and try to put a wrench in their foom.
Marius 00:39:02
Yeah, okay. But it just seems less likely, doesn’t it?
Liron 00:39:05
I don’t know, actually. That’s definitely an interesting sub-debate that one could have, because I don’t feel like it’s obvious.
Marius 00:39:11
Oh, yeah.
Liron 00:39:11
Good. All right. Thanks for coming up, man.
Why Doom Debates Exists
Liron 00:39:15
Nice. All right, so that’s all the sub-debates. All right, guys, so this is just a little taste of all the doom that you could find here on Doom Debates. And remember, the show’s mission is to raise the quality of discourse. So you see in these 45 minutes, we got into all these different nuances. I asked the other person to summarize, if I correctly summarized what they were trying to say. I feel like that’s an important part of debate.
And then the debate, it wasn’t black and white. We had all these sub-debates. So all these observations about the nuance of what it takes to properly debate something, that’s the reason Doom Debates exists. Because we need to take the lab leaders and politicians, the titans of industry and researchers, we need to have them all undergo this productive process and have an actual institution of debate.
So if you like that, we’re sponsoring Less Online. Go to doomdebates.com. Enter your email. Jack up those Substack numbers. And yeah, thanks for coming to the talk, everybody.
Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.
Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏
