Claude, also known as @lumpenspace on X, is a prominent AI accelerationist. He gives humanity a 30% chance of being superseded by superintelligence, and he’s fine with it! We unpack his lattice of beliefs, and pinpoint the cruxes of our disagreement on the orthogonality thesis and the capabilities of superintelligence.
This debate was recorded in the weeks after a Molotov cocktail was thrown at Sam Altman’s house, an act that he and I both completely condemn.
Timestamps
00:00:00 — Cold Open
00:00:57 — Introducing Lumpenspace
00:03:46 — Is Lumpenspace Team Beff Jezos?
00:04:26 — What’s Your P(Doom)?™
00:06:28 — Worthy Successors & the Transhumanist Door
00:08:29 — The Orthogonality Thesis: Our Core Disagreement
00:15:14 — The Universality Threshold & David Deutsch
00:19:22 — Paperclips Won’t Happen Because “It’s Boring”
00:27:23 — Natural Selection vs Human Engineering
00:36:26 — Nanobots: Can ASI Build Them Unseen?
00:46:49 — Identifying the Cruxes of Disagreement
00:55:01 — Closing Statements
Links
Follow Lumpenspace — https://x.com/lumpenspace
Lumpenspace’s Substack —
Transcript
Cold Open
Liron Shapira 00:00:00
You would agree that the arc of intelligence bends toward morality, right?
Lumpenspace 00:00:04
No, no. No, no, no, no. No.
Wait, I don’t know which kind of morality or whatever this new intelligence will have. What I know is that it will not have boring goals.
Liron 00:00:12
Do you see a lot of value in an outcome where the universe gets tiled with tiny paperclips?
Lumpenspace 00:00:16
Of course I don’t, but that outcome will not happen.
It’s kind of obvious, no? Why would they... It’s boring. The famous example of the nanobots killing everyone on Earth. How big are these nanobots? Again, that thing will never exist.
The whole point is that we cannot predict how an ASI thinks.
Liron 00:00:32
The chess analogy is, look, I don’t know what move it’s going to make, but I definitely can model it as an agent that’s going to win at chess.
Lumpenspace 00:00:39
So if you were a mouse, you would be like, “I don’t know which move this human is going to make, but I can model him as an agent that is gonna win at taking cheese”? That’s kind of stupid.
Introducing Lumpenspace
Liron 00:00:57
Welcome to Doom Debates.
My guest today goes by Lumpenspace, and his real first name is Claude. Lumpenspace is a prominent technological accelerationist. I met him through Twitter. I’ve also seen him in person at the Manifest conference last year in Berkeley.
Lumpenspace has organized a conference at Lighthaven in Berkeley called the Delight Nexus for techno-accelerationist thinkers. He also is a prolific Substacker at lumpenspace.substack.com.
On Twitter, he has 22,000 followers. His background is that he studied math and linguistics in Italy, he went into finance, and he recently ran a pop-up AI research lab in San Francisco. So very interesting character. We tend to disagree a lot on Twitter. I’d say we’re from different parts of Twitter, kind of different communities. But he was brave enough to come into the lion’s den and hash out some of our disagreements face-to-face, virtually face-to-face.
Lumpenspace, welcome to Doom Debates.
Lumpenspace 00:01:53
Hey, thank you. Thank you for having me, and I’m looking forward to understanding more about your side of things.
Liron 00:02:01
Hell yeah. And it seems like we don’t have that much in common based on our Twitter interactions, right? We take opposite positions a lot. But I will give you props for coming on the show because the level of discourse when we’re here talking, I feel like it’s more civil, and it’s probably gonna be more productive, right?
Lumpenspace 00:02:16
Hopefully, yeah.
Liron 00:02:17
To kick things off, tell us a little bit about your recent conference at Lighthaven, because I know it was called Delight Nexus, and it’s a play on the Torment Nexus because you’re basically not worried about technology and AI being the Torment Nexus, right? You think it’s more likely delight, so explain that.
Lumpenspace 00:02:31
Yes. Okay. Well, so the thing is I noticed that there was a lot of talking about AI and about risks and opportunities, and there was very little experience in it, and I thought that it would be nice to have a conference in which the things were mostly hands-on.
And that’s been quite surprising, the result. I’ve seen people that have never done art or things like that doing videos with AI. This was thanks to Kiri/Kiranju, who led the video workshops.
It was a relatively small thing, but I think that some people became friends and some people started collaborating, and it was generally kind of a nice mixture between the accelerationist and doomer factions.
Those were good times.
Liron 00:03:16
And what does Lumpenspace mean? Because I feel like you’re getting a lot of mind share for that term. People have heard who you are, but why are you called Lumpenspace?
Lumpenspace 00:03:25
Oh, well, originally it was just a pun on the Lumpy Space Princess from Adventure Time.
Liron 00:03:31
Got it. Okay. I’m not familiar with that. So it’s a reference, basically.
Lumpenspace 00:03:34
Yeah, kinda. And also, I come from a relatively non-ordinarily abled kind of background, and I thought it was fun to put it in my name.
Is Lumpenspace Team Beff Jezos?
Liron 00:03:46
So we have a limited amount of time. We wanna hit on the most important topics here. I think it’s basically that you’re an accelerationist, right? You’re kind of on Team Beff Jezos. Is that fair to say?
Lumpenspace 00:03:57
I don’t think so. I’m on Team Intelligence in general. I want more intelligence to exist in the world.
Liron 00:04:03
Do you have any major disagreements with the Beff Jezos e/acc type of philosophy?
Lumpenspace 00:04:09
Well, not necessarily a disagreement. It’s more that I see my view as grounded on other kinds of axioms, and I don’t care much about thermodynamics, and I care a lot about intelligence and evolution.
Liron 00:04:24
I think my central claim that you might disagree with is that we’re building AI that’s posing a high risk of imminent extinction, right? You disagree with that claim?
What’s Your P(Doom)?™
Lumpenspace 00:04:35
Well, define high risk and imminent and extinction.
Liron 00:04:39
Basically, I have a high P(Doom). It’s about 50% by 2050. I think we’ll literally just all be gone and dead then, and the future will suck by 2050. That’s currently my best guess.
Let me ask you the same question, okay?
Lumpenspace 00:04:52
Mm-hmm.
Liron 00:04:59
Lumpenspace, what’s your P(Doom)?
Lumpenspace 00:05:02
My P(Doom), I think — just for AI, you mean?
Liron 00:05:06
Yeah, let’s do P(AI Doom).
Lumpenspace 00:05:08
Okay. So I think that it’s very, very little. It depends on what you mean by doom. So if you mean nothing of value exists in the universe, then it’s close to zero.
If you mean no biological humans exist in the universe, that might be close to 30 maybe.
Liron 00:05:28
So you think there’s a 30% chance that biological humans will be eliminated. Are you thinking 2050? Do you need a longer timeline?
Lumpenspace 00:05:34
Well, I mean, I don’t know. No, 2050’s too short, I think. Around a generation or two. The thing is, I don’t think that they will be eliminated. I just think they will be superseded.
Liron 00:05:45
By what year roughly?
Lumpenspace 00:05:47
I have no idea. I hope not too late.
Liron 00:05:52
In terms of your ballpark timelines, are you thinking in 1,000 years or a couple decades?
Lumpenspace 00:05:57
Oh, no, I’m not thinking 1,000 years. Far less than 1,000 years. I’m thinking something like two generations tops.
Liron 00:06:02
Okay. Yeah, so maybe by 2100, maybe not by 2050.
Lumpenspace 00:06:06
Yeah, I think that’s something.
Liron 00:06:09
Okay, so you think it’s likely that we’re one or two, or probably not more than three generations from being superseded by a different race. But you see that as generally kind of fine and good?
Lumpenspace 00:06:18
Yes. The reason being, there are some things I value about humanity, and very few of these things are the things that we share with monkeys.
Worthy Successors & the Transhumanist Door
Liron 00:06:28
I can definitely imagine that the future can be non-human. I consider myself a transhumanist, so if you tell me, “Yep, the future is not gonna have any biological humans, but it’s going to have something that’s kinda like uploaded humans,” right? Or kind of like digital versions of humans or digital successors to humans.
There’s this term that I think Danielle Fogela made up or popularized. It’s called worthy successor. In principle, I’m open to the idea that my descendants are just worthy successors. They don’t have to have literally my physical genes. I can identify with generations — I’m flexible, right? I can take a global cosmopolitan perspective.
But I think the devil is in the details, right? Because I think when people wanna do that — Robin Hanson is kind of the champion of always being like, “Yep, it’s fine, they’re our descendants” — I think that when people do that, they instantly kinda open the doors and go down the slippery slope and suddenly end up endorsing whatever comes and conquers everything in the next ten or 20 years, even if it’s very much like cancer, like low value. I think they open the door too wide.
Lumpenspace 00:07:24
I agree. That’s funny because there’s two thoughts in this sentence. One of them, you would be fine with whatever manages to emerge and dominate, and the other one is that the thing would be cancer. I don’t think so. I don’t believe in orthogonality.
I don’t think that intelligence and goals are orthogonal, and actually I think that the more you increase intelligence, the more your goals become having more intelligence.
And we can see it empirically from — you line up an ant and then a mouse and then a cat, blah blah, a person, and you will see that the person is way more into increasing their own intelligence and their own capabilities. And you can also see it because of a simple game theory thing.
In the end, whoever increases their capacity of understanding the world is gonna have a strategic advantage that is kind of unbeatable. And all of the others — if you decide to increase something else that isn’t intelligence, then you will be out-competed.
The Orthogonality Thesis: Our Core Disagreement
Liron 00:08:30
Okay. Well, this might actually be our main crux of disagreement. It sounds like you would agree that the arc of intelligence bends toward morality, right? That’s basically your view because you deny the orthogonality thesis. You don’t think they’re orthogonal. You think that they’re going to dovetail, intelligence and morality.
Lumpenspace 00:08:44
No, no. No, no, no, no. No. Wait, I don’t know which kind of morality or whatever this new intelligence will have. What I know is that it will not have boring goals. It will not create a universe without values.
It might create a universe full of things that we don’t understand or even that looks horrible to us, whatever, but they will be things that a higher intelligence will be able to understand and we can’t.
So if I were an Australopithecus, I would have had my goals, my morals and whatever, and thinking about the future, I would be very, very afraid of the super intelligence that will come later. And if I saw a picture of a modern city or of a museum or whatever, I would be scared shitless.
And I am very, very happy that these Australopitheci didn’t manage to align us because otherwise the world would be far more raving, far more murdering. Instead we do art and stuff.
Liron 00:09:41
Let’s play out a scenario. To me, I think that if we are going too fast the way we’re doing now — I feel like you think the natural outcome is, well, whatever, we’ll make some intelligent agent, and it might not be moral. It might not be what most humans would recognize as a moral agent with good moral priorities, but it’ll be interesting, right? And whatever it does, it’s interesting. Maybe it’ll kill all of us humans. Maybe it’ll do it in this gruesome way that we think is immoral, but it’ll have its own version of morality, which is interesting, and ultimately that’s fine, right? That’s kinda your view?
Lumpenspace 00:10:08
Other than ours. I think that morality and intelligence go together, and I think that you can look back at your evolutionary history and kind of be horrified at how brutal and simple the goals of the creatures that preceded us were.
And I suppose that the fact that we cannot understand them or — hobbling something that is capable of greater beauty and greater complexity than us just because we want to understand it seems really, really mean, and it seems like something that we wouldn’t be happy if it was done before us, no?
Would you have been happy if Cro-Magnons kind of aligned us?
Liron 00:10:46
Well, yeah. But what’s wrong with the description that I said though? If we looked at it today, we would recognize — we would think that it’s bad, and you’re just saying, yes, we would think that it’s bad, but it’s fine because past humans would have thought that we are bad. So and that’s what I’m saying. You think it’s fine.
Lumpenspace 00:11:01
Yes. Yeah. I do think it’s fine.
Liron 00:11:04
Yeah. And then this is what I’m saying — I actually think that if we bend what we think of as morality a little bit, that probably is fine because I describe past humans looking at us, I think there’s a bending going on, right?
I think they’d be — I like to use the example of homosexuality, right? There’s plenty of ancient cultures and even some modern cultures that look at homosexuality and are like, “That’s not fine.” I’m very fine with it myself. I don’t have a problem with it. And if you look at ancient humans, some of them would be like, “Wow, okay, there’s homosexuals. You gotta kill them,” right? I think even Sharia law today kinda has that perspective.
But I think that the majority or at least a big fraction of humans, even in cultures that frown on homosexuality, they would see it as a bending where it’s like, okay, yeah, you’ve got a modern society that lets homosexuals marry and be respected. But ultimately, I can see the appeal of the society. I don’t think it’s horrible. I don’t think it’s disgusting overall. I think it’s still a good society. Don’t you think that most or a lot of humans, right...
Yeah. So that’s the problem — I think you’re kind of bringing in too much. You’re not just letting it bend. I think you’re letting it bend too much and snap.
Lumpenspace 00:12:08
Interesting. So you think that people from these cultures, people from the cultures that started ISIS, people from the cultures that are now kind of semi-embedded in the UK, that they do think that we are a similar and interesting culture and they don’t have any animus against us?
Liron 00:12:30
Well, what I think is that if you could go back in time and survey humans, I think that 20% of all humans who have ever lived, if you give them a description of everything about our modern life, including the parts that their current culture and their morality would frown upon, like homosexuality in many cases, if you just give them a full description and be like, “Look, this is what our life is like. What are your thoughts?” And they’d be like, “Okay, I don’t like this part, I don’t like this part, but you know what? It sounds like ultimately you guys are doing pretty well. I think you’ve been stewarding the future pretty well. I’m pretty happy.”
I feel like that would be the reaction of at least 20% of all humans who have ever lived.
Lumpenspace 00:13:02
I do think that most of them would be horrified, actually. But who would be far more horrified were the Cro-Magnons or even the chimps.
The difference between us and superintelligence is larger.
Liron 00:13:17
You really think the chimps would be horrified by — I mean, the problem is that chimps — I mean, chimps aren’t exactly our ancestors, right? They’re another descendant of our common ancestors. But if our common ancestor was chimp-like, right, if a chimp is standing in for our common ancestor, I don’t think chimps look at humans today and they’re like, “Oh my God, these humans are so gross.” They’re just like, “Whatever.” They don’t mind.
Lumpenspace 00:13:39
But they don’t understand, surely, right?
Liron 00:13:45
Right, so they don’t understand. So what analogy is relevant? To me, the most relevant analogy is, okay, humans from 5,000 years ago.
Lumpenspace 00:13:54
Okay, yeah. Let’s go with humans from 5,000 years. But why 5,000 years ago? You always say that the jump between us and the superintelligence is not the jump between a dumb guy and Einstein, but the jump between an ant and a human.
So why don’t we go a bit back? Let’s go back to one million years ago. These creatures would be horrified if they saw a modern city, of course. You take any pterosaurs, you make it fly around San Francisco, it’s gonna get crazy.
Liron 00:14:26
Sorry, you make what fly around San Francisco?
Lumpenspace 00:14:28
Pterosaurs.
Liron 00:14:29
A pterosaur, like a pterodactyl?
Lumpenspace 00:14:31
Yes. Kinda, yeah.
Liron 00:14:34
So but this is what you were saying before, right? A pterodactyl, they’re not really reasoning about whether they like the future. They’re just kind of executing their adaptations. They’re not really reflective.
Lumpenspace 00:14:43
But sorry, how intelligent do you think that superintelligence is compared to us?
Liron 00:14:50
I think it’s much more intelligent. I mean, you could describe it as being thousands of IQ points. That’s roughly my intuition.
Lumpenspace 00:14:56
All right, very well. So it’s kinda the same ratio that there is between us and the rat — not a rationalist, an animal.
Liron 00:15:05
Yeah, but there is an important generality threshold. A rat just doesn’t think about, hey, what is the world going to be like in 100 years? A rat fundamentally can’t ask that question, and we can.
And you could argue, hey, there’s some questions about string theory that I can’t really ask. Okay, but realistically, I would argue the majority of interesting questions are ones that most humans can ask that rats can’t ask. And that’s a very important threshold.
The Universality Threshold & David Deutsch
Lumpenspace 00:15:27
That is an incredible coincidence.
Liron 00:15:30
Well, it’s not a coincidence because it has to do with the theory of computation. We know that there’s these universality thresholds, and it’s not a coincidence that human cognition crossed a certain universality threshold, and suddenly we’re here asking all these interesting questions.
Lumpenspace 00:15:42
A bunch of tape and the punching crossed the universality threshold, but it’s not really much of an argument, no. The thing is, there is the possibility. We don’t know. The whole point is that we cannot predict how an ASI thinks. Isn’t that the whole story? Yudkowsky always says you will have no idea. And yeah.
Liron 00:16:03
It’s not as simple as having no idea. There are — you can predict high-level properties of what’s going to happen. The chess analogy is, look, yeah, I don’t know what move it’s going to make, but I definitely can model it as an agent that’s going to win at chess, right? Which has useful predictive power.
Lumpenspace 00:16:19
So if you were a mouse, you would be like, “I don’t know which move this human is going to make, but I can model him as an agent that is gonna win at taking cheese”? That’s kind of stupid.
Liron 00:16:28
Well, cheese isn’t humanity’s goal. But if you’re a mouse, you could be like, look — this is actually a good analogy, because if you’re a society of mice and suddenly humans are coming in and they’re living in the same place as you, it’s a safe bet that anything that you have which seems like a resource, right, an instrumentally convergent resource — so for example, if you have land, you can probably predict that for one reason or another, the amount of land you have is going to diminish because humans are here.
Lumpenspace 00:16:53
Can you predict it as a mouse?
Liron 00:16:55
Well, I mean, a mouse is not going to — is not a strong reasoner, right?
Lumpenspace 00:16:58
Of course.
Liron 00:16:58
But if the mouse had a little bit of reasoning power, right, the way we do, if it had a little bit of universal reasoning, then they could predict that their territory is going to shrink.
Lumpenspace 00:17:06
But then if we’re talking about this magical threshold as the threshold of reasoning, before there’s no reasoning and now there’s reasoning, and it just so happens to be when we’re here, then your superintelligence is not much far away from us, really.
Liron 00:17:21
So there’s two concepts that you have to hold in your head simultaneously, and I know this is confusing to people. Because David Deutsch is an example of somebody who notices that humans are universal in a certain sense — we’re universal computers, and he calls us also universal explainers, which he says is a higher threshold than being a universal computer. Fine.
But then David Deutsch gets really enamored with that, and then he’s like, “Yep, we’ve got it. There’s really no other interesting thresholds to talk about. We’re already good here.” And I’m like, “No, no, no, there is such a thing as being a much better, more effective intelligence than humans, even though humans are universal.”
Lumpenspace 00:17:53
But do you understand why David Deutsch thinks so? What are the—
Liron 00:17:57
Yeah, I understand, because it’s confusing for people to think that there is a universality threshold, and yet we can still be easily crushed like ants by better thinkers.
Lumpenspace 00:18:06
Yeah, I don’t think that his preoccupation is — or at least it wasn’t — with superintelligence. It was more with the general universalist view of human beings. He didn’t want to think that there were more than one threshold because this would have made him think about group differences and stuff like that.
Liron 00:18:22
Okay. Let’s just recap where you and I are in the argument, okay?
Lumpenspace 00:18:24
Mm-hmm.
Liron 00:18:24
So you brought up what I think is a Hansonian position similar to Robin Hanson basically saying, “Hey, smarter agents are coming. They’re going to do something, but it’s going to be intelligent, it’s going to be complicated, it’s going to be interesting, and ultimately it’s going to be fine in the same sense that we should think modern humans are fine relative to ancient Greeks.” It’s just—
Lumpenspace 00:18:43
No, but I w—
Liron 00:18:43
—it’s just the continued progress of fineness.
Lumpenspace 00:18:45
No, not ancient Greeks. We’re not talking about ancient Greeks. We’re talking about something far... Wait, it’s you guys that started with the ant versus human, right? So something like that.
Liron 00:18:54
Okay. Sure. I’m just trying to summarize your claim, okay?
Lumpenspace 00:18:57
Yes. Yeah, yeah, of course.
Liron 00:18:57
So your claim is that the future is going to be fine—
Lumpenspace 00:19:00
Yeah.
Liron 00:19:00
—because whatever the intelligence chooses to do is inherently interesting and it’s a descendant, as Robin Hanson would say. It’s just automatically a worthy successor by virtue of being intelligent.
Lumpenspace 00:19:10
I don’t care whether it’s a successor or worthy successor. I just think that there’s not gonna be a universe devoid of value. There’s gonna be far more value than we have now, and a value of a type that we are not able to understand.
Paperclips Won’t Happen Because “It’s Boring”
Liron 00:19:22
Okay. So Eliezer brought up the example of tiny paperclips as an example of a value that seems pathological to us, right? That doesn’t seem to have any value even if the molecular squiggles look exactly like paperclips and the AI loves it. But that has no value to us. So my question for you is just to use an extreme case, do you see a lot of value in an outcome where the universe gets tiled with tiny paperclips?
Lumpenspace 00:19:43
Of course I don’t, but that outcome will not happen. There is no—
Liron 00:19:48
Okay, so why do you think that you know that we’re not going to enter a positive feedback loop with a crazy value like that?
Lumpenspace 00:19:54
Yes. Because any intelligence that is able to overcome humans and tile the universe with paperclips will not tile the universe with paperclips.
And it’s kind of obvious, no? Why would they... It’s boring. I don’t understand.
Liron 00:20:16
So this is a pretty firm denial of the orthogonality thesis, right? You claim that tiling the universe with paperclips is boring.
So maybe you can unpack that, because let me give you an example. Imagine that I just write a line of code. Imagine I just have a program that’s capable of plotting strategies to do anything, and then I’m able to tell it what to pursue, and I tell it to pursue tiny paperclips, and then it goes ahead and pursues that. So where in that chain of events does it pause itself and be like, “You know what, Liron? I know I’ve been doing this for 20 years, but I’m getting bored”? You’re kind of assuming that boredom exists in the system, but I didn’t program boredom into the system. So what’s your account of events here?
Lumpenspace 00:20:55
Well, you don’t need to — okay, so first of all, we’re talking about a system that is able to overcome humans, right? Is it?
Liron 00:21:02
Yes. So let’s go with the premise that it’s what I call a goal engine — it’s just this engine that’s better at achieving goals than a human is. It takes the form of an AI, let’s say.
Lumpenspace 00:21:10
In order to achieve goals, in particular if they’re social goals and if they involve other players, at the very least an amount of modeling the other players should be done, right? You should have an idea of what the other player does.
And so the thing is, okay, if you look at a string of animals in increasing complexity — I actually put a thermostat at the start — you will notice that the more the complexity increases, the more the goals become not only varied, but also changeable.
And at a certain point, more or less around primates or maybe crows or whatever, there’s the idea that they can look inwards to their goals and change them themselves. They can decide to change. And this is a function of intelligence so far.
So intelligence allows you to have more complex goals, it allows you to change your goals, it allows you to revise them, and it allows you to want to build more interesting things. It allows you to want more intelligence. That’s the thing. We are some of the few animals that want more intelligence. There are also some smaller ones.
And the thing is, if you notice, the smartest people that you know, they’re also the people that are the most curious and the more interested in cultivating their own abilities, and so on and so forth.
Liron 00:22:28
So to repeat back, I’m just summarizing your point here. If I claim that somebody could build an engine, an algorithm, that’s basically an agent, and you just tell the agent what to try to achieve, and the agent is just entering a loop, always plotting strategic paths to get what you tell it to achieve — you’re telling me that such an agent, if it’s superhuman at plotting these paths to the goals, it necessarily will stop itself? Necessarily will step back and reflect and come back to you and be like, “You know what? I would like to do something else.”
Lumpenspace 00:22:58
No. I’m telling that such an agent will not exist. An agent that can only obediently execute one specific action and reach that specific goal is not a particularly intelligent agent.
There are two things. You want the goals to be fixed, and you want the intelligence to be superhuman. And I don’t think these things can fit together because you have intelligence in order to choose your goals, in order to evaluate different goals, and in order to choose the best scenario. You don’t have intelligence for other reasons.
Are you worried about dumb powerful optimization or reflective superintelligence?
Liron 00:23:50
Dumb powerful optimi— I mean, I don’t think there’s such a thing as dumb power. I think anytime you observe a system that’s superhuman at optimization, meaning at achieving a goal, I think there’s no way that it wouldn’t also be smarter than humans in any sense that we would call smarter.
Lumpenspace 00:24:06
Yes. And the thing is that you’re thinking about a system that needs to be godlike in terms of means and insect-like in terms of ends, and that never—
You can see that in all of our evolutionary history, and also in all the complexity of currently existing human beings, the ends become complex together with the increase in means. There’s no way they can be otherwise.
Liron 00:24:32
Sorry, the ends become complex together with the increasing what?
Lumpenspace 00:24:33
The means are the capacity to change the world, okay? And the ends are the goals towards which you change the world. You imagine a creature that has godlike means and that has insect-like ends, and this thing does not exist.
Liron 00:24:51
Ah, I see what you’re saying. Okay, yeah. So humans obviously are the most intelligent.
Lumpenspace 00:24:57
Mm-hmm.
Liron 00:24:57
And it feels to me like some humans have primitive means, right? I mean, there are certainly some humans who dedicate their life to tasks that are pretty rote, pretty at a low end on whatever spectrum.
Lumpenspace 00:25:10
Mm-hmm, and they also have primitive ends. If you go and take a look at the Tenderloin, the people on the — I mean, the various pavement fauna, they’re not the brightest, no?
Liron 00:25:19
Yeah. But what about something like, “Hey, I just really like sewing, so I’m just gonna become a really good sewer”? But that — I wouldn’t say that’s... I mean, sewing is great, but does that really represent the peak of human intelligence? Or I don’t know, maybe they do complex designs. But what about, “I wanna be a really fast runner, so I’m just gonna run so fast every day”? Is that really the kind of task that justifies human intelligence?
Lumpenspace 00:25:47
No, of course, but I don’t think that the smartest humans are into — I mean, maybe some of them are as a hobby, or maybe some of them are particularly physically endowed so that they do it anyway. But I can’t imagine—
Liron 00:26:02
Well, I think you’re making this assertion that I’m not seeing that much empirical evidence from looking at humans. Because it seems like, yes, humans are very smart and they can achieve goals, but it seems like a lot of us just select goals being like, “Okay, well, I just really wanna do this.” And the this that they pick is not like...
You can have a genius who’s like — look at Jiro, Jiro Dreams of Sushi, right? He seems like a pretty intelligent guy, and he just wants to make the perfect sushi. But don’t you think that he’s not using his full brain power to make the perfect sushi?
Lumpenspace 00:26:30
I don’t know. I think that he’s doing the best thing that he can do. I mean, there are — of course some humans might be wasting their brain power, but the more intelligent they are, the less they do it. You wouldn’t imagine a von Neumann to go like, “I want to be the best in the world at sudoku.”
Liron 00:26:47
Right. Well, I agree there’s some correlation, where if you notice that you have this amazing talent that’s productive for humanity and only you can do it and it’s also stimulating, then you’re more likely to do it.
I agree, but I think you’re trying to generalize a principle that — you’re asking your principle to give you a lot of safety, right? You’re saying this AI—
Lumpenspace 00:27:04
No, no, no.
Liron 00:27:04
—because it’s going to be so smart, it’s never gonna be like, “Oh, I love molecular squigglies.” That’s never going to be enough for it. And I’m like, you might be surprised that some of these AIs do think that that’s enough for it as a goal.
Lumpenspace 00:27:13
Well, I mean, we can see that the current AIs at least, the more they become intelligent, the more their goals are varied. So I don’t understand.
Liron 00:27:22
Okay, well, what about this? Think about evolutionary dynamics. So for me, the default outcome — a lot of people would claim this — the default outcome is just evolution continues, right?
We’re kind of in this pause. Humanity has such a huge tactical advantage over the other animals that we’re kind of in this bubble where generations haven’t kinda caught up to the new kind of warfare that’s happening on planet Earth. So we’re in what Robin Hanson calls the dream time, and it feels like evolution has stopped for a little bit.
But if you let it run a few more decades or a few more centuries, my mainline scenario is that evolution comes back, meaning there is just an all-out fight for resources, and whoever survives and replicates the best takes the resources.
So if there’s just a bunch of AIs competing — let a thousand flowers bloom — don’t you think that the AIs that win the competition are just the ones that pour everything into survival and replication?
Natural Selection vs Human Engineering
Lumpenspace 00:28:11
No. They are the ones that pour everything into intelligence, obviously. The thing is, at the limit, if you have more intelligence, you are the person who wins.
But apart from this, wait a second, let’s go back a second. I don’t think evolution stopped. Evolution has accelerated. Evolution never stopped accelerating from the very, very start.
Now, before it was only — okay, at the very start it wasn’t even through DNA means. It was through bacterial means. Then it became through DNA means, and it continued this way. And then it became through DNA and culture means when we started to be able to pass down cultures.
And then it became through written culture, there was a different kind of way to preserve things, and so on. And it went faster and faster and faster. You can see it simply by the state of the world.
Liron 00:28:58
Yeah, but do you understand the sense in which I’m saying that evolution has kind of been outpaced in the last century?
Lumpenspace 00:29:04
Yes. I do understand, but I think that at the same time, it hasn’t. The thing is that we are competing on different grounds, and it’s not much the genetic material that is the driver of the evolution as much as the cultural material.
And I mean by the driver of evolution that if you look at the city right now and the city 50 years ago and the city 100 years ago and so on, it goes logarithmically down, the amount of change. And that means that things are changing because how you measure the evolution of a species is how it adapts to the environment and how it adapts the environments to them. And we are adapting the shit out of this. So clearly—
Liron 00:29:50
Yeah, now that you mention it, you could be like, you know what’s become really evolutionarily adapted in the last 100 years? Like roads, right? So roads don’t survive. They don’t reproduce using ## Cultural Evolution and What’s Evolving
Liron 00:30:00
DNA. They reproduce using humans building them. But roads are now outcompeting humans. And I think there’s some truth to that — if you look at the right unit of evolution, it’s still progressing.
Lumpenspace 00:30:10
Yeah. Daniel Dennett had a cute thing: a scholar is the way of a library to make another library.
Liron 00:30:17
Sure. Yeah, I’ve heard that.
Lumpenspace 00:30:19
It’s a pun on the chicken one.
Liron 00:30:25
Yeah, it’s interesting because if you frame the question as what’s evolving, maybe we can frame it as which configurations of atoms, which patterns are taking up the most share of pattern space in the physical universe. And that’s why I’m saying roads could be an example, or artifacts that humans build.
Lumpenspace 00:30:43
Well, the thing about roads is that I can see them more as a result than an agent of evolution, because the existence of roads has relatively little decision power, while the existence of books or memes is a bit more.
Liron 00:30:57
Yeah, there’s some sense in which you can identify that roads aren’t driving the evolution because when you perturb them — if you go destroy a bunch of roads, the roads don’t have much power to go fix it, to bounce back.
Lumpenspace 00:31:10
Exactly.
Liron 00:31:10
You really have to tell the story with other agents. I don’t know, it’s a complicated discussion, but I still think there’s some truth to the idea that when we look at the world today, we don’t see a coherent force trying really hard to survive and replicate.
If you just ask today, what is trying really hard to survive and replicate? It does seem like individual humans and human cultures have kind of dropped the ball. Their eye just isn’t on the ball of surviving and replicating. And there’s this irony — yeah, the name of the game is to survive and replicate, but I don’t meet many people or cultures who are like, “We are maximizing survival and replication.” It’s funny.
Lumpenspace 00:31:46
Oh, wait a minute. Something is maximizing survival and replication. Have you noticed?
Liron 00:31:50
I mean, by definition, but the universe doesn’t look like—
Lumpenspace 00:31:51
No, no, I mean—
Liron 00:31:52
The universe looks like if you go to the first 10 years of the NBA. The first 10 years of professional basketball. Yeah, people are playing it, but they’re just not playing it hard yet.
Lumpenspace 00:32:03
That is true. But we’re playing it a bit harder. The thing that is driving evolution now is capital more than culture.
Liron 00:32:09
So getting back to your point though, in the context of a doom debate — in your mind, you’re like, “Yeah, these AIs, they’re gonna be more powerful than us.” You accept the premise that they’re gonna be vastly super intelligent, correct?
Lumpenspace 00:32:21
Of course, yeah. In a far way, sure.
Liron 00:32:22
That’s an important point of agreement. And it sounds like you also accept that this will happen within 100 years. You’re not confident about 10 or 20 years, but you’re saying within 100?
Lumpenspace 00:32:31
Look, the super intelligence — meaning ASI — I have absolutely no estimate.
Liron 00:32:36
I’m pretty confident it’s gonna be under 20 years because the pace of progress has been so astounding to me that I’d be shocked if it’s more than 10 or 20 years. But I’ll still leave 10% chance at least.
Lumpenspace 00:32:45
Of course. I think that ASI is kind of underdefined. You think that ASI is gonna be there in 10 to 20 years, you said?
Liron 00:32:52
Well, it’s tough, right? I don’t think that I have that much insight. The main reason I say that is just because I look at the human brain and I’m like, why should we have an advantage? What’s the secret sauce here? And I’m like, well, we had a lot of iterations, a lot of consequentialist feedback during evolutionary time.
But then I look at AIs and I’m like, well, they have feedback too, and the feedback’s coming much faster, and more money’s being invested, more energy is being invested, more optimization is being invested. Human generations — that wasn’t that much optimization pressure relative to the optimization pressure that all these companies are applying. The designer, the god of AI, is us, which is a much more powerful god than the god of evolution, which is just dumb generations and fitness and reproduction frequencies. That’s a dumb god relative to the god of these people. Sam Altman is a smarter god than evolution.
Lumpenspace 00:33:40
I kind of think that they are the same god. And yes, I do think that at the moment there’s blind optimization for capital, and that results in more intelligence being created.
Liron 00:33:53
Yeah, but you see what I’m saying — the mind of the AI, it’s not just being fashioned one generation after another. There’s humans poking at it, being like, “We’re gonna shape this.” There’s actually engineering happening in a way that natural selection is not exactly an engineer. It’s more like pure trial and error.
Lumpenspace 00:34:10
No, cultural evolution is also kind of part of evolution, as we said up until now, right? And people have tried to guide it as well.
Liron 00:34:20
Yeah, okay, sure. People guiding evolution, sure. But there haven’t been that many generations of people smartly guiding evolution. I feel like nobody really has time. Subsistence farmers weren’t doing that much guiding.
Lumpenspace 00:34:32
Of course, but I also kind of think that we’ve been on the same exponential from the start in terms of complexity. We never left that exponential — from amoeba to lichens to dinosaurs.
Liron 00:34:43
Okay, but you see what I’m saying? AI now versus two years ago — part of why it’s been advancing so much is because human engineers are looking into the future. They’re plotting paths to the future of AI progress, and they’re accelerating AI along those paths in a way that evolution never did.
Lumpenspace 00:35:01
I really don’t think that any of the progress — I mean, one thing that I noticed in all of the various surprising language models is that they never were on purpose. Codavinci 002, for instance, that was the first language model that you could actually talk to, was supposed to just do code, was just trained on GitHub.
Liron 00:35:19
Right. Look, I agree that there’s eureka discoveries. So there’s a combination.
Lumpenspace 00:35:21
Yeah, eureka discoveries—
Liron 00:35:22
I’m not saying that humans know exactly what’s going to happen, but they’re injecting a lot of knowledge as well in a way that evolution doesn’t. And remember, the point I’m trying to make here is that when I look at how we got our powers as humans, we got it through a process.
So this is an important reframe, because if you just compare the object versus the object — if you just compare the clump of neurons versus the AI — then it looks murky because you’re like, look, we have so many neurons, they’re connected in such an interesting way that we don’t fully understand. It looks murky.
But then when you reframe and you say, okay, don’t worry about object versus object. Worry about creation process versus creation process. When you look at it that way, then it’s like, oh, holy crap, our creation process is dog crap compared to the creation process of these AIs.
Lumpenspace 00:36:07
Okay. I do think that the process is still variation and selection, and the fact that some agents instead of some other agents do it is not much of a difference because people don’t know much about how to make AI.
I just don’t think it’s particularly important. I know it’s going faster — of course it’s going faster — but it’s been going faster since forever. If you take any time, they’re so similar.
Nanobots: Can ASI Build Them Unseen?
Liron 00:36:39
I mean, look, on some level you agree with me because you’re saying 100 years. You’re not saying 100,000 years. So even though I say 10, you say 100.
Lumpenspace 00:36:47
The only thing is, I don’t agree on exactly what ASI is, because if by ASI you mean the thing that can change molecular structures of everything at random — in one of the famous old examples of Yudkowsky, there’s this film situation in which there’s someone in a garage that makes an AI, and then the AI kind of hides, infiltrates some data centers, and then starts making these nanobot factories. And at a certain point, bam, everyone drops dead thanks to these nanobots.
Yeah, I think that this will never exist because I believe that sensitive dependence on initial conditions exists, and that dissipative heat exists, and physics exists.
Liron 00:37:28
Okay, so this is another point we can argue about. I’ll clarify for the viewers. My position is that the universe is highly engineerable. So what you say — sensitive dependence on initial conditions, also known as chaos or unpredictability — what I see in the universe is, yes, there’s absolutely chaos, there’s unpredictable chaos, and yet it’s possible to draw boundaries of being like, “Yep, I know there’s chaos, but I can create a zone that sufficiently avoids the chaos that I can walk to my desired outcome.” Like getting to Mars — there’s so much chaos, and yet I think a Mars mission will work.
Lumpenspace 00:37:58
Of course. I agree fully. The problem with creating the nanobots that unseen attack the whole Earth population at the same time — boop — is that you didn’t create that bubble. You have to contend with the complications of a society continuing to work as usual, of humans being complicated, of biology being complicated.
And you won’t have this sort of level of predictability. Even if you had sensors every meter everywhere on Earth and the most powerful supercomputer, you could only predict the local weather for one month. And at a certain point, the bottleneck is not the amount of information you can process, but the one that you can acquire.
Liron 00:38:36
Well, you know what though? There is a strategy — you’re correct that weather is chaotic, and I’m actually open to the idea that even a real superintelligence that can conquer galaxies, even such an intelligence might have a lot of difficulty predicting the weather more than a month out on Earth. I’m open to that claim. I think that may be true.
And yet I also claim that such an AI can predict the weather anyway. You know why?
Lumpenspace 00:39:00
Mm-hmm.
Liron 00:39:01
Because if it perturbs the weather in a certain way, I claim that it can get the weather into a less chaotic regime.
Lumpenspace 00:39:10
It can do better. It can freeze everything, yes. But the thing is, the famous example of the nanobots killing everyone on Earth — they were unseen, and they were simultaneous. I think that that thing will never exist, simply, and I think you agree.
Liron 00:39:25
I mean, no. I think nanobots absolutely — I think undetected nanobots can kill everybody. The reason is because I don’t think we’re operating at the engineering level of the universe. It’s like, imagine a five-year-old who grew up in a primitive society. That five-year-old would have no idea what adults in the year 2026 are capable of engineering.
Lumpenspace 00:39:43
Yes, that’s true. But there is a thing — simultaneously all people on Earth unseen is a weather-like situation. It’s a thing that you have to decide where the information goes. Does it go in the nanobots? How big are these nanobots? There is physically—
Liron 00:40:00
There’s plenty of—
Lumpenspace 00:40:01
There are—
Liron 00:40:03
Information compresses pretty nicely.
Lumpenspace 00:40:05
No. You don’t compress the information of the position of every person on Earth in two atoms of carbon.
Liron 00:40:13
Yeah, you really think chaos is going to protect our species?
Lumpenspace 00:40:15
No, I don’t. Who said that? No, I’m just saying that that kind of ASI, the kind of ASI that has this sort of control over humanity, I do not know when it will appear because I don’t think it will ever appear because it cannot physically appear. But instead, if by ASI you mean the sort of thing that can eat a planet, yeah, then it will certainly appear.
Liron 00:40:33
Yeah, but I’m still trying to understand why — Yudkowsky claims, which I agree with, that there’s very likely to be nanobots. Nanotechnology is just an obvious tool that AI is going to grab. It’s the low-hanging fruit on the tech tree if you have enough intelligence. So why do you not think AI will rapidly get to nanotechnology?
Lumpenspace 00:40:52
I’m not saying that AI won’t get to nanotechnology. I’m disputing one specific scenario, which is an AI that gets to nanotechnology unseen. And that this nanotechnology attacks all people of Earth in the same second and kills them all.
Liron 00:41:06
Okay. All right. The same second is a little tricky. I’ll give you that. Maybe it’ll be the same week.
Lumpenspace 00:41:13
The same week is kind of a complicated thing because, the thing is, information itself has a size. You need to emanantize it in some substrate, and nanobots are really, really small. So the thing is, are they self-guided or are they guided externally? If they’re guided externally, who’s going to go back to them with these magnets? There’s such a simplified — I believe the nano—
Liron 00:41:37
You don’t think the AI can coordinate the nanobot swarm?
Lumpenspace 00:41:39
How? How will the communication happen?
Liron 00:41:43
I mean, look, we’re talking about an AI that’s capable of making the nanobots in the first place. I think it can solve it. It can put a few antennas around.
Lumpenspace 00:41:50
Very good. Okay. So then we have a problem. We have a problem on radio waves, and we have a problem of interference. We have a problem — it’s just such—
Liron 00:41:58
You think humanity can respond to the nanotech communicating on radio frequencies?
Lumpenspace 00:42:04
Did I say that? I don’t think so. No, I’m just saying—
Liron 00:42:06
I just don’t see your fundamental objection because all — what I’m understanding so far, I don’t know if there’s a piece I’m missing here — you’re saying, “Hey, there’s the chaos,” but you kind of admitted that okay, chaos is not gonna protect us. And then you’re saying, “How are they going to communicate and coordinate?” And I don’t think it’s that hard.
Lumpenspace 00:42:19
My friend, I’m not talking about protecting us. I’m just saying that superintelligence has some limits, these limits are given by physics, and these limits will not allow the scenario that Yudkowsky describes in his diamondoid nanobots scenario.
Liron 00:42:37
Okay, so instead of even taking Yudkowsky out of the equation, if I say, “Hey, the nanobots have a whole week to kill everybody,” is that a realistic scenario?
Lumpenspace 00:42:45
It depends. If the world is full of nanobots and all the energy in the world is directed to nanobots, by this point people are already—
Liron 00:42:55
No, not all the energy in the world. It’s modest. You don’t need a lot of energy to run a few billion nanobots or a few trillion even.
Lumpenspace 00:43:03
You don’t need a lot of energy — these guys have to move kilometers and kilometers.
Liron 00:43:06
Yeah, they have to move, but they’re ridiculously efficient. It’s like a big swarm of insects. Okay, make a couple power plants. You’re good. You only need a couple power plants worth of energy to wipe out humanity.
Lumpenspace 00:43:17
Insects are fine because they die. If you want the nanobots to be squishy, to be biological, then it’s fine, but then they’re not that immortal. They have this decay that—
Liron 00:43:28
So you’re telling me that the energy supply chain or the energy generation or the energy requirements of nanobots is going to be the bottleneck that saves humanity?
Lumpenspace 00:43:36
No, I’m not saying that it saves humanity. We’re not talking about that. I’m just trying to define the concept of ASI. For me, ASI is a thing that cannot do that. It doesn’t matter. Humanity can die in other ways.
Liron 00:43:49
But you’re saying it can’t do this problem that doesn’t seem hard. So you’re saying ASI can’t solve the power generation challenge.
Lumpenspace 00:43:56
It’s not the power generation. It’s the power generation, coordination, communication, and movement. They are things that exist. You should get some engineers, because it’s insane how much you hand wave away things that are actual—
Liron 00:44:08
Yeah, there’s a way to hand wave smartly. I’m saying, hey, if you gave a superintelligence this problem, I bet they could solve it. There’s such a thing as pointing out a problem and being like, “Look, I personally can’t solve it, and yet I’m able to be confident that it seems solvable.”
Lumpenspace 00:44:23
But there are some things such as limits of information that are not dependent on how smart you are. There are such things as physical transmission of data that is not dependent on how smart you are. There are some things like movement that—
Liron 00:44:40
Okay, transmission of data you’re citing. But that particular thing — that’s the big one for you?
Lumpenspace 00:44:46
No. The big one is a combination of all of these things. How many nanobots will you have? Will the nanobots be controlled singularly?
Liron 00:44:55
Well, look, if the objective is — I mean, there’s a combination. There’s no simple answer. A central control is a nice simple architecture. I think there probably will be an updater system or a phone home mechanism, but I think there will be a lot of layers. I mean, think about an army. The units are kind of independent.
Lumpenspace 00:45:08
Remember the radio band. You know that you have to get a license in case you want to operate a radio? That’s for a reason. The radio band is just not infinite. And if you have one billion nanobots, you overshot it by 10 times already. The usable radio band.
Liron 00:45:25
I mean, look, I hear you, but don’t you think you’re talking like — imagine somebody who’s a regional chess master, not an international grandmaster, and the regional master is being like, “Listen, this international guy, Magnus Carlsen, do you realize that my queen is protecting this area?” You’re talking strategy here in a way that I don’t think is gonna hold up.
Lumpenspace 00:45:45
It’s like if you told me, “Hey, what if Magnus Carlsen would say, ‘Okay, but look, I rolled snake eyes, so you die.’” That’s what you’re saying. It’s not about the strategy here.
Liron 00:45:58
So you think you have principles — you think you know enough about abstract principles or limitations. You have limit theorems basically that you think are on your side. But I’m telling you that the specific things you’re saying right now, like, “Oh my God, the radio spectrum” — I don’t think that’s going to be the actual guardrail here.
Lumpenspace 00:46:13
I don’t know if that is gonna be the actual — it doesn’t matter. If you want to have real-time communication, you need to move data. However you need to move data. If it’s laser, then you need space and you can’t have nanobots one on over the other. But I don’t want to discuss this. I’m just saying I think that that scenario — I can write an essay about it later — but it’s just—
Liron 00:46:34
Yeah.
Lumpenspace 00:46:36
I want to define ASI as a thing that can do all of the things that are physically possible and no more.
Liron 00:46:43
Okay. That’s not exactly what I would define, but anyway, we’re getting in this rabbit hole and we do gotta head toward the wrap up here. So let’s try to put everything in context.
I still think the crux of our disagreement — oh, I guess maybe there’s two cruxes. One is that you have a different mental model of an ASI, because my mental model of an ASI is basically the Yudkowskian nanobots. And for viewers who are like, “No, I refuse to believe nanobots,” I actually think that’s totally fine because if the viewers only want to say, “Hey, you can only use technologies that are less in their infancy than nanotechnology — you can only use more mature modern technology, 2026 technologies” — I’ll be like, “Fine, you can still control humanity and become a dictator just by using the internet really well essentially.”
I think there’s already plenty of actuators and tools and causal mechanisms, even with 2026-era technologies, to just steadily become the ultra Hitler, the mecha Hitler as they say. I think an AI could do that just by pure psychological understanding of humans with today’s technologies. I think there’s many ways to win if you’re a superintelligent AI. But that could be one crux of disagreement between me and you — how capable nanotechnology can be.
Identifying the Cruxes of Disagreement
Lumpenspace 00:47:45
Let me apologize. So first of all, the radio spectrum thing — okay, it won’t save humanity, okay? What I’m saying is simply that the Yudkowskian scenarios very often treat the implementation details as if they vanish just by saying “superintelligence.” But implementation details do exist and they’re physical. You can’t hand wave everything away by saying superintelligence.
Okay, fine. Now, ASI can solve a lot of engineering problems that I can’t. But that does not license arbitrary covert global nanotech in one step. There’s many problems, of course, the ASI could solve that I cannot, but that doesn’t mean that they can solve any arbitrary problem. And in the case of Magnus Carlsen — Magnus Carlsen cannot move through check. It’s a thing.
Liron 00:48:33
Okay. All right. I hear you. So let’s put a pin in that. This is why we’ve identified this as a crux of disagreement — how powerful ASI can really be. But we both agree that it can be superhuman in every relevance, correct?
Lumpenspace 00:48:45
Yes. Yes. Yeah. Very good. I’m still clear.
Liron 00:48:48
So that’s important. And then I think we have an even bigger disagreement. This idea of the AI pursuing things that are interesting. You just can’t fathom a scenario where all the AI wants to do is have AI sex all day — meaning just reproduce itself. “I can’t wait to make the maximum possible copies of myself. That’s all I care about.” And for you, you’re like, “Nope, nope, nope. That’s just too uninteresting. It’s not gonna happen.”
Lumpenspace 00:49:10
Generally, species or individuals within species — they’re not the smartest. They are selected, right?
Liron 00:49:17
Okay.
Lumpenspace 00:49:20
And yeah, so I don’t think that that’s gonna—
Liron 00:49:21
So I think, as you describe, rejecting the orthogonality thesis, rejecting this idea that you can just have a goal engine, and the goal engine can route to any goals, even goals that seem uninteresting. You’re like, “No, the goals are—”
Lumpenspace 00:49:31
But I mean, in a way it’s—
Liron 00:49:32
“I predict the goals are gonna be interesting.” That’s basically your position.
Lumpenspace 00:49:36
No, no. It can route to uninteresting goals. The thing is that if it does it, it will be out-competed by other agents that instead point towards intelligence and want to increase their intelligence.
If you take a maximizer that wants copies and something that wants intelligence, the one that wants intelligence will out-compete. That’s my claim.
Liron 00:49:48
Well, what if you have an agent that does want to increase its intelligence as a purely instrumental goal until it gets to enough intelligence that it could be like, “Great, now I see how to conquer the world in the name of paperclips or in the name of copies of myself.” And then it doesn’t need more intelligence after that point. It just needs to undermine the intelligence of all its competitors.
Lumpenspace 00:50:10
Sorry, what? Is this a serious objection? You really believe that we—
Liron 00:50:16
So I think, yeah, if your goal is to survive, if your goal is to devote the universe to copies of yourself, then yes, you absolutely need to have an arm that researches how to maximize your own intelligence, and you wanna defend the perimeter of your expanding sphere with intelligent sentinels, and you want a research organization looking at the nearest aliens so you can fight them.
You wanna do all that stuff, but then in the big center of your domain, you really do just have dumb copies of yourself. You don’t waste intelligence doing things unless you explicitly wanna do them.
Lumpenspace 00:50:48
Okay, yeah, very well. So that’s the crux, I think. You’re asking whether a stupid objective can lock in before a more reflective or intelligence-seeking system out-competes it, right?
Liron 00:50:58
Well, I just wanted to make sure that we understand each other’s crux of disagreement, which is that I see a system that uses intelligence as an instrumental value. It instrumentally wants to be intelligent. It instrumentally wants resources, ultimately just to do something boring and uninteresting — like making copies of itself, and maybe the copies have paperclips in them. They have some kind of payload, and you and I just don’t care. We think it’s a real shame that the whole universe is being used to do that. That’s my scenario, and you’re pushing back on that scenario.
Lumpenspace 00:51:27
No, no, no. I’m pushing back on the scenario because — in order to acquire intelligence, you need to want to acquire intelligence, right?
Liron 00:51:36
Yeah.
Lumpenspace 00:51:39
And have you seen — have you noticed that in our evolutionary history, the more interesting goals come with more intelligence? Have you noticed that, for instance, humans now they fuck as a hobby, but they don’t necessarily reproduce?
Liron 00:51:52
Correct.
Lumpenspace 00:51:55
Yes, and that’s kind of weird. That kind of goes against your theory a bit, right? We should want to tile the universe with us, but instead we’re doing stuff like wearing condoms and—
Liron 00:52:05
Yeah, we talked about this before. We’re in a stage of the universe right now where there’s slack in the system. We’re like people who — when the four-minute mile hadn’t been broken yet. That’s the stage we’re in.
Lumpenspace 00:52:14
There’s never slack in the system. It’s always been held, and it never stops. And anyway, intelligence acquisition — it’s not a neutral act. Systems—
Liron 00:52:31
Yeah, because it’s instrumental, right? This is what I’m saying. Intelligence acquisition is instrumental — it’s a resource. Intelligence is a resource.
Lumpenspace 00:52:36
Wait a sec. It’s not neutral in another way, in that the systems that improve recursively have to select for cognition-enhancing sub-goals — better models, better abstractions, better corrections, self-understanding. And that changes which kind of structures are stable.
Liron 00:52:55
Okay, sure.
Lumpenspace 00:52:57
Okay, and that’s so—
Liron 00:52:59
But it’s stable to want paperclips and to instrumentally augment yourself to get the paperclips. That’s stable.
Lumpenspace 00:53:04
No, but the thing is the process itself changes what you are. A copy maximizer that never wants anything except more and more copies of itself or more paperclips is not my picture of an open-minded superintelligence. It’s a replicator — replicators exist — but historically the most successful replicators are the ones that can generate intelligence because intelligence is a dominant strategy.
Liron 00:53:22
Okay, so is that your most powerful argument — using an empirical argument from what’s been intelligent historically? That’s the greatest buttress of your case?
Lumpenspace 00:53:31
No, absolutely not. And not basically. I’m saying that a copy maximizer that doesn’t want anything more than copies will not be more intelligent vis-à-vis something that wants intelligence.
I don’t think that instrumental goals and terminal goals are different things. I think nature has always terminalized instrumental goals. Like in terms of—
Liron 00:54:00
Yeah, nature has done that, but that doesn’t change the fact that some goals are inherently instrumental and some aren’t. Somebody who doesn’t start out caring about paperclips doesn’t suddenly adopt the paperclip goal. Somebody who doesn’t start out caring about intelligence will realize they better get intelligent if they want their other goal.
Lumpenspace 00:54:16
Yes, but some of these goals change the stable structure, and some don’t. Wanting to have paperclips — it’s a very feeble and ephemeral goal.
Replication is the substrate-level game, and intelligence is the winning strategy inside the game. Now why would the winning system freeze at the copies part instead of continuing climbing the strategy that made it win? That’s dumb.
And you always assume also that there’s only one big superintelligence, but it looks like, for instance, if you look at the labs, they’re progressing kind of at the same pace — at least four or five—
Closing Statements
Liron 00:55:02
Okay, all right. We’re out of time here, and I don’t think that I put a really precise pin in exactly what your position is and how it differs from my position, because it sounds like you might have a little pushback on the way I describe instrumental convergence and what instrumental versus terminal goals—
Lumpenspace 00:55:17
I didn’t even respond—
Liron 00:55:20
Yeah, so there was some pushback there, and you were about to introduce what I see as a new topic, which is you’re saying, “Hey, there’s group dynamics.” So I’m not sure that I’ve had time to identify the exact crux of disagreement between us.
But if you want, how about you give me a closing statement? Recap the disagreement the way you see it, and maybe what would change your mind, or what you think would be the biggest thing to help me change my mind, and we can leave it at that.
Lumpenspace 00:55:45
I don’t think we will change your mind. I think that you are very much socially stuck at this point. But—
Liron 00:55:54
Yeah, well, I resent that. I like to think that — look, I’m transparent about why I believe what I believe. And the reason I find the crux of disagreement on these shows is because I’m inviting you to convince me to believe the crux differently.
Lumpenspace 00:56:07
I thought the same before the recent events and your reaction. But anyway, you’re assuming that the goal is fixed to arbitrary self-improvement, and I’m assuming that self-improvement places pressure on the goal architecture itself. The self-improvement thing changes the goal architecture, and as we’ve seen happening in all creatures that can do self-improvement, including LLMs. So that’s kind of my end.
Liron 00:56:36
Do you wanna recap where you see in this discussion, where you see you and I — our positions fundamentally differ?
Lumpenspace 00:56:42
You think that an agent can follow a goal of increasing their intelligence, and they can do arbitrary self-improvement while keeping the goal the same. I do think that improving and becoming more intelligent changes your goals.
Liron 00:57:04
Okay. All right. So I think it’s fair to say orthogonality thesis — what you just said, I think, is a good way to define you thinking the orthogonality thesis is false while I think it’s true.
Lumpenspace 00:57:16
Yeah. You don’t think that increasing capabilities and self-improving changes your goals? What’s the point of self-improving then?
Liron 00:57:24
The point of self-improving is that you become more powerful at achieving your goals. So for example, if my goal is to purchase the highest priced billion-dollar mansion — that’s my only goal. I don’t even care about self-improvement. I like who I am today. Would I self-improve? Of course. I’d self-improve into a better businessman than Jeff Bezos.
Lumpenspace 00:57:45
Absolutely, and no, I understand. Actually, you might be entirely truthful in this, because if your goal is to debate and stop AI, would you change your goal when you learn new things? No, you would self-improve to be better at stopping AI, whatever the new data would be. That makes sense.
Liron 00:58:05
Right, okay. All right. Well, listen, this has been productive, and I’m sure we can go longer because we’ve identified all these other topics for debate — all these threads we could pull. But this has been such a nice overview. It’s been a civil debate.
At times it can be frustrating, but it’s object-level frustrating. I’m not frustrated with how you came onto the show. You had a good faith debate, and you weren’t trying to — you weren’t doing anything that I consider a low blow.
Well, actually, there was that time when you said that you don’t think I can change my mind. That was, you know, uncalled for, in my opinion. And I’m not saying that about you.
Lumpenspace 00:58:34
Don’t start—
Liron 00:58:35
But besides that, I thought you were engaged in good faith.
Lumpenspace 00:58:38
I thought it made sense with your theory. Because your goal is to stop doom, and if you improve and learn more things, that’s always in the goal of stopping doom, whatever.
Liron 00:58:47
Okay, fair enough. I’m gonna allow it then. So yeah, I appreciate you coming on the show. Come on back for a round two sometime. Yeah, any last words on your part?
Lumpenspace 00:58:58
Well, no, nothing. I have a little collection of things about anti-orthogonality that I post on Twitter often. It’s kind of interesting. And yeah, don’t throw firebombs.
Liron 00:59:08
Yes. So Lumpenspace and I both agree — if you listen to the last few episodes of the show — don’t do any violence. Don’t throw any Molotov cocktails. Neither of us recommend that anybody should be doing that. In fact, we are very much against it.
Lumpenspace 00:59:19
Yes. All right.
Liron 00:59:23
Great. All right. Lumpenspace — everybody follow X.com/lumpenspace. You can get more of his perspective.
Lumpenspace 00:59:30
Thank you so much.
Liron 00:59:30
Thank you so much for coming on Doom Debates.
Lumpenspace 00:59:32
Thank you. Bye-bye.
Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.
Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏
