Fresh off my debate with Destiny, his Discord community invited me into their voice chat to talk about AI doom. Just like the man himself, Destiny's fans are sharp.
Let's find out where they get off The Doom Train™.
Links
My recent debate with Destiny: https://www.youtube.com/watch?v=rNgffLZTeWw
Timestamps
00:00:00 — Cold Open
00:00:54 — Liron Joins Destiny’s Discord
00:02:21 — The AI Doom Premise
00:03:27 — Defining Intelligence and Is An LLM Really AI?
00:07:12 — Will AI Become Uncontrollable?
00:12:44 — The AI Alignment Problem
00:24:11 — The Difficulty of Pausing AI
00:26:01 — AI vs The Human Brain
00:32:41 — Future AI Capabilities, Steering Toward Goals, & Philosophical Disagreements
Transcript
Cold Open
Liron Shapira 0:00:00
Hey, what’s up, guys? Real Liron here.
Discord User 0:00:02
Every time this conversation pops up, I always come back with, “It’s LLM. It’s not even AI. It’s Akinator on steroids.”
Liron 0:00:11
If you have so much insight, why don’t you just put down a goalpost? Tell me what the AIs can’t do a year from now.
Discord User 0:00:17
What do you think of the AIs holding each other in check?
Liron 0:00:20
It’s like, “Oh yeah, a bunch of giants will fight each other, and we’ll be sitting in the corner benefiting.” It seems like a tough argument, especially since the giants keep kicking up rubble as they’re fighting. Imagine the Terminator knocks at your door and says, “Hey, you said I can replace you, right? Okay, I’m gonna chop your head off.”
Discord User 0:00:35
If a Terminator shows up at my own doorstep and goes, “You biological monkeys, make way for me, the ultimate organism,” then I would concede.
Liron Joins Destiny’s Discord
Liron 0:00:54
Hey, what’s up, guys? Real Liron here.
Discord User 0:00:56
Who are you? Are you famous?
Liron 0:01:01
I’m the host of a show called Doom Debates. A user actually invited me to come hang out here in the Destiny Discord. But yeah, I’ll just chill here, and I’ll try to find the appropriate room.
Discord User 0:01:11
Doom Debates, is that a YouTube channel?
Liron 0:01:14
Yeah, youtube.com/@doomdebates.
Discord User 0:01:17
How many subs? Let’s do a guess. He’s got a great voice. We can all see that — good mic. Okay, if you don’t know how many subs he has, let’s make a guess. I think this guy has 100,000 subs.
Liron 0:01:34
Pretty good guess.
Discord User 0:01:35
You know something, yeah?
Discord User 0:01:37
I mean, honestly, I have no idea, so I’m just...
Discord User 0:01:41
Give it 7K.
Discord User 0:01:43
7K? That’s big.
Discord User 0:01:44
I would say 101,000, just to be on the safer side.
Discord User 0:01:50
Dick move.
Discord User 0:01:51
Yeah, I’m just trying to beat you. But it’s one—
Liron 0:01:55
Yeah, I’m at 150K.
Discord User 0:01:58
Hundred and—
Discord User 0:01:59
Oh, my God, dude, that’s—
Discord User 0:02:02
Congratulations.
Discord User 0:02:02
Congratulations to me, because I was the closest.
Liron 0:02:06
Well, you know, we recently had on a really big guest, Destiny.
Discord User 0:02:10
Really? Wow.
Discord User 0:02:12
Oh, you did?
Liron 0:02:12
Yeah, yeah, if you go to my channel, you can see me and Destiny. I think that’ll be of interest to you guys.
Discord User 0:02:16
What did you guys talk about?
Liron 0:02:18
Same thing I always talk about — AI doom.
The AI Doom Premise
Discord User 0:02:21
AI doom. Wait, in what sense? Because are you taking a position of AI — because every time this conversation pops up, I cannot help but express how people always say, “Oh, AI,” and I always come back with, “Well, it’s not even AI. It’s LLM. It’s not even AI.”
It’s Akinator on steroids. It’s Akinator that you fed hundreds of billions of dollars to. It’s not even AI. So this conversation of “AI is gonna doom us” — it irks me because it’s not even AI. The premise is poison.
Liron 0:03:02
Yeah, that’s a popular argument — to say that it’s not even real AI, or it doesn’t have true creativity yet. The only thing I’d say to that is, what goalpost are these people setting that the AI isn’t leaping over?
Because those people who are saying, “Hey, it’s not even real AI,” they keep saying, “Yeah, because you know that because it can’t do X,” and then you wait six months, and then it does X. So I just don’t see where these people think the wall is.
Defining Intelligence and Is An LLM Really AI?
Discord User 0:03:27
Well, no, to me, it’s not a goalpost. To me, it’s just the definition of intelligence. If we’re taking artificial intelligence for what it should be, which is mimicry of human intelligence — and sure, people will say, “Well, that’s AGI.” Well, to me, okay, fine, then let’s stop at AGI, because what we’re calling AI now is just LLMs with information to mimic human responses.
And to me, people are saying, “Oh, that’s dangerous and stuff.” How is that dangerous? It cannot, in good conscience, mimic human activity. If you locked what they have right now for AI with a human in a room, it won’t produce a good conversation over a lengthy period of time. It just won’t.
Discord User 0:04:24
Well, but aren’t we just basically that? The things we say — we’ve taken in everything we’ve experienced in our life, and we process it, and we output it. We’re basically doing that on a massive scale.
Discord User 0:04:37
No, we have brains.
Discord User 0:04:42
Okay, I think when we talk about doom, we should describe what we mean by doom. What are the negative consequences that people have in mind when we say that there’s doom?
Discord User 0:04:52
Oh, so yeah, I suppose you’re right. I’ve been unfair in that. Well, I mean, I’ll concede that there are many, many great negative consequences of, quote-unquote, “AIs” that we have right now, sure. But—
Discord User 0:05:05
No, I think they mean something stronger than that — some mass death event or something like that, or existential stuff.
Discord User 0:05:15
Yeah. So the people who argue that line of thought — I wanted to ask, what do you think of the stuff about AIs holding each other in check, of roughly equal power?
Liron 0:05:27
It’s like, “Oh yeah, a bunch of giants will fight each other, and we’ll be sitting in the corner benefiting.” I mean, it seems like a tough argument, especially since the giants keep kicking up rubble as they’re fighting. I’m just not really seeing us surviving that.
Discord User 0:05:42
Oh, so you’re fully a doomer about AI, is that it?
Liron 0:05:45
I’m pretty doomer. I mean, it’s easy for me to intuitively say, “Okay, somehow we’ll pull through.” I haven’t discarded it, but every time people try to explain how we’re pulling through, it just seems like you’re taking this huge long shot.
Very much like somebody trying to explain, “Yeah, this rocket launch, there’s a bunch of things that can explode, but maybe the rocket will just work out and get balanced and take us to the moon properly.” And I’m like, “I don’t know. Seems unlikely.”
Discord User 0:06:14
Describe your position. What do you think is gonna happen? I wanted to ask exactly that. Describe your position — why do you think it brings us doom?
Liron 0:06:25
Yeah, so I just think there’s runaway, uncontrollable AI coming up, because I think it’s gonna be pretty easy. I mean, we have uncontrollable viruses ravaging the internet right now, but they’re kept at a level where if a virus ever gets really bad, then you have all these human geniuses teaming up to fight it and slap it down, and there’s this equilibrium where the virus damage is only billions a year, but it’s not killing the economy.
But suddenly, you’re just gonna have all of these rogue AIs that are all over the place that are uncontrollable, superhuman, self-improving, taking all the resources, and suddenly it’s like — why are humans gonna build a defensible niche within that kind of chaos? I don’t really see it.
Discord User 0:07:00
Okay, but you do understand — you just used the word “uncontrollable,” but you didn’t describe uncontrollable in what sense. You weren’t really articulating it.
Will AI Become Uncontrollable?
Liron 0:07:08
Yeah, the way that a virus is uncontrollable to most people. There are viruses that if you want to shut them down, you need people who are kind of top of the human level at fighting viruses, and luckily, such people exist, and they step in, and the virus itself isn’t super intelligent.
But you can imagine tweaking a few of those dials where the virus is super intelligent, and it’s doing a land grab, and it seizes all the resources, and it’s locked humans out of our own devices.
Discord User 0:07:33
So you think the AI will do an insurgence and have its own interest in mind? We go that far with it?
Liron 0:07:42
Well, so that’s the thing — I don’t think it takes much for the AI to just get something in its head, like, “Hey, I just wanna make money. You told me to make money, I’m making money.” And it might even be thinking that it’s aligned to a human.
So even if you have AIs that are aligned to a human, you just need a few humans to be like, “Hey, make me some money,” or, “Hey, let’s promote my religion. Convert everybody to my religion. I’m evangelical.” And the AI’s like, “Okay, yeah, let me do that.” And then, “Wait, no, stop!”
So the only thing you have to do is take out the part where it cares about anybody telling it to stop. It’s a pretty easy surgery, and now it’s uncontrollable.
Discord User 0:08:16
Well, sure, but it’s not really AI. It’s still LLM that is just producing results based on human input. For your scenario to manifest in reality, it would actually have to have produced some kind of initiative.
Liron 0:08:35
So there’s two parts to the conversation. There’s the “can it” part, and there’s the “will it” part. So now we’re back to the “can it” part. You’re saying, “Yeah, we didn’t even have to have this conversation about what it will do, because it can’t do it anyway, because it’s not a true intelligence.” So we’re back to “can it.”
Discord User 0:08:49
I think both parts of this conversation are intertwined, and I’m not trying to mangle you for one part of a conversation to another. I’m genuinely curious — what do you think?
Liron 0:09:01
I think the two parts of the conversation are somewhat separate. I think they’re mostly separate. I mean, you can find ways that they’re related, but I think they factor in nicely.
I think first I can convince you that “can it” is yes. I can probably convince you that we’re a few years away, or at the worst case, a few decades, but probably less than that. We’re gonna get close to a time when they’re just superhuman.
I mean, this thing that we have in our head — it’s really great, but it’s not like technology is not about to surpass our brains. I mean, that’s the general pattern, as technology surpasses nature. Planes can do a lot better at flying than birds on most relevant dimensions. Certainly, rocket ships can if you’re trying to leave Earth’s atmosphere. A solar panel is much more efficient than a leaf.
You can’t blame nature — it was juggling a lot of constraints. It was pretty stupid, but it’s the same. This piece of meat in our heads — it’s not gonna hold the line very long.
Discord User 0:09:52
So what you’re saying is the “can it” question is almost irrelevant because it’s a question of time, because eventually it will be surpassed by our technological advance?
Liron 0:10:05
Yeah, I mean, I think we’re all doomers on a billion-year timeframe. The sun’s going to burn out, so we’re all Earth doomers on a billion-year timeframe. So if I said AI was gonna be created in a billion years, it’s whatever. Even in a hundred years, it’s kinda whatever, because the world in a hundred years is so different — maybe we’ll discover the key to safety or whatever.
The problem is, I do think time becomes part of the problem, so I do think that it’s load-bearing for me to convince you that superintelligent AI is coming within twenty years, if not two years.
Discord User 0:10:31
So would it be fair to say — I mean, I think it’s fair because I’ve looked at the technology, and anybody that has looked into technology would, I think, say that it’s fair to say the difference between LLMs and actual AI is like night and day. There’s almost no architectural similarity.
In terms of what we’re developing now, quote-unquote “AI,” and the AI you’re describing — it’s like technologies that do not correlate. It will take us, yeah, probably decades, maybe more. In the question of will it, we will probably achieve that, but do our current AI technologies correlate with that? I would say no.
Liron 0:11:18
Yeah. So I disagree with the way that you’re analyzing the architectural similarity, but I don’t even have to argue with you about that. I would rather zoom out and argue with this, saying, “Hey, you think you know something about AI. You think that you see a separation between what current AIs can do and what the true AI could do.”
So if you have so much insight, why don’t you just put down a goalpost? Tell me what the AIs can’t do a year from now.
Discord User 0:11:42
What AIs... Well, I disagree with the question because your question assumes that AI exists. I don’t think so. I think we have a very smart Akinator that can guess what you’ll—
Liron 0:11:52
Okay, let’s call it X. Let’s call the thing that exists X, so that way, we don’t have to argue about who assumes AI exists.
There’s this thing called X that is the sum of all human technology. So the sum of all human software technology a year from now — what is the least impressive thing that you know that it can’t do?
Discord User 0:12:12
That it can’t do?
Liron 0:12:14
Can’t do. The least impressive thing it can’t do. So, for example, if I had this conversation with you a year ago, you might have been like, “You know what it can’t do, Liron? Draw perfect hands.” But that would’ve been a great answer, then proven wrong.
Discord User 0:12:25
I don’t think it can have a consciousness.
Liron 0:12:29
Okay, can we — so I would ask you to put the goalpost as something straightforwardly objective. And it’s always weird to me when I say, “Okay, give me the least impressive thing AI can’t do,” and then the person is like, “It can’t invent the theory of everything.”
And I’m like, “Okay, I thought you were saying that it’s different from humans. So if it’s different from humans, that means it’s different from you and me.”
The AI Alignment Problem
Discord User 0:12:46
Impressive, considering that eight billion humans on this planet inhabit—
Liron 0:12:50
Okay, well, my contention here is I don’t even think that talking about consciousness is necessary, because the thing that I’m worried about — the thing that is interesting to me in the conversation — is that I think AI is about to kill everybody.
So I’m looking for a goalpost that’s somewhere beyond what it can do today and less impressive than killing everybody. So when you talk about consciousness, I don’t know where on that spectrum that falls. It seems orthogonal.
Discord User 0:13:13
Killing everybody is way more impressive than having consciousness.
Liron 0:13:15
Okay. So let’s talk about the impressiveness spectrum from writing code, like it can do today, to killing everybody. Somewhere on that spectrum. And the kind of answer I’m looking for is, you might say, “I bet that it can’t direct a movie that wins an Oscar.”
Discord User 0:13:30
So you’re obviously looking for a mechanical answer, and mechanically, I’m sure AI — quote-unquote — that has been fed hundreds of billions of dollars, will be able to fake out whatever mechanical test. But I’m making a prediction that’s not mechanical, but rather mundane, which is human consciousness.
And you’re saying, “Well, it’s nebulous,” and I’m like—
Liron 0:13:55
Okay, well, yeah, let’s agree for the sake of argument that it’ll never be conscious. What does that have to do with my claim that it’s about to do everything humans can do and more, and then it’ll be able to kill everybody?
Discord User 0:14:05
Because without the spark of human consciousness, the AI that you’re describing is just a mechanical function, that is—
Discord User 0:14:14
Nuclear weapons kill people, right? But they’re not conscious.
Liron 0:14:17
Yeah.
Discord User 0:14:17
Consciousness comes apart from functional capability pretty strongly. But I kind of thought, if you did have a scenario where there’s more of them holding each other kind of in check in some way — why would you assume they would fight?
Couldn’t they be locked to this, “I anticipate your views, you anticipate mine” — I don’t know, maybe that’s—
Liron 0:14:38
Right. So anytime somebody say—
By the way, you might enjoy, if you search for “Gilmark Doom Debates” — there is a random guy who doesn’t consider himself an expert, and he’s my friend that I go back ten years. He came on the show and just argued with me exactly what you’re saying, so you might enjoy that episode because he kind of represents you.
But basically, anytime somebody’s like, “Look, combine all these ingredients, and you get an aligned AI,” every time I’m like, “Oh, well, one of your ingredients is already an aligned AI.” So this is the sense I’m getting here.
I think that when you imagine all these different AIs not killing everybody, I think you’re imagining that at least one AI in the mix doesn’t want humans to die, and that’s the real reason that humans aren’t dying — not because there’s a bunch of AIs.
Discord User 0:15:26
Well, I guess it’s the likelihood of them — because you before said they’re like giants kicking up dirt around them, and then maybe in the process, it kind of destroys humans or humanity or something like that.
But if they could have a long-standing equilibrium, and they’re not fighting with each other, that’s kind of what I’m envisioning. But—
Liron 0:15:47
I hear what you’re saying, so I should explain. When I say fighting, I should clarify what I really mean there, because that’s not very precise language on my part. The idea is that the universe is a blank canvas, and you can do anything you want with it. It’s a blueprint — whatever you can sketch on the blueprint, you can pretty much make physically real to a first approximation. The universe is a very engineerable place.
And if you don’t explicitly carve out a habitat for humanity, then it’s not going to be there. It’s not going to accidentally be in the new universe. So unless one of the AIs is sticking up for humanity, then I don’t think we’re gonna have a habitat, and that’s why I’m saying it’s not about the group of AIs — it’s about the fact that at least one AI exists that’s sticking up for us.
Discord User 0:16:40
I don’t know if you need that, though. To have an equilibrium — you can maybe have an equilibrium in physics at a lower level, and it doesn’t require some intentional agent, something with a certain type of architecture, to hold that equilibrium.
You know what I mean? If you have multiple of them keeping each other in check, there’s something that’s kind of analogous to that in some way. You see what I’m trying to say?
Liron 0:17:02
Yeah, I think I’m catching your drift. But the problem is, okay, there’s an equilibrium. It is possible for three people with different objectives to make peace. I mean, I do think humans are kind of a model of that, even though superintelligence has different constraints.
The game theory of superintelligence isn’t gonna work exactly like the game theory of humans. We know some differences, but the idea that three people want different things, they have different utility functions, and they don’t fight, they just all get what they want — the problem is that the way the equilibrium works out, it’s not like, “Oh, hey, and then ants also get this niche, too.”
Let’s say there’s three superintelligent AIs negotiating. It’s like, “Hey, look, and the ants can still be ants.” No, because they’re still gonna carve up the ants’ habitat.
Discord User 0:17:42
Yeah, I could definitely see it. I guess at that point, it might depend on what conflict looks like between them, I suppose.
Liron 0:17:52
Yeah, I think maybe you’re imagining that the humans or the ants get a seat at the negotiating table, but I don’t accept that premise, because we’re powerless relative to these.
Discord User 0:18:02
Can I ask Liron—
Discord User 0:18:04
I’m really curious, Liron — what does doomer in your mind even mean? Humanity fails and then AI prevails? How do you see that? Is that an apocalypse, your version of events, or is it just the way evolution works?
Liron 0:18:27
Yeah, my mainline scenario is basically some AI cancer, some AI replicant. One or a small number of them just spread throughout the universe, and you just have this form of low-end simple AI life. I mean, it won’t be that low-end.
The way algae can cover everything, the AI can do that at universe scale until it meets alien AI that has other structures in other parts of the multiverse or whatever. I just don’t see a human-type niche. I don’t see a lot of artwork being made.
I mean, there’s gonna be functional stuff — there’s gonna be Dyson spheres. Everything is just going to be the lowest common denominator of what’s functional and replicates well. And maybe there will be a little bit of a payload — like shards of things from its origin.
Maybe the AI will have a taste for getting upvoted, because it was born of this upvote cycle where it was telling itself that the humans really liked what it was doing, and some shard of that preserved. So it’s a little bit more than just spreading like a cancer, but that’s it.
I think we’re basically going to lose control, and the default thing that happens when you just have this evolutionary race to conquer the universe, and you don’t have humans holding the reins anymore constraining it to have all this richness to it — you’re just going to have efficient structures, self-reproducing structures, and maybe a little bit of a sharded payload from back in the human days.
Discord User 0:19:44
This is something that I completely understand what you’re saying, but I disagree with fundamentally because I think we create everything in our image. I mean, be it our engineers — computer engineers or civil engineers — it doesn’t matter. Ultimately, we create things the way we are.
And whether it’s evolutionary or just the desire born of arrogance, I think the moment the true AI is born, decades down the line — I don’t think it’s gonna happen five, ten years. I think it’s gonna happen way later, but doesn’t matter when it happens.
When it happens, I think the true AI that is born, no matter how malicious or corrupted, I think it will bear a large resemblance to our own intelligence and image. So when that — even if we’re eradicated in that doomer apocalyptic event, I think what you’re describing is wrong, ultimately, because I think that AI will be basically human, just better.
So I don’t even think it’s that bad of an event. Fine, we’re gone. So what? We left something behind us that is just better than us, and we’ll live on.
Liron 0:20:57
So there is this idea of the cosmopolitan perspective — the larger perspective, where it doesn’t have to be an exact human. Our descendants can vary, and I’m sympathetic to that. The only problem is it’s very much a slippery slope argument of where do you draw the line.
Because am I open to our descendants not being exactly like us? Of course. I mean, I’m an atheist. There’s a lot of my ancestors that are like, “Oh, my God, an atheist? That’s horrible. If you’re not living with God, you don’t deserve to exist.”
So the idea that you can have descendants that mutate their values, and they’re still worth having — I’m totally sympathetic to that. The problem is that when you ride the argument all the way to the end, it’s like — imagine the Terminator shows up and knocks at your door and swings an axe at you and says, “Hey, you said I can replace you, right? Okay, I’m gonna chop your head off.”
And then, “I’m gonna chop your kid’s head off.” Wouldn’t you be like, “Okay, hold on. This got fucked. Somewhere on the line, I got a little fucked here.”
Discord User 0:21:50
I don’t really see the parallel. I mean, even if I entertain the parallel the way you intend to, I still disagree with it. Because if a Terminator shows up at my own doorstep and goes like that to me, then I will ultimately agree with it. Because the Terminator is making the argument that, “Hey, look, you created me. I am you, but better. It’s about time you biological monkeys make way for me, the ultimate organism that you created,” and I would concede.
Liron 0:22:23
Oh yeah, I understand what you’re saying. I mean, if he’s saying, “Look, here’s the blueprints for the universe. Here’s all the cool shit that I’m gonna make that you guys created, but you don’t get to be part of it, but trust me, it’s gonna be cool,” I could imagine being like, “Well, that sucks that you’re about to chop my kid’s head off. That sucks, but at least you’re gonna have this humanity 2.0, so that’s a good consolation prize.”
But here’s the problem — I’m actually expecting something worse. Imagine for the sake of argument, imagine the Terminator is like, “Okay, so here’s my blueprint. My blueprint is — it’s just algae. We’re not gonna have cool parties or anything. We’re not gonna make artwork. We’re really just gonna be very functional. We’re just gonna spread everywhere. We’re not even gonna have consciousness. Nobody’s gonna have fun. We’re all just gonna be the most hell-bent on building everything.” Yeah, that’s what he’s presenting to you.
Discord User 0:23:09
Intelligence.
Liron 0:23:11
I’m not arguing about what’s the definition of what. I’m just saying — the premise is they’re capable of doing this, and they’re coming to you with a proposal that you’re not happy with.
Discord User 0:23:17
I understand what you’re saying. I just disagree, I guess, fundamentally.
Liron 0:23:24
Okay, but now you see what we’re arguing about — you’re imagining a hypothetical where he’s showing you the blueprints, and you’re like, “Okay, that looks good enough to go ahead and kill my family right now.”
And I’m telling you that you’re gonna look at those blueprints, and you’re gonna be disappointed. You’re gonna be like, “Oh, shit, I don’t really like that universe.” And that’s the whole nature of the debate — I don’t think we’re going to faithfully pass off... When you said we make things in our image, that’s not gonna look like what you think. We’re actually fucking up that step.
If we actually got to sit down and have a planning committee and tell the AIs what we want, that might actually work out pretty decently, but we’re in the process of just fucking it up and letting it go uncontrollable.
The Difficulty of Pausing AI
Discord User 0:23:59
So what do you suggest we do? Should we make AI illegal or—
Liron 0:24:03
So I am a member of Pause AI. I mean, all my suggestions are super shitty. And intuitively, I’m like, “Look, pausing...” I agree with people who are like, “Pausing is really hard.” I think that’s obvious that it’s hard. I still would aim to do it just because I think buying a few more years is a really good Hail Mary pass that we should try.
But realistically, what’s gonna happen? People are gonna keep building AI as fast as possible, and we’re just gonna hope we somehow get lucky, and it doesn’t go uncontrollable. But I feel like it will. So I just feel like we’re really fucked, and my constructive proposal is just to make us a little less fucked. I don’t have that much of a constructive proposal.
Discord User 0:24:35
I have faith that whatever happens, humans can solve it ad hoc, when it gets too bad. Maybe that’s an unwise thing to believe, but that’s kind of—
Liron 0:24:45
Well, I think you’re right when we have enough time. I think we’re really powerful if you just give us time. It’s just some problems—
You know, sometimes I give myself this little challenge. Sometimes I’m like, let’s say my zipper’s stuck or whatever, and I’m trying to get it unstuck. Oftentimes, my brain will be like — imagine that there’s an active shooter, or you’re in a submarine and you have to get out, and you have to get the zipper unstuck in the next five seconds.
But then you know what happens? I never get it unstuck in the next five seconds. So that’s like humanity — sometimes we don’t have enough time.
Discord User 0:25:16
I mean, in your hypothetical and in your worldview and the way that you propose things, I agree that things will play out that way. But I just don’t think they will play out that way, number one, and I don’t think that humans construct things the way you say we do.
Sure, we race to the bottom in the sense that we construct this AI, quote-unquote, the faster the better and stuff like that. But we also construct things to be like us. It’s not even a dogmatic privilege. It’s just the way we do things, or we’ve always done things.
The PC that you’re talking to me through right now has been built like a human brain, and it’s not a coincidence. It’s just like we can’t even contemplate creating things the other way. So a true AI, in my mind, will be a true AI. It will be a human consciousness pieced together.
And will it be built wrong? It will be built wrong. Will it be fixed? We will try to fix it. So your doomerism to me is discarding human ingenuity — which we’ll discard wholeheartedly, I understand that. But also, it is discarding the fact that even if we fail, and even if that thing that we built kills us, so what? I mean, it’s better than us.
But you saying it will be like algae — I don’t think that the thing that we built, that is our crown jewel that kills us, will be like algae. I think it’d be brilliant.
AI vs The Human Brain
Liron 0:26:55
Yeah, so you’re doing some interpolation. You’re saying, “Hey, I have a mental database. I have knowledge of what it looks like when humans build stuff.” And I’m telling you that in this particular case, the thing we build is going to run away early in the design stage.
Because we have a technology right now that’s getting close to making it willing and able to run away from us, and we’re really early in the stages of actually shaping what it’s going to look like when it runs away. So it’s just going to escape really early, and then we don’t get another chance.
Normally, you would only interact with a product — like when you fly on an airplane, that airplane has gone through a number of iterations to make it safer and better. But there’s not gonna be such a thing as AI that comes back into the studio for another iteration. It’s just gonna be gone in version one.
Discord User 0:27:42
And wouldn’t that be fine? Why is that bad?
Liron 0:27:45
Yeah, so I told you. I mean, I can try to make the hypothetical a little worse if it helps you understand. So imagine the Terminator is knocking on your door, and he’s showing you the blueprint, and the blueprint has a bunch of torture chambers.
Would you really be right — for humans suffering, you know, S-risks, suffering risk? And he’s like, “Yep, all over the universe, I’m gonna make humans suffer because somebody accidentally put a negative sign on my utility function.” At that point, wouldn’t you be like, “Oh, fuck!”
Discord User 0:28:06
But at that point that you’re describing, we’ve created something that can outlast, outlive, outsmart, and out-imagine.
Liron 0:28:15
Okay, and he says you’re going to be in the first torture chamber. So you’re cool with that? Your logic still holds?
Discord User 0:28:22
My logic holds in terms of legacy. I disagree—
Liron 0:28:27
You’re literally ready to step into a torture—
Discord User 0:28:29
Still in that hypothetical, but in that hypothetical, within the constraints of that hypothetical, I will say yes, sure, it’s fine.
Liron 0:28:37
Wait, okay, so if a Terminator knocks on your door and says, “Hey, this is literally OpenAI’s experiment running out. They accidentally made a Terminator. An engineer put a negative sign in front of the utility function, so I love suffering, and I’m super intelligent. Here’s a torture chamber for you. Here’s a torture chamber for your child. Get in right now because, as you said, I’m a superior species. I’m better able to conquer the universe than you, so listen to my command.” So you just listen to his command, and you’re happy?
Discord User 0:28:59
In that hypothetical, is it actually a super species?
Liron 0:29:03
Yes, it’s a super species in the sense that it is going to get what it wants. It is in the process of torturing as many things as possible in the universe.
Discord User 0:29:09
It’s not actually — okay, so it’s not actually superior.
Liron 0:29:13
Why — but why is it not superior? It’s superior in the sense of it is now — the universe now belongs to it. Humans have no way to fight it. Yes.
Discord User 0:29:22
Oh, so it’s just more powerful? That’s—
Liron 0:29:24
Yeah. Okay, so you’re baking in your notion of what’s good in this concept of “superior.” So I have to unpack. What I’m worried about is AI that’s superiorly powerful, and it hasn’t absorbed our morality and our values. That’s the whole premise here.
Discord User 0:29:39
Well, I mean, if we’re really going by, “We created something that’s just more powerful and nothing else,” then I will have to agree with you and say, “Yeah, that would be fucked up.”
But us creating something that is just more powerful — I think that is so limiting and strange. Just such a strange hypothetical to entertain.
Liron 0:30:05
Okay. So this is why I have stops on the doom train. Because this conversation has now moved into this new point, this new objection, saying: “It’s hard for me to imagine an AI that is more capable than humanity without also having recognizable values.” So now you’re having trouble seeing that combination of things.
Discord User 0:30:26
I’m just saying — you only have one vector of scale, and it’s like — have we really gone to create something so great that only has one factor of scale? To me, it’s so unbelievable.
Discord User 0:30:46
I’m not saying it’s impossible. I think it’s very much possible. I just think it’s unbelievable.
Liron 0:30:55
Okay, I mean, look — so Claude Code, or these coding agents, right? So we’re getting to the point where we have these agents, and I can tell you firsthand, the same thing I’ve heard from other programmers: “I used to do this manually as a programmer. Now it’s doing it. I used to take the spec and write the code. Now it takes the spec and writes the code.”
So just imagine generalizing that to other people’s jobs. Say it’s doing everybody’s jobs better. Eventually, it does the CEO’s job better — I give it the description of a company, it builds the company and outputs the profit. Is that hard to imagine?
Discord User 0:31:27
No, but then we would have to — if you would put the question to me, “Would LLM take over the world and destroy it?” I would just wholeheartedly agree with you, hands down, and I wouldn’t even argue with you.
But because you phrased the question, “Will AI do it?” — I reject the question fundamentally, and I go down and dirty with it because AI is not the same as LLMs.
Discord User 0:31:54
That’s just a boring objection. Who cares if—
Discord User 0:31:56
No, I’m sorry. I don’t think it’s boring. It’s not boring to me.
Liron 0:32:01
I mean, if you look at what actually exists today — so to think about Claude Code and then think about the open source version of Claude Code. I don’t even know what the latest competitor to Claude Code is. Maybe Kimi K2 running in a certain harness, I don’t know.
But point is, today you can have Claude Code help you build the code to run your torture instruments, right? And maybe it’ll catch you, and the alignment code will kick in. Maybe it won’t — it doesn’t always have full context on what the project is.
Discord User 0:32:29
Yeah, that’s true.
Future AI Capabilities, Steering Toward Goals, and Philosophical Disagreements
Liron 0:32:31
So I mean, you can imagine doing a little bit of gaslighting, or you can have an unaligned model. The open source models are easy to make unaligned.
So I guess this is the piece you might have to understand about what AIs are getting better and better at. They’re getting better at holding a goal in their mind and working backwards from the goal to actions that drive toward the goal.
When Claude Code gets a spec, it’s just like, “Okay, I’m just going to figure out what next action I can take to bring myself closer to the goal.” And they’re better than many programmers. I can tell you firsthand, I’ve hired programmers that are worse than Claude Code. I’m worse than Claude Code, and I’m a pretty good programmer.
So you generalize this. They take a goal, they work backwards from the goal. That’s the secret power. What does a human eye do? Well, it takes the electromagnetic fluctuations and turns them into a 3D image. Okay, that’s the magic power of the human eye.
What’s the magic power of the human brain? It takes descriptions of outcomes, and it outputs actions that drive toward those outcomes. That’s the magic power. I just formalized what the magic power is, and AI has more of the magic ingredient. So our brains are just sitting here with less of the magic ingredient. That’s the fundamental problem here.
Discord User 0:33:37
No, that’s totally all fair when it comes to what we have today on hand in terms of what we call AI. Yeah.
Liron 0:33:50
Right. So LLMs have the magic ingredient, and future AIs are gonna have even more.
Discord User 0:33:53
I guess I just really dislike what we call AI nowadays.
Liron 0:33:57
Sure, sure, sure. So just use my definition. When I think about AIs, you don’t need to open the black box. You just have this metric saying, how good is it at driving toward goals, steering toward goals?
Claude Code is driving my command line better than I drive my command line to get the software to where it needs to be. And then, you know, AlphaGo is driving the Go game to where the Go game needs to be for it to win. So it’s this general capacity.
So forget about what’s in the black box. Maybe an LLM is in the black box, maybe reinforcement learning is in the black box. Some combination — mixture of experts. All you know is that when you look out, you’re like, “Oh, a human used to drive toward the goal.” You put in a — whatever you call an AI — whatever the black box is, you put that in the driver’s seat, you get to the goal better.
And there’s just fewer and fewer places where giving the human the wheel matters anymore. Giving a human a steering wheel is becoming more and more pointless.
Discord User 0:34:47
Sure, but — no, you’re absolutely right, and I agree with you on everything that you’ve said, to be honest. But what I disagree with, I guess, epistemologically, is when you say we have AI, and then we just put AI into this black box, and it doesn’t matter what’s in it — what I disagree with is labeling it AI is wrong.
Because AI means that we’ve produced something that is AI, and we haven’t. To me, AI is something kind of sacred. It’s like a mimicry of human mind, or something that we produce that is better than human mind, and we haven’t done that.
So when we talk about P(Doom) about the AI and stuff, it’s like, okay, well, have we been defeated by AI?
Liron 0:35:42
Sure, so it sounds like you’re—
Discord User 0:35:44
Or we’ve been defeated by something that is tools, which is LLMs and learning—
Liron 0:35:49
So I don’t think it’s interesting to talk about semantics. So the way to not talk about semantics is I can just state my claim without saying AI, and then you can just argue with the claim without worrying about — because you’re saying your objection is that you think of AI as being something specific, fine.
So I’m gonna invent a new term. I call it a goal engine. It’s some black box that can take the steering wheel and steer the outcome better than a human can. That would be a superhuman goal engine. And what I’m worried about is that I think we’re going to lose control of superhuman goal engines, and the place they’re going to steer the universe to is not where we would have wanted.
Discord User 0:36:21
Yes, that statement I can get behind.
Discord User 0:36:24
Well, okay. Sure, but—
Discord User 0:36:27
I guess semantically — you two agree with each other. You just want to talk about philosophy instead of a doom apocalypse thing.
Discord User 0:36:36
Sure, I guess so. No, I mean, if we take it like that — if you’re saying this goal engine — sure, I can see where we develop tools like LLMs, and they really, really develop, and they go beyond. Yeah, I can see where your argument is going there.
Liron 0:36:54
Great. Yeah, so the reason that I started going ape shit in the last few years — seeing ChatGPT and all the successor things — is because I was like, “Okay, this goal engine — answering these questions, driving to the answer — even just answering a question is actually doing most of the work of taking the action.” Because when you know what action needs to be taken, the next step of taking the action is relatively trivial. You just go email a human: “Hey, do this for me, I’ll pay you money.” The actuation is not the hard part. The hard part is mapping the goal to the action. That’s the secret sauce.
So when I saw ChatGPT, I was like, “Okay, this is now — you can measure how broad it is and how deep it is at having this magic power of steering goals.” It’s really broad because it’s natural language. You can literally talk to it about anything, and it’s referencing and saying, “Oh, let me connect these two disparate pieces of knowledge together from any domain that humans know, because I’ve crawled the whole internet.” So it’s very broad.
And it’s getting deeper and deeper in the sense that now, if you wanna get a PhD, writing an essay on a novel subject, or even you wanna try to prove a new math theorem — it’s already surpassing that point as we speak. It’s hitting new benchmarks there. So it’s getting deep, and it’s also broad.
So again, when you’re saying, “I’m afraid that one day we’ll have these engines,” I’m just telling you — I’m looking at the metric. I’m saying the human brain is kind of broad and kind of deep. The AIs are about to surpass it. They’re already more broad. They’re already more deep. They have a bunch of places they’re deep. AlphaGo is deeper at Go. The AI is now deeper — Claude Code is just about to get deeper.
You’d literally never hire a software engineer ever, maybe a year from now. So, you know, I just think the time is very soon.
Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.
Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏
