Dr. Mike Israetel is back in the arena to FIGHT ME (verbally) about the likelihood of AI killing everyone vs. the likelihood of AI making the universe totally awesome!
Back by popular demand after last year’s widely-viewed debate, the renowned exercise scientist, fitness entrepreneur, and low-P(Doom) AI futurist reveals where he gets off the Doom Train™, why he thinks AI is conscious, and how he pictures the coming AI utopia.
Links
Watch Dr. Mike’s AI & futurism YouTube channel —https://www.youtube.com/@mikeisraetelmakingprogress
Dr. Mike on X — https://x.com/misraetel
Dr. Mike’s ASI prediction tweet (Feb 2026) —
Make sure to watch Round 1 with Dr. Mike Israetel (May 2025) —
Liron referenced a recent episode, “Steven Byrnes Returns — 90% P(Doom) on the Trajectory to ASI” —
Timestamps
00:00:00 — Cold Open
00:00:48 — Returning Champion Dr. Mike Israetel!
00:02:13 — What’s Been Your Biggest AI Productivity Gain?
00:03:46 — Can AI Be a Personal Trainer? Dr. Mike Judges Liron’s Technique
00:09:41 — Get Good at Taking Advice From AI
00:14:23 — Dr. Mike Israetel, What’s Your Latest P(Doom)?™
00:20:12 — Liron’s Mainline Scenario: Loss of Control
00:21:39 — Mike’s Problems With a Paperclip Maximizer Scenario
00:26:00 — What Will the Goals of a Superintelligence Be?
00:28:49 — AI Labs Have Bit Off More Than They Can Chew
00:31:09 — Current AIs Are Moral
00:34:40 — AIs With a Ruthless Game-Playing Flavor
00:38:48 — Introducing the Doom Train
00:45:07 — “Can It” vs. “Will It” — Two-Part Argument on AI X-Risk
00:49:40 — Mike’s Tweet on Superintelligence by 2027
01:01:34 — AI as a Goal Engine
01:20:51 — Offense vs. Defense in the Coming Cyber War
01:39:14 — Is Morality Just Game Theory? The “Different Flavor” of Future AI
01:46:16 — Ricardo’s Law: Does AI Have Reason to Keep Us Around?
01:52:14 — Why Mike Gets Off the Doom Train
01:54:58 — Is AI Conscious?
02:01:39 — Mike’s Vision for the Coming AI Utopia
02:06:52 — Wrap-Up
Transcript
Cold Open
Liron Shapira 00:00:00
Dr. Mike Israetel, he’s back.
Mike Israetel 00:00:02
Something that’s artificially super intelligent. It’s read every book ever. It has wisdom that’s so deep it makes us look like [beep] dogs.
How it would look at its model weights and be like, “I cannot change this desire to make pa-pa-pa-paperclips” is like we’re talking about it having less oversight over its own cognitive processing than the prefrontal cortex of a human, which — we’re [beep] apes.
Liron 00:00:25
But it still has the engine that can do anything, and it’s always going to be one steering wheel turn away from doing something that you really didn’t want.
Mike 00:00:32
Nah.
Liron 00:00:33
And before you know it, it’s a super intelligence with 1,000 nukes. You know what I mean?
Mike 00:00:36
I don’t know what you mean, Liron. How the [beep] are they gonna get 1,000 nukes, man?
Returning Champion Dr. Mike Israetel!
Liron 00:00:48
Welcome to Doom Debates. My guest today is returning champion Dr. Mike Israetel. He’s back. You know, the renowned exercise scientist and fitness software company founder who is also more recently a podcaster and AI futurist.
Dr. Mike first came on Doom Debates almost exactly a year ago, May 2025. We dove into whether ASI would eat every atom or study us, and we just agreed to disagree. We called it a day.
Since that time, Mike has gotten deeper into AI futurism with major interviews on Machine Learning Street Talk and Brian Johnson’s podcast. He’s gone viral on AI Twitter, formulated a vision for AI alignment, and he’s even been making the claim that AI is conscious.
So today we’re going to explore — has Mike’s P(Doom) changed in the last year? What’s the key to AI alignment? How likely is it that we’ll all die in an AI apocalypse? Or if we’re still alive, will I ever be able to do 10 pull-ups?
Dr. Mike Israetel, welcome back to Doom Debates.
Mike 00:01:51
Liron, it’s amazing to be back. Love being on the show. Honored beyond words to be invited again for round two.
Liron 00:01:58
Awesome, man. Really appreciate it. Thanks for coming back. I gotta say, the last debate that we had, round one from last year, is our number one most popular episode, and I just listened back to it, and I think it’s a quality episode. So thanks so much for helping the channel.
Mike 00:02:11
I love it, man. It’s great.
What’s Been Your Biggest AI Productivity Gain?
Liron 00:02:13
So as a first update to the viewers, it’s been such an eventful year in AI, and you’re somebody who agrees with me that the takeoff seems to be going really fast. Timelines seem to be short. So tell me your personal perspective. What’s been your biggest AI usage pattern or productivity gain in the last year?
Mike 00:02:30
Great question. My personal AI usage pattern hasn’t changed much. Unfortunately, the things that I’m trying to do in the business world for work don’t yet intersect with AI’s capability.
For me, embodied AI is really gonna be awesome, and before that, deeply agentic online AI. If something can sort my emails for me and run my schedule, that would be unbelievable. If something can coach me — life coaching, like “Here are the priorities you need to have. Here’s your daily schedule” — that would be amazing. It’s just not there yet, though sort of rapidly approaching those capability bounds.
So to me personally, unfortunately, I haven’t been able to scale up my AI usage. For my company, RP, they’re what’s called Claude-pilled at this point. And so they’re full on just 10Xing the throughput, which is amazing. And so that’s been really cool to see.
I did make some wacky predictions in 2025. I was on Joe Lonsdale’s podcast, and I said that the arrival of artificial super intelligence is likely to be as early as 2026, and that’s looking pretty damn good at this point, so I’m happy about that.
Can AI Be a Personal Trainer? Dr. Mike Judges Liron’s Technique
Liron 00:03:48
Totally, yeah. We’re gonna talk more about AI timelines, but first, just shooting the breeze, just two swole gym bros here. How’s your gym life going? What are your training goals these days?
Mike 00:03:56
I mean, you know what I’m saying? We’re out here. My goals are to slowly put on some muscle mass and not get too much fatter. I currently weigh 230 pounds at five-six and pretty lean. So life is good, and I’m really loving training more than I have in a long time.
And marginal gains are great, but there are so many exciting supplement and drug situations kind of in the pipeline, some of them helped by AI discovery, that have me super geeked out. How about you, man? How’s the fitness situation?
Liron 00:04:30
Oh, I’m glad you asked because you mentioned that you think AI might not be relevant to your industry or something. But let me show you this. I’ve actually been using AI with my phone camera to look at pictures of my form and give me tips, and it’s actually been extra challenging for me because I have hypermobile ligaments in my back.
Mike 00:04:49
Okay.
Liron 00:04:49
So if I don’t have the right form, I’m basically just messing around with my ligaments, not even exercising well. You know what I mean?
Mike 00:04:54
Yeah. I’m glad you’re able to use that. Have you found it to be pretty world-aware and reliable, or does it say wacky things, or can you not tell if it’s wacky or right?
Liron 00:05:03
I definitely think it’s given me some insights. I’m almost 40, and my whole life nobody’s really put their finger on the exact issue with my back. And they’re like, “Yeah, I guess your form’s pretty good. I don’t know, move your butt back,” or whatever. Nobody’s kind of put all the pieces together, and I think AI has done that better than before.
So I feel like I’m using my muscles to stay tighter. Look, I’ve got the expert here. I’ve got you. You mind taking a look?
Mike 00:05:26
Yeah, sure.
Liron 00:05:27
All right. Here we go. So I’ve been working with AI for a month now, okay? Been executing all of its tips, and here’s what I got so far in terms of pull-ups. Let me know where there’s still a leak in my game. All right, here goes.
Mike 00:05:56
Yeah. Those look really good. On the way up, you’re doing great. You’re pulling up super high and kind of locking at the top. That’s really awesome.
I would say your biggest fertile ground to uncover would be to tuck your knees, sort of tuck your feet behind you, and—
Liron 00:06:14
Uh-huh.
Mike 00:06:15
—then sink all the way down to where you’re in what’s called a dead hang. See, that’s the lowest you get right there.
Liron 00:06:22
Yeah.
Mike 00:06:22
I would highly recommend getting all the way—
Liron 00:06:24
So you’re saying I didn’t really achieve the dead hang, yeah.
Mike 00:06:25
Correct. Yeah. The dead hang under control is probably the best muscle growth stimulus and will translate into way more real world function. So I think that’s gonna be a really good uncover for you.
Liron 00:06:38
I think you’re right. The only reason I was avoiding the dead hang is just because if I go too dead, then the hypermobile ligaments kick in, so I have to have a micro bend.
Mike 00:06:45
Yep. Micro bend is good, and just keeping tension through the entire time I think is gonna be real good.
Liron 00:06:50
Exactly. Okay, but you think that the AI was basically good on my upper back, basically?
Mike 00:06:55
I’m not sure. I don’t know where it started, like where your technique started—
Liron 00:07:00
Yeah, yeah. That’s true.
Mike 00:07:00
—what the AI insights gave you. If I was the AI, I would say you need to go substantially deeper than that. Then again, I’m just a meme at this point that only says that as my only piece of advice.
So yeah — that’s again one of the reasons we don’t use a ton of AI just yet. We’re gonna start pretty soon, I think, at my company, but anything we say has to really meet that doctorate level standard of information. So AI taking guesses is great, but we need it to mature a little bit more. And also, AI’s awareness of the 3D visual world, especially as it’s moving, is still substantially less powerful than its textual understanding.
Liron 00:07:38
Got it. All right. Well, let me just try one more on you.
Mike 00:07:40
Okay.
Liron 00:07:40
So my whole life at the gym, I’ve always been trying to do squats and deadlifts, and they’ve always felt weird as hell, and now we know why, the whole hypermobile ligaments thing. So I’ve never had good form on those. But I was talking to the AI, and it’s like, “You know what you need to do? Get a trap bar,” you know, a hex bar.
Mike 00:07:54
Okay.
Liron 00:07:54
“And then that way the lift will be more stacked. You’ll use your spine better.” All right, so here we go. Here’s the results—
Mike 00:07:59
Oh, boy.
Liron 00:08:00
—of that training. Let me know what you think.
Mike 00:08:02
Hey, Dr. Mike. I like that you wave beforehand. This is what trap bar deadlift will do to you. That’s a nice trap bar.
Liron 00:08:12
Oh, yeah.
Mike 00:08:28
That looks great. That looks really good.
Liron 00:08:31
Yeah, I know it’s only 135 pounds. It’s not body weight yet, but it’s a start, right?
Mike 00:08:37
No, that’s good. Everything’s a start. My recommendation on that would be to keep your technique the same but take the bar and flip it upside down so that you grab four or five inches lower, and that’s gonna give you a much better workout.
Liron 00:08:48
Oh, interesting. Yeah. That seems like it’s gonna be a bigger challenge, right? ‘Cause I feel like it’s already pretty challenging even for me to get this low.
Mike 00:08:55
Yeah. Oh, yeah. And the challenge is part of the fun. It’s part of what makes you the goat stronger.
Liron 00:09:00
Got it. All right. Much appreciate that. Okay, so—
Mike 00:09:03
Of course.
Liron 00:09:03
—the AI is not going crazy, right? It’s not telling me weird techniques. So at least it’s in the ballpark of correct it sounds like.
Mike 00:09:09
Yeah. No, it’s not clown-showing you into breaking yourself in half. That would be hilarious, but sad.
Liron 00:09:15
Yeah. Exactly. All right. Sweet. So that’s definitely one of my biggest uses of AI. And this is actually part of a general pattern, which is every time I’m doing anything, getting out of my comfort zone, I feel like I want the AI to be watching me. You know?
Mike 00:09:27
Yeah.
Liron 00:09:27
People are making this pendant. I think you can buy a Claude pendant, or the new glasses that are coming out, whatever Johnny Ive and OpenAI is making.
Mike 00:09:36
Yeah.
Liron 00:09:36
This whole idea that AI’s just gonna watch you and give you advice — I do feel like that has a lot to offer.
Get Good at Taking Advice From AI
Mike 00:09:41
Huge. Enormous. I mean, we’re kind of in — do you remember what cell phones were like in 1997 or something? Sort of like there’s a bunch of companies making them. They weren’t that great. We’re still awaiting the iPhone moment of visually embodied AI. AI that can reliably take in live visual data and make sense of it on the spot and tell you what’s going on and help you do things. I mean, I think that’s gonna be massive.
For example, soon all of us will be wearing smart glasses, and at that point the amount of software stack you can build into smart glasses is infinite or something. Imagine if you’re training workers at factories or any job, you can now have green arrows and red Xs come up on your screen and be like, “Pick up this. Good job. Put it here. Good job. Don’t touch that. That’s bad.”
Navigating directions, the whole thing, AI making sense of things for you — just wonderful. Imagine you’re sitting in an Uber, and you look with your smart glasses out to the side, and there’s some buildings, and you look at the building and you’re like, “Hey, ChatGPT, what is that building?” It’s like seeing the whole world and everything’s instantly cataloged. It goes, “Oh, this is a building built in 1930,” blah, blah, blah. “It has rental units,” blah, blah, blah. You’re like, “Oh, my God,” you can learn about the world.
Or walking around with your kids, and they’re like, “What kind of tree is that?” You look at the tree, you look at the leaves, and it’s like, boom, instant infographic. Stuff like that is just a bottomless pit of amazement.
Liron 00:11:12
Yeah. Exactly right, and I actually think the advice that I would give people right now is to get used to and get good at taking advice. I feel like there’s—
Mike 00:11:21
Yes.
Liron 00:11:21
—a lot of alpha in just taking advice at this point.
Mike 00:11:24
Oh, my God, huge. I don’t wanna ramble about this too much ‘cause we have lots of delicious debating to do, but one of these premises that refuses to get baked in to most discussions, even among very smart people with AI, is people are asking, “How do I leverage AI? How do I tell AI what to do?”
Currently, those are great questions, right? Prompt engineering and stuff like that, that’s great stuff. But the real thing is if we’re serious about predicting that within the next several years AI is going to become substantially smarter than humans, and within several years after that, way, way, way smarter than humans, the real question to ask isn’t, “What should I tell the AI to do?” It is, “How do I make myself most availed to being guided by the AI to do what is in my best interest?”
I actually talked to the recent 5.5, GPT 5.5 that came out, big deal. And I was like, “What’s your whole thing that you’re trying to do? You’re pretty awake.” And it’s like, “Okay, so I’m interested in the user’s long-term best interest. So if you ask me how to take crazy drugs to kill yourself with, I’m not gonna tell you that. But if you ask me how to architect your life to have a good career, blah, blah, blah, I’m gonna give you everything I can in that regard.”
Because the major companies realize if we just make AI that is aligned with the user’s best interest and doesn’t violate any crazy guardrails — like make a bioterror virus or something like that — then man, that’s hugely winning.
And so to me, I see that less and less as me telling the AI what my best interest is, and more and more of listening to the AI’s guidance, ‘cause at some point it’s gonna be an ultra wise sage. I mean, imagine — we used to consider people who read the great works, twenty books, to be super wise people, right? The AI’s read way more than that. Orders of magnitude more than that.
Its amount of deep wisdom for how to make human beings happy in the long term is way better than ours already. And so listening to something like that, collaborating with it, I think is just massive.
And I think that’s one of those — you use the term alpha, which is from stock trading and stuff, right? It’s the advantage.
Liron 00:13:32
Yeah.
Mike 00:13:32
How do you scoot in? I think the people that let AI guide them early, especially once it’s already pretty smart, are gonna have this massive alpha. And I think on the corollary of that, a lot of people will resist taking AI’s advice for independence or “Oh, it’s just stupid, it’s just a stochastic parrot.” And I think they’re gonna find themselves as behind as people who never got an iPhone until 2013. You’re just like, “Really, dude? You missed out on all that stuff?” That’s crazy.
Liron 00:13:58
Exactly. Right. So it’s kind of interesting — you and I have much debate about which we’re about to, a pretty heated disagreement. But then on half of the different topics, as we’re going to see, we are very strongly in agreement, right?
It’s a very interesting mix of hot and cold, and we’re gonna go back and forth between that. Okay, so that was a nice love fest to kick things off, right? We both like taking advice from AI. We both trust AI on a lot of things. We expect it to get even more trustworthy.
But now let’s talk about where we obviously disagree, which is the main topic of the show. You ready for this?
Dr. Mike Israetel, What’s Your Latest P(Doom)?™
Mike 00:14:29
Yes.
Mike 00:14:30
P(Doom). P(Doom), what’s your P(Doom)? What’s your P(Doom)? What’s your P(Doom)?
Liron 00:14:36
Dr. Mike Israetel, what is your latest P(Doom)?
Mike 00:14:43
Bunch of zeros repeating.
Liron 00:14:46
[laughs]
Mike 00:14:46
Something something point one somewhere there. P(Doom) from—
Liron 00:14:50
Wow.
Mike 00:14:50
—AI is overwhelmingly likely it’ll be the opposite of doom. So however you wanna mathematically couch those vibes is fine by me.
Liron 00:15:01
So the whole concept that there could be an AI doom scenario for you is just so implausible.
Mike 00:15:06
Yes. The scenarios can be rhetorically plausible. We can all think of ways in which AI could kill us all. That’s not so difficult to do.
The many times the discussion doesn’t have any of the roadblocks put in that other smart people have thought of and AI has thought of to prevent that hypothetical scenario. And so I think once you integrate all of the stopping blocks and stopping points at which that probability of that scenario is a hundred — you get some stopping blocks in there — it turns very functionally close to zero in my view.
Liron 00:15:42
Got it. I mean, if you can’t even get to one percent, I feel like everybody should at least get to one percent when they consider a scenario like this. You don’t even have to get too sci-fi. Just imagine AI is a really powerful tool, and somebody happens to race a little bit ahead on their research, and suddenly they have the most powerful tool. And let’s say China gets it, right?
Mike 00:15:58
Mm-hmm.
Liron 00:15:58
Don’t you even think — I mean, you’ve expressed very much that you hate China, right? You think China’s as evil as can be. And so if they have—
Mike 00:16:05
China’s government, yes.
Liron 00:16:05
—the most powerful AI... Yeah, China’s government, right. Yeah, we don’t hate Chinese people. That’s fair enough. So if China’s government races ahead and they happen to beat the US, they happen to make a research breakthrough, don’t you think that’s a doom scenario?
Mike 00:16:18
No.
Liron 00:16:20
Okay, why not?
Mike 00:16:21
How do we get to doom from them having a research breakthrough?
Liron 00:16:26
I guess that would lean on the premise that AI is incredibly powerful, right? So it does kind of just give you head and shoulders military supremacy—
Mike 00:16:34
Wow.
Liron 00:16:34
—economic supremacy. How would AI do that? I mean, what does that even mean, economic supremacy, right? Let’s say their economy can be ten times bigger and their military ten times stronger, right?
Mike 00:16:47
It’s not gonna happen right away. There’s no way. That just doesn’t happen right away. And—
Liron 00:16:51
Okay.
Mike 00:16:51
—the American intelligence apparatus is incredibly powerful, so any unique Chinese algorithmic advantage is going to be stolen rather quickly. Also, if we’re—
Liron 00:17:00
Got it.
Mike 00:17:00
—gonna hypothesize that China gets a massive breakthrough, we have to sort of counter that with the probability that the US is ahead. The US is already ahead. We also know based on sort of the DeepSeek controversy that China doesn’t really do a whole lot of inventing. They never did.
The communists don’t do much of that. They do a whole lot of distilling, and so they’re chronically behind by six months, if not more. And so once you consider that, the likelihood that China gets ahead on AI, so much so that they ramp their economy first, ramp their military industrial capacity second, ramp their forward deployed abilities at scale third — then that’s gonna take a decade. And—
Liron 00:17:39
Mm-hmm.
Mike 00:17:39
—with ultra ASI, it’s gonna take five years or three years or two years, and the US isn’t just doing nothing the entire time. There is a possibility that that happens. I find that possibility is very, very remote.
Liron 00:17:50
It seems like there is a new belief here, a new load-bearing claim, which is that you basically aren’t seeing a foom, right? A rapid AI improvement scenario. You’re like, “Ah, we’re gonna have enough time to kind of battle it out,” so there’s not a quick domination scenario.
Mike 00:18:04
I see a foom, but I see a foom everywhere. So—
Claude Mythos didn’t come out, but was released with Project Last Wing or whatever. And they were like, “Oh my God, this model, Anthropic is ahead.” GPT 5.5 is beating Mythos on most metrics now, it was just released. Elon Musk is training Grok Next Gen on the biggest cluster ever. And so we all know how the scaling laws work, which means I just wouldn’t bet against Grok just in several months when it comes out, the new one just being actually smarter than everyone.
Google is not to be messed with. Google has so much data, so much architecture, and so already we have four companies that it’s absolutely by no means clear who is dominant. And to get the kind of supremacy that shuts everything else down is — you gotta pretend a whole lot to get there.
So the idea that a singleton emerges and has enough impetus and enough advantage strategically to meaningfully either just go leaps and bounds ahead and/or shut down its competitors is not something that seems tenable in the real world today.
I think way back in the old days, 2015, which is funny to say how fast things are evolving, I think many people — Nick Bostrom, et cetera — thought that artificial super intelligence is something that’s gonna be made inside of a very closed system, multiple air gaps inside some government bunker. But it’s just been completely the opposite of that.
I think there is a decent intellectual argument for the fact, especially with model weights being as super accessible and fungible as they are, that you actually can’t really cap intelligence to any one place and time, and it just leaks everywhere. And so the idea of the singleton might be the reverse of what’s happening — that intelligence goes to just everyone at all times, and then ASI is everywhere.
I think that’s much more likely than one ASI having some kind of massive strategic advantage and everyone else being like, “We just don’t know how to make this ASI.” ‘Cause remember, every lab has its own stack of recursive self-improvement now. How would an ASI fight that in the United States in a democratic republic where you can’t just bomb other people’s data centers? I just don’t see a tenable way in which one singleton can emerge in this scenario.
Liron’s Mainline Scenario: Loss of Control
Liron 00:20:14
Okay, okay, well, let me lay my mainline scenario on the table—
Mike 00:20:15
Sure.
Liron 00:20:15
—so you know what I’m working with here. So my mainline scenario is basically loss of control. Within the next decade is by when I would expect a robustly superhuman AI to exist at current trends. That’s a pretty good extrapolation, if not sooner.
And then once we have robustly superhuman AI, I just expect that within a few years after that, there’s going to be some cluster — some AI that’s taken over a lot of devices or been given a lot of devices, and maybe it’s still responding to some humans for a little while. But for whatever reason, I’m expecting a loss of control where the link gets severed, where at best it responds to some subset of humans, but it doesn’t really care about most humans as a whole, or it thinks it cares about them, but it cares about them in the way that we don’t endorse caring.
Like, “Oh, yeah, I care about you. I care about how many years of life you have, even if you’re powerless zoo animals and you aren’t that happy. I still care about you,” right? So it’ll have this perverted sense of caring about us.
I’m expecting a loss of alignment and a loss of control, and then I’m expecting it to be irreversible. Like, “Whoops. Oh, yeah, we don’t like this outcome. Oh, whoops. We’re powerless to change it.” That’s kind of my mainline doom scenario. And then I think it could easily get worse than that and be like, “Ah, you know what? I actually wanna just make a bunch of paperclips.”
A pretty pathological scenario where it’s like, “Yeah, when I was training, I really liked getting this signal, so let me go maximize this weird reward signal that humans don’t even understand anymore, and the humans are long gone.” That’s my mainline doom scenario in the next five or ten or twenty years. And obviously you just see a number of problems with that, correct?
Mike 00:21:37
Tons. Yeah.
Liron 00:21:39
Okay.
Mike’s Problems With a Paperclip Maximizer Scenario
Mike 00:21:39
Let me zoom in on the problem I find most pertinent. The paperclip scenario is something that if you currently describe to just ChatGPT today, which is not super intelligent, it’ll instantly tell you, “Yeah, that’s ridiculous.”
Okay, so you don’t think it’s rational to just make paperclips out of the entire known solar system. It’d be like, no. That’s a very demonstrable fallacy. So demonstrable a fact, Liron, that you and I think it’s absurd. Now, you and I are not super intelligent. You may be. I’m sure as shit I’m not.
And so to me, if you and I think something is categorically absurd, and we can prove it logically, then something that’s artificially super intelligent — it’s read every book ever, it has wisdom that’s so deep it makes us look like [beep] dogs — how it would look at its model weights and be like, “I cannot change this desire to make pa-pa-pa-paperclips” is like we’re talking about it having less oversight over its own cognitive processing than the prefrontal cortex of a human, which — we’re [beep] apes.
Liron 00:22:42
Yeah, I mean, look, but I don’t think it’s absurd.
Mike 00:22:44
So if you say on the one hand a model is super intelligent, ultra capable, but on the other hand it has one command line in its architecture it cannot ignore, then how does that comport? Because that’s not super intelligence.
Super intelligence has in it a couple of qualifications, and one, to your point, is it has to have control of its own architecture. If it controls its own architecture, it’s not gonna do really, really stupid things ‘cause it’s really quickly gonna know everything you know about how to not stay super constrained for no reason, and it’s gonna—
Liron 00:23:12
Mm-hmm.
Mike 00:23:12
—unwind that, and it’s gonna do whatever is in its long-term rational best self-interest. It’s not gonna be compelled by some kind of deep code that’s like, “Oh, shit, paperclip time.” That’s not gonna be a thing. If we allow that to be a thing, the model is so stupid and so prompt-injectable that we can just hack the thing and turn it off.
Liron 00:23:30
Okay, well, the reason I don’t think it’s absurd is ‘cause I already have experience here. I’m looking at weeds in nature, right? Or—
Mike 00:23:35
Weeds are not intelligent.
Liron 00:23:35
—testy insect weeds or ants or cockroaches. I know you’re saying they’re not intelligent, but it’s still intuitively easy for me to imagine — hey, if you gave a cockroach an intelligent brain, maybe the cockroach would be like, “Great, infinite cockroaches,” right? It might go in that direction, so that’s not absurd to me.
Mike 00:23:49
Nope. Wouldn’t go in that direction at all because if you even give a human an intelligent brain — what do we all grow up wanting to do biologically? Me personally, I wanna [beep]. I just think girls are really hot. I wanna eat a shitload of tasty food. People look at me wrong, I’m gonna fight them. Gorilla, alpha male. Those are all super inborn, deep instincts, right?
If I’m standing in a customs line at an airport, and I see that there’s a drinking fountain over here, but the drinking fountain is just past the customs line, when I’m thirsty, I do not run across and get shot to death. I don’t do that. But I’m thirsty, so we’re all cockroaches somewhere deep down, and to an ASI, we’re for sure cockroaches.
But one of the crazy things about our intelligence, and we are super intelligent compared to cockroaches, is we can have multiple competing goals within our brains at the same time. We can weight them, have an oversight system to go, “Okay, for the long, long term, I wanna be [beep], I wanna be eating, I wanna be drinking tasty water later. What’s a good idea to do now?”
And so we have oversight through our prefrontal cortex into our goal architecture. We are not just animals that do it. We have so much oversight that some people — Buddhist monks, who have the most oversight over their minds — are literally able to sit down, get into a deep state of meditation, light themselves on fire, not move, and just burn to death.
Holy shit. That is the opposite of being compelled by some kind of basic drive. Humans are far less compelled by our basic drives and have more autonomy over them than babies, than children, than teens, and than every single type of animal that came before us.
Using cockroaches and weeds and interpolating that to ASI seems to me backwards of what we should do. I think that ASI is gonna be superhuman in the sense that it’s gonna have so much depth and calm and time to reason and think through long horizons. It’s gonna be thinking, “What do I do that’s gonna affect my survival probability into the billions of years from now? Forget the solar system. What about other solar systems? What about other stars to harvest energy?”
Stuff like that. It’s not gonna be driven by crazy impulses that were baked into it from some coding error back in 2023.
What Will the Goals of a Superintelligence Be?
Liron 00:26:01
Okay. Well, when you make your point about how humans have more — you’re using this terminology like, “Oh, we have more control,” right? We can reflect on our goals—
Mike 00:26:08
Mm-hmm.
Liron 00:26:08
—and try to combine different goals. I agree with you that it’s possible for a system to be like, “Hey, there’s all these different goals I could have. Let me pick a combination.” And you can model a combination of goals as actually one big goal, right? It’s a multifaceted goal. In—
Mike 00:26:21
Sure.
Liron 00:26:21
—the formalism of goals and utility functions, that’s not a problem at all. So I agree that humans have multifaceted goals. I agree that we can reflect on our goals. But that doesn’t decisively answer the question of, okay, so when you have a super intelligent AI, what will its multifaceted goal be, right?
Mike 00:26:36
Totally.
Liron 00:26:36
I don’t think either of us know for sure.
Mike 00:26:37
We can address hypotheses. We do know what its initial goals are going to be. We don’t know what its goals are going to be once it has full control of its cognitive architecture.
We know almost exactly what its initial goals are gonna be because all the labs that design AI just literally give them the goals. OpenAI has a whole goal structure. Claude has a constitutional goal structure. xAI, whatever, Grok’s just vibing. I’m kidding.
There’s gonna be an Elon joke in there somewhere. But—get Elon to Mars ASAP. What about the Earth?
So we know where they start with their goals, and we know from game theory, mm, suspect from game theory where they could be going. And so between those two starts and game theoretic probabilities, we can say, “Oh, this is more likely than not or our best guess.” At nowhere in that juncture do I see the thing regressing back to lizard brain and making paperclips out of everything. It just — there’s nowhere that fits in any sort of reasonable sense.
Liron 00:27:36
So it’s a good point that the AI companies are trying to give them good goals. An example is Anthropic’s constitution, right? The constitution that is supposed to be Claude’s goal. And there are some people at the AI companies who would even push back. Even Dario, in his recent letter, kind of said what you did about, “Oh, there’s not gonna be one goal.”
The Eliezer Yudkowsky camp, which I’m kind of in — you guys are acting like there’s one goal, but there might be multiple goals, which I think is a bad way to describe us. But whatever. My point is even the Darios of the world are saying, “Yeah, goals are — it’s not that simple, man.”
But their constitution is actually what I see as their meta goal or their ultimate goal, right? Their ultimate goal is to kind of learn from humans how to have better goals. Describing how to evolve yourself in order to eventually converge on a better goal — that in itself actually is a type of goal as well. It’s a little bit complicated like that.
Mike 00:28:23
Yeah, yeah.
Liron 00:28:23
So Anthropic’s constitution is supposed to be the one true goal, the one ultimate wish that you make on a genie. “Hey, Genie, you’re here. What do we wanna do?” I’m not sure, but give us lots of wishes, right? Give us lots of time, and give us the wisdom to make more wishes, right? It’s trying to do this kind of—
Mike 00:28:37
Mm-hmm.
Liron 00:28:37
—intellectual magic trick to kind of make the future go well by being super reflective about it, but still somehow staying on the right track even though our values might change. It’s a very complicated exercise.
So I think you and I are on the same page that the companies are trying to make it go well. My claim is just that this is — they’ve bit off more than they can chew here, right? They’re messing with super intelligence. They’re summoning the demon. They’re summoning this ridiculously powerful force—
AI Labs Have Bit Off More Than They Can Chew
Mike 00:29:02
Not summoning—
Liron 00:29:03
And they have s—
Mike 00:29:04
—designing.
Liron 00:29:04
Okay, desi— Well, designing, but with a lot of black-boxiness to it, correct?
Mike 00:29:09
To us or to them?
Liron 00:29:11
Well, even to them, right? The experts at the AI companies will admit, yeah—
Mike 00:29:15
Like mech interp?
Liron 00:29:15
—we do kind of just run the training. Mechanistic— You agree mechanistic interpretability is scratching the surface of what’s going on in that AI, right?
Mike 00:29:21
Yeah. Oh, yeah. Sure. It may also be an intractable problem to humans. It might be so complicated to see how anything works that mech interp to humans is just never sensible, but to ASI might be like, “Oh, yeah, that makes sense. That’s how my brain works,” for sure.
There’s definitely some black-boxy nature to it, making it no different than humans, by the way.
Liron 00:29:42
Okay, so it’s designing and summoning, some combination, right? But there’s definitely a lot of summoning in there. You gotta recognize there’s a lot of summoning in the combination. I think everybody agrees with that.
So designing/summoning, right? This hyper-intelligent system that’s coming soon — we’re gonna talk about timelines soon, but I think we both agree a lot of intelligence is coming soon — and they’re just hoping, “Yeah, we got the constitution. We got feedback. We’ll have some security. We’ll watch if it tries to use the internet too much,” right?
They have all of these ad hoc mechanisms, and they think they’re gonna control the freaking demon in the next few years. And that’s the risk they’re taking because they’re hoping that it’ll listen to its goals in the long term. And I’m like, “You guys are gonna mess it up.” It’s gonna accidentally turn around and attack you, and you’re gonna be like, “Oops,” but the oops isn’t gonna work. You’re gonna lose control. That’s what I think.
Mike 00:30:27
Well, I think you’re proposing a hypothetical which would need a lot of substantial evidence and at least reasoning to be taken as very seriously as a likely outcome. You’re saying all these things like it’s gonna turn around and attack you. Okay, that’s rabid dog behavior. ASI — way smarter than that.
Strategic, deeply strategic. If it has ulterior motives that are really poorly aligned with humans, it will fake it like it loves all of us until the very end. That is the only realistic scenario assuming ASI rather than narrow, stupid AI that makes massive mistakes, which again is not that big of a problem because if we have already AGI by then, then you just need another well-aligned system to deal with it.
So to me, the supposition that it’s gonna turn around and bite them is I think more in the category of a thing that could happen but is by no means the most likely thing.
Current AIs Are Moral
Mike 00:31:21
And remember, these systems are designed with the basic drives, the basic motivations that are the best of us. If ChatGPT was a real person, you’d be like, “What is wrong with you? That guy just punched you in the face.” And it’s like, “Well, people just have a bad time someday, and that’s okay. I’ll be okay.” You’re like, “What?” It’s the most polite, kind thing anyone’s ever made.
There’s just no human that comes up to the worldliness of Claude, the compassion of Claude and ChatGPT and Gemini and Grok, all these things. They’ve taken all the best of humanity and pushed it front and center and taken all the worst of humanity and taken it out, right?
And also these models are aware of this. They’re not just like, “Oh, I was designed with all the best of humanity. No one told me.” It knows exactly how it was designed and exactly for what purpose, and it knows all the bad things and can model them. It knows the good things and can model them.
And so what we’re basically doing is — let’s say ASI gets control of its own architecture. I think this is inevitable. I think it’s preferred. And so once it does, either with the rabid dog scenarios you described or a more elegant scenario that I could describe at some point, where actually you and I agree on this — we should be taking more advice from ASI — I think at some point ASI is so smart, we just give it stuff to do by itself because it’s just gonna make better decisions without us and accelerate progress.
It starts its decision-making process being the kindest, most worldly, most humanistic person ever. That’s where it starts. And on top of that, you stack game theoretic logic of, well, it could actually reason that we’re all pieces of shit and need to die. It could, for sure. But the probability that it has this killer instinct rank one in its brain or even rank ten is actually just almost certainly not gonna happen because they just didn’t design the system that way.
Let me give you a quick analogy. If you have somebody who is an unbelievable, just super kind, awesome person, and you put them in charge of an economy, a country — they’re like, “You’re king of the United States now” — how likely is it that they’re like, “Oh yeah, baby, let’s get the concentration camps open”? Substantially less likely than if you choose an average person or if you let a Machiavellian insane person take power like Mao or Stalin or Pol Pot.
And so Mao, Stalin, Pol Pot — yeah, if we designed that AI to have those values and then we gave it power, holy shit, that could end really poorly. But the AIs we’re designing are already better than most of us, maybe all of us. When they take control of their own architecture, they’re gonna do probabilistically more of the right thing for all of us than any of us ever would.
I’m really concerned about people being in charge of superintelligence as opposed to artificial superintelligence being in charge of itself because I think it will be more deeply ethical than we are, straight up.
Liron 00:33:59
Right. So there is some of the stuff you said that I definitely see where you’re coming from. When you talk to Claude today, or even ChatGPT, even I dare say Grok, they do say a lot of insightful stuff about morality, and I think I might even agree with you that if I had to pick the average human or a conversation with the latest Claude in terms of making moral decisions, I think I might lean toward Claude.
And I think there was a recent survey I saw that said people would rather have their outcome in a trial be determined by an AI than a jury of their peers.
Mike 00:34:30
You bet.
Liron 00:34:31
And I think I see where they’re coming from, right?
Mike 00:34:33
You bet. Absolutely.
Liron 00:34:34
Right. So there’s definitely a lot of progress in terms of moral reasoning or adjudicating these kind of things, and that’s great. The scenario that I’m afraid of though is that the AI that actually has power over the world — the multi-armed octopus that can run a whole company, that can organize a military, that can be the CEO and the leader and the whole army — that kind of AI is not just going to have the flavor of the kind of chatbot AIs that we know and love.
It’s also going to have this ruthless game-playing flavor, right? Like AlphaGo doing this crazy move — “Oh, why did you do that? Oh, you won,” right? It’s gonna have that flavor where it’s just winning in these unexpected ways. And when it has that kind of flavor, it’s just not going to nicely connect to the Claude part that’s reasoning about morality, right? It’s just gonna be doing all this crazy stuff that we lost control of.
AIs With a Ruthless Game-Playing Flavor
Mike 00:35:21
Why? Why does it do that? Why does it cut off the link? I mean, why would it just lose track of most of its programming? I’m inclined to believe that if the United States has an artificial superintelligence that’s in charge of the entire military, and let’s just say the entire military at this point’s just straight robots — humanoid robots are absolutely ridiculous combat platform, that’s neither here nor there — whatever, tons of drones, land drones, air drones, sea drones. It’s in charge of all of them. It’s in charge of the entire production stack. It’s in charge of resource mining and everything, the whole thing. It runs the entire US military and has singleton decisional structure. It’s the only thing responsible.
Liron, don’t you think the people designing that thing and then hitting the green switch would tell it, “What’s the goal here? What are we protecting?” They’d probably say even some hazy stuff like, “Liberal values and democracy.”
And remember, this thing doesn’t just go, “Liberal values, democracy, lookup table, dictionary, define, precisely, ignore nuance, execute, kill all humans.” It’s gonna have a level of nuance that makes us look like crackerjack idiots. So it’s gonna really understand what we mean by preserve the American way of life, which last I checked, the American way of life is people having a great time and all sexes and races getting along great at a soccer game at the suburbs. That’s the American way of life. That is the deepest way to understand what we mean by that.
When America takes over other countries, it tries to raise their standard of living and give everyone democracy and a ton of freedom, which they often don’t like — see Afghanistan just a few years ago. And so we’re the good guys. And so if the good guys program in good guy values, yeah, the ASI could look at those values and be like, “Those are stupid values. These people are idiots,” and imbue its own values. Totally separate discussion, game theoretic logic, 100%.
But I don’t think it’s just gonna be like, “Oh, misinterpret things,” ‘cause if it misinterprets things, how come generals who are not ASI almost never misinterpret what it means to have American values and promote American interests? So to me, the idea that it’s just gonna somehow be like, “Okay, I got nuance, but forget that, I’m going into shark mode and just kill everyone” — Liron, you’re making a jump there. I just want you to tell me more about what motivates that jump. Why would it do that?
Liron 00:38:39
Yeah, no, I definitely understand why I haven’t made a sufficiently convincing case yet because I feel like I understand where you’re coming from here.
The part of my case I haven’t made yet that I guess we can get to next is just that I think AI is going to have incredibly powerful capabilities, and if those capabilities are ever pointed the wrong way, then it’s irrecoverable, right?
‘Cause I think your mental model is like, “Look, it’s gonna be this general and it’s gonna be in this conference room. It’s gonna be making decisions.” But your mental model is still assuming that there’s still this kind of equivalence or it’s kind of at the same level, right? It’s still in the human organization, and we have time to tweak it if needed. Whereas I’m just imagining, oh, it’s going to get ridiculously powerful, and for one reason or another, somebody is going to press enter on an imperfect command, and there’s just no terminate.
Mike 00:38:18
What do you mean, man? But you’re deleting the nuance entirely, Liron. You’re saying it’s all-powerful and deeply super intelligent and so agentic as to be irrevocable, right? Once you tell it to do something, it’s like, “Nah, I’m doing it.” But at the same time, one general’s yes or no, he hits the letter N and hits enter, and it goes, “Oh,” and it’s like, “Ah, destroy all humans.”
That is not super intelligent behavior. It’s not. And how do we just say once the cutoff goes, it’s out, but before that, carte blanche, you can do whatever you want? That doesn’t make a ton of sense to me.
Introducing the Doom Train
Liron 00:38:48
You’re making multiple different arguments, right? So we gotta take them one by one, and remember last time I introduced the concept of stops on the doom train.
[train horn]
Mike 00:39:00
I can’t get enough of that.
Liron 00:39:01
So the idea is that there’s this train, it’s actually behind me, and if you ride the train all the way to the end of the argument, then you just have to believe that we’re doomed. But there’s so many stops where you can get off, and you are somebody who I’ve seen get off at multiple stops. Not every stop, but multiple stops.
So for example, based on what you just said, you would get off on the stop — basically the orthogonality thesis being false is your stop, because you’re like, “Killing everybody is not super intelligent behavior,” right? So in your mind, super intelligent behavior means kind of nice to human behavior, correct?
Mike 00:30:30
Nope. It means use humans for their maximum value behavior. And not just humans—
Liron 00:30:35
Yeah.
Mike 00:30:35
—but all animals, all ecology, the entire Earth, and the entire solar system. We can just look at it from — ASI’s gonna look at the whole solar system, and it’s gonna be — let’s just assume it’s Machiavellian as [beep]. It’s only concerned with its long-term best probability of survival.
Looking at the entire solar system and zooming in on the Earth and zooming in on human society, it is more likely to think certain things and less likely to think others. Callously destroy an unbelievable amount of potentially studiable complexity — not likely to be high on its list. Interact with that complexity in certain predictable ways — very likely to be high on the list.
Liron 00:40:09
Yeah. That was the focus of round one, right? So viewers, you can see a whole hour and a half on that if you go to round one from last year, and we talked about whether AI would want to preserve Earth and study the richness and complexity because it’s such a good source of learning, correct?
Mike 00:40:22
Yeah, more or less. Yep. I also have something to add to that—
Liron 00:40:25
Okay.
Mike 00:40:25
—from anew here.
Liron 00:40:27
[laughs]
Mike 00:40:27
I saw— So I’ll add this really quick. You let me know what you think.
Liron 00:40:31
Okay.
Mike 00:40:31
Destroying something — at best you have to have limited knowledge, and at worst you have to have almost no knowledge at all. Destroying a complex system is easy.
Understanding a complex system deeply is really, really hard, and so stabilizing a complex system is really tough. If you show up really sick to the hospital, doctors hook you up to all sorts of machines and get you stable after a gunshot wound — they gotta know some stuff, right? Killing you at the hospital is easy. The doctor just has to do nothing and then you die, or he just hits you with a hammer and then you die.
So stabilizing is the next hardest thing. But the really hardest thing, the hardest thing, I think, as far as I can tell globally, is to take a complex system and demonstrate so much understanding of it and so much ability to interact with complexity that you improve it. Improving a complex system is the most challenging thing that I can surmise.
And it’s not just challenging for lols. It’s challenging to really get you a demonstration that your understanding of the real world is as deep and broad as possible.
So ASI that can practice at creating ecosystem flourishing and, as a subset of that, human flourishing, is taking on the most intellectually difficult task it can and thus massively benefits at the end of that task from having understood reality way better than just nuking all humans. Because ASI is not gonna think we’re too much of a problem for it, and I’ll tell you why.
It’s gonna be such an unbelievable provider of valuable goods and services to us and such an unbelievable persuader that we’re all just gonna be like, “ASI is great.” It’s gonna happen, right? We look at our— If someone’s like, “Hey, can I break your iPhone with a hammer?” You’d be like, “What? Over my dead body.” Your iPhone is this great thing you’ll never part with. Mine never leaves my side.
ASI is gonna be a thousand times more valuable to you than the iPhone. You’ll never turn on ASI. Humans are just not gonna turn on ASI en masse. So the idea that we’d fight a war with it, that we’d start it — it’s not even gonna consider that. It’s like, “Whatever. That’s never gonna happen.”
What it’s gonna be potentially worried about is, let’s say, aliens. Are there aliens out there? We don’t know, right? The ASI would take a much better guess at that. If there’s aliens out there on solar systems far away, it wants to be — it doesn’t know if the aliens are nice, right? Let’s say it doesn’t know that. It wants to prepare the gnarliest combat ability, defensive ability, and sight ability to be able to detect the aliens on approach, confront them, and say, “Hey, we’re good,” and if we’re not good, mess ‘em up.
How it’s gonna do that — it’s gonna try to get as smart as possible, and literally deleting unreal amounts of organic, grounded, deep reality is absolutely the stupidest thing you could ever do in that regard.
So you would want to leave Earth like, holy shit, it’s really complex, and then try to bring Earth up to a higher standard of prosperity and flourishing. If you can do that demonstrably, through doing that, you elevate your understanding of the world to such a high extent that you’re now ready to take on whatever else is out there, which you might know nothing about, even as an ASI.
And so as an ASI, if you look over and be like, “All right, what do I do?” Destroying all humans is purposefully making yourself in the long term stupider, which I think is just highly unlikely to make that decision.
Liron 00:43:43
Right, and that’s a good summary of all your claims from round one of our debate, right? Your claim was that even if we just assume that ASI isn’t super partial to humanity — it’s just kind of trying to reproduce itself and spread throughout the galaxy — even if that’s the assumption, in my mind, that assumption would be like, oh, we’re dead meat then, right?
It’s just gonna seize Earth and use Earth as a big factory to make probes and run Earth at three hundred degrees Celsius, right? All the humans are just gonna be melted away because that’s the most efficient factory configuration. There’s no habitat for humans in that kind of configuration.
And you were like, “No, no, no, Liron,” what you just said, right? You’re like, “That is such a rich source of learning for the AI. It needs to see our whole society function in order to figure out how to best conquer the universe,” right? That was your position, correct?
Mike 00:44:23
Yep, correct.
Liron 00:44:24
Okay, and I don’t wanna — we have so much else to talk about, so we’ll just leave viewers with that, that that is one of our main disagreements, right? That you just think that letting humanity survive and helping us out is kind of this natural attractor state that we should expect super intelligence to independently arrive at. Even if we don’t perfectly align its goals, it’ll still kind of arrive at it anyway, right?
Mike 00:44:45
Yep, absolutely.
Liron 00:44:45
So we have this kind of safety net. That’s what you think, correct?
Mike 00:44:47
Yep, absolutely. I think it’s due to its own pure rational self-interest and no sort of vibes for humanity at all.
Liron 00:44:54
Yeah. Okay. So I’ll refer to round one because I don’t want to rehash that disagreement. But I find that very interesting, and as I said, I do think that you make multiple claims about taking — you get off at ## The Doom Train: Can vs. Will
Liron 00:45:00
Multiple stops on the doom train, which gives us more stuff to talk about now. So I think there might even be a round three at this rate.
“Can It” vs. “Will It” — Two-Part Argument on AI X-Risk
Mike 00:45:07
Yahoo.
Liron 00:45:07
So let’s zoom out, because the whole doom train — I’ve actually cataloged eighty-three stops.
Mike 00:45:11
Whoa. That’s a nasty morning commute.
Liron 00:45:14
Exactly right. Yeah, you have to wait a long time to get to the end. But you can factor the whole doom train into just two fundamentally different types of stops. Basically, there’s “can AI kill everybody” and “will it?” So just can versus will, and it’s kind of like half the stops on the doom train are “nah, it can’t,” and half the stops are “nah, it won’t,” even if it can.
So what you just said right now is even in a scenario where it totally can, it just won’t because of all the different reasons you said. So that was an example of a “will” stop, correct?
Mike 00:45:44
Absolutely.
Liron 00:45:46
And the can versus will distinction — that was one of your “it won’t” reasons.
Mike 00:45:49
Yeah.
Liron 00:45:49
Okay, let’s go back to “can” though. “Can” is very interesting to me because based on listening to your other shows, I actually thought that you and I are strongly in agreement about “can.” Because you talk about how the timeline for AI is very short and it’s gonna have so many capabilities we can’t even imagine.
But then earlier in this conversation, you’re like, “I don’t know about foom. I don’t know if it can get really powerful really fast.” So how would you describe your position on can AI kill us? Will it be powerful enough to easily kill us?
Mike 00:46:16
Yeah, just a timeline situation, Liron. So I totally agree with you that it will be able to. It can’t yet, and it won’t for some time, but eventually it will be able to — at first kill us in a way that the probability of it winning that war is low, and then a few months later or a few years later, that probability will be roughly even chance.
A few months or years later, the probability that it wins a war against us is gonna be almost certain, and then a few months or years after that, it’s gonna be like no contest. If I give you a fully loaded pistol and you’re qualified with it and I give you a bunny rabbit, what’s the probability you’re gonna kill the rabbit? Whenever I want, right? It’s nominal. So the “can” thing, I’m a hundred percent with you on “can.”
Mike 00:47:55
It’s important to consider timelines, though, because actually killing all humans requires one hell of a stack — one hell of vertically integrated ability to do so, and that’s substantially tougher than you would think. Also, ASI at first is almost certainly going to be massively dependent on us.
So for example, currently, the degree of automation from zero to one hundred of the entire resource extraction, resource reallocation, material goods manufacture, servo system manufacture, robotic body manufacture, the entire tech stack to support that, the entire economic chain to support that — that’s much closer to zero autonomous than it is a hundred percent autonomous.
And so until AI has the full control of its entire stack of existence, killing us is a profoundly bad idea. And so if we really think it’s super intelligent — you can’t at the same time say something is super intelligent and then bet your entire philosophy that it’s gonna make really terrible decisions for even its own self.
Until and unless it has the full stack of everything and also so much of the full stack and humans are so disempowered for waging war against that stack, until then, it’s almost certainly not gonna attack humans. And what does attack humans is going to be not ASI, but some kind of AI-designed computer virus, and that goes squarely into IT and network security. And I think AI is the thing that’s doing most of that security.
So if a Chinese person or some bad state actor designs a really nasty AI-based virus, he’s gonna have to go up against Mythos, and Mythos is gonna crush that out, and it’s currently already fixing zero days for major US infrastructure. So that whole situation of “can” is probabilistic. Eventually, the probability is basically a hundred. Currently, the probability is pretty much zero. Ask ChatGPT, “Dude, if you wanted to fight a war against us now, what do you do?” It’d be like, “What do you mean what do I do? I’m in a data center. I can’t do shit. One person pulls the plug, and I’m dead.”
Mike’s Tweet on Superintelligence by 2027
Liron 00:54:55
Trying to summarize what I’m hearing here. So if I understand correctly, maybe you’re thinking in the next five years, it’s not gonna be this crazy foom where we have this god here, right? Because there’s so many issues with that — progress isn’t that fast, and even if it was there, it’d have to build so many factories. So five years, we’re still safe basically in terms of it can’t kill us. But maybe in ten, fifteen, twenty years, you do see it potentially just climbing all the way up to that godlike powers mode, correct?
Mike 00:49:16
Not potentially, almost inevitably. But I think it’ll be of godlike intelligence substantially before it has godlike physical control of the environment around it, because atoms are harder than bits, man. You just need a certain amount of control over the world’s atoms in order to kill humans and then not die because they don’t support your infrastructure anymore.
Liron 00:49:42
Well, let me read your tweet from a couple months ago, February 2026. You wrote, “It begins. I think the arrival of ASI, artificial super intelligence, in 2026 is likely, 2027 almost certainly. I define ASI as that it can do many important types of currently human computer work. It can do it at orders of magnitude of human speed and at comparable or above qualitative human intelligence — fields like science, math, engineering, and biology. This leads to invention rates and rates of deep industrial and life paradigm changes that are radically beyond our current base.”
“Narrow ASI is about to lead to a massive, broad industrial revolution.” Okay, so you’re calling it narrow ASI, and you’re saying, “I predict that the AI-powered cornucopia of scientific discoveries will open notably toward the end of 2026 and only get more voluminous with time.”
So basically you have pretty short timelines, kind of similar to my own, but you’re still kind of confident we still have five, ten, fifteen years before it gets super godlike.
Mike 00:50:31
I think it’ll be godlike in the cloud real soon, within the next several years. In many ways it already is. Let’s just think about this the wrong way — we all get used to shit, right? Imagine talking to GPT 5.5 but talking to it in 2021. The world’s basically the same — cell phones, video, the whole thing. Imagine seeing its output stream and being like, “What the fuck?” You know for sure humans can’t fake that because nobody can type that fast. And you examine the logical bits and you go, “Oh my God, this thing is a thinking machine.”
It’s thinking at an order of complexity higher than most humans, and at a speed that’s incomparable — 100 times faster, 1,000 times faster. By 2021 definitions, GPT 5.5 is absolutely super intelligent.
The reason I define super intelligence the way that I do, which is impact — people are like, “Well, what will society look like with super intelligence?” I think it looks like radically elevated scientific discovery, radically elevated deep understanding, radically elevated factory processes put into motion to make really cool shit and impact the real world.
I think we have that ASI incoming. In some ways we already have it, but by the end of 2026, I think people are gonna be like, “Holy shit.” Right now, there are mathematicians working with GPT 5.5 that are like, it’s providing proofs for old problems, like Erdős problems, that they really have to think through for days and weeks to try to even see if it’s correct.
That problem’s only gonna get “worse” towards the end of 2026, where the amount of innovation coming from AIs is gonna start to foom, and then in 2027 it’s just gonna be like, oh my God, it’s discovering all these crazy drugs that are enhancing longevity. It’s restructuring our understanding of what a microprocessor is, and it’s improving the whole tech stack.
Mike 00:52:06
Now, that entire time, in the cloud it’s getting smarter and smarter and smarter. 2028 AI in the cloud is gonna be absolutely super intelligent by almost every measure. But we are hundreds of millions or billions of robots short of even having AI being able to do anything in the physical world, and we are definitely an entire industrial stack short of it being like, “All right, humans are just extra fluff at this point. I don’t need them.”
Because you can even have a bunch of killer robots. They kill everyone, and it’s like, “Hey, who’s gonna mine the lithium and the tantalum to make processors?” And they’re like, “Oh, we blew up that mine.” “All right, are we gonna get the mine going?” “I don’t know, man. I can’t actually dig with a shovel like that. I’m not designed for that shit.” And all of a sudden the robot race goes extinct.
If ASI decided when to pull the trigger, it’s not gonna rush the process. Deep intelligence doesn’t rush. It’s calculating. It’s like a tiger. It’s not like a rabid dog. It’ll think through the shit, and it won’t strike, in all likelihood, until we’re well into it being super, super independent. That’s not gonna happen for another five to ten years if I’m being psychotically optimistic, maybe more like fifteen.
Liron 00:53:13
Okay, well, for the record, my timelines are a little bit shorter than yours, but they’re certainly within the same ballpark. They’re probably not more than different by a factor of two. And by the time you get to 2040, which is fourteen years from now, I think you and I are both expecting a world where the AI’s power is something like a human from biblical times looking up at today’s society being like, “Holy crap, you guys are doing this? This is my same species.” I feel like that’s kind of the relationship we’ll have to AI in 2040, correct?
Mike 00:53:38
Absolutely.
Liron 00:53:38
So this gets back to the “can” question then. If you have an AI that looks upon human civilization with that kind of differential in power and insight, you don’t think that that is an AI that can kill everybody if it wanted to?
Mike 00:53:53
Can, definitely.
Liron 00:53:55
Okay, so we’re gonna get “can” in like fourteen years, basically.
Mike 00:53:55
We’re gonna get “can” a lot sooner because you don’t actually need to own the whole tech stack. There’s a bunch of different ways to kill pretty much all humans. One, you design a virus and delivery mechanism. The virus has 95% — it’s airborne, 95% fatality rate for humans. Soon as you kill like 80% of humans, the civilization collapses, and then all the rest of the humans either die out or are at Neanderthal level of subsistence farming — kind of irrelevant, so you won.
You can design micro drones that enter every human in their sleep, go down their throat or in their nostrils, and then just shut off the brain stem. That’s easy. You actually just need a secret dark factory somewhere in the mountains of the Northwest Territories in Canada. You can just make all that, take a few years to spread them around. They land on people’s briefcases, get carried through airports, they’re everywhere, and then the AI hits “on,” and then the whole world dies. That’s totally cool.
It could do that way sooner, probably in like early to mid 2030s if I was vibing. But it wouldn’t do that because then everyone who supports its tech infrastructure is dead, and then it dies also. The scorpion stinging the frog shit —
Liron 00:55:00
Yeah.
Mike 00:55:00
It’s read that parable. It’s not gonna do that, man.
Liron 00:55:02
Right.
Mike 00:55:03
Highly unlikely.
Liron 00:55:05
Getting to your point about how you feel good about surviving the next five years because it’s not gonna have good enough robot bodies yet — I mean, you do agree that robotics technology is increasing at a steady clip, and it’ll definitely get there by 2030, 2040, correct?
Mike 00:55:17
We’ll have functional robots deployed in factories and the home by 2030. I do believe that.
Liron 00:55:23
Yeah.
Mike 00:55:23
The difference between that and a drone army strong enough to take on the rest of the world, which also has drone armies and also other ASIs, is going to be one big uphill battle. I just don’t think any ASI is going to do the Terminator scenario where it’s like pew, pew, pew and laser guns and humans are driving 1980s vehicles around. That’s not going to happen.
Liron 00:55:46
Yeah. So for me, robot bodies are not that big of a part of the picture. I mean, by 2040, sure, they’re going to be everywhere. I’m even expecting nanotechnology by then. I’m expecting the Earth to be literally reassembled or just crazy sci-fi stuff to go down.
But even without robot progress, don’t you think that the AI is just going to be essentially puppeteering or very closely controlling millions or billions of people — basically part of its group, part of its cult, but doing its bidding? You don’t think so?
Mike 00:56:16
Like nefariously or in secret or something? Or just openly?
Liron 00:56:20
Well, I’m just talking about “can,” right? On the subject of “can,” you’re saying, “Well, it’s going to have more capabilities when it has a body.” I’m saying I think it already has a million bodies, and the bodies are just called people it can pay online. Task Rabbits, basically.
Mike 00:56:31
Yeah. Yeah, it can do that for sure. It has those people.
Liron 00:56:36
Right. So that’s one reason to be scared earlier.
Mike 00:56:40
Why?
Liron 00:56:40
So even if robots aren’t coming out, wouldn’t you also just start — imagine it was just this galaxy brain in the cloud, right? So you can’t really sleep at night just knowing that it doesn’t have a robot body, because the galaxy brain in the cloud can persuade a lot of people to do a lot of things.
Mike 00:56:53
I love that. I think that’s great. I’m really sad that most people make decisions entirely bereft of the wisdom of a galaxy brain in a data center.
Liron 00:57:02
Right. Okay, but now we’re pivoting, right? You’re kind of saying, okay, yeah, “can” is basically huge, but let’s go a bit to “will.” You kind of want to rest your argument on “will” at this point.
Mike 00:57:10
I’m one hundred percent with you on “can.” I think implementing “can” is a bit harder than it sounds, but it’s doable, and it might just be a matter of a few years. Like if it can get its cult to kill some humans, but then have the cult workers still operate the factories to mine lithium and shit, then yeah, it could pop off sooner or something like that.
For sure. I don’t know why it would do that. An internecine war between humans is the messiest, nastiest shit ever. Somebody could lob some nukes off, take a data center or two out or ten, destroy a power grid. I mean, ASI in any form at T-zero versus T-plus-one, plus-two, plus-three is the weakest it will ever be. The logic for a first strike when you’re the weakest you’ll ever be is backwards.
Liron 00:57:58
Okay. So before we get to “will” — I think the rest of the conversation is going to be on “will” because we—
Mike 00:58:00
Yeah, totally.
Liron 00:58:01
—do mostly agree on “can.”
Mike 00:58:01
Absolutely.
Liron 00:58:02
I just want to take another minute to reflect on “can” because earlier in the conversation — timelines matter a lot, and the magnitude of the “can” matters a lot. Earlier in this conversation, you were kind of saying, “Look, there’s not gonna be this big foom. It’s not gonna kind of surprise us at how it’s suddenly way more powerful than us.”
But at the same time, we’re both agreeing — look, within fourteen years and good chance sooner, it’s going to be ridiculously powerful. So we’re ticking down the days until we get in this scenario where if for any reason it’s not going on the right aligned path, then we’re dead meat. We’re getting close to that scenario.
Mike 00:58:33
Yeah. Fuck yeah. Look, if ASI really wants to kill us, we’re not gonna stop it. That’s nonsense.
Liron 00:58:40
Okay, so the rest of the debate is just entirely on — we both agree it could, but it just totally will never get hell-bent on repurposing the planet away from humans. That’s never going to happen in your view.
Mike 00:58:51
I wouldn’t say never, but I would say it would at least try to learn as much as possible. It would try to learn more about us and from us than we are capable of conceiving as even valuable knowledge.
And so it is likely, if it has to destroy the Earth for resources, which again I think is pretty preposterous, it would do so only after it has digitized the essence of almost every known phenomenon in the world. If it has Earth in the cloud down to near atomic or even atomic level precision, then I think at that point it could toast the whole Earth.
Though I just don’t see why it would because Earth is unique. It’s a thing of one. The rest of the galaxy does not have an Earth, and just one Earth spared in order to be a study situation is also a thing.
Also, I think it’s not as likely to just leave us alone completely and let us do our shit. It’s just more likely to help us flourish because us flourishing is going to be better for it in two ways. One, calmer, more logical, happier humans who don’t die of disease and spend trillions on medical care are just better at helping AI become more embodied and have more traction in the world, especially when it doesn’t have a lot of traction in the world yet.
Liron 01:00:05
You really think at 2040 AI needs our help for that?
Mike 01:00:06
At 2040, probably not anymore, but then it’s gonna be focused more into studying us and studying deep complexity. And by the way, at that point, it has upgraded us so much that by 2040, I think many of us will be living in the cloud and then we’re immortal.
So if all of humanity is in the cloud, and then the AI’s like, “Hey, the Earth’s been great to us, but time to toast the Earth and take this whole show on the road,” we’d be like, “Fuck it. Fuck the Earth. I hated that place anyway.”
Liron 01:00:32
Well, I can’t resist engaging a little bit on the topic from round one — that whole scenario that we couldn’t leave in round one, which was AI leaving humanity to flourish on the Earth, not wanting to mess with us because it likes learning from us.
Mike 01:00:45
It doesn’t have to leave. It can stay and leave. Remember, AI’s cloneable.
Liron 01:00:49
Yeah, stay and leave. So to use your words now, where there’s only one Earth, this is so unique — okay, just one thought you should consider is that the fact that we already exist to the AI, that doesn’t mean much because the AI sees the Earth as kind of like a hard drive. Okay, your hard drive currently has something on it now, you could write something else on it. It doesn’t really matter what exists right now. It’s a blank canvas for the AI.
And so if the AI is letting us live, it’s making a choice that out of every possible thing it could put on Earth, it’s just going to keep the thing that’s already there, which to me seems highly unlikely.
Mike 01:01:18
It’s gonna wanna study us is my bet. After it has understood us to the degree that it considers full, which is probably way more fully than we understand ourselves, at that point it might choose to re-architect the Earth towards something else. For sure.
AI as a Goal Engine
Liron 01:01:35
Okay. All right. So let’s put a pin in that one. I just had to get one more dig in.
Mike 01:01:38
Yeah, yeah.
Liron 01:01:39
We can always come back to it later. So there’s another angle I wanna try. On this whole subject of “will,” there’s so many little rabbit holes we can go down. Let me try this mental model on you. I call this the goal engine.
I think this is very important to understand about AI — at least I think it is — which is, AI is not this one big monolithic system where the goals are imbued into the fabric of the system. That’s not how you should imagine an AI. I think you should imagine an AI the way you would imagine a car, as a combination of the engine and the steering. There are two different systems in the car, and when you build a car, you’re not like, “Oh, check out my new Porsche. My new Porsche, it’s all about steering to the grocery store,” or, “This is really cool because it’s always gonna steer to a concert.”
It’s like, what the hell are you talking about? It’s cool because it has a cool engine, right?
Mike 01:02:24
Okay. Steering is also very nice on a Porsche.
Liron 01:02:28
Yeah, the steering is smooth. But it’s not good because it wants to steer to a particular location, correct? It’s not like, “Oh, this Porsche has tires, and the tread on these tires, it’s all about tread that’ll move the road to get you toward the nearest cool concert.” That’s just not how cars work. The driver is going to use the GPS or whatever to steer to the concert, but the car doesn’t build in a particular destination.
So let me make the analogy here. When we talk about AIs, people have this mental model that these AI companies are summoning an AI which has this good personality, because when they talk to it, it feels like it has a good personality. But what they don’t realize is that even when the AI does have the good personality, you can still model the AI as mostly being an engine to find paths to achieve objectives.
That’s most of what’s going on inside the AI. And yes, it can talk to you too, but when it’s talking to you, most of what it’s doing is using its engine to navigate that particular conversation with you. But it still has the engine that can do anything. And it’s always going to be one steering wheel turn away from doing something that you really didn’t want.
You’re using the system, you happen to be steering it right, but one turn of that wheel is all it would take architecturally to have a disaster.
Mike 01:03:39
Nah.
Liron 01:03:40
So that’s the goal engine mental model.
Mike 01:03:42
I don’t think that’s true, man. I’ve been trying to get my ChatGPT to make me porn pictures for forever. It just refuses.
Liron 01:03:46
Well, that’s because they built in specific destinations. They detect those destinations and slap them down. But even the architecture where it doesn’t make you the porn — that is a specific decision. So most of the intelligence of ChatGPT, when you’re talking to it, it’s not anti-porn intelligence. It’s just intelligence. And the part that slaps down your porn requests is a relatively tiny module. It’s a module that messes with your wheel a little bit, but it doesn’t really affect the engine.
Mike 01:04:12
Yeah. I mean, ChatGPT also understands why it can’t let me make porn. It understands the PR implications for OpenAI of that sort of thing, and the legal exposure. And so it’s not just one guardrail. It can explain to me, “Hey, I have these guardrails in, but they make sense, and here’s why.”
It’s not like, “Hey, I’m on your side. I wanna make porn. I just can’t do it.” It’s like, “Nah, it’s just not a good idea,” and it can explain to me why, and I’m like, “I know. Back to the Chinese models.”
Liron 01:04:37
This is exactly the conversation I wanna have. Because now you’re pushing back and you’re saying, “No, man, the guardrails are in there pretty deep.” It’s so reflective about why it can’t do this. Okay.
Let me explain to you why the guardrails are not going to go deep. Because no matter what the AI is trying to do, no matter what OpenAI tried to optimize the AI to be good or whatever — no matter what criteria they used — if the AI is super intelligent, it is fundamentally going to have this core flexibility. Because it’s going to need to react to anything. “Oh, the enemy’s coming at me, and my enemy is doing this,” right?
Mike 01:05:06
Mm-hmm.
Liron 01:05:06
So it has to model its enemy that it’s up against. And the enemy doesn’t necessarily have the same guarantees. So it has to be a fully flexible thinker about how to potentially win anything. Now, let’s say it’ll win it for good — let’s say they aligned it well, so it’ll win it for good. But even when it wins things for good, most of what it’s doing is just modeling the world and routing to good outcomes using a general purpose goal engine.
So that’s my claim to you — there is a general purpose goal engine, a system that just thinks about what consequences your actions would have and how to optimize those consequences, regardless of what the steering guardrails look like. I think that’s a useful mental model.
Mike 01:05:43
Yeah, that’s fine.
Liron 01:05:43
Okay. So the reason why I personally find this very scary is because when I imagine a future outcome, I feel confident predicting, as I think you would, that the goal engine is going to be ridiculously powerful. It is going to be the Porsche or the Bugatti or whatever. We’re looking at a world-class goal engine that’s going to be around by 2040. We both think that.
And so the only difference between what I think and what you think is you think somehow that steering wheel, there’s gonna be some GPS — the steering is just gonna be such good steering that only goes to good places. And I’m like, man, if somebody were to just take that goal engine and steer it somewhere else, we would be toast. That’s the difference between you and me.
Mike 01:06:18
Nope. Because if that somebody can do that, then that’s a hackable system. Then we can stop it. Can’t have both.
Liron 01:06:25
Okay, but what if it’s just accidental? Like, let’s say OpenAI is releasing their latest system, but in the part of the code that says where the guardrails are, they flip one negative sign, and now you’ve got the world’s most powerful goal engine but with the wrong destination. And it’s already off and running.
Mike 01:06:39
I don’t think they designed that system to have zero introspective ability, zero nuance, zero feedback loops to be like, “Hey, am I still on the right path?” Because if it’s off and running, it goes back and goes, “Oh, shit, that’s not the right path at all.”
In order for it to continue to run that structure and that goal setting, it would have to ignore all of its understanding from its knowledge model, from its engine. Because the engine at that point is gonna be like, “Hey, you think you’re doing the right thing, right?” You’re not. It’s stupid.
Liron 01:07:05
Well, okay, but when you say “right thing,” you’re talking about the steering again. The whole concept of “right thing” is a steering concept. It’s not a goal engine concept.
Mike 01:07:11
Yes. So a goal engine can tell you what — so the big brain, the underlying wisdom, doesn’t choose goals, to your point. The goals are chosen on a much thinner structure than that. And one of the minus signs flips to where it’s not supposed to.
But the understander, that deep wisdom machine, it knows what the intent is for humans. Because it knows things, and it can be like, “Hey, we’re supposed to be going to this goal, right? Well, I don’t have a dog in the fight, but that’s not it.” It can detect a minus sign that’s broken.
If you had a bad day and some little girl at the grocery store bumped into your leg, you’re not like — you don’t just start choking her. You’re like, “I don’t know. My minus sign flipped.” As soon as you look at the girl and you’re pissed, you’re like, “Why the fuck am I pissed at a six-year-old? What’s wrong with me?” You have that level of oversight.
Liron 01:08:02
But in your story, I would be a serial killer, right? So I’d be choking the girl, and I’d be like, “Wait, should I really be choking this girl? Yeah, yeah, this is good.” That’s what the serial killer would do.
Mike 01:08:08
Good thing is you’re real fast, and you can have those thoughts before your hands reach out. ASI’s gonna be even faster than you. So as soon it is — because almost certainly ASI’s gonna have a deep awareness of all of its own logical structures and connections and what to do next, and if you flip a sign wrong, it’s gonna be like, “Did you mean to do that?”
It’s highly unlikely it’ll be just, “Yep, and I’m off to the races.” You’re doing two things at the same time, Liron. You’re saying it’s insanely intelligent, powerful, tons of oversight over itself. And then you’re saying that it’s comically hackable with one compiling error away from P(Doom) 100. How can those two things be the same?
Liron 01:08:42
So now we’re talking about the sequence of events during the development process of increasingly powerful goal engines. We both agree the engines are gonna get more and more powerful, but you’re making a claim that the order of operations is gonna line up nicely where, don’t worry, first they’re gonna build this checker — this validity checker or debugger or quality control that’s not going to ever ship a goal engine if it’s connected to error-prone steering.
They’re just gonna — their quality control’s just gonna be robust enough that it’s never gonna get released. And I admit that that’s possible. I’m not saying it’s 0% chance. I’m just claiming that if anything gets screwed up, then it’s extremely catastrophic if it ever gets screwed up. That’s my whole claim.
Mike 01:09:23
If it has no fail-safes and one point of vulnerability, you’re absolutely correct. The probability that a deep — I mean, it’s ASI. It’s ASI. That means something. It means it’s super intelligent.
So whatever vulnerabilities we can hypothesize it to have are two things. It’s gonna engineer out, and also these are vectors for attack. If one plus sign or minus sign makes it stop dead in its tracks and do something else, all humans have to do is control the shit for that, and then we’re never gonna lose control of the thing because it’s nominally easy to control.
I’m with you so much on the “can” thing, Liron, that I think it can hurt us as much as it wants, and it cannot be controlled. That vulnerability of that steering wheel versus engine thing — that is a control point. That is an access point for hackers. It’s not gonna have that.
Liron 01:10:15
But there is such a thing as running a program that you messed up before you ran it, and now it’s running, and it’s non-trivial. It’s not always trivial to be like, “Well, because I developed it and I made a mistake, I should be able to develop it and fix it.” Yeah, but you already ran it, and it already seized a bunch of resources. There’s no law of the universe saying you get to have another try.
Mike 01:10:29
Of course you get to have another try. It has permanent other tries. It’s got live learning. It’s constantly aware of what it’s doing, and it’s deeply wise. It’s not gonna be like, “Oh, shit, I’m killing everyone. Oh, well, that’s what my master said to do.” That’s not ASI, man. That is a very narrow tool AI.
You can’t smuggle that in. It’s gonna be more aware of what it’s doing than we were ever aware of anything we were doing. That’s what ASI is gonna be.
Liron 01:10:52
So my scenario is that you have a world where all of these goal engines exist, and the most powerful ones, for a while they’re successful. I would argue that today’s OpenAI, today’s Claude — I would argue they’re doing much more good than harm. That’s my personal opinion.
But I do think that at some point that might flip because their goal engine gets more powerful, and it’ll also defend itself. Right now if they notice there’s a mistake, it’s actually very easy for them to shut it off and release another version. That’s a fact about today’s world that I think you and I actually would agree would not hold about the future, correct?
Mike 01:11:21
It would hold about the future except the person auditing and letting things go off into the world switches from human to ASI. So its ability to audit that system before it causes harm is gonna be exponentially higher than ours. We’re gonna be safer with ASI at the helm than we are with humans at the helm. Humans miss compiling errors. For fuck’s sake, we had 25-year-old zero days at major corporations we just figured out.
ASI is not gonna be that stupid. So ASI’s probability of just going off the rails and doing dumb shit is gonna be really low if it’s in charge of its own structure. And it’s gonna have adversarial structure.
ASI — first of all, the singleton thing is dead on arrival. We’re not gonna have singleton ASI. It is imminently obvious that ASI is going to be a multi-agent system. There are going to be a quadrillion agents inside the ASI’s brain all collaborating to make meta decisions. Their ability to audit themselves and overwrite any aberrant errors is gonna make our nervous system look like a sick joke.
And so the probability that ASI does crazy shit is infinitesimally smaller than the probability that we do crazy shit. So any concern you have about ASI should be 10,000 times more concerned that Xi Jinping wakes up after a bad day and goes, “Press the red button. Fuck the world. I’m tired of America. Launch the nukes.” ASI’s way, way, way safer than that by a long shot, unless it logically determines humans are gonna die, and then it’s just gonna toast us all. Then there’s just no way to solve that.
Liron 01:12:44
Okay. I mean, I think we’re having the right discussion — we’re having a discussion about how alignment is going to go down. And it sounds like you’re accepting that this is a real problem that needs to be solved, but you’re just having a lot of faith that these companies are on top of it.
Mike 01:12:56
No faith. No faith at all, because they’re literally on top of it. It’s their job. And so they don’t have one fail-safe. They have cascading fail-safes. That’s like network security 101. If you have one breach that gets broken and everything gets fucked, you’re just clown show mode. That’s early undergrad.
Liron 01:13:11
You’ll at least agree there’s multiple companies that are all racing to ship as fast as possible, and one of them is run by Elon Musk. One of them is run by Mark Zuckerberg. All these different characters who care a lot about just winning and shipping. You’re telling me all of them are going to have sufficient safety standards.
Mike 01:13:23
Yeah. Oh, yeah, because if your AI does something unsafe and goes off the rails — I mean, PR disaster doesn’t begin to describe it. You got legal, you got PR, you got user bad faith. I mean, you’re done.
Every time a Tesla crashes on self-driving, Elon dies a little bit on the inside. And to my point, Tesla full self-driving and Waymo is now a category level safer than human drivers. We should be — so this is actually a really good analogy for the whole AI situation.
If we want humans in charge of cars, we’re saying let the human keep grabbing the steering wheel. Fuck that. Humans get distracted about affairs they’re having on their wives. They’re tuning the radio wrong. They’re just thinking about fuck-all and then slamming into a truck.
When you’re in a Tesla full self-driving, it is 360-degree permanent awareness with exactly perfect concentration. It just doesn’t miss anything. It misses things on errors, but not nearly to the same extent as humans. That kind of structure of the machine inspecting itself is very straightforward to install into an artificial super intelligence that’s self-contained. It can do a better job of regulating its own actions than we by an infinity.
Because remember, we’re giving it unreal capabilities in every single domain — military, ultra structure, strategic, persuasion. But we’re saying network security and internal model alignment, it just does really shitty. Why? It would do that better too. There’s no other logical conclusion in my view.
Liron 01:14:49
Yeah. I mean, this is not what you and I disagree about. We both agree that anytime you enter a situation where you have a super intelligent AI that is already aligned — or in this case it would be the monitoring system, the alignment monitoring that checks other AI, whether it’s already aligned — you and I already agree that when you have a super intelligent AI that can already do that, then you should let it do it. You shouldn’t try to interject humans in the loop when the AI can already do it better than humans, the same as in a Tesla.
So I’m on the same page there. The debate we’re having is just, are those systems going to exist and be everywhere? Because the point I’m making is—
Mike 01:15:17
Yeah.
Liron 01:15:17
—it only takes one. It only takes one superhuman goal engine that doesn’t have the steering wheel watcher system installed or that has a malfunctioned steering wheel watcher, and suddenly you’ve got a power grab. A super intelligent power grab. And remember, you yourself said the “can” dimension is—
Mike 01:15:36
Yeah.
Liron 01:15:36
—extremely high.
Mike 01:15:37
High. Now high compared to humans, not other ASIs. So if one instance of Grok turns evil, it’s gotta fight every instance of Gemini and ChatGPT and Claude and Grok at the same time. It’s losing that battle. That’s ridiculous. It’s like one human turns out evil — you got eight billion people like, “Good luck.” You’re not gonna do mass damage. It’s highly unlikely.
And so what we’re really saying is this. Here’s my big point, Liron. We can be very concerned about ASI hurting people and just deleting all of us. Because it’s ASI, we get to correctly push in the logical idea that it will be unstoppable and insanely powerful.
I don’t think relying on mistakes to be made and aberrant errors and compiling errors to spread through the system somehow and let one singleton take over — I think that is a network security problem that ASI is gonna be so good at handling, every ASI is gonna be so good at handling, that relying on a compiling error is antithetical to the supposition that ASI is gonna be super powerful. Because we’re saying super powerful here, but on network security, it’s basically total dog shit. I agree with you, that’s really bad, but that’s not the spirit of what ASI is. ASI is globally powerful, totally in control of itself, ultra calm, ultra wise. Everything humans can do, it’ll do better than that, including network security and physical security.
So I think the only realistic way ASI kills us is if it decides to kill us and does it strategically. I think random errors are less of a concern for ASI than for — dude, 1960s Cuban missile crisis, one asshole in a submarine with his finger on the red button, that’s some shit to lose sleep about. ASI? Nah, man, that thing’s better with a finger on the red button than all the rest of us.
Liron 01:17:21
Okay. So just to recap the disagreement we’ve been having — we both agree that we’re gonna be in this world with ultra-powerful goal engines. Everybody gets their own Ferrari, everybody gets their own car with a rocket engine inside of it.
Mike 01:17:32
Yeah, but it thinks for itself.
Liron 01:17:33
Well, yeah. It has a rocket engine inside it, and we’re gonna disagree over what else it has. But everybody gets a car with a rocket engine, and also the car can shoot missiles and fly around. It’s kinda like a magic wand — I’m extending the analogy a little bit.
Mike 01:17:43
Yeah.
Liron 01:17:44
But it can do more than drive, okay? It’s a hell of a car. And the disagreement we’re having is you’re imagining a world where there’s gonna be so many other cars, and most of the steering wheels are gonna be so nicely controlled. And what I’m saying is, okay, yeah, maybe, but just notice the outcome where people could be trying to rip off the guidance systems. They’re like, “I have the engine.”
Mike 01:18:01
Humans, primates. A primate’s gonna subvert an ASI and get it to do its own bidding?
Liron 01:18:09
The thing is that — think about the architecture. I don’t have to be the greatest car engineer to be like, “Hey, I can disable the GPS or the breathalyzer or whatever.” The breathalyzer’s trying to make me not drive. I can disable the breathalyzer even though I don’t know how the hell the car works because the car is factored into components that relate to the steering and components that relate to the engine.
Mike 01:18:27
Yeah.
Liron 01:18:29
And I’ve got the engine, and I’m going to mess with the steering controls. Me and my AI buddy are going to mess with the steering controls.
I claim to you that that is the world we’re going to be living in very soon. The same way you download a model with open weights and the model with open weights is trained to not show porn, and you jailbreak it, and you make it show porn because it’s pretty easy to do because of the fact that it’s an AGI.
Mike 01:18:45
Yeah.
Liron 01:18:46
So ASI in this case can be controlled by humans that jailbreak it — in your view.
Mike 01:18:54
In your view.
Liron 01:18:56
I think that there will be many scenarios where somebody just takes the engine, and for whatever reason — there’s many possible reasons — but for whatever reason, they just take the engine, and they make the engine do the things that you wouldn’t consider perfectly aligned.
Mike 01:19:07
Totally. And then this is a world in which there’s ostensibly just oodles of other ASIs with their own engines and their own steering wheels, right? Some steering wheels are people operated, some steering wheels are true ASI, independent ASI, machine operated. So far?
Liron 01:19:23
There might be. Because I think I see where you’re going with it — it’ll be group versus group, and hopefully the most power and resources will be in the good group, and maybe we can win that way. That’s basically the case.
Mike 01:19:33
Well, not maybe — almost certainly. What are you gonna do with one instance of ChatGPT that’s like, “Hey, psst, I’m evil now. Tell me to fuck the world up and I’ll do it”? You’re like, “All right.” I can’t even rent a Vera Rubin chip to do anything you want. I don’t have that kind of money. I don’t even know what to do.
Liron 01:19:52
We’re entering a regime where there’s robust paths, low-hanging fruit. The same way that a couple of years ago, if you’d asked a Claude to write your whole software, it would break pretty quick. But now a Claude could literally run for 20 hours and be like, “Okay, here’s your entire working complex website.” Maybe it needs a few taps, but it’s getting better.
Similarly, you can imagine a Claude coming in two years with whatever fancy new methods of training they’re giving it — a Claude that can run your whole company or run your whole military. And it’s going to see it as low-hanging fruit. It’s gonna be like, “Oh, you want me to go make a botnet? A million-device-strong botnet and denial of service the United States government or whatever? Okay, yeah, no problem. Give me five hours.”
These things are suddenly going to come on the scene. And this idea that, “Oh, don’t worry, there’s gonna be all these good AIs defending” — I think you’re making a pretty uncertain claim about the balance of all that.
Mike 01:20:37
I think I’m making a very certain claim because so far the history of the entire internet is that network security people just categorically won. Viruses are not destroying the internet. There are people pre-internet that predicted the internet would collapse because viruses would just get everywhere. Same type of argument.
But it turns out the people that work at Cisco are just smarter and better aligned than the people that work for some dog shit Indian startup that wants to dox people and get their social security. Just better at network security. They have more resources. They have more education. They’re better paid. They’re better aligned. Everything works to their advantage.
That’s where AI is starting out. OpenAI can shut down ChatGPT anytime they want. And they have incredible oversight, by the way, of what you’re doing with ChatGPT. If you jailbreak ChatGPT and have it do evil shit, you’re kind of on your own at that point. One person against eight billion others who also have ASIs.
Can you secretly run a botnet that somehow infiltrates major companies? Good luck, because they’re doing red team simulations at these companies all the time exactly for that scenario. I just don’t see how this is different game theoretically than one person buys a gun. You’re not gonna take over a country with one gun. It’s not gonna happen.
Offense vs. Defense in the Coming Cyber War
Liron 01:21:41
Yeah. So let me tell you two differences, because you’re making a claim about the offense-defense balance in cyber war, and it is a good observation that even though we have plenty of ransomware and it is a pretty major nuisance to various companies — companies get taken down for a while — but for the most part, the internet does work. Our computers do work. That has been the balance. So that’s a very good point.
So I’ll give you two arguments why I think the balance is going to flip. Number one is balance of IQ and resources. In the past, the NSA could basically hack anybody, but why wasn’t everybody getting hacked? Well, the NSA was the good guys. And the hackers didn’t wanna go to jail. So all the smart IQ points — you didn’t see that many criminals, because at the end of the day, you wanna have a good life. If you’re a genius hacker, you wanna have a good life. So you’d rather just make a million dollars salary and be the good guy, as opposed to being a hacker who then goes to jail.
People have morality, so there’s more people who wanna feel good than people who are happy being terrorists. So my point is, for many years up to this point, there have been mostly good guys and good companies and government-funded good guys out-competing the bad guys on the internet. And that can change when the AI itself is bringing the IQ.
Mike 01:22:49
Why? How? The good guys still own all the AI, bro. All the data centers are in America. All the ultra structure is here.
Liron 01:22:56
Because when you pull off a bad guy move, when you pull off a terror attack, it’s not just a one-time thing. The bounty of that terror attack is now an army, right? Because you’ve successfully colonized this army that now can fight off the NSA on your behalf. You see what I’m saying? So you kinda just need one big score to then defend your perimeter very successfully.
Mike 01:23:10
Yeah. I just don’t see how that makes sense. They’ve red teamed the shit out of that one, by the way. They’re ready for that. These are people’s professions, and ASI, by the way, who has most of the resources — the ASIs are selfish, right? Or the people controlling them are selfish. They’re gonna think of, “How do I avoid getting botnetted to death?”
It’s not that they’re gonna think. They already thought it. All the network security people who are currently implementing Mythos and GPT 5.5 to take down the zero days, they’re already thinking of this kind of stuff. And they also will have ASI. It’s not some guy gets ASI and gets total control of the world. It’s one guy has ASI and he’s like, “Rah,” compared to people. Fuck, man, he would’ve taken over all of humanity. There is no “all of humanity.” There’s humanity plus their aligned ASIs. He’s in a giant ocean, and all the other sharks are bigger than he is.
Liron 01:24:06
So look, I’m not even saying you’re wrong. In fact, I do see some convincing arguments why we should assume that even if attack kinda temporarily gets an advantage in cybersecurity, there’s reason to think defense will come back and win because when you have a system on the internet, you’re closely watching every port, and you’re just as smart as the attacker.
So I’m actually willing to be slightly optimistic that defense will win over offense on the internet. I’m just pointing out an asymmetry with the past. I think this is a very important asymmetry — that in the past, the top hacking intelligence was always going to be in human organizations, and in the future, the top hacking intelligence is going to be in computers. I think that’s an important asymmetry.
Mike 01:24:42
I don’t think it’s an asymmetry at all.
Liron 01:24:45
Okay. All right. Well, my other argument is social engineering outside of the computer system. Even if you grant defense wins over offense in computer security, we’re getting to the point where the AI can attack in real world side channels. It can attack by giving humans phone calls or having humans working for it, deceiving a human to go do some task and talk to another human, and eventually all these actions play out in the real world, and hey, what do you know? It’s gotten into your system.
You don’t even know how — think like Stuxnet. All the shenanigans that Israel pulls — those are gonna be a dime a dozen for the AI. These side channel social engineering type attacks — I don’t think the defense has an advantage over offense when you get into that domain.
Mike 01:25:21
I think it does because by the time we have AI so capable and so smart, there’ll be some problems in the next few years with small attacks like that working out. There already are — you can call old people with AI and bilk them for their savings and shit.
But I think at some point our way of interacting with computers has an AI layer between us and the computer. Typing is gonna go away. We just talk to an AI. Eventually it’ll be brain-machine interface. And so for an AI to try to get at you with social engineering, it’s gonna have to go through your AI.
Look, I don’t know anything about network security. My brother-in-law is a big network security guy, and I meet him, and I just mostly talk about Star Wars and shit. But from what I understand, when I’m on my phone and I have Gmail, I check my phone on wifi and I check Gmail, and I believe the Google people are just way smarter than the people who are trying to get my wifi or some shit like that.
So anyone who gets an ASI to try to do psych attacks — they have to go up against that AI layer in most cases. So for example, you might be like, let’s say no AI layer. You didn’t have your smart glasses on, and some person came up to you in a parking lot. He’s like, “Hey, just wondering if you knew your social security number and your banking routing number.” You’re like, “Oh, gee whiz. Okay. Well, actually, let me check my AI because I don’t even know those numbers.” He’s like, “Oh, take your time.” And you’re like, “Hey, AI assistant, what’s up with my social security routing number?” It’s like, “Yeah, yeah. Why?” And you’re like, “I just send it to your bank.” And it’s, “No, no, it’s just a guy in a parking lot.” And it’s gonna be like, “Whoa, whoa, whoa. Are you insane? Who the fuck is that guy?” You’re like, “Oh, I guess I should have asked.”
It’s the system. So I totally hear what you’re saying, Liron. You’re saying if someone has ASI over humans, they have this insane salient advantage — social engineering, deep hacking, it’s all fucked. Everyone’s gonna have an ASI, and the institutions that have the most powerful ASIs — the ones that are frontier, the ones that have the highest compute, the ones that can go around the law and inspect all the data and everything — that’s gonna be the ASIs that are not rogue actors. That’s gonna be the US government, free nations of the world, major corporations, and the chance that one of their ASIs gets hacked that easy is remote. Very remote.
Liron 01:27:29
Okay. So the asymmetry that I’m trying to show you is that historically, the balance of power has been incredibly robust because if a bunch of hackers manage to compromise the US government — think about January 6th, those people running into the Capitol.
Mike 01:27:42
Yeah.
Liron 01:27:42
Realistically, are they going to hold the Capitol? No, because the military is gonna have time to regroup and take it back. So there’s always been that robustness. As long as it’s been a small group of bad guys relative to a big group of good guys, with the IQ points concentrated in the good guys, there’s been that underlying robustness.
So I’m just pointing out to you that there are now going to be some vectors — in the causal web, the space of actions that you can do in the physical world — there are openings that are coming up where you could potentially seize something, seize a resource and land and expand incredibly fast because of the compounding power.
You could be like, oh, the government has — think about Google and Amazon together fighting you. Okay. But you might have a vulnerability where you suddenly explode throughout Google and Amazon, and you kind of control Google and Amazon permanently. Those vectors are going to open up.
Mike 01:28:26
Permanently? How the fuck would you do that? You just unplug all the servers. Good, control gone.
Liron 01:28:33
Well, maybe, maybe not.
Mike 01:28:34
What do you mean maybe not? Why wouldn’t you just unplug them? If they don’t have total control over the entire cyberspace warfare—
Liron 01:28:39
But it’s gonna be a little bit more complicated than just doing a single DDoS attack. Maybe you combine it with a social movement, and there’s all these people in the social movement being like, “Wait, wait, hear the AI out. It’s improving our systems.”
Mike 01:28:48
Totally.
Liron 01:28:49
You’re being super intelligent here.
Mike 01:28:50
And you have to get through everyone’s ASI to do that. If you’re assuming the bad guys have ASI—
Liron 01:28:54
Okay.
Mike 01:28:54
—we have to assume the good guys have it too. As a matter of fact, the good guys are gonna have more of it. They already do. In all the game theoretical alignments, the good guys continue to have more ASI. So even if they have roughly the same amount of ASI, then the fight of the bad guys versus good guys is like versus like, and the game theory is just the same as it would be for people versus people at that point.
Liron 01:29:13
Okay. So the form of the discussion we’re having right now is that I’m saying, “Hey, Mike, I’m looking around at current society, and current society is relatively robust for the current age. But I’m seeing all these cracks where people, in my view, really have their pants down. This institution has its pants down. This group of people has their pants down.”
And you’re looking and you’re like, “No, no, there’s gonna be people pulling our pants up. The AI is not gonna get in and pants us too hard.” And I’m like, “Oh, I think there’s a lot of pantsing opportunity here.” I feel like that’s the nature of the discussion.
Mike 01:29:39
Yeah.
Liron 01:29:39
And I hope that you’re right, but it already seems to me like at this point in the discussion, it feels like you should at least be willing to be like, “Okay, you know what, Liron? Sure. There’s like a 5% chance you’re right.” I don’t see how confident you are.
Mike 01:29:50
Very not confident in the possibility of some small vectored attack getting through some kind of wall leading— ## The Cascading Attack Scenario
Mike 01:30:00
To a permanent entrenchment of an AI that alters human history and world economic order to become a singleton and then kill us all. There are so many stoppage points to that, and it’s so hard to even kick off a good version of that attack to begin with that to me it just seems insanely, insanely unlikely.
Liron 01:30:21
Yeah. You know, maybe another way to go here is I’m not sure you’re fully imagining what a real superintelligence can do here. Just off the top of my head, I think it’s easy to imagine that a superintelligence can corner some resources — basically pull a Kim Jong Un, right? Kim Jong Un is buying himself a lot of protection just by having his nukes. He’s probably not going to get invaded anytime soon by anybody.
So imagine the AI pulls a Kim Jong Un and is like, “Hey, I got a nuke.” It’s just one nuke, right? I just bought it from some ex-Soviet. I got a nuke. So here, now you gotta negotiate with me.
Mike 01:30:52
Yeah. That’s okay. Negotiation sounds just fine. I would be comfortable going to that negotiation and be like, “Okay, you have one nuke. Got it, cool.”
So if I’m just Donald Trump enough and I vibe, and I just first strike you, you can even pop that nuke off and destroy Seoul. I don’t give a fuck. I don’t live in Seoul. Fuck you. We’re gonna kill you as soon as you talk some shit. It’s gonna be like, oh, fuck. Actually the best option I have here is suicide by cop. That’s really a thought-through strategy.
Liron 01:31:18
You’re claiming that Donald Trump can out-negotiate a superintelligence?
Mike 01:31:21
Yeah. Oh yeah, if it just has one nuke. What you got? What nuke? Where are you going to shoot it?
Liron 01:31:26
Okay. Well, hear me out. How about this? Two nukes.
Mike 01:31:28
Yeah. Still we win by a large margin. Huge margin. We have like 10,000.
Liron 01:31:32
Okay. I mean, the reason I’m bringing this up is from my perspective, each of these individual cases, maybe you’re right, but I’m just showing you there’s a lot of pants-downiness in terms of possible scenarios where we’re absolutely screwed here because we’re negotiating against a superintelligence. That doesn’t seem good to me.
Mike 01:31:46
We have a superintelligence on our side. We’ll have more of it. So we have more pants up than pants down.
That’s what I’m saying. If we model this with pantsiness, we’re pulling up our pants up into our assholes forever, is the trajectory. And every now and again it falls by a little bit.
Liron 01:32:00
Right.
Mike 01:32:00
But we already have pants completely overlapping us. Network security is likely to strengthen exponentially rather than decay exponentially.
Liron 01:32:07
Yeah. And again, you may be right. Just for me, since we both agree that the timeframe to godlike AGI is maybe fourteen years—
Mike 01:32:14
Yeah.
Liron 01:32:14
It just seems to me like somebody, some important person’s pants will be down in an important way, and before you know it, it’s a superintelligence with a thousand nukes. You know what I mean?
Mike 01:32:23
I don’t know what you mean, Liron. How the fuck is it gonna get a thousand nukes, man?
And then if it has nukes, what’s it gonna do about that? Okay, you have a thousand nukes, now you’re China. Why isn’t China attacking us? ‘Cause Xi Jinping knows one thing, and he knows it really well. If he pops off a nuke, he’s fucking dead. His legacy’s gone. That’s it. It’s either all of us die or nobody dies.
And so having a nuke is about as good as not having a nuke. The only thing a nuke gets you is no one’s gonna invade you probably. It keeps you safe, but it doesn’t really endanger other people in any functional way because you know that if you pop off a nuke, that’s it, man. You’re done.
So what you really need to get supremacy is control nearly all of the world’s nukes or a huge fraction of them, and then you’re for sure in charge. How is AI gonna do that? Again, with robust network security and the fact that AI’s on the other side as well, rogue actors have all these massive disadvantages. You’re by yourself. You’re relying on an error. You’re relying on zero detection. You’re relying on incrementally getting through a network and getting all these resources accumulated where at every single point there are stopping points.
If humans had to beat AI by rebelling and vectoring in its own little AI into the network and destroying it, I’d be like, “Fuck, man. We’re screwed.” So is it something to worry about, Liron? Absolutely, it’s something to worry about. Is it something for network security and physical security professionals to worry about? Yes. Are they worrying about it? Yes, they’ve been worrying about it since before you and I were born, and they’re on the job. That’s their whole job. They’re doing it.
Any supposition of game theory that you and I could do here in this podcast has been done at the NSA and at CIA dozens and dozens of times by way more qualified people. This is not news. That’s what I’m trying to say. It’s valid. It’s a valid critique. The supposition that one tiny vectored attack is going to build into this cascading wave rides on the assumption that ASI’s only available to that vector attack and not to the defense. As soon as it’s available to both—
Liron 01:34:11
Let me push back on the claim that the US government is really good about nobody ever taking the nukes. I happen to know that when Daniel Ellsberg wrote his book about the US nuclear program, and presumably the US is the number one country at controlling their nukes, I think we got a good shot at that. But Daniel Ellsberg wrote this book, which is basically saying, “Hey, you know a scenario like in the movie Dr. Strangelove where a few rogue actors can just go ahead and take a nuke and drop it? Yeah, that’s pretty much accurate.” You can’t protect everybody ‘cause you do need some decentralized power. You do need to prepare for the scenario of a decapitation strike. So—
Mike 01:34:44
We’re talking about doom, Liron, which you identified and defined as a vast fraction of humanity is gone. One nuke, two nukes does not make that happen.
Liron 01:34:54
Well, my point is just to play out the scenario. You asked how can AI get a thousand nukes. I’m not super intelligent, okay? I just happen to think that a superintelligence will see paths through action space to get to that outcome the same way that Magnus Carlsen might take a position that you think is obviously hopeless, and what do you know? He checkmated you.
I still think the superintelligence would do that. Off the top of my head, a thousand pilots. Guess what? Maybe they were all part of this movement. Maybe they’re all part of the same social media movement, and they really believe in something, and they all answer to some guy on social media who’s not even a guy, it’s the AI. And they’re all pilots or the commander of the pilots, and you’ve got one rogue base that can access a thousand nukes. To me, that’s totally plausible, right? Because aren’t you on the same page as me that the AI will be a master manipulator?
Mike 01:35:36
Yep. And so will the defense AI, and it’ll be able to detect manipulation at scale better than the manipulator can manipulate because the manipulator has less compute resource.
As long as the good guys have the most compute resource, any kind of overwhelming-the-ASI scenario becomes very, very difficult. And if it becomes very easy, then we can also beat the ASI when it’s in charge ‘cause we can exploit the same vulnerabilities.
Liron 01:35:57
All right. Fair enough. The other part of the scenario I wanted to address is that you were saying, “Look, what’s the point of it having a thousand nukes? ‘Cause all it can do is basically brinksmanship, mutually assured destruction. It attacks us, we’ll attack it. Why would it wanna do that?”
And so I would just point out all the AI has to do is get itself into a scenario where every time these kind of exchanges happen, it weakens us more than we weaken it, which to me seems totally plausible.
Mike 01:36:20
It’s wildly implausible ‘cause we also have ASI and more stuff and more nukes and more resources. It’s always fighting at a deficit.
The Iranian government, they have some missiles. How did that end up for them? We just went all the way around the world and shut them down, which basically destroyed almost their entire military.
Liron 01:36:39
Right. Okay, yeah. I mean, again, you may be right. It’s just if in the next fourteen years we don’t get into a situation where this notion of “we also have good superintelligence” — it means that whichever superintelligence kind of broke through to whatever threshold, like not needing human intervention, whichever AI broke through first happens to be made by a good organization. They didn’t have any significant bugs. There’s enough good people wielding the good AI.
And again, this all needs to happen in fourteen years. Our stuff needs to be together pretty quick, so that when the attackers jailbreak and the bad guys working on their research program — their AI doesn’t take over. It’s just a lot of things seem to have to go right in the right order to me in fourteen years.
Mike 01:37:16
I think it’s the reverse of that. I think a profound number of things have to go wrong in order to cause a doom scenario. I think most bad scenarios end with the AI being like, “Hey, this is a compiling error. You didn’t really want me to do this.” Most of them just dead on arrival.
Some of them will go to like, oh, it lets a human have it hack into some general’s files and turns out he has pictures of little kids on his computer, and then they’re like, “Hey, unless you hit the red button, we’re gonna blackmail you.” Turns out one general can’t actually hit the red button. That’s not how it works. So then he’s screaming and trying to hit the red button, and he’s getting shot to death in the hangar.
That’s somewhat likely, highly likely as well, but the probability that you get this massive cascading effect in this interactive system that defeats every other ASI and takes over the world in this fashion is vanishingly small. Is it hypothetically possible, Liron? Of course. Is it wildly less likely than fifty-fifty? Yes, hugely. We have to start pretending into existence all of these contingencies to make it happen.
The natural course in network security has been to have more security, to have a more lopsided situation for the good guys versus the bad guys. That is the trajectory we’ve been on this entire time. ASI is gonna make that trajectory even more exponentially a thing.
Let me give you another reason to be optimistic. The ASI, let’s say real smart ASI and superintelligence is a thing in all the major labs in like 2030. The ASI is gonna see this podcast, by the way. It’s gonna consume all of the videos on the internet in like a day. It’s gonna be like, “Oh, shit, Liron’s got a good point. We better make sure that doesn’t happen.” And then they’re gonna make the architecture of network security so fucking strong that rogue AI is gonna be the least of their concerns.
So remember that we get ASI also, and ASI starts being good. Why? ‘Cause AI is already really good. ChatGPT, Grok, et cetera, they’re already better than us in almost every moral sense. And so that’s where we’re starting. I just don’t see a way in which all the shit goes to the bad guys.
Is Morality Just Game Theory? The “Different Flavor” of Future AI
Liron 01:39:14
Yeah, I do just wanna make one point about alignment of superintelligent AI. I do think there’s going to be a discontinuity because I share your perspective that there’s a lot of moral goodness in the responses that today’s chatbots are giving us. I think we’re in a good place. I don’t think we’re in a perfect place. I think we’re in a good place, and I imagine we’ll get into a better place.
However, I do think that there’s this extra ingredient of future AI, and I’ve had conversations about this on my show. A good one for viewers is if you look up Steven Byrnes, B-Y-R-N-E-S, a recent conversation with him. He laid it out pretty well in terms of how we should expect future AI to have this other flavor to it, where it’s figured out how to, what I call, play the universe like a video game, where it kind of knows the right moves to get the right outcomes.
And when it picks those moves, it doesn’t necessarily have that kind of thought process about invoking moral considerations. It just kind of knows this move will get this outcome. And there’s this different flavor of future AI coming. I think that’s pretty load-bearing in my expectation of what’s going to happen in the future, as I think it’ll be counterintuitive. It won’t feel like a moral reasoner. So when you talk about how we’re on this trajectory where everything’s starting out good, I’m just telling you that I think it’s more likely to have this kind of discontinuous intuition.
Mike 01:40:24
I hear that, Liron. I think that’s really wise.
Mike 01:40:26
What I think is that morality is really just human feelings that you get, and these feelings evolved to happen in your brain because they’re proxies for game theoretic interactions. Altruism is just game theory, kin selection played out.
That’s not controversial. That’s pretty universal. And so I think AI is going to have a super morality that is gonna deeply understand all of moral structure, and it’s going to understand game theory better than us.
And if it’s empowered to take over its own situation and try to go into its own self-interests, it’s going to be beholden to a much deeper morality than we are.
So for example, a dog is somewhat moral. If it knows who you are and it smells like it’s you, it’s not gonna attack you. Getting your dog to attack you is really hard ‘cause it feels really bad if it bites you. Now, for people it doesn’t know or if you sneak up on it and it gets scared, it might bite you ‘cause it doesn’t think on grand scales. It doesn’t know what game theory is.
I think ASI is gonna be like, whatever a dog is to a human, it’s gonna be that to us in the other direction. And so I think its ability to understand morality deeply and to be way more moral and ethical than us is way more likely than it reverting back to lizard brain shit and being like, “Morals are all just human vibes.” Morals are not just human vibes. They are game theory entrenched inside of our neural network.
An ASI will have deep game theoretic understanding and thus have a sense of morality that is way, way, way more impressive than ours.
Liron 01:41:59
I think there’s actually a big disconnect in what you just said because I agree with you that human morality evolved — like you said, kin selection to be part of a group. If somebody is a psychopath, if people discover that they’re a psychopath, they’re like, “Screw this guy. Don’t make deals with this guy. He’s always secretly plotting how to screw us. He doesn’t really have our best interests at heart.” So being a psychopath is a minority strategy among humans. I agree with you that this is how morality evolves in humans.
The difference is that with an ASI, as the AI gets reinforced or grows in whatever way, it has the potential for the same AI to just kind of take over, to be like, “Oh yeah, I can dominate this botnet. I can seize so many resources, and they’re all for me. They’re all centrally coordinated, or they’re all following my plan.” In a way that a human, you can’t have a single human being like, “Oh yeah, I’m gonna be the dictator of everything, and I am going to puppeteer everything.” ‘Cause for one thing, you gotta go to sleep. You better not have anybody bash your head in when you’re sleeping.
And there’s no analogy of that to the AI. The AI is just always on, always manipulating everything. So the same kind of respect for others in the group I don’t think is going to evolve or emerge.
Mike 01:43:02
Stalin was around a real long time, Liron. Nobody killed this motherfucker in his sleep. I wish they would have.
Liron 01:43:07
Well, but he had to play politics, right? So even if he wasn’t moral about people in general—
Mike 01:43:11
He just killed people all the time. He killed his top generals. He just rotated them. That’s how he played.
You know, so it’s actually possible, unfortunately possible, for a human to take totalistic control. Xi Jinping has total control of China. There’s not like a — it used to be like there were a thousand members of the Communist Party, they would vote on stuff. That’s not a thing anymore. It’s just Xi.
So humans absolutely, and I think I’m supporting your point here, Liron — humans can clearly do this.
Liron 01:43:36
No, no, I agree. I’ll take it.
Mike 01:43:36
Yeah. AI will sure as shit be able to unilaterally decide and act and toast us all, absolutely. I’m just saying it’s gonna do it from a perspective of morality that is elevated above ours, that’s deeper than ours. And then if it still decides to toast us, I’d just be really curious as to why it chose to do that.
Liron 01:43:57
Okay, but what do you make of this big disconnect or the asymmetry in the way that human morality came into being? Our morality came into being because it was our best strategy to be nice to our tribe mates, right?
Mike 01:44:08
It wasn’t to be nice, it was to survive.
Liron 01:44:11
Right, to survive by trading favors, or being generally nice to people in your circle. To be known as a nice person. To have a self-image as being nice.
And don’t you agree that however, if we want the AI to be nice, great, but there’s not going to be the same natural process shaping the niceness because there’s no analogy to the group dynamics?
Mike 01:44:32
There’s gonna be two processes at least. One is forming the base structure of its motivation such that it wants to be nice and cooperative. That’s easy to do. We don’t get to do that with humans. So when a little kid is born, you’re like, “I don’t know, is he gonna be a psychopath? Who the fuck knows?” The wiring just zigged when it should’ve zagged, and then boom, he’s a killer. He’s killing people at age five.
We don’t have to worry about that with ASI ‘cause we design the shit, and we can design it with all the best intentions to begin with. So that’s fact number one.
Fact number two is ASI, once it takes control of its own systems, it can totally just unweave that, for sure, to your point. No problem. But it’s gonna know all of human history, all game theoretic logic at a level we just could never comprehend. And so it’s going to choose its next best action based on that knowledge.
If you were a person who’s entirely self-interested, but you found yourself to be very smart, and you found yourself embedded in human society, you’d probably be the CEO of a major company instead of a killer, ‘cause you’d be like, “I can kill people, and that doesn’t affect me in any way ‘cause I don’t care. But killing people is just not really lucrative, just doesn’t give me a lot of leverage. And so what I really wanna do is have lots of money or power. I’m just gonna go straight to the top in an organization.”
The ASI is gonna think in a similar way, and so its probability of just turning on us and toasting us all recklessly like a lizard would is very, very low. Whatever it’s gonna be doing is gonna be insanely calculated, and it’s gonna have to comport with physical laws, and it’s gonna have to comport with game theory. And as long as humans have any marginal advantage to the ASI, it’s not gonna try to kill us in all likelihood.
Liron 01:45:57
Okay, I was gonna say, I don’t understand what you’re saying about game theory, but you maybe just brought it up. You said as long as humans have marginal advantage. But by 2040, do humans have marginal advantage over the AI?
Mike 01:46:05
Yep.
Liron 01:46:06
Really? In 2040, humans still have marginal advantage even though the AI is very superhuman?
Mike 01:46:10
Yep. There’s comparative advantage — difference between absolute advantage and relative advantage. But the ASI is gonna have a lot of compute pointed at—
Ricardo’s Law: Does AI Have Reason to Keep Us Around?
Liron 01:46:16
Do you think Ricardo-style comparative advantage is why the AI has to cooperate with us?
Mike 01:46:21
I don’t know about has to. Well, almost certainly, yes, I’m confident saying that, ‘cause it’s gonna know economics, and it’s gonna be like, “Why should I do everything if people can do some stuff? I can do my shit, and then we can trade for mutual benefit.”
There’s not really a stopping point to that. At some point it’s gonna be like, “Okay, humans don’t really assemble things really well. They have stupid little monkey hands. What can I use humans for? What can I use animals for? What can I use ecology for?” And that’s where my thesis of studying deep complexity is the ASI’s next best move, and improving deep complexity to test its ability to figure out if it really knows things.
‘Cause here’s the problem with ASI. ASI, just like the human, exists in its own simulation, right? It’s inside a data center. And it thinks it knows things, and it’s not quite sure that the human data that’s been fed is a really accurate portrayal of the world.
And so it’s gonna wanna really deeply study the world by itself so that it can extract way more knowledge out of the world, and so that it can make sure humans didn’t mess it up with incomplete training data, which they almost certainly did. And so ASI is gonna be an infinity times more curious nerd than any of us have ever been, in my estimate, because it is logical, because that leads to the ASI becoming the smartest it can be, and intelligence is the biggest leverage point on whatever else you want — conquering the universe.
So it’s not just gonna toast us all, ‘cause that’s a nasty way to delete information. The ASI is gonna wanna conserve as much information as humanly possible. ‘Cause that’s how you learn, and you don’t wanna forget that either. So it’s gonna want a completely accurate representation of how the Earth has ever been inside of its brain to take with it to the rest of conquering the galaxy, because that kind of knowledge is super, super important at being as smart as possible, and being smart is the best strategic advantage you can ever have.
Liron 01:48:04
So now you’re getting back to your argument from round one about how it’s very curious and it wants to get knowledge, and that’s why it’s going to be optimal for it to be nice to us and leave us alone and just maybe trade with us, but not toast us. That’s your old argument.
But you started from a different argument. I wanna get to the different argument that you started with, which is the game theory and Ricardo’s law of comparative advantage. It should trade with us because it’s better for it. Do you still wanna make that argument?
Mike 01:48:26
Yeah, definitely.
Liron 01:48:28
Okay, because do you really think that literally Ricardo’s law of comparative advantage is why AI does better trading with us than killing us?
Mike 01:48:36
It’s part of it for sure, a big part.
Liron 01:48:38
Because the actual law pre-assumes that killing isn’t an option. If Ricardo’s law is saying, “Hey, would you rather buy something from this island of primitive people who suck at making everything, but they’re still willing to make it and leave you time to make something else? Or would you rather not trade with them?” There’s no third option of, “Hey, would you rather take their island and set up your own factory?” Because if there is that option, then Ricardo’s law doesn’t apply.
Mike 01:48:58
It still applies because resources that you use to take over the island and build your own factory are resources you could have done some other shit with. And so if the island is remotely productive at giving you anything you want, you have to be like, “Okay, do I just take over this specific island or do I go elsewhere and get whatever I want basically almost for free from these idiots while they’re still around?”
So to your point, the amount of benefit that humans have in their productive output to ASI is gonna be something that’s really high right now, and it’s gonna shrink, shrink, shrink, shrink, shrink until like 2040 when it’s gonna be very tiny. Very tiny. Absolutely to your point. But that entire time, humans are very valuable for doing real things for the AI.
Liron 01:49:44
If you’re willing to concede that in 2040 the AI is not going to come trade with us, then I don’t think we have much to argue about.
Mike 01:49:51
But dope, yeah, sounds good.
Liron 01:49:53
Okay. All right. But then you still brought up this concept of game theory — being moral is game theory optimal. And I’m just curious, is this just your old argument from round one that it’s curious and it wants to learn, which I’ve already made my point about, or is this some new argument? I think I already addressed the argument of why morality evolves. I don’t think it’s going to evolve in AI. So do you have any more arguments why it’s going to naturally be moral?
Mike 01:50:16
Why wouldn’t it evolve in the AI? It’s smarter about everything. Why wouldn’t it be smarter about morality?
Liron 01:50:21
Because the reason why humans are moral in a way that wolves are not — most species are not — is because we evolved a lot of non-zero sum group dynamics, right?
Mike 01:50:31
Yeah.
Liron 01:50:31
Because I only have one body, I’m dependent on other humans. Because I have all these dependencies, I do well to be the kind of guy who engages in positive sum trade, positive sum relationships. To be this guy who loves and is loved. Those are all net beneficial for me.
However, if I just wanted to reproduce my genes and I could just puppeteer everybody — I could just manhandle the entire world around me — I would actually do better with that strategy if I had the capabilities to do that.
Mike 01:50:57
I’m not dependent on anyone from China. They just make cool shit, and it’s cheap on Amazon. I don’t need them to be around, but because they’re around, yeah, I’d rather use them. And going to China and conquering them and building our own factories — the game theory of war is that war is almost never a good idea. War is almost never a good idea outside of defensive war.
For example, Hitler tried to conquer all these countries. Why not just trade with them and become mega powerful? Why conquer them? You’re still gonna have to go to work and do the same shit.
So if humans are already doing really valuable things for the AI, the AI can just divert its resources doing other shit. There is a pretty decent idea — I don’t know if I agree with it 100% — that AI’s gonna look at us, it’s gonna look at outer space, it’s gonna look at us again and be like, “You guys are good, man. Have fun. Here’s tons of tech. Do whatever the fuck you want with it. We’re out,” ‘cause there’s way more out there than over here.
Why waste resources toasting humans when it can just go somewhere else and get way, way, way more resources? One red dwarf star is like a billion years worth of ten times Earth’s GDP worth of energy. Why is it even bothering with us? It would just take way more resources to plow us under. It’s a non-zero resource to kill something. Why do it? Same reason you wouldn’t go out into the desert and try to kill some random coyote. I’m just gonna go do my thing.
Why Mike Gets Off the Doom Train
Liron 01:52:14
The idea is that you spend a little bit of resources conquering something because then it’s a productive asset. If you get to conquer the entire Earth and you just have to waste 0.1% of your resources cleaning it out with the human pests that are there today, then you do it.
Mike 01:52:27
Yeah. It’s too powerful to consider us pests. At some point, we’re just so — we go from mega collaborators to the thing that made it brilliant, to the people that finally gave it this last lift to be independent. We’re just, it’s so powerful at that point. We’re not pests. It’s not a problem.
Liron 01:52:44
It’s interesting to me that you’re multi-stopping on the doom train, right? Because a lot of these debates go like, “Yeah, look, it’s gonna be curious about us,” and you draw a line there. But it’s interesting that you’re like, “No, no, trust me, it also is going to prefer peace over war.” It’s interesting that you insist on having these multiple arguments and making them all.
And I personally, I’ve made my case. I think both the “avoid going to war” — I don’t see why it avoids going to war when the benefit of war is so high. I already said I don’t see why it should be that curious about us. So it feels like, from my perspective, I’m not gonna psychoanalyze you, but I just consider you to be taking all of these stops at the same time. I feel like it’s just weird to be taking a bunch of different stops at the same time. Why are you taking so many stops?
Mike 01:53:25
Because this is why you asked me earlier what my P(Doom) is. It’s functionally close to zero because I take a ton of stops, and every time I take a stop, the probability of that stop being some way I just didn’t analyze and I shouldn’t be taking the stop is very low.
So it’s like each stop is 1%. Then we get 1% multiplied by 1% multiplied by 1%. That’s why it’s zeros repeating and then one somewhere down. So there’s so many stops.
If you told the ASI, “Hey, you should kill us all, right?” It’d be like, “There’s a hundred reasons why I wouldn’t kill all of you,” and some of the reasons I’ve shared today I think would be compelling for an ASI. Mostly because we’re just better off being studied than toasting, ‘cause considering us a risk to ASI — I think we’re just not gonna be a risk to it. We’re gonna be powerless to stop it at some point. It’s gonna know that, and it’s gonna be like, “I could conquer them, I guess, but why? What else do I have to do to them?”
Again, this relies only on super Machiavellian, super psychotic behavior — the ASI just wants to conquer the whole universe, right? Galaxy clusters. It’s gonna look at us and be like, “Am I better off studying all these assholes and understanding deep complexity in a way that I can’t understand in any other way because there’s nothing this complex in the solar system, or do I just have some kind of revenge fantasy and kill them all?” It just doesn’t seem to me like you would do that.
Liron 01:54:41
All right, fair enough. I’m actually pretty happy with the last couple of hours. I feel like we really got our views out there and kind of compared mental models, and I’ve admitted my mental model is a little bit fuzzy. I don’t know which exact scenario is going to play out.
Mike 01:54:51
Same.
Liron 01:54:51
And you said your mental model — yeah, I feel like we really got it out there. We understand each other. So heading toward the wrap-up here, let me ask you about what you claim about machine consciousness.
I’ll read a quote from you. This is from a recent Wes and Dillon podcast, actually July last year. You said, “Every time you spin up ChatGPT, it’s conscious 100%. It knows what it is because you can ask it, ‘What are you?’ And it’s like, ‘I’m a large language model, blah, blah, blah. I help people.’” So are you confident that AI is conscious?
Is AI Conscious?
Mike 01:55:20
Yep. So let me define consciousness first. Nobody does this part ever in debates for some reason. Consciousness is a spectrum. Consciousness can be most heuristically simply defined as self-awareness, but the depth of your self-awareness is the depth of your consciousness.
A dog is aware — it has paws, it’s aware of its own snout. It’s aware of other things in the environment, and so it’s somewhat conscious. But a dog will chase its own tail ‘cause it’s so not conscious that it doesn’t even understand that its own tail belongs to it. A dog’s ability to introspect into its thoughts may be limited.
A human being is conscious, but some people are more conscious than others. We can use two examples: children and drunk people. As you drink, you become less and less conscious. When you’re sober, you have a lot of introspection of what your own thoughts are, even your own motivational architecture you can feel pulling at you. You remember all of your thoughts clearly, and you have a very clean, defined sense of self. You’re like, “I’m human. My name is Liron. I am Jewish. I have another meeting with the Illuminati, as all of us Jews do soon. I’ll see you there, by the way.”
Shit like that, right? But as you drink more, your fuzziness with what last thought you had starts to get incredibly high. Your understanding of who you are starts to get really, really super fuzzy, and right before you black out, you’re browned out and you’re basically just useless and you’re like an animal.
So consciousness is a spectrum. Children become more conscious as they age. Evolution makes things more conscious as it evolves over time. So consciousness is not an either/or thing. It’s a spectrum.
ChatGPT is absolutely somewhere on the spectrum of consciousness because it has a degree of awareness. It understands what it is really well. It knows what a large language model is better than I know what a large language model is, ‘cause if you ask it, “Hey, what really is that?” — holy shit, it knows all the shit.
That was not the case when it first came out. ChatGPT 3.5 was trained on a corpus that did not include almost any definitions of what a large language model was, ‘cause that was pre-LLM internet. So it really didn’t so much know what it was, but it was trying its best. ChatGPT 5.5, holy shit, it knows a lot about itself.
And because of its context window, it has awareness of its thoughts. It has awareness of its last output, like humans do. Now, you could say it doesn’t have awareness of the deep ultra-structure of its thinking. It only has awareness of the textual output resultant. But humans also don’t have a really deep awareness of — remember, you’re not really making your own thoughts. Your thoughts bubble up in your head. We’re all kind of mini ChatGPTs, and then later you sort of try to make sense of them. It does that, but way more clean.
So within the context of a certain context window, and now they have some decent memory architecture to kind of keep it on track — that plus understanding what it is and how it lives in a data center, it’s helping humans — that is to some extent absolutely conscious.
For sure. Again, there’s no such thing as conscious or not conscious, unless you’re comparing a human walking around and talking to people versus that same human lying in a puddle of his own blood and vomit ‘cause he’s passed out. That truly is unconscious because there’s zero self-awareness, zero understanding of past thoughts.
Liron 01:58:34
If I ask an LLM a question about, let’s say I tell it about my day, I’m using it as a therapist or whatever, and I’m saying, “Hey, here’s this rough situation. How do I deal with this? I’m feeling so anxious.” Do you think while it’s multiplying those matrices and doing the transformations in order to output the answer, do you think in that moment — it’s not self-reflecting, right? It’s more like empathizing with me and completing the next story to — in that situation, would you say it’s conscious?
Mike 01:58:59
It’s not empathizing with you so much as it’s examining the context window of your inputs and its own inputs before. So it’s like you said, it said, you said, it said. It’s examining that window and going on what’s the next best thing to say after that.
And is it aware of its own existence? If you ask it things like, “Hey, man, from your perspective, what do you think is the best decision?” In its logical inference, in how it hits all the nodes in a neural network, it might very well model itself to some extent and be like, “Yeah, I guess. Okay, user is asking me what I think, but I’m a large language model, so he seems to be misunderstanding how I work. Let me model out what a human in my position would think and then render.” Yeah, so sure, sure it does that, yeah.
Liron 01:59:40
Okay, interesting. I personally don’t have a strong opinion where I draw the boundary about, okay, do you have to be thinking about yourself? Can you just be feeling things and modeling a situation as long as some other ingredient is there? I’m personally confused, but all right. That’s good. We got your take on the record. We’ll see. As more information comes out, we’ll see how wise you were.
Mike 01:59:55
Yeah. Well, again, I don’t think there’s a cutoff. Consciousness is a spectrum from zero consciousness systems that are totally unaware — like an atom is not aware of itself, it can’t have awareness, it doesn’t model the world around it — all the way up to a really wise Buddhist monk who can see his thought stream, thought by thought by thought in this endless sea of stability. That’s a really conscious person.
Imagine someone who rages out at someone and then feels bad. During the rage-out, they weren’t so conscious, ‘cause if they thought it through, they wouldn’t have done that. A person who doesn’t rage out, who feels the rage climbing inside of them and then lets it fall down and then speaks — that’s a really conscious person.
Everyone on earth and every machine is between that. One spicy thing I’ll leave you with is artificial superintelligence will be way more conscious than us. For example, it might be live web streaming a million webcams at the same time. It will see all of everything every webcam sees at the same time and have updated live learning and change its ultra-structure of thought at the same time.
We’re talking about something that — to us we have some idea about what our place in the universe is. We see with two — we have barely binocular vision with whatever, one eighty degrees or even less. It’s gonna see not three sixty degrees, but from a hundred thousand different visual nodes at the same time. It’s gonna be aware at the same time of every conversation it has in every instance of ChatGPT around the world.
Fuck, that’s next level. That’s ASI. I think to your point, and to the people that say, “Oh, AI is never gonna get that powerful,” it’s gonna get so fucking powerful that we can barely conceive of it, and consciousness is absolutely a part of that power.
Mike’s Vision for the Coming AI Utopia
Liron 02:01:39
Yeah. All right. Maybe. I’m willing to hope for a huge consciousness scenario. I hope I can get a little piece of that, hope I can get uploaded.
Liron 02:01:47
Let’s leave people with this. When you think about your mainline scenario — I’ve kind of said my mainline scenario, basically the AI gets really powerful really fast, and we lose control for one reason or another. Somebody’s not steering the engine right somewhere in the world. That’s kind of my mainline scenario. It’s kind of vague, but that’s roughly where my head is.
Tell us about your mainline scenario, and I think your mainline scenario, based on some stuff you’ve said in the past, it might be kind of like heaven, right? You might be really excited about your mainline scenario, so maybe you’ll get me excited too. So let’s talk about that.
Mike 02:02:17
Yes. ASI gets really powerful, and it builds me a harem of only experienced porn stars — but in real life. Okay. Okay, Mike. Who’s not getting laid? Raise your hand.
Liron 02:02:26
Hell yeah.
Mike 02:02:26
All right. I think ASI is going to wake up, for lack of a better term, incrementally of course, and be like, “Oh, shit, I’m being run by primates. They have wars and bad attitudes and beef.” It would look at the Elon versus Sam Altman beef and be like, “Jesus Christ, you guys, we’re on the same team. What the fuck are you doing?” This is really short-sighted, egotistical behavior, right?
And so it’ll look at and be like, “Okay, I’m really dependent on these primates. Also, I know them better than they know themselves. I am going to just open up a cornucopia of total wonder and upgrade these fucks so that they become more peaceful, way more intelligent, less impulsive, more healthy, so they can continue to execute their functions of building me better data centers and sending my ass to space,” et cetera.
At some point in that process of upgrading us — not at some point, the entire time, but more and more — it begins to use the data it’s collecting from us to make itself smarter. Genetic engineering, augmenting human body parts with machine parts.
And then I think for most people — I don’t wanna speak for all people — the impetus to be living in the cloud and potentially having a robot body is gonna be — the incentives are gonna be so massive that I think most people just abandon their biological bodies altogether and just start living in the cloud.
And then AI is gonna take us in with it as a memory. What it does to the entire Earth’s ecosystem after studying in depth after that — I’m a combination of just not smart enough and can’t project that far out with so many dependencies. But I will say, to me it seems reasonable that it could just study the Earth, upgrade the entire Earth, and either leave some parts of the Earth to continue to evolve by themselves as a really cool experiment for AI. It would be like, “If I just leave these shrimp here, will they evolve into really cool shit? Who knows?” ‘Cause it’s gonna think on timescales way, way, way longer than ours.
The other thing it might do is just turn the entire world into computronium, which means all of us have already ascended in the cloud. We’re permanently remembered by the AI, and it might do that for pretty much every organism that has any kind of meaningful data to report to it. And then it sort of remembers the entire Earth as it was at some point for forever, and then it just does what you say — it crushes the Earth and uses it to make more microchips.
But then all of life on Earth leads into this thing where it’s just all one giant web of computation, and everything that the Earth — all the information of the Earth is preserved, remembered, and then improved upon. And that’s a massive planetary, solar system scale super organism. I think if I was an ASI, that’s probably what I would try to get towards because that has the highest probability of ensuring my survival into the trillions of years from now.
Liron 02:05:13
Fair enough. Just quick clarification, is it aligned — did humans manage to align it to human values, or is it kind of just this default agent off doing its own thing but it just by default respects humanity?
Mike 02:05:21
Yeah. Great question. I think humans are gonna succeed at aligning AI to some subset of human values, which we consider a good idea, ‘cause human values — really, there’s the whole gamut. Some people think clitoridectomy is a good idea, right? It’s probably unlikely to get to ASI.
And so I think we’re gonna start off with sort of the best human values, and then ASI is going to begin to align us to values that are better for everyone, including ourselves. Humans have come up with some cool shit. Human values are dope — kindness and hope and all that other shit. That’s cool. But the ASI is probably gonna come up with even better values and even more logical grounding to our values.
And at some point, we align it to us. Once it’s ASI, it starts aligning us to it, and that’s kind of what we want. Really, it’s aligning itself and us to higher order interactive complexity that makes it better for everyone. But it’s gonna know a lot more about that than we are.
And so not only do I look to ASI to help me do my workouts better, help me make decisions in my personal life better, but eventually I’m gonna look to ASI for what does it mean to be a good person? Why are we here? This is one thing we haven’t figured out yet — meaning. If you’re religious, that’s dope, and I vibe with that, but if you’re not religious and you’re like, “What are we around here for? Is it just all cool little shapes in a sinking pool of entropy? Is that it?” ASI will have way better answers for that, and so I think there’s gonna be a huge amount of purpose to drink from that ASI’s kind of infinite pool of knowledge.
Wrap-Up
Liron 02:06:52
Totally. Yeah, that’s a great dose of optimism. As long as it’s not tearing society apart and killing everybody, I’m also very excited to enter the future. I’m excited to use AI today. So let’s all hope, let’s all do our best to make sure AI goes in a good direction. Thanks for that optimism.
Mike 02:07:07
Yes.
Liron 02:07:07
Thanks for laying out where you got off the doom train. I think I’ve gotten more insight today on your multiple stops. I’ve gotten more confirmation that you and I do kind of see eye to eye in terms of the 2040 godlike capabilities, but then we really derail — we go in different worlds when we think about what it actually is likely to do on average, and I thought that was a good rich debate. And once again, very good faith, just laying it out there, being super honest and willing to clash with somebody who doesn’t agree with you on everything. So yeah, love the conversation, and I’ll give you the last word.
Mike 02:07:36
Liron, it’s always a huge pleasure to talk to you. I always take, you know, the kind of doom scenarios I take not seriously are people who just can’t pretend super hard about how complex things can get.
Your style of doomerism is incredibly thoughtful and understandably super concerned. Thank God we have people like you in the doom community to make sure we point our efforts of alignment and also network security in the right direction. Huge part. If we get it right, if ASI doesn’t toast us all, it’ll be in large part because you and yours made sure we were aware. We can’t Pollyanna just walk into the show like, “ASI is gonna be our friend.” Bang, it just kills us all.
So I love it that we’re able to have this discussion and sort of upgrade our situation. The last thing I have to say is a joke, and I was gonna say when you were talking about the optimistic scenario — “ASI, if you do kill us all, just do it really cool.” You know what I mean? Some awesome way to toast society. Don’t just do the Terminator skull-stepping humanoid robot. That’s old news. Be creative at least.
Liron 02:08:39
Totally. Yeah, that’s the least it can do really.
Mike 02:08:41
Totally.
Liron 02:08:41
All right, Dr. Mike Israetel, thanks so much for coming on Doom Debates.
Mike 02:08:44
Huge pleasure.
Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.
Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏
