Casey Muratori is one of the top programming educators on YouTube, with a four-decade career building videogames and popular videogame software.
We cover his positions on crypto, programming, AI’s impact on programming, and whether he takes the AI doom threat seriously.
Casey’s instinct is to break technology down like an engineer until he knows exactly how it’ll operate. Does that approach break down with the modern AI paradigm?
This is a ride on The Doom Train that lands on a wholly unexpected destination.
Watch on YouTube
Links
Casey’s podcast, Wading Through AI (with Demetri Spanos) —
Casey’s YouTube channel, Molly Rocket — https://www.youtube.com/@MollyRocket
@cmuratori (Casey Muratori) on X — https://x.com/cmuratori
My recent conversation with Steven Byrnes on the reinforcement-learning phase shift —
My older conversation with Jim Babcock on the most likely AI doom scenario —
Timestamps
00:00:00 — Cold Open
00:00:31 — Introducing Casey Muratori
00:02:07 — Recap of Crypto Series: Is Crypto Dead?
00:11:46 — On Programming & Why Coding Agents Don’t Faze Casey
00:18:51 — Coding Is a Craft: Drum Machine Example
00:36:24 — Pivoting to AI: Can We Forecast Its Capabilities?
00:43:36 — Debating if AI Can Acquire Taste
00:54:59 — What’s Your P(Doom)™?
01:04:11 — Even 0.1% X-Risk Is Unacceptable
01:07:34 — Modeling Uncertainty: Bayesianism vs “Tie Goes to the Catastrophe”
01:09:34 — Is This Pascal’s Wager?
01:17:12 — Casey’s (Cynical) Mainline Scenario
01:24:22 — Would You Un-Invent the Internet?
01:27:18 — Let’s Ride the Doom Train 🚂
01:27:51 — Can AI Become Ridiculously Powerful?
01:29:11 — Will AI Preserve Humanity (Orthogonality Thesis)?
01:32:26 — Will AI Seek Power (Instrumental Convergence)?
01:36:06 — Wrap-Up
Transcript
Cold Open
Casey Muratori 00:00:00
Well, let me ask you a question to see if you really believe what you’re saying.
Liron Shapira 00:00:03
Okay.
Casey 00:00:03
If you have a car, and you know that when you get in the car, there’s a 0.1% chance that you will crash the car. If crashing the car destroys all of humanity, the world, Earth is gone the instant you crash the car, are you gonna drive the car to work every day?
Liron 00:00:18
Okay, fair, fair enough. I wouldn’t.
Casey 00:00:20
Then why are we talking about this car analogy? You keep bringing it up.
Introducing Casey Muratori
Liron 00:00:31
Welcome to Doom Debates. My guest today, Casey Muratori, is a video game developer, writer, and one of the top programming educators on YouTube. He is responsible for developing software that’s been used inside thousands of games, like Destiny, Gears of War, and Age of Empires. He helped make The Witness, a game that’s been regarded as one of the top releases of the 2010s.
He posts programming interviews and insights on his channel, Molly Rocket, and is a regular guest on The Standup podcast, hosted by The Primeagen. He recently started Wading Through AI, a podcast he co-hosts with AI engineer Demetri Spanos. It’s a grounded, hype-free tour of AI and what it means for artists and programmers, but for an AI doomer like me, you know there’s a big topic I’m dying to hear about.
So I’m excited to talk to Casey and see what a world-class coder makes of the AI doom claims. Casey Muratori, welcome to Doom Debates.
Casey 00:01:24
Thank you. It’s great to be here.
Liron 00:01:26
You’ve had an amazing career in software and video game development. What do you consider to be the core skills that have led to your success?
Casey 00:01:33
Gosh, I don’t know. I mean, I guess I would say just being around a long time. That’s kind of the main thing for software development and games. A certain amount of intellectual curiosity, just learning about things you don’t know.
I got into it a long, long time ago. I started programming back in the early ‘80s when I was just a little kid, and ever since then, I’ve just liked working on games. I’ve liked learning things about how games are made, and eventually, you stick at it long enough, you get good enough to contribute. At least that’s the way I look at it.
Recap of Crypto Series: Is Crypto Dead?
Liron 00:02:08
Totally. I caught up with you in the crypto arc. This was a few years ago now. So you did the series where you tried to figure out what the heck is going on with crypto. You remember that?
Casey 00:02:17
I do, because I felt like — so obviously, the technical aspects of crypto are easy for me to understand because that is in my domain, meaning understanding technology. I read a lot of research papers. I understand the technology behind things like cryptocurrency.
But the technology is only one part of it, much like AI now. The technology, literally how the thing works, is something that I can understand because it’s within the domain I normally work with. But what people think it’s going to do, how it’s going to affect society, what they’re going to try to encourage other people to do with it — that is a completely different question.
And so with the crypto side of things, I looked at the technology and was like, this is not very interesting technology to me. I assess the technology based on what it does, and I was like, this is not how I would want this stuff to be. If we were designing a new way of doing payment processing, this is probably not how I’d do it.
And so what I wanted to do was just hear from people in that space about why they think this is good. And although it didn’t really change my mind on cryptocurrency — I didn’t come away from that thinking it was great or anything like that — I still really enjoyed doing that. I enjoy hearing from people. I enjoy hearing from people I disagree with. I enjoy hearing from people who have different opinions than me. So I’m happy I did it even though I would say, in general, I came out of it still going, yeah, I still don’t like this technology very much, but at least now I kind of understand the goals of the people who do and their mindset and so on, and I think that’s valuable.
Liron 00:04:09
Yeah. Well, I think you came out of that looking really good because now it’s been, what, four years already since that whole bubble? And the Bitcoin price is lower than it was three years ago. It’s had a negative return while the S&P and the NASDAQ have been on a tear. The crypto funds have all pivoted into AI funds or shut down altogether.
Technically, a16z is still launching new crypto funds because there’s some people who won’t give it up, who are still HODLing. But is it fair to conclude, stick a fork in it, the crypto dream is kind of dead?
Casey 00:04:41
Well, at the risk of talking too much about crypto on an AI podcast, I would say, as with anything, I think there’s an incredible amount of subtlety that needs to be appreciated. I think that certainly the hypesters of that era, they were wrong. But that’s not the same thing as saying there isn’t — technologies don’t simply die just because they had one failed push. Or in crypto, two failed pushes, you might say, because there was one kind of an earlier one than that.
And I think that there’s still underlying goals to cryptocurrency that do make sense, and I disagree with them about the best way to solve those things. But I don’t think that’s the same as saying that it’s just that someone was right and someone was wrong. I don’t see it as black and white as that. I wouldn’t use the phrase, “I came out of that looking good,” as if I had this specific point to make.
What I would say is I think maybe I was right about what I think are the ways in which society actually processes these kinds of transitions, and I think they were maybe a little bit more — they were thinking of it from a much more abstract standpoint that didn’t deal with some of the on-the-ground realities. But I don’t think that means that there isn’t a future for different kinds of digital payment. I don’t think that means that all of the dreams of cryptocurrency are dead, and we’ve proved it, and here is their gravestone. I think that would be premature.
So I’m not only not willing to put a fork in it, I also wouldn’t say that there’s nothing to learn from or no path forward. I didn’t see it that way. There are some hypesters, there are some bad actors, and those people are as bad as you think they were. But that’s not everybody, and I think it’s important to point that out.
Liron 00:06:42
Got it. Okay, so my own position — I don’t know if some viewers remember this, but back in 2022, before the ChatGPT breakthrough and before people like me really perked up and started being obsessed with AI, I was spending a lot of time pointing out how dumb crypto was, and I liked Casey’s channel during that time. Maybe he wasn’t as opinionated as I was.
But I just saw so many insane pitches — a16z and Chris Dixon being like, “Listen, you’re not gonna have to rent things in your life. You’re gonna be able to own things. You can finally own instead of just renting.” And I’m like, “Couldn’t I already own a house and a car? What specifically am I going to own?” He’s like, “Tokens, man. Tokens are going to revolutionize.”
So from my perspective, I thought it was interesting also on an epistemological level. It told me this meta insight about people’s knowledge state, specifically that I can basically debug why something is dumb and publish a simple proof, and yet the dumbness will continue, and billions of dollars will be thrown at it. And when I talk about a simple proof, my proof that crypto is dumb is this: blockchain is just a really elaborate decentralized double spend prevention. That’s all Satoshi’s paper was, and we’ve just never mapped decentralized double spend prevention to any useful applications. So everything else is just hype.
Casey 00:08:05
Yeah, I mean, I don’t disagree with the fundamental take that the technology itself is largely useless. I do think that blockchain is not particularly interesting for a number of reasons.
But again, it doesn’t mean that it’s the end. There’s no question in my mind that if you look at the arc of history, it tends to look like having systems which hit a peak and then sort of have a slow decline, and eventually are replaced by other systems.
And if you think about how do we deal with things like allocating human effort, which is largely what money is about, or scarce resources — I don’t know that the one we have today is the final one. It’s very possible — there’s so many things you could look at in our current system and say, “This is not optimal,” that it’s easy to imagine that at some point we will have some evolutionary process that replaces the current system with a better system, just like the system we have today replaced earlier systems that were less good.
And so the reason I say that I don’t like to think of cryptocurrency as just one thing that failed, and someone was right and someone was wrong — that is something you could say about specifics, but it’s not something that we might say about what the future of digital currency and cryptography’s involvement in currency is. I just don’t think it’s fair to count it entirely out. That’s what I would say. Although I obviously agree that I don’t think Bitcoin solves a real problem personally. But that’s separate from dismissing the entire thing or thinking that it might never evolve into something eventually.
Liron 00:10:07
Okay, well, I think you’ve just pulled a high ground maneuver. That’s a term coined by the late Scott Adams. So I’ll be like, “Wow, this whole idea—”
Casey 00:10:16
Well, again, and I—
Liron 00:10:16
Yeah, so decentralized ledgers. Yeah, go ahead.
Casey 00:10:19
This is why I say that I like talking to people about things and having an earnest conversation about what’s going on because I feel like if they’re not trying to debate you directly, if they’re actually having a conversation with you, then you can find out what the common goals are that you have or the common problems that you see.
And I definitely talk to some people who I felt had a very sane opinion about what the technology actually did and saw it as sort of a stepping stone towards things that we could do in the future. And so I don’t want to throw those people under the bus just because there are tons of people who I might throw under the bus who are also in that movement.
And that’s all. So it’s not really a high ground maneuver. And I don’t really disagree with your point from a technological standpoint. It’s just, I think there’s always something to be gained from thinking about what was trying to be done, and is there are avenues for pursuing that in the future that maybe look a little different from what we have currently. So that’s all.
Liron 00:11:33
Yeah, no, sounds good. Well, we’re—
Casey 00:11:35
Don’t wanna make the point too strong.
Liron 00:11:35
Soon we’re gonna get into the whole AI. No, no, you’re good. When we get into AI doom, we’ll see who can take the higher ground on the other person. I think we both have the ability to be gracious and open-minded.
On Programming & Why Coding Agents Don’t Faze Casey
Casey 00:11:47
Okay. Okay.
Liron 00:11:47
Let’s get a little background on AI. Obviously, you’ve got a rich career in software engineering, and you are somebody who really loves the craft. It’s kind of your identity, and you just like doing it for the sake of it, even though you do it and you get great results. But you also love the craft. Is that a good characterization?
Casey 00:11:58
Yeah, I mean, I think I said in previous interviews where people ask about this topic — obviously it’s the hot topic today, so everyone’s always asking about AI. What I’ve said many times is just that it’s sort of an accident that I get paid to do this. I could imagine a different world in which this was more like something I had to do for a hobby because it wasn’t big business.
So it’s kind of just a question of when I was born happened to land in a timeframe when people were paying a lot of money for people to do computer programming. And so this is also how I try to explain to people why I don’t really care about LLM code generation very much. It’s because I don’t really think of — I’m not someone who just does this to try and get the paycheck as fast as possible. And so I don’t really care that much about speeding up a task that I wanted to do.
So it’s less usually about me assessing their claims about whether it really would speed me up or not, and more just about — this is what I like doing. So if it has to become a hobby for me instead, that’s fine too, but it doesn’t really affect my opinion of this thing. I’ve likened it before to things like a drum machine. If I enjoyed playing the drums, what would be the point of a drum machine? Nothing, really.
Liron 00:13:26
Yeah, and I’ve heard you talk. You seem to be on the really high end of having 100% of your job be enjoyable to you or very close to it.
Casey 00:13:33
Mm-hmm.
Liron 00:13:33
Because I’ve also been a lifelong programmer, and if you caught me at age nine, I’d be writing BASIC programs on my computer and my console, just printing out numbers in the shape of a triangle, stuff like that. I got into programming really young, and I wasn’t getting paid to do that. So I got to do a career that was the same thing as my childhood hobby.
But after a few years, I did get to the point in my career where the majority of the code I was writing was like, “Ugh, I don’t like this part. This is not worth my time. I wish I had a junior programmer.” But the junior programmer would still cost six figures, so I might as well just do it myself. I don’t have that much work. So I felt like I was spending a lot of time doing stuff that I’d prefer to skip over.
And then in this year, 2026, I actually am able to skip over it, which feels pretty crazy. But from your perspective, it’s like, eh, why skip over anything?
Casey 00:14:19
Well, I guess I would say it a little bit differently than that. I guess what I would say is if you find that there’s something that you would prefer to skip over, that’s just a good example of the fact that that part of the job was done very poorly by someone.
And this is also one way that I’ve described the current use of LLMs — that people created a terrible software ecosystem and now are trying to solve the problem that they created by making something that deals with the terrible software ecosystem for them. And while I don’t disagree with that approach from a purely pragmatic standpoint — it’s like, well, we created a huge mess that we don’t wanna clean up, so we’re just gonna paper it over with these LLMs — I certainly wouldn’t want anything to do with that.
I think the last thing you wanna do when you’ve created an absolute disaster is then to say that the solution to that disaster is to build another pile on top of it. Which is basically what LLM-created code to deal with the garbage human code that we created beforehand is. That does not sound like a solution to me at all.
And so that part I would state a little bit more strongly against. That is separate from “I’m using an LLM to create code, period,” which means I’m doing it to create code that I would have enjoyed writing by hand instead. And so what I’m saying is that the latter is the one that I think a lot of people now just — they maybe never liked programming at all, for all I know. Because they would rather just have the LLM create code for them, even if it was something that is not what I would describe as janitorial work or whatever the thing is that you were describing.
Liron 00:16:00
Right. There’s this idea of inherent versus incidental complexity, or — what’s your preferred — essential versus accidental complexity?
Casey 00:16:10
Yeah.
Liron 00:16:11
The way you’re framing things, you’re kind of saying all complexity must be incidental complexity, but don’t you think there’s also plenty of essential complexity which is just annoying to deal with?
Casey 00:16:22
So I guess what I would say is if you don’t like dealing with inherent complexity, then you don’t like programming.
Liron 00:16:30
Yeah, well, I’ll give you an example. So today, I whipped up a dashboard, and it was just a random dashboard — “Hey, I would like to see things this way. Can you make me a user interface where you present, you pull the week-to-week data of this particular aggregation? Can you do that?” And if it weren’t for AI, maybe I would have manually run a couple SQL queries and been like, “Okay, that’s good enough for now.” I’m not gonna go make a whole dashboard complete with progress bars, and I had it color-coded. I’m like, “Oh, can you render it like this?”
So do you really think that me wading through the code to make that particular dashboard would have been a good use of time?
Casey 00:17:01
Well, again, I would ask why would it be anything more than just typing in a couple lines that are basically APIs that look exactly like what you described? Most people, when they type in the prompt for what they wanted their AI to do, that’s what the code normally looks like for a properly written library. If what you wanted was “show this button, show this button, when you push this button, do this,” that’s what the code should look like.
If your code instead looks like 3,000 lines, then that just meant you did a really crappy job on everything below it. And so to me, I not only don’t want that papered over, I actually want specifically myself to have to experience that because that lets me know, “Oh, crap, this library sucks. It’s massively overcomplicated. I should go make a much better version of that thing first before I go forwards,” because otherwise all I’m doing is multiplying the size and complexity of the layer above it because the underlying abstraction was bad.
And so to me, none of that holds water, what you’re saying. Inherent complexity to me is the simplex algorithm. Or a linear complementarity problem. That is an example of inherent complexity because this is the only known way to do this kind of optimization.
And if the answer is, “I don’t want to do that,” then I would say that person must not like programming. Because to me that is programming — it’s learning about this type of problem, figuring out the ways in which we can solve it, and then making a nice, minimal program that does this thing well and efficiently. To me, that is what programming is.
And so suggesting that I would want a machine to do it just means I don’t want to do it. Which again, is fine if all you want is the end product. But if you’re talking to someone who is literally telling you that they like this procedure, hopefully you can see why having an LLM do the part they wanted to do is not a very compelling argument to them. You’re telling them to use a drum machine because it’s faster than learning to play the drums, but they’re telling you that what they want to do is play the drums.
Coding Is a Craft: Drum Machine Example
Liron 00:19:11
I totally understand what you’re saying, and I’ve internalized that message pretty young. When I was a cocky 21-year-old junior engineer, I remember my LinkedIn bio used to say something like, “I don’t write code for machines. I just write the most expressive possible code for other people and for myself to read, and it’s just a bonus that the machines can read it.” Because code — good code is just intent. So I know where you’re coming from. That, again, code should be expressive. But—
Casey 00:19:37
Maybe I disagree with that too, but okay.
Liron 00:19:38
Oh, yeah? Okay, how so?
Casey 00:19:40
Well, to me, programming is intimately tied to the machine. The idea of programming is telling a machine what to do. You’re trying to have a minimal description that maps to the minimal set of things the machine does to solve that problem. So those things are intimately coupled and they can’t really be separated, to me.
Liron 00:20:02
Yeah. So I guess there’s this concept of programming by spec document. And that’s kind of what I’ve moved to, and plus I have the additional bonus that I don’t even really write the spec document. It’s more like I just write a little kernel of the spec document, and then the spec document appears before me, and then I just press yes or no, and most often it’s yes.
So do you really think that this kind of spec language, which is human English — do you really think that machines — there is some machine-readable language, and when I say machine-readable, I mean using a compiler with an old-style, grammatically parsable syntax, much more constrained than English. Do you really think there’s a one-to-one mapping where everything I wanna express as a spec document has a programming language representation that’s as expressive to a human?
Casey 00:20:50
So I guess I’m not sure I understand the question. Again, I will say a spec document to me is not programming. A spec document is something that a manager does. If I wanna be a manager, then what you’re saying makes perfect sense. I’m gonna write a spec document. I’m gonna go to my clients or the customer. I’m gonna have discussions with them. I’m gonna say, “Hey, what sort of stuff do you need from this dashboard? What do you wanna see? What kind of stuff does it run on?”
Okay, I gather my requirements. I make a little document that says, “All right, here’s sort of the stuff that we need,” and then I hand it off to somebody and they make it happen. And that’s what you’re describing. If that’s what you like doing, again, great. Totally fine with me.
I may complain if the thing that comes out the other end, as it often does today with LLMs, is really buggy or slow or whatever. So I may complain about the current quality level of that thing, but that can be improved. As AI coding improves, that hopefully gets better. Maybe it doesn’t, and then we’re all in trouble.
But assuming that it continues to get better, that will get solved at some point. For someone who likes to program, meaning I like to make the machine do things by telling the machine what to do, literally telling the machine what the machine should do — none of that is interesting to me. I have literally no interest whatsoever in writing a spec document for anything. I’ve never written one in my life, I don’t think, and I don’t intend to start.
Liron 00:22:25
You say you’ve never written a spec doc, but you haven’t written assembly, correct?
Casey 00:22:29
All the time. Yeah. I wrote assembly last week.
Liron 00:22:33
Okay, okay. But most of your programming isn’t assembly, right?
Casey 00:22:36
I don’t mostly program in assembly, but I know the assembly for everything. So when I — I always am in the debugger with the assembly view up, and I often am one of the people who complains because I’m like, “The compiler did not generate what it should have generated here.”
So the only thing that I look at in those standpoints is the same kind of thing that I’m looking for when I create a library. I want the thing that I type in to map directly to something that I know what it will be before I type it in. I never want it filled in with things I didn’t anticipate.
I’ve talked about this with Demetri on the podcast. It’s why I don’t think AI currently is a tool, quote unquote, because tools to me — or at least my definition of a tool, people can define tool however they want, it’s an ambiguous term — my definition of a good tool is one that I know exactly what it will do. It just does that thing faster. There’s no ambiguity. So when I say, “Do this,” I will get exactly this behavior out. That’s what I would prefer the most.
And the degree to which I think that’s true extends so far down the stack. If you gave me the option of having a completely in-order processor, for example, I would take that. I would take one that I could microcontrol.
I would prefer that sort of thing to one that has more stochastic behavior. And if you look at the kinds of things I like to do in programming, they’re about avoiding those things, avoiding things like, “Oh, I don’t know. Sometimes there’s this frame hitch,” or, “I don’t know why the computer did that thing, but it’s rare, so I don’t care about it.” I’m not that guy. I’m the guy who’s like, “Hmm, let’s go investigate why there is this little anomaly. I want to know exactly why it happened.”
Liron 00:24:23
Exactly. Which is why you have a great channel. Do you avoid JavaScript because the compilation is so non-deterministic or rather just complicated?
Casey 00:24:32
I mean, I wouldn’t say that I avoid it because of that. I don’t really use JavaScript, but it’s less of a conscious decision and more that I just have avoided web development in general. It doesn’t really come up that much.
I’m happy to program in JavaScript if I have to to get something done, but by the same token, if I was in that space a lot, I would have less concerns about using something like an LLM because to me, at that point, I’m not really doing the kind of — if we’re talking about literally just programming JavaScript, you certainly can do things if you know that, “Okay, this is gonna be JavaScript run through V8.” I certainly could predict what it was going to do.
But if we’re talking about the broader scope of, “I’m using JavaScript to write some code that does a Stripe transaction,” then to me, I’m not really programming. To me, that is basically the same as using an LLM or writing the spec document because it’s so grotesquely poorly designed, and just — it’s a mess. There is no possibility of me bringing order to that chaos. It’s just a lost cause.
And so when I look at something like that, I’m like, yeah, if this was my job all day, absolutely. Because there’s nothing I’m going to do that’s gonna be any different than effectively the same process fed through English to an LLM. It’s that far removed and that divorced from the act of programming that I wouldn’t bother drawing a line myself about that. I just fortunately don’t have to do that kind of code very often.
Liron 00:26:17
Interesting. I was gonna try to trap you in the argument. I was gonna try to say, “Do you really hate compilers?” If you’re used to compilers and you’re used to high-level languages—
Casey 00:26:24
Yes.
Liron 00:26:24
—this is just another high-level language. But you bit the bullet and you’re like, “No, I always unpack what my compiler is doing.”
Casey 00:26:31
I do hate compilers. That’s strong. It’s not really true.
I struggle with compilers a lot more today than I used to, and the reason for that is that it’s common to be going through a compiler where the series of transformations it will do to the code are less predictable.
And there’s many reasons for that. One is that they’ve decided to try to squeeze every last little ounce of performance out of what they can to try and take various benchmarks and get better on them. So just kind of the march of progress, adding more things and more things to try and do.
But the other one is they’re revved very frequently now. And so one thing that I will find is that a compiler will generate a particular piece of code on this particular point release, and then in some other point release, it will generate a different thing for that same piece of code, and the different thing is very wrong. And that annoys me a lot because I want guarantees at this level where I can know that once I’ve inspected something and it’s being done properly, that I don’t have to look at it again.
And so yeah, compilers do piss me off a lot these days. I don’t get mad at the developers of this because they have a very hard problem. And also what they’re trying to do is, in a sense, noble. They’re trying to improve the generation of code over time.
But if it were up to me and compilers were somehow written only for me, which is never gonna happen because I’m in the vast minority of people, I would much prefer a far more direct idea of what compilation is. Because I know what I want to tell the compiler to do, and I know what I want it to generate for assembly language code.
So I would much prefer a direct translation layer. Much like assembly language is direct translation, I would prefer a higher-level language that’s basically still direct translation, and I’ll go ahead and in that high-level language, I will mark up for you how I wanted you to produce this. I don’t want you to perform the transformations of your own accord.
And I think I’m in relatively good company there. If you go back and read any of the forefathers of computer science — your Tony Hoare, your Dijkstra, your Knuth — all those people. If you looked at what they talked about sort of getting to eventually, it was not, “Oh, we’ll have this compiler that magically optimizes everything, and we have no idea what it did.” It was always like, “Oh, there’ll be a high-level language that lets us specify what we wanted the good code to be, and then it will do sort of the menial work that is just the automated process we were going to do anyway.”
And so I feel like to me, that is what I would have preferred. It’s just not what we ended up with, for obvious reasons. It’s because people want the thing that you’re talking about. They wanted the spec-based programming, so that’s what high-level languages has consistently moved towards — not the “we’re trying to program the computer more directly.” For a number of reasons, possibly very good reasons, but again, remember, I’m coming at it from “I like telling the machine what to do.”
Liron 00:30:06
Yeah. Maybe a good boundary between what you personally like and dislike is you’re okay with the idea of compilation. So you’re okay with not writing ones and zeros that go straight into the microprocessor.
Casey 00:30:06
Yeah.
Liron 00:30:06
But you really want the compilation to be very simple and predictable, as opposed to compilation as constraint satisfaction, where you have to solve a hard puzzle to optimally compile your program.
Casey 00:30:18
The biggest thing I just want is predictability. What I want is when I type in this thing in the high-level language, that I know what it will turn into in machine code. And if I can’t do that, we have a problem.
Now, I’m willing to accept some wiggle room, because it’s like — there’s probably some amount of flexibility where I wouldn’t complain. Is this gonna be an add or a sub here? Because I could add negative one or I could subtract one. Those are two different instructions technically, or something like that. Maybe there are some things that we consider equivalent down at the hardware level more or less when we look at what is actually going on there. If we boil it down and we’re like, “In this circumstance, it doesn’t matter which one is used,” we could sort of take those out. So there is room. I’m willing to concede that the space is not literally a point.
But the difference between the relatively small area that I would be willing to draw the line around and the utterly massive one that a spec document goes around is really just too far to span in terms of me getting on board with the other kind of thing.
Liron 00:31:37
Understood.
Casey 00:31:37
Which again, just to reiterate, it’s not that I don’t see the value in it. I’m not suggesting other people can’t or shouldn’t do this. I’m just saying for me — not very much fun.
Liron 00:31:48
Okay, yeah. So with the AI future, I’m envisioning where we’re going is that the code will just be a big comment with no code. It’ll just be the comments. Or you can imagine comments with code next to it. And today, the computer skips the comments and it only executes the code.
But ironically, I think we’re moving into a future where the comments stay, and then maybe every day at the beginning of the day, the code just gets wiped out and regenerated according to the latest AI on what satisfies all of those comments and constraints the best. Which I know is your nightmare. That’s the bad kind of compiler. But I think that’s where we’re headed because it’s incredibly efficient.
Casey 00:32:27
Well, define efficient.
Liron 00:32:28
It’s efficient for the human to write.
Casey 00:32:30
Yes. So I guess what I would say is, if I were to categorize the current arc of software — and I would take AI out of the equation entirely because I don’t even think we need it to make this statement — I would say that at least for the past 25 years or so, the arc has been making less and less efficient software on almost every axis.
Meaning less efficient running on the computer, less efficient in terms of the amount of time it takes someone to actually do things because the wait times for the user have gotten incredibly large nowadays. People are waiting two, three seconds for things, pages to load, when if you looked at the old days before you started having everything go to a cloud, it would have just been instant even on very old machines.
I don’t see AI as really that material of a change in that. They are just the latest way in which people are sacrificing efficiency of the outcome of the software for the efficiency of the people writing the software.
Now, that doesn’t have to be the case. We could imagine a hypothetical improvement in AI where the AIs are focused a lot on trying to put back in the efficiency that the humans have forgotten about. “Hey, you gave me this spec document and boy, the way you kind of described this was really bad. I’m gonna re-architect it because I’m an AI that’s been trained better than you. And I’m going to go make this software much more efficient than it should’ve been. You said run this thing on the cloud. I know that that was dumb. I’m gonna run it locally.” Whatever it is.
I could imagine AI eventually reversing that trend. So I don’t want to sound like an AI pessimist here — that’s why I wanted to take AI out of that equation. I don’t think AI is to blame for the current terrible software experience I have when I use my computer. I think humans are to blame for that, and I could believe — it hasn’t happened yet — but I could believe that in the future AI could be part of the solution to that problem if people started taking that seriously.
Liron 00:34:55
I have the project I was telling you about, my small business. It was performing slowly for a long time because I was in the failure mode that you talked about — I was prioritizing my own time. I was trying to get things shipped, and as a result, it had memory leaks, it had slowness.
And I did point the AI at the website and I was like, “Okay, this is slow. This part loads slowly.” And it’s actually done massive optimizations, rewritten big parts of the code. It’s like, “Look, you can do one big SQL query instead of ten SQL queries in a row.” Stuff like that. And I’m a programmer, so I actually understand this stuff, but I wouldn’t allocate my own time to do it because it wouldn’t quite be worth it. And the AI does it and I can code review it, and I’m like, “This is great.” What do you think about—
Casey 00:35:37
My hope would be that eventually the AI is, in general, just going to output performant code to begin with, hopefully. And then that, I think, would be an improvement over what we had because most code that I saw being written was not performant. So if the AI can reverse that trend, that would be a nice positive contribution in my opinion because otherwise, other than that, it’s been a pretty unbroken 20-year streak of just things getting slower.
Liron 00:36:08
Yeah, I’m telling you, you’re gonna see some fast software. I can tell you firsthand, the performance of my software is far better than it ever would be if I didn’t have this AI assistant.
Casey 00:36:18
Well, that’s good. That’s a plus.
Pivoting to AI: Can We Forecast Its Capabilities?
Liron 00:36:21
Totally. Let’s move into the future. That’s really what we focus on here on Doom Debates — this idea of where are things going? Where is the human species going? These are big questions.
Casey 00:36:32
Yes.
Liron 00:36:32
So a good starting point is where do you see AI’s powers in 10 years? And for context, there are super forecasters out there, the different AI companies have been surveyed, and it does seem like the median prediction about people who one would think would know what they’re talking about — the median prediction is that sometime in the next five years likely, AI is going to be able to literally drop in and replace most white-collar workers at their jobs, even software engineers. That’s the likely prediction for five years from now. So with that as context, what’s your prediction?
Casey 00:37:05
Well, I don’t really have a prediction because it’s not my field. And I think in order to have an accurate prediction about what AI will or won’t do later as opposed to what it’s doing today, you kind of have to be working with it day to day. And I don’t mean working with it meaning using it. I mean working with it as in doing the actual training steps, coming up with new architectures for it.
Or even just — one of the things that they’re trying very hard to do is to try and close the loop so that it can improve itself faster than they can improve it. And so in order to know how close various improvements are, what their likelihood of failure is, what the cost will be to achieve them, all that sort of thing — I think if you’re not actually working on it literally, I don’t think your prediction’s worth anything, and that includes myself.
I just think it would literally be pulled out of a hat. And the reason I say that is because having worked on lots of technical problems in the past, you can never know — we have a saying: interpolation works and extrapolation doesn’t. If I know a couple points in time and you want me to tell you where you probably were in between those points, sometimes that’s okay. We probably won’t be too far off. If you want to know where we’ll be outside of that range, it’s really hard.
And sometimes things progress in a smooth way, so they just look like they have been looking, and then your extrapolation will be okay. Other times you hit unexpected accelerants or unexpected decelerants. Really hard problems you didn’t think were there, or really fortuitous things that you weren’t expecting to have happen serendipitously that make things go much further.
And so I think the reason why I don’t think people outside of the people literally working on frontier models day to day trying to get them improved really have a chance of getting an accurate prediction is just because the only people who could really have a good guess on those things — and even their guesses aren’t going to be great, but they’ll at least be better — are the people who can actually look at the specifics and try to think about what the specifics, what they’re going to hit. Each specific thing that we’re doing and all the things that we will need to do — they might have guesses about where the gotchas might be or where the serendipitous parts might be. So they’re at least in a better position than I would be.
Liron 00:39:55
I mean, if your methodology is deferring to these experts, and these experts are saying median prediction, yep, five years — I mean, that is the official line from OpenAI, Anthropic. Dario’s warning everybody about mass unemployment. So do you personally then assign, let’s say, a greater than 20% chance that they’re correct?
Casey 00:40:15
Well, I think a greater than 20% is not very high. So I would say, yeah, it could easily be a greater than 20%. I mean, let’s just say 20% since it’s a number that you threw out. Is there a 20% chance that it replaces all white-collar work? Absolutely, there is. And just to give you some context for why I would say that that wouldn’t surprise me.
So there’s two possible things that can happen other than just we have a completely smooth interpolation from where we are now. A completely smooth interpolation from where we are now would suggest that maybe five years from now won’t necessarily be all white-collar work. So maybe it’s a little bit aggressive if we’re just going for a completely smooth interpolation from where we are now and what we’ve seen in the past.
But with any new technology like this, we simply have not tried that hard yet to find all of the things about it and all of the ways it works and all of the new ideas. We’re just starting to throw all the horsepower we have at it. It’s been like 10 years of people trying this stuff. That’s it.
The possibility of a breakthrough that has not yet been harnessed is still high, in my opinion. That is an actual opinion that I have, not deferring to experts. I’m just talking about the nature of technologies.
Liron 00:41:43
Yes, I completely agree. There’s rich ground for breakthrough. Yeah, go ahead.
Casey 00:41:47
So if you’re talking about a field which obviously goes back further than 10 years, but having trillions of dollars poured onto it — that only goes back a few years. And having this many people spend this much time trying to make it work — that’s all very new. So in such an early time period, I think it would be very naive to assume that all of the breakthroughs have already been found.
And so is the time between now and 2030 or 2031 — I think it was five years? So 2031. Suggesting that we know for certain that no breakthroughs that give us a little discontinuous jump that’s enough to do white-collar work in general — that sounds very presumptuous to me. I would definitely not say that.
Is 20% the right number? Well, okay, then I go back to the thing where I say that would be something you would need an expert to quantify, and they’re still gonna be wrong. Nobody knows that percentage chance. But I certainly can’t give you a percentage chance because I simply don’t know where we’re at on all of the little things. Nobody outside of the various frontier labs necessarily knows all those things either. So you’re only looking at your window into it.
Casey 00:43:19
Trying to put a specific number on that would be hard even for somebody who’s on the DeepMind team or somebody who’s on the Claude Code team or whatever. It’d be hard even for them, but it’d be impossible for someone like me who isn’t spending all of their time thinking about these things.
Debating if AI Can Acquire Taste
Liron 00:43:36
Okay. I agree with you. Let’s look 100 years out or the end of the century — just to get a time where plenty of tech progress can happen, okay?
Casey 00:43:45
Okay.
Liron 00:43:45
Is your default expectation when that amount of time happens — enough time for plenty of iterations of progress and plenty of capital to be poured in and improvements building on improvements — by that point, what do you think the human brain will still be able to do that a software-based system won’t be able to do, if anything?
Casey 00:44:03
Well, I guess that’s kind of a difficult question. Let me ask for a clarification in your hypothetical. So you are assuming that it’s not Mad Max at that point? You’re assuming technological progress continues, or what?
Liron 00:44:20
Maybe it is. I mean, let’s just say it continues enough that nothing stops software development. The problem is I would argue that it becomes a hellscape once the tech becomes too advanced. But sometime in the next century, it works kind of like buying an option, right? You just have to hit your price. The price is an intelligence level. So at some point in the next century, will it hit an intelligence level that’s just very cleanly beyond humans — I would even argue beyond the whole human species? Does that seem like a plausible outcome to you?
Casey 00:44:50
I guess I would say that it’s a two-pronged question, and I think you are collapsing them into one prong. I would like to split them.
So there are two things that a human is doing, broadly speaking, when they are — well, I guess we could say doing anything, but working certainly — but even when they’re just doing recreational activities or hobbies.
One is taking a series of steps designed to produce a specific outcome. This would be things, as we sort of talked about with the drum machine versus drumming, where it’s okay, we are trying to do something where we are placing this drum beat at this particular place, right?
Those are things that I think eventually, given enough computing power, you can do. So whatever they are — we wanted you to frame this movie shot so that it followed Hitchcock’s rule or something like that, right?
Then there’s another thing, which is discovering something that a human finds pleasing, distasteful, emotional, upsetting, et cetera. That I have yet to see any progress in at all with AI. And the reason for that is because currently AIs are based on looking at what humans have previously described as being those things and then basing their understanding on that.
So given that, if you had not ever had the invention of rock and roll, let’s say — right? So kick on one, snare on three.
Liron 00:46:56
Mm-hmm.
Casey 00:46:56
How does an AI come up with the fact that that is pleasing to a human?
And the answer is, as far as I can tell, that it can’t. And the reason for that is we have never built anything that attempts to simulate how humans determine that something that’s never been seen before and never been done before is pleasing to them.
So that prong I don’t know, because no one’s tried it yet, and as far as I know, no one cares. Because in general, what most people are talking about AI doing these days on the creative side of things still has a human on the other side of it. So I just say generate lots of random drum beats, and I’ll pick one that I think sounds good. That’s an implicit assumption in all AI work today, right? That the detector for “did I like it” is still in the human, right?
So—
Liron 00:47:58
Okay.
Casey 00:47:58
The problem there is I don’t know where we’ll get on that. It’s not impossible, because if you assume that humans are a biological process that can be simulated in some way given enough compute, then in theory, eventually maybe we can make a thing that models that. That before anyone has ever in the history of time determined that rock and roll could even exist, when a stochastic process generates the drum beat of kick on one, snare on three, this other detector thing, which is simulating a human’s biological response, will go, “Okay, I liked that.”
That work I have not seen done. I’ve not seen anyone even try so far to do that thing. So that’s prong two. Prong one, I think in 100 years I could believe you. That’s we’ve got something that can do all of the things that a human brain is doing based on things we already knew that humans liked.
The second prong, I haven’t heard anyone attempt yet. So I could see going 100 years without anyone even ever bothering because literally you might just argue what’s the point? We’re the humans, we’re sitting around. This thing can generate a bunch of stuff, and I just listen to some of them and I pick one, and that’s the one I liked. Why do I need the thing to — right? You know.
Liron 00:49:48
I see your construction here. I asked about being better than human and you said, “Well, what about pleasing humans?” It’s one of those scenarios where you kind of either have to open the black box of the human brain or have sufficient insight into what makes humans tick, and maybe you have an advantage to that if you are human. Okay.
Casey 00:50:09
Yeah. Once you know something, once you know the rules — once you’ve determined for a particular thing, we’ve defined rock and roll as the space of things that is this, then the AI can generate things in that space. But until you knew that was a space — that’s kind of the problem I’m talking about, and so far we’ve needed humans to do that.
Liron 00:50:11
Right. Now in principle, would it be that shocking to you if 100 years of technology allows AI to just scan the structure of the brain, perform many experiments on humans, have big focus groups, and using all of that data in addition to super intelligent algorithms, can then do better than Rick Rubin?
Casey 00:50:32
Rick Rubin. Do I know who that is?
Liron 00:50:35
He’s a famous producer, right? He’s behind a lot of hits in different genres.
Casey 00:50:39
Oh, sure.
Liron 00:50:39
He has taste, right? So suddenly it can out-taste a really good human producer. Is that plausible to you?
Casey 00:50:49
So again, it depends on what you’re talking about. If what you mean by taste is producing something that people who are already listening to a particular genre of music will also listen to, then no — certainly I think an AI could do that. If what you’re talking about is creating a new genre of music that is unlike any previous genre of music, then no, I don’t think so. And I also don’t think Rick Rubin could do that. Not that I know who he is.
Casey 00:51:14
But another way to say it would be you can kind of look at this as just the AI from zero problem. So currently we have no examples of AI from zero that can do things... Well, I guess you might say we have no examples of AI from zero that can do things at all. They always have to have something put in from the human world. Even something like AlphaZero did not invent chess. It took the rules of chess and then created extremely good ways of playing it, right?
So if—
Liron 00:51:46
Okay.
Casey 00:51:46
If you were able to do the thing that I’m talking about, the AI could start from nothing. You have a blank computer somehow, and some code that doesn’t encode any preferences or any previous knowledge whatsoever. It’s just a bare algorithm of some kind, right?
And it goes from there to having language and thought and things like this. We don’t have any examples of that currently. It all has to be based on previous human stuff. And so the reason that I’m concerned that the second prong will not be taken care of is because we’re nowhere on that front right now. If we were somewhere on that front right now, then maybe I’d have more confidence in saying, “Sure, in 100 years, definitely.” But when you’re nowhere on something, that’s much harder.
Now, it’s hard to define exactly what this thing means. And maybe the answer is just eventually our scanning technology gets so good that we can just recreate a crap ton of human brains by scanning a bunch of humans and creating this simulator that simulates lots of them, and then we just use that as our thing. I could believe that that would work, but we don’t have that technology yet. So again, it’s kind of just a crapshoot. You’re asking me a Boolean: will we have the ability to scan and simulate a human brain in 100 years? I don’t know. I have no idea if we would have that or not. It’s certainly not on the same trajectory as current AI research is, though. I can tell you that for sure, because that’s not what AI currently does.
Liron 00:53:15
I don’t share your assessment that today we’re nowhere—
Casey 00:53:18
Okay.
Liron 00:53:18
—on the path toward having AI create new things. I think we’re definitely somewhere. I’ll agree we’re not all the way there for sure.
Casey 00:53:25
No, I said from nothing. From nothing.
Liron 00:53:28
Yeah, from nothing. I don’t even think — I mean, if you define nothing as the LLM doesn’t have any training data to bootstrap with—
Casey 00:53:35
Correct.
Liron 00:53:36
—then okay, you win. But—
Casey 00:53:37
But that’s what I’m literally—
Liron 00:53:37
—it does seem to me—
Casey 00:53:38
That is literally what I’m saying.
Liron 00:53:40
Okay, but that becomes trivial, right? I’m not allowed to train the LLM. I don’t even have an LLM.
Casey 00:53:45
Correct. And that’s why I’m saying that I think that second prong becomes incredibly difficult to satisfy, because if you required training data, you’re a priori saying that you can’t generate it the way humans generated it. The way I know that a human can do this is because humans did it. In order to predict that an AI could do it... You didn’t say anything.
Liron 00:54:07
I’m picking a line of questioning that I think is highly relevant to what — this is Doom Debates, right? And where I’m hoping to go with this, time permitting, is I do think that there’s a series of arguments that one can accept. I call it the doom train. And it points toward humanity getting wiped out by computers very soon.
So you brought up this idea of, “Hey, what if it didn’t have any training data? What if it started from zero?” And you’re starting to ask a different question, but now you’re getting off the path of questions that I concern myself with. I just look at questions—
Casey 00:54:35
Okay.
Liron 00:54:35
—that seem highly relevant toward whether we’re imminently doomed.
Casey 00:54:39
Oh, sure. Well, I don’t think we need to talk about any of the things that we’re talking about then if you’re just asking if we’re imminently doomed, right? I don’t see why it’s even relevant if we have AIs that can do all the things a human brain can do. We can be doomed so much earlier than that, that it doesn’t seem necessary to consider whether or not an AI could have created rock and roll without hearing rock and roll.
What’s Your P(Doom)™?
Liron 00:54:59
All right. Well, I guess this is a good time to ask you the big question of the show. You ready for it?
Casey 00:55:04
Okay. P(Doom). P(Doom), what’s your P(Doom)? What’s your P(Doom)? What’s your P(Doom)?
Liron 00:55:10
Casey Muratori, what is your P(Doom)?
Casey 00:55:14
I hate to disappoint you. As with the things that I said before, I don’t think I can actually put a probability on it because of the inability to know all of the specifics of what people are actually working on, right? If I went to all the frontier labs and they showed me everything they were doing, so I had full knowledge of everything they were doing — “Walk me through everything that you’re doing” —
I maybe could come out with a P(Doom), but it could easily be way higher percentage chance or way lower percentage chance based on what I saw there, right?
Liron 00:55:56
You know, you’re used to being able to look at the system and imagine how it compiles and imagine stepping through it. But as you know, the whole thing about deep learning is that it’s this really intense constraint satisfaction where the system emerges, where all the humans can say about it, at least until they do a lot more study on it — do science on it — but all they can say about it on day one when it emerges from training is, “Okay, well, it’s really good at predicting these tokens,” right? Or it scored really high—
Casey 00:56:23
Yeah.
Liron 00:56:23
—on the objective function, and it has a lot of innards that make it do these complex, pass these tests in a very complex way. So that breaks your usual way of analyzing things, right? So you can expect to not be able to analyze it.
Casey 00:56:36
Well, I think that’s what I’m saying. What I’m saying effectively is that my P(Doom) — so you’re asking me to put a percentage number on how doomed I think or how likely we are to be doomed, I guess, is how you would say it? What is the percentage chance that we get wiped out? Something like that, right?
Liron 00:57:02
Exactly, and I get your uncertainty, but I’m saying maybe the approach can be to reason from the fact that it’s a black box that can hit these complex tests. Maybe that still tells you something.
Casey 00:57:02
It doesn’t. So maybe I can frame it this way. Consider the things that would have to happen for AI research even to continue, right?
So in order for AI research to continue, a lot of things have to continue to go right. A nuclear war tomorrow — all the data centers are gone, the electrical grids are wiped out, the satellites are destroyed. That would set it back a hundred years easy. The hundred-year horizon that you’re talking about is just over in the blink of an eye.
And so when we talk about putting P(Doom) on something, what we’re by definition talking about is that, A, it was possible — meaning it is possible to build the AI that wipes out humanity. That’s a given. You have to accept that premise obviously, because if you didn’t think that we could build an AI that wiped out humanity, then P(Doom) is zero by default, right?
Liron 00:58:06
Right, which I wanted to ask you about that as a factor in the equation.
Casey 00:58:06
Right. But let’s say we accept that, because I guess I could make a trivial argument for that where I don’t see how it’s arguable we couldn’t. Literally, all we need to do is hook a thing up to the nuclear weapons, and it’s bad, so it launches them all and kills us, right? By accident. It’s not smart. It’s not doing anything. It’s just War Games. It’s the movie War Games from the ‘80s, right?
So we don’t need intelligence, forget decision-making, forget anything that we would have to actually argue about how good the thing is. We could just accidentally wipe ourselves out using computers. I don’t think that’s that ridiculous. It’s not P zero. It may not be high.
Liron 00:59:02
I think that’s a good point that human civilization is more fragile than I think some people realize. I agree with you there that it might only take a few nukes at least to set us back to the caveman era, and then maybe we can rebuild, but it’s gonna be tough.
Casey 00:59:02
Yeah, and you put a hundred years on it. You didn’t say for all eternity forward might we get there, right? And then you have to start going, “Well, okay, will it always happen that we destroy the capability before finishing it?” And you know.
Casey 00:59:16
My perspective on that is just if you wanted me to think that P(Doom) was particularly high, you also have to convince me that the probability that we won’t screw things up in some other catastrophic way earlier — that doesn’t completely wipe out humanity but does kind of halt technological progress — couldn’t occur.
And I guess what I’d say is, in my mind, the way I have to frame this problem is not so much let’s look at what the percentage chance is of building an AI that can wipe out humanity in isolation. It’s what does the horse race look like between building that AI and actually spending the resources to build that AI and having enough people on building that AI and having enough power and all this other stuff — versus the probability that everything just goes to crap for some other reason before that, right? It’s a horse race to me.
And that’s a hard horse race to evaluate.
Liron 01:00:24
Totally. So just to lay my cards out on the table, I do think nuclear threat is underrated. I think something like 1% a year of a huge catastrophe — at least a billion people dying.
Casey 01:00:33
Yeah.
Liron 01:00:33
I literally think there’s a roughly 1% chance per year that at least 1 billion people will die because of nuclear war, which is pretty crazy. But if you look at a 30-year period, nuclear war — we still have better than even odds of surviving. That’s not too bad.
Casey 01:00:52
Yeah.
Liron 01:00:52
And then bio-terror probably the same order of magnitude as nuclear, maybe even worse. Okay. But AI, I think, is coming up fast. I think it’s coming to be super intelligent in less than a decade, more likely than not. And I even think that the chance that once it exists in less than a decade, the chances are actually better than even that once it exists, it will run out of control and permanently disempower humanity. So even though you’re right that there’s other risks I’m worried about, AI seems really front and center in the doom equation for me.
Casey 01:01:21
I don’t know that I’d push back that hard on it for a couple reasons. Meaning I don’t think I’m gonna disagree with you particularly strongly here to the extent that I’m gonna disagree with you at all.
So I’ll say reason number one. I would ask — let me talk to the people who would disagree with you instead first. So let me just agree with you, but agree with you to the audience as opposed to you.
For people out there who think that what you just said is stupid, or they’re saying, “You just don’t understand,” or whatever, “You got your facts wrong,” whatever it is that they’re going to say —
Liron 01:02:00
Yeah.
Casey 01:02:02
I guess what I would have to ask is what is the benefit of proceeding otherwise? Meaning why wouldn’t you just, out of an abundance of caution, say, “Let’s take this concern very seriously”? Because the kinds of benefits that people are talking about getting from AI, we don’t need them in five years from now. Humanity doesn’t literally need to have superhuman intelligence in the next five years or the entire race is doomed.
So it’s unclear why you wouldn’t take the possibility of doom extremely seriously, because the only thing it could do is help ensure that it doesn’t happen, which seems like a good outcome, right?
Liron 01:02:54
So just to be clear, your actual position is that the doom risk is significant. And normally I would say significant is 5% plus, let’s say, in the next 20 years. I feel like you — it sounds like you’re giving it 5% plus in the next 20 years. I know you don’t tend to put numbers on things, but that’s a pretty low lower bound, no?
Casey 01:03:12
Well, I don’t know that I can put a number on it, but what I’m saying is what is the point of putting a number on it? Why wouldn’t we just assume that this is a very — that why wouldn’t we just assume that it could happen, and any percentage chance, 0.01% chance, is still enough because it’s complete annihilation?
I hate to quote the 1% doctrine here because I know that nobody likes Dick Cheney, and I don’t necessarily like Dick Cheney either. But he was rather right about this. If you have an incredibly catastrophic thing, it doesn’t really matter how low you put that percentage chance. Unless you think it’s literally impossible, then the only sane way to proceed is to be concerned about it because you can’t really — the only percentage chance you could put on it that should make you feel comfortable is zero, and I certainly can’t put zero on it. I can’t prove it won’t happen.
Even 0.1% X-Risk Is Unacceptable
Liron 01:04:12
Well, the number matters a lot, right? I mean, if the chance was one in a million and there’s discernible benefits, then those do start to outweigh the cost.
Casey 01:04:20
No. I don’t think that — if you were to tell me that there’s a one in a million chance that all of humanity is wiped out in the next few years, but here are some benefits, I’d be like, you’d never take that. All of humanity—
Liron 01:04:35
Well, one in a million is literally the death risk of every ride in a car that you take.
Casey 01:04:37
Sure, but it’s only one human.
Liron 01:04:37
Okay. That is true, but the benefits also will distribute to a large amount of humans, right? Just like the risks do.
Casey 01:04:43
But we’re still here, right? What you’re talking about is a one in a million chance that literally we are gone. No one should take that chance.
Liron 01:04:58
Right. I mean, I agree, I agree that it’s a bad idea to—
Casey 01:04:58
Yeah.
Liron 01:04:58
—to have all of humanity take a car ride together. But the thing is we just—
Casey 01:05:01
Yeah.
Liron 01:05:01
—already are, right? It’s like—
Casey 01:05:02
No, we’re not.
Liron 01:05:02
—every day we live in the shadow of nuclear war and biorisk and asteroids. We’re already all on a car ride constantly.
Casey 01:05:10
Well, okay. But I guess the way I would say that is those things are incredibly highly regulated, and we all take them exactly as seriously as I’m suggesting.
Nobody goes —
Liron 01:05:22
Um—
Casey 01:05:22
No one looks at nuclear war and goes, “Well, we don’t need to do anything about that.” Citizens aren’t given access to nuclear weapons. Private companies aren’t given access to nuclear weapons. They’re highly controlled by the government under government regulation, and only DOD people are allowed to do anything with them. That is completely different from how we’re approaching AI. If we were approaching AI like nuclear weapons, I wouldn’t be that concerned, right?
Liron 01:05:45
But the reason we care so much about them is because we know with high confidence that they are at least catastrophically dangerous, if not existentially. And so that’s what I’m saying — I do think that it’s worth acknowledging or grappling with the idea that AI is also catastrophically dangerous. I’m just wondering if you personally are open to that claim instead of just going the 1% route. Can you just be open that it actually likely is catastrophically dangerous?
Casey 01:06:14
No, because in my opinion I can’t assess that, right? So when I look at things, I feel like I don’t — I don’t feel like there is a value in engaging in motivated reasoning to arrive at a false confidence.
If I look at something and I go, “I cannot assess this,” then that does not mean that there is a percentage chance or something — what it means is I can’t assess it. And so the way I like to make decisions in the face of complete uncertainty, meaning I have no way of assigning any probability to this whatsoever that I can have any confidence in, is to say, “Okay. What are the results of proceeding under the two different assumptions I would make were I able to make an assessment of it,” right?
If I was to assess it at some percentage risk — 5%, 1%, whatever it is — versus 0%, those are the two Booleans to me. Meaning if this were completely safe, how would I proceed? If this might have some chance of being catastrophic, how would I proceed?
And to me, I see essentially zero downsides in going with the “let’s assume that there is a percentage chance that it’s catastrophic.”
Modeling Uncertainty: Bayesianism vs “Tie Goes to the Catastrophe”
Liron 01:07:34
To clarify, would you be opposed to doing an expected value calculation substituting 50% of the probability of good, 50% probability of bad? Would that be consistent with your recommended decision theory here?
Casey 01:07:46
No, because to me, I think that a catastrophic risk to humanity is not something that can be weighed against something that doesn’t have an equal weight there.
Liron 01:07:59
No, for sure. But when you’re saying consider both possibilities, I’m just clarifying — is that isomorphic to a Bayesian who just plugs in 50/50?
Casey 01:08:06
No. I guess what I’m saying is “tie goes to the catastrophe” might be the right way to say it. If I cannot assess what is likely to occur, then that by default means I can’t rule out the catastrophic possibility. Therefore, the next thing I’m going to consider is, if I were to proceed assuming the catastrophe will happen without my intervention, which is what you would do if you can’t rule it out, what are the downsides to that?
AI proponents think there are huge downsides to that. I don’t think there are very many downsides to that actually, or at least I don’t think the downsides to that are particularly important relative to avoiding a catastrophe, which if I can’t rule it out, is still on the table. So decision theory isn’t really what I would call it. It’s being unable to assess because I don’t have the information, right? And I’m not sure anyone really does, but I certainly could have more information to make this decision on, in which case I might change my mind.
Liron 01:09:12
Well, yeah, I think when you make decisions, you have to distinguish between, “I can’t rule out this really low probability, obviously less than 1% thing — I can’t rule that out,” and, “I can’t rule out this thing that seems like it has a tens of percentage probability.” I think those are very different in practice cases of things you can’t rule out.
Is This Pascal’s Wager?
Casey 01:09:33
They’re not when it’s the end of the human race.
Liron 01:09:33
Yeah, I would argue it is because there’s a slippery slope to Pascal’s wager, right? You don’t literally—
Casey 01:09:39
Yeah.
Liron 01:09:39
—believe in Pascal’s wager as an argument for Christianity, do you?
Casey 01:09:42
Well, I guess what I would say is, you don’t believe me that if something is going to end the human race, that if I can’t rule it out, I should take it seriously.
Then I’m not really sure what to say. I don’t feel like that’s equivalent to Pascal’s wager because if I’m saying I can’t rule it out, I’m literally saying that. I’m not talking about I can’t rule out the possibility that the world is held up on the back of a giant tortoise that swims around because I haven’t been on a spaceship to look at it. I’m talking about an actual me thinking about this technology, knowing what I know about it, I can’t rule it out. That’s different than putting a percentage chance on it because putting a percentage chance on it is doing an actual calculation that gives you a number, and I’m not doing that, and I can’t do that. And anyone who says they’re doing that, I don’t know where they’re getting that number.
Liron 01:10:38
The load-bearing part of Pascal’s wager is can you rule out the Christian God or no?
Casey 01:10:43
Again, that’s just talking about an abstract thing, right? That’s saying, of all the things that might exist in the world, can you prove that one of them doesn’t exist? We’re not talking about that. What we’re saying is, here is an actual thing that I have analyzed. I’ve analyzed it. I’m not considering hypothetically all the things in the universe. I’m talking about I’ve analyzed this one thing, and I cannot rule this thing out even after knowing how the thing works. That to me means that, although I can’t put a percentage chance on it, that doesn’t imply that the percentage is zero, right?
Liron 01:11:25
Okay. I think you’ll find that the way you make decisions will have a lot in common with the way a Bayesian with tens of percents of probability makes decisions, as opposed to having a lot in common with a Bayesian who gives a 0.1% probability to something, even though it can be said that the Bayesian with the 0.1% probability on something is also not able to rule it out.
Casey 01:11:45
Well, you’re welcome to try to assign a Bayesian to it. I guess all I would say is that if I was to think about trying to encode everything that I’m saying in terms of some kind of Bayesian node graph sort of a thing, I obviously haven’t done that, so I clearly can’t be making my decision that way actually. I’m definitely not really doing that. I’m doing some other process. If you want to try to put it into those terms, I guess if it really makes you feel comfortable, that’s fine. But I don’t understand what the point would be, right?
Liron 01:12:27
Yeah, it makes me feel extremely comfortable to put things in Bayesian terms, yeah.
Casey 01:12:27
Okay. Then sure. I guess I’m not sure what that would change, but go for it.
Liron 01:12:33
I mean, because you get to plug it into decision theory, right? That talks about making decisions by multiplying probabilities of outcomes with the value of those outcomes.
Casey 01:12:41
But that implies that you know something about those probabilities, which I’m saying we don’t.
Liron 01:12:45
Well, knowing a probability is actually pretty easy. I claim that deep within you, you actually know that the probability is more than 5%. There’s some part of you that thinks that — based on what I’ve observed so far, I may be wrong, obviously you know you. But there are people like Robin Hanson and Yann LeCun who are openly saying, “Hey, the probability of AI doom is less than 0.1%.” And I think there’s some part of you who’s looking at those people and being like, “Don’t you think you’re underestimating...” Is that fair?
Casey 01:13:14
No, I think it might be that 0.1% is the right number to put on it, and what I’m saying is 0.1% is unacceptable.
Liron 01:13:22
All right. Fair enough. So I’m in the weird position then of — I disagree with the part of your logic that says that 0.1% is unacceptable. If I thought — because for me, it’s okay great, 0.1%. Again, I feel like I’m getting in a car, and I’m taking a journey which is giving me plenty of benefits.
So maybe the other part of the disagreement between me and you is you might want to argue that the net benefit of AI in the next year or the next 10 years while we’re building it — you might want to argue that the net near-term benefit is negative, in which case maybe your 0.1% risk avoidance holds better than that. Would you make that argument that there’s no immediate benefit on net?
Casey 01:13:58
Well, let me ask you a question to see if you really believe what you’re saying. If you have a car, and you know that when you get in the car, there’s a 0.1% chance that you will crash the car.
If crashing the car destroys all of humanity — the world, Earth is gone the instant you crash the car — are you gonna drive the car to work every day?
Liron 01:14:19
If there’s a 0.1%—
Casey 01:14:21
Yeah.
Liron 01:14:21
—and it’s just me getting in the car, and the only benefit is that one journey — okay, fair enough, I wouldn’t. But—
Casey 01:14:27
Then why are we talking about this car analogy? You keep bringing it up, but it’s completely wrong. Right?
Liron 01:14:31
But if the journey leads to a lifetime supply of free food for every human on Earth at the end of the journey, then I might consider it. I probably wouldn’t. One in a thousand is probably too expensive for that. But if it leads to heaven for everybody on Earth, at that point I’d be like, “Okay, well, you just have to make it to the end of this journey once, then you get heaven for all time.” That actually might be a pretty good deal.
Casey 01:14:56
All right, let me keep going on this train then. If you’re saying—
Liron 01:14:56
Okay.
Casey 01:14:56
—that you get in the car, you have a 0.1% chance that’s the end of the human race. But if you make it to the end of the journey, whatever you mean by heaven, humanity all goes to heaven somehow.
So you feel like it was okay for you to make that chance? In other words, you’re putting everyone in humanity’s lives on the line, and you’re fine with that. You didn’t have any compulsion. You weren’t like, “I should probably ask some friends.”
Liron 01:15:25
No, no, I definitely have that compulsion, right? And if somebody else did that, I’d be like, “Wow, that person is really a jerk for taking it upon themselves to make the decision.”
Casey 01:15:33
Yep.
Liron 01:15:33
Now, if they were put in a tough spot where they only had a limited time to make the decision, I think they should err on the side of not rocking the boat, right? Because that’s such a consequential decision, and they should be like, “Man, I shouldn’t rock the boat.” I definitely—
Casey 01:15:45
Yeah.
Liron 01:15:45
—think that’s a good deontological intuition that I would share as well. But after humanity all convenes and considers it, I think I might not be in the minority here to be like, “Oh, permanent heaven? That’s a pretty good deal for a 0.1% risk.”
Casey 01:15:59
And I guess what I would say is that, as an example, is something that would make me feel better about — if we move the analogy back to AI. If all of humanity was making this decision collectively, then it’s at least that was done in what I would consider a more proper and thorough way, right?
Currently, it’s not. It’s much more like the actual analogy you described, which is that some people are just deciding to take this car ride for all of humanity. And to me, that seems completely unacceptable at 0.1%, which is why I keep saying that you want me to ascribe a higher number to it, I don’t see why it would matter. Even at 0.1%, it’s a ridiculous thing to think about risking all of humanity at a 0.1% chance.
That seems — at a bare minimum, at 0.1%, you have to have all of humanity’s buy-in to do something that risks all of their lives.
Liron 01:16:52
Okay, well, I agree that the buy-in aspect is a good aspect that we should definitely think about and that we’re not getting from the AI companies, correct?
Casey 01:16:59
Absolutely. Not only are you not getting any buy-in, you’re getting pushback, and they’re still doing it anyway, right? So humanity — a lot of humanity is actively telling them not to do this, and they don’t listen, right? So that’s the opposite of buy-in, if you will.
Casey’s (Cynical) Mainline Scenario
Liron 01:17:12
Yes. All right, I’m glad you brought that up. Okay, so you personally in your life feel a good deal of concern about AI potentially taking over the world and disempowering humanity?
Casey 01:17:22
I guess I wouldn’t say I feel very much concern about that particular — the fact that I can’t put a percentage on it, as we’ve talked about, is kind of saying that I’m not that concerned about it personally.
If I may, I feel like there are a lot of risks that AI brings with it that have nothing to do with complete doom. Even basic disregard for the way markets get disrupted, the IP theft that occurred — those sorts of things are harms that are happening right now. And so for me, I have a lot of concerns about AI companies about literally harms happening right now. So to me, those are more concerning than the hypothetical future thing in the sense that I would prefer these companies to be reined in now. I don’t need to wait to see if the artificial intelligence destroys the world. I’m already unhappy with what they’re doing, right?
Liron 01:18:20
What do you think is the most likely scenario that’s going to happen over the next 10 or 20 years because of AI?
Casey 01:18:26
I am a pessimist and I’m generally cynical. And that is why I probably don’t share your concerns at the percentages you share them. My feeling about what is the most likely outcome is AIs will get roughly good enough for the people who make them to make a lot of money, and then largely they will stagnate.
And the reason that I say that is because that seems to be what happens with basically every technology we make. As soon as people can extract all the money that they wanted from it, they don’t seem very concerned anymore about all of the things that they previously said they were going to do. And it’s not that there won’t still be people with those particular goals creating artificial super intelligence or whatever it is. It’s just they will no longer be the people who get to spend any of the money because the people who are spending the money will actually just be making this sort of rent extraction intelligence service that you kind of use. It’s designed to be used by everyone and produce results just good enough so that you pay for it, but not good enough that it disrupts the economy any more than necessary to extract the money that they wanted. That is my — because I am a pessimist and a cynic, that is where—
Liron 01:19:43
Mm.
Casey 01:19:43
—I expect us to go. Great news for you would mean it doesn’t destroy—
Liron 01:19:47
Right.
Casey 01:19:47
—the world. Bad news for people with a lot of hope for AI because I’m also saying I don’t think it’s gonna produce great results for humanity. But again, it’s a cynical, pessimistic take, and I acknowledge that.
Liron 01:19:59
Exactly. So if you have to pick one scenario that could happen — of course, there’s so many countless scenarios, it’s a crazy exercise.
Casey 01:20:05
Right.
Liron 01:20:05
But you just have to pick one. It’s—
Casey 01:20:06
That’s it.
Liron 01:20:06
—it’s no less likely than any other. Yeah, you’re picking basically there — people employed or at least in upper management or ownership of today’s AI companies, maybe they get a compounding advantage, maybe the AI gets really powerful. But then the eventual equilibrium is basically concentration of wealth, a bunch of people getting screwed somehow — no access to wealth, maybe no jobs. That’s kind of your mainline scenario?
Casey 01:20:31
Yeah, and I don’t have — again, because I don’t spend a tremendous amount of time trying to model these sorts of things, I don’t have an exact scenario, and I’m not claiming that it’s perfectly reasoned and here’s why. No, I’m just saying cynically my brain goes, “Well, somehow this is what will happen,” right?
And so if I had to pick the most likely outcome, I would say that in 10 years from now, what we will see is AI is shoved into everything. There are some positive uses for it — people are getting some benefits from it. There are a bunch of negative things from it as well — a bunch of software got crappier because it has AI jammed in it now and it didn’t work that well. A bunch of companies make a lot of money off of it, and that’s it. And all the same things that we have today happen again. There’s all sorts of problems with AIs telling people to do bad things, and then there’s lawsuits about it, and then there’s regulation about it.
Liron 01:21:38
Okay, but don’t you think we also cure a bunch of disease?
Casey 01:21:41
It might. I think we could say that computers, and probably the internet, have certainly had a lot of positive benefits that we could find along those lines, so I think AI probably will do some. But it won’t be insane, right?
Liron 01:21:58
A lower bound for heaven is curing a vast majority of disease. Let’s say curing cancer as a shorthand, okay? So when you say, “Hey, do you really want to put all humanity in the car?” Well, my assumption is that the car also involves a cure for cancer.
So that’s the kind of thing that gives me the confidence — if I have to make the decision unilaterally, I’m not saying I have to, I would love to consult everybody. But if it’s a genie saying, “Look, Liron, it’s you in this moment, man. Humanity’s on your back. You’re not allowed to phone a friend. Decide or not, but getting in this car, yeah, it might crash one in a thousand, but I’m telling you, cancer’s cured forever.” I’m like, “Oh, that’s so many lives,” right? That’s more than 1/1000th of the human population who now doesn’t have to suffer, right?
So that’s why I think — that is actually my mainline scenario. As long as humanity still has some power over the AI, I literally think AI is gonna cure cancer. But you’re not so optimistic about that?
Casey 01:22:44
Well, I’ll say a couple things. One, I think we could’ve cured cancer without AI. Maybe it accelerates the timeline. I don’t think that it’s if we don’t have AI, we don’t cure cancer. I think that’s ridiculous. I think the reason we haven’t cured cancer is because we don’t spend that much money on curing cancer. Despite what people may claim, we don’t spend trillions of dollars every year curing cancer. We’re spending trillions of dollars on AI. If you wanted to cure cancer—
Liron 01:23:05
Yeah.
Casey 01:23:05
—you could’ve done that, probably.
Liron 01:23:07
I’ll accept that there’s a significant probability that with enough money and human effort with zero AI, we might cure cancer or at least take a giant chunk out of it. I think that’s very fair.
Casey 01:23:16
And so what I would say is I don’t think that it’s — I think it is likely that AI will have some positive benefits. I just don’t think that they will end up being all that much different from the kinds of benefits we’ve had from prior technologies, and we will suffer all the same kinds of downsides that we’ve seen that we have suffered because of other technologies as well.
I think it will go basically like the internet went, and I think it’s pretty obvious that we can point to positive things about what happened to the internet. It’s also obvious we can point to a bunch of negative things that happened because of the internet, and I think that AI will look kind of similar to that. It won’t look — and this is only because I’m cynical. It’s always a cynical take to assume that any new technology will go about as well as the most recent one that wasn’t that good, right?
Would You Un-Invent the Internet?
Liron 01:24:23
But hold on. Don’t you love the internet?
Casey 01:24:26
No. I don’t love the internet. If I could snap my fingers and go back to the ‘80s, I absolutely would. I hate to say it.
Liron 01:24:36
Really?
Casey 01:24:36
Yeah.
Liron 01:24:37
But you’re such a good YouTuber.
Casey 01:24:38
You can — and this is not just me saying this. You can actually just ask people who are close to me, and it absolutely kills me that this is what I do for a significant part of my time instead of spending time typing on my computer. That’s just the reality of how our business works these days.
But if I could’ve un-internetted, I would absolutely take that bargain. Now, to be fair, when I say I would take that bargain, what I mean is I would take it if you’re talking about just for me. As I just said, if I was in the car and you’re asking me I have to reverse the internet for all of humanity, that is not a decision I feel I should make myself. So the rest of the world would have to weigh in on it.
Liron 01:25:24
Wait, hold on. Now I’m just curious. Can’t you just go find a job today where somebody will employ you at a pretty lucrative salary to go down in the basement and sit at a computer all day that doesn’t have internet access and maintain some old bank software or something, and then you don’t have to do a stream? Isn’t that still possible today?
Casey 01:25:41
I think that if there was such a job, I don’t know exactly where it would be, because any bank software that I can think of today is gonna say, “Well, you gotta do this with AI,” right? I haven’t heard anyone saying that the current way you could go get a job is by saying, “I would like to sit at a computer that is not connected to the internet and program it.” I don’t know any job where that exists.
Liron 01:26:03
Yeah. Let’s go back. Well, today AI’s taking over anything. Okay, but technically, sorry, I’m trying to make a point about how you love the internet. So what if we go back—
Casey 01:26:11
Oh, okay.
Liron 01:26:11
—five years, right before AI was helping with coding? And let’s say it’s some old computer where reference manuals tell you everything you need to know, Stack Overflow doesn’t even help. Don’t you think there used to be plenty of jobs in the internet age where you didn’t have to live stream? So it’s kind of weird to me for you to be saying, “Yeah, I don’t like the internet because I’m trapped here live streaming.”
Casey 01:26:29
No. I mean, well, if I knew about those jobs, probably, but unfortunately you also have to remember that it’s a path-dependent question, right? So one of the reasons that I do the kind of content that I do is because it does allow me that flexibility. I can spend 50% of my time recording videos, and then I can spend 50% of my time doing the kind of programming that I want on the problems I want.
If I was to get a job in the sort of bank basement that you’re talking about, that would be a very unfulfilling problem to be solving. In fact, I’d probably solve it fairly quickly and it’d be done, and I’d just be sitting around in the basement doing nothing. And so if you’re suggesting that there was some job I could have had that was not connected to the internet where I’m just programming things all day that are interesting, I absolutely would’ve taken that job. I have no idea where that job would be. I certainly was never offered that job.
Let’s Ride the Doom Train 🚂
Liron 01:27:19
Got it. Fascinating. All right. Last scenario for you, kind of riding my doom train.
Casey 01:27:25
Mm-hmm.
Liron 01:27:25
Gotta blow the whistle.
Casey 01:27:27
Oh, I love it.
Liron 01:27:28
So the next stop talks about what a truly super intelligent AI would want to do. Because—
Casey 01:27:34
Mm.
Liron 01:27:34
—you can factor the whole doom train into what can super intelligent AI do and what will super intelligent AI do. So my premise is that it can do quite a lot — a single data center can kind of make a blueprint for how all the atoms in the Earth should be rearranged, and then go and execute that blueprint using nanomachines.
That’s my impression. That’s my best guess as to what a truly super intelligent algorithm can do. You can just humor my premise. I think your position on that is that it’s maybe, right? You don’t know. Is that a good characterization?
Can AI Become Ridiculously Powerful?
Casey 01:28:04
Well, could we use a better example? Because assuming that it can make a plan for all of the atoms in the universe — are you assuming that the—
Liron 01:28:12
Okay.
Casey 01:28:12
—super intelligence is able to determine — because that presupposes that there’s something about the universe that allows you to represent the state of all atoms in less than—
Liron 01:28:23
No, no. I didn’t mean fully literally. I just—
Casey 01:28:24
Okay.
Liron 01:28:24
—meant a spec for, “Hey, I want you to — the Earth has to satisfy these properties.—
Casey 01:28:30
Okay.
Liron 01:28:30
—Here’s a grid, this land has to be used for this, and—
Casey 01:28:32
Okay.
Liron 01:28:33
—the earth has to be these structures. You get the idea.
Casey 01:28:35
Okay. Yeah.
Liron 01:28:38
With pretty high precision, but not atom level. Okay.
Casey 01:28:39
Okay. Yeah. Sure.
Liron 01:28:41
So you’re a maybe on that, right? Okay. So let’s—
Casey 01:28:42
Yeah. Okay.
Liron 01:28:42
—take it as a premise and—
Casey 01:28:42
Just make sure we — I understand, because I want to make sure I answer your question properly. So a data center, a single data center, was what you started with, is able to create a plan for how Earth is run. That’s what we’re talking about, roughly? All the things that people should be doing and machines should be doing — what all the robots should do, what all the people should do.
Liron 01:29:01
Yeah, pretty much. Yeah. Kind of take over the planet—
Casey 01:29:03
The overlord.
Liron 01:29:04
—and kind of have its way with it—
Casey 01:29:04
Yeah. Okay.
Liron 01:29:04
—with a lot of control, basically.
Casey 01:29:06
Understood. Okay. Sure. That’s plausible, I guess.
Will AI Preserve Humanity (Orthogonality Thesis)?
Liron 01:29:11
Nice. Okay. So we’ll put a maybe on that. And then accepting that premise, we go to the second half of the doom train, which is: okay, so it can do it, but then what will it actually do? So my question for you — it goes by the name—
Casey 01:29:21
I see.
Liron 01:29:21
—orthogonality thesis, right? Do you think an AI in that situation is then likely to realize that helping humans is good, right? And it should be nice. Or do you think there’s a good possibility that it just sees humans as random things in the environment? Or not random things, but just not its priority. Its priority could just be, “Look, I’m all about copying myself, just like most life forms on Earth are,” right? “And I’m just gonna do that, and the humans are irrelevant to me in that sense.”
Casey 01:29:45
Well, this is one of those ones where I think it depends a lot on what AI developers do in the next few years. I think this is also why I think it’s something worth talking about, because I think it depends a lot on the way AI is created.
So currently, with the way that AIs work, I would say that it’s highly likely that if you were to just keep going exactly the way that we’re going right now — without fundamentally changing which algorithms we’re using, how training works, without significant deviation from the current path that, say, frontier models like Mythos or something like that, that basic idea.
Casey 01:30:32
If what we’re positing is something like that eventually becomes this thing that’s the overlord, then my assumption is that it would feel a lot like very little has changed. And I know that’s a very strange thing to say, but my experience with how these sorts of systems work and how they are made suggests that they are based very much on prior things.
So a lot of what they do is based on things that have been done, and there are often a lot of pretty bad, sometimes hilarious, sometimes not so hilarious, weird holes or artifacts of the representation or how it’s executing. Which is kind of how it feels when you live under a current government. It’s a lot of stuff that’s based on things that were done previously, that got codified in various ways that are kind of weird and diffuse. There’s a lot of truly, sometimes even horrific things that occur as byproducts of that creation.
And so what I would expect the AI overlord to be like is a more science fiction version of that thing. You’re saying there’s robots and things like this, right? So I expect it to be kind of like, yeah, there’s a robot that helps me across the street — because I’m gonna be older at this point, I’m getting pretty old now. So there’s a robot that helps me cross the street, and sometime, for whatever reason, one time it just beat the crap out of me or kills me or something. And I have no idea why that happened. All of that stuff seems completely plausible to me.
That is sort of where I see it.
Will AI Seek Power (Instrumental Convergence)?
Liron 01:32:28
What about the idea of optimizing for reproducing itself? What if it just gets a desire to reproduce itself, or its programming says it should reproduce itself, and it’s better at that than we are at not getting stomped by it? Isn’t that a plausible scenario for you, that it just kind of harvests the Earth to reproduce itself throughout the galaxy?
Casey 01:32:47
I think it’d be a more plausible scenario if the way that these were constructed changes more. That’s why I say it depends a lot. I don’t want to dismiss that concern because I totally see where it’s coming from, and I don’t disagree that there are definitely ways that you could construct the AIs now that as you follow that to its logical conclusion, it will do exactly what you’re saying. So I wouldn’t dismiss that possibility at all.
That, to me, is not as likely because a lot of the current ways that AIs are made and improved don’t involve a lot of, for lack of a better term, hard constraint satisfaction or hard optimization. The kinds of what we would call operations research in computer science — they seem to have strayed away from that a lot.
And so while I don’t disagree that there are parts of AI research that look like that, and that if those became more dominant and central in the way these models evolve, you have a totally legitimate concern — I really don’t want to dismiss it or sound like I’m downplaying it. However, modern AI development seems to be deviating from that a lot, and so I would expect something much less predictable, much less consistent than what you’re describing.
Liron 01:34:23
Okay, fair enough. Yeah, that’s a great place to wrap it. Viewers, if you want to see my position, which is that we will actually see AI that has that kind of—
Casey 01:34:31
Right.
Liron 01:34:31
—flavor, that’s more of the reinforcement learning optimizer. I’ve got a recent conversation with Dr. Steven Byrnes where that’s kind of the focus of the conversation, and there’s an older conversation I had — I’ll stick up a link in the show notes — with Jim Babcock, where we also talk about this kind of phase shift that we think is coming. We basically think the kind of AI that’s winning at Go, that flavor of AI, is going to play the universe like a video game and win against you there too. That’s consistent with what you’re saying?
Casey 01:34:58
Very consistent, and that’s why I say both things. It’s why I say I think this is something that’s important to talk about. So I would never tell someone like you to stop saying what you’re saying, because I’m like, “This has to be talked about.” And also why I said the thing about it depending on how AI research goes.
Because the more it goes towards operations research and goes away from just, more of a “let’s try to encode the way humans think about things as large stochastic networks,” the more I think you run the risk of creating the kind of thing that you’re talking about. So what path you take and what the AIs are mostly built out of, I think, changes.
You always want to put that P(Doom) on things, right? I think that number, while I can’t say much about what its absolute value is, I could say its relative number, I think, changes dramatically depending on that approach. Decisions about approaches like that I think could make a pretty big difference in what that P(Doom) actually ends up being at the end of the day when we finally know what it is.
Wrap-Up
Liron 01:36:06
Awesome. All right. Yeah, this was such a great conversation. I wish I could get you to have doom as an even more salient mainline scenario in your worldview, because it seems like it’s coming fast and hard from my perspective.
But that said, I find you super reasonable at dignifying the question, being modest about what you know and don’t know, and pointing out that a lot of these stops on the doom train aren’t really compelling stops to get off on — that it is plausible to ride the train all the way to the end, to doom town. So yeah, appreciate the conversation. Casey Muratori, thanks for coming on Doom Debates.
Casey 01:36:39
Thank you so much. And yeah, I hope that I could contribute something. This is not my area of expertise, so I apologize for anywhere that I really could have given a much better answer if this was my domain.
Liron 01:36:55
No, fair enough. Well, that’s what we’re all about. We always get the cross-domain guests, and I can honestly tell you, you are definitely nowhere near the worst.
Casey 01:37:03
Okay, good. There we go. That’s my goal. I just wanna be a little bit above baseline. Whatever the minimum bar for the show is, if I could just be a little bit above that, we’re good.
Liron 01:37:14
Mission accomplished.
Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.
Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏
