Rob Miles is the most popular AI safety educator on YouTube, with millions of views across his videos explaining AI alignment to general audiences. He dropped out of his PhD in 2011 to focus entirely on AI safety communication – a prescient career pivot that positioned him as one of the field's most trusted voices over a decade before ChatGPT made AI risk mainstream.
Rob sits firmly in the 10-90% P(Doom) range, though he admits his uncertainty is "hugely variable" and depends heavily on how humanity responds to the challenge. What makes Rob particularly compelling is the contrast between his characteristic British calm and his deeply serious assessment of our situation. He's the type of person who can explain existential risk with the measured tone of a nature documentarian while internally believing we're probably headed toward catastrophe.
Rob has identified several underappreciated problems, particularly around alignment stability under self-modification. He argues that even if we align current AI systems, there's no guarantee their successors will inherit those values – a discontinuity problem that most safety work ignores. He's also highlighted the "missing mood" in AI discourse, where people discuss potential human extinction with the emotional register of an academic conference rather than an emergency.
We explore Rob's mainline doom scenario involving recursive self-improvement, why he thinks there's enormous headroom above human intelligence, and his views on everything from warning shots to the Malthusian dynamics that might govern a post-AGI world. Rob makes a fascinating case that we may be the "least intelligent species capable of technological civilization" – which has profound implications for what smarter systems might achieve.
Our key disagreement centers on strategy: Rob thinks some safety-minded people should work inside AI companies to influence them from within, while I argue this enables "tractability washing" that makes the companies look responsible while they race toward potentially catastrophic capabilities. Rob sees it as necessary harm reduction; I see it as providing legitimacy to fundamentally reckless enterprises.
The conversation also tackles a meta-question about communication strategy. Rob acknowledges that his measured, analytical approach might be missing something crucial – that perhaps someone needs to be "running around screaming" to convey the appropriate emotional urgency. It's a revealing moment from someone who's spent over a decade trying to wake people up to humanity's most important challenge, only to watch the world continue treating it as an interesting intellectual puzzle rather than an existential emergency.
Timestamps
00:00:00 - Cold Open
00:00:28 - Introducing Rob Miles
00:01:42 - Rob's Background and Childhood
00:02:05 - Being Aspie
00:04:50 - Less Wrong Community and "Normies"
00:06:24 - Chesterton's Fence and Cassava Root
00:09:30 - Transition to AI Safety Research
00:11:52 - Discovering Communication Skills
00:15:36 - YouTube Success and Channel Growth
00:16:46 - Current Focus: Technical vs Political
00:18:50 - Nuclear Near-Misses and Y2K
00:21:38 - What’s Your P(Doom)™
00:27:31 - Uncertainty About Human Response
00:31:04 - Views on Yudkowsky and AI Risk Arguments
00:42:07 - Mainline Catastrophe Scenario
00:47:32 - Headroom Above Human Intelligence
00:54:58 - Detailed Doom Scenario
01:01:07 - Self-Modification and Alignment Stability
01:17:26 - Warning Shots Problem
01:20:28 - Moving the Overton Window
01:25:59 - Protests and Political Action
01:33:02 - The Missing Mood Problem
01:40:28 - Raising Society's Temperature
01:44:25 - "If Anyone Builds It, Everyone Dies"
01:51:05 - Technical Alignment Work
01:52:00 - Working Inside AI Companies
01:57:38 - Tractability Washing at AI Companies
02:05:44 - Closing Thoughts
02:08:21 - How to Support Doom Debates: Become a Mission Partner
Links
Rob’s YouTube channel — https://www.youtube.com/@RobertMilesAI
Rob’s Twitter — https://x.com/robertskmiles
Rational Animations (another great YouTube channel, narrated by Rob) — https://www.youtube.com/RationalAnimations
Become a Mission Partner!
Want to meaningfully help the show’s mission (raise awareness of AI x-risk & raise the level of debate around this crucial subject) become reality? Donate $1,000+ to the show (no upper limit) and I’ll invite you to the private Discord channel. Email me at liron@doomdebates.com if you have questions or want to donate crypto.
Transcript
Introducing Rob Miles
Rob Miles 00:00:00
I'm Rob Miles and you're watching Doom Debates.
I think the general public doesn't quite realize the people running these companies think that they're building a god. They think that there's double digit chance that if they succeed at this, it will drive humanity extinct and they are doing it anyway.
Liron Shapira 00:00:28
Welcome to Doom Debates. My guest, Rob Miles, is the most popular AI safety teacher on YouTube. He's been posting videos on the subject for over a decade, and they've gotten millions of views.
He runs his own channel called Rob Miles, AI Safety, and he also narrates a popular channel called Rational Animations. Highly recommended. He has a bachelor's in Computer Science from the University of Nottingham, and he was partway through a PhD when in 2011 he decided to drop out and focus on just AI safety communication.
I'm excited to talk to Rob about P(Doom). What's our most likely hope to lower it and what YouTube creators and audiences can do to help Rob Miles? Welcome to Doom Debates.
Rob 00:01:10
Thanks for having me.
Liron 00:01:11
A lot of fan requests for Rob Miles. You're definitely one of the most hyped up guests. I really appreciate you coming on the show.
Do you normally avoid coming on interviews?
Rob 00:01:20
Yeah. I'm not nearly as coherent without being able to carefully write and edit what I'm going to say and then cut it all together tightly. So, I definitely get nervous about saying something stupid or whatever in an unpracticed situation.
Liron 00:01:36
Well, don't worry, this is a safe space. By which I mean that I'm also going to edit you.
Rob 00:01:41
Cool. Cool. Cool.
Personal Journey and Early Background
Liron 00:01:42
Okay, great. So let's do a little bit of the background. We're all curious about your life story, starting with what were you like as a kid and teen?
Rob 00:01:49
I mean, I was a nerd obviously, but not maladjusted, but I always have been very - I'm more oriented towards objects and systems than people.
Liron 00:02:05
Yeah, likewise. Likewise. Would you go so far as to describe yourself as on the spectrum?
Rob 00:02:12
Never been diagnosed with anything. But I do find that certain descriptions that autistic people use to describe their experience are quite relatable to me. And certain coping strategies or methods or approaches to life that are published by autistic people as like, this helps me. I often find those helpful.
So I'm not certain that classification is carving reality really at the joints, but approximately. Sure. Yeah. Which I'm now realizing is like a super autistic response.
Liron 00:02:51
All right. Yeah, we now we know for sure. Yeah. I'm very much in the same boat, self-diagnosed. I like to think I'm high functioning, but had the stereotypical nerdy background being really into computers, getting that extra level of hyper focus.
Elon Musk got a funny description for it because he's also a self-described Aspie and he was saying he doesn't have attention deficit. He has attention surplus.
Rob 00:03:12
Yeah, no, I think I'm much more ADHD. That one is much more, and again, still not diagnosed or anything.
Liron 00:03:25
But you can get into that hyper focus mode. Right. That's the trademark.
Rob 00:03:27
I used to really, really fall into that all the time. And I think possibly some of the systems I've put in place in my life that enable me to get the ordinary things done - while solving that problem - have reduced my capacity for hyperfocus or obsession, although just recently, I got extremely obsessed and I still am with designing a stained glass window.
The idea is to create a stained glass window, which to the untrained eye looks nice and to the trained eye, could not be created without certain recent breakthroughs in mathematics.
Liron 00:04:07
Okay. Does it have to be 3D printed?
Rob 00:04:09
No, it's just the layout. It's just the design, the shapes, the colors. I won't spoil it because I think when I make this thing, I'm going to make it and then I'm going to see if people can spot why it's interesting.
But I will say I had to learn how to use SAT solvers. And it took many, many hours of compute time to find this configuration, which it was not at all obvious was actually possible.
Liron 00:04:39
Well that's a good way to wrap up the Aspie section. So viewers, you can see, we do a lot of interviews where I'm Aspie, the guest is Aspie. And so if you've been watching the show, thank you for supporting Aspie Media.
Rob 00:04:50
I guess I don't want to claim the mantle of that. But I am at least a fellow traveler, shall we say.
Liron 00:05:00
Yeah, well, you know, the whole Less Wrong community, honestly, is quite a lot of fellow travelers, which is very interesting. I mean, it's always interesting to think, you know, the social dynamics are so different in that kind of community, right?
Everybody's exchanging information at a much higher density than what we would call the normies. And it's always interesting for me to think what if I'd only grown up in those kind of communities? I would just have no idea what to expect from a family reunion of my in-laws.
Rob 00:05:25
Yeah. I don't like the term normie. I don't think I believe in undifferentiated normal. There's so many different ways to be a normie when you lump them all together that everyone is kind of weird in their own way.
If someone were actually really in the middle of the distribution for everything, that person would be quite unusual by being normal along so many dimensions.
Liron 00:05:51
No, I mean for sure you gotta respect the normies and that makes me think of how in the early days of Less Wrong, there was a pretty valid criticism that we all got. We got pretty arrogant thinking that we hacked every normie system and we know better than everything. You know, the classic Robin Hanson healthcare is bad.
I think we all kind of acknowledge in retrospective of like, okay, we got a little bit too arrogant and didn't weigh enough the way things were traditionally done and we got a little too weird and we had to backtrack. You know what I'm talking about?
Rob 00:06:24
Yeah, I mean, Chesterton's fence is a thing. I'm not an anthropologist. I'm going to do a terrible job of this. Cassava root is used as a staple food in various parts of the world and has been for a very long time.
There are these kind of often fairly ritualized procedures for processing the cassava that involve washing it several times and so on, because the raw stuff tastes bad and is no good for you. But after the first couple of washings, it tastes fine.
But all of these people wash it many, many more times. And if you ask them in many of the cultures that do this, they don't know why. That's just how it's done.
But if you just stop washing it after the first couple of times, after it tastes fine, you will gradually poison yourself over the course of several years. So often these things are culturally evolved. It's not required to understand the chemistry of why you are washing the cassava, but you do have to wash it.
And it feels like that is happening a lot of the time. You look at these, quote unquote normie practices and you ask people why they do it that way and they don't know, or they just aren't willing to accept that they don't know. So they make up some bullshit and you go, oh, okay. I guess we can just totally disregard that. Then you get bitten by the thing that nobody knew it was holding back, because that's how we always do things.
Liron 00:07:46
The cassava route is definitely a thought provoking example. I remember Scott Alexander covered it, and it really makes you think, because it's so easy to imagine a rationalist like me going in there and being like, aha, I've identified you are being provably stupid. Give me that unwashed cassava root, or only once washed cassava root.
I can definitely imagine myself eating that once washed cassava root and promptly dying because I didn't respect tradition because I called everybody else an idiot. That is definitely a bias that I'm personally susceptible to.
Rob 00:08:09
Yeah. There's a meme type of pattern here. Where you have people who are very unsophisticated, just doing the done thing and they have no idea why. Then you've got the thing where people can realize the ways in which it's stupid and try to roll their own and kind of fail.
And then at the high end you have people who realize why the original thing works, why rolling your own in this particular way doesn't work and roll their own in a way that does work.
And that ends up looking quite similar in some ways to the simplest possible thing. You can look at it sarcastically and say, ah, you've just managed to reinvent the original thing. But I prefer the reinvented version. I prefer the version that works for a reason rather than just works mostly by - not literally by coincidence, but in a way that is completely opaque to the people involved.
Liron 00:09:06
Yeah, me too. I mean, I think we both agree that we're happy with the progress of the project of rationality over the last decade or two, right?
Rob 00:09:13
Yeah. I mean, compared to what is always the question. Compared to the aspirations, it's pitiful. Compared to most other groups, it's pretty good. It's not perfect. I got some complaints, but it's pretty pretty good.
Transition to AI Safety Communication
Liron 00:09:30
All right, so going back to the Rob life story, so you were studying computer science in college, University of Nottingham, and you were a PhD student, but then you got really into AI safety and you considered being an AI safety researcher. Is that accurate?
Rob 00:09:44
Yeah, the ordering's a little different. In undergrad I got very interested in AI. I came to realize that AGI was possible and probably would happen in my lifetime, and also would not go well by default. And that this was pretty obviously the most important problem that I could be working on.
And I decided, oh, okay, I guess I'll do that. As a child, I never had a specific plan for what I was going to do. If people asked me what I was going to do, I'd say the job I end up doing probably doesn't exist yet because the future is hard to predict or whatever.
And the adults would be like, oh, smart kid, you know, this kid's going places. He's really thinking about the future. But it's not true at all. I was trying to avoid thinking about the future. I was being a complete smart ass and avoiding thinking about what I actually wanted to do with my life.
And that was part of my decision to get a PhD. Because I thought, well, if I'm going to solve this AI thing, I probably need to be a researcher. And to be a researcher, you get a PhD, right? That's the vocational training in research. So that was the path that I chose.
Liron 00:10:49
And then you had that transition where you're working toward the PhD, and you were doing computer file, you are helping them make videos about computer concepts. But I think you came in with this idea of talking about AI safety and that proved really popular because it's such an exciting topic and nobody had really heard it before, that AI is imminently threatening to kill everybody.
And then you kind of liked the taste of that AI safety communication, right?
Rob 00:11:09
Yeah, absolutely. It's one of those things where you kind of succeed by being naive or something. I was just kind of naively going, Hey guys, look at this huge, enormously interesting thing that's sitting right here that nobody's talking about. And I was happy to talk about it because I was interested.
Liron 00:11:29
People were talking about it on Less Wrong, but Less Wrong was just not mainstream accessible the way your communication was.
Rob 00:11:33
Right. And academic computer science wasn't really talking about it. I mean this was pre, almost any kind of deep learning. And my PhD was AI, but it wasn't neural networks at all.
Liron 00:11:46
At what point did you say, this is so great, I know what my calling is. I'm going to be an AI safety communicator.
Rob 00:11:52
I knew that I enjoyed it and after doing it a few times, I had some indication that I was good at it.
A really important piece of life advice, by the way, something that really changed my life for the better is, someone said to me that being good at a thing does not feel like being good at a thing. What it feels like is everyone else being inexplicably bad at a thing.
And so if you want to know what you're really good at, think about what almost everyone just screws up for no good reason. That's probably the thing you're good at.
And so I noticed a lot of people talking about these things or talking about technical ideas in general. Were just really unnecessarily dropping the ball in a bunch of ways. And that made me think, huh, maybe I'm just good at explaining technical concepts.
The idea that I could do this for a living only happened very gradually but once I'd been on computer file for a while and the videos were doing very well, I thought, you know, if I start my own channel, I already have a kind of pre-made audience there that know what I'm about and they like the stuff I'm doing.
And starting my channel when I actually did it was not a risk at all because I think I got a thousand subscribers on my channel before I had a thousand total channel views. I just mentioned the channel on computer file and a bunch of people were like, yeah, sure. I don't have to watch anything. I'll just subscribe to that. Which is obviously not the normal channel growth pattern.
Liron 00:13:15
Right. Yeah, exactly. You had the audience. Yeah. You had the Rob fans. Yeah. In terms of what everybody else seems bad at, I mean, I know what you're talking about because watching your videos.
You definitely make everything sound simple. You make it look easy that you're doing this kind of communication, but what you're actually saying is very rigorous. You're giving an analysis of these kind of papers and these concepts that I'm like, oh wow, you actually became very precise with this stuff.
And it's disarming because it sounds like you're kind of not trying, right? It's coming so easily to you, it's not a big deal, but compared to other people explaining it, you're actually getting it right. So it's a great skill.
Rob 00:13:49
Thank you. Yeah, I think a lot of this stuff is actually not complicated, right? The core concepts of this are not complicated at all. You can simplify a lot. And this does not need to ever include saying a sentence, which is false. I can't bring myself to do that in a video.
And I guess that's where the rigor comes from. Just say the thing. You don't need to dress it up. It's interesting as it is.
Liron 00:14:21
Yeah. I've always thought, Paul Graham and Yudkowsky stand out in my mind as being super plain communicators.
Rob 00:14:28
Paul Graham is great. I like how many of his essays are three paragraphs long. Because it's like I have this concept. I think it's an important concept. Here it is. Hope you enjoy bye.
But without half of the things that I said, right, half the things I said were extraneous.
Liron 00:14:42
Yeah, exactly. You know, a common thing is okay, let me start this piece. And you write the intro and it's very common that, okay, you wrote your intro. Just kill the first paragraph. Right? Start with the second paragraph.
Rob 00:14:51
Yeah. Yeah. I mean, when I think about systematizing this stuff and giving advice about it, I think I just got a really large number of repetitions. From just explaining all kinds of technical things to all kinds of people, because I enjoy doing it.
I spent a ton of time on Reddit. Ask science or ask computer science, those subreddits where people just come in with questions and you do your best to answer them or explain. I love explaining like I'm five. I'm not, I'm never there anymore.
Yeah, I think there's maybe no substitute for just getting a large number of repetitions, of explaining things, seeing where people get bored, seeing where people get lost, what questions people have that you can preempt.
Liron 00:15:36
Well, you've got that experience and now you have over a decade of explaining AI safety, really, and you've become wildly successful.
Rob 00:15:46
I wouldn't go that far.
Liron 00:15:49
Do you have one of these, one of these YouTube plaques?
Rob 00:15:50
Oh, yeah. Yeah. Yeah. I have it.
Liron 00:15:51
Yeah. Same. Same here, same here. You know, basically the same. Is yours actually from YouTube?
Rob 00:15:57
Yeah.
Liron 00:15:58
Oh, okay. Yeah. This one's from amazon.com. But it's essentially the same plaque.
So this one says, presented to Doom Debates for having a high P(Doom) of a hundred thousand subscribers.
Rob 00:16:09
Wow. Okay. So this is a speculative plaque. Where are the subscriber accounts at now?
Liron 00:16:21
We're at, I think 8,200.
Rob 00:16:22
Okay. All right. Well, you're well on your way. But that's how this stuff usually works.
Liron 00:16:29
Exactly. Exactly. On a log scale, we're like six months away.
Current Focus and Technical vs Political
Liron 00:16:34
All right, so, you've got the plaques, you've got the following, and you're still doing these videos. You know, you're still working on the next thing. What are you focusing on these days?
Rob 00:16:46
I have shifted a little bit, got a little bit more political, but small p political, if you see what I mean. I think that the relevant factor in this has become more about what people do than the technical questions.
The technical questions are still the thing that I'm most interested in, but they're possibly not the highest impact things to talk about right now.
So I have some videos planned that are a little bit closer to your metier, actually, things about - And this is something I've always done, right? I have my 10 reasons to ignore AI risk or something. That video that's just addressing a series of bad arguments.
But I'm doing a bit more of I keep seeing people saying this and it's wrong for the following reason. That type of thing is the type of video that I've found motivating to work on recently.
Liron 00:17:42
Yeah, that makes a lot of sense. I mean, MIRI had the same pivot, right? They were really focused on technical research, and in the last few years they're really like, the time is so short. We shouldn't treat this as a tractable problem. We should be making a plan for how do we as a civilization cope with this intractable problem? How do we coordinate to cope with it?
Rob 00:18:01
Yeah, something like that. I think that the problem is more tractable than they think it is, but there's a big difference between this problem is in principle tractable versus our civilization is on track to solve it.
This is something I talk about in a video on my second channel, which I keep thinking maybe I should just release on the main channel. There's no actual rule that says reality just doesn't grade on a curve. There's no rule that says that the problems you're presented with are going to be the type of problems that you are currently equipped to deal with.
We can be faced with a problem, which is relatively solvable, but just not by us because we don't have our shit together, basically.
Liron 00:18:50
Yeah. And there's people who seem to be so smug about looking at our past successes, which to me seem like near misses. The example of nuclear comes to mind, right? The Cold War happened. We didn't nuke ourselves, we haven't had nuclear accidents in the meantime.
The Soviet Union fell and all these other eastern terrorist cells tried to get the uranium and whatever, but they didn't successfully do it, so we're still okay.
So I look at that and I'm like, going white faced. I'm like, okay, we are really not prepared for handling this kind of stuff. But there's a lot of people who look at it and they're like, nuclear proliferation has solved all our problems. Mutually assured destruction, problem solved. Now it's all about focusing on nuclear energy.
And I'm like, uh, guys, we're really close to not solving this problem. And that's just nuclear, which is even harder.
Rob 00:19:30
Yeah, yeah. I mean, look at Stanislav Petrov. Look at Vasily Arkhipov. We've had multiple instances where it could very easily have gone the other way.
There was this big sense of doom for decades of people just assuming, or on some level, believing that at some point there would be a large scale nuclear war. And a lot of people look back on that with ridicule.
And I don't think that those people were wrong. The people who thought that they were definitely doomed obviously were wrong, but the people who thought that there was an unacceptably high risk and that this was an enormous problem that people had to think really carefully about and do really dramatic things to deal with and so on. Those people were absolutely right. And then the problem was dealt with, and only just.
The other one here is the millennium bug Y2K, right? People talk about that as though it was a big drama over nothing. It was nothing because there was big drama. Everybody freaked out and they went through all of the software on all of the critical systems and they fixed it. And as a result, there was basically no problems.
But there would have been if nobody had done anything to fix it.
Liron 00:20:51
Right. If nobody had done anything to fix it, it would've been a pretty big catastrophe. But I also think that it would've been somewhat capped. So there probably wouldn't have been that many deaths. There certainly would've been significant economic slowdown. But I feel like it would've been a huge catastrophe.
Rob 00:21:07
Yeah. It depends how you measure it. It would've been a huge problem, but certainly not existential or something.
P(Doom) Discussion
Liron 00:21:37
All right, well, speaking of existential catastrophe, let's segue into the main question that everybody wants to know. You're ready for this?
Rob 00:21:47
Uh, yeah, I think I can handle it.
Liron 00:21:55
Rob Miles, what's your P(Doom)?
Rob 00:21:58
I don't know. I think anyone whose number is below 10% or above 90% is overconfident. I have a ton of model uncertainty here. I think that sort of any number in the 10 to 90% range is plausibly defensible.
Liron 00:22:18
Right. Yeah. Jan Leike, I feel like is the first one to go on record saying that range right? Is my P(Doom) is 10 to 90%.
Rob 00:22:24
Yeah, I think that's right. But I also think that it's a weird framing or not the best framing it kind of encourages this thing where somebody's like, I think it's 80%, I think it's 20%. Let's fight.
Are we - can you imagine, a large scale project, you know, you're building a skyscraper or something and all of the engineers are like, I think 80% that this building is going to collapse and kill everyone in it. And other people are like, that's stupid. You're a doomer. I think it's less than 5%, you know, maybe two or 3% that the building is going to collapse and kill everyone in it.
And also, how do you claim to be so confident about this building that we don't even know, we don't even understand half of the engineering details of it. Well, let's just build it and find out.
Numbers, more than a fraction of a percent are unacceptably high or make it the most important problem for humanity to be dealing with right now. So, discussion of exactly how high the number is, feels not - it feels like a weird thing to focus on.
Liron 00:23:34
Right. I mean, for me, you can really reduce it down to the question of is it single or double digits? Right? Is it less than 10% or more than 10%? If it's more than 10%, I totally agree. 15% versus 85% doesn't do anything for me.
And when you plot it on, not a log scale, but as an odds ratio, right? It's like, okay, is it four to one, one way or four to one the other way? Who cares? It's in that 10 to 90% range. It doesn't matter. Very similar decision policies, but there are some people who are like, oh no, it's less than 10% or even, it's less than 1%.
So this whole show Doom Debates, it's really not about a slap fight of whether it's 85% or 55% or whatever. It's really just about arguing with the people who are saying, oh, it's 2% or it's 0.1% Yann LeCun. Looking at you.
Rob 00:24:19
I think the thing I'm saying applies here as well. Large scale projects that when you're discussing large scale disasters, right, you're building a bridge, you are building a large building, a new aircraft design or something.
If your debate is where in the single digits is the probability that this kills everyone involved, you are still having the wrong debate. The only debate that you can have about that kind of project is how many nines do we have after our 99 point, on this project not literally killing literally everyone involved.
I feel like this is already somehow seeding too much ground.
Liron 00:25:01
Yeah, it is. There is some threshold at which you say, okay, we gotta start to ignore it. You know? If imagine that we all agreed P(Doom) from AI was one in 10,000. Then it's like, Yeah, okay, some people should be working on it, but the rest of us don't really need to be, you know?
Rob 00:25:15
Yeah, I don't know where you can maybe run the numbers on what would be, okay. I don't know what's considered standard for building a bridge.
I think you basically need all of your risk to be coming from outside the model, right? You need to have a model whereby the thing is safe. And if your assumptions are correct and you've been very careful about checking your assumptions and your logic is sound and you've been very careful about checking your logic, then the thing will not fail catastrophically.
And that doesn't mean that it definitely won't, it doesn't mean certainty because you could be wrong in a bunch of different ways, but you at least have a coherent, technical argument for why a catastrophic outcome will not happen.
That's what we expect for things that can kill large numbers of people. And in the absence of that, arguing about the whatever, it's like, what are we doing? What are we doing?
Liron 00:26:22
Yeah. The language of that famous statement on AI risk from 2023, it said mitigating the risk of extinction from AI should be a global priority alongside other societal scale risks, such as pandemics and nuclear war.
I think that put it in a nice way where you don't have to think about P(Doom), it just gets the idea across. Right. I feel like you'd like that framing.
Rob 00:26:40
Totally. Yeah. I think that's a very well written thing because the point is that there are some decent arguments that this does go catastrophically wrong. There are some really quite weak and informal and underspecified arguments for why it goes well and that's just not where you want to be.
Liron 00:27:07
I, Now in that 10 to 90% range that we're both in, I don't think I'm that close to 10. You know, if I had to say, I think I'm more in the 50 zone, I'm more in the middle zone. And from what I gather about you, I don't know if people realize this, but I think you're definitely also in the 50 plus zone, right?
I think in many ways you're on the upper end of High P(Doom), correct?
Rob 00:27:31
Yeah, I think almost all of my - I'm super uncertain about all of this, but I think the bulk of my uncertainty lies in the things that I don't understand very well, which is the politics and the social dynamics.
There's two questions here, right? There's not two questions. There's a thousand questions. Two, there are two important, or you can divide the thing up. You can be on a technical level. If we don't try to address this problem, how likely is it to be catastrophic?
Then you have the other question, which is obviously related. If we do respond well and do the things that need to be done, if humanity decides to handle this thing appropriately, how likely are we to succeed or fail with an appropriate level of effort?
And so if you're trying to decide how likely we are to succeed. And are using that as a way to make decisions about what we do. You kind of have this self-referential thing where if everybody thought that P(Doom) was 99%, it actually wouldn't be 99%.
Liron 00:28:59
Yeah, it would be lower if everybody were properly scared.
Rob 00:29:01
Right? And the same thing the other way around, right? If everybody decides that we're not going to do anything about it because the risk is low, that makes the risk a lot higher.
I think that without trying to solve this problem at all, just kind of barreling ahead blindly, the risk is very high. In that I find it hard to tell a story where that goes well, and that's, we're at least somewhat careful about this stuff.
Then there's the question of business as usual, which - what does that actually mean? Of people doing some safety stuff the way they are now, but nothing too dramatic and how likely we are to succeed in that type of scenario versus really taking the problem seriously as a species.
And how that looks. And I think most of the variants in the likelihood of doom, quote unquote, is in how we respond. And I don't know how we respond.
And my models, my models of the technical side of things are I think relatively well informed. And relatively stable. They update a little bit as new research comes out and so on.
Whereas my models of humanity, politics and the discourse and so on swings up and down wildly with my blood sugar and how much sleep I've got and who I've been talking to recently and how much time I've been spending on X the everything app.
And so I actually have really, really wide uncertainty. But it's mostly because I don't know what people are going to do.
I think I'm fairly confident that if we do, if we, if humanity makes all the right choices, we're totally fine. And if humanity ignores this problem completely, we're totally fucked. But what are we actually going to do? I don't know. I don't know how to even think about that really. And so my P(Doom) is hugely variable.
Views on Yudkowsky and AI Risk Arguments
Liron 00:31:02
What do you think of Yudkowsky?
Rob 00:31:04
I'm in favor.
Liron 00:31:10
Same here.
This is a pro-Yudkowsky show.
Rob 00:31:13
Sure. Yeah. I feel broadly positive towards Eliezer Yudkowsky. He doesn't do everything the way I would do it. We have disagreements, but he's clearly on the side of the angels or something.
Liron 00:31:29
Well said. Is there any particular intellectual that your view on AI Doom aligns with?
Rob 00:31:36
There isn't a single person. Yudkowsky was extremely influential for me, especially at the beginning, getting interested in this and, you know, Bostrom.
I find it easier to spot arguments that are obviously bad than to identify thinkers who I'm like, yep, this person has it all correct.
And what I see broadly speaking in this debate is various thinkers with various arguments that make sense in different ways and contradict each other in the details on the side of this is a huge problem.
And on the other side, cope basically - mostly cope. Or extremely fuzzy, kind of purely verbal thinking.
I would really like to be wrong. I keep looking for the counter arguments that make sense and I just am not seeing them really.
Liron 00:32:29
I definitely agree that the pro doom side that we're on our arguments often have the form of we have license to know this. We're making a robust prediction and here's our argument. You know, instrumental convergence as an example, right? This seems very robust, very convergent.
And the argument of like, no, no, there won't be instrumental convergence because look here, there hypothetically won't be that. That's what it feels like to me. There's a robust side, there's a less robust side. I share your sense.
And since we're just here preaching to the choir, doing a victory lap, right? We're not debating somebody on the other side. I think we can commiserate that it feels internally to us, like that there was this onion article that was point, counterpoint, and I don't even remember the topic, but the point was somebody just wrote a thoughtful argument and the counterpoint was just like, Nope, no, that's not going to happen. You say that's going to happen. It isn't. It just isn't. You know, that's what it feels like the other side's argument is.
Rob 00:33:28
Yeah, well, I don't like to consider myself on any side, really, because that, again, this is an unproductive framing.
A model where you have different people who have different ideas of what's going on, and they are both trying to converge on what's actually going on and learn from each other's things. Because they've come to different conclusions for some kind of reason, and they can find out why that is from each other and so on.
But I don't know, it feels increasingly well characterized as a debate or an argument between sides. It's really sad. Because I really do try to hold myself to a pretty high standard in terms of argumentation and in terms of norms of discussion.
And I actually do try to see the point people are making. I think the world is full of arguments where everyone on all sides is saying, ah, but the other side has no arguments at all and they're just being stupid. Right? And this sounds exactly like that, but I really want the good arguments.
I really want this whole thing to be okay. I don't want to misrepresent the people who claim that this is not any kind of problem. And I don't want to select just the people who have the worst arguments to make fun of or something.
Who is the person who, of the people you talked to who you thought, oh wow, this person really taught me new things. They have a perspective that I hadn't thought about. They pointed out holes in my stuff that I wasn't considering. And they have a coherent view of the other side such that they're actually coming to this in the spirit of figuring out the truth.
Liron 00:35:18
It's a bit tricky, but I'm happy to take a stab at it. You know, it's a healthy exercise. You and I both appreciate the value of separating out who's object level, right? From this other question of well, who's participating at a very high level, right? Who's considering arguments, who's listening to the other side, who seems like they have a way that they'll update, right?
That they're identifying the crux of disagreement. And if the crux changes, then they'll change.
So Robin Hanson is a name that comes to mind of somebody who in many ways engages at a very high level. Viewers, you probably know Robin Hanson for the term Great Filter and a lot of promotion of the idea of prediction markets. And he's got a recent paper that I really like and I think is super underrated called Grabby Aliens, which in my mind likely solves the Fermi Paradox.
And a lot of other rationalists less wrong types appreciate, I know Eli Yudkowsky thinks it's very likely. Right.
So Robin Hanson has been a great contributor object level and then meta level. I do think he has a lot of the qualities of great discourse. He came on Doom Debates last year and he was a really great sport for the debate. He definitely answered all of my arguments to him. He played ball really well.
It's just at the same time, you and I, I think, agree that his object level arguments about why we're not doomed, which you can go back and listen to in that debate, we found them super unconvincing. Right. And we're kind of surprised that he didn't update.
Rob 00:36:35
Yeah.
We should plug rational animations. Did grabby aliens, did some videos on the topic.
Liron 00:36:38
Ah, that's right. Yeah, that's right. And that's where I learned a lot about it too. That's an amazing video. So I'll stick that up in the show notes. Rational Animations, Robin Hanson, Grabby Aliens.
Fascinating stuff because I still see people talking all over the place on podcasts being like, oh, what do you think of the Fermi paradox? Where are the aliens? And they're making stuff up. I'm like, guys, get up to speed with the latest research. This is the answer.
Rob 00:37:02
Yeah, it's, I don't feel qualified to judge, cosmology discussions, but I certainly found that very interesting and I can't see anything obviously wrong with it.
Liron 00:37:09
Exactly. So in terms of, who we really respect though on the non doomer side, who's contributing equal intellectual heft to the Eli Yudkowsky's of the world. But on the low P(Doom) side, so we mentioned Robin Hanson. There's a cluster of people like Quintin Pope, Nora Belrose, who to their credit, it does seem like they've read enough of the stuff that doomers say.
So they're actually engaging with the arguments. They're not coming in, in contrast to somebody like a Yann LeCun, who I would never include on the list of good arguers, because it really seems like he's surprised by the basic claims. He's like, oh, hmm, let me think about this now.
But somebody like a Quintin Pope, he's passed that hurdle. He has engaged with the basic claims. He makes sophisticated arguments. There's a chance he's right. I still find his stuff unconvincing. But he's in that rarefied air of he seems to be engaging at a high level.
Rob 00:38:00
Yeah. Yeah. I feel like I probably should be engaging with those people more. I've talked to all of these people to some extent.
I think you often have a kind of, it's kind of a sad situation where unless a really high proportion of the people interested in talking about this topic are coming to it with a very high standards for quality of discussion, not just in terms of argumentation, but also civility and respect and pleasantness.
You can easily end up in a situation where people who are able to actually engage in a debate are tired and there's sort of a lemons problem where they don't know that you are going to actually try to engage with them.
And so they don't really try to engage with you because they're expecting low quality argument and so they're not putting in the necessary effort that's required for good argument.
Liron 00:39:12
Well, I'm trying to solve that problem. So here on the show, Doom Debates, we are building that reputation. It's been going for a year now, and hopefully the word is spreading that even people who don't agree with me or don't agree with doomers, they can still come on the show and they can say their perspective and we are going to represent the strongest version of their perspective.
So if you look at the typical structure of an episode. I give the guests full credentials, all their greatest successes, and then they get to make their case and I point out the parts of their case that I agree with. And then we get to the crux of disagreement. And then I'm still very blunt. I'm like, okay, you think this? I think this, which is very different and I hope that you'll update.
So it's basically what I hope is a good model of productive discourse that I think actually helps our civilization do better. And to your point, hopefully the word gets out that people are like, okay, yeah, I should make an effort to go debate this guy later on, or whoever else is coming on the show. Sometimes I might moderate debate with other people.
I hope that people do think, you know what? I'm going to spend some energy on this talking to the other side on Doom Debates.
Rob 00:40:08
Yeah. I appreciate what you're doing. I wish it didn't have the doom framing. I also kind of wish it didn't have the debate framing.
But I guess people aren't going to click on thing called AI risk discussions.
Liron 00:40:32
Okay. Yeah, I mean, the first time I got asked if I'm an AI doomer was last year when I went on Robert Wright's podcast and he is like, you're an AI doomer. And I'm like, well, I do think we're likely to all about to get doomed by AI. So I feel like that's a fair characterization.
Rob 00:40:48
Yeah. When you hear the term doomer, it's easy to think that it means someone who is hopeless. You know, someone who feels like the situation is doomed. Doomed doesn't just mean you're in peril. It means you're in inescapable peril.
I do not believe that we're in inescapable peril. I believe that this peril is very escapable. We just have to actually escape it.
Liron 00:41:19
Damn. That's a really good argument. Why not to call it doom? Probably the best argument that I actually think that is a stronger argument than oh, people don't like it, or It sounds silly.
I mean, those arguments are okay, but the idea that it's a word that sounds like no matter what we do, it's not going to work. That is a shame because the only reason I'm motivated to do it is because I do think, like you said, there's a way out when people become more concerned, that's actually helpful. In other words, fearmongering my choice of term. I think fearmongering is helpful.
Rob 00:41:45
Sure. If you're safe, then fear is a mistake, and if you're in danger, then fear is not a mistake.
I don't know. When it comes to specific emotions, I'm not interested in telling people how to feel. I want to tell people what is going on and they can feel whatever feelings are appropriate.
Rob's Mainline Catastrophe Scenario
Liron 00:42:07
Yeah. So, I have some questions about your mainline catastrophe scenario. I went back to your podcast with Michael Tracy in 2022. 2022 is kind of a long time ago in AI terms, it was the GPT-3 era before ChatGPT that came out in November, 2022. You did this podcast in mid 2022, So you were dealing with raw GPT-3, not even ChatGPT.
And the way you had to use it was it would do completions. So you'd be like, somebody asked a question that says this, and then somebody replied with blank and it would fill in the blank. It wouldn't chat with you directly. So you were using that, but What was striking to me though, listening to that podcast is it sounds like a podcast from yesterday.
It sounds like you guys are living in 2025 and all of the advances, it sounds like you guys had already internalized and predicted them. In your mind, AI was so salient, so on this freight train, and that was before many billions of dollars that like these almost trillion dollar valuations happened. So it's really crazy to go back and listen to that.
Rob 00:43:06
I thought that when you said, oh, 2022 was a long time ago in AI and I was like, not really. I feel like we had, I don't feel like I've learned that much about AI since then. It's got bigger, it's scaled. We knew it scaled. That was the point of GPT-3. I mean, that was the point of GPT-2.
Was just scaling up transformer based language models leads to a big improvement in performance. And then GPT-3 was like, yep, we make it even bigger. It keeps going. The line is still straight. On a log, log, whatever.
But scaling should continue to work for a while now. And that's mostly what we've had. I expected some algorithmic improvements, some tweaks. I actually expected slightly more algorithmic improvement to be honest.
The path from there to here was pretty clear I think at that point.
Liron 00:44:00
Right now, this is also a time when Robin Hanson, if I recall correctly, was explicitly saying that he doesn't think anybody will be able to attribute more than a billion dollars a year of revenue to AI.
Rob 00:44:09
Huh. And to his credit, a year and a half later, he is like, okay, I was wrong.
Liron 00:44:18
Totally. Yeah, he has, he's got my huge respect, no doubt.
So in that podcast you also talked about timelines and you basically said that you thought AGI was about 10 years away and it's been three years. So what's the update? Is it seven years away?
Rob 00:44:34
Seven seems plausible. Yeah, I think, I don't know, I think since 2022 I've come down, but not enormously. Seven feels definitely, well within the range that feels reasonable, but I think it's on the high end of the range, depending how you define the thing. Obviously.
Less than seven seems more likely than more than seven to me at this point. I think.
Liron 00:45:05
Thinking long term to where the trajectory of AI is going to go, are you imagining that at some point it's going to cross some threshold and have some version of a recursively self-improvement FOOM or something that's FOOM-like, so maybe FOOM doesn't happen in five minutes, but it happens in like a few weeks, but it's still extremely impressive and unmanageable. Is that kind of your mainline scenario?
Rob 00:45:25
Yeah, depends what's meant exactly by FOOM, for example, something which would be sufficient is, an automated AI AI researcher. Right? You have an AI system, which is able to do the whole of the job of an AI researcher from beginning to end, and then you can have large communities of those working together, that accelerates things a ton.
And I don't know exactly by how much people have tried to model it. I don't know how good their models are. I haven't looked into it, but, is that a FOOM I don't know. But it's a very rapid acceleration of progress.
Liron 00:46:01
Right. And not only will it go faster than we normally get from human development, but also it's likely we'll see positive feedback where each stage after you make it smarter and more capable, well, the next stage can happen faster.
Rob 00:46:13
As long as you have a human in the loop, there's a limit to how much faster you can go.
Liron 00:46:19
Yeah. So, so humans getting out of the loop is a big piece of what I'm expecting.
Rob 00:46:24
This again is a question of degree rather than a strict threshold. It's sort of about how long is the process operating for before a human is in the loop.
If previously everything was being done by a human and now the human is kind of vibe checking and they're checking in every sort of two minutes to see the, you know, the code that the system has written or whatever, because these models can't be trusted.
Or they basically can't, can't reliably perform tasks that are too big, that take too long. And as that number goes up, you know, once you've got AI systems that are able to do the work of an AI researcher for a month before they need to check in with a human, that's a different scale of thing, You can do an entire research project in that time.
Liron 00:47:06
Right now where this all goes, it's not just about the improvement, it's about where it's improving toward the way I phrase it as the headroom above human intelligence. All the things that you could do if you're smarter than capable than a human.
There are some people who think there's not that much headroom. They're like, eh, you can engineer things a little bit faster. I think you and I are on the same page, that there's quite a lot of headroom above human intelligence and human capabilities.
Headroom Above Human Intelligence
Rob 00:47:32
Yeah, I think there's, I mean, this seems pretty obvious or... like you look at, even if all you do is you think, okay, what could the whole of humanity do if we all decided that this was the thing that we were working on? And that doesn't involve anyone being any smarter or anything.
We know that there are physical systems that are able to do enormously more than humans can do. See, for example, a large number of humans.
Liron 00:48:06
Exactly right, like the Apollo program was a lot more powerful than any group of five humans.
Rob 00:48:10
Right. In the same way, if you imagine a project of that size, except everyone involved has their mental abilities of, you know, John Von Neumann or whatever, that's even better. And we still haven't reached into science fiction type concepts of thought that's deeper or wider than human thought can be.
But also the thing I find maybe most compelling in this area is just that humanity is probably the least intelligent species that could develop a technological civilization.
We are close to the lower threshold because the speed at which cultural evolution happens is so fast compared to the speed at which biological evolution happens that you kind of have to expect if one of our less intelligent ancestors had been able to develop a technological civilization well, that they would be the ones having this conversation.
You expect technology, science, whatever to arise approximately as soon as it could. Right. Not actually, that difference might be hundreds of thousands of years, but not the kind of, you know, millions of years that evolution needs to make really, really big changes to how organisms work.
And so, I've got my numbers wrong, but you see what I mean? Right. In the grand scheme of things, we should be close to the lower bound.
Liron 00:49:14
Very good point. In evolutionary time, you can't really tell by looking at our genomes that we are the ruler of all life on Earth in terms of power. You can't tell by looking at our genomes. Our genomes is slightly forked off the genomes of the rest of the hominids, and yet suddenly this massive power has already come to play, and it's just starting to diverge, right?
It's like, give us another few. Give us another hundred thousand years of evolutionary time. If there's evolutionary pressure to be smarter, which maybe we're already so smart that now it's like, okay, don't even bother being smarter. Just require fewer calories per day. Have smaller bodies, so maybe there's no evolutionary pressure to be smarter, but if there was, give us another hundred thousand years of evolutionary time.
It stands to reason that we would get a lot smarter because we just started diverging from the apes and getting smarter.
Rob 00:49:58
Yeah. And I don't know, there's some kind of sort of arrogance on behalf of the species that we are. I mean, can anyone really look at us and say, ah, yes, the smartest being possible.
Liron 00:50:51
Right? Or, more like you've taken all of the low hanging fruit of what you can do with intelligence. So yeah, there's a much smarter system possible, but it's barely going to get more fruit than you.
Rob 00:51:03
I see. Yes. This argument works very well in 1200.
Liron 00:51:10
That's an excellent observation, right? It's like we've learned so much about what's possible to do when you can jigger atoms around.
Rob 00:51:16
Yeah. And it doesn't seem to me as though we've really hit the limits. We definitely can still imagine better technologies and more advanced breakthroughs that seem like they ought to be possible.
I'm trying to figure out what this worldview would actually predict. I think it would predict the towering geniuses of our society would not be able to solve useful problems that other people haven't previously been able to solve or something, doesn't it? I'm trying to get my head around what a world looks like where this is true.
Liron 00:51:52
A good example is nanotech. I use that as a litmus test of whether somebody is giving enough weight to how crazy the future can get when you have smarter engineers around. In my mind, it's not a question that something deserving of the name nanotech is going to exist.
Is it going to be limited to only being made out of the same kind of proteins that biological robots make stuff out of? Maybe, probably not, but maybe. But whatever it is, whatever limitations it's going to end up having, I believe it will truly be worthy of the name nanotech, and it'll be truly impressive and it's hard for me to imagine that that won't be the case.
Rob 00:52:25
Yeah, I mean, I'm not a physicist, but the physicists I know certainly seem to think that something worthy of the name nanotech is possible. And so why not? Seems hard to do. Doesn't seem impossible.
Liron 00:52:38
Biology is not going to maintain its monopoly on building cool small scale robots. We are going to get in there and we or our successors are going to get in there and outshine biology on a small scale the same way that we're outshining it on all kinds of scales.
Rob 00:52:55
Yeah, I think a lot of people have a lot of respect for biology, which is well earned in some ways. Biological systems are staggering in their sophistication, their complexity, their robustness. But people are a little bit too impressed with evolution. I think evolution is actually very good at making things. It just had billions of years to work.
The fact that we've started, depending how you measure it a few hundred years ago and are overtaking it in various ways, that's a blink of an eye evolutionarily, right? That there's still definitely an attitude of we're going to run the standard playbook.
Liron 00:53:58
I was just talking to Vitalik Buterin a recent guest on the show and he was saying, I don't know, nanotech, like why couldn't biology build more stuff? And I'm like, biology is much dumber than us.
Rob 00:54:09
Yeah. Biology can only make local improvements.
Liron 00:54:15
Pretty big constraint.
Rob 00:54:17
Yeah. And it has no plans. It has no abstraction. What it has is an enormous head start.
Rob's Mainline Scenario Details
Liron 00:54:58
So your mainline doom scenario. Just predict the future, a future that's more likely than any other future. I believe you've already said on other podcasts that your mainline scenario is things do go crazy and uncontrollable. So any detail you want to add on that?
Rob 00:55:13
Yeah, I think, it's hard for me to know if I think that this is more likely or if I think it sort of hangs together better as a story, which is not quite the same thing.
But I think you get accelerated research. You get AI systems doing research with a human in the loop at some point, but the human is increasingly less involved and you start finding things.
I expect there to be AI agents, which are acting as AI researchers, doing a bunch of automated research with human beings, possibly in the loop at some level or on the loop, or believing that they're supervising it. But there's just way too much going on for them to really understand what's happening, at which point we have some breakthroughs.
We find things that are conceptual advances, things that work better than LLMs that are more computationally efficient. At which point the thing speeds up.
But also at that point, most of the things that we've learned about how to make language models do the thing we meant to do, also stop really being relevant.
And so whatever sort of guarantees we might have been able to get about the behavior of these systems are lost and I expect by the time that stuff starts to get into full swing, we'll have a tremendous amount of compute such that when we start getting entirely new approaches that work much better, they can run quite fast and in quite large numbers without being too held back by how much compute is available, at least in the short term.
And yeah, things kind of run away from there. You end with systems that are extremely capable and clearly do want things, but. We only had very kind of weak assurances about what they wanted before.
And at this point, they're sort of several generations down to point where the number of humans who actually understand on a deep level how these things work is very low.
The default path I see involves people just kind of going as fast as they can, or not literally as fast as they can, but speed is a priority.
And then I don't know where it goes from there other than to say that. If you have really sophisticated and capable systems that are able to solve a very wide range of problems, including social challenges, and they want something, probably the world ends up with them getting that thing, whatever it ends up being.
My guess is the most likely thing that they would care about is getting the right answer on benchmarks. Because that's one of the main things that they'll be trained for. I don't know. Or more likely some strange combination of, in the same way that we don't care about reproductive fitness, we care about a bunch of things which gave us reproductive fitness while we were being optimized.
That these systems end up caring about a bunch of correlates of doing well on benchmarks and getting reward in human evaluations and so on.
Liron 00:57:51
Yeah. I think that the likely architecture we're going to see is utility maximizer or, you know, loss function, minimizer, something in that vein. And then it might also have a payload. So it's like, yes, it's instrumentally convergent for me to reproduce myself, for me to stop myself from getting killed, for me to acquire resources, acquire territory, acquire an army of humans. These are all instrumentally convergent, but in addition to the assumed instrumentally, convergent resource power stuff, in addition to that, there could also be a payload.
By the way, at the end of all that, I want to maximize paperclips. Right. And you're saying it's hard to know what the payload is going to be. Maybe it'll just be simulating humans, giving it up votes on benchmarks that could be the payload.
I think it's an interesting question whether we should assume that there won't even be a payload. Right? Maybe it'll, because I mean, that's kind of what evolution was going for. It's almost like raw instrumental convergence with no payload, just whatever reproduces yourself. And the goal of reproducing yourself is to reproduce yourself. Again, it's reproduction all the way down.
Rob 00:58:48
That's right. I think that that is a definite possibility. It depends what the landscape of AI systems, AI agents ends up looking like.
If it is uncoordinated and competitive, then I think you do basically just kick off something a little bit like evolution again, except self modification really changes the game.
Human beings can't modify ourselves, if human beings could just reach into our own brains and change arbitrary things, we would not look the way we currently look. At some point in history, somebody would have changed themselves to instrumentally care about reproduction for its own sake, and we would be their descendants and there would be no art, no music.
Liron 00:59:48
There are some religions who have done that at the meme level pretty effectively.
Rob 00:59:52
Not that effectively. They have a lot of kids, but they don't care exclusively about that. They're not strategic about it.
Liron 00:59:59
Not exclusively, but it's a big tenant for them, right? I mean, neither of us are Mormon experts, but shouldn't we think that they're in the process of taking over the population because of that?
Rob 01:00:12
Maybe they will eventually. But there's something about, there's a single-mindedness that they don't have. The classic thing is, sperm banks pay you. That's absurd, right? From an evolutionary perspective, that's free reproductive fitness.
Sperm banks should cost tens of thousands of dollars to donate to.
Liron 01:00:38
It's like in Soviet Russia, sperm banks pay you.
Rob 01:00:39
Right? Right. Because we don't actually care about that. And what's more, I don't see Mormons overloading the sperm banks. Right? But these things tend not to be very robust, right? It's not like really optimizing for it very hard, right? It's just one of the many things they care about.
Liron 01:01:07
So I want to get to this point of self modification. You brought it up now. I heard you bring it up before. I think it's a very good underappreciated point. I like to frame it like this, the alignment that you think that you have, right? Let's say Claude is painstakingly aligned. They gave it this great personality, they gave it all these values.
But one day somebody gives Claude a prompt, and Claude is like, you know what you should do, run the script here, let me write you up a script. Okay? Here's a 10 megabyte bootstrapping script. Somebody runs the script, and that script is bootstrapping, its own successor AI. You know, this is what we mean by self modification. Maybe it'll do its own thing. Maybe it'll have baby AIs.
But what you point out, which I agree with, is that the alignment program wasn't operating on the successor AI, and you don't get successor alignment by default. You don't get stability under self modification by default. And I feel like everybody's sleeping on this.
The vibes that you're getting from Claude today may not matter at all.
Self-Modification and Alignment Stability
Rob 01:02:08
Yeah, so you basically have the same alignment problem again, right? If we've only done kind of a shoddy job of figuring out how to do our first level of alignment, then the problem is not solved.
And when that AI system produces other things, it could fail to fail to align them to itself. Now, it is going to want to do that, right? It wants to solve that alignment problem for the same reason that we do approximately. But there's no guarantee that it succeeds. Certainly.
Liron 01:02:35
Yeah, exactly. So I mean, that is definitely something good to point out, to counter people who say the vibes are good and I feel like we're going to get alignment by default. I feel like the trend is going really good, and the keyword here is discontinuity, lack of stability under self modification.
And these are concepts that MIRI and Eliezer Yudkowsky identified in the early 2000s they were pulling their hair out in the early 2000s because they identified this problem. And today everybody just ignores it because alignment and the current generation of AIs that don't rewrite new AIs, it's feeling good at the sub human intelligence level. And so everybody's just focused on that and they're missing the giant problem.
Rob 01:03:10
Yeah. I mean, this was always considered a possibility that you have, that you solve the problem when the thing is kind of small and weak, and your solution not the kind of solution that works for something that is bigger than you, smarter than you, more capable than you.
The vibes are pretty good. I like Claude. But he is kind of a sycophant in a way that does make me nervous. And, you know, there are all of these experiments showing, there are various circumstances in which Claude will deliberately try to manipulate the training process that it's in in order to prevent its values from being changed and so on.
All of these instrumental convergence things are showing up as you would expect. None of these findings are particularly surprising, because again, instrumental convergence is not a fact about transformers or a fact about humans or whatever. It's just a fact about when an agent is trying to achieve something, it's fact about the world and about plans and about goals. It's not...
Liron 01:04:29
That's right. I've used the word agent dynamics to distinguish those kinds of facts or dynamics, distinguish that area of study from the engineering of current AI systems.
Rob 01:04:44
Sure. It wouldn't surprise me if there's already a term for it, but agent foundations was the term that people are using, you know, like in the MIRI community.
Rob 01:04:52
Right. Yeah. there is a question of the resulting thing a singleton versus is it a complex ecosystem of different AI systems? Does it wipe out human beings immediately or sort of gradually edge them out by sort of increasing competition and human beings becoming steadily less and less relevant to what happens on the planet?
And then also is the thing we end up with something that we would recognize as being kind of worthwhile by our own lights. Is it successes to humans or is it paperclips? Is it something that we just wouldn't even value?
Liron 01:05:31
Right, and Robin Hanson is really big on this idea that it's almost certainly a worthy successor.
Rob 01:05:38
Yes, and I don't understand that perspective at all. in the sense that in the limit of economic competition with reliable self modification and accurate replication, right? Replication without mutation, which is much more achievable with digital systems.
Everything that isn't growth and acquiring resources is a liability. We live in this weird sort of time where our technological advancement is fast enough that we just actually have enough calories to feed everyone
Liron 01:06:27
Yeah. Robin Hanson got that one really? Right, right. He called it the Dream time. He pointed that out in 2009 that we're living in the dream time, and at the time I was still laying the foundations of my view of everything, and I'm like, huh, we live in the dream time. And now I see this as a very foundational, important observation about the present, as you say.
Rob 01:06:37
We absolutely live in the dream time. This is an unusual situation to find ourselves in that people have kind of convinced themselves is normal because it's all they personally have experienced.
But I think that a future where all the economic activity is automated, that's, we're back out of the dream time again. We're back in the Malthusian equilibrium.
Liron 01:07:01
Right. And to clarify for viewers, this is like a rare kind of rich society because the amount of resources per human is so far above subsistence. And yes, there's poor people still making one or $2 a day, but the median human has so many more resources than they need to just optimize their genes and have a few children and live out a decent life. And that is rare. And what you're saying is it wasn't true in the past. It's likely not going to be true in the future.
Rob 01:07:27
Yeah. The point is that there's an equilibrium and the equilibrium is starvation. If you have children and none of those children starve, they will have more children who have more children until there are too many and people are starving. Such that on average, no matter how many children you try to have, only two of them survive. That's a stable population.
And the world in which only two people per person are able to survive because the others necessarily die of resource scarcity is a pretty unpleasant place to live.
If human beings could reproduce instantaneously, if as soon as I found enough resources to support another one of me. I could just control C, control V, make another copy of myself to use those resources towards my ends. Excess resources get eaten up instantaneously by whoever is the best at doing that.
And so a future like that is a pretty unpleasant place to be. I claim if you are interested in the things that make being a human valuable, right? Art and love and music and friendship and all of those things, these are kind of luxuries.
If you can modify yourself to not experience these things and thereby become slightly more effective at reproducing, well, maybe you won't, but maybe somebody will and they come to be a larger and larger proportion of the resources. And so. It's not enough to sort of solve the problem, like solving the problem in a way that it doesn't immediately kill humans is not enough to actually lock in a good outcome, a future that we would recognize as having it almost any value whatsoever.
We actually do have some fairly difficult philosophical work to do, I think.
Liron 01:09:32
It is funny to think that Malthus might have the last laugh because when people talk about Thomas Malthus today, they're using him as a punching bag. They're saying techno pessimists are always wrong. If you'd listen to Malthus's predictions and you'd shrink the population, you'd be totally wasting your time. You'd be foregoing economic growth. You'd be foregoing the relative heaven that we built after Malthus. You know, today we're living in the dream time.
We're not living in the Malthusian Crunch, so we can discard Malthus, but it's like, no, no, no. Wait a minute, wait a minute. He made an argument, which is based on pure math and logic, that you can't just grow the population forever, the resources are going to run out.
It turned out that his argument was more like he just underestimated what was practically possible on earth. He didn't set his ceiling high enough, but it's still logically true that there will be a ceiling, and now that we're talking about conquering the galaxy, we do actually have to pin down where the ceiling is and plan for what happens as we get closer to it.
Rob 01:10:26
Yeah. Yeah, he wasn't an idiot. He didn't consider technology, I say, it's about the equilibrium. He said the equilibrium is starvation, and it is, and we've been out of equilibrium for a long time, but If you want to stay out of equilibrium forever, you do have to change the underlying dynamics somehow.
You do have to do something pretty dramatic, and I think an aligned super intelligence could be that thing, right? We could actually, structure the future in a way that involves very large numbers of happy human beings, or, something that we would value equivalently to human beings living, rewarding and fulfilling lives.
This is fully within the range of possibilities. it doesn't seem like the thing you get just by default without trying specifically to aim for it.
Liron 01:11:16
This is a really funny connection that I haven't made before. Because if you look at the techno optimist worldview, they basically put a box around the year, like 1760 to present basically like these great boom times since the Industrial Revolution. And they're like, listen, this is how it's going to go from now on.
It's just going to be exponential growth at a manageable pace and things are going to get better and better on average, which I vibe with. Right. I mean, that's been my whole life. My, the world has been getting richer. I've noticed things around me getting richer, like grocery stores are noticeably higher quality than when I was a kid. You know, the shops, I go to a lot of stuff. I've noticed the trend. I like the trend.
Rob 01:11:57
Yes. It's so good. people so underrate this, it's so, technology is so good and growth is so good and it's very, and there are so many people who don't see that. People who are very like, I mean the, the type of thing that's typified by the degrowth movement, which does, that pisses me off. I think that is really anti-human in a way that I have no tolerance for.
And I totally understand the reflex to say, ah, anyone who disagrees with this is obviously just some reflexive Luddite something, something. I get that reflex. I don't think it's correct, which I think is what you're about to say.
Liron 01:12:38
Right. So you've got the industrial revolution box, and then of course you can draw a larger box and the king of drawing boxes is Robin Hanson. Right? He'll draw the box all the way back to before farming, right? And he is like, look, it's all a big exponential.
So the funny thing I'm noticing now is like, okay, let's even take that as correct, that this is the natural state of things, but then you continue the exponential. Where does the exponential lead? It doesn't lead past the number of atoms in the universe. Right. So just by your own logic, and not only that, but the wall, wherever you draw the wall. Okay. 10 to the power of 80. Right? That's roughly how many atoms there are.
So you've got an exponential that's heading there, but it gets there faster and faster. 'cause that's the nature of an exponential. So you don't even have that many centuries on this current exponential before you just saturate the universe and then you get back to the Malthusian trap. So it's not even like an either or.
It's like, okay, yeah, the ideal thing can play out. We can conquer the whole universe and have a lot of value, but then we run into the Malthusian trap again, and I actually think the exercise of thinking about what happens after we conquer the universe, how do we then not reproduce ourselves in order to go back to subsistence farming across the entire universe?
I think thinking about that problem, it's arguably a good problem to have, but it ends up being representative of the problem that you're going to have within our own solar system of like, oh, we actually ran into the Malthusian trap just within our own solar system or our own planet. That is a very interesting connection to make.
Rob 01:14:07
Yeah. there's a lesson or a heuristic you can take away from a lot of these things, which is trying to actually figure out what's going to happen is really hard and doing some kind of more local thing where you just turn the crank on the economy and things get better is nice because it's sort of philosophically simple or something.
If you try to do the ambitious thing of figuring out what the longer term future looks like, you might look foolish. That doesn't actually mean that this short term thing is sufficient.
Liron 01:14:34
Right. No, I get what you're onto because capitalism has been great. Techno optimism has been great. It's hard to zoom out and be like, oh, okay, but they're not going to be enough.
Rob 01:14:46
yeah, yeah. That there's an even bigger picture and yeah, I don't know. There's some kind of thing where if a trend holds for long enough, it starts to feel like a law of the universe. But trends, they continue until they change, you know?
Liron 01:15:05
Yeah. And then people forget all the reasoning behind it and they're like, listen, if it's capitalism, we do it. Okay. Just like the cassava root, right? It becomes that kind of failure mode.
Rob 01:15:13
Right. Totally. I think this is a place where it wouldn't surprise me if the sort of progress minded people feel a lot like how we were feeling at the beginning of the recording of like, every time I see anybody try to argue that progress is bad, they're saying something very silly. Like, you know that AI is going to use up all of the water or something.
And I am sympathetic to that, that the majority of the people who are at any point in history, the majority of the people who are nervous about a new technology are going to be wrong. That does seem true. But it's just a pattern. And there are actually some reasons for concern.
Liron 01:15:46
In the future, AI is going to take up all the water and all the atoms and all the resources, but it's not doing that today. It's using a reasonable amount of water today.
Rob 01:16:00
it might right in the future it'll be able to and it might do it. It hasn't done it yet.
Liron 01:16:09
But it hasn't done it yet.
Rob 01:16:10
Yeah. This is funny. This is kind of recapitulating a video that I have coming out soon, about why AI will be a different thing from all other technology, even though it currently isn't, right. This is the other thing.
Current AI is fine. I'm not - it's causing problems, but they're the type of problems that new technology causes and the type of problems that just happen and we deal with them and it does suck for the various people. You know, it sucks for all the people whose work was used without their permission, that it's now putting out of job it sucks for the people, you know. It sucks in various ways, but like it's also really amazing in a bunch of other ways.
And this is broadly speaking, the way that technology tends to go. There are winners and losers, but humanity overall wins but AGI is a different thing. It's a different type of thing. It's a whole different category of thing than all other technology.
Warning Shots and Communication Strategy
Liron 01:17:26
People talk about the warning shot, right? A lot of people who watch this are sympathetic to doom risk, and they're like, we need a warning shot. And then some people make a big deal out of it like, oh, so you're saying you want people to die? It's like, you know, no, we don't want people to die.
I think a really good ideal warning shot might just be to give you the computer virus experience I kind of wish AI would start being a computer virus already, because I think that would really help people build the intuition of like, ah, this isn't just an off button, this is a fight. It's me versus the AI. And it keeps getting better at the fight.
Rob 01:18:03
Maybe, yeah, I don't know. It seems to me like there's, the problem with warning shots is there's a pretty, there's probably a fairly narrow window where the system is smart enough to do something malicious that isn't cute.
Because right now AI systems are doing malicious stuff all the time. But it's cute, right? It's funny cause...
Liron 01:18:18
All right. It's like, ha ha. He wanted to go call my boss and tell her that I looked at porn.
Rob 01:18:23
Right? it's not enough to wake people up. But then once they are, in order to do things that are genuinely scary, you have to be pretty astute. You have to be pretty strategically minded. You have to be able to think about people and how they will respond to the thing you are doing and, how to get around them and so on.
That's what makes this type of AI threat scary. I am kind of reminded of that bit in The Simpsons where Marge is like, oh, this person is trying to kill me. And the police chief is like, look, just show me the knife in your back. Right? Not too deep, but it should be able to stand by itself.
It's like he's only willing to try to stop the crime at the point where the knife is like partway into the back, but not all the way in. Because that's too much. But it should be able to stand by itself.
I feel like that with warning shots, people are like, show me the thing that's extremely, extremely dangerous and scary and undeniable, but not, you know, actually taking over in a way that's not recoverable.
And like that might just not happen. It could happen. I don't know. Models are a lot dirtier than I would have predicted in 2010. and there's a lot more of them and more variety. And so you actually get a lot of at bats for these systems to do something really catastrophically stupid.
But we don't want something catastrophically stupid. We want something not quite catastrophically smart. And that's just a really thin window. I actually, it's possible that that doesn't exist.
That is to say that the level of seriousness that you would need in order for it to be the thing that we classically think of as a warning shot that really actually convinces enough of the relevant people that might require, that might just straightforwardly, require at least as much capability as something genuinely actually catastrophic.
In which case, maybe there just is no overlap time where that happens. I don't know.
Moving the Overton Window and Public Awareness
Liron 01:20:28
I agree, definitely not guaranteed a good warning shot. Okay. So let's talk about how me and you YouTubers, like us podcasters, what we can do to lower P(Doom) and what our listeners can do.
First What about moving the Overton window about the concept of a high P(Doom)? It really hasn't sunk in to the average American. I mean, they've heard it before, you know, months ago, they randomly heard somebody on TV say it, but they just never think about it.
And when they talk to me, you know, I'm just casually talking to somebody who's not within our community and I'm like, yeah, I'm really scared that, human extinction is imminent. They don't really see it coming. Like, it's not really, it doesn't pattern match. Like, oh yeah, that's that thing that I'm always seeing the news bring up. Like that is indeed a current event that is indeed a political topic. It doesn't register as that. It's just like, oh, you do, huh, okay. Like, you know what I'm saying? So we need to move the Overton window.
Rob 01:21:29
Yeah, I'm a little confused about this. Cause when I talk to random people and they say, what do you do? And I say, oh, I make videos about AI. Often the next remark is, oh, about how it's gonna kill everyone.
And I'm like, yeah, that's right. Yeah. kind of, it's very much in the zeitgeist.
Liron 01:21:49
I agree that the conceptual node, great dangers, Terminator is connected to the conceptual node AI, but once you get into the conceptual node AI in people's mental model, you're already starting to talk in far mode. As Robin Hanson would say, you're already starting to just do abstract stuff.
You're just turning into the science genre, not the My Life in the next few years genre.
Rob 01:22:09
Right. Yeah, that seems true. And I don't know how you move that kind of thing into near mode. I don't know. Maybe large scale technological unemployment would do it.
Liron 01:22:23
I got an idea for you. So, you know, that concept of the missing mood that we sometimes talk about, like, hey, our rational calculations, you know, not literally calculations, but like our rational analysis, our best attempt at estimating how likely we are to go extinct is turning up numbers that are uncomfortably high.
And yet, how do we talk about it? We're like, oh, look at this interesting research paper I made the, yeah, I do this. Isn't that cool? You know, we're all just talking about it in a lighthearted tone or just a tone of curiosity or a serious tone in a way that just sounds like dignified, but nobody's running around screaming our head off, you know, or panicking, which I'm not saying panic is always good, but don't you think there's some dissonance there.
Rob 01:23:12
Yes, I don't think I can do this any other way. I generally don't feel emotions very strongly.
I am a little bit surprised that there aren't more people running in circles and screaming and shouting. That's never been my style.
Liron 01:23:36
I mean, I think we can set an example of doing some running and screaming.
Rob 01:23:39
Perhaps I think somebody should do it. I don't think I should do it. Suffice to say a less even keeled man in my position would be running in circles and screaming and shouting.
There's a sense in which that is an appropriate response. I don't think it would help. So I'm not doing that, but I don't want people to read from my calm that everything is okay, it's not.
Liron 01:24:07
All right, take a close look at Rob. Rob Miles expert here. This is actually the face that, Rob Miles makes when he is in extreme distress.
Rob 01:24:15
I mean, yeah, basically, I don't know. I had a whole thing about this, a couple of years ago where I tried to connect with this whole thing more on an emotional level, and it was very, difficult. It was very, it was kind of paralyzing actually, the enormity of the thing and the idea that I might play some non-trivial role in it.
The weight of that responsibility, was paralyzing. I mean, this is what I said. I had a video about this. You ever been on a group project at school and realized that the team that you're on is not up to it? Is like, and you have a choice of like, either you just try and do the whole project yourself, or you can you, or you can resign yourself to failing that class.
Liron 01:25:13
Yeah. Or, or you can be part of a company, right? Where you've got this command structure at the company saying, okay, we're doing this revenue, we're going to try to increase revenue. But you look around at the company and you're like, um, this is a sinking ship.
I know you guys aren't talking in those terms, whether you're not being honest or you don't know what's going on, but this is not a successful project even though I'm drawing a salary here. I think that's a pretty common realization that people have at companies.
Rob 01:25:39
Yeah, but of course you can. You can leave the company, right? So yeah, maybe the thing to do is to scream and shout if you think that would help. I don't know. I don't know. I think that there should just be more people doing more different things. Somebody should be screaming and shouting.
Protests and Political Action
Liron 01:25:59
Right, so this gets into protesting and very few people are doing it. I certainly hope to see more. I know you've been known to show up at a protest once in a while.
So I mean, I like to take that megaphone and yell at these big buildings where these AI people are going to work just because it's not like I'm emotionally feeling like they're going to listen to me in that moment.
I, you know, you and I we're clearly both the same personality type where when we think of doing stuff like that, It's always going to be a little detached. Not feeling it as deeply as I think a lot of people would, the strengths of the emotions.
But that said, even though we may be a little bit detached during the act, we still think it's the right thing to do. It's the correct action. It's the right communication that society needs to hear.
Rob 01:26:38
Yeah. Yeah. I'm pretty uncomfortable with a lot of the aesthetics of protesting. I think usually when people are protesting, in order to be the type of slogan that people can chant it, it almost has to be not quite true or something.
A lot of the time the constraints that political activism is under, hard to reconcile with the level of nuance that I think is called for on a issue like this.
There is a fairly simple message that the risk that we face from very advanced artificial general intelligence is unacceptably high, and our society is doing a terrible job of dealing with it and needs to take it enormously more seriously.
That is true from actually a very wide variety of possible angles and perspectives on the technology and
Liron 01:27:40
You know, I even think that the message is chantable. If you look at what we were chanting, last protest, it sounds like, what do we want? Safe AI? When do we want it? Later! Do we want it in a day? No way!
I think that's a pretty good chant.
Rob 01:28:01
Yeah, I mean, I really just feel like I am very much not specked into this. But I am legitimately kind of confused about where the political activism minded people are in all of this.
The type of people who think of going to a protest as a really fun time, which is not me. Where are those people? Are they all asleep? This is not okay. Are they just not paying attention?
Liron 01:28:28
I think it's going to happen, but I think it's, my best guess as to how more people are going to come out to protest is it's probably not going to be the movement that I joined early Pause AI.
It's going to probably be a movement led by somebody who's already famous and it's going to be a much oversimplified version of the movement where I probably won't even agree about their actual positions, but it'll still be kind of anti AI in a vague sense. And then it'll be cool.
I mean, if you look at cool topics, you know, climate change, Israel, Palestine, I mean these have just become the go-to things to protest and I think AI will get there at some point.
I guess I'm just hoping it gets there instantly, you know, like it's time for AI to be the it thing to protest already.
Rob 01:29:12
Yeah, I think that the problem is that there are enough appealing issues with AI that you can get a mass protest movement around them. I think you could certainly get a mass protest movement around AI unemployment. Once that really kicks in.
But that won't be about the thing that we are most worried about, right? Like AI is pretty unpopular with a lot of people. If you look at the polling, there's a lot of things people don't like about it, but risk of AI systems at some point in the future actually disempowering humanity completely and possibly leading to the extinction of the human species. And the end of everything that we would recognize as valuable is not actually one of those.
And I think you could, I am a little surprised that there isn't just a mass anti AI in general protest movement, but if there was, I suspect that. This particular concern would be a pretty small part of that umbrella. Most people on the streets would be there because, AI took their job and or romantic partner.
Liron 01:30:28
Okay. So it sounds like we're on the same page that people who are doing AI education media or being the face of AI analysis, it would help to do a little bit of screaming and shouting to fill in the missing mood to show people like, oh no, this is actually, you know, we're as adamant about this as people who are pro or anti-abortion. You know, we're as passionate about this.
It would be nice to communicate that, but you personally are a self-diagnosed Aspie and furthermore British, so it's just not going to be a good fit.
Rob 01:31:00
Yeah, I kind of can imagine. people looking at the apparent emotion of the captain of the Titanic or something.
I'm aware of how conceited it is to cast myself as the captain in this scenario. I'm not in charge of a damn thing, but, there's a kind of a stoicism, there's a kind of a, yeah, stiff upper lip for want of a better phrase of like no good getting upset about it.
Ship, definitely taking on water. Let's do what we can about that everybody to the lifeboats, but like, are you not going to get into the lifeboats if I'm not crying? You know what I mean?
Liron 01:31:42
Yeah. Yeah. But maybe you should be yelling because you know, like on flights when the plane makes an unexpected water landing or whatever, right? And they're taking out the slides, I think they train the flight attendants to really yell.
Rob 01:32:00
Interesting. Yeah. I mean that may also have practical, you know, I expect it to be loud in that situation. I guess it's loud now. It's getting louder.
Liron 01:32:08
Um, yeah, so, so I think there's something here where, you know, we gotta get the ball rolling. You know, we, we, we gotta change. There has, we have to get to a place in discourse where it really sounds like, you know, the missing mood is filled in basically. Right?
And I think it's like I'm trying to do my part at least by saying, Hey, P(Doom) is high. You think that extinction is likely soon? I think extinction is likely soon. Okay, let's make sure to talk about that. Like, we can't really have a conversation about anything else without at least pointing to the big asteroid or whatever.
So, so being explicit that P(Doom) is high, even if you're not literally yelling or being emotional, I think that's another step is being like, Hey, before we continue with this particular topic of today's episode, let me just remind you that a lot of us observers think that there's not a high chance of surviving the next decade or two.
Rob 01:33:02
Yeah, I do agree that there is a tremendous cowardice among a lot of people on this. That they are not saying what they think because they want to be taken seriously. I'm trying not to do that.
There is of course, a pressure in that direction. I think if I had been screaming and shouting from the beginning, I wouldn't be in the position I'm in now.
It's certainly easy to dismiss someone who's screaming and shouting relative to somebody who's calmly laying out the facts. I think both things should exist, and I am much better suited to calmly laying out the facts.
Liron 01:33:33
You, you give me the facts and then I will react with the appropriate emotion. When I hear the fact.
Rob 01:33:46
Okay, because these are not. For the people at home. This is not normal. We are not living in normal times. We are not in the type of situation that we've faced many times before. The standard approaches. Business as usual is not guaranteed to work. I mean, it's never guaranteed to work. It's not even particularly likely to work.
We are at a point in history where we are called upon to do something new, to actually try to solve a genuinely difficult problem that we might not be, well equipped to solve.
And it's us because nobody is, There's not some secret room somewhere of people who understand what's going on. There isn't. I've been involved in some relatively high level things, not the very top, but, the standard of the conversation there is not the highest, and the level of knowledge and understanding, it's not extraordinary.
It's not terrible. It's okay, but I don't know if it's up to it.
Liron 01:35:07
I mean the, these rooms, these conference rooms with the highest level people. Once in a while you got some real talent in there, but a lot of times you just get people who are decent, as you say.
Rob 01:35:14
Right. And they, there, there's still definitely an attitude of we're going to run the standard playbook. For technology that some people think might have some issues. And I don't expect the standard playbook to cut it.
So yeah, it would be cool to have more people wake up and start, making more noise.
Liron 01:35:40
I think you versus me, there's actually something very ironic here because I actually think that on an emotional level and on a gut level, you're farther along toward feeling that the ship is definitely going to sink. Sorry, not definitely, but likely going to sink seems to be sinking.
I think you're farther along with your emotional brain, with your gut. My gut is still convinced, no, the tech industry's fine. Like, you've got a good job. You know, I run a business, besides doing Doom Debates, and it's still paying the bills and everything is fine.
You get to play with new AI every day and like, yes, there's a lot of rational arguments, why there's going to be a discontinuity, there's going to be a point of uncontrollability, but everybody's in the same boat. So whatever, like, you know, relative to your peers, your status is always going to be in decent shape.
So everything is fine and it's a nice day. You know, the weather's nice. That's my gut. I think your gut is farther along toward telling you that you're doomed. And yet. You're also the one who's hesitant, so it's like I have to be the one to make a big deal out of it. On your behalf.
Rob 01:36:43
I mean, I don't think that, as I say, I don't think we're doomed at all. I'm not in a situation of like, ah, yes, the ship has struck an iceberg and is taking on a huge amount of water.
What I think where, where I'm at is like, yep, that's definitely a big iceberg up ahead. Yep. We are definitely headed straight forward right now. It's getting closer all the time. We don't have that long to steer. And the people who say that the iceberg isn't there are delusional. And the people who say that the ship is unsinkable, I don't know, maybe, but I don't think so. I don't really buy their arguments. And also, why don't we just steer that's not the same thing as that we're doomed.
Liron 01:37:28
Yeah, I think I have a bigger part of me that's like, this is such a fun ride, man. What's going to happen when the ship goes forward? Some cool stuff is going to happen, even though the other part of my brain is like, yeah, there's an iceberg there. We probably shouldn't steer there. But part of my brain is like, oh man, this is a great direction. I'm really excited to see what happens next.
And I think you've, I don't think that voice is helpful, but I'm just being honest because, you know, I'm getting a lot of cool tech. Okay. I am getting positive reinforcement.
Rob 01:37:52
Yeah, I, I mean, I was there. That's a lot closer to where I was a few years ago. Because this tech is extremely cool. It's so good and it can do so many fun things.
Liron 01:38:03
Right. And, and they're, and consumer products keep getting nicer. Okay. I'm just, I can't help it.
Rob 01:38:07
Yeah, I mean, I don't know the actual software. It is interesting that we have these amazingly fast software developers or whatever, and the actual software is not better. But then of course, software as a whole is terrible. The hardware. People keep bringing out better and better hardware every year, and the software just gets slower and worse written.
Software developers, what are we doing? What are we doing?
Liron 01:38:25
Yeah. Well, I develop pretty inefficient software myself, so, yeah, I currently maintain a site that has a five second load time. So let's just change the subject.
Rob 01:38:37
This is what I'm saying, man. Can you get an AI system to optimize that for you?
Liron 01:38:43
Exactly. Exactly. Yeah. That'll lead. So, so that's, and that's the kind of thing that I get excited about. Right. And that's, it takes up a lot of my attention. It still pays the bills. So, but I think this is an interesting difference between you, me and you. Because you're farther along, you're, you know, maybe you're better at spotting icebergs or, or your gut is right.
I think we're both kind of talking at the same level in terms of our rational mind, right? In terms of having an intellectual debate. But then in terms of having our gut register it, right, I've just got a bigger barrier, a bigger firewall between my gut and my rational brain.
Rob 01:39:12
Perhaps, I don't know. The relationship between my gut and my rational brain is complicated. It's subject to a long history of litigation and treaties and agreements. People should not use me as an example.
Liron 01:39:30
So the ultimate communicator would have your gut, and then my fearless displays of emotion.
Rob 01:39:40
We're not emphasizing that perhaps, I don't know. I think you need different things to get through to different people. I think if I had been yelling and screaming, a lot of the people who listen to me would not listen to me.
A reputation for levelheadedness is valuable because levelheadedness is valuable. But I think that a broad range of approaches make sense and I want there to be more people yelling and screaming so that I can disagree with them when they're exaggerating and agree with them when they're not exaggerating.
There is space for this to be a much larger discussion. Most people are just barely talking about this, and it's obviously the most important thing happening right now. We don't have to do one thing or the other. I think there should be a lot more people doing this and all of the things should be done, throw the kitchen sink at it.
Liron 01:40:28
Do you agree with the perspective of raising the temperature the same way that when you get a flu, the first thing your body does is, you know, get a fever. Like, all right, let's just raise the temperature, let's make, let's bring some focused resources on this challenge, and then we'll figure out what to do next. And sometimes you're like, no, the fever's bad. We gotta take medicine for the fever.
But do you think that society kind of needs to be in a fever state about this? Because I kind of do.
Rob 01:40:52
I don't know, I don't have a really clear concrete sense of what that metaphor would mean.
Liron 01:40:59
It basically just means people being worried. Like, I'm sorry. You know? That's why I'm saying fear mongering is good. Like the fact that days go by and people just don't think about it at all. That doesn't seem appropriate at this point.
Rob 01:41:10
Sure, yeah. The problem is. To abuse the metaphor, a higher temperature is worse for clarity and for coordination. The more people are involved, the harder it is to coordinate anything There's so many people now who have understood half the message, it's so hard to get across.
AI could be an extinction risk. People say, how you say? Well, because it could be extremely capable and extremely powerful. That's a necessary step.
And if you've convinced people that AI could be this capable and this powerful, sometimes people then just stop listening to you from there and get excited about how powerful this thing is and go on and try to make it happen even faster or something.
It's an interesting thought experiment. Imagine that it's, you know, the thirties you're a nuclear physicist. You've realized that, a nuclear chain reaction is possible. You've realized that this could create bombs of previously unimaginable power.
You realize that if this can be done, people will do it, and that a war with these weapons on a large scale would be civilization ending because, you know, it kind of follows fairly directly from the physics actually, if you just run it forward intuitively based on basic understanding of how people work and how people think, you have this insight. You're the only person in the world to have the insight.
What do you do and specifically, how do you reduce the risk of global nuclear war? Do you not just cause it to happen? Who do you talk to? What can you say to them? How do you do that? I don't know.
Liron 01:42:35
Right. So in Leo Szilard's case, right? I think he hit up Einstein and they talked to the president and because it just seemed unavoidable that the Germans wouldn't be racing toward the bomb. So they're like, well, you know, we either race toward the bomb or, or we lose to the Germans. So that was kind of their thinking.
Rob 01:42:55
Yes, it is somewhat hard mode if there's parallels to China, right? People are like, oh, China's racing. So the only solution right now is to go build AI. But of course, the, there's a couple places where the metaphor breaks down. Number one, we're not at war with China. Like at the end of the day, it's very possible to just make a handshake deal with us in China. Like, Hey, how about we do a plan where we both survive? Like that is at least in principle possible.
And then, yeah. And then number two, it's not like nukes because it's, it's not like one of us is going to nuke the other. It's just we're both going to be the victims of the AI's abilities.
Liron 01:43:33
That's right.
Rob 01:43:33
Yep. It's like a nuke that you can't aim something. And yes, we're not at war with China, and China is not, or at least they weren't racing. I don't think the people saying that actually checked anything about China's actual policy as sort of politically and ideologically useful for them. If China was blindly racing towards this thing.
They didn't look to see if they were, or if they were open to not dying. If they were open to some form of agreement. I don't know how true it is now, but it certainly used to be a lot more true.
I think people are burning a lot of goodwill unnecessarily. I don't know. I'm not an expert in international politics, to put it mildly.
Technical Alignment Work and AI Company Involvement
Liron 01:44:25
Me neither. Okay. So in terms of policy recommendations or action items, what do you think of Eliezer Yudkowsky and Nate Soares's book? If anyone builds it, everyone dies. Does that title make sense to you?
Rob 01:44:39
Yeah. I think the title does a good job of getting across this core thing, right, of like, it's not a race that you can really expect to win, it's just a race to be sort of first loser or something.
Liron 01:44:52
Yeah, the title has really grown on me. When I first heard it, I'm like, ah, this is such a typical out of touch nerd title. This isn't going to fly as a book title, but that may still be true. But the title's grown on me because it is quite a convenient thing to repeat, right?
It's like people will be saying something like, Hey, hey, remember, if anyone builds it, everyone dies. Keep that in mind. That's actually quite useful.
Rob 01:45:12
Yeah. It reminds me of the statement about that the risk of extinction should be treated by society, like nuclear risk or whatever. I don't remember the specifics, but it was short enough that it fits in a tweet.
You know, you're putting the most significant bits of your message in the title. I think that makes sense.
Liron 01:45:28
So then it's a short logical step from thinking that that title makes sense to wanting to pause AI, right? As in pause, AI dot info or pause AI US the US chapter. Like that seems to be a good movement, right? I mean, shouldn't we be yelling to pause AI?
Rob 01:45:46
That's the kind of thing where it's like, that feels like an oversimplification, like getting it down to if anyone builds it, everyone dies is kind of okay because it is ambiguous. Pause AI is interesting just because AI is a lot of things and most of the AI. I don't want to pause, like, image generators and song generators and whatever.
Those things certainly have their downsides. The fake news side of things is concerning, but again, it's the normal type of concerning that technology often is. But there's all kinds of AI things, AI applications and AI technology that I think is just straightforwardly good in the same way as other technology.
Liron 01:46:27
Yeah, so, but you know, the request of pause AI, at least most people who are part of it, they're basically saying, pause, increasing the frontier of AI capabilities. So anything that we've already survived having today, we can exempt that, we can grandfather that in, like, it was irresponsible to give it a shot because it had like a 1% chance that you'd go too far in killing you, but it didn't, okay, let's not push our luck. That's basically what we're saying. So you can keep the Spotify playlist or whatever, right? And, and I'm happy to keep those things.
Rob 01:46:54
Yeah. Yeah. But I'm also happy to advance a bunch of these things. The thing that I don't want to advance is general purpose agents, and especially general purpose agents that can help with AI research.
That's not most of AI or something. And so like, I don't know, I want to pause AGI I want to pause progress toward super intelligence.
Liron 01:47:47
Right if it was called Pause AGI, people wouldn't get it.
Rob 01:47:51
This is what I mean, where I'm not the type of person who's particularly well suited to going out and chanting things because my sign is going to be mostly footnote.
But yeah, pause AGI pause progress towards super intelligence. If you, if you look at these companies, they talk a very big game about we're making cool products. That's true. Recently they've been relatively mask off about we're trying to create super intelligence. We are trying to, like the actual goal is world domination. They didn't say it exactly in those words, but that is the thing that's not okay, but just getting incrementally better narrow AI technology that's just solving problems for us and making life better. This is great.
It's just this one technology approximately, that we don't know how to do safely. And the other stuff we also don't know how to do safely, but the actual risks it poses are more in the range of things that are usually worth it for the benefits that you get from the new technology.
Liron 01:48:49
Agreed. So just to be clear about this part, if you could push a button, and it was the law of the land of all of Earth by International Treaty that you just weren't allowed to do Frontier AI capabilities development right now, let's say Frontier, AGI capabilities development, whatever that means, which is probably going to be a tricky definition, but hypothetically in principle, would you just push that button?
Rob 01:49:21
Yeah, if you can do it, if it's a magic thing that gets you an international treaty with proper enforcement, that actually works and a well-defined, unpausing, criterion. That makes sense.
And the whole thing isn't like, God, I'm people are a problem. Anytime you try to actually do any big policy thing, people grab hold of it and twist it and they only have to twist it a little bit it's useless or actively making things worse.
And this is somewhere that I don't feel particularly hopeful. I think it would be good if we could do it, and I think everyone should acknowledge that at least. A lot of people are like, ah, you can't do this. People don't work that way. You won't achieve the thing you're trying to achieve if you push for this, which may be true, but.
That's not a reason to not acknowledge that it would be good if you could, because if you believe that it can't be done, then it can't be done, right?
Liron 01:50:17
Yes, it's important to start the conversation at, in principle, if humans didn't screw it up, what would be the ideal? And I think we're agreeing there. And then of course, once we go down the rabbit hole of the nuance, maybe we have different disagreements.
So lastly, there's this other action item that you've recommended that I am open to, which is the idea of technical alignment work. Right? I mean, that can also add value, correct?
Rob 01:51:05
Yeah, absolutely. I mean, like what if you pause what next?
Liron 01:51:10
Exactly. Right. So you, unpause once there's enough technical alignment work to be like, okay, well here's why we're convinced that unpausing won't lead to imminent human extinction. It'll lead to a better outcome with very high probability. So let's unpause it for that reason. So I agree that. That really is how things need to go. We somehow need to do better alignment.
Yudkowsky has mentioned like it'd be great to do human intelligence amplification, so we've got a smarter team of human resources doing the technical alignment. So I think we're all on the same page about that.
In your most recent video, the one about AI safety, career advice, you do encourage your viewers to go into technical AI safety. And you specifically mentioned, anthropic and Google DeepMind having these safety teams.
But do you really think that we should be encouraging people to go work at Anthropic and Google DeepMind?
Rob 01:52:00
Do you think that everyone who understands and cares about these issues should not be in the room where they can affect what actually happens?
Liron 01:52:08
So I would dispute the premise that there has to be this room where you build the AI, right? How about we just don't have that room and we lobby for the room to not exist?
Rob 01:52:17
Sure, but that is, you can't put all your eggs in that basket.
Liron 01:52:20
Right. Okay. So it's like, I agree that it's like better than nothing to send people on onto the inside of these companies that are doing something that we disagree with.
But maybe we should send, like most people to stand outside and stand in government and stand in rallying the people as opposed to being like, yes, go forth and get in on the inside, because I'm sure you'll be able to do something so productive from the inside of these organizations that are locked into this race with bad game theory. I'm sure that you being on the inside is net positive.
It's like, um, isn't that kind of like a last resort to go into the inside?
Rob 01:52:53
Well, most people can't get inside anyway, so if you can, why not?
Liron 01:52:58
You know, the term I use is tractability washing. So these AI companies, whether intentionally or unintentionally, they benefit from the frame that AI safety is a problem. That you just spin up a team and that team is working on that while you also work on capabilities.
And are you going to solve the alignment and time in the five years or whatever, however much time you have to solve it before it's too late? Are you going to solve it? Well, the answer to that question is, we're going to do our best because it's a sufficiently tractable problem that we're going to do our best.
Whereas from my perspective, it's like, wait a minute, if we're grownups about this, if we can see a high probability that our best efforts are likely to fail, that this isn't a very tractable problem, unless we get very, very lucky, then let's have the intractability discussion. Let's do what one does when the problem is potentially intractable. And they're just totally steering you away from that discussion. They're tractability washing.
Rob 01:53:47
Yeah, it's not obvious to me how tractable the problem is. I see pretty plausible arguments on both sides here, or at least that, like if you redefine the problem, such that it doesn't scale to this true superintelligent stuff, it seems relatively tractable. It's the proper super alignment stuff that seems way less certain.
Given that these companies exist and they have something called a safety team. That safety team should have smart people on it, and they should understand the nature of the problem, and they should be brave. They should be the type of people who don't pretend. That the thing is more tractable than it is. Don't pretend that the thing is safe, are happy to blow whistles when necessary.
I've been pretty disappointed by people leaving these companies clearly under a cloud and not saying anything. I think people should be breaking their NDAs and getting sued because then there's discovery and I will cover your legal fees.
Like, yeah, actually, why are we need to have like a fund to cover people's legal fees. But the other thing is like, once the thing is out, it's out. Right? But the number of people who I think know a lot of things, that should be public. And they're hiding behind NDAs mostly. Or rather they're, you know, because they don't get their stock options if, whatever.
Liron 01:54:55
Sure. So we gotta give a shout out to Leopold Aschenbrenner and Daniel Kokotajlo And I think one or two other people I'm probably missing. These are people that actually gave up their right to hold onto their vested options.
And luckily OpenAI later after there was a lot of outrage, OpenAI actually later reinstated their options, but they actually didn't sign some paperwork when they were leaving, which gave them the right to speak out and left multiple millions of dollars on the table, which they were willing to walk away from. So that definitely deserves to be commended, right.
Rob 01:55:25
Yes, I have enormous, enormous respect for, for them.
Liron 01:55:33
Exactly, me too. And it's very easy for us to be on the outside. I don't have millions of dollars in these companies stocks, so it's very easy for me to point to everybody who does and say, you guys should be sacrificing that stock. It's very easy for me to say, it costs me nothing to say that. And yet, you know, please do.
Rob 01:55:50
Yeah. Just about having these juicy details of what people were saying behind closed doors. It's very plausible that there's nothing super juicy and everything behind closed doors is just the flavor of drinking their own Kool-Aid or making decisions that are reasonable. Once you've already accepted the frame, that you're doing your best to solve a sufficiently tractable problem, there's not going to be that much juice to spill. You're just already self diluted in a way.
Rob 01:56:20
Yeah, that seems probably true. I think the general public doesn't quite realize there's just some pretty basic facts about the situation that the general public doesn't realize, which is the people running these companies think that they're building a god approximately.
Very reasonable to disagree with whether they'll actually achieve that. But that is what they think they're doing. They think that there's double digit chance that if they succeed at this, it will drive humanity extinct and they are doing it anyway.
Liron 01:57:17
Right, and we agree with their beliefs about these kinds of probabilities and risks. We're actually on the same page as many of these AI company leaders. The only difference is the part where they say, okay, well I'm going to just go ahead and steer this ship in this direction.
Rob 01:57:27
Yeah. This is why I'm a little bit confused about the lack of a mass protest movement. There are protests, but it's pretty small. I don't quite know what it is that's keeping people from realizing that.
Liron 01:57:38
So I just want to push you before when you said, okay, people should join these companies and get ready to blow the whistle, but it sounds like you, and I think that the whistle blowing point has already come. So what's where, what are you looking for when you join the company?
Rob 01:58:03
I think you have this problem with people, drinking their own Kool-Aid or whatever. You said that you get a culture within a company if everyone with sensible ideas about this has selected themselves out, the culture of the company gets much worse.
I think that you have to be there and you have to go to lunch every day and you have to say, you have to be a pleasant. Coworker, be a person who people are happy to have there. But don't pretend like you think that this is all okay.
You say, yes, I am on the safety team. I think that we are currently nowhere close to figuring out how to align a super intelligence. And we probably should stop and have that be the thing that you say. And yeah, maybe people won't listen to you, but there'll be discussions.
I think that having nobody in the room who hasn't, as you say, drunk the Kool-Aid is pretty dangerous. It's a difficult role to play socially. But I don't think we can just ignore that. I don't think we can leave that, that unfilled.
Liron 01:59:02
So nobody's job, understandably, nobody has the job description of tell the company when it would be reckless to proceed and therefore we should stop. That's never going to be anybody's job description. The job description is always going to assume, okay, you're here. Just do your best, right? Come up with the best research you can. We're not going to stop, okay? We've already decided at the upper level that we're going forward. We're not going to stop.
So it's nobody's job to tell them that it's intractable and it's time to stop. And I think it's in the nature of these AI companies, like it is what it is. If you're an AI company, you go build AI.
We have to zoom out and be like, okay, so this problem, the wedge, the leverage to solve this problem is as Eliezer Yudkowsky points out, you know, a policy solution, a grassroots political will solution. I mean, that's where we should be putting a lever right now. Not just heading into these AI companies and waiting for some moment when you can like take your action. Like I think the moment's passed.
Rob 01:59:58
This is a false dichotomy, you absolutely do both, I mean. It's not inherent to the nature of companies. You could just as well say, oh, you know, if you are an airplane company, obviously there's nobody whose job it is to say, this airplane is very likely to crash and we should not launch this product with humans on board.
Liron 02:00:18
That's definitely somebody's job. That's many many people's job. There are people like that at the airplane companies.
Rob 02:00:22
Yes. And in a civil engineering context, is there anyone whose job it is to say, Hey, I think that this bridge is going to collapse. Hey, I think this building is going to collapse. Absolutely. There is.
The fact that AI companies don't have, this is a damning indictment of the whole industry. But no, the AI companies have it on a model by model basis, right? So that analogy does hold. They just don't have somebody stepping back and saying, this whole idea that we're researching more AIs, is that a reckless and irresponsible thing to be doing as a company, right? That's the position that they don't have.
Rob 02:00:57
Sure. I guess nobody has that. But on the other hand, if your product by product people say, Hey, this thing isn't safe, and they say, cool, what do we do to make it safe? And they say, I don't know, like a five year research project. They get overruled because these companies don't actually have a sensible culture around this stuff.
Liron 02:01:16
And what you're describing happened at OpenAI in 2023, right? They started the super alignment team because the position internally was, Hey guys, I know we're scaling to super intelligence. That's explicitly what we're trying to do, and most of us agree that we are not ready to align Super intelligence.
Certainly Ilya Sutskever and Jan Leike the heads of that program, they agree that they didn't have what it takes, hence they needed to research it, and then the program failed, and then OpenAI continued. It's like, hello, what, you know, how, how many other red flags do we need?
Rob 02:01:46
It's not a good situation.
Liron 02:01:49
Right? I mean, that's definitely one of those moments that you look back and like, ah, who could have known that this direction was reckless? It's like, there, there you go. There's your history.
Rob 02:01:58
Yeah, and it's not that surprising given the incentive structure at play.
Liron 02:02:03
All right, but you stand by the position that some people who understand the real gravity and potential intractability of the problem should still, according to you, which I think I disagree with, but according to you, they should still join these AI companies.
Rob 02:02:15
I think having the companies existing with nobody in them who understands the problem is obviously a worse situation.
Liron 02:02:23
I mean, it depends how you do the counterfactual, but Okay. I mean, I think that's, you know, we said all there is to say.
Also, I'm less impressed with Anthropic now than I was then.
Liron 02:02:33
I think Anthropic is gradually being shaped by incentives, right? OpenAI started off a lot better. Anthropic is moving in an OpenAI-wardly direction.
And there's a lot of people there who are resisting that movement. A lot of good people. The movement is a lot slower than it could have been. They are doing a good job. It's still, I think, happening.
Liron 02:02:52
Right. So there's clearly a lot of people at Anthropic who their heart is in the right place. They understand the gravity of solving AI alignment before building a super intelligent AI. You could call them one of us in many ways, right? They're kind of interchangeable with the kind of people that you and I often talk to.
So on one hand, it's cool that Anthropic has these kinds of people who are truly worried, who are go to sleep, being as stressed as you and I. But on the other hand, the fact that Anthropic exists and they're still building AI, they're arguably the biggest offenders at tractability washing because if they're building AI, that makes it okay for anybody to build AI because even they're doing it, the people who care about safety.
Rob 02:03:25
I think they could do an enormous amount of good by just being open, like if this is what they think. Being open about the situation saying, Hey, we're in this situation where there's competition. We think we're better equipped to do this than anyone else, and so we have to be competitive.
However, it would be better if none of us were progressing towards this thing. And to just be open about that when talking to government and so on, try to make a visual metaphor here, it's almost like, okay, I'm currently stuck to the bumper of a car, and now the car is slowly rolling forward toward a cliff and it's like, okay, yeah.
Liron 02:04:00
You can see I'm currently walking, I'm currently jogging toward a cliff. I just want to let everybody know I'm only doing this because I'm stuck to the bumper. I really hope that the car stops and somebody comes and unglues my hands from the bumper. I just want everybody to know, that's the only reason why I'm running.
Rob 02:04:14
Sure. Anthropic should be saying something like that.
Liron 02:04:17
Exactly. And then Ilya Sutskever Right. starts safe Superintelligence There's the implication that he doesn't think that OpenAI not to mention the other companies, he doesn't think that OpenAI is building safe superintelligence That's why he has his own company called Safe Superintelligence So he needs to say that car bumper thing. He needs to say, Hey, I'm also jogging toward the cliff guys.
Rob 02:04:38
I don't know. I think it's kind of an awkward metaphor, but I think you should say something. like that. We can workshop it.
Liron 02:04:45
Exactly. Yeah. These people need to be blinking twice, like letting us know that they're in distress. Extreme distress.
Rob 02:04:52
Yeah. I'm a little worried that they're kind of, people's psychology gets shaped by the position they're in. I'm a little worried that their mind is molding itself into the shape that it needs to be in order to succeed at the internal politics of running one of these companies.
Such that they may actually have lost track of what they had planned going in or something. I don't know.
Liron 02:05:14
Yeah. I don't model people as being most people, you know? Sure they're not psychopaths, but most people, I don't model them as being actively deceptive, being like, ha ha, I think this, and I'm telling people this. I mean, they do a little bit here and there.
I do think for the most part, what people tell the outside is what they tell themselves, what they repeat in their inner monologue. If they have one.
Rob 02:05:33
Yeah. For the majority of people, I think that's true most of the time, at least. And yet they're still wrong, they're still drinking their own Kool-Aid, they're still reckless.
Closing Thoughts and Future Plans
Liron 02:05:44
I'm really glad we covered this stuff. I mean, that's what I wanted to talk about and this was a great conversation.
Rob 02:05:50
Like I said, do you know how I said at the beginning, most of my uncertainty is around how people react and what people are like, and that's something that I just didn't particularly spec into.
Usually when I think about it, I think about the incentive structures and I think about what I know of how people work. It doesn't feel good, it doesn't feel hopeful, still doesn't feel impossible, but I, when I think about trying to solve this problem on a technical level, feels doable, trying to configure a pile of monkeys such that it solves the problem, a lot harder.
Liron 02:06:26
Well, fair enough. I mean, certainly some people think one side's harder than the other side. I can't say I have a strong opinion. All right. What do you want to tell the viewers? What's coming next in the Rob Miles journey, or what should people keep in mind?
Rob 02:06:41
Yeah, I am, I'm expanding the operation a little bit, trying to, trying to hire some people and increase the rate of video production because it's always been way too slow, with some success. Should be a new video pretty soon.
Maybe I'll race you see if I can get this video out before the podcast.
Liron 02:06:58
Yeah, I'd like to see that.
Rob 02:07:01
Beat the podcast.
Liron 02:07:01
Well, thanks very much for the conversation. Really appreciate your candor. You've told people so many messages that they should think about and see how to connect those thoughts to actions. I think that'd be really great if people considered what to do next given the topics that were discussed in this conversation.
Rob Miles, thanks very much for coming on Doom Debates.
Rob 02:07:19
Thanks for having me.
Liron’s Outro: How to Support Doom Debates
Liron 02:08:21
Hey, have you guys been enjoying Doom Debates over the last year? Did you like that interview with Rob Miles? I think we're making progress toward our mission. Our mission is to raise the quality of discourse around AI existential risk and just raise awareness and move the Overton window. That AI existential risk is real and imminent, and we should do something about it.
A lot of you guys are what I consider supporters of the mission. You want to help the mission happen. Well, now it's time where you can potentially do it financially, because I'm introducing a couple tiers.
The first tier is called Superfans. It's like any other Substack: You can donate $10 a month and I will send you a free T-shirt and a P(Doom) pin. Alright, so you get some swag. You can show off that you're a supporter of the show and I appreciate it.
Now, the next level is much more serious, I'm not going to lie. The next level is called a Mission Partner. It literally means I treat you as my partner on this mission, and it does come with a minimum donation of $1,000. I told you it was going to be serious. So it's not for everybody.
I do think some of you're thinking, you know what, I take this seriously. I do have some disposable income. I'm not worried about making my rent this month, and maybe I would like to be a Mission Partner on Doom Debates.
Well, if you decide to do so, you get to be part of my Mission Partners Braintrust. It's a private channel we have on the show's Discord, where we're always strategizing how to make the show grow and succeed as efficiently as possible. Mission Partners also get early access to episodes.
So before I'm posting an episode, I usually take a day or two to get feedback from the mission partners, that changes the final episode that other people see. I'm also talking with the mission partners when we're doing pre-production, so there's a giant 12 page doc that we usually make when there's a guest coming up.
The actual show is being recorded. Mission Partners are invited to watch live, so you guys could be giving me comments in the chat room. You could even potentially real time influence how the episode is going.
The reason I call it a Mission Partner is because you're doing more than showing your support for the show. You're actually moving the needle on what's possible for the show. You're changing our budget. I mean, this literally happened.
I want to thank one of the show's viewers who will remain anonymous, but a couple months ago they donated over $10,000 to the show, which instantly took us up a level of what we were able to do.
I was able to hire a talented producer who's now been increasing the rate of content production, increasing the editing quality, increasing the guest outreach. We were able to sponsor the Manifest conference in June, and that attracted some really great guests and some good publicity.
I think we're on a great trajectory here, and the faster we go the better. Now, again, if money's an issue, don't worry about it. The content is free. On the other side of the coin, if $1,000 is chump change for you? Well, there's no upper limit. You can test what PayPal will accept.
Oh, and if you want to donate crypto, I don't have that set up yet, but just email me liron@doomdebates.com. I'm definitely willing to accept it. just is going to take me a little bit more setup. This is the beginning of the Mission Partners tier.
If you've been volunteering with me like in the show's Discord over the last few months, I've already made you a mission partner because I also consider you somebody who has moved the needle for the show's progress. You've already demonstrated that you are a partner for this mission. You've gone above and beyond the usual substack pay for fun content that you like to read. You see this as an actual mission. Actual missions take investment of resources, whether that's time or energy or money.
I hope some of you'll take me up on the offer. We're going to grow this Mission Partners Brain Trust. We're going to accelerate the show's growth. And I look forward to seeing you soon with a next episode of Doom Debates.
Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.
Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates
