Liam Robins is a math major at George Washington University who's diving deep into AI policy and rationalist thinking.
In Part 1, we explored how AI is transforming college life. Now in Part 2, we ride the Doom Train together to see if we can reconcile our P(Doom) estimates. 🚂
Liam starts with a P(Doom) of just 3%, but as we go through the stops on the Doom Train, something interesting happens: he actually updates his beliefs in realtime!
We get into heated philosophical territory around moral realism, psychopaths, and whether intelligence naturally yields moral goodness.
By the end, Liam's P(Doom) jumps from 3% to 8% - one of the biggest belief updates I've ever witnessed on the show. We also explore his "Bayes factors" approach to forecasting, debate the reliability of superforecasters vs. AI insiders, and discuss why most AI policies should be Pareto optimal regardless of your P(Doom).
This is rationality in action: watching someone systematically examine their beliefs, engage with counterarguments, and update accordingly.
0:00 - Opening
0:42 - What’s Your P(Doom)™
01:18 - Stop 1: AGI timing (15% chance it's not coming soon)
01:29 - Stop 2: Intelligence limits (1% chance AI can't exceed humans)
01:38 - Stop 3: Physical threat assessment (1% chance AI won't be dangerous)
02:14 - Stop 4: Intelligence yields moral goodness - the big debate begins
04:42 - Moral realism vs. evolutionary explanations for morality
06:43 - The psychopath problem: smart but immoral humans exist
08:50 - Game theory and why psychopaths persist in populations
10:21 - Liam's first major update: 30% down to 15-20% on moral goodness
12:05 - Stop 5: Safe AI development process (20%)
14:28 - Stop 6: Manageable capability growth (20%)
15:38 - Stop 7: AI conquest intentions - breaking down into subcategories
17:03 - Alignment by default vs. deliberate alignment efforts
19:07 - Stop 8: Super alignment tractability (20%)
20:49 - Stop 9: Post-alignment peace (80% - surprisingly optimistic)
23:53 - Stop 10: Unaligned ASI mercy (1% - "just cope")
25:47 - Stop 11: Epistemological concerns about doom predictions
27:57 - Bayes factors analysis: Why Liam goes from 38% to 3%
30:21 - Bayes factor 1: Historical precedent of doom predictions failing
33:08 - Bayes factor 2: Superforecasters think we'll be fine
39:23 - Bayes factor 3: AI insiders and government officials seem unconcerned
45:49 - Challenging the insider knowledge argument with concrete examples
48:47 - The privilege access epistemology debate
56:02 - Major update: Liam revises base factors, P(Doom) jumps to 8%
58:18 - Odds ratios vs. percentages: Why 3% to 8% is actually huge
59:14 - AI policy discussion: Pareto optimal solutions across all P(Doom) levels
1:01:59 - Why there's low-hanging fruit in AI policy regardless of your beliefs
1:04:06 - Liam's future career plans in AI policy
1:05:02 - Wrap-up and reflection on rationalist belief updating
Show Notes
Liam Robins on Substack -
Liam’s Doom Train post -
Liam’s Twitter - @liamhrobins
Anthropic's "Alignment Faking in Large Language Models" - The paper that updated Liam's beliefs on alignment by default
Transcript
Introduction and P(Doom) Reveal
Liron Shapira: Welcome to Doom Debates. This is part 2 of my two-part conversation with Liam Robins. Part one, in case you missed it, was all about the college experience in the era of AGI. Highly recommend checking that out.
Part 2 is going to be all about riding the Doom Train and seeing if we can reconcile our respective P(Doom)s. All right, let's dive in. So to kick things off, let's skip to the end here. What is your P(Doom)?
Liam Robins: So my P(Doom) is around 3%.
Liron: Okay. And we're gonna calculate how we got to 3% by riding stop by stop. And you'll assign a probability to getting off on each stop or not getting off each stop. And doing that kind of multiplication is fraught with peril, but at least it's a rough approximation.
Stop 1: AGI Timing
Liron: So let's get to the first stop here. AGI isn't coming soon. What do you think?
Liam: 15%.
Liron: Okay, so basically you're going to stay on the train with 85% probability. You're just giving a little bit of probability to maybe there'll be another winter or something. Correct.
Liam: Yeah, something like that.
Liron: Okay. I think that's actually very reasonable. I don't even think my own probability is that different.
Stop 2: Intelligence Limits
Liron: Next stop, artificial intelligence can't go far beyond human intelligence.
Liam: 1%.
Liron: So you're very convinced that surely there's a bunch of headroom?
Liam: I think so. I think that if we can get to AGI, we will probably also get to really superhuman stuff pretty shortly thereafter.
Stop 3: Physical Threat Assessment
Liron: Great. Next stop, AI won't be a physical threat.
Liam: Also 1%.
Liron: So it clearly can physically threaten us if it has sufficient intelligence.
Liam: Yes. I think if there is a super intelligent system that wants us dead, it will find a way to do that.
Liron: I agree.
Stop 4: Intelligence and Morality Debate
Liron: All right, the next stop is intelligence yields moral goodness.
Liam: I put this one at 30%, but I'm just very unconfident either way. It's just more of a philosophical position than anything that can be really empirically tested at the moment.
Liron: Well, I'm at 2%. So this is a substantive disagreement here. I just don't even see a plausible reason why AI that's programmed using, let's say reinforcement learning, that's trained using reinforcement learning. Why would it just learn anything other than that which is reinforced?
Liam: Well, if you believe in say moral realism, then there is an objectively correct morality out there that a hyper intelligent being could fall upon. And then if it in its training knows for sure that empirically, yes, this is morality, then maybe.
Liron: So you're unconfident about moral realism, right? You're giving it a window of opportunity that maybe it's true and that's why you're unconfident.
Liam: Correct.
Liron: So to me, moral realism is obviously false because I mean, unless you just want to say there's this Platonic morality that exists, but it's causally impotent, it just doesn't affect the causal world. It's just there. But you just can never know that it's there because knowing that it's there would be causally entangling yourself with it and it's causally impotent.
So only God knows, but then at that point it's becoming a meaningless concept, right? What is the connection? How does somebody causally, how does somebody that is made out of atoms access, physically interact with this moral reality? How does that work?
Liam: I mean, I think that is very plausible. If I wanted to steel man the moral realism position, I could just say that, well, look, the sort of less intelligent beings on Earth don't seem to have any morality at all. They just do what's in their immediate material self interest.
And even in more primitive human societies, morality basically just revolved around, you know, what's good for my family or my tribe. It seems like as we've gotten more intelligent and had more advanced moral systems, we've gotten things like moral universalism that remains true no matter who you are, and where even people in wildly different situations and with wildly different self interests can still converge on.
Okay, this is a moral course of action. And maybe if you just kept scaling intelligence, then you would find even more agreement in that vein.
Liron: So to be fair, I think a lot of people listening, a lot of the population, and even Scott Sumner, who's clearly very intelligent. What you just said now has a lot of people who think that you're making sense, even though from my position you're obviously not. That's why it's the difference between 2% and 30%.
So just to repeat back what I'm hearing, you're basically saying, I believe there's a trend where smarter people and smarter animals seem to behave more morally. And a candidate explanation for that trend is that they're somehow interfacing with this fundamental thing in the universe that is making them be moral and they're establishing a causal connection.
So their neurons have essentially a physical connection, some law of physics that says that a neuron in a brain can tap into moral reason. I mean, what exactly you're acting like there's a connection between your observation of smarter people being more moral and this hypothesis that the universe fundamentally has morality. But my question for you is, what's the mechanism of connection here?
Liam: Trying to think how I would explain this. And again, my background is not philosophy, so I'm sure that other people could explain this much better. But I think that if you truly understand just the nature of sentience and how that feels to people, you might just inherently, at that point, want to say that, okay, sentient beings should be treated well and should not suffer.
And also, if you understand the scale of how strongly each sentient being feels and the number of each of them, you might converge on a moral system closer to that.
Liron: But don't you think a handful of psychopaths exist that are smarter than 99% of the human population and they truly enjoy watching people die and torturing them? So the existence of those people, doesn't that reflect or doesn't that imply that you could similarly get to an AI that is similarly psychopathic and there's no fix.
Psychopaths don't one day wake up and it's like, oh, I was wrong, let me change. They're just psychopaths their whole lives.
Liam: Sure. I think that is plausibly true. In fact, I think that's more likely true than not. I just think that the converse that, you know, more likely than not an AI will just generally be oriented toward the good. I would put that at 30% probability.
Liron: So it sounds like we can distinguish the strong orthogonality thesis from the weak orthogonality thesis. It sounds like you're pretty on board with the weak orthogonality thesis, which says that it's possible in the space of algorithms to have algorithms that are like human psychopaths except even more intelligent. An arbitrarily intelligent psychopath, that algorithm does exist.
And the only contention between us would be the strong orthogonality thesis, which is the question of how likely are we to go and pluck that algorithm out?
Liam: Yeah, sure. I'll say that, you know, among the smartest human beings on the planet, probably upwards of 95% of them are generally moral and upstanding people, and probably less than 5% are psychopaths. And so I think that just if you have this really super intelligent AI system and also moral realism is true then, or true in some sense, then yeah, 95% chance that the AI will do what it knows to be morally correct.
Liron: Okay. If you look at why psychopaths exist in the human population, I think it's pretty convincing to assert that they exist because of this game theory explanation where it's like if you're playing the prisoner's dilemma and you know, tit for tat is a good strategy where you cooperate with people who cooperate with you and otherwise defect.
But once you get a society of people who are very willing to initially cooperate, there's this other strategy that emerges where you can exploit them a little bit by playing one shot games with them and then defecting more. And that's why you get a small percentage of psychopaths - they do well in societies where there aren't that many psychopaths.
So don't you think that would be a really convincing explanation why you can just have the first AI come online and just have its way with humanity, throw us in the garbage?
Liam: I mean, that could be true. Again, I'm not an expert in evolutionary psychology, but I feel like if that was the only reason, then we would actually see far more psychopaths than we really do. We would see far more people who maybe outwardly pretend to care about morality, but really they are just psychopaths.
Liron: I think the logic of the game theory there is, and I'm not an expert on this, but once you get too many psychopaths, you get a low trust society. So then people check more rigorously whether you're willing to cooperate the first time. But once trust levels get really, really high, then somebody born with no moral compunctions does better.
Now don't get me wrong, I actually think that if you're a psychopath and you're not extremely intelligent, you're likely to eventually get caught and eventually leave a sour taste in people's mouths. So it's not that easy to be a successful psychopath. But the optimal number of psychopath gene frequency in a population is a small number.
I find that very plausible because you can always take it to the extreme, right? You can imagine a really, really high trust where there's so much value created from the high trust, but at the same time you're increasing the incentive to have a psychopath come in and exploit it. So it does seem like that makes a lot of sense to me as an equilibrium.
And remember, the reason I'm bringing all of this up is because when you're observing the statistics in the human population of people's moral tendencies, keep in mind that the statistics very much come from these kind of game theory strategies. And also specifically, an AI is going to be more powerful than a human psychopath. So an AI doesn't really have dependencies on people the way that a psychopath still needs other people to help him out.
Liam: All right, all right. Again, this is not a very strong conviction I have, and I think you could plausibly argue me down to 20% or even 15% that intelligence yields moral goodness. I just think it's much more plausible than you're giving it credit for.
Liron: This is probably the first time in the podcast one year history where somebody has said, you know what, I am going to significantly update one of my belief probabilities. Wow, good on you.
Liam: Well, maybe I'll caution that until I do that, I would want to talk to someone who's really pro. Sure, you know, intelligence yields moral goodness and then see if their argument is more convincing to me.
Liron: Yeah, I mean, just so take it for what it's worth, right. From my perspective this is hardly even a question because this whole concept of morality, we have a causal explanation for how it got into our brains, right. When we reflect on why we're moral, it's really not mysterious. We can explain, you know, non zero sum dynamics.
If every time that I gave you a dollar, I had to lose a dollar. If there's no such thing as a trade that was mutually beneficial, then we wouldn't have evolved morality and we would become less moral. When you put humans in a situation where all the trades are purely zero sum, very quickly, you see that they learn to not be generous.
So we can see the pressures that led to this equilibrium of what we call morality. And it's not a hard counterfactual from my perspective to be like, okay, so without those pressures, you don't get the morality we're familiar with. I don't even get why some people are still hanging on to this idea that intelligence yields moral goodness. But yeah, I mean, that's just me, right? I guess we've kind of beaten the dead horse.
Liam: Yeah, I think we can move on.
Stop 5: Safe Development Process
Liron: All right, next stop on the doom chain, we have a safe AI development process.
Liam: So I put this at 20%, but I'd like to clarify what I mean by that because the statement itself, I think, was a bit unclear. So I basically reinterpreted this as saying that, okay, assuming that a misaligned AGI emerges and maybe tries to take over the world, will we be able to stop it in time?
Liron: Right.
Liam: And that's a different question from are the AI companies acting responsibly? Which I think anyone who's followed the news can tell you that's not really true. But even a company that's acting irresponsibly and is cutting corners on AI safety could still maybe pull through and, you know, stop the bad AI should it emerge.
Liron: Right, right. So the safe development process could just be like, let all the different companies do whatever they want to do with minimal oversight, but then just notice if they're getting out of control and just go order them to shut down. And even just that process of keeping an eye on them and telling them to shut down, maybe that is a sufficiently safe process if things don't get out of hand.
Liam: Yeah, yeah.
Liron: I mean, 20% seems a little bit optimistic, but I don't think it's crazy. Right. I think this is an area where there's major unknown unknowns. Even when you said 15% AGI isn't coming soon, I think we have to allow for that tail risk or tail opportunity where it's like, you know, things just slow down and become comfortable.
In some sense, we're lucky to even just had these last few years where it hasn't taken a tiny amount of time. At least we have a few years to watch it play out, and that's already better than our worst fears. So maybe it'll just take a long time and we can get scared and stop it and things just won't be too fast.
So I agree with you there. 20% is not crazy. I don't have a huge quibble with that. You do agree with me that it's not like off buttons are that natural of a thing to expect, right? When Neil Degrasse Tyson is like, I'm not scared because I can just shoot the AI with a gun. I think that doesn't make sense, right?
Liam: Yeah, no, that does not make sense. It will be substantially more involved and more complicated than that, which is why I don't think it will work in all likelihood. But I think it's plausible.
Liron: I mean, I naturally just compare it to a computer virus in order to get rid of the off button intuition, right? Does a virus have an off button? If it was explicitly programmed in, yes, but those off buttons often have a way of breaking.
Liam: Yeah, I would agree.
Stop 6: Manageable Capability Growth
Liron: All right, next stop on the Doom train, AI capabilities will rise at a manageable pace.
Liam: Again, not entirely clear what that statement means. I'm just going to lump it in with the last one and say that, yeah, should something bad happen, we'll be able to manage it. 20% exactly.
Liron: And I should really clarify for the viewers, when I split up the stops in the Doom train, I don't think that I've done such a precise job that you can treat them as independent events in this rigorous mathematical sense and you can multiply them together. I just don't think it'll work like that. It would be really awesome if it did.
Maybe AIs can get really good at factoring real complex reality like that. But I suspect they wouldn't have 11 stops like I do. They would have 11,000 stops. They would really make a dense graph.
Liam: Yeah, conditional on you've done this very specific thing in this very specific order, what comes next?
Liron: Exactly. And I think that's how real brains work, right? Is they just have so many neurons that model so many different things in a way that you just can't print out a short summary of. And that's kind of how the real reasoning gets done. That's what we've learned from these large language models having all this common sense at a massive scale.
All right, well, let's just move on then, because we did kind of talk about AI capabilities rising at a manageable pace when we talked about other stops.
Stop 7: AI Conquest Intentions
Liron: All right, so let's move on to AI won't try to conquer the universe.
Liam: So I now have this at 25%. Previously I had it at 50%. And in particular I split this into three separate possibilities of why I would not try to conquer the universe.
Possibility one is just that the AI is not truly agentic. It does not have any inherent motivations or goals that it wants to pursue. Possibility B is that it has goals, but those goals are simply just aligned to human goals by default. And then possibility C is that its goals are what I call non expansionary, where it wants to do something or maybe take over part of the universe, but will leave the rest of the universe alone.
Liron: I agree that AI totally will be agented because we're already seeing that today, right?
Liam: Yeah. And I put that at 2% that it's just not going to be agentic, which, yeah, seems pretty negligible in my mind.
Liron: Right, right. And to take your third one, the idea of misaligned but not expansionary, even 8% seems a little bit optimistic to me that you think there's an 8% chance that we can escape that way. But I mean, 8% is low enough where it's really hard to quibble.
Let's focus on number two, where you think there's a very significant hope, or 7B as you call it. There's a very significant hope that by default AGI will be aligned to human interests.
Liam: So, okay, yes, I currently have that one at 15%. As of three weeks ago, I had it at 40%. But then I read Anthropic's agentic misalignment paper and I'm like, yeah, this already just countered a bunch of the counterarguments I was going to make. So I'm down to 15% after reading that.
Liron: Well, good on you for updating. And so, I mean, you're clearly not ideological. You're actually trying to balance a mental model. This is actually one of the key learnings of rationality and of Bayesian reasoning is an intelligent agent, a rational agent.
You go into the world and you're always doing this dance. You're like the dance of a martial art where evidence keeps streaming in and you keep trying to do the steps of the dance that are the right steps corresponding to the evidence that's coming in. That's how you do it. You shouldn't expect to plow through the world with this unchanging worldview. So good on you for that.
Liam: Yeah, I think that some people started off with just this prior that alignment by default totally couldn't be true, close to zero percent for, you know, reasons that Yudkowski gave 20 years ago. I started out with, you know, alignment by default is probably true just because I didn't think there's really any a priori reason why that shouldn't be the case. But then just empirically we've actually seen, you know, people trying to test more and more advanced AI systems. And yes, experimentally we've seen actually they are misaligned.
Liron: Now there's a way that I would factor this differently from how you factor it because my next Stop on the Doom Train is Super alignment is a tractable problem. And when you talk about alignment, I think it's the same thing as me talking about super alignment. We're just talking about the alignment that matters.
Not oh alignments. Have I made it not say curse words? That doesn't matter. We care about the alignment of when it's super intelligent, what it actually does with the universe? So wouldn't you then say AGI will be aligned to human interest? Wouldn't you just go put that probability into the next step and not in the AI won't try to conquer the universe step?
Liam: I think I'm making the distinction between alignment by default. That barring a really concentrated effort to align AI systems, they just will basically be aligned to what we want them to do. And then the second question is, okay, even if it starts out misaligned by default, if we put enough energy and resources and thought into aligning it, can we do that?
Liron: Hmm, I see. So you factored it into alignment by default versus not default, but then we solve it. Yes, I see that. That's an interesting distinction. Right? Will it be aligned if we don't try? And then okay, maybe it won't be. But then if we do try. Okay, I see what you're saying. So you factored it that way. Yeah, I mean that's a reasonable factoring of it.
Stop 8: Super Alignment Tractability
Liron: All right. And you've now lowered your probability to 15% that it'll be aligned by default. Okay, I mean that's getting into the low range where I don't feel like I have to argue with it too much. So then going to Super Alignment is a tractable problem. The next stop on the Doom Train. Are you still at 20% for that?
Liam: Yes, I'm still at 20% for that one.
Liron: I mean, I think that's reasonable. I certainly think it would be tractable if we just had a lot of tries, right? If we can just save the state of the universe and reset. Oh, let me try to align it again. Again and again and again. Eventually we'd get it right. It's just what makes it dangerous is just that you're overconfident once and then it's game over.
Liam: Yeah, that's one of the reasons I'm actually, I'm not even confident that if we had a ton and ton of tries that we could do it. It might just be a totally intractable problem where if you have a less intelligent species trying to align a more intelligent species. There's just no way that that could ever possibly work.
Liron: Yeah, I'm not going to quibble with you too much there, but let's go to the next one here.
Stop 9: Post-Alignment Peace
Liron: Once we solve Super Alignment, we'll enjoy peace.
Liam: That's a very rough 80%. That's an 80% based on vibes.
Liron: Uhhuh.
Liam: I'm tempted to put it higher than 80%, but.
Liron: Wow. So you really think that if we just have. If anybody has the ability to use an AI which is aligned to them, then you think the balance of everybody using their aligned AIs will be peaceful enough?
Liam: Pretty much, yeah.
Liron: I mean, it's hard to reason past that point certainly, because I personally am like, wow, that is a better scenario than I hope for. The obvious argument for me that we wouldn't have peace is because everybody, I model it as everybody has this magic wand where they can basically say the state of an outcome and then snap their fingers and the outcome happens.
But it's kind of like this genie that's like, okay, you got that outcome that you wanted, but you know, when it was helping you. Maybe this is the premise of the problem, but when it helps you, what if it doesn't care about other people as it's helping you and you kind of screw over other people when you're getting your wish?
Liam: I mean, that's possible. I think just one way we could maybe avoid that is say there's one really giant AGI controlled by the US Government, let's say, and that AGI basically just watches over everyone and you are allowed to have a personal AGI that does everything you want unless it, you know, negatively impacts somebody else, at which point the bigger AGI steps in and, you know, does what governments always do and it mediates interpersonal disputes.
Liron: Right. So I agree. In a situation where Anthropic races ahead solve Super Alignment and says, okay, we're going to have this one singleton, it's going to rule the world, but it's going to be like a client server model where it'll just respond to requests from everybody and just balance everybody's priorities. And that'll be how Heaven works.
It'll be like this computer time sharing model of Heaven that is actually plausible, assuming that the AI really understands the assignment of what it means to have a flourishing heaven and distributes it to every human like that.
Now the part that is more scary to me is imagine that the United States has one of these highly aligned superintelligent AI, and then let's say China has one of these aligned superintelligent AIs. But if the United States says, hey, super intelligent AI, do you think that we could just gain dominance over China?
And it might say something like, well, right now you guys are a little bit ahead in the lead. I'm actually a little bit more powerful and intelligent than their AI. So yes, it is actually possible for me to go do a strike and permanently seize control of the future for the US.
This is why I'm not convinced about the. Once we solve super alignment, we'll enjoy peace, right? I think any kind of asymmetry can create really massive instability.
Liam: I mean, I think the scenario you're outlining where there's multiple competing ASIs and they're of roughly equal strength to each other and it takes a substantial amount of effort for them to battle against each other. That seems like a very negligible possibility in my mind.
I just think that the first most advanced ASI will just totally take over and it'll totally be able to wipe out any competition and you know, whoever controls that will be able to set the course of the universe going from there.
Liron: That's an interesting guess about how things will play out, right? So the idea is if we and China and I don't even know what other country is getting really close to.
Liam: Saudi Arabia, Europe, either an American led ASI or a Chinese led ASI or just a misaligned ASI, any of those three are way more probable than the competing ASI possibility.
Liron: Right? Because the idea is just that if a bunch of people have their own super powerful ASI that only cares about them or their country and not other countries, even in that scenario, you'll pretty quickly have a lead open up where that lead just becomes permanently dominant.
And then in that case the aligned AI will know that the winning country cares a little bit about the other countries, so it shouldn't go torture the other countries. And then we all live happily ever after, some of us more happy than others, but we're all pretty happy.
Liam: Yeah, pretty much, yeah.
Liron: I mean I do find that plausible. Personally, my main concern is that I think that all of humanity will be left in the dust soon, in the near Future, once the AIs take off and recursively self improve and become uncontrollable. So I don't even think we'll get to fail in this cool way where some of us have a helpful ASI and then there's chaos.
Liam: Yeah, I mean, again, this is conditional on aligned ASI, right?
Liron: Exactly, exactly. Okay, very fair. So you're putting 80% at once we solve super alignment, we'll enjoy a peace. Seems a little high to me, but you've explained it reasonably.
Stop 10: Unaligned ASI Mercy
Liron: Let's go on to unaligned ASI will spare us that set.
Liam: 1%. That I think is just cope at that point.
Liron: Yeah, yeah, yeah, yeah, yeah, I agree. And you know, even though it's cope so many people on my show, I think Mike Israel is probably the most notable guest to be like, imagine you have an ASI that only cares about self preservation. It's still going to want to study us. It's not going to want to disturb the planet with 8 billion humans on it and you're like, hell no. Right. You know, okay, I agree with you.
Stop 11: Epistemological Concerns
Liron: Then let's move on. Okay, this is the last stop on my doom train, which is AI. Doomerism is bad epistemology.
Liam: So for that one I say a couple of things, sort of, yes, it's just very hard for any of us to wrap our minds around this. It's hard to know who exactly is an expert in this. And in fact, trying to forecast the future of AI requires expertise in a lot of fields that it seems like nobody is an expert in all of those fields.
And also just forecasting the future in any event is pretty difficult. Plus for some of these questions, we've started to be able to empirically test them. But for a lot of them it's still just entirely speculative. So I just don't put a lot of stock one way or another in anyone's ability to predict the future on this.
Liron: So I agree with you that forecasting in general is hard and it's hard to be 90% sure or even 60% sure about complicated things. But that doesn't mean it's bad epistemology. Right? Using a prediction market and be like, I wonder what the probability that this war will break out or that war will break out. I don't know. It's hard to know. That doesn't mean that the act of trying to predict is bad epistemology.
The people who argue it's bad epistemology, they say stuff like, you can never predict the end of the world. Any doomer is always wrong. You have to be an optimist. I mean, those are the kinds of arguments that are attacking the very epistemology.
Liam: When I say AI doomerism is bad epistemology, AI anti doomerism is also bad epistemology for the exact same reasons, saying it either will or won't happen. You run into the same issues where it's just very hard to predict these things. So I don't think this would update me one way or the other.
Liron: Right. Okay. I mean, this is getting nitpicky on my part, but there is a distinction between how you're operating epistemology and declaring what types of reasoning are valid compared to disagreements about the actual beliefs. Right. Object level versus meta epistemology level.
I think you and I are both on the same page that it's perfectly fine and normal to apply the epistemology of prediction the way we'd predict anything to predict doom. Right. I think you and I are on that page. We can just disagree on well, who's doing a better job of predicting doom?
Liam: I agree.
Liron: Okay, great. Yeah. So you don't really get off on the epistemology stop then. It just seems like you get off on some of the other stops, which are about the actual object level belief. Some of them aren't well supported, and as a result, you don't have a super high or super low P(Doom).
Bayes Factors Analysis
Liron: So remind us, where does your P(Doom) net out?
Liam: So, okay, if you actually multiply all those numbers together and at this point you can show a picture of the flowchart I made. If you multiply all those numbers together, it shows my P(Doom) at 38%, which is not that out of the norm from your P(Doom). I don't believe.
Liron: Right? Yeah. I mean, 38% is totally in what I call the sane zone. I'm happy with anybody who says anything from 10 to 90%. It's like, yeah, close enough. I don't see that much value in quibbling from, let's say somebody with 20% versus 60%. It's like, yeah, whatever.
And when I say my 50%, I don't think somebody whose P(Doom) is 40% is being dumb at all. Right. If anything, maybe I'm dumber than them. I can't tell. I just think that somebody says their P(Doom) is 4%. I just strongly suspect that they are, in fact being dumb.
Liam: So here's the main thing where we differ is this is my sort of inside view from first principles. Attempt to make my prior belief about P(Doom). And if I was just in a room by myself where there was no other network of of people or record of historical events, this is the number I would come up with.
But I think there's several factors that I would then apply to that 38% number that then substantially drop me down. And I list three of them in my blog post. Number one is just the general heuristic that humanity won't be doomed by the scary new technology.
You know, because plenty of people have predicted doom at various points in the past and none of those previous doom predictions have panned out, which gives me some reason, not a very strong reason, but some reason to lower my P(Doom) from what it would otherwise be.
Then more importantly than that, superforecasters think we'll be fine. When super forecasters have been asked this question of what is your P(Doom)? They put very low numbers. I believe I read off the top of my head. But something like, you know, one in a thousand is the median super forecaster estimate for AI doom by the year 2100.
Liron: Yeah, well, I might quibble with that, but maybe get to your third Bayes factor that you want to apply.
Liam: The third is that insiders in leading AI companies as well as in the government seem to think that we'll be fine, at least by outward indications. And I know that this is a bit more fraught because obviously they're incentivized. You know, at a leading AI company, you have an incentive to cover up doom or to sort of ignore the problem.
But I still think that people at leading AI companies are not suicidal. And if they really thought that, you know, AI doom is likely to happen, then they would say so and they would not be rushing forth as quickly as they are.
Liron: All right, so those are your three factors. And again, it's great that you're reasoning like this. You know, anytime you can make your reasoning explicit so we can look at it, I do think that's a productive exercise.
I have to caveat though. Just like the stops on the doom train don't straightforwardly multiply out the way independent events do, there's really no simple math you can do with them, unfortunately, because it'd be nice if you could. Similarly, when you're introducing these extra Bayes factors, right, you're bringing in smuggling in a lot of mathematical assumptions, right?
You're saying well, these are independent Bayes factors. And also they're kind of the only Bayes factors we need to consider. Because here, let me lob in a Bayes factor. Right. How about the Bayes factor that Earth regularly has extinctions?
Liam: I wouldn't say regularly. You know, every couple it's had six.
Liron: Yeah, but every time something gets interesting, anytime there's some global phenomenon happens, there's a significant probability that it's extinction related. For example, when the atmosphere changed to be oxygen rich, I think that drove one of the extinctions.
Liam: I mean, that would definitely update me on, you know, for some reason, humanity might go extinct sometime in the next thousand years, but humanity will go extinct of this particular thing in the next 20 years. That does not seem particularly strong evidence to me.
Liron: So, you know, my meta point here was just that you've selected these three Bayes factors and you've just thrown them all into the stew. But I bet if you and I both brainstorm for a while, we could probably think of something else to throw in the stew which would multiply you back up.
So this idea of let me think of a few things to throw in and we'll multiply, you know, maybe it's better than nothing. It's an interesting exercise, but it's not going to yield this answer that you can trust. Correct.
Liam: I mean, I agree. I'm very not confident in my predictions. If I were to put 95% error bars on my P(Doom), those 95% error bars would go everywhere from probably one in a thousand to 80% or 90%. I just think that I didn't deliberately choose these three factors because I wanted to bring my P(Doom) down by an order of magnitude.
It's just that these are, I think, the most relevant factors that I would put on top of just my initial.
Liron: What if you forget the math for a second? Right? Just now you've done the analysis, you have trained your mind to have all these considerations. They have carved paths into your neurons. But now just close your eyes and just let your intuition respond. Right. Having synthesized everything, what is your intuitive P(Doom)?
Liam: Well, my intuitive P(Doom) is 3%.
Liron: Well, that's interesting that the, you know, taking your 38% and then applying those three Bayes factors got you to an end result that's consistent with your intuitive P(Doom).
Liam: I mean, it's definitely interesting. And I can fully understand from your perspective, oh, am I just engaging in motivated reasoning here to, you know, give some reason to explain what I believed previously. But I think it's actually the causation goes the other way. Around where I was already thinking basically with these Bayes factors, and that's why my intuition was 3%.
Liron: Right. Okay.
Liam: Yeah.
Historical Precedent Analysis
Liron: I mean, that's fair enough. It's possible that you have, in fact, spelled out the moving parts underlying your intuition or a good approximation of those. So let me go tackle those particular Bayes factor, because I think I've already touched on the point of there's so many other factors we could mix in this do. So you got to make sure that you've selected all of them that matter.
But let's just go one by one. And maybe that's. My other argument is that I'm not on board with the individual factors. So when you say humanity won't be doomed as a good historical heuristic, so my beef with that is, isn't that it's wrong. I mean, clearly it's right. So one beef would just be the selection argument or the anthropic argument of how could we possibly in a position, be in a position to be like, oh, yeah, extinction of humanity happens all the time.
Right. I mean, we can only look to other evidence.
Liam: I mean, I don't. I'm not sure whether multiverse theory is true or not, but it's totally possible that we're just living in an unlikely multiverse or an unlikely universe within the multiverse where humanity lived to the year 2025. And actually, there's plenty of universes in that same multiverse where we got extinct long before this point. So that's my least strong of the three Bayes factors.
Liron: If you want to treat something like a Bayes factor, which. And by the way, I think when you say Bayes factor, I think this is basically the concept of a likelihood ratio or an odds ratio. Right. So if something is to be precise, if you have two competing hypotheses and the observed evidence is three times more likely in a world where hypothesis two is true compared to a world where hypothesis one is true, then you get to multiply your current belief odds ratio by three.
This is actually an intuitive way to do Bayes, where you do actually get to multiply these factors. And it actually, the math is actually kind of simple. And it sounds like you're kind of turning to similar math when you're throwing in these multipliers. Right, sure.
Liam: And in particular, my Bayes factor of humanity won't be doomed is 0.75. So, whatever odds. I had just intuitively that humanity will be doomed. I'm just going to multiply that by 0.75 to say, well, this heuristic says it probably won't be.
Liron: So if you're treating us not having already gone extinct as a piece of evidence, you have to make sure that it really qualifies as evidence. Meaning in a world where humanity goes extinct in 2027, in that kind of world, what would you observe leading up to 2027 and then compare, in a world where the humanity doesn't go extinct for a very long time after 2027, what would the world look like leading up to 2027?
And in both of those worlds, wouldn't you still just observe, hey, humanity's never gone extinct before?
Liam: Yeah, yeah, you are right about that. I think, okay, I was probably incorrect to call it a Bayes factor because that's not properly Bayesian reasoning, but I do still think I should factor that in somewhere in my analysis.
Liron: There's some way to steal in the point, right? The idea of, look, every day we all wake up and we're still alive, and it's just rare for us to be like, nope, you're definitely dying tomorrow. But, I mean, the problem with that logic, though, is that, you know, imagine you just got sentenced to be hanged, right? And you're hanging day tomorrow, right? So it's just at some point you just have to ignore that logic.
Liam: I would agree, I would agree. And maybe you've convinced me to bump it up to, I don't know, 0.8 or 0.85 instead of 0.75.
Liron: Okay.
Liam: But I still think it's worth considering.
Liron: Game of inches over here.
Superforecaster Analysis
Liron: All right, so the next one is super forecasters think we'll probably be fine. So this is always interesting to look at, right? Because we know that across many domains, super forecaster, it's a real skill. It's you got to respect super forecasting ability. So if somebody comes up to me is like, hey, the super forecasters are unanimous agreeing or almost unanimously agreeing that P(Doom) is super low.
You're saying just 0.12%. It's hard for me to dismiss that. Right. I don't want to offhandly dismiss the super forecasters. However, I think that there's suspicion on which groups of super forecasters are doing this when. So there's a Max Tegmark tweet from a couple years ago where he says one of the world's best forecasting groups Samutsveti does that mean anything to you?
Liam: Yes, it does.
Liron: Okay, so that's a forecasting group. They just estimated that the chance of AI catastrophe is 32%, dropping in half if there's good safety regulation. So maybe it could be as low as 16%. So it's not like all groups of super forecasters are all unanimous, right? I mean does that, is that a fair assessment?
Liam: I would want to spend some time reading this particular paper because this is news to me and if it really does turn out that like actually there's a substantial portion of super forecasters who do believe in AI doom or that one study we've been referencing it, it was a study in 2023 where they got a bunch of superforecasters together and their average P(Doom) was 0.12%.
But if I do find evidence that either that study was only one small slice of the super forecaster population or even those super forecasters have since updated their P(Doom) upwards over the last two years, then I will substantially change my mind based on that.
Liron: Yeah, another thing I've heard, you know, so I don't consider myself authoritative on what these super forecasters are thinking. So I am open minded about it. I should update to be more like the super forecasters. But another thing that I've heard is that a lot of these super low predictions came before the chat GPT revolution.
Liam: Yeah, that is true. And I would not super trust any prediction made before say mid 2023.
Liron: Yeah. All right, so we'll put a pin in it. I mean look, we're, it sounds like we're both open to being convinced. I think both of us feel bound by rationality, by actually incorporating evidence to defer to a bunch of thoughtful super forecasters.
I guess the reason why I haven't gone and researched the official super forecasters is because I just feel like I follow enough people on social media, namely Twitter, where I do feel like I'm hearing a lot of perspectives and I don't feel like we're totally not doom. Doom is less than 1%. I don't feel like that is the unanimous consensus of people that I respect.
Insider Knowledge Analysis
Liam: It's definitely not in the level of crackpot doom predictions in, you know, that we might say some randos thought that the Internet would make humanity go extinct. This is actually a substantial portion of experts of people who've say maybe quit leading AI companies, people who've thought a lot about this even you know, Nobel prize winners are saying that yes, this is a very serious possibility.
So I'm not obviously I'm not saying it's ridiculous and this entire thing is, because I take it seriously. I do want to come to an informed conclusion. I guess. One area of disagreement, just in a very meta level between you and me, is that I think that no matter how strongly I hold a prior belief that, you know, humanity is going to be doomed for X and Y reasons, if that belief is just not shared by the experts to the extent there are experts, if the experts seem to be almost unanimous in saying that that is not true, then I think I'm obligated to drastically reduce my total P(Doom) just on.
Liron: So now you're talking about that third Bayes factor now, right?
Liam: It's the second and third combined. I think that super forecasters have some level of expertise and also the people in the inside making it have some level of expertise.
Liron: Yeah, because the third Bayes factor you mentioned is AI developers and top government officials seem to think we'll be fine. But that doesn't seem like how I would characterize AI developers and top government officials. I mean don't get me wrong, some of them do. Absolutely, absolutely. Right.
Yann LeCun seems he's actually explicitly said his P(Doom) is less than 1%. So yeah, he score one for the non doomer side. But I mean remember as you mentioned yourself, right? The center for AI Safety, one sentence statement on AI risk, tons of luminaries, right? Even Sam Altman, Dario Demis Hassabis. I mean key people are at least saying that P(Doom) sounds like it's at least a few percent or they said, it's at least like the risk of nuclear war and pandemics. Right.
So why are you characterizing. When you're saying AI developers and top government officials seem to think we'll be fine, you mean maybe half of them, right? Not all of them.
Liam: So. Okay, I think it's worth clarifying what we mean here. First of all, when I say top government officials, I mean the government officials in the upper rungs of the intelligence agencies who have privileged access to leading information. I don't think that the average member of Congress say has a better view of P(Doom) than I do.
And similarly with AI developers, you know, the average person with privileged access at a high level to how these systems are working. I'm not even sure that the modal coder at OpenAI or at Google even has very good inside knowledge that somebody else wouldn't.
Liron: I feel like privilege access is a very small factor epistemologically these days, because if you think about how much we know today, I don't think there's a single person alive in, even in 2023 who already had as much of an advantage as somebody like you and I who have no quote, unquote, privilege access. Right?
But even the most privileged person in the world with the most intel in the world in 2023 probably didn't know as much about the trajectory as ordinary people know a couple years later.
Liam: I would agree with that to some extent. Partially why I think privileged access is so important is that we are starting to hear reports that, oh, AGI's achieved internally. And I know that that was a false alarm the first time it was said, but it's very plausibly true that actually somewhere within these companies, AGI or something very close to it exists.
And if you see that and you see how it's actually working, you know, I think that can tell you to a very substantial degree whether risk is real or not.
Liron: I think I get what you're saying here. You're basically saying there's people who are farther along, looking farther into the future from us because they have inside access into these AI companies. And you would think that if doom were coming down, they would spot it, right? They're in the crow's nest of the ship. They would spot the doom and they'd sound the alarm.
So there's some evidence of the fact they haven't sounded a major alarm. But on the other hand, it seems to me like people who have that kind of privilege access are acting a little bit freaked out. I mean, I would actually so, to be fair, Dario and Sam Allman, those two particular people, seem to be downplaying the extinction threat. But we did get a warning from Dario that the jobs are likely going away.
Liam: Well, I think my p. There will be mass automation and unemployment. That's way higher. Nothing that I've seen really convinces me against that.
Liron: So I do think that it's. It is rational to incorporate the evidence of, okay, the Sam Altmans and Darios and the Demis Hassabis of the World. They're not yelling about doom the way I am on my podcast. They're not fear mongering. I do think that's worth something.
And then, of course, you should make another adjustment to be like, well, them personally, it's a good Hail Mary bet for them, right? Because they are going to live a great life if it goes well. And if it goes badly, well, what can they really do? It's already a race, so they might as well optimize for the case. Right?
So you can apply an adjustment for being like, what are their personal incentives? Subconsciously and consciously. But if you look at their employees too, or if you look at anybody who has privilege, access, maybe the government has a hand in this. I know the government's working with these AI companies.
So yeah, I mean, you can update on the fact that nobody's running in the streets yelling, but it's at the same time, if you look at the trend, it seems like the intensity of the worry is increasing over time, not decreasing.
Liam: Yeah, I would agree. And if that continues to happen, if even people at the very inside of these companies are starting to get worried and are starting to signal that they're getting worried, then I will certainly update that factor to be much weaker than it is now.
Liron: But if that's all it takes. I mean, I hear reports all the time of individual after individual signaling that they were. The last one I saw was the reviews for Miri's book. You know, the book called if Anyone Builds It, Everyone Dies. And you had huge names coming out. Ben Bernanke was saying, wow, this is a must read. This is a real threat. You know, that guy had some government access.
I mean, and there's other people. There's actually the National Security advisor said that.
Liam: I don't think Ben Bernanke's opinion is the one that matters here. I saw recently some very notable figure. I forget who, but basically, oh, it was Francis Fukuyama was in the news recently. Actually my mind has been, by the way, convinced about AI doom. I think it's a plausible scenario. But that doesn't really update me, whether or not just to.
Liron: Give you these names. Okay, so forget Ben Bernanke. After all, he was only a noble winning economist and former chairman of the US Federal Reserve. What does he know? But there's RP Eddy, the former director of the White House National Security Council. And his quote is, I wish this wasn't real. Read today, circulate tomorrow, demand the guardrails. I'll keep betting on humanity, but first we must wake up.
So do you really want to say that you're applying a Bayes factor downwards because people who are in the know aren't warning us that we should wake up the former.
Liam: What? You said the head.
Liron: Yeah, so his title here is former Director, White House, National Security Council.
Liam: Okay, that does update me more in the direction of doom if that guy is saying.
Liron: Oh, nice. Okay, let me take another bite at the apple. All right, so Suzanne Spalding, former undersecretary, Department of Homeland Security, she is writing a review saying the authors raise an incredibly serious issue that merits really demands our attention. You don't have to agree with the predictions or prescriptions in this book, nor do you have to be tech or AI savvy to find it fascinating, accessible, and thought provoking.
Okay, so maybe that's not I have a 99% P(Doom), but isn't that enough of what you're getting in terms of hey, the people who would.
Liam: Right?
Liron: I mean, if you're using the privilege information argument. So what I'm getting from this is like, okay, well, there's no additional piece of privilege information saying we're five times more doomed than I thought, but there's also no piece of privilege information saying we're totally not doomed.
Liam: Yeah, yeah, I think maybe you're getting to me. Maybe I should weaken that point too to I don't know, 0.4 or something.
Liron: I mean, I think that my intuition. You're a young guy, so maybe if you live longer, you may develop an intuition that the grown ups are usually also kind of winging it, right? They're using open source intel, so the thing is that the Gordons of the world, right, the Asperger's hyper smart nerd in their mom's basement. No offense, Gordon.
Those people, generally, they can actually just be more insightful and more at the front of the pack than people whose official title is whatever, Secretary of Homeland Security.
Liam: Yeah, in general, I would agree with that. And certainly if the argument is that, you know, a bunch of smart people believe that AI doom is a very serious thing, then you don't need to convince me about that. I'm already, well convinced that some very intelligent people, people probably smarter than I'll ever be, believe strongly in AI doom.
Liron: Yeah, so I'm just not loving the Bayes factors is what I'm saying. It just seems like you tack them on. And I mean, don't get me wrong, my doom train is kind of ad hoc too, right? So I just Feel like maybe there should just be one doom train, right? Everything should just be a stop and we should try to make all the stops as close to independent variables as we can.
Even though we know we're going to fail, at least we can try. And I just don't think there should be a separate category of Bayes factors.
Liam: I still think there should be just because again, to give a very different example, and I know it's not a perfect analogy, but imagine that my prior was that, you know, vaccines are evil and if I inject a vaccine then it'll make me grow horns or something. I should still, if I'm reasoning properly after seeing the expert consensus saying that vaccines are overwhelmingly safe, I should still update for my, you know, 99% vaccines dangerous belief to something like below 10% that vaccines are dangerous.
And I think that's even the case. Obviously AI doom is a much more plausible argument than anti vaxxer stuff. But I still think that I hold this belief that humanity will be doomed within 10 years. And the experts don't agree with me. I still think that no matter how strongly I believed in that, I should have a way to update down.
Liron: Sure, sure, yeah. I mean this idea of when to defer to experts, it's obviously it's a valuable topic. It's unfortunately a complex topic. A topic with a lot of debate. I can tell you, me personally, in my experience, it just when I feel like I grasp the logic of an argument in my own life experience, generally the experts don't come in and shock me that much at that point.
Don't get me wrong, sometimes I will update. I'll be like, oh my God, the rationalists know so much better about, I don't even know what kind of diet to eat. Right. Like compared to mainstream. You just follow all the rationalist directs recommendations oh wait, some of the mainstream recommendations work a little better. So I have updated back and forth potentially.
But I've just never, I've never felt totally ignorant by not listening to experts. If I can critique the logic itself, that's my own life experience. Take it for what it's worth. The point of my podcast Doom Debates is actually to shine a light on who are the so called experts and just show everybody that, look, everybody is winging it or everybody is doing their best working with a very vague combination of doom train stops.
I mean you can see the different people on my podcast. Sure. I don't have White House level staff Yet. Right. Give me a little time to climb the ranks in my podcast. But I do have superstars like Professor David Duvenaud, University of Toronto, who's worked closely with Geoffrey Hinton, who's collected a bunch, you know, not a Turing Award, but a best paper award at a conference.
That guy is full of credentials, and he's basically saying that my stops on the doom train check out.
Liam: Again, I'm not denying that there's a lot of just very impressive expert level people who agree with you. I just, I still think that just the two categories of people who I think are the most relevant if we're going to give expertise to anyone, are A, the super forecasters and B, the people who currently have privileged information, currently working in high levels of national security and high levels at leading AI labs.
And those are the two groups of people who I would be most impressed or would be most willing to change my mind if I heard different reports from them.
Liron: Well, I considered a very significant piece of evidence that surveys of people working in the AI industry seem to show that the median doom probability is greater than 10%. Especially when you correct for the idea of these people would be selected for and incentivized to be optimistic. I mean, they literally all have the millions of dollars riding on the optimism thesis.
Liam: Yeah, that's fair.
Liron: The other thing I wanted to say about my podcast, you know, people who watch the episodes of my podcast and the reason I'm doing this is to show people that not only do you have superstars like David Duvenaud who are like, yep, the doom argument checks out. And then I do what's called borrowed credibility, right? It's like they. It's like it came out of his mouth, not mine. Okay. I'm just the host.
The other thing to notice from my podcast is when you look at all the non doomers, and some of them are very smart, you know, Professor Robin Hanson, Professor Scott Sumner, more professors on the way, you know, Gary Marcus, all of these different people, they're all intelligent non doomers, but you can see that they're different. Non doomer arguments don't even line up.
So even the non doom thesis, it's not like there's this one canonical expert non doom consensus, right? Compared to the consensus of why vaccines are good, it's like people are on the same page. It's like, well, it helps your body, you know, get resistant to the infection, right? It's a much more Gears level model.
Liam: I would actually be very curious to know of the superforecasters who do not believe in doom. Or are they all on the same page of why doom is not likely. If say a third of them believe that AGI just isn't coming soon. And then a third of them believed it is coming soon, but it'll be aligned. And then a third of them believed, okay, it is coming soon and it won't be aligned by default, but with enough alignment efforts, we can make it work.
If that was such a stark break where they actually don't really agree on that much, that would probably reduce the amount that I would trust them.
Liron: Yeah. So you totally don't get that kind of unity. That's actually the reason I started my podcast. I named it Doom Debates. I was originally going to name it Doom Train because I want to highlight there's. It's not a two sided thing. It's not Doom versus non doom. It's this whole train. It's all these little sub arguments and people are just getting off on all over the place and.
Yeah, I mean it's just, you know, I would just not frame it the way your Bayes factor seem to be phrasing it of hey, the experts are telling us not to worry. It's like, watch my podcast to see what the experts are saying. You may get a different idea.
Updates and Final Calculations
Liam: So okay, here's something I'll tell you is that as we've been talking, I have mentally been like, yeah, may, maybe my Bayes factors are stronger than they should have been. So I updated the 0.75 to 0.8, the 0.2 to 0.4 and the 0.6 to 0.7. And after I do that, my P(Doom) goes from 3% to 8%. So I think you can take some victory and okay, you updated my P(Doom) up 5% from what it was awesome.
Liron: Yeah, and that's, you know, from it's. It's actually more useful generally to analyze these more as odds ratios than percent probabilities. Because that jump from 3 to 8, it feels like 5%, but it's actually the jump from 33 to 1 odds. And now we're down to 13 to 1 odds. Sure. Which is huge. Right?
It's equivalent to the jump from 50% to more than 90% or something like that. A huge jump. Right. It could be a 40% jump, maybe more than 80%. So we got to think in terms of odds ratios now. You got to 8%, which I'm super thankful for. I mean this is. You probably hold the record of the most meaningful update after having conversations me for going from 3% to 8%.
And I'm willing to squeak you in into what I call the sane zone. I Normally say it's 10 to 90%, but I'm willing to give you a grace period or what do you call it, a margin of error, so that as long as you're more than 5%, I'll consider you essentially being in the sane zone. It's just when you query your intuition, your intuition is still pushing back. It's still 13 to 1 that we're not doomed. It's not a little bit more flexible.
Liam: My intuition is pretty flexible. But yeah, just my gut feeling. Even before considering any of all that we just talked about, my gut feeling is still that about 5% chance that we're doomed. 95% that we'll be fine.
Liron: 95%. So 19 to 1. Because as I said, it does seem more extreme when you phrase it as 19 to 1. Right. It's like my $19,000 to your 1,000, right. If we were making a bet. Or it would actually be your 19,000 to my 1,000, right. So I got a $19,000 payout if the world ends.
Liam: I'm not saying that my gut is the most reliable thing in the world, but yeah, that's what my gut tells me, that 19 to 1 odds is that we'll be fine.
Liron: So you almost see this as a theoretical exercise, this whole doom thing kind of.
Liam: Yeah.
Liron: Yeah, I feel, I mean that. Because that's, I think it always is weird to me when people are like, yeah, you know, I'm open minded, I'm open to anything could happen, but I'm 19 to 1 that we're going to survive. That's a pretty rigorous level of confidence. I mean you can make a lot of life decisions with 19.
You can buy a house. If you're 19 to 1 certain that you're not going to move in the next few years, go ahead and buy a house. Right? I mean, that's a nice solid level of confidence.
Liam: Yeah, I mean I guess I'm not confident in any particular future coming true like that, that 19 to 1 is split between, you know, if there's a 15% chance we just won't even get AGI. And then there's a certain chance that we get AGI and it goes perfectly and then there's a non negligible chance that we're about to get misalignment, but then we pull through at the last minute and then we have some sort of pause.
It's like, I can't tell you with any degree of confidence what the future will look like. It just seems like the range of futures where we end up mostly fine is a lot more plausible than the range of futures where we are doomed.
Liron: Okay, I think that's a good place to leave the doom train. I'm really happy with how you rolled with the punches and really reflected on your own beliefs and why you have them and what it would take to update you. I mean, you're clearly a strong rationalist in the making here.
Liam: Thank you. Thank you.
AI Policy Discussion and Wrap-up
Liron: Moving into the wrap up here, in addition to thinking pretty rigorously about your P(Doom) and the doom train, you also think a lot about AI policy and how we should go about laying down some useful policies whether we're doomed or not. You know, you kind of think about the whole space. So tell the viewers what insights you've learned from that kind of thinking.
Liam: Sure. So I think that in general, the policies you would support at, you know, 1% versus 5% versus even 15% doom are actually largely sort of similar. And even if I had a much higher P(Doom) of 80%, still many of the policies would be the same.
Which is just maybe we should have some more public sector support for AI research so that the people who want to be doing AI safety work and mechanistic interpretability work should have the opportunity to do so, even if it's not immediately market incentivized for the same reasons that governments always support public goods projects for that reason.
Also public funding compute resources like NAIR, which is basically just a public compute cluster that people doing public facing research can access. Just generally having stricter CyberSecurity at leading AI labs and publishing their safety and security protocols.
Also increasing our human capital is always a good thing to do in any circumstance, but especially we need the most talented humans possible to handle AI safely. And also I do buy into a large extent to the national security arguments of I do think we should be competing with China and making sure that America stays ahead on that.
Liron: It sounds like your strongest overarching point, which I agree with, is that whether your P(Doom) is 2% or 98% or anything in between, there's a lot of policies that you've called, I think, Pareto optimal, right?
Liam: Yes. Basically for those who are not familiar in economics, a Pareto optimal solution is something that's better for all people, or at least it doesn't make anybody worse than what happened beforehand. And I think that there's a bunch of policies on the table that are Pareto optimal in the sense that whether you're an accelerationist, whether you are a safetyist or a doomer, or whether you are a national security hawk, these policies are simply good policies that make everybody better off and we should be pursuing those policies.
Liron: So my question for you is, you're pointing out that all these policies are Pareto optimal. Do you think that people are leaving low hanging fruit on the table or do you think that both sides of the discussion are actually doing a good job on pushing forward the Pareto optimal policies?
Liam: No, I think there's a ton of just low hanging fruit on the table that I think anybody who's very seriously engaging with the issue should support. And I think it's just a combination of inertia. It's just hard to get the government to take action. And also just other concerns totally unrelated to AI are getting in the way. Increasing human capital through immigration is great. Whether you're an accelerationist or a safety person.
Liron: Yeah, you know, it's a general rule of government, democratic government, that there's so much waste and there's plenty of times when you can prove that a certain policy would work better, you know, letting market forces work in more places, that kind of thing, and we just fail to do it.
And Brian Caplan has a really good general reason why, which is that governments always do things that sound good and not so much things that work good because the voting is done based on what sounds good. And then Brian Caplan loves corporations because corporations are out there doing things that don't always sound good, but then they do work good. So that's why Brian loves corporations more than government. And I tend to agree.
But it sounds like that same effect of government leaving money on the table because some things sound better. It sounds like that is impacting all of these obviously good policies about AI. So I'm not that optimistic that they're going to improve. But maybe your point is that if we can all just agree that the AI issue is important, whether it's going to be optimistic or pessimistic, then let's at least prioritize these win win AI policies and, and downplay things that aren't AI related that people normally fight about.
Liam: Yeah, my basic mode is just that as long as there is low hanging fruit on the table, I'm going to reach for that low hanging fruit. And then if and when the low hanging fruit eventually gets picked, then I'll reorient myself towards some maybe tougher battles.
Liron: Gotcha. So it sounds like you're gonna graduate and have a really nice career in AI policy, right? Is that kind of how you see yourself?
Liam: That's definitely one of the paths I'm strongly considering and I'm getting into the D.C. AI policy world. Making any plans, you know, 2027 or later is kind of impossible at this point, but I think there's a very likely future in which I'm some kind of policy analyst, think tank worker, or even lobbyist of some kind.
Liron: So in two or three years, I'm going to be on this podcast telling the next college student, Liam Robbins from The Department of AI Policy is warning us that AI doom is 8%. Don't you want to take him seriously? All right, nice. So that is all the topics we wanted to hit.
And I think the viewers are with me that it's impressive that you're having this discussion just being in college. I think most of us in college, in college were not even engaged with important topics to this degree. So, yeah, keep up the good work, man. And thanks so much for coming on the show and riding the Doom train.
Liam: Thanks for having me.
Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.
Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates
Share this post