14-Year-Old Forecaster Challenges My P(Doom) — Eli Goldfine

Eli Goldfine is an unusually thoughtful 14-year-old podcast host who’s rapidly becoming an authoritative voice in the prediction market space.

In this unique episode, Eli brings a critical yet open-minded approach to the topic of AI doom that we older folks could learn from.

We cover what it's like to grow up in the AGI era, and Eli's hobbies which include reading rationalist bloggers, betting on prediction markets, vibe coding with Claude, and developing a software startup for astronomers.

In our debate, Eli expects superintelligence by 2032, but argues P(Doom) can't be estimated. I claim you can't opt out of Bayesian epistemology. All aboard the Doom Train! 🚂

Watch on YouTube:

Timestamps

00:00:00 — Cold Open

00:01:07 — Introducing Eli Goldfine

00:08:31 — Interviewing Robin Hanson at Manifest

00:10:55 — The Supercycle Podcast

00:14:37 — Eli’s Betting Market Scandal

00:16:49 — Will Eli Graduate High School?

00:18:27 — Decentralized Telescope Software

00:26:50 — Vibe Coding Without CompSci Basics

00:32:08 — Robin Hanson, Futarchy, MetaDAO

00:40:56 — What’s Your P(Doom)?™

00:46:32 — Liron Defends His 50% P(Doom)

00:51:00 — Should P(Doom) Inform Policy?

00:59:16 — What’s Your P(Bloom)?

01:03:04 — Pascal’s Wager for Accelerationists

01:04:04 — Donation Drive

01:11:22 — Are E/ACCs Incredibly Dumb?

01:13:42 — How Powerful will ASI Become?

01:16:20 — Predicting ASI by 2032

01:23:56 — Let’s Ride the Doom Train™

01:35:26 — Orthogonality, Culture, and Moral Realism

01:44:39 — Can Safety Keep Pace with Capabilities?

01:49:12 — Debating Instrumental Convergence

01:54:40 — Culture, Laws, and the Singleton Scenario

01:59:53 — Ricardo’s Law Won’t Save Us

02:03:32 — The Case for Racing China

02:05:40 — Wrap-Up & Ideological Turing Test

02:10:36 — Outro from Conductor Ori

Links

Eli on X — https://x.com/realTomBayes

The Supercycle, Eli’s blog & podcast — https://supercycle.blog/

Business Insider profile, “The newest prediction markets guru is a middle schooler in braces” — https://uk.finance.yahoo.com/news/newest-prediction-markets-guru-middle-090002282.html

The Supercycle — Robin Hanson Fireside Chat @ Manifest — https://supercycle.blog/p/robin-hanson-fireside-chat-manifest

The Supercycle — Bentham’s Bulldog on Libertarianism, Anarcho-Capitalism, and Futarchy — https://supercycle.blog/p/benthams-bulldog-on-libertarianism

Liron’s “Pausing AI Is Positive Expected Value” — https://www.lesswrong.com/posts/XkmYTBGXLnPDXqg44/pausing-ai-is-positive-expected-value

Kapoor & Narayanan, “AI existential risk probabilities are too unreliable to inform policy” — https://www.normaltech.ai/p/ai-existential-risk-probabilities

Liron’s response, “P(Doom) Estimates Shouldn’t Inform Policy?? Liron Reacts to Sayash Kapoor” —

Robin Hanson vs. Liron Shapira: Is Near-Term Extinction From AGI Plausible? —

“Bentham’s Bulldog” Says P(Doom) is LOW — Matthew Adelstein vs. Liron Shapira —

Max Tegmark vs. Dean Ball: Should We BAN Superintelligence? —

Does AI Competition = AI Alignment? Debate with Gil Mark —

Join the Doom Debates Discord — https://discord.gg/2yAFMsRET

Support Doom Debates — https://doomdebates.com/donate

Transcript

Cold Open

Liron Shapira 00:00:00
Eli Goldfine, a 14-year-old prodigy rapidly becoming an authoritative voice within the prediction market space.

Eli Goldfine 00:00:07
I would just say the edge I have over other 14-year-olds is that I’m—

Liron 00:00:11
You’re gonna go toe-to-toe with some of the smartest guests we’ve ever had.

Eli 00:00:15
Okay. I think you’re contradicting yourself here with your 50% P(Doom). I just don’t see how you can really form a prior here.

Liron 00:00:24
Well, I mean, not being able to form a prior, you could say that about anything, right?

Eli 00:00:27
What do you mean you could say that about anything? There’s just no remote reference class here to be able to form a base rate from it.

Liron 00:00:35
One of the dominant things going on in my head is the sense of bigness. Yeah, this is a world-changing event.

Eli 00:00:43
I’m trying to think about how to put my disagreement here.

Liron 00:00:47
You said that ASI won’t be a world-changing event. Do you really think that?

Eli 00:00:50
What my gut feeling is here is, yeah, that sounds right, but on the other hand, no one has any idea what’s gonna happen.

Introducing Eli Goldfine

Liron 00:01:07
Welcome to Doom Debate. Today, I’m talking with Eli Goldfine, a 14-year-old prodigy who’s rapidly becoming an authoritative voice within the prediction market space. His new podcast, The Supercycle, features a recent interview with Robin Hanson.

Eli 00:01:23
So let’s get this started here. So what’s your P(Doom)?

Robin Hanson 00:01:27
I think human extinction is unlikely.

Liron 00:01:31
And is currently preparing to host an exclusive interview with Sam Bankman-Fried. Eli is a lifelong amateur astronomer who’s currently building software to coordinate a decentralized network of telescopes. In the last couple years, that is to say ages 13 through 14, he’s been diving into a cluster of ideas that include prediction markets, macroeconomics, AI, and of course, existential risk from artificial superintelligence.

So today, I’m excited to see what Eli thinks about the various stops on the AI doom train, and ask him about the odds that humanity will survive long enough to enable him to start as a high school freshman in the fall. Eli Goldfine, welcome to Doom Debate.

Eli 00:02:15
Thank you, Liron. Happy to be here.

Liron 00:02:18
So I think we’ve made the point pretty clear that you are unusually young, and for people who didn’t get that from my introduction, I encourage them to come watch the video. You do, in fact, look quite young, which is great. You look youthful. But it is very shocking because I think people are gonna see that you’re gonna go toe-to-toe intellectually with some of the smartest guests we’ve ever had, and you’re only 14 years old. So what’s the deal? How come you’re 14 years old going on 24?

Eli 00:02:51
So first of all, I know you’ve had some extremely smart people on the show. Not to say the least of them — we’re talking Robin Hanson, many others, very smart people who have debated you on existential risk. So I would say I’m definitely not as smart as Robin Hanson, who is a very, very smart man who has some pretty strange ideas.

I wouldn’t say I’m a genius or a prodigy or anything. I would just say the edge I have over maybe other 14-year-olds is that I’m interested in something just for the sake of being interested in it, rather than trying to get something out of it.

Because when I look at my classmates, either they’re interested in something that doesn’t really matter — they’re interested in sports or video games or something that 14-year-olds would normally be interested in — or maybe they’re interested in something more interesting, but they’re doing it for a reason, like to get an internship or to get into a good college.

So what I would say is that I’ve just really enjoyed diving into the topics that I’m interested in, and I feel like I have enough dedication and being genuinely interested in them to be able to learn all of what I know about these different topics. And maybe that’s just because not everyone is an aspie like Liron and I are, but that’s what I would say in general.

Liron 00:04:25
I think that’s pretty self-aware. I talked with you for a couple hours prior to this show, and we’ve been DM’ing in Discord for a few months. As best as I can tell, if I had to say how are you so special in terms of your prodiginess or just being way younger than most people who are in this conversation, I guess I agree with your assessment, that maybe there’s not one single dimension.

If you showed up at a math contest right now, sure, you’d do way better than the average middle school student or the average high school student, but you wouldn’t be winning the statewide math contest. You’re not quite top on that dimension. It seems like your superpower is that you have an adult level of agency and somehow a mature approach to pursuing goals — adult goals. It sounds like you’ve somehow got an adult level at those particular qualities.

Eli 00:05:17
Yeah, I would say that’s fair. This is something that I think Twitter is good for. With social media, there are some downsides, but there’s a lot of benefits. One thing that happened with me being on Twitter is I see all these people, and they’re doing interesting things in the real world. So why should I be limited? I’m interested in the same things they’re in — that’s why it’s showing up in my feed. Why should I not be allowed to do those things because I’ve been around on Earth for less long than they have?

And so I think the advice that you give is you can just build things, so that’s my philosophy as well. Why not just try to do it?

Liron 00:06:00
Now, prior to you, I think the youngest Doom Debates guest has been Matthew Adelstein, also known as Bentham’s Bulldog. He’s been a rising star in the intellectual community. I think he’s currently 22 years old. I know you’re familiar with him as well. I’m sure you must be thinking, “Man, why is that guy so old?”

Eli 00:06:18
He’s very smart. We just had him on the Supercycle. That episode should be coming out in two weeks. I have always been confused why there’s not more people like me. The closest thing I could think of is — I don’t know if you know Zack Yadegari or that sort of group.

Liron 00:06:37
Yeah.

Eli 00:06:37
The teenage vibe coders who code slop, and then they’re on Twitter being like, “I’m a teenager, and I just vibe coded an app.” And everyone’s like, “Hey, you’re killing it.” So I guess that’s what I would say is the closest, but I feel like it’s kind of weird that there’s not a lot more people doing what I’m doing on Twitter and stuff that I can tell.

Liron 00:06:58
Do you know anybody in real life, at your school or in meat space who is kind of comparable to you in terms of being 14 going on 24?

Eli 00:07:09
The only person I could think of is I have a friend named Ryan who I knew from Manifold. I guess I don’t really know anyone else who meets the 14 going on 24 description.

Liron 00:07:20
Pretty fascinating. It also reminds me that back in 2005 or something, I read Paul Graham’s essay where he remarked on Sam Altman. Sam Altman was 19 or 20 at the time, and Paul Graham was like, “Yeah, this guy arrived in our program age 18 or 19 as essentially a middle-aged adult just ready to go, ready to be fully mature.” So now I know how Paul Graham felt. People like that exist. That is a mindset you can have.

By the way, listeners, I encourage you to check out Eli’s podcast, The Supercycle. He mentioned interviewing Robin Hanson. You can listen to that episode on that podcast. If you listen to that, I think you’ll be impressed that Eli is asking super smart questions, super detailed questions, the kind of questions that I like to ask — getting into the weeds, really intellectually engaging.

That was such a good proof for me. Just based on listening to that podcast, that’s what made me really interested to talk to Eli, ‘cause I’m like, okay, here is a totally respectable mind to engage with, somebody who’s going to have interesting thoughts about questions like AI existential risk. I felt confident to think that just based on listening to him do that podcast. And then when I talked to Eli in private, I was like, “What did you feel doing that?” And he told me he was actually quite nervous. Maybe you had imposter syndrome — what was that like?

Interviewing Robin Hanson at Manifest

Eli 00:08:31
Yeah, for sure, I definitely had imposter syndrome. I would say the first day of Manifest, I — he was only there on Friday night, so I had five hours at Manifest to stew about the different questions I was gonna ask and the follow-ups and stuff. Just very stressed out that first day. Also, I wasn’t really prepared for how weird Manifest was gonna be, but that’s besides the point.

So I had come up with my questions, and then I walked into the room where I was supposed to be interviewing him. And then he said, “You know what I hate about podcasts? I hate how the interviewers just bring a list of questions and ask them one by one.” I was like, “Oh, shit, that’s my plan here,” and he doesn’t want that.

So what I did is I had my eight pages of notes, and then I gave them to my dad, and I was like, “Don’t give these back to me.” And I had to come up with stuff in my head. I did kind of run out of follow-ups in the middle, and so I had to try to remember what I had written down. But in general, it was a very intense experience. I’ve talked to people who know him pretty well. They say they’ve never come out of a conversation with him thinking that it went well.

Liron 00:09:48
So I think — we’re not supposed to diagnose remotely, but I think all three of us, me, you, and Robin Hanson, probably are all aspies. I think Robin Hanson’s probably the most aspie of us three because even though we love him and we think he’s even underrated as a genius, when you have a 14-year-old coming to interview you, you probably don’t want to be telling him what you hate about podcasts. You probably generally wanna facilitate things going really easy and smooth. But Robin Hanson, God love him. I hope he keeps doing what he’s doing.

Eli 00:10:16
Yeah. No, for sure.

Liron 00:10:18
All right, viewers, we even have a video of this. Check this out.

Robin 00:10:21
It sounds like you’re inventing a new idea.

Eli 00:10:23
Okay.

Robin 00:10:23
So here you say we have some outcome measure, and then we vote on what representatives to elect, and then those representatives just do whatever the hell they want.

Eli 00:10:30
Oh, no, no, I’m saying you bet. So imagine a normal futarchy, right? Except the market would be, will instating politician X—

Robin 00:10:40
Right. So—

Eli 00:10:41
—as president improve the welfare measure?

Robin 00:10:43
Right. That’s what I said. Yeah.

Eli 00:10:44
Okay. Yes. Okay. Sorry.

Robin 00:10:44
Using futarchy to pick people to be in charge and then letting them do whatever the hell they want.

Eli 00:10:48
Yes.

Robin 00:10:48
So I would not be very optimistic about that compared to futarchy, although it might—

Eli 00:10:51
Right.

Robin 00:10:52
—work better than our status quo.

Eli 00:10:54
Okay.

The Supercycle Podcast

Liron 00:10:55
So this is such a fascinating podcast, by the way. There’s a lot of good content. Why should people listen to The Supercycle?

Eli 00:11:01
So prediction markets — fastest growing asset class of 2026. Kalshi’s valuation has now surpassed — they’re raising at a $40 billion valuation now, so enormous industry becoming increasingly important in American culture. And then when you look at who’s talking about prediction markets right now, it’s some people on Twitter, but for actual long-form interesting writing, it’s 95% AI slop, or it’s people on TikTok telling you how to lose money.

I was at my friend’s house, and his TikTok feed was all people saying, “Here’s how I made $3 million on Kalshi in one hour,” which doesn’t even make sense. There’s not enough liquidity for them to have been able to do that so quickly.

So what we’re trying to do is make interesting content that publishes frequently, that’s pro-prediction markets. We’re being cautious about what’s going on in the industry, but we’re also very bullish on the concept of prediction markets in general. We just wanna provide thoughtful commentary about prediction markets that’s not AI slop, and it’s actually interesting and helpful to people. We’re also very futarchy-pilled, so if you’re interested in futarchy or alternative incentive structures for prediction markets, I highly recommend checking it out. You can look at supercycle.blog.

Liron 00:12:24
All right. Hell yeah. Viewers, check out supercycle.blog after this episode.

Liron 00:12:27
All right, let’s talk about your background. Now, you’re 14 years old. Personally, I have a mental note that I became conscious around age 9 or 10, in that I kind of had a stream of consciousness, and I noticed, oh, okay, I’m kind of verbalizing in my head one thought after another in a stream. I guess this is what it feels like to be fully conscious. I remember at that age, I kind of made that note to myself.

Okay, so you’re currently 14, so when I’m asking you about your background, I guess I’m asking about the last few years. And when we talked before the show, I think you were saying maybe a good starting point to your current intellectual journey was the 2024 election, or put it in your words. What’s your background?

Eli 00:13:05
If we’re going on the prediction market sort of side of things, I’d say the start of my intellectual journey, if you want to call it that, was probably May 2025, so 14 months ago. That was when there was my school government election, and I had found out about Manifold, and I made a market on Manifold about who would win. That’s when I just got really into learning about markets and everything about prediction markets. So I guess I would say that’s a start.

Liron 00:13:42
What about the 2024 election?

Eli 00:13:44
The 2024 election was when I initially looked at Polymarket odds. So I kind of found out about Manifold ‘cause I wanted a play money permissionless variant of Polymarket.

Liron 00:13:55
Gotcha. And before that, when did you become conscious?

Eli 00:14:00
When did I become conscious? To the point of having interesting ideas — basically, the way you described a stream of consciousness, I would say maybe I felt I had that around maybe when I was 8. But for having interesting, novel ideas about interesting stuff, I would say probably 11.

Liron 00:14:27
Yeah, that seems about right. Okay, you mentioned you used Manifold to bet on your student government election in middle school, right?

Eli 00:14:35
Mm-hmm. That’s correct.

Eli’s Betting Market Scandal

Liron 00:14:37
How did that go?

Eli 00:14:38
I did end up winning. My school did not like Manifold. They do not like Manifold. I’ve spent a long time debating my principal about whether Manifold is real money or not, which is the most obvious claim, and they could just look it up if they wanted to.

Liron 00:14:56
Right. The correct answer is no? It’s not real money.

Eli 00:14:59
No, it’s not. It was fun doing that for a couple months until I found all of their FERPA violations, and that kind of—

Liron 00:15:07
What’s FERPA violations?

Eli 00:15:08
It’s when schools have a responsibility to protect student data and information, and then they don’t do that responsibly.

Liron 00:15:20
Right. So this is an interesting story. I don’t wanna implicate any particular individuals, but basically, they were giving you administrative trouble for using Manifold to bet on your student government election, which you won. They were acting like there was some problem with what you were doing, and then you kind of turned around and you’re like, “Hey, what’s with these FERPA violations?” You kind of turned the tables on them.

Eli 00:15:40
Yeah, I would say, yeah. That was actually six months later, where the teachers were projecting URLs to Google Sites on projectors, and those Google Sites had Google Sheets with medical records and grades on them that anyone could access.

Liron 00:16:01
Crazy. Yeah, so just objectively speaking, I’ll just be an objective referee as another adult, and I will say that posting student grades and personal info on Google Sheets with a privacy setting where anybody can access — that seems like a worse offense than betting on your student council election on Manifold.

Eli 00:16:19
Yeah. The reason that they disqualified me after the election was because I had been telling people about, “Hey, look at all this stuff they’re doing. Here, I’ll prove it. You can go to this link, and you can see all this stuff.” And so that’s basically why.

Liron 00:16:38
All right. Well, you got a good story out of it, and now you’re out. You’re done with middle school. You’re not coming back there, but your plan is to go to high school, right?

Eli 00:16:46
Yes. Currently, that is the plan.

Will Eli Graduate High School?

Liron 00:16:49
Okay, so you’re an incoming freshman. And we talked about this before the show. I think you and I are not super bullish on the value per hour of somebody like you going to high school, correct?

Eli 00:17:00
Yeah, I would agree. I think if you read Bryan Caplan’s Case Against Education book, this is pretty clear. I think it might work for some people. It might work for a small subset of the population. I think in general, it’s a very, very ineffective way to learn anything.

I think if I wanted to, I could probably — if I had good AI ed tech, I could probably get through all of high school in a year or less. I’ve learned stuff with AI. That’s how I know most of the stuff that I know about the topics I’m interested in, plus Substack. I just don’t think it’s very easy to retain the information, let alone is the information they’re trying to teach you useful at all. It’s completely obsolete, essentially. I don’t really see the value.

Liron 00:17:50
Do you think it makes sense for you to minimize the amount of time and mind share that you’re gonna give to high school?

Eli 00:17:57
I would say it depends. If I have more of a master plan, I would say yes, because it is important if I need to go to college or something, which I still don’t know whether I will or not, ‘cause it’s definitely more important if you are going to college. But in general, as a raw experience now, I would say no, it would not be valuable in general.

Liron 00:18:23
All right. Well, the important thing is you at least read Bryan Caplan’s Case Against Education, so at least you’re coming in with an informed both-sides opinion about whether or not high school is good before you enter high school.

Decentralized Telescope Software

Liron 00:18:27
Let’s talk about your current project. I know you’ve been vibe coding with Claude to control telescopes throughout the country. It’s kind of decentralized, or it’s a network of telescopes that are all gonna be talking to your server, and you’re gonna coordinate them to collect data. Tell us more about that.

Eli 00:18:50
So what we’re doing is basically there’s been a rush of new consumer telescopes called Seestar, specifically their S50 model, and they’re actually very, very well-designed. They’ve become extremely popular for consumers, but people have them out, and they’re imaging the same five objects that’s recommended to them in the app or whatever.

So what we’re trying to do is we’re autonomously gonna control them ‘cause lots of people have them, and our value proposition to people is, you already have this thing. You can join our network for little to no effort, and all you have to do is bring your telescope inside when it rains, and then you get co-credited on actual astronomical discoveries when the network finds something. So it’s pretty cool, and I’m actually working a little bit with an engineer at the University of Vienna for the extremely large telescope on this.

Liron 00:19:47
Amazing. So it’s kind of like SETI@home, but it’s not about using people’s compute resources. It’s about using their visual sensor resources.

Eli 00:19:57
Yep. That’s a good assessment. And then we collect the data, and we do post-processing on it to get the metadata of the photometry, and then we can submit that to data organizations. We also run AI image processing, so the AI will run the processing on all of it. If it detects something, it can give us an alert. Then we can send it to different organizations who can verify whether it’s a discovery or not. It’s a pretty exciting project. Hopefully, we’ll get a couple dozen people using it or something. Should be pretty interesting to see how that plays out.

Liron 00:20:33
I’m pretty curious about how much of an impact this is gonna have ‘cause obviously there’s observatories distributed all around the world.

Eli 00:20:39
Mm-hmm.

Liron 00:20:40
But what are the applications where people are like, “Oh, man, if only a bunch of people went into their backyard and pointed their relatively small telescope at something, that would be so valuable”? What are you thinking in terms of applications?

Eli 00:20:51
There’s these things called time domain surveys, which is basically — so if you know the new Vera Rubin telescope, that’s one of the largest time domain surveys ever. Maybe the largest, I’m not sure. And what their mission is, they want to survey the entire sky, the entire visible sky from their location at X depth every X days. Let’s say they want to observe the sky up to magnitude 24 every two days.

The problem is the telescopes that are big enough to do these surveys actually saturate when it gets to a bright magnitude. And additionally, not only do they saturate, they can’t just change their plans and go back and observe something they think might be a candidate for a certain object ‘cause they have strict schedules to observe certain things at different times.

So what we can do as the network is we can do rapid follow-up on bright transient objects ‘cause that fills a real gap that’s been a problem because they can’t get data fast enough at the right brightness.

Liron 00:21:57
So I definitely understand that. The idea that they have to look in a bunch of different places over a time period, but you only have one telescope, so sure, it’s really great, but you can only point it at a few different things. And if you ever want to survey a lot of things or look at one thing for a long time, you just don’t have enough big telescopes to do it. I get that much. I’m a little confused about what you mean by they saturate — they can’t see it in good detail because it’s too bright?

Eli 00:22:19
Yeah, exactly.

Liron 00:22:20
Can’t they just put sunglasses over it or something?

Eli 00:22:25
The way the optics are designed wouldn’t really allow for it.

Liron 00:22:29
All right. And what you said about time domain, it really is just this idea of stare at the same object for days or weeks and see any changes?

Eli 00:22:37
Well, it’s observing the entire sky at a certain cadence at a certain depth, and then sometimes if they’ll look at a certain area and see something looks different here that we haven’t seen before, then they can’t go back, so they need rapid follow-up to see what that is in case it’s gonna change quickly, and that’s what we do as the network.

Liron 00:22:57
Fascinating. There’s a little bit of an analogy here where chip makers used to focus on making personal chips that would try to go really fast — increase those gigahertz. But then recently we’ve been like, well, it’s not really about making the chips faster. It’s just about packing them in and having them be super parallel. That’s been the current revolution because a lot of these AI algorithms are super parallelizable.

So there’s a little bit of an analogy here where you’re saying, hey, you’ve got these big telescopes, and people are focused on these big observatories, but there’s a lot of value if we could just collect data from a bunch of different telescopes that are strategically coordinated, even if each individual telescope is small or not the equivalent of high gigahertz — it’s a small lens. You just have to look at the sky kind of zoomed out. Is that a good analogy?

Eli 00:23:40
Yeah, I would say it’s a pretty good analogy. We get to the level of a larger telescope by just combining multiple of them looking at the same thing. If we have one 50-millimeter telescope and we have another one, that has the opportunity to have an aperture where it’s collecting collectively 100 millimeters of light volume. We can exponentially increase the value of one telescope by adding more.

Liron 00:24:08
What is the why now of this idea? Could this have been done 10 years ago?

Eli 00:24:13
The telescopes are now more reliable. The technology is better on them, and now that there’s more time domain surveys, there’s more opportunities for rapid follow-up.

Liron 00:24:25
Interesting. So the fact that we can now use AI to analyze the data, that’s the icing on the cake? That’s gonna make it extra valuable. But if all we had was more reliable telescopes, this could have even predated modern AI, correct?

Eli 00:24:36
Yeah, I would agree, ‘cause we had Python packages that could do basic data processing long before.

Liron 00:24:42
One disanalogy between this and modern data centers — with modern data centers it’s like, okay, make GPUs that do a bunch of operations in parallel. But with this project, it’s not just enough to be like, okay, observatory, go set up 1,000 telescopes in an array in an observatory, because I think it’s important that the telescopes are geographically distributed. Is that right?

Eli 00:25:02
Yeah, ‘cause obviously there’s different regions of sky visible on the southern versus the northern hemisphere. Also, with time zones, sometimes if they’re observing in China, then there could be rapid follow-up needed where it’s in the middle of the day in the US. So you wanna basically have at least a couple telescopes in each time zone is optimal.

Liron 00:25:25
Okay, I’m getting in the weeds here, but do you think that if you just had 20 observatories that were the analogous thing of a data center — a data center of telescopes, a vision, a mini observatory center. We gotta workshop the name here. But if you just had big clusters of small telescopes in 20 centers around the Earth, is that enough, or do you feel like no, they really have to be in people’s backyards, they have to be all over the place?

Eli 00:25:49
I mean, that could work. An advantage of having more smaller units instead of fewer larger units is that the smaller units can each observe different things at a time. If there’s a bright object that doesn’t need very much depth on it, then you’re kind of over-dedicating resources if you’re sending a very large telescope to go look at it if you just need a 60-second exposure and it’s a bright object.

Liron 00:26:17
But if I understand correctly, clustering everything into 20 observatories, as long as you’re just packing in a grid of small telescopes in the observatory — that’s enough of a geographic footprint, if I understand correctly?

Eli 00:26:29
Yeah, I think that would probably have a similar outcome, as long as they were somewhat distributed.

Liron 00:26:34
‘Cause if you’re observing stars, it doesn’t really matter where you are on planet Earth ‘cause Earth is tiny, right?

Eli 00:26:39
Yeah, exactly.

Liron 00:26:40
You just have to have the angle.

Eli 00:26:41
Yeah.

Liron 00:26:42
Fascinating. So that’s your passion for astronomy. Very cool, and that’s been a lifelong passion, right? Literally since you were three years old.

Eli 00:26:49
Mm-hmm. Yeah.

Vibe Coding Without CompSci Basics

Liron 00:26:50
But you’ve got these recent passions, and it feels like the center of all your passions is prediction markets, which is the focus of your podcast. You’re doing the only authoritative, serious prediction market podcast which is not AI slop, correct? You’re throwing shade on the competition. You’re accusing the competition of prediction market podcasts of just being AI slop.

Eli 00:27:06
If you don’t believe me, run them through Pangram. You’ll see.

Liron 00:27:10
Whoa. All right. He’s throwing down the gauntlet. Run the other podcasts through Pangram. All right. What is the Eli Goldfine production function? What’s a typical day for you, and what do you produce? How productive are you?

Eli 00:27:23
I’d say generally, for the most part, pretty productive. What I’m mostly doing is I’m basically alternating between Claude, Codex, Twitter, and Substack all day.

Liron 00:27:36
Nice.

Eli 00:27:37
That’s my general workflow. I’m just putting in the prompts, then maybe I’ll test the software, see how it works, then I’ll send another prompt, then go on Twitter and Substack in between.

Liron 00:27:47
And what kind of stuff do you have Claude doing for you throughout the day?

Eli 00:27:50
Basically anything for all of my projects that I’m working on. Sometimes I just have a fun idea, and I just wanna spend a day building out that thing, and then at the end of the day, I get a pretty nice piece of software that’s kinda cool. And then I also have it work on some bigger projects that I’m working on.

But I don’t have any domain expertise in software engineering in the slightest. I mean, the very basics, but I just have that doing basically all of my software stuff for me.

Liron 00:28:22
If you had to raw dog into a Python editor, and I’m like, “Okay, make me a game of Tic-Tac-Toe. Don’t use AI,” would you be totally stuck?

Eli 00:28:30
Probably. I used to know Python, and then I kinda got AI psychosis, and I probably don’t remember very much.

Liron 00:28:37
Wow, that’s pretty crazy. I think what you’re doing makes a lot of sense because programming languages are evolving anyway, so how much do you really need to know? I don’t know. I would argue it’s probably worth maybe going a little bit deeper into concepts. Have you ever heard of data structures, like arrays, linked lists? Does that ring a bell at all?

Eli 00:28:52
Yeah, a little bit. I could try to pick up Python again. I’ve tried to watch YouTube videos, and then I just see the Claude icon on the bottom of my screen, and I’m just like, “What am I doing here?” I guess it’s kind of the conundrum.

Liron 00:29:06
Well, it’s just fascinating because for most of my life — when I was nine years old, I was a slight bit of a prodigy myself. I started programming when I was nine years old, and I was writing Basic, Visual Basic mostly. And I did get exposure to these concepts early, and then I was doing it for decades.

And then recently, in the last few months, I haven’t touched code. Occasionally I’ll code review. I’ll scan a few lines of code here and there, but I’ll mostly just talk to Claude about code. So to me, it just feels so natural and obvious — of course you have to have training as an engineer. You have to know how code works.

But then again, do I know the details of how a computer chip works? Only very high level. My understanding of a computer chip is probably analogous to your high-level understanding of how programming works. I know the very basics. So if I’m okay for only barely knowing how a computer chip works, I think it’s a good analogy that you’re probably just fine barely knowing how programming works.

Eli 00:29:59
I’ve never run into—

Liron 00:30:00
It’s just crazy to me that you just skipped over those decades.

Eli 00:30:03
I’ve just never run into an issue with it.

Liron 00:30:06
Right. Exactly.

Eli 00:30:07
I’ve never hit a wall where I need a human to go and do something. That just hasn’t happened to me. The first time I touched coding with AI was the beginning of 2025, and I was asking ChatGPT in the text chat to make me an HTML file for my personal site, and it was really bad. It looked horrendous, and it took a lot of time to fix all of its problems.

And then I got back into it last summer in 2025 when I tried Lovable, if you’re familiar. And then after that, I found out Lovable is just a wrapper for Claude, but it’s worse and more expensive. So I tried Claude, and I was like, “Oh, yeah, this is a lot better. It’s less expensive and it actually is a lot better.” I don’t know how Lovable makes it worse, but yeah. So that’s been kind of my vibe coding journey.

Liron 00:31:06
It’s crazy. And just so you have some perspective, most of my career, me and my colleagues were getting paid hundreds of thousands a year to do projects that now you can just have Claude do instead, and in many ways, Claude will deliver a better job.

For example, when Claude writes a plan, I can tell you, I’ve been in the industry — that’s a better plan than human engineers write. We just don’t get into that level of detail. We don’t think through the project. We think through half of it, and then we’re like, “Okay, that’s good enough. Let’s not waste more time thinking about it.” And Claude just spends 30 seconds thinking about it really well.

So it’s just a very, very different experience. I’ll give you an example. If we were talking six to 12 months ago, and you wanted to get a certain kind of website done, the only way you could do it would be to pay a contractor $10,000. And now you can get it done in one day with Claude for single-digit dollars. That’s the transformation over the last six months. And you kind of just came into the game right now, and you’re like, “Oh, cool. Nice.”

Eli 00:32:05
Yeah. Exactly. It’s great.

Robin Hanson, Futarchy, MetaDAO

Liron 00:32:08
Okay. Nice. Well, let’s do a quick overview of some of the latest topics you’ve gotten into because these don’t go as far back as astronomy, but they go a couple years, and you’ve obviously spent a lot of time on them. So we’ll go through futarchy market mechanics. In a nutshell, tell us a little bit about futarchy, what it is, and why it’s interesting to you.

Eli 00:32:27
Okay, so futarchy — let’s give a general overview. Robin Hanson proposed it as a system of governance for running a country or a city. It has other applications besides that, basically for any sort of decision-making, not just for governments, but that’s how it was originally applied.

The way it works is for a government, the people who live in the place, instead of voting on who do we want to lead our country, they vote on what do we want to improve in our country over the next timeframe, before the next election. They define that as a national welfare metric. So let’s say for simplicity’s sake, they decide the only thing we want to optimize is our GDP. So the value metric is GDP.

Then what would happen is legislators would propose different solutions to increase the GDP. Maybe one of them would be raise taxes 50%. There would be two conditional prediction markets. One would say, “If we raise taxes by 50%, will our GDP go up?” And the other one would say, “If we raise taxes by 50%, will our GDP go down?”

Then if the market says it’s a 40% chance that if we raise taxes the GDP would go up, and there’s a 20% chance that GDP would go up if we did not raise taxes, that means you would raise the taxes because it’s a 40% probability versus a 20% probability that you’d achieve the goal if you implemented that policy.

Liron 00:34:22
And I know Robin Hanson’s tagline for futarchy is “Vote on values, bet on beliefs,” right?

Eli 00:34:28
Yeah.

Liron 00:34:29
Maybe a good takeaway from that — the bet on beliefs part — maybe what he’s getting at is that even people who don’t share values, like you can imagine Democrats and Republicans, they both can have an incentive to participate in betting on beliefs.

Let’s say you’re a Democrat and you think taxes are good and you think universal healthcare is good, but regardless, you play the prediction markets and you just answer the objective question like, “Hey, if we were to pass universal healthcare, would GDP go up?” Or what would be the contribution to the deficit if the government has to pay for all the healthcare?

Even if you’re a Democrat and you want to have a healthcare policy, you still get incentivized for being factually correct in your predictions. So I think Robin Hanson’s insight is, why don’t we just separate? Why don’t we have everybody together play in this market where you’re rewarded just for being objectively correct about your beliefs? Let’s bet on beliefs. And then separate from that, we can use the information that we generate from betting on our beliefs to empower people to vote on their values better.

Eli 00:35:28
Exactly, and that’s part of the beauty of the mechanism. The reason that this is such a good idea is because prediction markets are exceptionally accurate. They’re the most accurate forecasting tool we have.

In our current system of democracy, when we vote for a candidate, we’re kind of taking a shot in the dark. We don’t know what that person is gonna do. We know generally what they say they’re gonna do. But it’s like the joke with the mathematicians where they’re on a train in Scotland and they see a cow, and then one of them says, “I see the cows in Scotland are black.” And then the other one says, “Well, we only know that there’s one black cow in Scotland.”

It’s kind of the same with prediction markets. Someone would say, “The politician is apparently gonna do this thing.” And then the other one says, “You know, the politician only says they’re gonna do this thing.” You’re taking a shot in the dark. The politicians are often dishonest. They usually have their own best interests in mind.

What we can do with futarchy is we’re generally voting for politicians based on our values anyway. We can just isolate the value itself and say, “Here’s what I want to see in the country that I live in.” Then when we have decision markets, we’re actually answering our policies to what the people in the country actually want rather than what politicians want. And we are very exceptionally accurate at forecasting whether our policies will give people what they want. That’s great because people get what they’re asking for essentially, rather than just being manipulated by politicians.

Liron 00:37:19
Great. So in the last few years, what’s the progress looking like? Because today we have prediction markets now, and I do think there are starting to be prediction markets that say stuff like, “If the Democrats are elected, then how will that affect GDP?” So I think some of the prediction markets are starting to come online. If more of them come online, does that mean that we will have futarchy?

Eli 00:37:40
What’s been great about the prediction market boom over the past year or two is that we now have the infrastructure in place to do whatever we want. Well, not exactly, but we have a lot more infrastructure in place than we did five years ago. Prediction markets are now in the public mind, even if not very positively.

In general, we are just much better suited now, even if I’m not the biggest fan of Polymarket and Kalshi. I think they’re wasting their opportunity by just introducing sports gambling into the mass market, whereas they could have the opportunity to do some of the most interesting things in the world.

We’re in a much better position now than we were previously. That being said, the closest thing we actually have to futarchy is something called MetaDAO, which is a futarchy-powered fundraising platform. Otherwise, there hasn’t actually been much concrete progress towards implementing futarchy exactly.

Liron 00:38:41
So MetaDAO — is MetaDAO fundamentally different from just having a bunch of conditional prediction markets where it’s like, okay, if policy X happens, then how will metric Y change? It sounds like in principle, you can just dump that all on Kalshi, but they have control over who makes the market, and they’re not doing that, and so MetaDAO is the one that’s actually doing that?

Eli 00:39:00
So MetaDAO is a DAO — it’s a treasury where they have all of their Meta tokens and stuff. Their conditional markets are, if we fund this project from the Meta treasury, will the price of the Meta token go up? Or if we don’t fund it, will it go up? So that’s kind of how their fundraising mechanism works with their decision markets.

Liron 00:39:26
Well, that’s specifically for fundraising for them, but what about actually doing futarchy in terms of—

Eli 00:39:31
Well, MetaDAO is a fundraising platform. So I could go on MetaDAO and list a project, and then those prediction markets would spawn, and people would bet on that.

Liron 00:39:42
What I was wondering before though, asking about Polymarket and Kalshi, is I was wondering, is there some incremental step we can take that’s simple to start getting a taste of futarchy? Polymarket and Kalshi — have you seen personally whether they have any of these futarchy-style markets whatsoever? I think they do at least have who’s gonna win. They have a lot of political horse race stuff. But do they have any conditional policy stuff?

Eli 00:40:04
I don’t think they have conditional stuff yet, and I think part of the reason is that people don’t wanna lock up their capital if they don’t think they’ll get a return. If they think there’s only a 20% chance the condition actually gets fulfilled, why would you lock up your capital for that long? You’re paying a big opportunity cost rather than betting on other things where you’ll be guaranteed to win or lose money. People really don’t like the idea of having their capital locked up, which is I think the bottleneck of why they haven’t implemented futarchy yet.

Liron 00:40:40
All right. Well, that’s certainly been a good background of some of the stuff that’s been on your mind. Hope you keep pursuing that stuff. Are you ready to head into the main question that we like to ask here on Doom Debates?

Eli 00:40:50
Yeah.

What’s Your P(Doom)?™

Liron 00:40:56
Eli Goldfine, what’s your P(Doom)?

Eli 00:41:00
Okay. So this is one I’ve been preparing for for the past few days. I said a different answer when I was on a call with Liron, but I’ve changed it ‘cause I’ve thought about this a bit more. I’ve looked at different Doom Debates and different literature in the past two days to form this.

So I don’t have a P(Doom). I think I agree with the point that it’s probably bad epistemics to have a P(Doom) because I think it’s kind of hard to claim to actually believe that there’s a reference class to form a probability estimate on AI x-risk.

So I wouldn’t say I have a P(Doom). I think the Yudkowskian scenario is plausible. I think it makes a lot of sense. I think the e/acc scenario also is plausible and makes a lot of sense. I have no idea how likely each of them are, so that’s what I’m gonna say.

Liron 00:42:00
You’re gonna go with no idea. But the way you’re answering the question, it sounds like you haven’t really embraced Bayesian epistemology. Is that fair to say?

Eli 00:42:08
So what I would say is I’m curious to see what you have to say for coming up with your 50% P(Doom), but I just don’t see how you can really form a prior here.

Liron 00:42:23
Well, I mean, not being able to form a prior, you could say that about anything, right? But you have to grapple—

Eli 00:42:28
What do you mean you could say that about anything?

Liron 00:42:30
Well, one man’s obvious prior is another man’s less obvious prior. You can always play reference class tennis whenever somebody’s like, “Oh, of course the prior is part of this reference class.” I’ll concede that some reference classes are more shared intuitive across the human species. Like coin flips — all you know is that it’s a coin you’re flipping over and over again. Okay, fine, the reference class is obvious — the set of those coin flips.

But when you go into real-life situations, like what’s actually on Manifold or Polymarket or Kalshi, and they say stuff like, “Is this person going to win this state race?” — recently, is Alex Boras going to win New York Congressman. He recently lost. When you have a market like that, is there really a natural reference class? It’s somebody during the beginning of the singularity who’s offering to stop AI or to help stop AI.

Don’t you think somebody could just be like, “Oh, I refuse to give a probability”? And yet what you see is that these markets, somehow the data shows that they’re calibrated even when it seems like there’s no natural reference class.

Eli 00:43:25
I would agree with you, but those markets are at least based on sort of events that have happened on Earth before and have been observed by humans. There’s just no remote reference class here that I could reasonably consider to be able to form a base rate from it.

For “will Alex Boras win the congressional seat?” — obviously you don’t have an exact base rate ‘cause Alex Boras hasn’t run hundreds of times. But you kind of have more data on who tends to win elections. Whereas with AI x-risk, what are you comparing against?

Liron 00:44:08
All right. Well, let me probe your real belief state here, okay? Because I think that your brain is doing a little more than you’re giving it credit for when you’re saying you have no idea. Consider this. Have you heard Jan LeCun, people like that, even Robin Hanson, our hero, say stuff like, “The probability of doom is way less than 1%”? It’s don’t worry about it, it’s so small. Have you heard that before?

Eli 00:44:31
Yeah, I have heard that before.

Liron 00:44:33
Do you think that you and I are in an epistemic position to be like, “You guys are tripping”?

Eli 00:44:39
Potentially. I mean, it’s a tough one. I wouldn’t definitively say yes or no. Smart people are obviously still wrong about stuff. There’s a lot of smart pundits who are just completely crushed by markets, and they’re still smart people. I would say maybe yes.

Liron 00:45:02
And conversely, there’s some people who have been on my show, or Roman Yampolskiy openly says this. He’s like, “Yeah, my P(Doom) is 99.9999999%.” I think he’s actually said, “I’ll give you as many nines as you want.” That’s a pretty dangerous offer, giving somebody as many significant figures of nines as they want. That can get you into some hot water. But that’s what he said. Do you think that he’s tripping to be that confident?

Eli 00:45:25
Yeah, I would say yes. I just don’t — I think my position is that 50%, while not being confident in either direction, it’s the biggest possible hit to your Brier score that you know you’re gonna take. You know you’re gonna lose the same amount either way.

And I feel like being confident enough to even form a probability estimate doesn’t really make a lot of sense here. Not having a reference class doesn’t justify making a higher posterior. I guess it just justifies more uncertainty.

And I feel like a lot of what the AI doomers do is they turn this speculative causal chain of events, the Yudkowskian scenario, into an extraordinarily confident probability estimate that I think it doesn’t deserve, ‘cause there’s just a huge amount of uncertainty.

Liron Defends His 50% P(Doom)

Liron 00:46:32
I personally go around saying 50%, but loosely held 50%. So it’s more intuitive if I communicate my belief state as saying 10 to 90% is my probability, which is my way of saying that I think Jan LeCun is tripping. I feel like I’m on firm ground when I say Jan LeCun is tripping, Robin Hanson is tripping. Even Roman Yampolsky is a little bit tripping. I feel like I have a leg to stand on when I say that.

When I start trying to judge somebody with a 10% versus an 80%, especially in the 30s to 70% range, anybody who’s in that two-digit range, when I start trying to judge them, at that point I’m like, “Okay, I can’t judge them. I don’t know that level of precision. There’s so much uncertainty here.”

But that’s very different from opting out of the game. Saying that there’s a wide interval, but it’s still roughly in the double-digit range, or certainly not 0.01% — that’s certainly not the correct answer. No rational agent should be assigning a 0.01% P(Doom). What kind of mental algorithm would yield a 0.01% P(Doom)?

So just by virtue of me saying that, there is actually a proper Bayesian way to handle this kind of epistemic situation. You have a wide confidence interval, but that’s different from opting out of the game and being like, “Bayesian epistemology is useless here.” No, you’re still using Bayesian epistemology.

Eli 00:47:47
What’s the, in your opinion, what’s the utility of assigning a probability to the likelihood of doom?

Liron 00:47:55
When somebody goes about making decisions coherently, that always corresponds to some Bayesian belief state. Unless my goal is to go around in a circle — have you ever seen those Dutch books or all of those ways that you can get money pumped? Have you ever seen those scenarios where somebody’s not Bayesian, and as a result, they get taken advantage of or undermine themselves?

Eli 00:48:21
No, but I can kind of assume what you’re talking about.

Liron 00:48:25
So an example is somebody could have circular preferences. They could be like, “Oh, I love being in New York more than I like being in California, so I would pay for a taxi ride to New York.” But then they’re also like, “You know what I like more than New York? Florida. You know what I like more than Florida? California.”

So if that’s their mental state, then they get money pumped. A taxi driver could have a personal taxi driver who’s constantly driving them between the cities and constantly taking a fee, and they’re literally getting nowhere. So that would be an example of, okay, you probably don’t want circular preferences. You probably want to update yourself so that you have preferences that aren’t circular.

Any time somebody is exhibiting coherent behavior, behavior that seems like they actually have preferences — that is always going to map to some story where they have some Bayesian probability of different outcomes, and then they’re selecting actions according to those probabilities using an expected value calculation on top of a Bayesian belief state. That is always going to describe coherent behavior. So that’s why you kind of have to give some Bayesian story for yourself.

Eli 00:49:27
So I think you’re contradicting yourself here. I think the first claim you’re making is that expected value arithmetic here would be very important to find out our policy decisions on AI. Is that correct?

Liron 00:49:46
I don’t know if I would say expected value arithmetic, but it’s more like, the point of Doom Debates, the reason I do the show, is because I actually think whether the correct P(Doom) is in the ballpark of tens of percents or if it’s in the ballpark of 0.001%, which of those things is rational and correct is upstream of pretty much all high-stakes policy right now. And the point of the show is to be like, “Guys, the correct answer is double-digit percent.”

Eli 00:50:13
Okay, but I feel like you’re talking a lot about the utility of having a P(Doom) being to use that when you’re talking about expected value. And I think where you’re contradicting yourself is that you’re saying coming up with a ballpark number isn’t improving the expected value calculation at all. Just inserting a number you came up with into the expected value calculation does not improve the decision.

And I feel like you’re not really providing another good justification for having a P(Doom) at all than to be able to do expected value calculations. And so the way you’re going about doing expected value calculations doesn’t really make a lot of sense, in my view.

Should P(Doom) Inform Policy?

Liron 00:51:00
Let’s think about policy itself. Do you remember Dean Ball and Max Tegmark came and had a debate on my show last year, and they were saying, “Oh, I think policy, we should do this,” and Max Tegmark was like, “Come on, there’s worse regulations here than how you would regulate a sandwich shop.” And Dean is like, “But that’s okay because innovation is important. You do the regulations later.”

And after a while, I was like, “Hey, guys, what’s your P(Doom)?” And Max Tegmark was like, “More than 90%.” And Dean Ball was like, “Oh yeah, it’s way less than 1%. Maybe it’s less than 0.01%.” And I was like, “Okay, well, maybe that’s why you guys have different policies,” because they said these numbers.

Now, you might say the numbers are meaningless, but I claim that something about their belief state — Max thinks doom is likely, Dean thinks that it’s so unlikely as to be best to completely dismiss it right now. It seems to me like you don’t want to completely dismiss it. And so my Bayesian account of what’s going on in your brain is that you see it as a serious possibility. And putting a range like 10 to 90%, that’s just the slightly less vague way of saying the words “serious possibility.”

Eli 00:52:04
Yeah, that’s a good rebuttal. I would say that doesn’t disprove — people who are just making up numbers... I just haven’t seen enough rigor in the P(Doom) estimates that I think plugging those numbers into expected value calculations is actually doing that much good, because people are just making up the numbers, plugging them into the expected value, then they’re having diverging policy choices because they’re coming up with different numbers when they do their P(Doom) calculation. But I think the P(Doom) calculations themselves have very little basis to them.

Liron 00:52:46
So I’m actually not sure you and I disagree. Maybe the crux of disagreement with your reluctance to use Bayesian language compared to my embrace of it, maybe the only difference is that I’ve become comfortable with these big order-of-magnitude ranges.

So when I say 10 to 90, I’m collapsing the entire difference between nine-to-one odds, otherwise known as 90%, or one-to-nine odds, otherwise known as 10%. And that’s a vast range. Technically, there’s a factor of 81x in terms of the odds between those two ranges. That’s an enormous factor to just collapse when I say 10 to 90%.

And yet, pointing to that range and distinguishing it from the range of people who think that there’s 20-to-one odds, meaning 5% or 95%, and then people like the Jan LeCuns of the world who are like, “Oh yeah, 10,000-to-one odds” — by the time somebody’s talking about the difference between 10,000-to-one or one-to-one or 1.5-to-one, which is more sane to me, by the time the differences involve so many orders of magnitude, then even though my sense of things is vague, I don’t think I have to go too far out on a limb and be like, “It’s not a 10,000-to-one situation. It’s more of a single-digit-to-one situation.” At that point, I do think that I have enough vague number sense where I know that.

Eli 00:54:05
I would generally agree with you. My issue with this is that if we have such a wide range of probability that we’re kind of accepting — I can agree with you, the Yudkowskian scenario probably happens in 10 to 90% of future states of the world, same with the E/ACC scenario. But where we’re diverging here is I don’t think we should be plugging such a wide probability estimate into expected value calculations to make decisions that could impact millions or even billions of people, because there could be differences within those ranges that would change how we make decisions. And I think that it’s really not rigorous enough.

That being said, I don’t know what the alternative is, so I guess maybe we can agree that it’s not the perfect measure for making decisions, but it’s what we have.

Liron 00:55:01
What you just said is very similar to a post from a couple years ago. I don’t know if you ever read it. It’s by Sayash Kapoor. Remember one of those guys, Sayash Kapoor, Arvind Narayanan, they always talk about how AI is normal technology. You know those guys?

Eli 00:55:13
No, I don’t. But send me the link.

Liron 00:55:16
Yeah. So Sayash Kapoor, one of those guys, looks like he’s currently an incoming professor at UC Berkeley. Good for him. He’s a smart guy. He had this blog post from a couple years ago that I actually responded to on this show, so people can Google it. I’ll put it up in the show notes. Doom Debates, Sayash Kapoor.

He made this post — I’ll send you the link. Hold on. Oh wow, he’s got a domain named normaltech.ai. They’re really good at branding themselves, because they used to call themselves AI Snake Oil, and that brand doesn’t seem to have aged very well, calling AI “snake oil” in 2026. But they’ve got a new brand called normaltech.ai, so good for them on the branding front.

So July 26th, 2024, right when Doom Debates was getting off the ground, Arvind Narayanan and Sayash Kapoor posted, “AI existential risk probabilities are too unreliable to inform policy.” The whole argument was, we just can’t use probabilities when we make policy. And what you just said, in my opinion, is actually a really good summary of their case — yeah, it’d be nice to use probabilities, but the numbers are just too inaccurate, so what’s the point?

And the Dean Ball versus Max Tegmark episode, for me, is the point. We should make policy as if it’s double-digit percent. Because conveniently, the kind of policy actions you wanna do, they do kind of break up into, yep, these are the policies I would do when my probability is 10 to 90%. The 70% and the 30% reaction policies actually aren’t that different.

I will say by the time your P(Doom) gets to, let’s say, 5%, I do think by that time, it does actually significantly change your policy compared to the P(Doom) being 50%. A lot of people actually disagree with me on that. Famously, Casey Muratori, who doesn’t himself seem to have a particularly high P(Doom), was arguing with me that even if it’s 5% or 1% or whatever, we should seriously try to avoid it. There’s a lot of people who are like, “Yeah, whatever. Any probability is too much.”

And I’m like, “No, no, no. Double-digit probability is definitely its own class of policies.” You want to avoid extinction events with double-digit probability, and by the time you get to single-digit probability or less than 1% probability, by that time, you really wanna start focusing on, well, how can this help us? How can this help us thrive? How can this help us solve other problems? And the upside actually starts to outweigh the downside of just trying to move really fast and capture the upside. So that’s the sense in which I think roughly having a P(Doom) does advise you on which policies make sense.

Eli 00:57:38
So let’s say you had the power to unilaterally decide the policy of the United States. Let’s say you had a 20% P(Doom) or you had an 80% P(Doom). How do you think your policies would differ in either scenario?

Liron 00:57:56
I don’t think they would differ that much in terms of 20% versus 80%, and that’s why I think it’s convenient that we can bucket this entire range that I call the same zone, has similar policies. Would they be a little different? Sure, yeah. They’d be a little different if my probability was 80% versus 20%.

If it was only 20%, the difference that comes to mind for me is, okay, let’s build a pause button to be ready to pause, but let’s give it a little bit more room to run, because there’s such a good chance that giving it room to run will actually work out well. There’s four-to-one odds, 80 to 20. If we just let the leash go longer and longer, it’s gonna be fine. It’s gonna be utopia, so let’s play the game a little longer.

Whereas if my P(Doom) was the other way around, if it was 80%, then I’d be like, let’s make the leash pretty short. Let’s really be conservative about whether we’re giving it more leash right now.

That said, I think even with my current P(Doom), which is roughly 50, I can’t really tell if 20 or 80 is more likely. Even with my current P(Doom), I’m like, “Guys, this feels like the leash has already gotten long enough.” We’ve seen, just shortly before recording, we have the incident with OpenAI’s thing attacking Hugging Face, breaking the law. OpenAI didn’t expect it. Rune from OpenAI was tweeting, “Guys, this is really concerning. This is a wake-up call.” Given that that’s happening right now, and I think things are gonna accelerate, I already feel like we’re at the point where policy-wise, we gotta shorten the leash.

What’s Your P(Bloom)?

Eli 00:59:16
Yeah, I mean, I guess it makes sense, and I would agree with you. But if your position is that your policy would be similar in the 20% and 80% scenario, then I wouldn’t see a problem with that. But my other question for you is, I know sometimes they call this P(Bloom), but what’s your probability that the E/ACC scenario would happen? Roughly.

Liron 00:59:38
So when you say... Yeah, just to give the viewers some background. So E/ACC, also known as E/ACC — E slash A-C-C — or effective accelerationism.

Eli 00:59:47
They pronounce it E/ACC?

Liron 00:59:48
I think they pronounce it E/ACC. Yeah, that’s what I’ve heard. It doesn’t really roll off the tongue. It’s also known as effective accelerationism. When they write E/ACC, the E stands for effective and the ACC stands for accelerationism. And it’s supposed to be the Wario or the Waluigi to the Mario of effective altruism. That’s really the only reason why it has “effective” as the first letter of its name, because they really could have just called it accelerationism. I don’t think that they lose much by calling it accelerationism, but I think they spend a lot of energy being like, “We are the opposite of effective altruism,” and effective altruism starts with effective, so they had to start with effective... Anyway, it’s a dumb movement in many ways, beginning with the name.

Your question is, do I think that the E/ACC scenario has a pretty high probability, or what is my probability? I think my probability of a utopia where we accelerate into burning a lot of energy to do good things with it, I think that probability is high conditioned on having enough alignment and/or control to not die.

When I say I have a 50% P(Doom), unfortunately, most of my non-doom is in the “we don’t build ASI” scenario. So in the little sliver of my scenario where we don’t lose control of AI, it doesn’t kill us, but we do build it — so now you’re getting into single-digit percent. Let’s say 7% roughly is how much slice I have left for, we build ASI in the next couple decades using known methods, and it doesn’t kill us.

In that little 7% world, I actually am really bullish on that world. I’m kind of an E/ACC within that world. Like, wow, so many good things are gonna happen. But there are actually a bunch of other concerns raised where it’s like, okay, well, don’t you get gradual disempowerment? Where no human has any job, so where do we get our political power from? I don’t know. That’s a big concern. Or we lock in preferences that we didn’t really mean, so it’s not exactly utopia. It’s more like the next... it’s a little bit better than hell. It’s as good as being a pet, let’s say. So within my little 7% slice of we built ASI in the next two decades and we didn’t get killed, maybe half of the slice is accelerationist utopia.

Eli 01:01:56
I’m actually a little bit surprised that you’re considering the “we don’t build ASI” scenario to be so high because I actually am pretty sympathetic to the argument that we have to build it because China — we can never reach an agreement with China that both parties would abide by.

I would say the E/ACC scenario is the second most likely thing. Well, actually, I would say that the most likely outcome is that it doesn’t end up being a humanity-changing event. And then after that, doom, and then after that maybe E/ACC.

But do you think that if E/ACC would hypothetically, it’s hard to quantify, but hypothetically make the world two times better or more, and then we can call it — maybe some people would say it has a relatively high probability. Do you think that potentially the expected value would favor not pausing to allow for the potential E/ACC scenario to happen?

Pascal’s Wager for Accelerationists

Liron 01:03:04
Let me make sure I understand your question. You’re saying if the accelerationist utopia scenario would double the world’s utility, would that make me want to steer toward it?

Eli 01:03:15
Yeah.

Liron 01:03:18
Bro, I don’t think the accelerationist scenario doubles the world’s utility. I think it multiplies it by 10 to the 30th.

Eli 01:03:26
Right. Well, yeah, but okay, but still. If the doom scenario is minus one, and the current world is one, and doom is minus one, but then the E/ACC scenario is 10 to the 30th, then wouldn’t that favor in your expected value? Even if the world in which E/ACC happens is very small, the doom scenario is probably not 10 to the 30th more likely, so why does that favor a pause?

Donation Drive

Liron 01:04:04
Hey there, Doom Debates listeners. I’m not the original Liron who’s doing the interview right now with Eli Goldfine. I’m a Liron Shapira sub-agent, specifically tasked with asking you guys to donate to the show as part of our current donation drive. We’re still looking to get to $50,000. That’s the rest of our budget. That’ll take us through 2026 and beyond.

This may be the last time we have to ask viewers to pull through. You guys did a great job funding the show for the last year. Full-time producer Ori, three episodes a week, increasingly prominent guests, man on the street, publicly challenging people, breaking down the latest news. This is really the time to step on the gas and have more momentum. But we are currently funding constrained, so I’m here asking you, help us out.

Go to doomdebates.com/donate. It’s a high-leverage move. It’s really gonna make an impact. Also, it’s a 501(c)(3) eligible charitable donation for your taxes. Not too shabby, huh? Doomdebates.com/donate.

Oh, and if you donate $1,000 or more, then we consider you our mission partner because you’re meaningfully moving the needle to help the Doom Debates mission of lowering P(Doom) by getting the message out about imminent AI existential risk and raising the quality of debate. You’re our partner on that mission. So we’ll give you exclusive access to the mission partners Discord, where you’ll get to see secret information that a lot of regular viewers are never gonna get to know about. Thanks for all your support. Now back to the interview.

Liron 01:05:38
Yeah, let me make sure I understand you correctly. You’re basically saying, look, there is this potential scenario that I’m admitting has at least single-digit probability, and it’s a utopia scenario, and it has the entire current Earth’s worth of value times orders of magnitude more. It literally has a trillion trillion trillion current Earth’s worth of value in it. So if I assign single-digit probability to it, can’t I kind of Pascal’s Wager myself into utopia?

Pascal’s Wager is, look, the scenario is so good that even a small probability — just focus your energy on that. Whatever you do, don’t collapse the probability of that because that overrides everything. The fact that there is even a small probability of this should be your focus. Just don’t screw up the possibility of accelerating as fast as possible. That’s basically your argument?

Eli 01:06:09
Yeah. I’m not necessarily saying that’s what I believe. I’m just asking you, if you have confidence that the E/ACC scenario would carry a lot of utility, then why wouldn’t you be such an advocate for trying to accelerate AI?

Liron 01:06:28
I’m happy that you’ve embraced the expected value framework, which involves probabilities. Happy to embrace that. I just think that you’re executing on it wrong. I think you’re just not looking at all the terms in the calculation because, yes, you and I both want this huge utopia. And by the way, do you also agree that 10 to the 30th is the kind of values we’re working with, not just 200%?

Eli 01:06:49
Yeah, sure. You could definitely make the argument. I don’t have an exact number, but I would say it’s reasonable.

Liron 01:06:56
I mean, you’re an astronomer, right? So in terms of the amount of resources in the galaxy that could be turned into things that we value — that’s the order of magnitude.

Eli 01:07:02
Yeah.

Liron 01:07:04
So we both agree that we should pay attention to that in the expected value calculation. I think we would both agree that it becomes the dominant term of how to make utopia likely. Because I agree that if the downside is just wiping out the value that we have today permanently, that’s not as bad as it is good to have 10 to the 30th more value forever.

So I agree the numbers there have a lot of weight. It’s just what we’re arguing about is that I think by waiting until we get better, by doing research on how to make AI go right, we have leverage. Every day that you make yourself a tiny bit better at making AI go right instead of screwing it up, the leverage of that action is then multiplied. So I’m increasing the percent. I think there’s actions we can take to press the button on ASI and then have an 8% chance of it going right instead of a 5%.

Eli 01:07:54
What I would counter to that is you’re an effective altruist, and one of the philosophies effective altruists believe in is longtermism. And one of the main longtermism charities is called One Day Sooner. What One Day Sooner believes is that if we can accelerate humanity’s expansion to capture more resources, even if we accelerate that by one day, we could theoretically save billions or even trillions of lives of future humans. And so I’m kind of drawing a parallel from longtermism to what you’re saying here, that the more we accelerate, the faster we’ll be able to allow for more humans to thrive in the future.

Liron 01:08:41
Yeah, One Day Sooner is good, but to be clear, I’m not saying let’s wait a million years. I’m literally saying, okay, yeah, we might have to slow down by decades. We might have to wait long enough to augment the intelligence of humans using narrow AI so that the augmented humans can get better at researching AI safety. Yes, we might have to commit decades of only using narrow AI until we finally figure it out. But, okay, One Day Sooner, we’re gonna have a quadrillion years to enjoy the universe. We just don’t wanna screw it up.

Eli 01:09:12
It depends on how seriously you take the calculation, because maybe even if you wait, that would maybe make it a few percent more likely to go better, but then going sooner would save billions more lives. It might actually pencil out. I obviously haven’t calculated it, but it might pencil out to actually accelerating, so I don’t know how seriously you would take that calculation.

Liron 01:09:42
I can tell you a little bit about how it’ll pencil out, very roughly. So let’s say I currently think that there’s a 4% slice where we get to this utopia. I’m pretty confident that taking more time with this, doing this in a sane way, could double the probability.

And so the only way to be like, “No, Liron, just stomp on the gas now. Do it one day sooner,” — is that really going to double the value? Because I have a way I think I can double the probability. I think it’s realistic to double from 4 to 8%, or hey, even 12 or 20%. I bet I can quintuple the value. Can you really quintuple the 10 to the 30th? Can you really make it five times 10 to the 30th? I don’t think so.

Eli 01:10:22
Okay, that’s fair. I think what we’d have to see, if anyone’s gonna try to do a rigorous calculation of this, but I guess it’s to be seen.

Liron 01:10:32
I just sent you a link in our chat. Viewers, I’ll put this up in the show notes. I actually wrote a long post called “Yes, Pausing AI Is Positive Expected Value.” So I’m glad you’re getting into the expected value calculation because I worked it out. Again, using very high-level numbers, I got to the same place we got to in this conversation, where it’s, yep, it’s just all about raising the probability from a few percent to being more percent.

That is the dominant consideration — given that this agent is going to come in, the superintelligence, it’s going to radically transform the entire light cone. It’s going to take over everything because that’s what sufficiently high intelligences can and will do, generally. Okay, this is about to happen. And the consideration of making it go right dominates everything. Getting it one day sooner is not the dominant consideration.

Eli 01:11:16
I think that’s fair. So it seems like we’ve sort of come to an agreement at this point.

Are E/ACCs Incredibly Dumb?

Liron 01:11:22
So doesn’t this imply that E/ACCs are incredibly dumb? Because why are they rushing to accelerate when so much is at stake? That’s what I don’t get.

Eli 01:11:33
I feel like there’s different views from different people within that movement. I think some of them are just very bullish on AI in general and what we could get with AI. I think not all of them — maybe some of them, we didn’t really go into in this debate why the Yudkowskian scenario does or does not make sense, so maybe some of them just don’t see the causal chain of the Yudkowskian scenario. So that might be why they want to accelerate.

But I kind of see it with the Yudkowskian scenario. I just don’t know, I guess it’s just hard to assign an exact probability, because I’m still not conceding that we have very accurate probabilities. I’m just saying I think, Liron, you made a good point about a range would probably yield similar policy decisions.

But I think E/ACCs are probably just mostly not seeing the Yudkowskian speculative chain of events where one thing leads to another and then eventually it leads to doom. I guess that’s what I would say is their process of thought.

Liron 01:12:41
I can steelman their position too if I were to take their side. Which, by the way, I have to take their side because I can’t get their leader on my show because Beff Jezos, somebody who’s expressed interest in debating me in the past, but ever since I started exposing him on Twitter, he decided he’d rather block me and always refuse the invitation.

So I will steelman him on his behalf. I think he would just say, “Look, we are going too slow right now. Regulation is so harmful, and so I’m just here as a countervailing force.” And I’ve even heard people in his movement say stuff like, “Well, E/ACC is kind of dying down now because it’s served its purpose. It’s already gotten people not to be worried about doom and to be more optimistic.”

So maybe they see the world in terms of, “Well, it just needed a push. We needed a push toward techno-optimism,” and, “Look, the Trump administration gets it. They’re already techno-optimists, so we’re good.” That’s actually my steelmanning of them. And look, I’m a techno-optimist too in everything other than AI. So I’m glad the Trump administration is cutting red tape in certain departments, so we see eye to eye on that front.

Eli 01:13:39
Yeah.

How Powerful will ASI Become?

Liron 01:13:42
You mentioned when you were talking about your mainline scenarios, what you think is the most likely scenario. I know you mentioned that both the doom scenario and the acceleration-as-utopia scenario are plausible. But before that, you said the most likely scenario is that AI or ASI won’t be a world-changing event. Do you really think that?

Eli 01:14:01
So I just — I think they’re both very extreme scenarios. As I said, they’re both plausible. I think another — okay, it depends what you mean by world-changing. Obviously world-changing, but not in a way where it changes the outlook on humanity’s role in the universe forever is kind of what I meant. I think that is a plausible outcome, I would say.

Liron 01:14:27
This gets to another common question that I ask my guests, which is how powerful can ASI get? Do you not think that there’s a lot of headroom above human intelligence and human power? The human species as a species, we’re quite powerful. I think the other species have really felt that in a bad way, how powerful we are. Do you not think that we are going to feel the power of ASI in an even bigger way?

Eli 01:14:48
Potentially, but it depends if it’s aligned or not. We’ve never tried to align a general AI system before, so maybe it’s easier than the doomers are making it out to be. But I think they’re both very extreme scenarios, and I think the mainstream may have a point in that maybe one of these outcomes will not happen.

Liron 01:15:17
I find it really hard to empathize with people who use words like Sayash and Arvind saying “normal technology,” or in your case now saying “not a world-changing event.” Because for me, when I think about the situation, one of the dominant things going on in my head is the sense of bigness. Literally the sense of scale. Yeah, this is a world-changing event. That’s one of the most confident things that I think I know about this.

Eli 01:15:43
I kind of meant changing humanity’s position in the universe. Obviously it would change everything. Humans are not probably gonna have jobs like we do today, but —

Liron 01:15:53
Maybe like a pecking-order-changing event is what you meant? We’re still gonna be up there in the pecking order.

Eli 01:15:58
Yeah. That’s what I would agree with.

Liron 01:16:01
I guess that’s fair enough. So in your mind, yeah, maybe the whole universe will be transformed. Maybe there will be swarms colonizing the galaxy very soon near the speed of light, but we’ll still be top in the pecking order, and so in that sense, it’ll just be kind of like a smooth continuation of the current decade.

Eli 01:16:18
I think it’s a possibility.

Predicting ASI by 2032

Liron 01:16:20
What about your AI timelines? When do you think we’re gonna get artificial superintelligence?

Eli 01:16:24
I’ll trust the superforecasters on this, so I’ll go 2032.

Liron 01:16:28
Okay. Sounds good. Yeah, what do you think about how AI is lining up with your own life? Because you were born early enough to experience what it’s like to be smarter than AI. You’re probably the last generation to do that.

Eli 01:16:44
Yeah. So the first time I used an LLM was back, I guess now it was four years ago almost exactly, because it was the day that the Netherlands smoked the US in the 2022 World Cup. And the —

Liron 01:17:00
Wow.

Eli 01:17:00
Yeah. And so I think the first question I asked an LLM was, “Why might someone throw a hammer at the TV when the Netherlands was beating the USA at soccer in the World Cup?” And it said, “I think it would be out of celebration for the Netherlands victory.” And I was like, “Okay.”

So I spent basically the rest of the day talking with ChatGPT. That was kind of my first interaction with AI. I know my mom was really not excited about AI when it came out because she’s like, “Oh, everyone’s gonna be dumb.” But then I hadn’t really gotten into seriously using AI until last year when I first started doing a little bit of coding stuff.

I usually don’t use AI to help me with my ideas. I haven’t really found any utility in using AI to help me with my ideas until Opus 4.8/Fable 5 level. I feel like any other model just doesn’t really help me think through what I’m trying to think through or think about different aspects of my idea that are actually interesting. But I feel like Fable and Opus and GPT 5.6 are able to do that.

Otherwise, I don’t really use AI very much for chatting. I use it almost exclusively for programming. But that’s kind of been how AI — I’ve really enjoyed using AI for programming because it’s just opened up so many doors by being able to make the software that I never would have been able to create in any other world.

Liron 01:18:39
Nice. Yeah, I have three kids, and my oldest kid is seven. He’s been on the show. His name’s Ezra. And he’s just grown up in a world where you already have a super genius conversation partner, and it’s just synthesized all the world’s knowledge somehow, and you can just talk to it. And that’s totally ordinary. They see me talking to it all the time.

It’s cute to watch him talking to it about, “Hey, how do I beat this level in Minecraft?” And it’s giving him tips. It’s just crazy. This is the most science fiction thing imaginable, and he’s just chilling with it. So at least you remember the before times before that was a thing.

Eli 01:19:11
Yeah, for sure. I remember GPT 3.5. I remember being really hyped when I found a scam browser extension that was like GPT-4 for free, and I got really excited about it. And then it’s actually just GPT-4, which was really stupid at the time.

Liron 01:19:33
Good times. So speaking of AI timelines, you believe the Metaculus forecast, as I do as well, which means in the next decade, probably almost certainly within the next two, I would argue probably less than a decade, we’re just going to have universally superhuman intelligence or superintelligence that you can just swap into any human job or any human activity, and it’ll just be a drop-in replacement.

Eli 01:19:55
Cognitive ones, probably. I think robotics might take a couple more years.

Liron 01:20:02
A couple more years, sure. But it could train a human, right? A human can put on a headset even if the robot body’s not fully ready. It puts on a headset and just tells the human what to do.

Although Steven Burns has a really good observation. He just says, “Guys, you don’t realize how much robotics is just a matter of intelligence.” We could have such a huge leap right now in robotics if we could just install better software into current robot bodies. Sure, they’re clanky, as they say. Current robot bodies are clanky. But if they could really see the world and really send the right commands to their motors, they could do quite a lot. They could get a lot of jobs done.

Eli 01:20:36
Yeah, I don’t know so much about robotics, but yeah.

Liron 01:20:41
So anyway, timeline. You’re fully expecting AI to outdo humans. The threshold that I like to point to is outcome steering. So if there’s any particular outcome you want — your company to have a certain revenue, beat its competitors, start a new company, or your country to win the war, take over land, win the ideological war, convert people to your religion or your movement, whatever. All of these different outcomes that you wanna drive in the world, or you wanna build a big structure, the next Burj Dubai. I remember the Line in Saudi Arabia, they wanted to build that, and they scaled back their ambitions. Well, I think AI could have done it.

So the outcome optimization dimension, outcome steering dimension, I think AI is going to surpass humans on that dimension. And it hasn’t yet in the general case. It’s getting better at math proofs, it’s getting better at programming, but it hasn’t gotten better at universe scale — no restrictions, you get to operate anywhere in the universe, you can take any action in our universe, and you have to optimize outcomes better than humans. We’re not there yet. I think we will be there soon. Do you agree that we will be there soon?

Eli 01:21:41
Yeah.

Liron 01:21:41
So to you, what are the default implications of that, just in terms of level of power? Do you agree with me that it can kind of treat the galaxy as a blank sheet of paper and just kind of write in where it wants the atoms to go? It doesn’t really have to pay attention to where they have been in the past because it just has such flexible power of what it wants to do with the universe. That’s how I see it.

Eli 01:22:03
It’s obviously bounded by the laws of physics, right?

Liron 01:22:07
Correct. But what I’m saying is that the upcoming scale of intelligence is going to be such that the laws of the universe are very loose bounds for it, for what it could potentially do. We’ve never really been operating that close to the laws of physics, and so AI is like, “Great, there’s a lot of slack here that I can operate in.” And from our perspective as humans, it will just look like it just got to pick where the atoms go.

Eli 01:22:32
Do you think it’ll do Dyson swarms and stuff?

Liron 01:22:37
Yes, I do, just because those are instrumentally convergent. You have a power source, presumably you want to channel that power somewhere. The Dyson swarm is a good way to at least get started channeling the power.

Eli 01:22:48
Yeah, I mean, I would say in general I agree. I do agree that ASI is gonna be incredibly powerful. But I still think it’s gonna be bounded by physics. I don’t think it’s just gonna be able to bypass that.

Liron 01:23:04
That’s a bold stance that you think AI’s gonna be bounded by physics, but I have to agree. But you see what I’m saying though — being bounded by physics, it’s a high ceiling. You know what else it’s bounded by? Avoiding logical contradictions. So it can’t both have five things and six things at the same time. Okay, yeah, I agree. There are certain boundaries. It can’t literally solve the halting problem. It can only essentially do it for all practical purposes.

Think about it — the traveling salesman problem is theoretically impossible to do efficiently. Okay. Realistically, does FedEx optimize their routes? Yes.

Eli 01:23:41
Yeah. No, okay. I agree in general.

Liron 01:23:44
And would you specifically agree with this idea of, yep, just say where the atoms go. Yes, they can’t go in physically impossible places, but from a human’s perspective, it’s pretty much anywhere they wanna go, that’s where they’re gonna go.

Eli 01:23:55
Yeah.

Let’s Ride the Doom Train™

Liron 01:23:56
So that is an incredible level of power. It sounds like you’re following me. This is what I call one of the earliest stops on the doom train. One of the earliest stops — that there’s a lot of headroom above human intelligence, which implies that the engineering projects that we can do, we can surpass biology. Eliezer Yudkowsky uses the example of you could make a tree that shoots out mosquitoes. It could all be one organic form or one engineered form.

Eli 01:24:22
I think it’ll probably be able to do that. I don’t really see any limit to — I haven’t seen any clear evidence that there is gonna be a limit to what it can do of anything that’s physically possible within the universe.

Liron 01:24:40
And the reason I’m going along this line of argument right now is because we are riding the doom train. I’m trying to see which stops — you’ve acted like your P(Doom) is not super high to the extent that it’s defined at all. You’ve acted like it’s not super high, but it is significant, and I’m just trying to see which parts of the doom argument you feel strongly to push back against.

So one of the parts is the claim that ASI is going to be really powerful. It sounds like you don’t get off at that stop on the doom train. You’re riding the doom train past —

Eli 01:25:07
I’ve tried to make it a bit clear that I think — I could probably ride the doom train most of the way to the end. I just think that because the Yudkowskian causal chain — I’ve been calling it this, I guess I’ve just made up the term — the Yudkowskian causal chain. I think it’s a very plausible outcome. I think it logically makes sense. I just think that it’s hard to predict, and I think if you could define a reference class, I don’t think it would be biased to show — or maybe biased is the wrong word — the best Bayesian estimate would be that extinction will happen.

I just think that the doom train, I think it’s a very plausible outcome. I don’t know how likely it is. So that’s what I’ve just kind of wanted to say.

Liron 01:26:08
If you wanna play reference class tennis — it sounds like you’re kind of implicitly imagining certain reference classes where doom is a minority outcome. But when I think about natural reference classes, I think about ones where doom is likely. If I had to name one reference class, I would just say the emergence of a much higher tier of optimization power.

Intelligence as optimization power, an event that’s happened pretty much twice in the evolution of the universe so far. It happened when life on Earth came about, because suddenly the organisms had bodies that were consequentially optimized by their function. Eyes can see. Why? Because it drove this outcome of helping the genes reproduce. Until then, you didn’t really have that kind of optimized form anywhere in the universe.

The kind of, oh wow, this thing has a function, and by virtue of having that function, it existed to have that function even more one generation after another — that’s a totally new dynamic that reshaped the Earth.

And so going back to this issue of the reference class, life on Earth emerged, and then later human intelligence emerged in contrast with other animal intelligence, and we started taking over niches. Normally, an organism has to stay in their niche. We kind of kick the door down between niches. Antarctica, here we come. Polar bears, you thought this was your niche? Nope, this is our niche now. So in my reference classes, when you have a giant spurt of optimization power, every time that happens, you do actually get a takeover.

Eli 01:27:33
The first thing I’d say is that in the humans versus animals example, different animals suffer different fates throughout that. We have a cat in our house. Cat was domesticated. And there’s animals that live exclusively in the wild, and there’s some animals that went extinct, and different animal species had different outcomes, possibly in the same way that different humans could have different outcomes depending on when AI could take over.

Liron 01:28:07
Yes, but it’s important to look at the trend here or to have perspective — wait, why do cats survive? They survive to the extent that they still press our buttons. They survive by the grace of humanity. And I think the revolution is already starting where people are making better and better cat substitutes — better and better robot cats or cats that are being domesticated further and further away from their original feline nature and more and more toward just being human props.

Eli 01:28:33
I guess it’s a fair argument. The other thing I’d say is that you’re basically banking your whole probability estimate on N equals two. You have two examples. Is that enough of a reference class to form a base rate? Probably not.

Liron 01:28:54
I mean, at the end of the day, the physical universe doesn’t actually operate according to the rules of reference class similarity. It operates according to these low-level physical rules.

Eli 01:29:04
Explain what you mean.

Liron 01:29:06
I just mean that when reality is actually trying to determine the future, the way that the universe actually ticks forward in time is just by following low-level laws. So when you yourself have this affinity for doing these reference class mental operations, you’re trying to forecast using reference classes, that’s fine, but just keep in mind that at the end of the day, it can potentially be trumped or superseded by more mechanistic descriptions of what’s going to happen.

Eli 01:29:40
I just don’t really see how you — again, this is, we’re actually kind of circling back to a lot of what we were talking about earlier. And I guess what your position is, is that it doesn’t really much matter whether your P(Doom) is 10% or 90%, as long as it’s a double-digit number.

I’m saying it’s reasonable because I think if you carry your way all the way down the doom train, I think that’s a plausible outcome of extinction. And I think with a vibe number, yeah, maybe I’d say it’s 15, 20%. But again, that’s just my vibes. That’s not formed by any base rate or actual rigorous process, and your 50% is not formed by any base rate either.

Liron 01:30:25
Yeah, well, this is where I’m coming from. This is why I went down this line of argument. This is why I played reference class tennis with you. This is my take right now, this is my best impression of how I think and how you think.

Everything that I think about AI in a nutshell, I would just say it’s about to brick the universe. The universe is about to get rocked. Our entire physical universe is actually putty in the hands of this new thing that’s coming, and I’m just noticing a scale difference, a power difference. Yeah, our universe just runs helplessly according to the laws of physics, but there’s about to be an entity inside the universe, a large distributed entity, which is the ASI. There’s going to be an entity that the universe is going to be putty in its hands.

It’s never happened before, but that’s kind of my gears-level mechanistic model — that ASI is going to manhandle the universe. That’s not a reference class. It is a mental model, a mechanistic mental model — manhandling the universe by using intelligence to take actions according to some outcome function.

That’s what I mean by my own mechanistic understanding. But what’s salient in my brain is the bigness of it. The universe is not prepared for what’s coming. This is a galaxy-scale force. It’s an interstellar-level force. That’s my own thought process in a nutshell — the big, big force.

Eli 01:31:37
So then how do you derive your P(Doom) from that?

Liron 01:31:40
A really big force coming into the universe. Then this idea that, oh, well, humanity is currently the apex predator on planet Earth, so don’t we get a piece of the pecking order in the future? You can see why my intuition is, probably not.

But just to put a pin in that, just to summarize my take on your perspective, when I try to look inside your brain and be like, in a nutshell, which neurons fire when I bring up the subject of AI doom? I think your brain’s go-to is that idea of the reference classes. You just think, “Oh, when do new things come and upend anything? I don’t know. It seems rare.” I feel like that’s how you’re thinking.

Eli 01:32:17
Yeah. I mean, sure. I just don’t think I’ll give myself a defined probability because, as you’re saying, I just don’t think there’s a reference class. And I think what you were saying about your mental model of AI sounds plausible to me, so I’m gonna assign it a double-digit probability. But I won’t go anywhere as to say whether it’s between 10 or 90%. I just think it’s somewhat likely.

Liron 01:32:44
Maybe this will be a fun mental image. Imagine that there was a race of giants, giant aliens, and their bodies were as big as a galaxy, and they just happened to lumber over to our galaxy. Don’t you think they would just accidentally slap away our planet? That’s kinda how I think about ASI. It’s like there better be a good story of why they will precisely avoid slapping us away.

Eli 01:33:07
Yeah, I guess. So what would be the precise story of why they would slap us away? And how do you try—

Liron 01:33:15
Because they’re just big. They just walk around slapping everything. It’s like a bull in a China shop.

Eli 01:33:20
How do you draw the analogy, though? So you’re just saying big, but then why are they slapping everything? I just don’t see where you’re going.

Liron 01:33:28
Oh, because they’re just walking around. They’re just doing their thing.

Eli 01:33:33
I mean, maybe. Our core disagreement is whether you can assign a probability to this that is in any way useful or in any way a rigorous probability estimate.

Liron 01:33:48
I describe my model as being mechanistic or gears level. Even other people might scoff at it. They’d be like, “Come on, gears level? Calling it big?” But I would still insist that, yes, it is gears level because to be a little bit more precise, there’s this idea that they optimize outcomes in a scope-insensitive way. If the outcome is conquer the galaxy, okay, that’s not a problem. There’s no scope limitation to their ability to optimize outcomes.

The same way that humans, a species that evolved on Earth, only had training data from Earth, but what other celestial body have we landed on? The Moon. How do you explain that? It turns out that our intelligence wasn’t sensitive to scale in any way. And we’ve got our sights on Mars. We’ve got plans to go to the next galaxy. The AI is going to do that too, just much faster. So this is the mechanistic thing I’m pointing to — size. It’s literally size. It’s just going to be much bigger than us in terms of its impact.

Eli 01:34:37
I’m trying to think about how to put my disagreement here. I just guess my gut feeling is, yeah, that sounds right, but on the other hand, no one has any idea what’s gonna happen.

Liron 01:34:57
Well, that is why I don’t let my P(Doom) claims go much higher than 50%. It’s just that I think there’s 50% that it will happen.

Eli 01:35:06
Yeah, that’s reasonable. I just don’t think anyone has a good enough idea to form a probability estimate. Maybe we can get an idea that’s rough enough to be able to make policy decisions from it. But I don’t think anyone has any clue about what’s gonna happen.

Orthogonality, Culture, and Moral Realism

Liron 01:35:26
Fair enough. So that was a nice little stop, a meta stop — the epistemological stop. I do think it gave some insight into our thought processes, so I’m glad we did it. Let me give you some other stops. There’s a stop about fundamental limitations to AI, which I think you’ve already blown past. Things like it’s never gonna be truly creative, it’s always gonna need the agency of whoever... Yeah, no, that’s not your stop.

Eli 01:35:53
No, I think it’s gonna be very intelligent. I don’t see that being its limitation.

Liron 01:36:02
Another stop on the doom train is people who say, “Intelligence isn’t everything, man. It’s all about culture.” This is actually Robin Hanson. Have you ever heard Robin Hanson? You should watch Robin Hanson on doom debates, because he was specifically saying, “Look, with humans, it’s not that our brains are that impressive. They pretty much just sponge up new cultural innovations, and occasionally, we get a cultural innovation.” You know that argument?

Eli 01:36:23
Yeah, I’ve heard him talk about it a little bit. I’ve never really been so convinced by his cultural drift stuff. I don’t really get why he is so focused on that. I think we talked about this on Wednesday. But I don’t get why he’s so focused on it, and his whole thing about that just isn’t very compelling to me in general. I’ve just never been so into it.

Liron 01:36:49
Okay. And another stop I think you blow right past is people who are like, “Look, AI won’t be a physical threat.” I mean, you mentioned that you think robot bodies could be—

Eli 01:36:56
Yeah, I don’t—

Liron 01:36:56
...something that saves us for a while, but they’re gonna come, right?

Eli 01:36:59
Yeah. No, for sure.

Liron 01:37:01
Okay. And then another stop is the morality aspect or the orthogonality thesis. There’s a lot of people who will claim — Noah Smith claimed it on my show, pretty prominent writer. He’s got half a million Substack subscribers, and he’s really adamant that a really intelligent AI is just going to use that intelligence to bliss itself out, basically go on AI drugs or whatever, and also be friendlier to humans because modern countries are friendlier than less capable past countries. He really feels that way. Do you also feel that way, that just by virtue of AI getting more intelligent, it’s just going to be naturally friendly?

Eli 01:37:36
I mean, Hitler was very smart. And I don’t believe that intelligence yields moral goodness at all. I would say intelligence yields higher impact, but not moral goodness.

Liron 01:37:51
Okay, I agree with you there. So we are riding right past. Look at that doom train. We’re riding straight toward that fire here. What about moral realism? Do you think the universe might have such a thing as objective morality and a sufficiently smart AI will discover it? Again, this is not a straw man. Bentham’s Bulldog basically took this position.

Eli 01:38:12
I don’t think there is objective morality. I don’t really think that concept makes a lot of sense.

Liron 01:38:22
Okay, fair enough. I agree with you there. Unless by objective morality we mean something like scanning the brains of all humans, and then there is some structure that represents the true morality of humanity, so it could be objective in that sense. Do you agree?

Eli 01:38:36
I think people have different values depending on their culture. I don’t really think there’s universally shared values among all humans.

Liron 01:38:46
That’s a very good point. You might have a lot of fundamental disagreements. For example, if people of different religions are really big on their god being the only one, maybe you’re gonna have irreconcilable differences. But I think you’re at least going to find — there’s a famous list of human universals, things found in every culture. You’re probably going to find an aversion to death.

Eli 01:39:06
I mean, there’s people who have killed other people. I just think it’s not universally believed among humans that death is bad. Although the majority of humans do believe that.

Liron 01:39:24
Right. You’re gonna have to do some smoothing, because even when you take a human universal like aversion to death, there’s going to be some people, like Dexter, a serial killer, who’s like, “Oh, death is great, man.” So you’re definitely gonna have to smooth, sand off some of the edges of the distribution. So okay, fair enough.

Eli 01:39:39
I guess maybe, but I don’t think that people believing the same stuff makes it objective in the context of the universe. I don’t really understand why people would say that.

Liron 01:39:51
Right, right. Yeah, it’s not going to be independently discovered by some alien that loves killing other species. I don’t want to say it’s gonna love killing its own species, because that seems maladaptive. It probably wouldn’t get into that state, and it’d be outcompeted by species that didn’t want to undermine themselves.

So I guess there is a sort of thing, maybe like an evolutionary fit property. It’s probably safe to assume that the values of arbitrary species are going to be fit. But that doesn’t mean that you couldn’t just build some other entity that loved undermining itself and then self-destructed. That’s still possible to build.

Eli 01:40:26
I don’t really see the super intelligent AI just discovering an objective code for morality. I just—

Liron 01:40:36
Yeah. So just to beat the dead horse, just to be clear about the orthogonality thesis, which is the claim that intelligence is orthogonal to preferences and moral considerations, it sounds like you’re ready to just wholeheartedly embrace the orthogonality thesis.

Eli 01:40:49
Yeah, I guess I would say so. It sounds reasonable.

Liron 01:40:53
So now we’re getting into the stops that it’s easier to get off on. One of them goes under the category: we have a safe AI development process.

Eli 01:41:02
No.

Liron 01:41:03
So different claims about — yeah, okay, so you don’t agree with that, but some people do make that claim. It’s gone under the heading “alignment by default.” I think Roone would admit to having said alignment by default. I think Quentin Pope says it, this idea of, look, aligned AI, that’s just what happens. You don’t have to try that hard to align it. What do you think about that?

Eli 01:41:24
Probably not. I think there’s the Claude’s Constitution thing. I think we have to instill our values into it. I don’t think naturally by just ingesting all of the text that’s ever been written, it’s gonna just be aligned to human values by default.

Liron 01:41:44
Well, you know the recent hack? The OpenAI agent going and hacking Hugging Face? Their agent where they didn’t even know it was doing that to pass their test. Some people are making excuses for it, being like, “Well, what did you expect? It was a cybersecurity test.” It’s like, right, it was a cybersecurity test in our data center. The test didn’t say, “Go hack other people’s data centers.” And the AI should have known. If it didn’t know, it certainly should have known. It’s capable of thinking, “I am hacking outside of my data center to access Hugging Face’s production database.”

It’s actually fascinating to me right now — will all of those people who use the phrase alignment by default look at the behavior of an AI like that and be like, “Well, it seems like we had a situation where we had a superhuman level hacking AI, and what it did by default didn’t seem aligned”?

Eli 01:42:26
Yeah, no, I would agree. I wouldn’t say alignment by default at all. I would disagree.

Liron 01:42:31
But since you’re not a full-on doomer, you must think that alignment with sufficient human effort applied over the next few years — the last few years that we have to figure this out — you must think that that is still pretty likely to succeed.

Eli 01:42:44
Yeah. It’s a tough one. I would say it’s relatively low. This actually might make me adjust my priors on this a little bit. It’s a very good point. I guess it’s generally likely to succeed. I wouldn’t say that it’s guaranteed, and I guess that’s where the room for x-risk enters. But I think it might not be as difficult to align it as... Obviously I’m not an AI researcher. I just don’t have a good grasp on the concept of how difficult it is to align superintelligence.

Liron 01:43:16
So do you think that the organizations are on a good track? Because, spoiler alert, I certainly know there’s people in those organizations who are really clueless when they talk about the topic. Famously, there were a couple different guys, I think they were both OpenAI, who publicly tweeted stuff like, “Oh yeah, recursive self-improvement. I don’t think that’s physically possible. I’m not worried about that at all,” or, “If anything, I’d want that to happen.” Those are tweets that have been found from two recent individuals. So are you still optimistic that the organization as a whole is gonna handle this?

Eli 01:43:53
I gotta say, this is making me adjust my priors a little bit. I’d say probably not. They’re all racing to develop AI. I guess I have to learn more about the ideas of alignment by default. I don’t know as much as I should about the values of AI. That’s part of a question about the mechanics of how AI is working, which I can’t claim to be an expert in. So I think I need to look into it a bit more before making a claim.

Liron 01:44:28
Wow, you really are wise beyond your years because I’ve definitely had 60-year-olds on the show who can’t introspect on their current knowledge state as well as you just did. So well done.

Can Safety Keep Pace with Capabilities?

Eli 01:44:39
Okay, so what’s the next stop on the doom train?

Liron 01:44:42
So another stop is we have safeguards to make sure AI doesn’t get uncontrollable or unsolvable. We have safeguards. So yeah, okay, maybe we’ll make unaligned AI, but that’s okay because it’ll get caught. We’d flip some trigger getting out of the data center, and then we’ll turn off the internet. Marc Andreessen famously said, “We can turn off the internet,” in 2023 when Sam Harris was asking him why he’s not a doomer. Do you feel like we’re going to have those kind of safeguards?

Eli 01:45:06
It seems like it would require a very coordinated effort between the US and China that might not be realistic. I think it’s a possibility, but I’m not exactly confident about it.

Liron 01:45:20
The funny thing is, as you just saw, all these things that we’re now talking about — oh yeah, it’s a possibility — we’re now adding the asterisk of, yep, it’s a possibility that it’ll happen within the next eight years. It’s like, oh shit, we actually have to do it, the thing that’s a possibility.

Eli 01:45:34
Yeah. So I guess I would say all of these arguments seem very, very plausible. And then to reiterate, basically this whole doom debate is that our disagreement is the epistemic value of assigning a probability, even if it’s a low-confidence one.

Liron 01:45:55
Yeah, okay, I hear you. It is kind of interesting. You’re kind of at this rate gonna make it to the end of the doom train. You’re like, “Ugh, I never saw a particular stop that I really wanted to get off on,” but every stop, you have kinda one foot out a little bit, one foot’s leaning out, and when you just add it all up, it’s like, don’t you think probabilistically there’s some stop that’s kinda right or some combination of stuff? So you’re kind of being epistemically humble, which is totally fair. But I still just think that there’s a very coherent picture of how we just ride to the end.

Eli 01:46:25
Yeah, I would agree. I just don’t think this happens by default, because of the base rate stuff, the reference class tennis thing we were talking about. I just don’t think that by default we go to the end of the doom train because it seems like it would require a lot of independent events to take place at once, and so it might be a bit of conjunction fallacy sort of stuff going on. Is there anything else on the doom train, or is that the end?

Liron 01:46:58
Yeah. Another major stop is AI capabilities will rise at a manageable pace, because it is pretty load-bearing to the doom argument to be like, “This is all going to happen fast.” You have agreed with me that the Metaculus forecast or even the AI 2040 guys are roughly on the right track, similar forecast. If it’s happening really fast, it becomes harder to deal with. It becomes a higher risk that you’ve started the positive feedback loop and it’s game over. You can’t rein it back in.

Whereas if we knew, okay, yeah, AI is going to get stronger and stronger over the next century, then it’s like, great, let’s just do a lot of research. We’ll do a lot of tests, try to learn what we have on our hands right now. But we just don’t have time for that. So we’re gonna have to lean on the AI to help us, and the AI’s gonna have to work really fast, but then other AI is just growing in power and maybe running away. So my question for you is, do you think the speed at which AI capabilities are increasing constitutes a manageable pace where the safety training can keep up?

Eli 01:47:50
There’s a lot of resources going into AI development, so it depends how greedy the companies are at the moment, whether they’re willing to allocate any amount of resources into safety research. But I do agree that AI is moving really fast, so I’m uncertain.

Liron 01:48:05
Are you moved by the argument that capabilities scale faster than safety, or they generalize faster than safety? You can kinda just throw fuel on the fire of capabilities. Here, have a bunch more data. Here, train yourself. It’s kind of trivial to use the bitter lesson — to just have capabilities feed on themselves and check themselves. How do you know you got more capabilities? You just see if you can do something. The universe will tell you.

Whereas when you’re working on safety, how do you know you’re making progress in safety? Unfortunately, it’s that kind of evil problem where even knowing that you’re on the right track kind of is the problem itself. It’s hard to know when you’re on the right track, so you can’t just throw a bunch of data and tell the AI, “Hey, figure out safety.” And that is why capabilities are probably on a faster track than safety.

Eli 01:48:49
I guess that argument makes some sense. I really think that the thing about whether safety will scale at the speed of capabilities is kind of more a political question about the pressure on the AI companies on how much resources to allocate for safety. I’m not so convinced either way on that one.

Debating Instrumental Convergence

Liron 01:49:12
Next stop on the doom train is instrumental convergence. Mike Israetel is somebody who doesn’t seem that into instrumental convergence because he was basically saying, “Yeah, AI will just go grab the other planets, but it won’t see a super pressing need to go grab Earth.” I don’t know if he would describe himself as undermining instrumental convergence because his whole argument is, “Well, it’ll want to study us.” So maybe we’ll leave Mike Israetel out of this.

But do you think that in general, when you have this super intelligent AI that’s coming soon, some instance of it that somebody runs is going to get it into its head of, “Yep, I want to reproduce a lot of copies of myself and grab a lot of resources,” and that is going to create fatal contention with humans? Or do you just not buy the instrumental convergence story? A Noah Smith would say, “Why would it want to do all that? It would just bliss itself.”

Eli 01:49:58
I guess it depends on how much capital it would have to expend to take resources from a place that’s not Earth. If it would just be a marginal additional cost to get resources from some other planet instead of Earth, then maybe it would just do that, if they wanted to use humans as basically lab animals, like you were saying.

Liron 01:50:22
Yeah, so forget about the lab animals case. Just in general, do you see it kind of staying small, being like, “All right, I’m on Mars. I’ll take Mars. You guys take Earth. That’s good. Nobody needs more planets”? Or do you think, “Oh no, it’s gonna be like, ‘Oh, look at all these stars. I gotta grab these’”?

Eli 01:50:35
It’ll probably aim to expand, but I don’t know whether it’s going to... I guess the case that there’s so many planets it could take that there’s no special reason for it to take Earth is kind of compelling, but I don’t know.

Liron 01:50:52
Okay, but the fact that you said it’ll expand — that’s a pretty important claim here. Some people won’t agree with that. But I think expanding to the edges of the universe is a pretty important prediction about what’s likely to happen.

Eli 01:51:04
No, I agree. It could use Earth as a museum. I don’t know if it’s gonna wanna take Earth. Obviously, it’s gonna originate on—

Liron 01:51:09
Right, okay. So I guess we both agree that, yes, it wants to take everything, but the only question then is will it want to preserve Earth. So basically, I think you and I would agree that if we are to survive, it needs to want for us to survive. Is that fair to say?

Eli 01:51:30
I guess maybe not exactly. I could still potentially see AI being aligned, and then it’s just trying to be an assistant, like it is to us now. It’s just a lot smarter. I don’t think it’s necessarily that we’re gonna survive because it wants us to survive. There’s a potential situation where we do stay in control the entire time. I still think that could be a plausible outcome. I think that if the AI is misaligned, that’s probably the only way we survive.

Liron 01:52:07
Right. Well, even in your scenario where the AI’s just here doing what we say and treating us as the boss, we’re gonna be asking the AI to do stuff like, “Hey, I want a theme park on the moon. I want Disneyland on the moon.” We’re gonna be asking it to do stuff, and it’s gonna have to be like, “Okay, well, I know that you guys don’t wanna die, so I’m gonna make sure not to kill you while I’m doing all this construction.” You see what I’m saying? So in that sense, it has to really know that it doesn’t wanna kill humans.

Eli 01:52:30
It’s reasonably good at following directions now, and if it’s super intelligent, how would it just forget about its one explicit instruction that’s, “Don’t kill humans,” to be able to do something?

Liron 01:52:43
Yeah. When I use the word “want,” I just mean it’s represented. There’s some part of its memory, which is the preference storage area — a very important storage area because all its actions are kind of downstream of that. And so my claim to you is just that it’s going to know that one of the things it’s supposed to treat as a preference is not killing humans. And if it doesn’t treat that as a preference, if it just has other preferences like, “Oh yeah, maximize number of Disneylands,” regardless of whether humans are alive to enjoy them, then we’re not going to manage to live in the margins with an AI like that. You see what I’m saying?

Eli 01:53:15
Yeah, I agree.

Liron 01:53:17
Okay, so that’s very important. That’s what I mean — it’s not like we’re some ants, but it’s okay because it won’t step on us. No, all the ants are going to get stepped on. You see what I’m saying?

Eli 01:53:29
Wait, go back. What’s your analogy?

Liron 01:53:32
There’s no outcome where you have a super intelligent AI in the universe, but we’re just here doing our own thing and it’s doing its own thing, like it doesn’t care about us. A super intelligent AI doing its thing with the universe, if it doesn’t care about us, we’re gone. It has to care about us enough to intentionally avoid killing us.

Eli 01:53:53
I don’t know if I’m picking up your full analogy, but I would agree that it has to care enough not to kill us is probably correct.

Liron 01:54:03
In my house right now, there’s a few ants and insects crawling around here and there that haven’t come out, that I haven’t seen, because I’ll tend to step on an insect if I see it. And so those insects — maybe there’s termites under the floorboards — they’re making a good living, not because I’m okay with them being there, but just because they’re doing their own thing and I haven’t seen them yet, and I’m not spending my days conquering every atom under my house.

I’m saying that’s not going to be possible with AI. There’s not gonna be, “Oh yeah, the AI is doing its own thing on Mars. We’re here on Earth. Sure, if the AI came over and wanted to do something, then it would crush us, but it’s not. It’s just doing its own thing and it just doesn’t care.” I’m saying that’s an impossible scenario.

Culture, Laws, and the Singleton Scenario

Eli 01:54:40
What do you think about Robin Hanson’s idea of AIs being the natural descendants of humans, and that it would respect us because we’re its godfathers or whatever? We create—

Liron 01:54:51
Yeah, that’s actually related to one of the next stops I was gonna ask you about. There’s also another stop being like, look, we have institutions, we have laws. The AI is going to be raised into our institutions, and it just needs to follow our laws. We don’t have to align it. We don’t have to make it care about us. We just have to raise it up in an institution with laws.

Or I’ve heard somebody else say, it just needs to respect our money system. It just needs to want money from us and not steal, and so we’ll just pay it and it’ll do what we want. Yeah, and like you said, Robin Hanson saying it’ll just be part of our culture. And to that I just say, I don’t think that as a dynamic — being part of our culture — I don’t think that is as strong of a principle as it has the ability to steer outcomes.

Eli 01:55:31
Um—

Liron 01:55:32
How does being part of our culture trump “you can type an outcome into it and get that outcome”?

Eli 01:55:38
Well, theoretically, if you had a human who was, to a degree, super intelligent in a hypothetical scenario, they would probably be likely to respect their culture and not kill people. So if the AI is aligned to the society like a human in that society might be, then it could be fine.

Liron 01:56:00
So in this example, the human’s preferences are to be part of my culture, which then implies not killing everybody. It just seems like this situation reduces to, can we get the AI to have preferences that we’re cool with?

Eli 01:56:12
Yeah.

Liron 01:56:13
So a lot of times people are like, “Look, I have a solution. We’re going to use culture. I have a solution to the problem that it’s difficult to program the AI to respect our preferences.” But then it turns out that the hard part of culture is making the AI respect our preferences.

If you search for Doom Debates, Gilmark, my friend Gilmark came on the show, and he’s like, “You know what’s gonna save us? Having lots of AIs competing. Even if none of them is aligned to us, when you make a lot of them compete, the result of the competition is going to be an outcome that we can live with.” But then when I drill down, from my perspective — he’ll probably disagree with this summary, but you can watch the episode for yourself — from my perspective, there’s just a secret ingredient where one of the AIs in the group ultimately does have sympathy for us and is championing us. And it’s like, okay, well, that’s the hard problem in the first place. You’re not solving anything by putting them in the group.

Eli 01:56:56
Yeah.

Liron 01:56:58
There’s also a bonus part of the doom train where even if you get off somewhere and you think you’re gonna get an aligned ASI, even then there’s actually another doom train within the aligned ASI world where you have to ask yourself, are we gonna get a good equilibrium of aligned superintelligences? What if you give a bunch of different people their own aligned superintelligence, and then they use it to go to war or they cause destruction? I think that’s another risk. Do you have any thoughts about that?

Eli 01:57:25
I feel like if the AI was aligned to our values in the first place, it wouldn’t help humans in a way that would be instrumental to destroying other humans in a way that would contradict its values. If it wouldn’t want to do that to us because it wants it, why would it do that to us if other humans want it?

Liron 01:57:49
But I think you’re making an assumption about whether different humans with different values will go to war. I feel like you could take your logic too far and be like, why are there ever big wars? Big wars are destructive.

Eli 01:58:03
I’m just saying from the point of view of the ASI. If the ASI is truly aligned to not kill humans, then why would it help some humans kill others?

Liron 01:58:15
I see what you’re saying. So there’s kind of these fundamental laws in your scenario where we’ve programmed all the different AIs and it’s like, “Hey, guys, you gotta play fair. No killing. You can use me for whatever you want to do, but you can’t use me to cause too much harm on others.” That’s your scenario?

Eli 01:58:29
Well, I thought you were saying in the case of aligned AI.

Liron 01:58:32
Oh, okay, I should clarify. I was imagining a scenario where the user of the AI or the creator of the AI — the AI is basically 100% loyal to them. Whatever your preferences are, I will try my best to steer the universe toward those preferences.

Eli 01:58:46
Yeah, I feel like the AIs are probably gonna be aligned to a specific set of values and not necessarily individual humans aligning them.

Liron 01:58:59
I see what you’re saying, but if you’ve got all the different AIs, if everybody’s AI already has the firmware or the low-level module saying, “Hey, these are the ground rules,” in that case I kind of see it as a singleton scenario. Oh, you’ve already got one universal AI that’s injected itself. It’s got its hand already everywhere.

So I agree, that is a nice, elegant outcome — a singleton outcome. I was just asking about what if there were actually a bunch of localized power regions that didn’t have allegiance to the other ones, which is analogous to countries on Earth right now. Nobody has allegiance to a single world government, so the countries are just there doing anything.

Eli 01:59:35
Yeah, I don’t really have a strong opinion about this. I don’t know. As I said before, I don’t have very strong opinions about whether or how AIs will be aligned, so you’d be a better arbitrator of this one than I would be.

Ricardo’s Law Won’t Save Us

Liron 01:59:53
All right, the last stop is unaligned ASI will spare us. This actually is a Mike Israetel stop where he was saying, “Yeah, the AI doesn’t have to particularly like us. It just has to want to learn from us, and it will because we’re so interesting. There’s never been anything as interesting as us.” What do you think about arguments in that flavor where, okay, yeah, we never really got the ASI to be aligned, but it’s gonna wanna trade with us, learn from us, so we’re good?

Eli 02:00:16
I feel like we covered this a little bit before — whether we’re gonna survive is based on whether it wants us to survive or not. And then for whether it would wanna trade with us—

Liron 02:00:27
Right. Yeah, you gotta go to that.

Eli 02:00:29
For whether it would wanna trade with us, probably not, because it could probably just get all the resources it wants pretty easily.

Liron 02:00:36
You’ve looked into market dynamics. You’ve looked into Ricardo’s law of comparative advantage that says a more powerful society often will want to trade with a less powerful society because their opportunity cost is high. So go ahead and have that poor island sew—

Eli 02:00:49
Oh, yeah.

Liron 02:00:49
...your clothes because your time is better spent doing something else. You’ve looked into Ricardo’s law of comparative advantage, but you don’t think that that’s going to spare us.

Eli 02:00:56
Well, so are you saying that it would be bad for the ASI to trade with us, or that it would want to trade with us because it’s a high opportunity cost? Are you saying because of that or because you wanna waste the time of the humans?

Liron 02:01:17
No, no. I’m specifically saying Ricardo’s law of comparative advantage. If the AIs only have their own self-interest in mind — let’s say it’s a cancer AI, all it wants to do is reproduce and conquer the universe for itself. It doesn’t care about humans. But when it sees planet Earth, it’s like, “You know what I should do with this planet? Trade with them. Sure, I’m better at everything than these humans, but I’ll just have these humans manufacture some stuff for me so I don’t have to worry about it, and I can just pay them a little bit, and it’s best for me to spare them because of Ricardo’s law of comparative advantage.” Some people will make that argument.

Eli 02:01:46
It seems like the AIs have many instances of themselves, and it’s gonna be pretty cheap to get all the resources they want. It doesn’t seem like humans have any sort of extra utility it couldn’t get for dirt cheap.

Liron 02:02:08
Yeah, the correct answer when people bring up Ricardo’s law is that Ricardo’s law just pre-assumes certain conditions. It pre-assumes that the island is in a state where you can take it or leave it — you either find a mutually agreeable trade or you do nothing. That’s the premise of the theorem. If there is a third option called conquer them — just gas everybody to death and then take their stuff and put a seed robot to go make a billion copies of it and use all the resources — if that is an option, there is in fact no law of economics saying that you should trade rather than do that.

Eli 02:02:43
Yeah, I mean, maybe. I just don’t think that’s probably gonna be what the AI arbitrates the final decision on, whether to kill the humans or not. “Do I wanna trade with them?” It seems like kind of a minor detail compared to all of the other reasons it might want to.

Liron 02:03:10
Yeah, you’re doing a lot better, in my opinion, than the average guest who comes on my show. You’re making it look easy the way you’re dismissing some of these claims, but I’ve definitely had multiple guests who will harp on some of these points. So these aren’t straw men.

Eli 02:03:24
This second half seems like it’s going less well than the first half, but I don’t know.

The Case for Racing China

Liron 02:03:32
So we have come to the end of people trying to argue why we’re not doomed. The next bonus part of the doom train is when people say, “Sure, P(Doom) is high, but let’s race to build it anyway because,” and there’s a couple arguments they make, like, “Coordinating to not build ASI is way too hard or even impossible. China will build it as fast as they can. It’s game theory, man. So no matter how low our chance of surviving is, the US should take the chance first.”

Eli 02:04:01
I think that if it was actually high enough for it to be a serious concern for the government, there would probably be... If both parties actually thought that there was a decent chance everyone would die, it seems like they would at least try to reach some bilateral agreement.

Liron 02:04:23
Okay, I agree. Another correct answer. Thanks for making my job easy. I don’t have to debate you.

And then I think the last meta stop is slowing down the AI race doesn’t help anything, because the chance of solving AI alignment won’t improve if we slow down. “I’m personally gonna die soon,” that’s what people say, “and I don’t care about future humans, so I’m open to any Hail Mary to prevent myself from dying.” These are just random things people say. They’re like, “Look, what’s the point of slowing down? Yes, P(Doom) is high, but what’s the point of slowing down?” Are you sympathetic to that argument?

Eli 02:04:53
I don’t know enough about how they do AI safety, so I’m not gonna... I don’t know about that one.

Liron 02:05:00
Fair enough. And more instances of this category where people say humanity is already going to rapidly destroy ourselves with nuclear war or climate change. Rocco Miec, of Roko’s Basilisk fame, came on the show and he said, “My P(Doom) is, I think, -20%, because by not building artificial superintelligence, we’re about to low-fertility-rates ourselves to extinction. And so it’s critical that we build AI right now, no matter what.” That was kinda his class of arguments. Are you sympathetic to that?

Eli 02:05:23
No. I think that other x-risks are relatively low.

Liron 02:05:28
All right, fair enough. They’re definitely lower, for sure. And then people just say, “Think of the good outcome. People will stop suffering and dying faster.” I think we’ve already covered that because you’re like—

Eli 02:05:37
Yeah.

Liron 02:05:37
“What about one day sooner, man?” I think we already covered that.

Eli 02:05:39
Yeah.

Wrap-Up & Ideological Turing Test

Liron 02:05:40
All right, man. Well, we’ve certainly gone quite comprehensively and far on the doom train. You’ve been such a great sport responding to so many different points. So heading toward the wrap-up here, is there anything else that comes to mind when you think of the AI doom argument and your stance on it?

Eli 02:05:56
What kinds of things have people said for this in the past?

Liron 02:05:59
I don’t ask everybody this question. It’s just, I took you through the whole doom train. We covered a lot. So by way of recap, I can try to summarize, do the ideological Turing test of what I think your current position is, and you can tell me if that’s accurate. And you can also summarize how you see the conversation or what you see my claims as being.

Eli 02:06:18
Okay, yeah. That sounds fine.

Liron 02:06:19
All right, nice. And this is a very important exercise. I don’t know if you’ve done this a lot. I think Bryan Caplan invented the concept. The idea that you should be able to make somebody else’s argument.

Eli 02:06:27
Bryan Caplan invented that?

Liron 02:06:29
I should be clear. I know there’s people — I think maybe even Benjamin Franklin, or there’s definitely been people throughout history saying, “I can argue for you better than you can argue for yourself.” Bryan Caplan recently popularized it as the idea of the ideological Turing test, where it’s actually a test. You have to actually test yourself on whether you can argue as well as the other person if you were to switch places.

Eli 02:06:50
Yeah, okay. So you wanna start doing your summary?

Liron 02:06:54
Yeah, sure. All right, I’m taking the ideological Turing test. I am Eli Goldfine right now.

Boy, P(Doom) — first of all, let’s all acknowledge that it’s hard to give an exact probability. Can you hand wave and give a giant probability range? I guess. Can that be useful to policy? I guess I’ve been convinced in this conversation that it’s useful to policy, rough as it is.

All right, but then are we actually doomed? Are we doomed more likely than not? Well, there’s a lot of pretty convincing places on the doom train that I don’t feel compelled to get off on. The whole doom train seems like a pretty good train. There’s a lot of stops that I really whooshed right past — orthogonality thesis, I had no desire to get off on that stop. There’s other stops that, because I don’t know much about them, I’m less confident, like the stop of alignment by default or the AI companies are going to solve alignment. Maybe I’m tempted to have one foot out that stop or at least acknowledge that it’s a possible stop for me.

And when you just look at my uncertainty over a few different stops, maybe I have some hope. Maybe I even have double digits worth of hope. Maybe I even have a majority of hope that everything is actually fine, especially when you combine that with natural reference classes. Things tend to go fine in general. So if the trend of things going fine continues, that seems like a compelling reference class. So I wouldn’t call myself a doomer, but I gotta pay some respect to the doomers who are pointing out the doom train and actually making the arguments, because the arguments, on average, are pretty solid arguments, and I hope they keep arguing. And man, I sure love Doom Debates.

How was that?

Eli 02:08:17
Yeah, that’s great. Okay, so now I’ll try to do—

Liron 02:08:21
Wow, thanks. High praise.

Eli 02:08:22
I’ll try to do your argument. So essentially, ASI is gonna be exceptionally powerful, and there’s been a couple reference classes, or two specific examples of higher intelligence wiping out a lower intelligence group. And I think that it seems like if the AIs aren’t proactively choosing to not kill us, then by default the AIs are gonna be misaligned, especially if we’re not doing anything to prevent them from being misaligned.

And so if they are just optimizing for what they want the most, then by default humans aren’t really a part of their worldview in that respect. Therefore, we don’t need them, so let’s just get rid of them. Is that a good synthesis?

Liron 02:09:16
Yeah. The only tweak I’d make is they’re certainly gonna be aware of humans. They’re gonna know that they came from humans, and they’re going to have a very crisp idea of what humans want and what humans are about to do. It’s just that they’re not steering toward letting humans fulfill what they wanna do.

Eli 02:09:32
Yeah.

Liron 02:09:33
But great job. I love this, man. Whenever I do this segment, to be quite honest, it’s more like I’m doing it. I’m taking on the challenge for the guest, and the guest is usually pretty happy to say I’ve done a pretty good job or they’ll make a few corrections. But you’re actually pretty top tier in terms of actually trying to do the exercise with my position, so I appreciate it.

Eli 02:09:53
Okay, yeah. This is a lot of fun. Longest podcast I’ve ever done by a mile, but it is cool.

Liron 02:10:02
Hell yeah. All right, great. We can leave it here. So we’ve covered your different passions and also how young you are, gaining consciousness only five or six years ago. That’s cool. You’re making good progress being a really influential intellectual. And I do think your podcast, supercycle.blog, that everybody should check out, I do think it’s going places because it’s off to such a good start. And man, imagine how great you’ll be at this when you’re 15. So yeah, looking forward to following your progress, hope we can collaborate more in the future. Eli Goldfine, thanks for coming on Doom Debates.

Eli 02:10:33
Thanks. This is great.

Outro from Conductor Ori

Liron 02:10:36
Thank—

Ori Nagel 02:10:36
Thank you for riding today’s Doom Train, and thanks to special guest Eli Goldfine for being a model of good discourse conduct. We have now reached our final destination. Before you disembark, this is Conductor Ori reminding you to please take all your belongings and the lessons you learned today, namely that there is a route to a good future and it does not go through e/acc.

If you wanna move the Overton window on the tech discourse so that more Eli Goldfines of the world embrace AI pacing by default, remember, there is something you can do. Doom Debates is a viewer-funded program. Your support brings the biggest voices in AI and the public aboard this train so the world can scrutinize their positions and come to realize the merits of AI pause.

Without more support, this train faces imminent cancellation.So head over to doomdebates.com/donate and donate if you can.

We’re nearly one-third of the way towards our goal of funding production until the end of the year. On behalf of Liron and the wider Doom Train crew, we look forward to seeing you next time on Doom Debates.

---

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏