Liron explains the 80-year-old Erdős conjecture that a GPT model just cracked, Scott Aaronson ponders "the last days of human relevance”, multiple live callers debate me, plus my 7-year-old son Ezra drops in to share his P(Doom)!
Timestamps
00:00:00 — Cold Open
00:00:42 — First Donation & “Why Is a Duck?”
00:02:54 — Can AI Draw “Colorless Green Ideas Sleep Furiously”?
00:08:26 — Doom Debates Sponsors Less Online — Be Our Intern
00:10:43 — Who We Still Need to Get on the Show
00:26:44 — Claude Opus 4.8 & the Rising Waterline of Intelligence
00:28:48 — The 80-Year-Old Geometry Conjecture a GPT Model Cracked
00:44:50 — My 7-Year-Old Ezra Joins: ChatGPT, Minecraft & His P(Doom)
00:58:46 — How the AI Actually Beat Erdős’s Grid
01:03:50 — Scott Aaronson: “The Last Days of Human Relevance”
01:07:53 — Why the Foom Is Taking Years, Not Hours
01:10:47 — Is Ori Just a Yes Man?
01:12:41 — Does P = NP + AI?
01:23:43 — METR’s Beth Barnes: “We Are Not On Top Of It”
01:28:31 — Liron’s Vibe Coding Confession
01:31:21 — Alverin Joins: Can We Wait, Then Hit the Off Switch?
01:48:01 — AI in a Box & the Super-Persuasion Threshold
02:06:24 — Brian Joins: The Ways It Could Go Right
02:11:30 — Jack Joins: How Fast Will the Foom Be?
02:18:48 — 80,000 Hours, Inventing Erdős Problems & Holly Elmore’s Warning
02:29:42 — Wrap-Up
Links
LessOnline 2026 — June 5–7, Berkeley — https://less.online/
Lighthaven Summer Camp (June 8–11) — https://less.online/summer-camp
Manifest 2026 — June 12–14 — https://manifest.is/
Timothy B. Lee / Kai Williams — “OpenAI’s Milestone Math Breakthrough,” explained (Understanding AI) —
Liron’s follow-up: grilling Claude about the Unit Distance Conjecture —
Shtetl-Optimized — “Dispatches from the possibly last days of human relevance” —
https://scottaaronson.blog/?p=9782
Futurism — prankster posts real Monet, calls it AI — https://futurism.com/artificial-intelligence/real-monet-ai-chaos
Anthropic Alignment Science — “Teaching Claude Why” — https://alignment.anthropic.com/2026/teaching-claude-why/
AI 2027 — Daniel Kokotajlo & the AI Futures Project — https://ai-2027.com/
Global Call for AI Red Lines (Bengio, Hinton, Harari) — red-lines.ai
Order the 80,000 Hours book — Benjamin Todd (Penguin) — https://80000hours.org/book/
Transcript
Cold Open
Liron Shapira 00:00:08
May 29th, 2026, Doom Debates Live Nation.
How are you?
Producer Ori 00:00:15
What’s up? Dun, dun, dun, dun, dun, dun.
“What’s your P(Doom)?”
Ori 00:00:23
Hey, what’s your P(Doom)?
Liron 00:00:24
There’s a new sound effect for you.
Ori 00:00:25
Nice. New sound effects.
Liron 00:00:26
Yeah. Nice. Okay, I see we announced it in the Discord that we’re going live. Let’s do a tech check here, see if we’re live on YouTube.
Going live on YouTube. Never thought I would be a YouTube live star, but here we are.
First Donation & “Why Is a Duck?”
Ori 00:00:42
That was not on my list for what you were going to be also.
Liron 00:00:47
Yeah, exactly right. That was not on Ori’s list when we first met each other in seventh grade.
Ori 00:00:55
Yeah, we met seventh grade. Yeah.
Liron 00:00:59
Yeah.
Ori 00:00:59
But started hanging out more in eighth grade than ninth grade.
Liron 00:01:02
Exactly. We went to the same youth group.
Ori 00:01:05
Yep.
Liron 00:01:06
That was before AI Doom was a thing, but at that time, Eliezer Yudkowsky had already made himself aware of the upcoming singularity, because that was around 2000.
Ori 00:01:17
The wheels were in motion. There was bubbling going on that was making it happen.
Liron 00:01:23
Yeah, I don’t know why we were wasting time dissecting a frog when we could’ve been talking to Eliezer about the singularity.
All right. Enough banter.
Liron 00:01:32
Let’s debrief the week. I think really the big news coming up is Less Online. Where’s that soundboard, everybody?
Ori 00:01:42
Let’s go.
Liron 00:01:44
All right, and we actually got our first donation on the stream, so I don’t have this kind of train whistle sound effect in the soundboard, okay? So you might hear a train approaching, but you’re not going to hear— This is 100% organic train horn. It’s made out of wood.
So there’s only one reason why I would blow the train horn, and that is because a user, Gadzooks, has donated Canadian $13.99.
I don’t know if you type that in manually when they ask you how much you want to donate, and you’re like, “Oh, I’d like to donate $13.99 Canadian, thank you very much.” But we do appreciate it. Definitely keeps the lights on here.
Ori 00:02:24
Thank you, Gadzooks.
Liron 00:02:27
Exactly, yeah. I think some of the lights might be flickering, guys. Let’s see. What do we got here? Uh-oh. We need some help keeping the lights on.
Ori 00:02:34
Hey.
Liron 00:02:36
So yeah. It’s all because of your donations that we can keep these on.
Okay, so yes, this question is, “Why is a duck?” I might take a pass on that one. Yeah, Doom in the dark, exactly, or Doom Debates after dark.
Ori 00:02:49
Wow, he bought it. He donated. He bought a question just to troll you.
Can AI Draw “Colorless Green Ideas Sleep Furiously”?
Liron 00:02:54
Yeah. “Why is a duck?” Yeah, that’s a good question to ask AI. It reminds me of how I think there was in the old philosophical literature, people talk about these meaningless questions that you can’t possibly answer, or these sentences.
What I mean is, I think it was Noam Chomsky who said there’s these grammatical sentences that are valid syntactically, but they have literally no meaning whatsoever. And the famous example was, I think, “colorless green ideas sleep furiously.”
That sentence has no coherent meaning, and yet it’s grammatical. The adjectives are modifying the nouns. Everything makes sense.
And then I remember when the AI was first doing— What was that first AI that was drawing things? Midjourney, right? They had Midjourney draw “colorless green ideas sleep furiously,” and it actually drew a pretty good picture of a colorless green idea sleeping furiously.
I don’t know if you want to pull that up. We’ll start the screen-sharing journey soon. Here, let’s get it going now.
Ori 00:03:53
Leave it to AI to solve all our philosophical conundrums, right? And it’s like you said with Eliezer Yudkowsky, the purpose of a driving animation for philosophy can be, how do we want to train AI?
Liron 00:04:11
Yeah. All right, here, I’m going to screen share the colorless green ideas tab.
All right, so I’m just having GPT draw it over again, because I couldn’t find it on Google Images. So we’ll see.
So yeah, we’re going to be taking questions from the live audience. We’re going to be talking about the Less Online conference. Oh, man, what a cop-out. Look at this. It just wrote the text. It made an image of text, and the text says “colorless green ideas.” I think we’ve gotten to the point—
Ori 00:04:37
You got trolled.
Liron 00:04:37
Right, I got trolled. The AI is so wise to us. It probably knows the irony of asking an AI to do that, and it’s just like, “Okay, I’ll just draw text.”
It’s meta-aware of the fact that I’m asking it that, right? But the original Midjourney was more naïve, and it just drew it. So I’ll leave that to you guys to do your own image.
Or I’ll just ask it another—
Ori 00:04:58
Well, no. You could do it again without the quotes, and then it would work.
Liron 00:05:02
Right. Oh, you know what? You’re absolutely right. I put it in quotes, so it literally just drew this on.
Ori 00:05:05
Yeah.
Liron 00:05:07
Right, yeah. So it knew my own meaning better than I did. Exactly, yeah. Okay.
Ori 00:05:13
It was aligned, as the OpenAI people would say. Baby aligned.
Liron 00:05:20
Yeah, exactly. Right. Here, “depict the content of this sentence.” See, I’m saying the content, not the words. The content, the meaning.
Here, I’m saying “depict the meaning of the sentence.” There. “Colorless green ideas sleep furiously,” right? And the whole premise is that this sentence has no valid meaning. In the space of meaning that the syntax maps to, there’s nothing. There’s a void.
And yet, I think we’re going to see a depiction, and I think that the trick to this is that in the space of ideas, it’s going to draw a point that is somehow adjacent to— It’s somehow combining the concepts and it makes sense to us. Okay, well, sleeping is depicted, green is depicted, colorlessness is depicted. So it’s all depicted in the same image, so it gets credit. That’s how it’s going to work.
Yeah, there we go. So that’s kind of what I remember. I think the picture I had had a little bit more of— there was maybe a bed and part of it was colorless and part of it was green, whatever. But you get the point. This kind of looks like a colorless green idea sleeping furiously.
You know what I should have done is I forgot I got to go to the API platform. When I go to the API platform, it lets me generate 10 images at once. So that’s how we got to roll here. We got to use the API.
Ori 00:06:31
And three’s not enough for you?
Liron 00:06:32
Three. Are you doing it in three?
Ori 00:06:34
I thought I saw “generate three options” when you were at a different layer.
Liron 00:06:42
I think I only get one. No, I think I’m only seeing one. Yeah. No, I need 10. You can do up to 10, and of course, you pay for each one, but I’m such a baller, I can afford to pay 5 cents times 10.
I roll deep.
Liron 00:06:54
Also, I just noticed I’m looking at the screen on my YouTube and it says YouTube Canadian. So as you guys know, I live in upstate New York, and recently something happened with my ISP where my IP address is being classified by databases of IP addresses like I’m in Canada, and Canada’s a couple hours drive away.
So I’m just randomly getting the experience of the Canadian internet right now. I went to buy some shoes and the checkout didn’t work because the whole time it thought I was trying to buy them to a Canada address. So I had to go start over again and find the US site.
So yeah. I’m experiencing Canadian life right now, and I got to tell you, it’s pretty annoying. I don’t know how you guys do it. Oh, you know what? Maybe that’s why I’m seeing that the donation is Canadian $13.99. Maybe it’s just $10 US.
Ori 00:07:40
Oh, yeah. What YouTube are you on?
Liron 00:07:43
Wait, Ori, can you see it? Can you see it in the stream, or do you have the YouTube open?
Ori 00:07:46
Let me see.
Liron 00:07:48
Yeah, go look at Gadzooks’ donation. Tell me if you see it in Canadian or US currency.
Ori 00:07:52
I also see it in Canadian $13.99.
Liron 00:07:56
Okay. Got it. Yeah, everybody sees a different internet. Okay, so anyway, where were we? Okay, so we got to talk about Less Online.
Liron 00:08:03
And there’s just one thing I want to test real quick. Now that I have the screen share up, tell me if the sound effect is echoing for you or crisp, because last week we had some technical difficulties. All right, so let’s see. What’s a sound effect that we need right now? Here’s people who are too chicken to debate me.
Ori 00:08:18
I can think of some very obvious names.
Liron 00:08:18
Are those some good? Okay, but you heard the sound right?
Ori 00:08:24
Yeah, no echo.
Doom Debates Sponsors Less Online — Be Our Intern
Liron 00:08:26
No echo. Okay, great. All right. I think we nailed it.
Nice. All right, so let’s talk about Less Online. Okay, so screen share. We’re going to head over to less.online. Let me share this tab.
All right, here we go. This is crazy. We’ve been anticipating this a very long time. We’ve been talking about this for months, and now we are T-minus one week, literally one week from now. One week and two hours is when the conference is officially starting. It’s in Berkeley, California, and I think we said this last week, we have an amazing deal for you guys watching the stream right now.
Or actually, do we still need this? I think we’re already good, right? The deal’s off the table, right? Of being our intern?
Ori 00:09:05
No, I think we’re still looking for people. I think we’re very close to nailing it down, but we could still get applications for it, for sure.
Liron 00:09:14
Got it. It felt like it was such a long time last week, but here we are. It’s already days away.
Ori 00:09:18
Yeah.
Liron 00:09:18
But yeah, okay, you should still ping us if you want to do this.
And actually for Manifest too. So I’ll share that tab too. Manifest.is. It’s kind of funny. If you guys don’t know about this, look at this. Manifest, June 12th. So they’ve got two conferences at this place in Berkeley called Lighthaven, the rationalist mecca, if you will, back to back next week and the week after next.
They’re both in Berkeley, California. Same venue, great venue. Kind of different crowds of people, but a lot of overlap. And there’s this thing called Summer Camp where you can stay between Lighthaven.
We’re going to be at both, and we’re looking for interns for both. And I don’t know if we said this explicitly, but we’ve got the deal where we’re going to actually pay you $500 for either one or both, whichever one you help us out with, right, Ori?
Ori 00:10:00
Yes, exactly. Yeah.
Liron 00:10:00
And if you’ve ever considered buying a volunteer ticket where instead of paying $675, you only have to pay $350 and you have to volunteer for them, don’t volunteer for them, okay? Volunteer for us, because we’ll make your ticket $0, and we’ll pay you $500, okay?
So that Pareto dominates what they’re offering.
Ori 00:10:23
It’s an amazing offer. You get to hang out with us. You get to talk about your favorite show. It’s a great vibe. There’s so many reasons to want to go to this.
Liron 00:10:34
Exactly.
So yeah, so Doom Debates, we’re going to be rolling deep, because look at this. Scroll down on the home site for Less Online. Okay, some writings we love. A lot of these are really good blogs. I subscribe to a third of these myself.
But then you scroll down farther, this is where it gets good. Okay, cabinet of attendees. All right. Scott Alexander, Eliezer Yudkowsky, okay, yada, yada. The all-time greats, and also myself.
Who We Still Need to Get on the Show
Ori 00:11:00
Hey.
Liron 00:11:01
Check this out. Oh, nice, it links to Doom Debates.
Yeah, they gave me an image here. It says sponsor. That’s right. So Doom Debates is the— Apparently there’s no other sponsors. We are the official sponsor of Less Online. That’s right, everybody.
Ori 00:11:15
They may as well say Less Online presented by Doom Debates.
Liron 00:11:20
They really should, yeah. Honestly, I wish I was above the fold, where you don’t have any Doom Debates representation above the fold. We got to fix that.
So yeah, they really should call it Doom Debates presents Less Online. That should be the name of the conference.
Yeah, so Less Online, it’s a play on LessWrong.
Liron 00:11:38
And what does that mean that we’re sponsoring it? Well, when you go to Doom Debates, we are going to have a merch table where we’re going to be giving out shirts like this and encouraging people to subscribe to Doom Debates.
And we’ve got some cool merch. Some of it is Doom Debates themed, some of it is inspired by Eliezer Yudkowsky. We have the, spoiler, if you’ve been following the show’s Discord, I’ve got a funny image of my kids trying on the kaleidoscope glasses that we’re going to be handing out.
So yeah, you’re going to want to get those. They’re going to pay your cost of the conference just to get cool merch like that.
And we’re going to be doing talks, we’re going to be doing mini debates and mini episodes. So this is your chance. If you ever wanted to be on Doom Debates or have your friend be on Doom Debates, this is definitely the way to get that access right now.
And we’re going to be doing a Doom Debates talk. It might be other people debating. It might be me versus everybody. That’s a classic type of debate we do, survey the whole audience, what their position is.
Yeah. So Less Online, so we talked about be our intern. Do we have a link for it? Go to our Discord. If you go to our Discord in the general channel, we’ve been announcing how you can be our intern. That’s the right place for them to go, right, Ori?
Ori 00:12:51
Yeah. In the Discord. I’ll reply to that message that had the offer there, that we’re looking for help there. So I’ll reply to that so it goes back to the top of the thread.
Liron 00:13:06
Right, yeah. We’ll bump it up to the top.
Yeah, so it’s great fun. If you don’t have any plans during this weekend, and you can get yourself to Berkeley, because we’re not offering transportation, okay? That’s all you. Or lodging. If you can do transportation or lodging.
I don’t think that they have a tent encampment outside, so I think you’ve got to really actually crash on a friend’s couch or something.
As long as you handle that, you can get in for free. Actually, okay, I just thought of this. What if you take the $500 that we pay you and you just use it to pay for a bus ride and lodging? There you go.
Ori 00:13:38
Someone could for sure do that.
Liron 00:13:42
Exactly, right? So we just solved your housing and transportation problem, just like that.
Yeah, so we encourage you to be our intern for Less Online and/or Manifest. It’s going to be great fun. There’s lots of great talks, there’s lots of great networking. And just so you understand why the conference is good, it’s just a combination of the people. People like me and Ori. If you think me and Ori are a cool type of person, where we engage with ideas and we’re friendly, you’re going to find a lot of that on Less Online. That’s the type of person.
And also, the venue, I don’t know if it’s award-winning technically, but a lot of people are praising it because think about the best part of a conference where you’re in the hallway talking to people who aren’t currently listening to the speaker. So they turn the idea of a hallway into the entire venue. There’s a lot of courtyards, a lot of nooks. There’s a lot of, quote-unquote, hallway conversations.
And I can tell you personally, I’m not somebody who likes to listen to talks just because. I don’t get a lot of social value from being among people. It just doesn’t push my buttons the way that it would for a more social person. And also, the 1X speed is kind of a killer for me. I’ve just trained myself to listen at 2 to 3X speed. So the 1X is kind of painful.
So you’ll mostly just find me chilling outside of the talks, which is where a lot of people are chilling. So I recommend having a conference experience that mostly involves not listening to talks, except the Doom Debates talk. You’ve got to come to that one.
Ori 00:15:02
That one will be very fast-paced. Liron versus everyone.
Liron 00:15:06
Right, because I think that I can just talk at 2X, and that way everybody can win. Because I know you have to listen live, but I’ll meet you halfway. Or I’ll talk at 1.8X, and we’ll—
Ori 00:15:15
Oh my God.
Liron 00:15:17
All right. Yeah. Is there anything else we should let people know about Less Online? We said why the conference is good. We said why they should be the intern.
Maybe we should go over the people again, right? So you mentioned Eliezer. There’s Aella. I think she’s best known as a sex researcher.
She used to be an escort, and she was an innovator in the field of escorting. And she’s written a very interesting post on how to be an escort. She’s a combination of somebody who’s rational and introspective and also into research. Anyway, interesting person. We could deep dive on her.
Actually, I want to invite her to come on the show. I haven’t gotten around to that yet. Yeah, she’s going to be a good guest. Hopefully.
Ori 00:15:54
Yeah.
Liron 00:15:54
Let’s see. Katja Grace, AI Impacts is her organization I think she co-founded. They do great surveys. Yeah, let’s just talk about every single person. Okay, Gwern. Remember last year, this is crazy, but I think that Ori and I got to meet the real Gwern in the flesh.
Ori 00:16:13
Yeah. He was a very nice guy. Really smart and really sharp.
Liron 00:16:15
I don’t know what we’re allowed to say.
Ori 00:16:16
Yeah.
Liron 00:16:16
Should we reveal their gender? No, we can’t. It could be any gender.
Ori 00:16:21
Oh. Yes. Right.
Liron 00:16:23
They could be any gender. But look, the photo is just a letter G, right? No headshot or anything.
I don’t know if it’s overstepping to say this, but the individual that is supposedly Gwern in the flesh, I guess we didn’t get proof, but I will say the individual seemed like the kind of individual that you wouldn’t expect to be Gwern. Is that fair to say?
Ori 00:16:47
I disagree, actually. I think the individual who may or may not be Gwern is kind of exactly what I would expect.
Liron 00:16:56
Really?
Ori 00:16:58
Yeah. I don’t know. What can I say? I can’t say anything about it. But they were just—
Liron 00:17:05
Right, yeah. We can’t say anything. You can’t leak any information.
Ori 00:17:07
They were just incredibly well-read, really sharp, book smart, articulate. I don’t know, interesting to talk to.
Liron 00:17:14
Yeah.
Ori 00:17:15
Yeah.
Liron 00:17:16
This list is also making me realize that we haven’t been doing the best job getting guests for Doom Debates, because Scott Alexander, I’ve tried a couple of times. He’s only done one podcast ever, the anonymous appearance on Dark Ash.
Eliezer technically can never come on a show with Doom in the name, but I’ve done other content with him on my other channel. Sv, I actually have told him that I’d love to have him on the show. It’s just a question of when he’s available and what the theme of the episode will be. But yeah, big fan of Sv.
Talked about Aella, got to invite her. Emmett, we invited him to the party last year. He came, he was on for a few minutes. So check that box, I guess.
Ori 00:17:52
Yeah. That was great.
Liron 00:17:52
Daniel Kokotajlo, he’s expressed interest in coming on the show, right?
Ori 00:17:56
I think we’ve invited him, but he’s never been on the show. It’s too bad.
Liron 00:18:02
I know. AI 2027, what a superstar. His background is he used to work at OpenAI, and then he left, and he famously wrote on LessWrong that he doesn’t trust them to act responsibly at the time of the intelligence explosion. I think he got their number on that.
Ori 00:18:18
Yes. And it’s—
Liron 00:18:18
And then he went on to write AI 2027, yeah.
Ori 00:18:20
It’s amazing how prescient AI 2027 has been. People keep saying, “Oh, yeah.” People keep pointing out how the timeline that we’re in, how effective agents are, has just been mirroring AI 2027. So, yeah.
Liron 00:18:35
Yeah. I recently actually went back and read all of AI 2027 all over again. And it definitely holds up, and the timeline, agents working. That’s probably their number one prediction of agents will work, and I would say they definitely work at least. I definitely have a daily experience of a 10-minute agent really truly working.
Ori 00:18:57
Yeah. And that’s pretty unsettling if we’re on that timeline, right? They were like, “We’re a year away from the point of no return,” in their timeline.
Liron 00:19:11
Right, 2027. Exactly, right.
Liron 00:19:14
Okay, so let me just scroll back here what the viewers are saying. Let’s see.
The Inside View, that’s Michael Trazzi, friend of the show. Hey. He’s saying, “Are you guys just promoting Less Online?” That’s just the teaser, okay? This is a variety show, okay? These Friday shows, they defy a simple explanation like that.
Ori 00:19:33
We’re the sponsor of Less Online, Michael.
Liron 00:19:35
Highly recommend it. Michael, are you... Yeah, the sponsor. That’s right, look at this.
Okay. Just in case, for those of you who missed it, I know you guys are just joining the stream now. People pile into the stream. All right, look at this.
Take a good look. It’s me, and you know this photo is from Doom Debates, so this is like Inception. You’re on Doom Debates looking at a photo of me from Doom Debates.
Yeah. He’s asking, “You’re paying them?” Yeah, we’re paying them. I will say it’s a sweetheart deal because it’s not like we have a revenue model. Our revenue model is actually just asking you guys, the viewers, to donate to the show because you support the mission. So it’s not like we’re flush with cash. We want to be responsible stewards of your cash.
So, yeah, we negotiated a price that we thought was worth it. Because, look, we’re transparent with how we run the show. We do think it’s strategic to just make sure that we have mind share, even if it requires spending some of our budget.
We think it’s important that people on Less Online know, “Hey, we’re doing this project. We’re trying to have high-quality discourse. We’re trying to raise people’s P(Doom).”
We want everybody to know about this. And there’s this effect of creating common knowledge in a technical sense. Not only do we want you to know about Doom Debates, but I want you to know that this other person here knows, that this other person here knows, that Doom Debates is a thing. That has a lot of overlap with things that people in this community care about. It’s creating common knowledge.
Liron 00:20:54
All right. TheRugbyProp is saying, “You should take the raised money to build a misaligned AI, which then gives more credence to the doom hypothesis.” Wow, so many galaxy brain takes. That reminds me of that galaxy brain take that everybody loves to say of being like, “The real misaligned AI is just AI reading doomers saying how misaligned AI is bad.” Yeah—
Ori 00:21:12
Oh.
Liron 00:21:13
It wouldn’t have been misaligned if we hadn’t tipped it off to the idea that it could be misaligned. Sorry, guys. It’s really all our fault. So convenient.
Ori 00:21:19
I couldn’t even buy that argument. Do you think people really mean that when they say that? I thought it was such a cheap shot at the quote-unquote “doomers.”
Liron 00:21:28
Well, Anthropic is always messing with us. Because recently there was a paper from Anthropic and they were saying, “Hey, look, we found evidence that the only reason this particular AI is behaving misaligned is we traced it down. We traced it in the weights to these articles about misalignment.”
And a lot of people had a field day with that. A lot of people were like, “We knew it. See? The doomers are just creating their own problem.” It reminds me of that. I feel like this is as counterproductive. Was it FDR who said, “We have nothing to fear but fear itself”? Who said that?
Ori 00:21:57
Yeah, I think that was FDR. Or Kennedy.
Liron 00:22:00
Yeah. What was the context in which FDR said it? Was he basically just saying, “Don’t be a pacifist”?
Ori 00:22:07
Don’t know.
Liron 00:22:07
I know the day that will live in infamy. I got that one. He’s talking about Pearl Harbor. Okay, hold on. I’m looking at the context here.
I’m looking on Reddit here. “I’m certain that my fellow Americans expect that on my induction into the presidency, I will address them with a candor and a decision which the present situation of our nation impels. This is preeminently the time to speak the truth, the whole truth, frankly and boldly. Nor need we shrink from honestly facing conditions in our country today. This great nation will endure as it has endured, will revive and will prosper. So first of all, let me assert my firm belief that the only thing we have to fear is fear itself, nameless, unreasoning, unjustified terror which paralyzes needed efforts to convert retreat into advance in every dark hour of our national life.”
So this is 1933. Okay, so World War II was not even... This is his first inaugural address. So I guess he’s talking about the Depression.
And it’s saying, “The United States has suffered through more than three years of the most appalling economic catastrophe.” Okay, so he’s talking about the Great Depression.
I think that this quote has been counterproductive to people’s thinking. “The only thing we have to fear is fear itself.” Because it’s an example of what I call recoil exaggeration. Or actually, is it even recoil? No, I think it’s a different bias.
You know how people are biased to think that if something rhymes, then it’s more true? And your brain is already accepting it as true even before you’re critically thinking about it. It’s like, “Oh, it rhymes. I’ve already accepted it, sorry.”
Well, similarly, I think that people have a bias to accept things that are conveniently self-referential. “Oh, the only thing to fear is fear itself. Okay.” No, fear itself tends to be pretty low down on the list of things to fear, in actuality.
Ori 00:23:41
Okay. Fair point.
Liron 00:23:42
Yeah. And so similarly, when you’re thinking about why AI becomes misaligned, it’s so convenient to be like, “Oh, look, the only reason it was misaligned is because people talked about misalignment.” No, that may be a reason on the list of reasons in some cases, but I don’t think it cracks the top 100 reasons.
But yes, did it happen right now when the AI is dumb and it’s still a little bit of a parrot? A little bit of regurgitating its data. Yeah, okay. But that’s super misleading. And Anthropic is often really pissing me off to be focusing on things that they should know are red herrings.
Ori 00:24:15
Yeah.
Liron 00:24:19
All right, let’s see some of these audience comments here. Live audience.
Okay, Alvarin is saying, “How do we call into the show?” Okay, great question. We haven’t shared the link yet just because we want to do a little bit of news, but we will give you guys a call-in link. We’re going to do live call-in. Let me just see anything else the audience has been saying.
Somebody’s saying— No. Okay, no recent comments. Oh, “I imagine Gwern is a Scott Alexander type, but I don’t know.” Fascinating person, I’d love to meet them. Yeah, okay, we can’t. Our mouths are zipped here.
No comments. Okay, yeah, so we are going through the list of people on Less Online, and we are saying how these are almost to a T people who are welcome on Doom Debates, maybe even 100%. And pretty much everybody I’m seeing on my screen right now, with a couple exceptions.
Okay, Max Harms, shout out to Max Harms. He had a great appearance on the show. Go search Doom Debates, Max Harms.
Wayne Evans, I feel like I got a good shot at getting him to come on. We got to hit him up.
Ori 00:25:16
Nice. That would be awesome.
Liron 00:25:16
Oh, okay, Rob Miles. Bam. That’s nice that we were able to check a couple boxes here. Rob Miles has been on the show.
Ori 00:25:25
He’s been on the show twice.
Liron 00:25:26
That’s right, yeah. Been on the show twice, that’s right. Search Doom Debates, Rob Miles. All right, everybody.
And I recently tweeted an invite to Kelsey Piper. I don’t think she replied to that one. But the reason is because she recently had a good tweet on why she personally is concerned that we do kind of seem to be getting close to doing something insane the way we’re going about AI, and she’s just hopeful that we won’t for some reason.
And yeah, I just offered her if she wants to turn that tweet into an episode, I think that’d be great. But yeah, we’ll keep trying.
Let’s see. Okay, New Place To Frown is saying, “Matuszek, Andy Matuszek is an interesting guy. He taught me a lot about note-taking. Very cool website.”
Okay, Andy Matuszek, if you’re on the stream right now, you might be one of the 23 people watching, or you might be one of the couple thousand people who are watching asynchronously. You are hereby invited, and if we see you in person, we’ll invite you in person.
Somebody’s saying Paul Christiano. And yeah, Paul Christiano, you are hereby invited. Yeah, that’d be a good get.
All right. All right, guys, so that’s Less Online. Let’s move on. Oh, check it out, pics of Lighthaven. Don’t you guys want to go in this amazing, warm, welcoming space? Look at this space. Great space. Lots of nooks, good weather, California weather. All right.
And don’t forget about Manifest as well. It’s another good conference.
Claude Opus 4.8 & the Rising Waterline of Intelligence
Liron 00:26:45
All right. So after Manifest, I think we wanted to do news of the week, right?
Ori 00:26:53
Yeah, news of the week. What’s the big news this week?
Liron 00:26:55
There’s Claude 4.8, Claude Opus 4.8.
Ori 00:26:59
Yeah.
Liron 00:26:59
I’m always talking on the show about how I spend a lot of money on Fast Mode, so I want to let you guys know that Claude Opus 4.8 promises to only charge one third as much for Fast Mode. So it’s pretty funny. 4.6 charged about $200 an hour, 4.7 same price, roughly $200 an hour. Suddenly, 4.8 supposedly is going to be only about $66 an hour, so that’s good.
Only caveat is I started using it, and it still feels slower. I’ve actually been stuck on 4.6 because 4.7 and 4.8 Fast Mode, they felt slower to me. I don’t know why.
But 4.8, I’ve been using a little bit, and it does seem more thoughtful. It does seem a little bit smarter as far as I can tell so far. So I think I’m going to use 4.8, but it’s just weird that it feels slower. So now you guys know what my current experience is.
But I said this on Warning Shots earlier today, which is I think the end game of all of this is that we’re all just going to have cheap, fast programmers, and there’s not going to be that excuse of, “Oh, I can still program because it’s too expensive to use AI.” I think we’re rapidly converging to a place where superhuman AIs are going to be affordable.
Ori 00:28:05
Yeah. I saw a take. Someone gave a take where it was the only two companies that are going to continue to exist are OpenAI and Anthropic because they’re able to just eat into what so many other application companies are doing.
Liron 00:28:17
Yeah. I don’t know if they were being ironic or not, but I would say that it’s plausible what they’re saying.
Ori 00:28:25
Yeah. They were exaggerating clearly, but you look at how quickly they’re deploying and all the functionality that they’re able to deploy also, it’s pretty insane.
Liron 00:28:39
Uh-huh. Okay.
Liron 00:28:42
I’ve got a really interesting follow-up from last week. Do you remember on the stream last week, we covered the OpenAI math breakthrough, and then I tried to scroll through and give you guys some insight about it, and I really had nothing whatsoever?
The 80-Year-Old Geometry Conjecture a GPT Model Cracked
Ori 00:28:55
Yeah.
Liron 00:28:55
So this week, somebody published— Timothy B. Lee, right? Let me pull this up. His publication, Timothy B. Lee, he says... Let me pull it up on a screen share.
But basically, he published something about it, and I actually read it because I’m thinking, “Hey, this is my chance to understand it and tell you guys on the stream. I can make up for last week.”
All right, so he writes, “Other news outlets hand-waved past the substance of OpenAI’s big math breakthrough last week, but I realized I had a math major on my team, and I could get him to actually explain the result.” Okay, great. Let me just make my browser window a little bit bigger here. Bam. All right.
Liron 00:29:24
So he has a screenshot, and so I actually dug into this a little bit so I could explain to you guys. So let me pull this up.
All right, so we’re going to learn a little bit of the math. We’re finally going to understand what was going on with OpenAI’s math. And the funny thing is this is coming a little bit late because since the time that last week’s news was announced of OpenAI disproving a longstanding Paul Erdős conjecture — we said it was standing for 80 years or something — and they disproved it last week.
And then since that time, I think Google DeepMind came out and said, “Hey, look, we proved and disproved nine of these.” They literally one-upped, right? If I remember correctly, they’re saying, “Yeah, here’s nine more.” And everybody was over it already. “Yeah, yeah. We already know. That’s not news. Only OpenAI doing the first one is news.”
Ori 00:30:33
Well, the AI models have been solving these Erdős problems for a little time now. But this was a pretty significant one, a bigger prize, I suppose.
Liron 00:30:46
Right. Okay. Min Wu Kim is saying: “Didn’t Liron take a lot of math during college?” Yeah, I almost double-majored in math. Ended up just minoring in math. So, yeah, I know undergraduate math, and I do happen to know a little bit of extra theoretical computer science-type math, and a little bit of logic meta-mathematics type math.
But I’m pretty uneducated when it comes to number theory, and also, I haven’t done anything except watch 3Blue1Brown for the last 20 years. So you’re not exactly talking to a math pro.
But this is kind of a one-eyed man leading the blind situation. I think that I have a little bit that I can offer to the average viewer of the stream, so let’s give it a shot.
Liron 00:31:28
Okay, so understanding AI. So I’m just going to scroll through this post. I think the diagram is where it’s at.
Okay, so the unit distance problem. First, we start with some points connected by lines. A, B, C, D, E, we’ve got these five points, and you can see in this diagram he’s got A and D connected with a line, which is exactly length one. So it’s one unit.
And C to E, that’s one, B to E is one. And you can see here, A and B, they’re connected by more than one unit. And he’s saying, okay, well, as a starting problem, you can imagine just squishing A and B together, and then you could score another— You can count another one-unit line. And the name of the game is how do you just have the most possible one-unit lines?
Ori, are you following?
Ori 00:32:07
Yep.
Liron 00:32:08
Right. So it’s the unit distance problem. Given a set of five points, how do you arrange them so that you get the most possible one-unit lines?
And this is very interesting. There’s a table of the best solutions for five, six, seven, eight, and nine points. So look at this. When you have five points, you can’t do better than making them into this pentagon thingy. For those of you who are listening audio-only, just take three triangles and smash them up into a pentagon shape.
And is this— No, sorry, it’s not even a pentagon, it’s a trapezoid. Sorry, misspoke. So yeah, you smash three triangles into a trapezoid, and you can see here’s a unit, here’s a unit. So if you count up the number of one-unit lines, it’s just all the triangle sides basically, and you get seven. Seven one-unit lines.
Okay, and then it starts getting a little bit more complicated when you have nine. It’s interesting to me because all of these configurations are equivalent in terms of how many unit lines there are.
This triangle looks the cleanest to me. Everything’s perfectly stacked and symmetrical. And then this one is like a Tetris type of shape, a parallelogram. And then over here, this is a weird shape. It doesn’t really have a simple description. It actually kind of looks like a triangular prism, but in two dimensions.
It’s weird. There’s no simple description, and yet it also has nine unit line segments. So this is where it starts getting counterintuitive, because it’s when things stop being clean.
Ori 00:33:27
It seems like the one on the left there, the one that is almost like a 3D prism that you described, that’s kind of where the solution goes, right? The ultimate solution to this problem.
Liron 00:33:40
The... Oh, not really. Not really.
Ori 00:33:43
Oh, okay.
Liron 00:33:43
Are you just thinking about the mention of projecting it into higher dimensional space?
Ori 00:33:46
Yes.
Liron 00:33:47
Is that why you’re saying that? So that’s a good thought because they do mention higher dimensional space. I actually asked Claude about this, but what you’re thinking of — “Oh, it kind of looks like a 3D triangular prism projected down into 2D” — that’s actually a coincidence. That’s not what they’re talking about in terms of projecting it into high dimensions.
Ori 00:34:04
Got it. Okay.
Liron 00:34:04
Yeah. It’s something else entirely. Yeah. I’ll tell you more. We’ll get to that. But it’s a red herring, basically.
Okay. And then look at this. Back in 12 dimensions— no, back in the diagram with 12. It’s just this perfect symmetrical shape. There’s only one option. This is the only thing you can do, which is basically you take six triangles and smash them together into a nice symmetrical hexagon. And if you do that, then you get 12 unit lines.
Okay? I encourage you guys to pull up the video if you’re watching on audio only. All right. And then finally, we got eight and nine. So this is when it really starts just getting super trippy.
I don’t know if I have any insights to add. I guess, would you say this looks like a prism projected into two dim— I guess this looks like a prism still. That’s interesting.
Ori 00:34:43
Yeah. The eight-point object there on the left looks like a prism again. Some kind of 3D object.
Liron 00:34:47
Yeah. I suspect this is the last time that this looks like a projection of higher dimensional geometry, though. I’m not aware of it still looking like that beyond this point.
Ori 00:34:56
Mm-hmm.
Liron 00:34:57
But then look at this. It’s equivalent in terms of number of units. It’s equivalent to taking the hexagon from the last one and sticking on another little triangle point on it. So it’s really just kind of ad hoc doing whatever works here. I don’t know if there’s much of a method to this besides guess and check.
Ori 00:35:16
Uh-huh. Okay. All right. I’m with you.
Liron 00:35:17
Yeah. All right. And then there’s a weird shape I can’t even describe. It’s just, okay, they got a bunch of points. They put them in a really weird arrangement, and sure enough, there’s 14 unit lines, just like in the nicer-looking one. They somehow also got 14 unit lines. All right.
Ori 00:35:29
Yeah.
Liron 00:35:29
And then finally, they get to the one with nine points, and I guess this is kind of symmetrical. I can’t really tell if it’s symmetrical or lopsided. I think it’s lopsided. Yeah. Anyway.
Ori 00:35:38
Yeah.
Liron 00:35:38
Basically, it starts getting past my own understanding, and yet they’re somehow pulling out a maximum number of unit lines. But this was something that I didn’t have any intuition for last week, so it’s nice to see it. Okay, there’s this game. You start to lose all intuition for the game, but it seems complicated.
Ori 00:35:55
Yes. Okay. I’m following.
Liron 00:35:56
Okay. And the other thing that’s interesting to look at is the pattern of number of unit segments. So here it goes from five points to six points to seven points to eight points to nine points. And the number of unit segments they’re able to achieve, the maximum number, it goes from seven to nine to 12 to 14 to 18.
So seven to nine, that’s a difference of two. Nine to 12, that’s a difference of three. Twelve to 14, it’s only a difference of two again. Fourteen to 18, suddenly it’s a difference of four. So the differences, as they step up, again, counterintuitive. There’s no simple pattern here.
Ori 00:36:28
Hmm. Okay.
Liron 00:36:28
Yeah. It can never go down. You can only go up. I wonder if you can prove that you always have to go up by at least one. I don’t know if that’s provable or not. Maybe you always have to go up by at least two. I think you always have to go up by at least two, because when you tack on an extra point, you can always do it where it’s just another triangle sticking out. Probably. I don’t know.
Okay. So you should be able to always go up by at least two. That’s my conjecture. Feel free to prove that. Prove or disprove my conjecture.
Liron 00:36:54
Okay, so now we get to the Erdős theorem. The Erdős theorem is a theorem about what is the lower bound and the upper bound for the optimal way of arranging N points so that you get the maximum number of unit distances.
And again, so far, looking at these examples, it’s roughly the relationship between five points gives you seven units, nine points gives you 18 units. It’s roughly 2N. So N is nine, the number of points, and then 2N is the number of unit distances. So it’s roughly 2N. It doesn’t seem to be scaling much more than linearly here.
All right, so here’s where the Erdős thing comes in. He’s saying, what if we just make a naive grid? Okay, we make a grid of points. So look at the grid on the left. It’s a three-by-three grid. And what if we just say, hey, there’s the point in the middle. It’s just one unit away from its four neighbors on the grid? This is the simplest solution imaginable.
And you can imagine if you make the grid bigger — if it was 10 by 10 or whatever — you could go to each of these points, and you could be saying, “Hey, this has four neighbors, this has four neighbors, this has four neighbors.” And all the points that are in the center regions of the grid, they just all have four neighbors. That would be the simplest solution.
And even the simplest solution is actually not bad because we’re already getting a number of unit lines that’s scaling with N. If there’s N points, you’re still getting roughly 2N, I think. What is this? This is nine points, but it’s only four unit lines. So, bad example, I guess. But I feel like if you had 100 points, you could probably get close to 200 of these. I don’t know. I haven’t done the math, but point is, it scales linearly with N. That’s all I know.
Ori 00:38:27
Okay.
Liron 00:38:28
All right, and then now we get to another interesting detail. So Erdős is saying, what if instead of making these dots a distance of one apart from each other, a whole unit, what if you made it fractional dots?
So in this grid, this is a five-by-five grid, but each of these dots is actually one-fifth apart from each other. So it’s actually a one-by-one unit grid. Or hold on, did I even get that right?
I think it might be the square root of five. I think each of these dots might be the... Because hold on, he configured it so that the radius of the circle is one. So if this radius is one, yeah, that must mean roughly two and a half dots. I think he’s using the square root of five.
So the idea is that you can have these diagonal... Look, we can prove it, right? Because one squared plus three squared, if these were length one, then this would be the square root of 10, but we don’t want it to be the square root of 10, we want it to be one.
So I guess this has to be one over the square root of 10, and this has to be two or... Wait, one and two. One squared is one, two squared is four. Oh, no, square root of five. Okay.
Yeah, I think this is one over the square root of five. This is two over the square root of five. And so when you do one over the square root of five squared, that’s one-fifth. Plus two over the square root of five squared, that’s four-fifths, so then you get a radius of one.
Okay, this is why some people say I don’t try to do math in public.
Ori 00:39:49
Do you know, this is something I could have seen you doing when we met in high school or middle school.
Liron 00:39:57
Yeah.
Ori 00:39:57
This is very— So you’re saying that the radius of the circle is half of 0.5?
Liron 00:40:08
So yeah. Well, no. The radius of the circle is one. Because remember, the whole game that we’re trying to play here is we’re trying to get a bunch of these unit distances, okay? And these—
Ori 00:40:13
Yeah.
Liron 00:40:14
Black things are points, and they’re points on the grid. And these hollow circles are also points on the grid. So you got all these points on the grid, and this central blue point on the grid is a unit distance away from all eight of these outer points.
So yeah, Erdős is saying, I’ve got this grid where I’ve strategically spaced these points where each of them is one divided by the square root of five, a distance away from its neighbor.
So then when I make a five-by-five grid of these points that are fractionally spaced apart, then the central point is connected to eight of its neighbors because you can make eight of these diagonal lines.
Yeah, I’m starting to think that explaining all this might be beyond the scope of today’s live stream, but I can at least give you a high level. It’s, he did better than unit distance one. Because the problem with unit distance one is you never get a diagonal. Because if you took this blue dot, you can see where my mouse is, right? So if you took this blue dot—
Ori 00:41:10
Yeah.
Liron 00:41:11
And you tried to connect this corner dot, now you got the square root of two, but you don’t want the square root of two. You want one. You want another unit distance.
And look, this cross that looks like a plus sign, you’re only getting four unit distances, but when you can go diagonally, instead of getting four straight up and down, straight left and right, now you’ve got eight diagonal ones. So basically diagonal is better than going straight up and down and straight left and right.
Ori 00:41:35
Yeah, he’s making a minimum viable shape for maximum one-unit distance connections between dots. The minimum viable shape, or the simplest shape, is a circle.
So let’s see, for four dots, a circle gives you two — what do you call them? — segments? Two segments. And when you have eight, then you get three segments. And when you have 10, it gives you five or something. So he’s starting to prove the theory clearly with a circle as the first example of the shape that you can create from the dots.
Liron 00:42:15
Yeah. Now, this is an interesting one because he managed to find a grid distance where you can get the plus going, the straight up and down plus, and you can get diagonals going. I think this is the optimal solution.
I’m not sure. No, sorry, it’s not optimal because there are solutions where you can even have more diagonals. You can have more than eight different diagonal segments. I think it’s actually unbounded how many of these diagonals you can have, as long as you have fine enough grids.
But the problem is that in order to make really fine grids, you need to have enough points, too. So the idea is the more points you have, the finer you can make your grids and the more options you have for your grids. And so the more options you have to find a way to have these circles that have lots of diagonals that happen to land in the right place on the circles.
Okay, yeah. So I think I’ve already kind of gone beyond my depth on explaining this, so I’m going to—
Ori 00:43:12
Okay. But, so he did the circle, so now what?
Liron 00:43:16
Right. Okay. So let me try to give a high-level sketch of my takeaway from all this.
So, the idea is, Erdős is saying, “Okay, yeah, we have this game of trying to maximize unit distances.” And he did actually make a pretty big assumption here, which is the whole idea that you can have a grid. Because think back to over here, you don’t even see a grid. When you see these weird constructions that I showed you before — nine points in a weird construction — there’s literally no grid here. You just place the points wherever you happen to move them around when you’re playing the sliding the points around game. They did not land on a grid.
So Erdős is saying, “Look, I bet I could just have them land on a grid, and even though I’m adding a constraint which I didn’t need to add, it’s going to make my life easier, and it’s not going to sacrifice my performance in terms of number of unit distances.” That was basically his conjecture.
Ori 00:44:14
Okay.
Liron 00:44:16
Yeah. And so far, he’s making a pretty compelling argument. He’s saying, “Because look, I’m not saying my performance is optimal, but I’m saying I can scale it pretty nicely, and I bet nobody can scale their performance that much better than I can scale it using this method.” That was basically his conjecture.
Liron 00:44:16
Yeah. So then it just gets kind of complicated where they teach you what the deal is. They do a little bit of math of, okay, how many diagonals are there? Well, it gets to the question of when you’re using the Pythagorean theorem and you’re getting these whole number solutions, how many whole number solutions are there?
If the radius is five, we saw that one squared plus two squared worked, but how many sums of squares can you get? And it turns out that the number of sums of squares that you can put together corresponds to the number of diagonals you’re going to be able to make that are unit distances.
Yeah, somebody’s saying, “New place to find,” and saying, “Pythagorean triples.” I think that’s the idea, how many different Pythagorean triples are there that add up to a certain number? And number theory tells you it has a formula where the number of triples grows— I don’t even know. There’s some ## Special Guest: Ezra
My 7-Year-Old Ezra Joins: ChatGPT, Minecraft & His P(Doom)
Liron 0:45:00
formula of how much it grows.
All right, one sec. I think there might be a special guest here. I think my son was mentioning that he wants to get on the stream. My almost seven-year-old son wants in here.
Ori 0:45:12
Oh, hell yeah.
Liron 0:45:15
Bud. You want to come on “Doom Debates,” Bud? All right. All right, everybody. This is our latest guest. All right, Ez, say hi to the live stream.
Ezra 0:45:31
Hi.
Liron 0:45:33
Okay, one sec. So I’ve got to set up the AV situation here. Great. I’m using my earphone. One sec. All right, Ez, wear the earphone. I’m going to stick this in your ear, okay? All right. Ori, talk to him.
Ori 0:45:46
Ezra, hey, can you hear me?
Ezra 0:45:49
Yeah.
Liron 0:45:50
Okay, nice. He can hear.
Ori 0:45:51
Hey, all right. I saw you were getting ready for this. You had some special goggles on.
Ezra 0:45:57
Yeah.
Ori 0:45:59
Yeah. I heard you like ChatGPT Harry Potter stories. Is that what we should do?
Ezra 0:46:06
Yeah.
Ori 0:46:06
Little Ron, that’s what people are saying.
Liron 0:46:10
Yeah, exactly. That’s right. Yeah, Ezra, tell him — what kind of Harry Potter stuff are we talking about these days?
Ezra 0:46:17
The interview.
Liron 0:46:19
The interview? Oh, yeah. He’s been watching the Harry Potter cast doing interviews. Yeah.
Ori 0:46:26
Oh, nice.
Liron 0:46:26
So we’ve been talking about Harry Potter.
Liron 0:46:29
Ezra, so as you know, this show is all about debating whether computers like ChatGPT, also known as Compy—
Ezra 0:46:36
Yeah.
Liron 0:46:36
—debating whether computers are going to take over the world or be bad for humanity. So what has been your experience using ChatGPT?
Ezra 0:46:45
Um.
Liron 0:46:46
Would you say it’s mostly positive?
Ezra 0:46:48
Yeah.
Liron 0:46:50
What was the last thing that ChatGPT helped you with last night?
Ezra 0:46:54
Well, the last thing was the interview.
Liron 0:46:57
Okay, we watched the interview of the Harry Potter characters, but before that, right? I think you were trying to do something in a certain game.
Ezra 0:47:03
Oh, Minecraft.
Liron 0:47:05
Okay. Yeah. So tell us, how did you use ChatGPT to achieve what you were trying to do in Minecraft?
Ezra 0:47:10
Make a portal.
Liron 0:47:12
Okay. Yeah. Tell us the story.
Ezra 0:47:14
So, a lot of days I was trying to make a type of portal, and then I asked him, and he taught me.
Liron 0:47:27
That’s right. Yeah. So you’re playing your Nintendo Switch, and you’re in Minecraft, and you’re trying to make a portal. I don’t even understand Minecraft, by the way. It’s all him.
Ori 0:47:35
Yeah.
Liron 0:47:35
You were trying to make a portal, and then I saw you take the iPad, right? You stood up the iPad next to the Nintendo Switch, so you were second screening. And were you typing or using voice mode?
Ezra 0:47:48
Both.
Liron 0:47:48
Okay, so you were typing and using voice mode, and you asked it what?
Ezra 0:47:53
How do you make the portal?
Liron 0:47:54
Okay, and then what did it say?
Ezra 0:47:56
It said you get obsidian, and you get the flint and steel, and it’s four blocks wide and four blocks tall.
Liron 0:48:08
Yeah, I was pretty surprised. It was literally using it to be like, “Okay, here you have to move these blocks together, and then you’ll get a portal.” And then you took the advice, and you made the portal, right? It worked perfectly.
Ezra 0:48:16
Yeah.
Liron 0:48:16
All right. So that was—
Ori 0:48:17
Wow.
Liron 0:48:17
—a positive experience.
Ori 0:48:19
That’s impressive. Already doing tech support. You’re already troubleshooting. That’s cool.
Liron 0:48:29
Exactly. So do you think that there could potentially be any downside to AIs like ChatGPT?
Ezra 0:48:36
Are you talking to me?
Liron 0:48:37
Yeah, I’m talking to you.
Ezra 0:48:39
No.
Liron 0:48:40
Okay. You heard it here, guys. There’s no downside. Well, let me ask you, out of all your experiences with ChatGPT, has it ever done anything bad, or what’s been the worst thing it’s ever done, if anything?
Ezra 0:48:54
Well, I never seen it done anything bad.
Liron 0:48:56
Yeah, you’ve never seen... No, and that is fair. And I do admit on the show that I do think AIs today are quite helpful, and it’s hard to even imagine how they’ll turn bad. But then when you think about this show, “Doom Debates,” right? Because you were talking about being on “Doom Debates” — do you want to basically debate that AI is a force for good?
No? Okay, so what is your position? You got to take a position.
Ezra 0:49:25
I think it’s going to be good.
Liron 0:49:27
Okay, fair enough.
Ori 0:49:29
I feel like that’s — is that Mommy’s position? Is that what Mommy thinks also?
Liron 0:49:34
Yeah. What does Mama think?
Ezra 0:49:36
She never told me, so I don’t know.
Liron 0:49:39
What does Mama think about Dada even doing this podcast?
Ezra 0:49:42
She doesn’t like it.
Liron 0:49:45
Okay.
Hey, buddy, do you know what P(Doom) means?
Ezra 0:49:50
It means... No.
Liron 0:49:53
It’s the probability of doom. Do you know what probability means?
Ezra 0:49:55
No.
Liron 0:49:56
Well, you know probability because we often do a bet, right? Do you know what it means to have a 10% chance of rain?
Ezra 0:50:02
It means that it’s probably going to rain for two minutes.
Liron 0:50:06
Yeah, something like that. So, a 10% P(Doom) means there’s a 10% chance that there’s going to be doom, really bad stuff happening. So your P(Doom) is low, right?
Yeah. Okay. Well, look at this, I got sound effects. You can pick if you want to do... You could do a whip, or hasta la vista, or a train.
Ezra 0:50:31
Whip.
Liron 0:50:31
Okay.
Ezra 0:50:32
Oh. Is there any more I can do?
Liron 0:50:36
Yeah. Here, you can use the mouse. You can pick one of these sound effects.
Ezra 0:50:41
Where is it?
Liron 0:50:41
Right. The mouse pointer is right here.
Ezra 0:50:45
I don’t know where the mouse is.
Liron 0:50:46
Here.
Ezra 0:50:47
Oh, there it is.
Ori 0:50:50
I think we should hear — Ezra should say his P(Doom) again, and then we need a sound effect for it.
Liron 0:50:58
Yeah, okay. One sec. Here, let me help you, okay?
Ori 0:51:05
Do the awe.
Liron 0:51:07
Yeah, we need an awe sound effect. Here, I think you’re going to like this one here. Just use this one to click. I think you’re going to like this one. You have to double click. Here, I’ll do it for you. Oh, no, I can’t. Here, just double click. You got to use this finger here. Double click.
Ezra 0:51:23
Ugh, a fart?
Liron 0:51:24
Yeah.
Ezra 0:51:27
What about...
Liron 0:51:29
Oh, that’s a good one.
Ezra 0:51:31
Oh, I like that.
Ori 0:51:34
Looks like Ezra won this round. Ezra, we heard that you keep beating your dad at bets. You bet it was going to rain, and then you got a lot of money from him.
Ezra 0:51:43
Whoa.
Liron 0:51:45
All right, don’t play with the gun. That’s not safe.
Ezra 0:51:48
I’m not. That’s the gun?
Liron 0:51:51
Hey, bud. Hey, tell Ori about the bet that you won.
Ezra 0:51:55
Oh yeah, that’s the gun. But did you already tell him?
Liron 0:51:59
I think he knows. We talked about it last week.
Ori 0:52:01
We want to hear from your side. What happened? Because your dad is only taking bets. He thinks he’s going to beat you.
Ezra 0:52:09
So one night we were reading Harry Potter, and I said, “I want to do a bet.” So he said, “How about we do is it going to rain tomorrow?” And then I just did it.
And then he said he knew it wasn’t going to rain, and so then he said I should win big because I probably won’t. So, it was 12 against one. So then, it sprinkled a little bit, and I won.
Ori 0:52:44
You got $12 from that?
Liron 0:52:47
You made $12?
Ezra 0:52:48
Yeah.
Ori 0:52:49
Yeah.
Ezra 0:52:51
Oh, man.
Liron 0:52:54
That’s right. Yeah. And actually, you know what? There’s another. So, that is true. You were riding high on that particular bet, but there is actually another bet that was made. Do you remember that? Do you remember one of the other bets that we made?
Ori 0:53:07
Yeah. What happened with you and Mommy on that bet?
Ezra 0:53:10
So, wait. Which one, the blink one?
Liron 0:53:14
Yeah, the blink. Tell him about the blink one.
Ezra 0:53:16
Oh. So, okay, I will do it. So, it was the first bet.
Liron 0:53:27
Yeah, you were still learning. I don’t blame you.
Ezra 0:53:28
No, it was the second bet, so I didn’t know.
Liron 0:53:32
Okay. You’re new to betting.
Ezra 0:53:34
Yeah.
Liron 0:53:41
Okay. Are you okay if I play the video?
Ezra 0:53:42
Yeah.
Liron 0:53:42
Okay. All right. But you got to explain what happened, and then I’ll play it, okay?
Ezra 0:53:44
Okay. So, I said, “If I don’t blink, I’ll give you $5, and you will give me $1,” which was not a good bet. If I don’t blink for one minute. So—
Ori 0:53:59
Should we try the bet again then?
Ezra 0:53:59
So that’s what... And then I did it.
Liron 0:54:03
Okay, let’s roll the video of what happened last week.
Ezra 0:54:07
No, that was two weeks ago.
Liron 0:54:08
Two weeks ago.
Ezra 0:54:10
That’s funny.
Ori 0:54:11
We can’t hear the audio. I don’t know if there’s audio coming from—
Ezra 0:54:14
Macy did that too, and she actually made it.
Liron 0:54:16
Oh, you can’t hear the audio? All right, give me a second. Let me try again.
Ori 0:54:17
Yeah, just audio.
Liron 0:54:17
Yeah, the audio is critical. All right, one sec.
Ezra 0:54:19
You want him to hear me crying.
Liron 0:54:22
No, come on, you didn’t cry. You just were a little—
Ezra 0:54:25
I was like, “Oh.”
Liron 0:54:27
It’s actually funny because I think at first you kind of laugh. You were able to laugh at yourself, but then you also...
Ezra 0:54:35
I was like, “Oh.”
Liron 0:54:36
Okay.
Ori 0:54:38
Okay. Now we hear it.
Liron 0:54:38
Versus your five. We shook on it.
Ezra 0:54:41
I can hear it.
Liron 0:54:42
Yeah, because you’re trying not to blink.
Ezra 0:54:44
It’s funny.
Yeah, I laughed.
Liron 0:54:48
All right, so you’re trying to make it one... Oh, you just blinked.
Ori 0:54:50
Okay, blinked.
Liron 0:54:52
All right, buddy. $5 for me.
Ezra 0:54:55
No.
Liron 0:54:55
Okay, buddy, you owe me $5. Your facial expression changes. It kind of changes from smiling to sad.
Ezra 0:55:04
I’m not sad, it’s funny.
Liron 0:55:06
Okay, instant replay here. See? Look at that.
Ezra 0:55:09
Yeah.
Liron 0:55:10
Okay, but then what happened after?
Ori 0:55:12
Can we get an instant replay where you say, “Okay, now you owe me $5”?
Ezra 0:55:17
I said, “Blink.”
Liron 0:55:17
Yeah, here.
All right, buddy, $5 for me.
Ezra 0:55:22
No.
Liron 0:55:24
Okay, buddy, you owe me $5. Sorry, bud. But then what happened after?
Ezra 0:55:34
Oh my God. That day is like a different bet?
Liron 0:55:37
Yeah, what happened later? I said you didn’t have to pay me, right?
Ezra 0:55:42
Yeah.
Liron 0:55:42
Yeah. Okay, you happy now?
Ezra 0:55:43
Yeah.
Liron 0:55:43
And you learned your lesson, right?
Ezra 0:55:45
Yeah.
Liron 0:55:45
Yeah. What lesson did you learn?
Ezra 0:55:48
That, well—
Liron 0:55:50
That you should try again right now.
Ezra 0:55:55
No.
Ori 0:55:57
The bet. Isn’t the lesson that if you make a bet, you can win real money, and you can lose real money?
Liron 0:56:04
Yeah, and don’t be overconfident. Okay, well, I think we’re—
Ezra 0:56:07
Hey. Well, I didn’t know, so I didn’t know to pay more money for a bad bet.
Liron 0:56:15
Yeah. That’s right. You shouldn’t bet at one-to-one odds, right? It should’ve been my five to your two, not my five to your five.
Ezra 0:56:21
Yeah.
Liron 0:56:22
Yeah. You deserve to win big if you’re able to hold out a minute. Okay, well, how about this?
Ori 0:56:26
And you know your dada is going to follow through on the bet. If you bet money and you win or lose the bet—
Liron 0:56:34
Yeah.
Ori 0:56:34
—he’s going to say, “Pay up.”
Liron 0:56:37
Exactly. Yeah, you better pay up. But you got to give it to me in escrow next time. I need the cash upfront.
Liron 0:56:43
Here, let’s do the over-under. So we name a number of seconds where we think it’s equally likely that you will be able to not blink for that long versus you won’t. And then that way we can pick which side of the bet we take, and it’s one-to-one odds.
So do you think 20 seconds is realistic to keep your eyes open?
Ezra 0:57:00
No.
Liron 0:57:01
No? Do you think 15 seconds?
Ezra 0:57:04
Yeah, probably.
Liron 0:57:07
Okay.
Ezra 0:57:08
No, I’m not doing it though.
Liron 0:57:09
You don’t want to do it live on the stream? Here, just look at this nice white light where the camera is and try not to blink.
Ezra 0:57:14
The light?
Ori 0:57:17
Here, I have an idea for a bet, okay?
Ezra 0:57:20
I want the lamp.
Ori 0:57:21
How about — is mommy going to appear on the stream? When is Mommy going to appear on the stream, if ever?
Liron 0:57:28
Right. Yeah, that’s a good point. Yeah, she’s been pretty clear that never is the likely option.
We can make a bet on what time will I get a text asking if I’m still streaming.
Okay. All right, next week on the show, my four-year-old.
Ezra 0:57:45
No.
Liron 0:57:47
Yeah, so he has a sister, Mitzy. She’s four years old, and she doesn’t even use Compy at all, right?
Ezra 0:57:53
Yeah.
Liron 0:57:54
But actually, our youngest kid, Raz, who turned three—
Ezra 0:57:58
Yeah, he loves it.
Liron 0:57:59
—well, he hasn’t been using Compy, but he’s been able to talk to YouTube, right?
Ezra 0:58:03
Yeah.
Liron 0:58:03
Yeah, I see. How does Raz use it?
Ezra 0:58:05
And then sometimes he’s like, “Play cars.”
Liron 0:58:14
Exactly, yeah.
Ezra 0:58:14
Because he keeps saying it over and over again. So then I started playing it over and over again and then something weird after, and then Mitzy started doing that, “Play, play, play.”
Liron 0:58:24
I remember that, yeah.
Ezra 0:58:25
“Play, play, play.”
Liron 0:58:26
Exactly. So basically all our kids are able to use voice mode to talk to that. It’s pretty funny.
Ezra 0:58:30
But usually I type it because he’s very annoying.
Liron 0:58:33
Yeah. All right, buddy. Anything else you want to talk about, or are you good for today’s Doom Debate?
Ezra 0:58:37
I’m good.
Liron 0:58:38
All right. Thanks for coming on, bud.
Ezra 0:58:39
I’ll come back next time.
Ori 0:58:40
Bye.
Liron 0:58:40
All right, come back next Friday. Give us the latest updates.
Ezra 0:58:43
Okay.
Liron 0:58:44
Okay. Bye.
How the AI Actually Beat Erdős’s Grid
Ezra 0:58:46
Bye.
Ori 0:58:46
Oh, man. We needed to bet. We needed to have you guys do a real bet.
Liron 0:58:51
Yeah. My wife just came in being like, “Where were you? I was scared.” Okay.
Liron 0:58:57
All right, everybody, so that was our drop-in six-year-old news segment. We got to learn from the youth. Gen Alpha, Gen Beta.
Ori 0:59:05
Hell yeah. That was awesome. You’re teaching him some pretty complex betting concepts.
Liron 0:59:11
Totally. Yeah. It seems like a little bit of it has stuck. I don’t know. We’ll give him this much. He remembered the sequence of events that happened.
But did he know any of the takeaways? Yeah, I like to think that some part of his brain has remembered it.
Ori 0:59:25
This comment, Lexar, had a good comment. “Lesson that Ezra learned. Lesson: price in the possibility of bailouts to your bets.”
Liron 0:59:34
Yeah, exactly, right? Which we covered that last week. And maybe on a subconscious level, I feel like kids do that, right? That’s why he felt safe plunging into the bet.
Ori 0:59:42
Right. He didn’t know who he was up against. You caved, though.
Liron 0:59:47
Yeah. It’s like everything I learned about prop betting I learned from a six-year-old. We should write a book of wisdom like that.
Liron 0:59:54
Yeah. All right. I think we should quickly finish out the math thing. I feel like we shouldn’t dwell, but there probably is one final takeaway here.
Ori 1:00:02
Is there... Okay, I feel like, yeah, maybe you could wrap it up.
Liron 1:00:06
All right. Let’s wrap it up. So we tried to maximize the unit distances. And I’ll just give you context on what was solved. So Erdős basically conjectured, look, my grid with all the diagonals, I have enough diagonals going that it’s — and remember, the length of the diagonal radii is one. So the number of diagonals is so many that even if you try to not use a grid and have some different configuration and try to get clever and do all these crazy designs, you’re not going to significantly beat just having the right grid with a bunch of diagonals. That was his conjecture.
And it did seem pretty plausible. I agree with Erdős’ intuition that that was a plausible guess. But then the AI somehow came out and said, “No, I have a configuration that actually beats a grid.” So in that sense, they disproved the conjecture. Remember, we were confused about whether it proved or disproved it.
So there’s a diagram here that gives you a little bit of an intuition for the AI’s approach, courtesy of the author of this piece. It looks like this. The AI managed to invent some configuration of points that doesn’t look like a grid. You can see it has all these mini circles of points.
The idea is that if you start from any of these black points in this diagram and then look at all the unit distances from those particular points, it turns out you’ll just be able to have more of these red lines. But there’s a caveat where this particular diagram doesn’t demonstrate that, but if you make the diagram so big that it’s bigger than the universe — much bigger than the physical universe — that’s how big the diagram has to be before the AI solution starts overtaking the Erdős solution. So it’s crazy stuff.
There’s more details. But yeah, the point is it found — and remember what we were saying about high-dimensional space. So it’s not a high-dimensional space in terms of 10 dimensions or 50 dimensions or whatever. It’s high-dimensional in the sense that there’s this thing where you add a square root to a bunch of integers. So instead of just looking at the integers, you look at the integers plus some square root, and in that sense, it’s another dimension for numbers and it used the... Anyway, never mind.
I don’t think any of these details can survive the simplification of a stream like this.
Liron 1:02:06
Hopefully I added one minute’s worth of explanatory value to the disproven conjecture. I encourage you to read the post. And then actually let me link you to this, because if you are a little bit math inclined and you read the post and you’re still a little bit confused as I was, you can get a little bit less confused if you go and read what I tweeted. I tweeted out here, let me share the tab.
I tweeted a link. You can go to my Claude here, where I was basically grilling Claude about this article saying, “Hey, Claude, can I ask you questions about this?” And I was asking it questions like, “Hey, was Erdős conjecturing that literally just a grid of points is the optimal arrangement of points for the most unit distances?” And it was saying, “No, not exactly,” blah, blah, blah.
But yeah, you can see I was saying, “I explained the intuition of why extra diagonals aren’t worth—”
Ori 1:02:58
We don’t see your screen.
Liron 1:03:00
Oh, you don’t see the tab? One sec. All right. So you can see here.
Ori 1:03:03
Yeah, there it is.
Liron 1:03:05
Yeah, I was talking to Claude, and you can go see my Claude conversation on Twitter. You can see I was asking, “Explain the intuition of why extra diagonals aren’t worth at super exponential scale.”
And it went ahead and explained. And yeah, this is definitely a peak experience using AI because I read a post, I half got it, and then I pull up an AI, and I can converse to my heart’s content, and literally within one and a half seconds, I get a perfectly prepared paragraph that’s exactly what I want to know about the piece.
And I know this is old news. I know this already happened two years ago, but I don’t study that much math. So the opportunity to use it to ask math questions was pretty amazing. So yeah, I encourage you guys to go read this Claude thing.
Liron 1:03:49
All right. Now we can move on.
Scott Aaronson: “The Last Days of Human Relevance”
Ori 1:03:52
Yeah. If we’re talking about it, isn’t it an interesting post? How about the Scott Aaronson post? Did you retweet that? Or I’m sure you saw it at some point.
Liron 1:03:59
Oh, I don’t think I retweeted. I asked Scott Aaronson a question, though. And I feel like that’s actually kind of interesting. If not the post itself, the question I asked him is interesting to me.
Ori 1:04:07
Sure.
Liron 1:04:07
So let me pull that up.
Ori 1:04:08
Yeah. Okay. Yeah.
Liron 1:04:09
Scott Aaronson is actually somebody I’d love to get on the show. I just haven’t gotten around to it. But I’ve started reading Scott Aaronson when I was in high school, so he’s definitely an intellectual hero.
Ori 1:04:19
Yeah. The gist of his article was sort of, “Hey, we are now at a point where AI is able to do the kind of scientific research. Yeah, possibly the last days of human relevance.”
Liron 1:04:31
Right. Yeah, dispatches from the last days of human relevance. So he’s talking about the unit distance problem, and he had an update with more problems getting solved.
What was your takeaway from the Scott Aaronson post? Just we might be at the last days here?
Ori 1:04:44
Here he is, a scientist in the frontier of this kind of research, and he’s like, “Maybe the thing that I work so hard on, this tool that we’ve made, can do everything that I’m trying to do. What do I do?” He was kind of contemplating that.
Liron 1:04:59
Right. Yeah. All right. So yeah, I agree. That was basically the takeaway.
Liron 1:05:07
Let’s see. Oh, look, I haven’t read this end note yet. He says, “End note. I should have foreseen, but didn’t, that the comments on this post would be dominated by people looking for ways to minimize whichever specific AI accomplishments I blogged about. Thus, it turns out the ability of AI to solve Erdős problems just demonstrates that Erdős problems were never serious math in the first place. Nothing like algebraic geometry or Grothendieck-style theory building.”
Should’ve censored that.
“Which remains untouched. Likewise, the story I shared was obvious AI slop. I had taken it as obvious that when assessing AI’s impact on the world, one needs to look at least somewhat into the future to remember where things were four years ago compared to where they are today, and at least try to draw a straight line through the data, if not the exponential that seems to fit better.
Does anyone seriously doubt at this point that major problems in algebraic geometry and other Grothendieck-friendly areas of math will fail to future AI models? Or that AI-written stories will improve not only to win literary awards from AI-native judges, but to avoid the features that commenters here are complaining about? And whenever that happens, there will be new confident reasons not to care immediately offered in the comment sections like mine.
Apparently, people do still doubt, hence the throwaway remark in my post about Penrose and Hameroff and microtubules.” Right? “And remember, search Doom Debates Penrose for Penrose’s objections.
If not that or something like it, what exactly do they think the ceiling will be and why? Recently,” — I should have mentioned this before — “I came across what I consider one of the greatest social experiments of all time, one that illuminates people’s reactions to every AI advance. A Twitter/X user named Schloms displayed the following AI-generated fake Monet painting and asked people to explain what made it worse than a real Monet painting.
If you haven’t seen this yet, I recommend that you try the exercise before reading further. As it was, numerous art aficionados responded at length, savaging the flat, lifeless, uncreative AI slop, the emotionless composition, the missing spark, the lack of tranquility, the harshness, the lack of depth and symbiosis, and on and on. Only after they had all said their piece did Schloms reveal that this is, in fact, an actual Monet painting.”
Yeah, he really told them hard. I love that.
Ori 1:07:18
Fuck yes, Schloms. Oh, God. It’s so annoying, the AI art. They’re just on a team. Wow, they looked at a Monet painting. They were saying it’s shit.
Liron 1:07:33
No, I know. These trolls are so obvious. So obvious.
Liron 1:07:41
Yeah. Look, it’s just a weird thing. I’m happy to be a little bit humble too, which is, I have been surprised by how many things AIs can do that take so much intelligence without skipping even farther ahead.
The existence of LLMs doing so many things that humans can do without — I’m happy to admit that I thought by this point there would be kind of a snap to the next paradigm where AIs are like, “Look, we figured out how to be intelligent. Intelligence is kind of this step. We got ourselves up to the step. We don’t have the same limitation as the human brain. We have way more. We can just throw more computation at it, and so we’re off to the races.”
Now, I think that is a description of what’s happening, but it’s happening on a multi-year timescale, whereas my intuition was that the snap would’ve happened on a matter of, I don’t know, hours? An Eliezer-style foom.
So, I’m happy to admit that we’re past the point where I would’ve predicted that a very rapid foom would’ve already started by. And I hesitate to speak for Eliezer, but I think that if he had to guess, as far as I can tell, I do think that if forced to guess, he would’ve been like, “Yeah, it probably would happen by now.”
But to defend both Eliezer and myself, I think we’re both saying that that wasn’t the main load-bearing prediction. The main load-bearing prediction is that it’s not about when your boat is going to get into the water, it’s the fact that if you keep pushing that boat into the water, it’s eventually going to float and chug around the water. We’re kind of pushing into this new regime of surfing. I don’t know if that’s the best analogy, but we’re in the foothills, right? We’re at the beginning of this new regime, and it hasn’t taken off yet.
But it seems like it’s — I still claim that it’s going to take off, okay? I still think there’s going to be recursively self-improving or at least vastly superhuman intelligence, and the fact that it’s just knocking off puzzles one by one on a matter of weeks or months or years instead of hours, I don’t think the big takeaway is that much impacted at the end of the day. Right, Ori?
Why the Foom Is Taking Years, Not Hours
Ori 1:09:27
Yeah, it makes sense. And the thing that stands out to me is recursively self-improving. That is happening right now. All the AI companies are using AI to program the next version of AI. It’s just not quite as fast of a loop because, as you said, it’s kind of a strange form of intelligence because it’s not like a human brain.
If a person could work as quickly as GPT to solve this Erdős problem, then they would be able to have so much power. Probably in clock time, it took GPT — yeah, the OpenAI model — to solve that problem. Probably in clock time it took five hours or something, right?
Liron 1:10:17
Mm-hmm. Right.
Ori 1:10:18
Uh—
Liron 1:10:18
It’s already going fast. Yeah. It’s getting there. It’s chipping away. Zoom out. If you just zoom out and give it a few years instead of giving it a few hours, if you just open the aperture a little bit, it does — I would claim that it looks like it’s fooming. I would claim progress is going really, really fast.
Liron 1:10:31
We should give a disclaimer to the audience right now because remember when I said my opinion and then I was like, “Right, Ori?” Okay, there was a commenter for last week’s stream who thinks that you are being held against your will to repeat opinions that I have.
Somebody thinks that you’re not being authentic to yourself, Ori. What do you have to say about that?
Is Ori Just a Yes Man?
Ori 1:10:52
Okay. Yeah, I saw that comment. And look, that guy commenting on my pres— This is the first time I’ve seen myself for an extended period of time on camera. And if he has criticisms, I’m assuming it’s a he — he has criticisms, you have no idea what I think.
When I look at myself on camera, I’m like, “Oh my God, this is how I come across?” So yeah, I think that guy, I could understand where that guy was coming from. Okay, yeah, maybe I’m being too much of a yes-man. But I am saying my authentic opinions. If I agree with you, then I say I agree with you, or if I disagree, I’ll say that I disagree or maybe just not say anything at all. But you’re not forcing me to—
Liron 1:11:41
Well, Lex here in the chat is saying, “Ori, don’t blink for two minutes if you need help.”
Ori 1:11:45
No, if I don’t blink for two minutes, then I win $100, right, Liron? That’s the bet that we made.
Liron 1:11:49
There you go. Yep.
Ori 1:11:53
Yeah, no, it’s just my style. I’m a hype man. What can I say?
Liron 1:12:00
Right. Yeah, Ori’s naturally a positive guy.
Ori 1:12:02
Yeah.
Liron 1:12:03
Look, I guess the thing to point out, too, in this case, for those who think I’m holding Ori hostage, is it’s not like — he got to choose which podcast he wanted to be part of, right? So, he could’ve easily gone and worked for David Shapiro, right? Or whoever, like Izzo Towel.
Ori 1:12:20
Yeah. And who knows? Maybe it’s a false flag. Maybe I do work for David Shapiro.
Liron 1:12:27
Yeah. Exactly, right. Okay. Yeah. Some people are saying that you’ve seen me, seen Davida. All right. So yeah. If you ever want to mix it up, don’t worry. I’m not going to stop you.
Ori 1:12:39
Thank you, Brian.
Liron 1:12:40
You’re not being detained.
Does P = NP + AI?
Ori 1:12:41
Thank you.
Liron 1:12:43
Yeah. All right. Let’s go to my comment here. So, this is unrelated to Scott’s post, but I said — I bring this up a lot on the show, and I was actually sloppy though. I should’ve known to put a better case for this. But I say, “I wonder if it might soon be the case in practice that P equals NP plus AI.”
That’s my equation. I’m shingying right now. I’m stoned. This is kind of a stoner equation, doesn’t really make any sense, but P equals NP plus AI in the sense that AI will be able to find a solution or an approximation, shortcut, or satisfactory workaround to any NP problem that we care about.
Then our only hope is that taking over the world is closer to PSPACE-hard, and AI algorithms won’t reduce problems so easily there. So yeah, I was basically telling Scott Aaronson, one of the greats of complexity theory, one of the living greats — I was basically saying, “Do you think P equals NP plus AI?”
This is only on the internet. Does somebody like me get to ask somebody like him a question like that? So Scott actually replied to me. He quoted me, and he’s like, “Well, I doubt this is true, if only because NP problems include all the cryptographic problems that were specifically constructed to be hard, like Bitcoin mining and LWE.”
What is LWE? Is it one of those logarithm problems? Man, I should know what that is. I’ve read a lot of his stuff. So anyway, there are problems in NP that are constructed to be hard — yeah, cryptography, even quantum secure cryptography, I think is still in NP.
And I actually have thought of that before. I just forgot. Somebody’s asked me the question, “Really, Liron? Do you really think that AI can break all cryptography that’s in NP?” And after thinking about it for two seconds, I was like, “Wait, no I don’t.” So Scott busted out the obvious correction that I forgot to remember again. I didn’t remember it when I asked him the question.
I guess what I should have asked is, “Okay, Scott, don’t you think that except for a few exceptions like this, which are designed to be pathological, don’t you think that all the practical problems...” Bitcoin mining is purposely saying, “Here’s a hash function, which is purposely designed to obfuscate. Do you think you can reverse this hash function?” And the answer is probably no. I think that security will hold.
But any problem that’s practical in any way, like traveling salesman, like shortest distance problems, find the perfect shortest distance — the idea that the AI will come up with some workaround that’s good enough. I do think there’s something there.
And sure enough, Scott humors me. He continues, he says, “While I can imagine AI proves P doesn’t equal NP and the Riemann hypothesis, it’s much harder for me to imagine an AI for which no natural open math problem, even let’s say among those that ultimately do have reasonable length proofs, ends up being too hard.”
So he’s actually saying that his intuition is that we won’t be able to shake out all the interesting results from NP. So I do think there is a little bit of a contrast between Scott’s intuition and mine.
I think I’m willing to grant superintelligent AIs a little bit more omniscience within the class NP than he is. And then he continues, he says, “On the other hand, yes, something like what you say will almost certainly become increasingly true as AI advances, at least outside the realm of cryptography. Indeed, that’s a huge part of the point of AI.”
So what Scott is saying is this idea of being able to solve problems that seem hard. The definition of the NP complexity class is problems that it seems hard to find a solution to, but once you find a solution, you’re like, “Oh yeah, that’s a good solution.” Solutions that are easy to verify, hard to find.
What’s a classic example of that? I guess a math proof. A math proof is a classic example because it’s really hard to search for a proof of all kinds of math, but then once you find a proof, you can look over it in a few minutes and be like, “Well, that’s a valid proof.” This took humanity 50 years to find, but here it is. You show it to an expert, the expert can verify it in a matter of minutes.
So that contrast between so much searching and racking your brain to find the answer versus finding the answer and verifying it quickly, that’s characteristic of this class NP. And I don’t know if I’d say a majority, but a really huge fraction of things that are interesting to do in the world have this property of being hard to find and then easy to verify.
So I asked one of the complexity theory greats, Professor Scott Aaronson, I asked him, “Hey, do you think AI is going to basically make NP easy? Not just easy to verify, but easy to find, effectively in practice?” And he kind of patted my head as he should, and he’s like, “Yes, Liron, that is kind of the point of doing AI.” A huge part of the point of AI is to take problems that we want to solve and try to easily solve them somehow. So, that’s the obvious thing to say.
And then what I said of every problem, he’s like, “Well, it just seems like there’s a whole ladder here, and it doesn’t seem like AI will just shoot to the top of the ladder. It seems like there’s a continuum where at the top you’ve got the cryptography, which is intentionally designed to be hard, followed by some proofs that just happen to be hard to find for a reason we can’t necessarily explain, but it can still be too hard for AI.”
And he’s probably right, but I also think that there’s a chance that there’s a hard separation between cryptography at the very top, followed by all proofs, where AI will just shake out the entire bag of proofs in NP. I think there’s actually a possibility of that.
Ori 1:17:37
It also seems like part of his response is the response that a lot of people give with AI, which is, “Yeah, AI can do a lot of things, but it’s not going to win in my field.” AI can never take my job.
Liron 1:17:51
Yeah. That’s a funny connection — “Oh, I’m a guy who searches for proofs, and so AI’s never going to search for proofs.” That’s a funny connection, like is he bringing his ego into this? But I do think that we’re just talking about the most basic theory of AI. I don’t think he’s brought his ego into this. I think he’s giving me a dispassionate analysis here.
Ori 1:18:09
Mm-hmm. Okay.
Liron 1:18:09
But maybe he is. You never know.
Ori 1:18:12
It seems like he’s saying that a little bit, like, “Hey, look, AI’s going to do a lot of things, but it’s not going to solve everything, Liron.” Which is kind of what you bring to different people that you debate against. It’s amazing how that seems to come in every debate, or in many debates.
Who was it recently that was saying how AI couldn’t do exactly what their job is at... Oh, man.
Liron 1:18:37
Yeah, Andreessen.
Ori 1:18:39
Andreessen says it. Sure.
Liron 1:18:41
I know somebody on the show said it recently. Yeah.
Ori 1:18:43
Someone on the show was saying it, yeah.
Liron 1:18:45
Okay. We won’t name names.
Ori 1:18:46
Journalism. That was it.
Liron 1:18:48
Oh, journalism. Right. Scott has a very self-effacing personality type. So if anything he would go the opposite way, right? He would tend to tell you that AI probably can do his job, and he’ll just be out on the street. That would be his natural inclination to take it that direction.
Ori 1:19:05
I think that should be our man on the street interview. It’s like, “Oh, are you worried about it taking your job? How about your job? How do you feel about it?” Everyone’s going to be like, “Well, AI can’t really do what I do.”
Liron 1:19:17
Yeah, exactly.
Liron 1:19:17
Me and Ori are making big plans for next week, okay? Because we’re going to be at Less Online. We’re going to have all this stuff that we’re doing, and when the conference finishes, we’re actually going to head out, spoiler, to the streets of Berkeley, California. UC Berkeley. Go Bears.
A little-known fact, I actually went to UC Berkeley as my college. And so we’re going to return back to the scene of my computer science education and go out and see today’s youth, right? Today’s young adults. We’re going to talk to them about what they think they’re going to do when they graduate.
Ori 1:19:50
Yeah. Then why are they booing AI at all these graduation speeches?
Liron 1:19:55
Exactly right. Yeah. So that’s coming soon. Because I know that a lot of you guys are fans of Man on the Street, so we’re literally going to go out on the street. So you can look forward to that.
Ori 1:20:08
Yeah. All right. Now that we’re almost an hour and a half into this, should we get to Twitter?
Liron 1:20:13
Yeah. Shall we get to the end?
Ori 1:20:15
Let’s get to the good stuff.
Liron 1:20:15
I think somebody’s been waiting patiently also to do a call-in. So we’ll—
Ori 1:20:19
Oh, okay. Yeah.
Liron 1:20:20
—do one tweet, and we’ll do one call-in. How about that? We’ll mix.
Ori 1:20:23
Yeah.
Liron 1:20:24
So here. All right. Call-in link. Here we go. All right, our VAMP.
Ori 1:20:30
VAMP. Okay. All right, let’s see what people are saying.
Roy Roy says, “I feel like recursive self-improvement is kind of taken for granted in a lot of scenarios and isn’t fully explained.” Yes. Totally agree with that. I think that a lot of scenarios have a sort of discontinuity phase where it’s like, “Eh, it’s pretty powerful. It does a lot of useful things.” And then discontinuity. It suddenly takes over.
Liron 1:20:54
Mm-hmm.
Ori 1:20:56
Okay. You just shared the call-in link.
Liron 1:20:56
All right. So we got a call-in link, so feel free to use that.
Ori 1:20:58
Yeah.
Liron 1:20:59
And then in the meantime, I’ll head over to Twitter. I’ll see what I bookmarked from this week.
Ori 1:21:05
Yeah.
Liron 1:21:05
But yeah, overall, it’s just kind of funny because the vibes this week are that it’s been a lull. Svie called it out in his update. Some people are calling it out, and it obviously reminds me of nine months ago or a year ago when people were like, “AI progress is slowing down.”
And typically what happens is, okay, yeah, an AI winter is now a month. And then a couple of weeks later, it’s like, “Oh, here’s a new breakthrough. AI winter’s over, guys.” Everything’s going at hyper speed now.
Ori 1:21:30
Yeah. And people are spoiled. This week, Claude 4.8 came out, right?
Liron 1:21:37
Right. Exactly. It’s like the IQ of the AI that’s available to everybody has noticeably increased. I told you I thought it was a little bit more thoughtful, noticeably. And it’s like, yeah, there’s been no news this week, just the waterline of intelligence making humans useless has just increased another few IQ points. It’s like, we’re not made out of IQ points, guys. I’m already tapping out here.
Ori 1:21:53
Right. Scott Aaronson was tapping out.
Liron 1:21:58
Yeah.
Ori 1:21:59
Or Scott Aaronson.
Liron 1:22:00
Yeah. You’ve got to know your Scotts. Know your Scotts.
Ori 1:22:04
My bad.
Liron 1:22:05
Yeah, Scott Aaronson. I’ve got to get him on the show though, for sure. And remember, he really gave us a nice comment because we actually did an episode where we reviewed him on a different podcast, and I just criticized him for being like, “Yeah, I was on the OpenAI safety team, and this is what we did.”
And my main criticism was, “Okay, look, you obviously had good intentions. It sounds like your boss, Ilya Sutskever, had good intentions. But it is kind of a Mickey Mouse attempt to solve this problem that’s clearly way above. You and Ilya are pretty clearly admitting that the problem is just light years away from where you need to be.
And yet, I don’t see you being one of the biggest doomers. You’re kind of being a measured doomer. You’re a little bit of a doomer, but also you’re holding out a lot of optimism.” So I was just challenging Scott. Why don’t you become an even bigger doomer given that your own description of the situation is that it’s so hopeless?
At least that was my read on his... So anyway, I made some claims. I don’t know if everything I said was totally fair. Maybe I let him have it. But he had a really gracious response afterward. This was over a year ago. Afterward, he actually posted on Facebook. He’s like, “Hey, I recommend you guys watch this. It was a really good critique, even the part where he tears me a new one.”
Yeah, he even said that. He’s super gracious, and like I said, an example of that classic Scott Aaronson self-effacement. So yeah, maybe I’ll see him again at Manifest, and then I’ll get him.
When you summon him to a court, I’ve got to give him one of those, right? I’ve got to serve him papers to come on “Doom Debates.”
Ori 1:23:39
Yeah.
Liron 1:23:39
That’s why I need to see him in person.
Liron 1:23:42
All right. Moving along. All right, so am I sharing my screen? Yes, I am. Okay. Bookmarks.
METR’s Beth Barnes: “We Are Not On Top Of It”
Ori 1:23:47
Yes.
Liron 1:23:47
Yeah, I have a major problem where when I open Twitter, it’s like I get so stuck on the home feed, right? It’s one of those things where it’s like, “Oh, I’m going to go to my Twitter bookmarks. Nope, I’m going to just scroll stuff.” Anyway.
Okay, let’s do an older one. Yeah, there’s so many options. We’re not going to get to all of them.
Oh, this is the Kelsey Piper tweet I mentioned. I’ll just read it. She says, “If we really built super intelligent AIs at our current ability to understand, and steer them, that’d be the end of humanity. It feels on a gut level like we won’t really do that since it’s insane.” But people are pointed right off that cliff and pressing the accelerator at full speed. That’s what she’s saying.
Yeah. Oh, and this was actually in reply to a Beth Barnes tweet. Elizabeth Barnes, she’s I guess the co-founder of METR. Let me show you her tweet here. I’ve got to share this tab.
Oh, it looks like my tab sharing is busted.
Ori 1:24:41
No, we see her tweet.
Liron 1:24:42
Oh, you do? Okay, weird.
Ori 1:24:42
Yeah.
Liron 1:24:44
Oh, because I’m sharing the window, not the tab. Oh, fascinating. Okay.
So Elizabeth Barnes says, “Our report,” the latest METR report, “focuses on claims that are solidly defensible and generally agreed within METR. Here I’ll give some personal opinions.”
Yeah, so METR released a frontier risk report, and you guys know METR. They do that graph saying that the time horizon of AI keeps increasing. So this is a very notable comment from Beth Barnes. This is arguably the number one most respected organization besides, let’s say, AI 2027, right? The number one most respected organization of telling us where we stand on AI progress.
And now she’s tweeting. She’s saying, “The report focuses on a narrow set of risks from current systems and on analysis rather than calls to action. It doesn’t really comment on how concerned should we be, does something need to be done, or are we on track to handle AI safety?
Sometimes people outside the field say things like, ‘The AI situation can’t be that bad. There must be experts who are on top of it.’ As an expert, I would like to be clear that we are not on top of it.
Some key aspects of the situation, in my opinion. Number one, we are likely on track to develop AI systems capable of causing human extinction or permanent disempowerment, quite possibly within the next few years.
Number two, things are chaotic and rushed. We aren’t on top of the basics. Models regularly violate user intent. Labs train on things they meant to avoid. Security probably isn’t good enough to prevent adversaries stealing dangerous models, let alone thorny questions of how to control and align superhuman AI.”
This is starting to read like a laundry list related to Eliezer’s from a couple of years ago. Remember it was a list of lethalities? A list of reasons we’re screwed. And here she’s recapitulating, which always goes to show Eliezer’s always years ahead of everybody else, in my opinion.
Liron 1:26:21
All right, so she’s continuing. “METR and other independent orgs, as well as safety and security teams at labs feel woefully under-resourced compared to the scale and pace of AI development. We’re struggling to build benchmarks fast enough, keep ahead of latest capacity developments, read and respond to all the safety-related claims that AI developers are making, run all the evaluations and assessments that companies and governments are asking us, plus develop the science needed to assess risk from increasingly capable AIs.”
What a statement, that the leading organization is struggling to create the next benchmark.
Ori 1:26:51
That’s a good point. We should dwell on that for a second. Oh, sorry, what was the — I didn’t hear it.
Liron 1:26:56
It was—
Ori 1:26:57
What?
Liron 1:27:00
I feel like the volume here is a little soft on that end.
Ori 1:27:02
Yeah, the volume on that was too soft.
Liron 1:27:07
Gotta work on that. Okay.
Ori 1:27:09
You could take the perspective of an OpenAI employee, or take the perspective of the rest of the world. That’s the what versus the yeah.
Liron 1:27:18
Oh, okay. Yeah, exactly. Right. OpenAI is saying yeah.
Yeah, so it’s crazy that they’re having trouble making benchmarks, right? It’s like, “Hmm, what should I do in the next two weeks? Should I try to spend effort at measuring how far we’ve come, or should I push even farther? Because it’s actually easier for me to just push the frontier than to even take stock of where I am.”
Ori 1:27:42
Or how about the notion of people anecdotally say, “AI? Oh, AI can’t do that much.” Literally experts can’t even find things that AI concretely cannot do. It’s saturating all the benchmarks.
Liron 1:27:54
Right. There’s the nuclear option of saying, “Okay, make me a million dollars.” I think AI struggled to independently make a million dollars.
Ori 1:28:04
Yeah, it seems like that’s where the benchmarks are going. They’re going to real-world impact.
Liron 1:28:09
Right.
Ori 1:28:10
You’re basically measuring an AI the way you evaluate an employee. You hire the AI. You’re like, “Hey, can you—”
Liron 1:28:20
Right.
Liron 1:28:22
This is random, but I’ve been having an experience with my own code. I’ve now been going a few months where I started with a code base that I worked on for eight years, for my own project, my own day job. We have a code base, we have a website that kind of runs the operations platform for coaching. It’s a coaching platform.
And I’ve been working on it for eight years of me handwriting code, artisanal handwritten expert code, and I haven’t touched it for literally three months now because I’ve been using Claude Code. I’ve just been talking to Claude about my code, and when I want to see code, I’m literally saying, “Hey, Claude, can you show me a bit of code from here?” I just ask it to show snippets of my own code, and I don’t even do that much.
And it’s just crazy how I’m now going farther and farther away from the code being in the state that I wrote it in. And I’m also caring less. I’m like, “Hey, Claude, in this file, are you roughly doing this?” And it’s like, “Yeah, we’re kind of doing this.” And I’m like, “Okay.” I don’t really know what’s going on in this file. I’m roughly kind of trying to give it hints and hoping that it is not creating too much technical debt, and I’m just testing the functionality. That’s all I care about at this point is testing the functionality.
Liron’s Vibe Coding Confession
Ori 1:29:19
Hmm. Wow.
Liron 1:29:21
I know. I’m just vibe coding. I’ve completely abandoned — not completely, but I’m definitely getting on that vibe. And, for the techie people out there, I migrated to a Postgres database, a Cloud SQL database. I used to use a NoSQL database because I think SQL is an annoying query language. I don’t like SQL. I never wanted to touch SQL.
But SQL is just really good for robust production infrastructure. SQL is not going to crash on you. You can easily fork the entire SQL database. You can have test databases. SQL is basically the gold standard of operations.
So I finally got us on a SQL database, and I haven’t written a single query. I just talk to the AI and ask it what’s in my database. And I still use NoSQL ## Vibe Coding with Claude (continued)
Liron 01:30:00
conventions. I’m like, “Hey, what is user.surveyfields.age?” I just ask it. I know that’s not even how you’re supposed to talk about SQL, but that’s how I like to talk, and it just translates.
Ori 01:30:13
Yeah.
Liron 01:30:14
So Beth Barnes continues, “Any reasonable civilization would clearly be taking things much more slowly and carefully with AI.”
Let’s see. I feel like normally when people have sound effects, there’s a third party doing the sound effect, right? Because when I get the idea for the sound effect, the moment has passed by the time I can actually go do it.
So she says, “Yeah, any civilization would be taking things much more slowly and carefully with AI. The benefits of getting upsides of advanced AI a little faster are small compared to the risks of getting it irrecoverably wrong, and we could lower these risks by going slower.”
Yeah. So that’s basically her take — she’s an expert saying AI should go slower. At the end of the day, are we going to listen? I guess not.
Ori 01:30:59
It’s hard to look away from all the benefits. In some sense, you want the benefits of Opus 4.8, right?
Liron 01:31:07
Right. Hell yeah.
Okay, so that’s Beth Barnes, and then Kelsey Piper was basically echoing what she said.
Liron 01:31:16
Alverin has been waiting patiently on the stream. Let’s take a viewer call. Is it going to be a debate? Is it just going to be a niche topic? Let’s see.
Hey, Alverin. Am I pronouncing your name right?
Alverin Joins: Can We Wait, Then Hit the Off Switch?
Alverin 01:31:29
Hey, how’s it going, guys? Yeah, you’re pronouncing it right. Great job. Thank you for that.
Liron 01:31:35
No problem.
Alverin 01:31:35
Hold on. I’ll try and center myself here. Do you guys hear me fine, and is my camera good?
Liron 01:31:43
Yeah. It seems like you’ve got some podcasting experience.
Alverin 01:31:46
Not really. I was on Destiny’s stream at one point, which you recently had them on, and I was talking about AI doom, actually, on his stream. So maybe you can credit me a little bit with putting that little earworm into his ear.
I was comparing the Trump administration’s — a lot of their decisions around AI, like the Stargate project — and I was just linking a bunch of different things together. But I know we’re kind of short on time here, so there’s two things that I just want to go through real quick with you.
The first one is just a quick fan question. It’s not super serious. It’s actually just kind of lighthearted. Whenever you ask people for their P(Doom), do you actually play the noise for them right there? Oh, did he just drop?
Ori 01:32:38
Liron just messaged me, told me his power went out. But we can keep going. I can answer this question though, actually, because I’ve seen what happens behind the scenes.
Alverin 01:32:49
Okay, so what is it? Does he actually play it for them?
Ori 01:32:52
No. It’s just a live conversation like this. So he’s like, “Get ready for the big question.” And yeah.
Alverin 01:33:01
It’s added post-production.
Ori 01:33:01
I guess we’re revealing some of the secrets of how the podcast is made. But some guests know about the big question, so they know what’s coming. But many don’t watch full episodes to know it’s coming.
Alverin 01:33:23
I was just curious if they heard the actual AI generated music that’s played in the actual videos. I honestly think it would be way funnier if he played it for them so that they could hear it before they answered, because it’s kind of ridiculous, but it’s funny. I love it.
Ori 01:33:44
Well, now that we have the soundboard, maybe that’s a possibility.
Alverin 01:33:47
Exactly.
Alverin 01:33:49
Yeah, and then my other question — you’re pretty well-read on this stuff, Ori, so you could probably...
Ori 01:33:58
I can take a shot at it. But Liron’s following also, or we could catch him up when he gets back.
Alverin 01:34:03
Okay, cool. So my next thing requires a little bit of scaffolding, conversational scaffolding to explain adequately. So I came up with this concept of what I call ASIQ — it’s a play on intelligent quotient.
The reason I developed this is because I was thinking about... just to give people an idea of where I’m coming from, my P(Doom) is like a three-layer cake. 49% of my P(Doom) is things are going to go not great, and we might have this extinction risk.
2% is actually we might be able to get a utopia out of this, so it’s the good outcome. And then the other 49% on the back end is we actually end up with AI models that are just permanently scaffolded over and over again, and we don’t ever actually figure out how to do recursive self-improvement. So that’s kind of where I’m at on the P(Doom) spectrum.
Ori 01:35:19
So your P(Doom) is basically 49%—
Alverin 01:35:20
Pretty much 49%.
Ori 01:35:21
Because that’s the doom side. Okay.
Alverin 01:35:23
Exactly. And then this ASIQ thing is basically — so an ASIQ of 100 to me would be like, okay, this model is now as intelligent as all of society. It is just as smart as if we had all put our brain power together to try to figure out some kind of problem, like whenever we all came together to try and solve COVID.
It’s something that can operate pretty much on a societal scale just as intelligently as we can.
Now, my question for Liron, and for you as well, Ori — do you think that researchers will actually know before we get to that point? Will they be able to recognize, “Oh, we’re actually at an ASIQ of 70, and so maybe we shouldn’t hit the gas anymore. Maybe this is a good point to pause”?
Do you think that we’re going to be able to have any insight into how close we are to having an ASI that is smarter than the rest of our society?
Ori 01:36:35
Well, I think that’s part of what people like Liron are calling for. Even Eliezer Yudkowsky said in his interview with Liron — he didn’t define what the red line is.
He knows that the AI that’s more intelligent than all of humanity is dangerous. He knows that the AI that builds that is dangerous. It’s like, what is the “it” in “if anyone builds it, everyone dies”? What is the “it”?
I was just looking back at what he said, and he’s like, “It’s the AI that’s more powerful than all of humanity, or it’s the AI that builds that.” But where’s the specific red line? He doesn’t define it, and unfortunately it’s hard to define.
And then there was recently a big public letter called “AI Red Lines.” I’m just on their website now — it’s red-lines.ai. And it’s got all these important people signing it, like Nobel Prize winners, Nobel laureates, Yoshua Bengio, Yuval Noah Harari, author of Sapiens.
The headline is, “We urgently call for international red lines to prevent unacceptable AI risks.” It’s not that they came out and had actual red lines. It’s like, we call to create the red lines.
I think there is a question of is it possible to define what those red lines should be? And yes, it’s possible. They launched this during the 80th session of the United Nations General Assembly. So there’s progress for serious, important people to say, “Hey, let’s get together and get really clear on where we define the threshold.”
People are working on that, and there are actually some policy papers that do set some thresholds. But I don’t think they’ve been part of the discourse in a real way, unfortunately, like they should be.
One example that is currently moving forward with some progress in the UK — Control AI is leading it — they’re trying to get the UK government to put forward a policy that says, just as a value, “We do not want to build superintelligence.” That’s not specific like no data centers above this size, or no AI above this many flops. It’s just declaring a value of we do not want to build AI superintelligence that’s more intelligent than all of what humanity can do.
So there is some progress to that, and people want to get together and talk about it. Also, I think that many of the AI lab leaders say that they want to do that. Famously, Demis Hassabis from Google DeepMind constantly says, “We should have the equivalent of the international agencies that we have for atomic energy and nuclear weapons — we should have that for AI also.”
People are trying to make progress on that. But it’s hard to know how genuine — this is my opinion — how much effort are people like Demis Hassabis putting towards it?
Ori 01:40:54
Oh, here we go. We got our guy back.
Alverin 01:40:56
Now he’s caught in between.
Ori 01:40:58
Liron is logging back in.
Liron 01:40:59
Power outage. Can you guys hear me?
Ori 01:41:03
Yes, but you sound a bit echoey.
Liron 01:41:07
Yeah, I sound bad. For some reason, my microphone isn’t working as usual, so I’m using a worse mic. Is it good enough, or should I try to fix it?
Ori 01:41:12
It’s good enough, yeah. But it’s like your laptop mic.
Liron 01:41:20
Yeah, it’s a low-quality mic for sure.
Ori 01:41:22
Yeah. But anyway, I was just complaining about how the AI leaders will make statements saying, “Yeah, we should get together, we should all work together on this.” And it seems like a lot of lip service.
Here they are, pouring billions and billions of dollars into making data centers, into making new AI capabilities, and what fraction of resources are they devoting to calling for policy change, to bring people together to work on this policy that they say is critically important?
Alverin 01:42:05
Right.
Liron 01:42:07
Yeah. I agree. It’s messed up.
Alverin 01:42:08
I guess I have a follow-up. The way I see this working in my head is at some point, we’re going to get the ability — well, arguably, it seems like we’re aiming towards recursive self-improvement. We want to create models that can improve themselves, and we don’t have to be in the loop as much.
But I guess the way that I see it is maybe the current researchers are like, “Well, we’re not really at the point where we’ve figured out true recursive self-improvement. We’re still very much in the loop. We are automating a lot of coding, but it’s not like, ‘Oh, I sent off 100 AI agents, and they gave me a new model on Monday.’”
I’m curious as to what you guys think about once we get to the point where we do have recursive self-improvement — whether the researchers might just be of the position that, “Oh, well, we’re going to know how close we are to a model that is actually dangerous for us. And the reason we’re not spending a whole bunch of money on throwing on the brakes is because we’re actually just not at recursive self-improvement yet.”
But once we get to recursive self-improvement, and then we’re like, “Oh, wait, hold on, now we actually need to throw on the brakes” — what makes you not confident that they will just throw on the brakes at that point?
Ori 01:43:23
You’re muted, Liron. It’s the famous Hail Mary.
Well, I think Eliezer Yudkowsky had a great critique of this, which was — well, they haven’t been prioritizing safety up to now. So suddenly, when things really get critical, then they’re going to pull back?
Oh, I guess we lost Liron. For context, everyone, Liron said he had a power outage, randomly had a power outage. So here we are. I guess he’s logging out and logging back in.
No, we don’t hear you. I see you talking.
Alverin 01:44:02
This is very convenient for you, Liron. No, I’m just kidding.
Ori 01:44:05
Yeah, we don’t hear you.
Alverin 01:44:07
A little bit of comm issues.
Liron 01:44:07
Testing.
Ori 01:44:09
Yeah, now we got you. Yeah.
Liron 01:44:10
Oh, weird. That is so weird. Is it coming in now?
Ori 01:44:14
Yeah.
Liron 01:44:15
Huh. Okay. Well, we’ll keep trying it. Yeah, no, I agree with what Ori said. There’s always that possibility.
And yeah, by the way, if it’s crazy loud, I can also turn down. Is it better if it’s this volume? Testing, testing.
Ori 01:44:34
Yeah, that’s better. That was a little better.
Liron 01:44:37
Interesting. All right. So the thing is that, yeah, you could always argue, hey, they’re going to get spooked enough, or some circuit breaker’s going to trip where they’ll be more sane, and so will everybody else.
But they certainly aren’t acting now like they’re preparing to break any circuits. They’re just acting like they’re racing. You see what I’m saying? So you can always hope, but there’s just not many signs of it.
Alverin 01:45:00
Right. I guess my position is, my skepticism comes in at — it could be the case that the reason they’re not focusing on safety right now is because the safety consequences are so limited right now. We don’t actually have any instances where the models are causing mass catastrophes yet because we haven’t implemented them in that way.
But once we get to the point where we’re like, “Okay, now we are ready to start implementing these models...” And I think what they’re really aiming for is recursive self-improvement. So once they get to actual recursive self-improvement and they cross that horizon, it might be the case where they’re like, “Okay, now stuff is serious.” We’re not just talking to the model. Now it has this ability to improve itself wildly beyond what we already figured out.
Alverin 01:45:54
One extra little thing to tack onto this. I imagine if we do get to this system, what do you think about them limiting what I call actuator channels? Making it so that the model can only produce text or actions within very constrained bounds.
So it’s a powerful model — superintelligent or close to superintelligence — but it can only generate and produce information that we basically allow it. For example, maybe we ask it a question, but we only let it answer in multiple choice, to protect against super persuasion, or to stop it from being able to break out of its box.
If we don’t ever actually program it or give it the ability to do those things, and we’re dealing with actual recursive self-improvement at this point, not just a chatbot — I feel like AI safety is going to scale up as the capabilities scale up, but we’re just not necessarily there yet.
It’s hard for me to imagine how a superintelligent system can bypass those actuator channels that we develop for it. I can imagine if we literally just put it on the internet, yeah, of course it’s going to be able to get past those actuator channels. But it’s hard for me to imagine if we literally just keep it in an air-gapped box in a bunker underneath the United States and we only let it answer us in multiple-choice questions.
How does it get to the point where it’s actually going to take over the world from that spot? Do you think it just convinces us over a long period of time?
Ori 01:47:24
You’re muted. Yeah, there you go. No, we heard you for a second, but you were muted at first. Now you’re muted.
Alverin 01:47:34
What a tragedy. No, Liron, you can’t do this to me.
Ori 01:47:40
Oh, he’s gone.
Alverin 01:47:44
I’ve been trying to get this conversation for... No, not really. But I have been watching you guys every Friday, and I was really hoping you guys would do call-ins again so I can get these questions out. There we go. We can hear you now.
Liron 01:47:55
Okay. Yeah, I think I’ve got a workaround where I’m just using a simpler configuration here. It’s coming in pretty crisp?
Alverin 01:47:59
Yeah.
AI in a Box & the Super-Persuasion Threshold
Liron 01:48:01
Sweet. Okay. This does mean that I don’t think I can do sound effects, so is there even a point to doing the stream at this point if I can’t do the sound effects?
But okay. You basically asked two questions that are kind of related. Can we stop it when it just starts getting more and more dangerous? And what if we only let it communicate multiple choice and air-gap it? The classic AI-in-a-box problem. It’s always good to revisit that occasionally.
I guess let’s take them in backwards order. The classic AI-in-the-box problem. Well, these days, it’s just so implausible that anybody’s actually going to keep it in the box. They’re all rushing to make it available by API. They all want to talk to it.
Imagine the intellectual daring to be an AI researcher today being like, “I’ve trained a new model, and I’m only going to let it talk to me with multiple-choice questions.” Even before we get into the effectiveness of that, imagine the boldness. And somebody’s boss would be like, “Well, your coworker here is actually training the AI in only two days. How fast are you training your AI? Because it’s bonus time, and who’s going to get the bonus?” There’s so much pressure.
“Hey, our competitor, Elon Musk with Grok, he says he’s going to train it in five hours.”
So yeah, the idea that somebody would have the discipline to only do that is starting to really stretch belief. But if all the AI companies agreed, “Hey, we can only talk to the AI in the form of multiple-choice questions” — okay, I think that would buy time.
But the problem is you still want it to give you elaborate answers. So enough multiple-choice questions adds up to anything — like what letter do you want to say next? Like a Ouija board. So by the time you’re getting useful information out of it, you’re just going to ask it more questions. You’re going to take longer, but the more useful stuff you want out of it, the more channels you’re giving it to manipulate you. Unfortunately, there’s a very tight correspondence between how much it’s helping you and how much it can hurt you. It’s like you let somebody in your life, you love them, but then you give them the power to hurt you. It’s kind of analogous to that.
Alverin 01:50:00
Yes. Quick clarification though. For me, the big thing that’s separating this in my mind is I think the reason models are widely distributed now is because they’re chatbots. That’s the extent of what they are. I think if Demis Hassabis was convinced that they had actually cracked recursive self-improvement, I don’t think he would be like, “Oh, cool. We’ve got to deploy this next Wednesday so that we can automate call center jobs away.” He would be like, “Let me get the president on the phone so we can talk about what the future of technology is going to look like.”
Liron 01:50:37
Okay, but are you kind of satisfied with the multiple choice? Why that doesn’t seem super promising?
Alverin 01:50:43
So for the multiple choice, I guess at the end of the day, they could super persuade us through multiple choice over decades and decades if they really wanted to.
Liron 01:50:51
But it’s not over decades. Remember, the relationship is that by the time the multiple choice is useful or interesting, by that time you’ve also opened the persuasion channel. So if you’re only asking 10 questions total, okay, maybe it can’t persuade us, but is that a realistic scenario? By the time we’ve asked it something interesting, it can say a lot.
Alverin 01:51:17
Let me think about this. I guess the way I see it — another element we should add to the conversation is, I think for us to be super persuaded, what if you need an ASIQ of 1,000 instead of 100 to actually super persuade us, just because the problem of super persuading us is so complex?
For example, I don’t think that all of our world could write a document that could make me do something that I know is harmful to myself, if I just read it. So super persuasion I think is possible, but I think you need an exponential amount of intelligence. It’s not like, oh, if this model is as smart as our society, then it can super persuade me. But if this model is as smart as our society, it could probably solve a lot of different forms of cancer. So we can limit how smart the model is getting, even though it’s smarter than us, ideally.
Liron 01:52:17
Okay. But it does seem like this is kind of a separate part of your argument. Maybe we can seal off the part of your—
Alverin 01:52:26
No, no, it’s part of the argument. Let me draw the connection. So let’s say you have this superintelligence, but it’s not smart enough to super persuade you, but it is smart enough for you to use. Super persuasion wouldn’t be an issue if it was not smart enough to super persuade us, because super persuasion is such a hard task.
Liron 01:52:43
Yeah. It does seem like you can already talk to today’s top AIs about how to go about doing marketing, doing persuasion. And to me it seems like there’s a pretty consistent connection between the fact that they can’t be self-driven doing large programming projects — they do kind of need to stop and I’m able to give them advice, so they’re not fully super intelligent there yet.
It seems to me like that is pretty closely related to how they can’t be super persuasive, effectively persuade over time. I feel like my best guess is at around the time that they can operate their own utility online, build it, make money from it, maintain it — that’s probably going to be around the time that they can operate their own influence campaigns online. Maybe not, maybe it’ll be lumpy. But it just seems like there’s kind of a connection there in terms of an autonomy level.
Alverin 01:53:37
I guess when I think about super persuasion, I’m not actually thinking about writing stuff online. I’m thinking of this model is inside of a bunker somewhere and Sam Altman or Demis Hassabis or the president is talking to it, and you’re saying this model is going to convince the president or Sam Altman or Demis Hassabis specifically to let it out of the bunker. Because it’s not connected to the internet. If it’s a model that can recursively self-improve, I don’t think they’re going to put that on the internet.
Liron 01:54:05
Yeah. So you’re imagining a scenario where there’s this super model, like a Mythos type situation. I like to think Mythos isn’t on the internet, even though hackers claim to have infiltrated the weights and everything. There’s a claim of a failure of security with Mythos. I don’t know how true it is, but let’s assume they have something like the next Mythos, or five Mythos’ in the future, and only the president has it and they manage not to get it on the internet.
Yeah, it might just be an advisor, especially if the ASIQ isn’t super high yet. It reminds me of the scenario in Max Harms’ book, where he paints a picture of only a couple of years in the future, and they unveil this next generation AI, which is super intelligent, but it’s not so super intelligent that it can just take an hour to kill everybody. It needs time. It needs to act relatively slowly.
So I can totally imagine that we get into a scenario where it’s just talking to the president. It’s not outputting that much persuasion. It’s mostly just answering the president’s questions in an authentic way, kind of similar to the way that GPT seems to answer most of our questions today. Yeah, I can imagine that scenario, and I can imagine that we buy ourselves time.
It just becomes harder and harder. The more and more intelligent something gets, and the more and more widespread it gets, then the less and less time that we can buy ourselves. But I agree that we might buy ourselves some time.
Alverin 01:55:14
And so this is connected to the actuator channel thing that I was saying too. Ideally, we don’t let this model improve itself once we’ve got it to 100 ASIQ. So we have it at 100 ASIQ — it’s as smart as the rest of our civilization combined. But we don’t let it go beyond that because we know that the further you get beyond that, the harder things become to steer.
And then another thing you granted me — you were like, “Oh, maybe it’s even the case that it’s answering us non-adversarially.” Just to be clear, I assume that all models are answering us adversarially right now. You say a lot, you’re like, “Oh, the models are really helpful.” It’s like, yeah, the models are really helpful because they have been RLHF trained to be super helpful. But if you didn’t RLHF train them, they would be adversarial or they would do random stuff that we didn’t like.
So even this super intelligent model — you can say it’s adversarial against us and it’s answering our questions adversarially in the bunker. But even if that’s the case, if it’s just stuck inside of a bunker and we’re not letting it improve itself, then all it can do is answer our questions. And then you might argue, oh, well, ASIQ 100 is enough to super persuade us. But I think actual super persuasion where it outputs text and Donald Trump reads it and he’s like, “I got to get this thing out of here” — I think that’s a really hard problem. That’s not something ASIQ 100 could do. That’s ASIQ 1,000 or 10,000. Does that make sense?
Liron 01:56:45
Yeah. It sounds like there’s two prongs to your argument. There’s the, what if we just stop it when it feels more dangerous? And then there’s the, we can get into this equilibrium where it only starts to feel dangerous, but it’s still super useful because it’s still giving us advice and it’s not free on the internet.
So I guess you have this mainline scenario where we have this AI that’s the next Mythos or whatever, and it’s super useful, and it got to the point where we decided it should stop because it’s starting to self-improve, and we should just aim for that point. Am I characterizing you accurately?
Ori 01:57:16
Just aim for that point. Yeah, we recognize that it’s—
Liron 01:57:18
The point where it’s starting to self-improve, yeah. And you know I’ve talked about the idea of edging your intelligence explosion. We’re just trying to edge here. We’re not trying to get the explosion, we’re just trying to edge.
Ori 01:57:29
Yeah, exactly. We get to the point where we’re edging and we don’t want the explosion. Sure.
Liron 01:57:34
Right. We’re just trying to edge AI to the point where we notice it’s swerving.
Ori 01:57:37
Wait, can I just ask — maybe this is why the metaphor is the metaphor. What happens in the edging situation? How does it usually turn out?
Liron 01:57:46
Yeah, exactly.
Alverin 01:57:46
I think it depends. If I’m edging and you take my hands away, then actually, it depends on what type of person I am, but I might not be able to complete my intelligence explosion.
So my point is just that I think the people who are working on it have better insight into it than we do, of how close we are to that. And I think as we get closer, they will want to slam the brakes on more and more. But right now, we don’t have a model that actually is that crazy. And so maybe that’s why they’re not slamming the brakes on it.
Liron 01:58:22
I agree. It would be really convenient if we can try to edge it, and then once we get into that edging scenario, the situation will look different. And so there’s a correlation between how scary it looks in the moment and whether that moment is the right time to stop. Because that would be very convenient — as long as you can just look around you and see how scary things feel and then act accordingly.
That’s a great way to go if reality lets you do that. Just my own concern is that I feel we’re just running out of time. I don’t think that the stop button is going to happen to be available at the time when it becomes intuitively clear that now’s the time to press it.
Okay, but let’s humor that. Let’s say we are going to get lucky. Reality is going to give us a time where it’s simultaneously feeling scary, it feels like we’ve gone too far, and there’s still time to stop. Let’s say we’re putting all our eggs in that basket. Then what I would ask for right now is prepare to stop.
Even if we’re saying, “Yeah, we’re going to stop later, we’re not stopping now, it doesn’t feel scary enough yet” — where’s the preparation to stop? And that to me exposes the lie of this idea that, “Oh yeah, we’re going to stop later.” What’s your excuse for not preparing now? Because the AI companies are clearly not preparing.
Alverin 01:59:31
When you say prepare to stop, just to be clear, you’re saying we need... I guess a lot of times people say a big off button, but the way I see it is research is a continuous development. If the order comes from Demis Hassabis to not develop the next model, the people at the company are not going to develop the next model. They’re just going to keep that model they already have in the bunker. And if Demis Hassabis says, “Don’t release it,” then they’re not going to release it.
Liron 01:59:57
Well, okay. So what does prepare to stop mean? There’s a couple different things. One sense of prepare to stop is, hey, we’ve been testing it and it’s getting too aggressive, it’s starting to copy itself. So one thing is shutdowns, off buttons.
And I actually think that there is a good chance that in some or even most, maybe all — probably not Elon’s — but in a large fraction of AI companies, there is a project by their internal computer security team, and the project is like, “Here’s the internal off button. Here’s how we quickly turn off our data centers.” I suspect they are planning things to that degree. I don’t know if they’ve got an emergency response for what if it’s gone out as a virus.
But so maybe they do that. This idea of let’s stop — one of the technologies is the social technology of, okay yeah, we’re stopping. And it’s kind of robust that the CEO... Well, you need more than the CEO’s order because you need a covenant with the board. You need to have advised the board. The social technology of stopping — you probably need to have coordinated with your competitors because otherwise the board or competitive pressure is going to be like, “Okay, you’re fired. We’re going to install a CEO who doesn’t think it’s time to stop yet because our competitors aren’t stopping.”
So to me, prepare to stop is more than just the internal off button that you think is secure. It’s also that coordination technology.
Alverin 02:01:08
I feel like the coordination technology, the social technology, actually comes from our government. I think if the government, if the CIA found out, “Oh, we actually have a recursive self-improvement model that Google DeepMind just created,” they’re going to go in there and say, “Hey, listen, we know where all your data centers are. We know all the people who are involved in this. It’s all public information to us. We’re going to stop this here. You guys are not going to develop the next model, and we’re not going to deploy this model until we’re 100% sure that it’s not going to recursively self-improve itself into 1,000 ASIQ.”
Ori 02:01:42
Wait, we got to jump in right there because that literally just happened with Mythos. It wasn’t recursive self-improvement, but Anthropic came out with this model and the government was like, “Oh shit, we should pay attention to it.” And then the profit motive came in and disrupted the responsible figure from giving a safe overview of all the models.
Now it’s back to a free-for-all, and who knows if the government’s going to go in and be some kind of intermediary there. So there was a test case, and it was a pretty strong test case too. They were like, “Look at all these zero days.” And even in that test case, it got close to some kind of responsible act, and so far the profit motive, the incentives have led to not some kind of responsible oversight.
Liron 02:02:37
Okay. We got to wrap it up, or the whole stream is going to end in 10 minutes. And yeah, Alverin, this has been great, man. Come back next time. These are good arguments and I think you’re representing the everyman — the everyman of being like, “Oh yeah, it’s scary, but can’t we just wait till it gets scarier?”
So I’m happy to engage with this. I think there’s definitely some good meat here. I would love to have better, snappier responses to this besides yeah, maybe it could work.
All right. So yeah, Alverin, we’ll let you go here.
Alverin 02:03:07
Can I have one more thing? Just one more thing, okay?
Liron 02:03:09
Okay.
Alverin 02:03:10
My last thing is, I’ve heard you say this a lot whenever you’re having conversations with other people, where you’re like, “Tell me one thing that AI won’t be able to do in the next year.” And you’re like, “Just tell me what it’s going to be.”
My thing — I work for a call center, basically, and I do customer service. I’m going to be honest with you, I’ve seen models that do customer service, and they actually fail. Specifically Robinhood’s customer service model just fails.
So if you are still doing that as a bet, please give me whatever odds you want, but I almost guarantee you that next year I will still have my customer service job, and it will not be replaced next year this time. Or even the end of next year. The end of 2027, they will not have automated my customer service job. I almost guarantee you that.
Liron 02:03:57
But is it a reliable — this is interesting because I employ some customer service people, and they’re augmented with AI. But I agree with you that pulling the trigger on... My experience has been that it’s a situation where if you had 100 customer service people before, you can probably go down to a lot less than 100 because they’re so AI-empowered. But if you had three before, you probably don’t want to go down from three to zero. You probably just want to keep the three.
Alverin 02:04:20
So at that point, you’re betting on how good of a customer service agent I am? Because I might be one of the guys left with the AI impact. I don’t know.
Honestly, we aren’t even using AI at my job yet, so it’s really hard for me to imagine that I’m even using AI by 2027, to be honest with you.
Liron 02:04:47
Well, don’t you think AI has made you and your team — so you’re saying you’re not even using it? So you’re saying it hasn’t even made you more productive, hasn’t made your team more productive?
Alverin 02:04:53
They’ve told us you can use Microsoft Copilot. They’ve said you can use it, but they haven’t given us any direction on how to use it. I don’t know what situation I would ever use it in. Most of my job is helping people log into their account.
I could see maybe you develop a model that explains to someone how to log into their account. But at the end of the day, people want to talk to other people to help them log into their account.
Liron 02:05:12
Are you comfortable sharing what industry you’re in, or you want to keep it private?
Alverin 02:05:16
Yeah, no, it’s fine. It’s healthcare industry, I guess.
Liron 02:05:20
Well, there’s a lot of private equity searches going out right now looking for companies like what you described that haven’t given their employees AI tech, because the play is to buy out your company, install AI best practices, and then suddenly have better profits.
Alverin 02:05:34
Well, it doesn’t even matter, but I see what you’re saying. I would love for that to happen, actually. Honestly, please automate away my job so I can collect unemployment for the next six months. I’m tired of working.
I really would like to see this automation happen quicker. I don’t see it happening, though, at least where I’m at in my position. So basically, my message to the researchers is hurry it up.
If we really are going to get AGI — also, that’s another thing. If we develop AGI, I don’t think it’s going to automate jobs. They’re going to use it for recursive self-improvement and keep it in a bunker somewhere. They’re not going to replace the McDonald’s worker. Why would you waste tokens on replacing the McDonald’s worker when you could spend those same tokens improving your next model and creating a superintelligence? I don’t know why you would waste the tokens doing that.
Liron 02:06:18
All right. Fair enough. Lots to talk about. We’ll let you go. Thanks so much for sharing, man.
Alverin 02:06:21
Thanks for having me on, guys. That was great. Bye-bye.
Brian Joins: The Ways It Could Go Right
Liron 02:06:24
All right. So yeah, my plan is we’ll do Bryan and Jack, and then we’ll just wrap it up. We’ll just do a couple minutes each. We’ll have more streams, guys. And actually, I think Alverin mentioned something about not being able to talk to me again. But you guys can all talk to me if you come to Less Online, keep that in mind.
All right. Bryan. Hey, how’s it going, Bryan?
Bryan 02:06:45
Oh, hey. What’s up? Hold on. Let me close the stream real quick. Okay. Hey. Hi. How’s it going? You guys rock. Really appreciate you.
There’s a lot to talk about. I kind of made a list. I think basically the main thing is, we got the 50% versus the 50%, right? And I’m pretty much with you. I think we’re at about 50/50.
But we’ve done a lot of talking about how it could go wrong, and I need more qualities and quantities about ways it could go right. How much is your 50% chance of it going right based on us stopping, and then what does stopping actually mean?
Because aren’t there ways to develop systems without being recognized and being noticed? And couldn’t those systems also turn into something more intense than we intend in secret? Of course, at the moment, we have big data centers and we sort of record what they’re doing. I get that, but I just don’t feel like saying stopping everything is robust enough to make up the entirety of the other 50%, for me personally.
And I think the other 50% has to do with a lot of things that we haven’t quite talked about yet — like making a corrigible AI versus a psychologically biased AI versus an internally resource-driven AI. I think talking about things like this can sort of get us to that other 50%, because I am under the impression that humanity doesn’t survive by doing nothing. I think we actually have to make effort. And I don’t think stops are entirely robust. I think pauses have a tendency to resume.
Liron 02:09:19
Okay. We’re just trying to keep it compact here. Because you’re saying a lot, you’re making good points, but—
Bryan 02:09:29
I’ll just list it and then we can look at it later. Data has latent etymological bias. We have to scrub the data. We could potentially send the models through counterfactuals, like perfect scientific counterfactuals, so that they have to counteract with science. We could attach counterfactuals to the vectors during pre-chain, if that’s possible. We could add extra empathetic data — so instead of adding more 4chan, we add more life skills and being a good person to the data itself.
We have to address brain-computer interfaces because there’s a possibility that could resolve the issue. That’s Kurzweil’s entire thing. We have to talk about Roko because even if that’s a 1% chance, that’s too much. We need to speed run alignment because—
Liron 02:10:20
We’re going to wrap the stream soon. I don’t disagree. You’re bringing up a lot of subjects. I just feel like this is a long conversation.
Bryan 02:10:31
Yeah, that’s what I’m saying. There’s a lot to talk about that we haven’t talked about. That’s what I’m saying.
Liron 02:10:37
Yes. Okay. All right. Sounds good. Thanks for coming by.
Yes, I’ll take the general point that there is a large universe of things to talk about. And I’ll yes-and and say, hey, they’re all reasons that we’re screwed. So even if you ignore all the reasons that we like to harp on — why we’re screwed, instrumental convergence, AI not having morality, or lack of knowledge — even if you remove all those from the picture, humans are just going to do something stupid. Just tell AI for fun to blow everything up.
There’s even other reasons, or some of what Bryan was alluding to — the AI is going to see bad characters in the data. Well, that’s the one that I don’t like when people talk about, blaming the doomers for writing bad characters. So let’s not talk about that one.
Ori 02:11:17
Well, I think he was talking about reasons that we’re not going to be screwed also, that maybe all of these things could be mitigated.
Liron 02:11:24
Yeah.
Liron 02:11:25
All right. Here, let’s take the last caller if they’re still... Oh, they’re still patient. I like the patient callers. All right, so we got Jack.
Jack, get ready. You got to hit the ground running. Don’t be surprised we’re letting you in.
Jack Joins: How Fast Will the Foom Be?
Jack 02:11:36
Hi.
Liron 02:11:39
Yeah, Jack. Got any video feed here? The viewers want video. This is a visual medium.
Jack 02:11:45
I don’t have my camera connected. That would probably take a few minutes. Is it okay if it’s just—
Liron 02:11:51
All right. No worries. So you got to win us over on personality.
Jack 02:11:57
Well, so I wanted to call in because — and I’ll try and keep it short. I work on a lot of deep learning projects, and I’m really focused on forecasting and researching recursive self-improvement, and that specifically more than probably anything.
And so we were having kind of a conversation in the chat and also, I think it was brought up a couple of times, but I was just curious — I think I heard you mention that you’re less certain or your probabilities might be shifting around how long foom will be, whether it’ll be a multi-year long foom versus a multi-week foom or something.
I’m just curious what both of your guys’ probabilities of a multi-week or multi-month foom are versus multi-year foom and recursive self-improvement.
Liron 02:12:58
Yeah, good question. I’m happy to give you a rough guess with wide confidence intervals.
I would bring in the thing that Steven Byrnes explained on our episode a few months ago, where he thinks there’s going to be a new regime where AI is going to work a little bit differently, and it’s not powered by this engine that predicts the next token. Because predicting the next token has taken us really far, and there’s a lot of ways to do it better and we’re making a lot of progress with this pre-training engine. But Steven Byrnes thinks that even the core engine is going to work a little bit differently.
There’s a new architecture that’s probably coming, which I think is the most likely option if I had to guess. And I suspect that the recursive self-improvement will also take a big leap when we get that next generation kind of AI.
I’m going to double down on the Yudkowskian worldview and say that the foom will be shockingly fast when we get this fundamentally next generation AI. I think there’s some patterns that support what I’m saying — like if you look at AI progress in the last three years compared to the three years from 2013 to 2016 versus 2023 to 2026, there’s a lot more AI progress in the recent window. No doubt about that.
And I suspect that there’s going to be even faster AI progress one way or the other. I think we’re scratching the surface here of how much headroom there is for AI progress. So I still feel a strong intuition that we’re still scratching the surface of how much improvement room there is. Even if you don’t call it recursive self-improvement, for one reason or another, there will be a lot of improvement getting unlocked in a way that’s discontinuous, or just the exponential continues, however you want to call it.
That’s what I think in terms of the qualitative magnitude of what’s coming.
Jack 02:14:39
Yeah.
Ori 02:14:40
Does that mean — I’m just curious now. So are you saying then that if we stay on this LLM paradigm, you’re like, “You know what? Maybe it’ll be safe”?
Are you coming out and saying that it’s not an autonomous, a rogue AI risk? There could be a misuse risk with the current AI systems clearly now, but the rogue AI— ## LLMs and Rogue AI Risk
Ori 02:15:01
The risk is kind of lower. So are you saying that LLMs aren’t a rogue AI risk?
Liron 02:15:07
Well, I’m not sure. I’m kind of 50/50. The longer we have LLMs being extremely impressive but still having this Achilles heel — every few minutes they still make a mistake — my day-to-day experience is I watch the LLM work. I’m like, “Wow, you nailed this. You really did all the work I asked.” But then, “What about this? Can I ask you a question about this?” And the AI’s like, “Oh yeah, you’re right. That makes sense.”
And it wasn’t that brilliant of a question. It was a pretty standard professional software engineer type question. And the AI agrees immediately. It’s like, wait — you clearly have broader skill than I have, and yet I’m still able to add a little bit of value. What’s up with that?
So as long as that’s still the case, every year that that’s still the case as the AI does more and more things, it makes me think maybe there is something with the paradigm that’ll always leave these little holes, and we need to shift to the next paradigm before it can be robustly autonomous.
But 50% chance I’m wrong, and it does just kind of fill the holes. So I hope that answers your question, Ori.
Ori 02:16:06
Yeah.
Jack 02:16:08
Yeah. I agree with a lot of that. I actually do also believe it’s going to be a pretty aggressive boom.
One of the intuitions — and also, sorry, let me know at any point if you need to go or end the stream, because I know this is something I’d love to talk about for five hours straight.
Ori 02:16:27
Come to Less Online. I feel like you might be into that.
Liron 02:16:31
Come to Less Online.
Jack 02:16:33
Yeah. That would be cool.
Well, there’s a lot of moving parts here, but I think that actually AI research is probably going to get easier increasingly. And I’m increasingly more convinced that there are actually a lot of paths to very powerful AI, and it’s more like reaching a threshold of criticality, like in a nuclear reactor, where you just get it to a level of robustness.
You used a word that I think you used a lot, Liron, which I really like — which is when you reach a threshold of robustness, you do get into that cyclical process. The flywheel starts going. And I’m less tied to the transformer versus some novel architecture, like test-time training, TTT type things. There’s a bunch of different ideas.
But I think increasingly it’s just more about getting past the threshold to where the AI is essentially competent enough to just fix itself almost right away, rewrite itself almost immediately. And then it really kicks off. That seems intuitively to me how it will play out.
Liron 02:18:00
Yeah. It’s an interesting question. It’s probably the last question, so it’s certainly interesting to me.
Jack 02:18:07
Yeah. But anyways, I agree with basically everything you’ve said on doom and everything, so I feel like there’s not really a point in me talking about that topic, but I’m very concerned as well.
Liron 02:18:19
Nice, man. And just to be clear, you are the great Jack-IIFI in the comments, right?
Jack 02:18:25
Yeah. I’m the person who had the name that’s in spirit with whatever that guy’s name was.
Ori 02:18:35
47f. Yeah.
Jack 02:18:36
Yeah. 47f.
Liron 02:18:37
Right. Exactly. Nice. Solid, man. Yeah, keep up the great commenting.
Jack 02:18:41
Thank you very much. Thanks for everything you guys are doing.
Liron 02:18:45
All right, man.
Ori 02:18:46
Thanks for joining. That was great.
Jack 02:18:48
Have a good day.
80,000 Hours, Inventing Erdős Problems & Holly Elmore’s Warning
Liron 02:18:48
All right, Ori. So yeah, before we wrap it up, I did just pull up a couple last tweets I want to end with.
Ori 02:18:53
Yeah, let’s hear. We’ve only gone through Twitter a little bit.
Liron 02:18:56
I know, right? Well, that’s good. We have so much to talk about. We’re never going to run out of ideas. All right. Here, I’ll share my window.
Can you see this one?
Ori 02:19:13
I see Gaz, 80,000 Hours.
Liron 02:19:16
Okay. There we go. So I want to put in a recommendation for this book from 80,000 Hours — their definitive explanation of how to use your career to try to force the world onto a better track. They’ve been thinking rationally about this question. I like to think that they acknowledge the urgency of AI careers, so let’s see what they have to say. I’ll play this.
Do you hear audio, Ori? Did you hear the tab audio or no?
Ori 02:19:39
Play it.
Liron 02:19:41
Okay.
Ori 02:19:43
No.
Liron 02:19:44
No? Okay, one second. Let me get that tab audio going.
[Audio clip from Rob Wiblin of 80,000 Hours plays]
“We have a new book out. It’s called ‘80,000 Hours: How to Have a Fulfilling Career That Does Good,’ written up by our founder, Benjamin Todd, and Penguin just published it around the world. A lot of listeners don’t actually know this, but 80,000 Hours has been running since 2012, trying to help people have a much larger social impact with their career while doing something that also delivers them, hopefully, a wonderful and well-rounded life.”
“Our latest editions naturally have a particular focus on AI, which is upending lots of people’s career plans and forcing a lot of people to rethink what they’re expecting to do. If you’re worried about AI yourself, the chapter covering which skills will be most valuable in the future goes through a lot of intuitive ideas, which I think might justify picking up the book just by themselves.”
Liron 02:21:18
All right. So yeah, it seems like a solid book. Haven’t flipped through it myself, but I’ve listened to a lot of their content and thought this was noteworthy.
80,000 Hours — it means you have 80,000 hours in your career. I would argue that’s an overestimate at this point for one reason or another, but they can call it 5,000 hours or whatever.
Ori 02:21:35
Yeah.
Liron 02:21:35
You ever check out their stuff, or are you—
Ori 02:21:38
I mean, I haven’t read it, no. You are very much an effective altruist. I will say, “Nope, not my thing.”
Liron 02:21:47
Okay. Yeah. I mean, a lot of people on the stream are trying to think how to make an impact before AI just does every job, or before everybody’s gone. So yeah, check it out.
Liron 02:21:57
All right, next thing. This is because a couple of funny and poignant thoughts stand out.
So Code Tarr on Twitter is saying, “Sure, they can solve Erdős problems, but can they invent new ones? Didn’t think so.”
So yeah, this is basically doing the David Deutsch move, where it’s like, okay, yeah, they solved an 80-year-old problem, but you know who’s the real genius? The guy who told them which problem to solve.
It’s like AI takes over the world — you know the real genius is the guy who said, “Take over the world.”
Ori 02:22:31
Right. It’s easy to identify these problems.
Liron 02:22:36
Well, I mean, Erdős was dribbling out problems. I asked GPT about this — yeah, Erdős, that was kind of his thing. He would love to talk with all these other mathematicians and pose problems.
Ori 02:22:45
Yeah.
Liron 02:22:46
But we were talking about P versus NP. A lot of interesting problems are actually — well, I don’t know if this has to do with posing the problem. P versus NP is saying, hey, NP problems, they’re easy to verify. They might be hard to solve. And then there’s the other end of the equation of, are they easy to pose?
You generally don’t think of posing a problem itself being a problem. Because if it was, you would just formalize it, and that would just be another problem. The problem posing problem.
Ori 02:23:15
No, I mean, you’re right. “Wow, it’s so hard to make a prompt.” Like, “Wow, make me $10,000.” And then the AI goes and makes $10,000. You’re giving all the credit to the guy who said, “Make me $10,000.” It’s the prompter.
Liron 02:23:27
Right. Exactly right. The prompter is such a genius. But as you know, this whole idea of prompt engineering has just gone down. It’s very rarely talked about anymore because you can just talk naturally and the AI will get you.
Ori 02:23:41
Dude. Pour one out for all the prompt engineers. They had careers for a year, and then another model comes.
Liron 02:23:47
Right, exactly. Prompt engineering used to be a future career, and now it’s dead on arrival.
Liron 02:23:53
Okay. Well, maybe we can wrap on this one. This tweet from Holly. She says, “My Twitter feed is desperate for AI to be a technical issue. They want to go through the solutions to the Erdős problems instead of deal with the real issue of letting this problem get worse by allowing bigger and bigger models to be built. Willfully missing the point.”
Holly has a lot of good points that don’t get said very often, because I am guilty as charged — I like analyzing the Erdős problems. I like coding with the models. I like getting excited. I like the horse race between Anthropic and OpenAI. I like trying to make money on calls when Google gets more valuable. I’m really enjoying the horse race, the Icarus flying closer to the sun. I’m enjoying my airspeed, my altitude here.
But Holly is not getting as much personal enjoyment, it seems, but I think she’s also correct in her perspective of: listen, there’s a long-term problem here that you guys are just not even addressing.
Ori 02:24:51
100%. I think she’s focused on the right thing. It’s what I’m concerned about also. How much time do we have?
Liron 02:25:01
Right. And it goes to what I was saying before. The guy — the gentleman who was here, who I had the long conversation with.
Ori 02:25:11
Alverin.
Liron 02:25:11
And made some good points. Alverin. Yeah, exactly.
So it gets to what I was telling him — we are not preparing the social technology to hit that stop button when it starts improving. We’re so focused on being like, “Oh, what does the improvement mean? Oh, we know it’s improving now. We have a benchmark that says it’s improving.” And meanwhile it’s like, okay, so what do we do? Keep improving it. Okay, now we’re dead.
Ori 02:25:30
Yeah, for real.
Ori 02:25:40
The benchmarking is a kind of a capability, so I’m confused why people are excited about benchmarking.
Liron 02:25:42
I wouldn’t go so far as to say that the benchmark is the capability, although it is true that if you make a benchmark, a lot of times you can train AI to it. That is kind of funny — if you have a better benchmark, that may potentially accelerate your capability development just by having the better benchmark.
Ori 02:25:54
Yeah.
Liron 02:25:55
That’s actually a very good point to bring up. But even if we grant that it’s not, or it’s only a slight help to increasing your capability — the benchmark — this is Holly’s point. I’m making her point for her because I mostly agree with it.
It’s giving you that feeling of, “I’m helping, I’m making a benchmark.” But as you saw with METR, Beth Barnes feels helpless. She’s like, “Look, I’m telling you there’s a benchmark and they’re running through it, but I’m not getting funded much to even make better benchmarks. I don’t feel like people are slowing down the way that the benchmark says that we should.”
The benchmark is kind of like a way to make people feel like they’re part of it. Like, yeah, you’re part of this picture. You have some power. You have some respect. But meanwhile, we’re here making money, raising money from our investors, feeling powerful, feeding our egos, and ending the world. That seems to be what’s happening right now.
Ori 02:26:39
Yeah. It’s a hard thing to talk about. It’s interesting to talk about what’s right in front of us, and the concern that you’re talking about — the nature of it is that there’s going to be a deception involved, and it also involves some kind of threshold change, which we’re not seeing right in front of us. So that makes this hard to talk about.
Liron 02:27:06
Exactly. So our friend Jack just commented. Jack double-I-F-I, accept no substitutes. He’s saying, “Benchmarks are like loss functions, but more general and with held-out validation sets.” Yeah, that’s basically what I was getting at. We’re now in an age where just imagine taking a fresh AI and showing it a benchmark for the first time — having the AI one-shot a new benchmark test. AIs can do that, as long as the benchmark is easy enough.
It looks at the definition of the benchmark and says, “Oh, you want me to do that? Okay, I will.” The benchmark is doing part of the work, which is posing the problem. You pose the problem, the AI will solve the problem. So there’s definitely something to that.
Benchmarks — it’s pointing the AI at interesting problems, and then by way of even trying to solve those, it can get some hints as to what to train itself on, what to spend the training cycles on, or what to spend the research and exploration cycles on just by seeing the benchmark. Seeing the target is surprisingly powerful.
Ori 02:28:00
Yeah. But Jack is at least—
Liron 02:28:04
Yeah, that’s interesting. It’s like he’s not on the screen, but I’m still talking to him via comments.
He’s saying, “And probably the model becomes the evaluator/validator in the end, which is RSI, but still lots of difficulties with robustness there.”
Yeah, so basically the benchmark people are part of the problem. I don’t know if I’d go that far, but I think the best takeaway is: if you are trying to help, if you’re early in your career, you’re looking at how to help the problem, and you’re like, “I know, I’ll join a benchmark org” — that feels like a good place to be.
I think Holly’s point, which I agree with, would be that we need people to attack the problem head-on. And the problem head-on is yelling that we need to pause, telling people that we’re doomed, doom debates. We’re part of the real solution, telling people that we’re doomed. But honestly, there needs to be a lot more of that. And when you’re just joining a benchmarking organization because it feels legit — Holly’s kind of trying to take an ax to the legitimacy of these organizations as tools of lowering P(Doom).
She would argue that benchmarks aren’t lowering your P(Doom). They’re doing things to hurt. They’re justifying AI progress as much as they are measuring it, or they’re enabling it because the same people who would’ve been like, “Shame on you,” are now saying, “Hey, I’m just here measuring you.” We just lost somebody who could’ve been yelling, “Shame on you.”
Ori 02:29:17
Yeah, totally. And you were talking about the social technology. It seems like the solution that we need is really the policy, the social technology in response to AI.
Liron 02:29:34
Yeah. Well, we’re social technology right here. You need a media arm that’s just yelling at people.
Ori 02:29:40
100%.
Wrap-Up
Liron 02:29:43
Exactly.
Liron 02:29:44
All right, guys. So yeah, we’ll wrap it up here. Next time you see us, I think we’ll be dropping another interesting intellectual episode in the early days of next week. I think that’s currently the plan. And then besides that, you will hear from us at Less Online. We’d love to see you guys in person.
Ori 02:29:58
Yes, hope to see you there.
Liron 02:30:00
Yep, all right. Bye, everybody. Have a good weekend.
Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.
Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏
