***ATTENTION: We’re looking for paid help at LessOnline from June 5-7. Apply now!***
OpenAI pushes the frontier of mathematics, Anthropic hires Andrej Karpathy to work on recursive self-improvement, live callers challenge the doom argument, and we've got a soundboard!
Doom Debates Paid Gig in Berkeley!
Doom Debates is proudly sponsoring The LessOnline conference this year and we got a merch table in the courtyard! We need 2 people to help represent the show — hand out T-shirts, talk to attendees, and spread the good word about Doom Debates. If you’re a regular viewer of the show, you probably know enough. Just be knowledgeable, friendly and helpful!
💰We pay you $500 to help out during the entire Fri-Sun event
🎟️ Plus you get FREE admission to the conference ($675 value)
This is best suited for someone local to the SF Bay Area or who can arrange accommodations. If you’re interested, send an email to internship@doomdebates.com and tell us a little about yourself and your availability for all 3 days. Thanks!
Links
Less Online conference (June 5–7, Berkeley) — https://less.online
Doom Debates Discord — https://discord.gg/g2X8h5UCrU
Liron’s hard question to Goertzel —
Eliezer’s job replacement progression —
Liron tags David Deutsch — “Is It New Knowledge?” —
Timothy Gowers on the Erdős breakthrough —
Nate Soares on Anthropic & recursive self-improvement —
Nan Ransohoff on the philanthropy tidal wave —
OpenAI — “An OpenAI model has disproved a central conjecture in discrete geometry” — https://openai.com/index/model-disproves-discrete-geometry-conjecture/
Nan Ransohoff — “The Third Wave of American Philanthropy” —
Rob Wiblin / 80,000 Hours — METR red team report breakdown —
Donate to Doom Debates — https://doomdebates.com/donate
Timestamps
00:00:00 — Welcome + New Soundboard!
00:09:27 — Doom Debates Is Sponsoring Less Online
00:17:30 — Ben Goertzel Episode Recap
00:27:30 — Goertzel's Alignment Blind Spot
00:34:21 — OpenAI Disproves an 80-Year-Old Erdős Conjecture
00:41:00 — David Deutsch Says "Amazing"
00:50:31 — Eliezer Yudkowsky's Job Replacement Progression
00:54:07 — Anthropic's $44 Billion Run Rate
01:07:45 — Schadenfreude Week: Liron Loses $10K
01:13:04 — Teaching Ezra Prediction Markets
01:18:11 — Caller Williawa: Is the Goal Engine Argument Effective?
01:37:00 — Caller Will: Brain Rot and Idiocracy
01:55:04 — Anthropic Hires Karpathy for Recursive Self-Improvement
01:57:16 — Rob Wiblin Says Recursive Self-Improvement Should be "Illegal"
02:01:37 — Caller Henry: The Chicken Riddle
02:08:41 — David Sacks Kills the AI Executive Order
02:11:20 — The Philanthropy Tidal Wave
02:15:56 — Donate to Doom Debates
Transcript
Welcome + New Soundboard!
Liron Shapira 00:00:08
That’s right, we got a soundboard. I gotta get a bleep. That’s another — I gotta bleep myself.
All right, we got a soundboard here.
Yeah, let’s see. What else we got here? You wouldn’t be a complete soundboard without Arnold Schwarzenegger.
Producer Ori 00:00:26
Get to the chopper.
Liron 00:00:31
Isn’t Arnold Schwarzenegger always a mainstay of these soundboards?
Ori 00:00:34
Oh, yeah, that’s true. That’s true.
Liron 00:00:38
When I was 14, I’d go on Kazaa or whatever — Kazaa, you know, programmed by our friend Jaan Tallinn.
Ori 00:00:47
Oh.
Liron 00:00:47
And I would get these prank phone call MP3s. It would be Arnold Schwarzenegger prank calling people. Somebody would use an Arnold Schwarzenegger soundboard to prank call people and be like, “Who is your daddy, and what does he do?”
Ori 00:01:00
Oh my God, I remember those. They’re so funny.
Liron 00:01:04
Yeah, good times. Early 2000s, man.
Ori 00:01:08
Yeah.
Liron 00:01:08
Watching those badly encoded Family Guy downloads. I don’t know what you were doing.
Ori 00:01:14
Yeah, those are early days of the internet. Prank calls. They’re funny. Gotta love that.
Liron 00:01:22
All right, let’s do an AV check here. So we should be live on YouTube. Let’s see if the crew is here. I need 28 people, okay? I miss the 28 people from last Friday.
I hope they’ll come back. Let’s see.
Ori 00:01:36
No chats yet.
Liron 00:01:38
No chats yet, okay. Hold on. Oh, God. Okay, so there’s a separate — in the first one. Okay, so there’s definitely, you know, Riverside has left some dog poop sitting around our YouTube, some episodes that are invalid, some corrupted episodes, so let me go clear those out.
Ori 00:01:55
Oh, okay.
Liron 00:01:55
Yeah, ‘cause Riverside was being buggy, basically, our streaming software, so one sec.
Ori 00:01:58
Oh, okay. Oh, here we go.
Hi, Liron and Ori.
Liron 00:02:05
Hi.
You know we got a soundboard?
Let’s see. Look at this, look at this.
That’s right. Ori doesn’t have it yet. Only I do.
Ori 00:02:19
Yeah. That was great. What was that, a whip?
Liron 00:02:24
Yeah, exactly. It was a whip. And let’s see what else we got here.
Ori 00:02:30
Oh, cha-ching.
Liron 00:02:31
Take that to the bank.
Ori 00:02:33
That’s every Casqui debate.
Liron 00:02:38
Yep. Cha-ching. We should do the cha-ching. And of course I’ve still got the old-fashioned sound effect as well.
We’re doing it manual here. Okay, so I see the YouTube. Helene is here. Griffin Detrick is here. All right, we got a first question. We’ll be getting to that soon.
Ori 00:02:55
Nice.
Liron 00:02:55
But we gotta get our bearings here. We gotta recap the week. Let me just make sure here.
Ori 00:03:03
Nice. These are — I feel like these are some questions hot off the Ben Goertzel episode.
Liron 00:03:07
Totally, yeah. All right, so I’m looking. Nice. It looks like we are in fact live on YouTube. And so if — I like to screen share. So I’m live on YouTube, live on Twitter. You guys know the drill. We’re gonna be going over the week. We’re gonna talk about what happened. This is a live call-in show. End of the month, we try to hear your voice, make your voice heard. This is your platform, okay? We’ll platform you here on Doom Debates if you have an unorthodox, a heterodox opinion.
Also, anybody, if you wanna jump in front of the queue, same rules as Eliezer. You can pay $10,000 and we’ll dedicate the whole episode to you.
Ori 00:03:40
That’s generous.
Liron 00:03:40
Ori and I were talking about making a GoFundMe. Yeah. We were talking about making a GoFundMe where we just raise $10,000 and we just get Eliezer back on the show, ‘cause it’s good content for us, and maybe other people wanna see it too.
Ori 00:03:51
Yeah. You just need a sacrificial lamb to debate him.
Liron 00:03:57
Right, exactly. Yeah, we need the next 47b. Somebody’s saying, “Any warning shot sound?” Yeah, hold on. I think I got one. I’m still getting the hang of the soundboard here. You guys will have to bear with me. I’ll be a soundboard pro before you know it. But, oh, here we go. Okay, you guys ready for some warning shots?
There it is. There it is.
Ori 00:04:18
That is a warning shot gun. That is a serious firearm right there.
Liron 00:04:23
So guess what? Last Sunday I actually went shooting, because in Upstate New York that’s actually something that guys do here. My friend took me shooting, and I fired a gun. I’m like, “Oh, this is so great. I’m gonna post this on Warning Shots. This is gonna be a real warning shot.”
And then John and Michael were like, “No, no, no, no, no. Delete that video.” Because think about it — you know Nerid Weissblatt, right?
Ori 00:04:48
Yeah.
Liron 00:04:50
The one who’s always — she’s probably here right now. She’s probably one of the 11 viewers.
Ori 00:04:50
She’s watching. Hey, Nerid.
Liron 00:04:52
Yeah, yeah. Nerid, hey, welcome to the show. We will not be showing the firearm video. But Nerid is gonna take the video of me at the firing range and then cut it with a, you know, terrorism at a concert or whatever —
Ori 00:05:07
Yeah.
Liron 00:05:07
And be like, “This is what the AI doomers are doing.”
Ori 00:05:09
Yeah, for sure. For sure.
Liron 00:05:10
Yeah. Exactly. So I’ve learned my lesson there. Poe’s Law is something that took many decades to sink in for me, ‘cause I used to post things and I’m like, “My humor’s good. People are gonna get my humor.”
And now, Holly Elmore was early to this idea that you just have to literally say what you mean with no humor. And if you can manage to be funny while saying exactly 100% what you mean explicitly, then great. But if you’re trying to use irony where you’re only saying the opposite of what you mean, then don’t do it. And I guess that’s why people invented the DM chats, right? That’s why people go on DM. They don’t post publicly.
Ori 00:05:43
Yeah. I mean, there’s some risks to humor. And now you’re a somewhat public figure, so anything you say, maybe people will try and use it against you.
Liron 00:05:54
Exactly. I’m a public figure with an enormous platform. Well, we’ll talk about that later, with the idea that platforms like this are actually poised to grow, ‘cause there’s a rising tide of prominence of the issue, and there’s a rising tide of money, right? There’s a torrent of money coming, so we’ll talk about that soon.
Ori 00:06:12
Oh, yeah.
Liron 00:06:12
But for now, yeah, in the meantime —
I was gonna do a sound effect. Damn. Okay. Well, the room is gone.
Ori 00:06:28
It’s the Mario coin. Look at all this money.
Liron 00:06:33
Right. Oh, here it is. Here it is. Okay. In the meantime, I was just gonna say — all right.
Ori 00:06:40
YouTube donations. You can make a YouTube donation.
Liron 00:06:41
Right. Exactly. Make your YouTube donations. I’m gonna set a goal here. The goal amount is we need 50 super chats, and each of them needs to be at least $100, and then we’ll extend the episode. Okay?
Here, I’m saying extend the episode. Start goal.
Okay. I know you guys are all getting your Anthropic secondaries here in the chat. I know you’re wasting time on Friday at Anthropic, so definitely we will take donations in equity, but I’ll sell the equity immediately. I don’t wanna be a schmo holding Anthropic equity. That would be a bad look.
Ori 00:07:20
Yeah. Hey, one thing about the AV is I think we should — you know how we put you on the left? You should do that. You adjusted the video.
Liron 00:07:27
All right. One sec. Yeah, you know, if you guys notice in the video, it’s kind of an Easter egg, but I’m always on the left because subliminally we want you to think that I’m your primary guide to the world of AI doom.
All right. Yeah, so let’s get to it. Okay, so first thing — hold on. Let me pull out the list of what we were gonna talk about here.
We’re gonna obviously get to the recap of the week. Oh, we gotta do Less Online, right? That’s number one.
Ori 00:08:00
There you go.
Liron 00:08:00
Okay. And also, now that I think about it, we gotta strategize the show because we talked about this live. For the recording, I guess it’s fine, but for the live audience, it’s a weird — this whole idea of burying the lede where you wanna get your best stuff up front at the very beginning.
Ori 00:08:13
Yeah. Yeah.
Liron 00:08:13
Well, if the live audience is trickling in, we actually get our highest audience halfway through, so it’s a weird format. I’m not used to it. You gotta drop the best stuff halfway through.
But I think what we should do is we should not bury the lede. We should start off with the best stuff, but then for the benefit of the live audience, we should just hit on the stuff again in the middle, and the recording audience won’t mind too much. So yeah, double tap.
Ori 00:08:34
Nice. This is sophisticated streaming etiquette. Yeah, strategy. I agree. I think start with the best, ‘cause you wanna have it be good for the on-demand listeners, so you should start with the best thing.
Liron 00:08:51
Yeah. You gotta please everybody in a subtle way. Have you heard about how Netflix specifically scripts their shows so that the characters are always explaining the plot and being super obvious because they’re purposefully catering to the audience of, “This is your TV in the kitchen while you’re cooking and you’re not paying attention”?
Ori 00:09:08
Okay. I didn’t know. Well, but that happens in every show, though. I don’t see how it’s so different.
Liron 00:09:14
Well, they’re making it extra obvious, right? They’re violating the rules of normal screenplays.
Ori 00:09:20
Oh, I see, I see.
Liron 00:09:26
Cool. All right. So yeah, let’s talk about Less Online for the benefit of the recorded audience, not the live people. The live people are all fashionably late, okay? We couldn’t care less about the people who are here right now at the early in the episode. You guys are not the intended audience for this, okay?
But for you guys on the recording, all right, just between me and you — Doom Debates is gonna be sponsoring Less Online in two weeks, June 5th through 7th. A really cool conference in Berkeley. It’s really the only conference that I plan to go to on a yearly basis. You — I may not go to any other conferences this year. We’ll see.
I know Producer Ori’s gonna go to a couple other conferences. So we’re gonna go to Less Online in Berkeley. It’s called Less Online ‘cause it’s kinda based on LessWrong. It’s put on by the people who run the LessWrong site. Lightcone Infrastructure is the organization. I think Oliver Habryka is the president and founder of that organization, and I know most of the people in the organization. Really cool people, really rational people, nice people.
So yeah, I’m a fan of what they do and their conference, and I consider myself part of the community. Okay, I’m a self-designated part of the community, and I have the karma to prove it. I have 4,000 karma on the site or whatever. And guess what? Doom Debates is an official sponsor, right? We can now announce.
Doom Debates Is Sponsoring Less Online
Ori 00:10:37
Slap.
Liron 00:10:37
The deal has been made. That’s right. Doom Debates will be sponsoring Less Online. We definitely negotiated hard to be the sponsor. There’s a lot of competition to sponsor Less Online.
And it’s kind of a sweetheart deal. It’s not like we have that much money to throw at this, right? I mean, we’re always asking for your guys’ money. We’re not wasting a ton of money sponsoring conferences. It was a relatively modest amount for a sponsorship.
But it’s a sweetheart deal because just like I appreciate LessWrong and Less Online, they have some appreciation for what we’re doing here at Doom Debates. They know that our values are consistent with what they’re trying to do. They know that we’re here to elevate the discourse. A lot of them happen to agree with our central claims that P(Doom) is high, and the ones that don’t think P(Doom) is super high, they still respect what we’re doing — look, it’s a good discourse, right?
So at least come for the high P(Doom), stay for the discourse. They get both prongs of the mission, and so that’s why we were able to find a mutually beneficial sponsorship arrangement.
And that brings us to our call for interns, or helpers, or whatever you wanna call it. Okay, we are offering, if you’re watching this stream right now — or I should say if you’re listening to the recording, right? That’s who I’m really talking to. If you’re here right now hearing my voice, you obviously have some affinity to the show and somewhat support the mission.
If you are thinking of going to Less Online or you think it’s interesting, check out this deal we can make you. We’re actually looking for two helpers Friday through Sunday, June 5th through 7th in Berkeley, California. If you can get there and handle your own accommodations, we will give you free admission to the whole conference if you’re our helper, and on top of that pay you — hold on a second — $500.
Ori 00:12:23
Let’s go. That’s money in the bank. That is money in the bank. Come on, do it for the dogs!
Liron 00:12:34
Yeah, so we’re offering $500 plus free admission to the conference. I mean, I think it’s a no-brainer. Honestly, I do think there are some of you who are like, “Man, go to Less Online, I looked at that conference and it looks interesting, but where am I gonna get $675?” Right? And we’ve solved that problem for you. We’re gonna get you free admission, and don’t forget about the $500. That’s just an extra bonus.
Ori 00:12:54
Can —
Liron 00:12:54
So, you know, I would argue it’s overdetermined that you should do this. Yeah, go ahead.
Ori 00:12:57
Yeah, and can we hype up who’s gonna be there, too? I mean, consider some of these names, okay? Eliezer Yudkowsky. Zvi Mowshowitz. Zvi Mowshowitz. Daniel Kokotajlo.
Robin Hanson.
And did I say Scott Alexander?
The one and only Scott Alexander.
Liron 00:13:24
All right, all right. Yeah, no, these are very high-quality people. And you know, of course we’ll be there, too. You guys already know us, but —
Ori 00:13:33
Yeah.
Liron 00:13:36
It’s something, you know? Yeah. So we’re gonna have a merch table, and you’re gonna be helping us staff the table, giving out the T-shirts. We should get sweatshirts, too. Hmm.
Ori 00:13:45
Hmm.
Liron 00:13:45
I don’t know, sweatshirts are more expensive, so we’ll have to think about how to budget this, okay? My family is starving. Help me budget this.
Ori 00:13:51
Does the sweatshirt have the P(Doom) thing on the back?
Liron 00:13:55
Yeah. Check this out, okay? Fashion, I’m gonna fashion model it for you. Here we go.
Ori 00:14:00
Oh, snap. It’s on the back, too.
Liron 00:14:03
Uh-huh. Uh-huh.
Ori 00:14:04
Hell yeah.
Liron 00:14:04
We should get the P(Doom) theme as a sound effect, you know? We’re really upping our game here.
Ori 00:14:10
Ooh. Oh, yeah. That’s great. Yeah, for sure.
Liron 00:14:12
Yeah. It’s okay. I’ll do it acapella. P(Doom). There.
Ori 00:14:18
There.
Liron 00:14:22
All right.
Yeah, so hope you guys will join us at Less Online. If you don’t wanna volunteer and be our wage slave for the whole conference, that’s totally fine. We encourage you to just go to less.online and come to the conference anyway and say hi to us and be independent. But then we also encourage you to wear our T-shirt. So yeah, great conference.
Hit us up if you have any questions or you’re considering coming, and yeah, hope to talk to you. Anything else to say about Less Online, Ori?
Ori 00:14:50
Yeah, if you’re interested in doing the paid gig, then email internship@doomdebates.com.
Liron 00:15:00
Yes. You can email internship@doomdebates.com. You can also talk to us in the show Discord, which is doomdebates.com/discord. That’s where the Discord is.
There’s something else we can say about Less Online, okay? Which is we are gonna have a live taping. Oh, yeah, soundboard. Yeah, I’m building habits here. Live taping.
It’s gonna be on the main stage at Less Online. The guest is to be announced. At Manifest Conference last year, I did a Liron versus everybody, so there was no guest. It was literally just me debating the entire audience, where I would be like, “Okay, raise your hand if you have this position,” and then I would call on people to make the argument on behalf of everybody with that position.
So we might do that, or we might have a special guest. We’re currently recruiting. So if you’re listening to this and you’re coming to Less Online and you wanna debate me live or debate somebody else live, definitely hit me up, ‘cause it’s currently to be determined. So that is one reason to come to Less Online. That is one reason to help out, is because we’re doing this cool program on the stage and we’re giving merch. All right. I think we’ve said all there is to say about Less Online.
Ori 00:16:13
Yep. It’s gonna be a good time. I mean, and it’s so soon.
Liron 00:16:15
Yeah.
Ori 00:16:16
Just two weeks. So you gotta get those applications in if you’re interested in going.
Liron 00:16:20
Exactly. It’s a great time. When are you gonna go June 5th to 7th to go somewhere to have a great time? Were you gonna go to Miami or something? No. Hell no. Go to Less Online. All right.
Ori 00:16:31
Yeah.
Liron 00:16:32
Um —
Ori 00:16:32
It is genuinely — the last thing I think that’s worth mentioning is that I do genuinely think it is a great event. I mean, it’s at an awesome venue, and the people are really smart, and it’s welcome to, you know, nerds and rationalists and normies.
Liron 00:16:52
That’s right. Rationalists and normies alike. Correct. It’s open to everybody on the entire autism spectrum.
And you know, by spectrum, I also mean not having it at all, right? So normies, that’s also a point on the spectrum.
Liron 00:17:14
Okay. Let’s move on here. We’re gonna do episode recaps and tweets. All right, so episode recaps. We started off the week with a real banger. Started off with a bang. It was Ben Goertzel.
Ben Goertzel Episode Recap
Ori 00:17:32
Yes.
Liron 00:17:32
He was definitely one of the most sought-after guests. And I guess we can screen share. We can read off some of the comments, because this is definitely a high-performing episode. It’s getting plenty of views and comments.
Ori 00:17:44
Yeah.
Liron 00:17:44
So you know it’s good. All right, let’s get the screen share going. One sec.
See, this is why Ori needs a soundboard, so he could be entertaining you guys during this.
Ori 00:17:54
Yeah, I know. It’s a last-second addition. I don’t have the soundboard. I myself am a human soundboard.
Liron 00:18:01
Yeah, Ori vamp.
Ori 00:18:02
Vamp. Okay.
Liron 00:18:02
Exactly. Ori, show us some of your tweets, too. I want — you should load up your browser. We really wanna see.
Ori 00:18:09
My favorites? They’re all, right now, this week, it’s all Ashley St. Clair gossiping about Elon. I just cannot get enough —
Liron 00:18:15
Oh, okay. I saw a little bit of that, yeah.
Ori 00:18:17
Yeah.
Liron 00:18:17
Wasn’t there a rumor that he impregnated an Asian person?
Ori 00:18:22
Oh, that could be. He now has children with, in the ballpark of 10 women. I mean, so why not throw another one in there?
Liron 00:18:35
Exactly. All right. So we’re gonna pull up the Ben Goertzel episode. Talking friendly to us. There we go.
Ori 00:18:43
Nice. We could hear the audio, by the way. I heard that.
Liron 00:18:44
You can? You can or you can’t?
Ori 00:18:48
I did, I heard it.
Liron 00:18:48
You — oh, you did. Okay, great. All right, so here’s the screen. Let me try to make it reasonable dimensions.
All right, so let’s see what viewers are saying. Let me give my own recap of the episode, okay?
Ori 00:18:59
Yeah.
Liron 00:18:59
I’ll be honest, between you and me on Friday, okay, I don’t say this during the episode, but I definitely disagree with his main claims. Is that fair to say?
Ori 00:19:09
I mean, it’s fair to say. It’s what you believe, so I don’t think it’s so wrong of you to say it.
Liron 00:19:18
Right. I mean, it’s just — he’s, I mean, now you know the history with Eliezer. We didn’t get into this too much in the episode, but he was one of the only people thinking about AGI back in the day, decades ago, and Eliezer was too. And it really was slim pickings in terms of who’s even thinking about this.
And there was a time when Eliezer thought that he was just gonna build AGI, and of course it’s gonna be good because the singularity’s awesome, right? Which was my default position before I learned about all the safety stuff.
And then Eliezer had this realization of, “Oh, this is probably gonna go bad. We don’t know how to align this thing, and it’s just gonna take over, and then we’re gonna be dead,” which is what a lot of us believe now. And so he basically tried to convince Ben Goertzel of that, and Ben is like, “No, man. Yeah, I don’t believe you, man.”
And so then they split off. And also, I don’t think that his research led anywhere from Eliezer’s perspective. I don’t wanna speak for Eliezer, and obviously Ben’s research is still going, but I think — put it this way. Whatever value they used to find in his research, they split ways, and so they’re not aligned, right? However you think that’s the case, he’s not part of MIRI now.
So now he just debated me, ‘cause I wanted to take his temperature and I wanted to see where he’s at. And from my perspective, I wouldn’t want him to be part of MIRI because I think he’s largely dismissing the urgency of the AI safety issue, right?
Ori 00:20:34
Fair. Yeah.
Liron 00:20:37
Yeah. I don’t wanna go all the way and say he’s a total doom dismisser, but he’s close.
Ori 00:20:43
Hmm.
Yeah. It’s strange the difference between the in-the-moment reaction and the holistic view. So it’s like, you got really specific with him and you asked him, you know, on the spectrum of orthogonality thesis, you agree with it in theory. And he’s like, “Yeah, yeah, I agree with it in theory. Yes, AI could be very powerful.”
But then on his holistic view, he’ll agree with you on those points, but holistically it’s like, “You know, no, I’m not too worried about it.”
Liron 00:21:15
Right. I mean, he’s managed to acknowledge some stops on the doom train. He rode with me past the stop of, “Oh yeah, it’s gonna be really powerful really soon.” But then the moment we started talking about, “Okay, well, don’t you think we’re just gonna have one that goes out of control?” He’s like, “No, I don’t think that.”
And he used the phrase like, it’s a narrow-minded or immature perspective or whatever, so definitely a big divergence. And then he even mentioned, “Oh, it’s gonna have this drive to transcend itself.” And I’m like, I’m not really seeing transcendence as a universal drive the way Ben does, right? So we were two very different people, different mindsets I guess.
Ori 00:21:48
Yeah.
Liron 00:21:48
And the fact that he’s built an AI really quick, and he admits to not being super focused on safety yet. So that’s — anyway, that’s my takeaway from it. I’m like, okay, well, I’m glad we got it out there.
As Zvi Mowshowitz says, “Sir, can you please speak that into the microphone?” I feel like we got him to speak into the microphone what his current opinion is, and you can take it or leave it.
I think that there was a skew in the comments as expected, where people mostly were on my team being like, “Okay, I don’t agree with this.” But there were certainly some people — I mean, there were a lot of reactions like, “Hey, he seems like a nice guy.” There were a lot of, I definitely know people who are like, “Man, I really dig his vibe so much. He’s a great guy.” So you know, at least he’s got that going.
Ori 00:22:36
Yeah. I think it’s interesting the — what I think is interesting is just that he’s still working on his kind of pet project to build AGI, right? And —
Liron 00:22:49
Right.
Ori 00:22:49
I mean, honestly it doesn’t seem like they have that much traction. He talked a little bit about the traction, but he’s been working on different angles for a while.
And yet his views are pretty important. Here we are now in the time of AGI because Ilya Sutskever and Transformers have found a pretty —
Liron 00:23:08
Talking friendly to us.
Ori 00:23:09
They found a pretty good approach, and it’s like we’re getting closer to this intelligence takeoff, right? And it’s like the philosophy really applies.
So all the strategy and philosophy that he’s thinking about is relevant. It just so happens that the technology approach that he’s taking is maybe not the one that’s going to build the AGI. So I just find that interesting because the way he’s talking now, he could’ve been saying the exact same thing 10 years ago. If Transformers were never invented, it would be the same conversation with Ben Goertzel, you know?
So the alignment philosophy and all of that — 20 years ago, the split between him and Yudkowsky, it’s like, oh, who cares? These guys are just off in fantasy land. Who cares about this rift that they have.
Liron 00:24:04
Mm-hmm.
Ori 00:24:04
And yet now here we are 20 years later and the actual technology is there, and the actual beliefs matter. So it’s important to hear what he has to say about it.
Liron 00:24:20
Yeah, fair enough. And you could argue Eliezer is also here saying the exact same stuff, right? So everybody accuses everybody of not updating on evidence. There’s that dynamic to it as well.
Ori 00:24:29
Yeah.
Liron 00:24:32
Yeah. All right, so the commenters are saying — let’s see, the top comments here. Sort by top.
“My friend just asked me if Ben was Ray Kurzweil’s dad. Hell, yes, Goertzel, let’s go. This guy is a good man. Keep them coming. You are doing good work.” All right, so in terms of substantive comments, let’s see.
“I cannot understand these AGI accelerationists. They are out there thinking these alien wombats are gonna be just peachy.” Okay. I agree with that.
Let’s see. “I like how he says ‘humans’ the same way as Yudkowsky.” Yeah, got a point there.
“I notice he called Eliezer ‘Ellie.’” I feel like that’s an affectionate nickname. Nobody else is allowed to call him that.
Ori 00:25:12
It sounded like that, yeah.
Liron 00:25:14
Yeah. Let’s see.
All right, people are commenting that my expression was too distasteful. I wasn’t poker-faced enough. Okay, fair enough.
Ori 00:25:26
That was a funny one. You never get that comment.
Liron 00:25:32
Yeah. I mean, he did talk for a long time, right? Part of it is honestly I do get bored, ‘cause I like to listen to podcasts at 3X speed, and that is part of my job that — that’s the worst part of my job as a host, is that when the guest is talking a long time at 1X, I do get bored, okay? To be honest.
Ori 00:25:49
Whoa.
Liron 00:25:50
I’m instituting a rule, okay? If you wanna be a good guest, you really gotta keep your part of the conversation to 90 seconds, right? You really shouldn’t be talking for more than 90 seconds. Ask the host a question, okay? I feel like that’s reasonable.
Ori 00:26:04
Or — but if they’re a fast talker, then you might be cool with it.
Liron 00:26:09
Yeah, I guess. But then you don’t need to say as much, right? You can talk shorter ‘cause you’re a fast talker.
Liron 00:26:16
Okay. Yeah, so that’s Goertzel. Let’s see if the live commenters are saying — our valued live commenters, our favorite people here that we’re definitely talking to — they’re saying, “What do you think? Are we gonna get one solid disease cure before we all die?” Yeah, good question.
All right. So I think we’re done with Goertzel. And then later this week, yesterday we dropped a 2024 episode that we are reposting, about Martin Casado. We can go to that —
Ori 00:26:44
Yeah.
Liron 00:26:45
I actually think that this is an important episode. I feel like it’s underrated. Yeah, go ahead.
Ori 00:26:49
Well, going back to Ben Goertzel, I actually think that — okay, so you disagree with him, but —
Liron 00:26:55
Yeah.
Ori 00:26:55
I actually thought it was kind of subtle where the difference was or what you thought was, you know, where you thought he was putting his cards on the table.
And I guess you’ve done a few tweets now to show, okay, this is where I disagree with Ben Goertzel. So I think it’d be interesting to talk about that for a second. What was it that he said that caught your —
Liron 00:27:21
Okay, the exact disagreement of the argument?
Ori 00:27:21
Yeah, what was it that he said that caught your attention?
Liron 00:27:22
Okay, you’re right. I mean, I didn’t say it right now. Yeah, okay, sure. Well, you’re right. We should just go do the clips, right? Because I pulled out moments that I thought were significant, and I posted them on Twitter, so let’s see.
Goertzel’s Alignment Blind Spot
Ori 00:27:31
Right. Right.
Liron 00:27:31
All right, one second. And that’s a brilliant segue, Ori, ‘cause we’re about to dive into Twitter. That’s a big mainstay of these kind of shows. So all right, I’m pulling up my bookmarks. One sec.
Ori 00:27:40
Give the people what they want. Give me the Twitter feed. I’m fiending for it.
Liron 00:27:47
Yeah, exactly. All right, hold on a second. Yeah, and I tweeted — I’ll share the tab in a second. Here we go.
So I got a caption. I thought this caption had viral potential, which is, “Doom Debates asks hard questions like,” and then my question is, “When you have your project to build AGI, at what point are you not just maximizing capabilities but making a trade-off to get more alignment instead?” And then I said, “Other shows, feel free to do the same.”
So in my mind, this was a mic drop moment. I don’t have a mic drop sound effect. What’s the closest thing to a mic drop sound on here?
Ori 00:28:20
Just literally hit the mic.
Liron 00:28:23
Yeah. I’m realizing we need so many different sound effects here. But what about this? No.
Ori 00:28:32
Yeah, yeah. That’s good.
Liron 00:28:32
Okay. Whatever. I thought it was a mic drop moment because it is kinda ridiculous how nobody asks the hard questions, right? Nobody gets somebody prominent on their show and then hits them with a hard question. That’s not really done in our society.
Ori 00:28:50
It’s a great question.
Liron 00:28:51
Yeah. Right. Everybody’s just, “Hey, welcome to my interview.”
Ori 00:28:54
Let’s hear what he has to say.
Liron 00:28:54
Yeah. All right. So here’s the actual hard question and his response, and you guys can judge it for yourself.
So if I just take the last thing you’re saying now of, “Hey, when we build super intelligent AI, we should really try to put caring for humanity or caring for sentient life as being near the center of its goal systems” —
Ori 00:29:13
Yes.
Liron 00:29:13
When you have your project to build AGI, at what point are you not just maximizing capabilities, but making a trade-off to get more alignment instead?
Ben Goertzel 00:29:28
So I would say we haven’t hit that point yet in my own projects, but it’s not hard to see how that would happen. Because we already have early experiments we’re running now where misalignment of various sorts happens, and then we have to sort of rein experimental agents in and reboot and realign.
We’re gonna get a practical sense of how to manage it during a pre-AGI phase, and I think that’s sort of how it has to be. I don’t think you’re gonna be able to solve that problem by theory alone any more than they could solve the problem of how to make a scalable transformer by theory alone, right? It’s AI developing by —
Liron 00:30:34
What?
Ben 00:30:34
Yeah, it’s developing by a mix of theory and practice.
Liron 00:30:38
Common thing we hear that you gotta launch these AIs and have their capabilities grow in order to effectively have a feedback loop that can help with alignment, right?
Ben 00:30:48
Yeah, but I mean, I didn’t want it to be that way. I spent a lot of time last year trying to do an in-depth mathematical analysis of self-modification and the stability of goals under self-modification, and I think I made a lot of progress there.
But what I found is you can make more progress mathematically under assumptions of very rich resources. When you get closer to super intelligence, it becomes more tractable from a math standpoint. And MIRI, Eliezer Yudkowsky’s group, kind of found the same thing. They did a bunch of nice papers about the AIXI type stuff, the case where you’re near infinite compute size. And you can do more nice theory there, but the nice theory doesn’t seem to tell you that much about systems that are just barely big enough to do general intelligence.
There it’s very messy, and it seems like you’re figuring things out through a mix of theory and practice. And that’s not satisfying to me intellectually. It just seems to be the real situation.
Liron 00:32:06
All right.
Ori 00:32:08
Yeah, I mean — it’s pretty damning that he’s like, “Okay, we tried it in theory, and the theory doesn’t give us very much. So just build it, I guess.” I mean, that’s why you think it’s so damning.
Liron 00:32:20
Right. Yeah, what he said is he’s saying, “Hey, the end game is — it’s easy to predict that it’s gonna be coherent in the end game,” or “I have math to show that,” which is true. Utility theory says when you have a utility function, your utility function says to preserve your utility function. Goals do tend to be stable in the limit. I believe that.
And he’s saying, “Yeah, we don’t really have math for the current situation.” And I’m like, right, so it’s probably gonna converge on unaligned goals, right? That’s the problem, and I think he agrees that that is a risk. He’s just more optimistic that somehow it won’t happen.
Ori 00:32:55
Hmm. Why is it more likely to converge on unaligned goals?
Liron 00:32:56
Well, the whole premise of “if anyone builds it, everyone dies” is we don’t have the tools to make it converge to aligned goals. And I think a lot of people will admit, “Yeah, we’ll just keep iterating. We’ll just keep trying, and if we see a problem, we’ll try to stop it and fix it.”
Which, when you dig in, you’re like, wait, are you gonna always be able to stop it and fix it, right? I feel like that’s the part that people are hand-waving about — “Oh, well, maybe we can.” We’re talking about getting into a messy situation in terms of predicting what stopping will look like. We’re starting to lose theory about what it means to stop an AI.
And it’s not intuitive yet because when you use Claude Code, there’s an obvious stop button, right? It’s not intuitive where Claude Code is gonna spin up an independent virus sub-agent. It spins up sub-agents, but they all respond to a stop command. So people don’t get how we’re about to lose that familiar property.
Ori 00:33:46
Yeah.
Liron 00:33:50
Yeah.
Ori 00:33:51
Yeah. Great question. Great exchange. Good to, like you said, good to get those views on the record.
Liron 00:33:58
Yeah. All right. Let’s go to Twitter ‘cause we got a bunch of bookmarks. As Zvi Mowshowitz pointed out, you might consider this a relatively uneventful week because all we got was a frontier progress in math proving. That’s pretty much all that. It’s not like there was a new model drop or anything. It was just proving something that humans weren’t able to prove for 100 years, so slow week.
OpenAI Disproves an 80-Year-Old Erdős Conjecture
Ori 00:34:22
Wait, wasn’t there also a model drop? All the new Gemini releases?
Liron 00:34:27
Oh, yeah, I forgot. I was like, “There’s no model drop. Oh, wait, there was one.” Yeah, Gemini 2.5 Flash, which is actually on some dimensions — there’s a little bit of doubt, but it’s arguably on the Pareto frontier of being slightly more cost efficient and somewhat more speed efficient for the same performance for some use cases.
So you’re right. No, we did get a model drop and an increase in the Pareto frontier. So I’ve just been, you know, my dopamine neurons have just been completely wiped out and unable to perceive that amount of progress anymore.
Ori 00:34:57
Right.
Liron 00:34:59
Yeah, so that’s as good a starting point as any, though. Let’s head over to OpenAI’s site where they talk about their math breakthrough because this was a big deal. I talked about it on Warning Shots. It’s gonna be dropping this weekend. Let me share the tab. All right, how are we doing with the screen share? One sec.
Ori 00:35:17
It’s just breakthrough math, Liron. Come on.
Liron 00:35:21
Yeah, exactly, right?
Ori 00:35:21
What’s the big deal?
Liron 00:35:21
I mean, that’s what Gary Marcus would say.
Ori 00:35:24
Is he saying that?
Liron 00:35:25
Yeah, he was somewhat downplaying it. He’s like, “I’m skeptical this was really LLMs.” But actually, I think Gary Marcus’s position is a bit hard to pin down, but my best guess, if I were to have him back on the show and be like, “So what’s your prediction here?” He’d probably be like, “Well, my prediction is just that we’re going to inject neurosymbolic AI into these LLMs, and so maybe the timeline will be kind of fast.”
He was a little bit skeptical on timelines, but he was willing to acknowledge a one-third probability that timelines would be fast. That’s what I remember when he came on Doom Debates, which, good for him coming on Doom Debates.
Ori 00:35:59
Yeah.
Liron 00:35:59
Yeah, so Gary Marcus is a friend of the show, so I’m not in the business of dunking on him. I will acknowledge that other people find him to be a good target for dunking. But when he came on the show, he set his position, and I think it was kind of a position that doesn’t make super strong predictions.
He has made a few predictions, and some of his predictions have come true. So he’s kind of a hard one to call him out as being like, “You were so confident and so wrong about this,” because he does have ways that he can say, “No, I wasn’t,” and it’s not super cut and dry. So that’s pretty much all I have to say about him.
Ori 00:36:41
Yeah.
Liron 00:36:41
All right. And I respect him for coming on the show. So here’s OpenAI’s website. Let’s see how they’re publicizing their recent math breakthrough. So where is it? It should be here. You’d think they would tout it. Is it already scrolled off the front page? Math breakthrough scrolled off the front page. Let’s see.
No, it should not scroll off the front page. Maybe it’s in their separate research blog.
Ori 00:37:07
Wait, I think I saw it. It’s the third item.
Liron 00:37:08
Third item. Let’s see. Okay, find the math breakthrough here.
Ori 00:37:15
I know. Advancing voice.
Liron 00:37:15
Advancing voice, no. Content provenance, personal finance. I mean, it kind of makes sense that on the homepage they’re only putting things that are relevant to the average consumer, right? So maybe — here, OpenAI research, right? So it’d maybe be under the research section.
Okay, “An OpenAI model has disproved a central conjecture.” So right, disproved the conjecture. I think most smart mathematicians were agreeing with Erdős’ original conjecture here, and ChatGPT came in and is like, “Guess what? I’m disproving the conjecture.”
Which I think is rare for conjectures to get disproven. I feel like they’re more often proven than disproven.
Ori 00:37:52
Interesting.
Liron 00:37:54
Yeah, here we go. “For nearly 80 years, mathematicians have studied a deceptively simple question. If you place N points in the plane, how many pairs of points can be exactly distance one apart?”
Right, when I first thought of this question, I’m like, “Shit, how do you...” I didn’t have much intuition about this question. My intuition is just like, I don’t know, don’t you just put them in a square and it is what it is? There’s not much you can do, right? That was my first reaction. Do you have any thoughts on this, Ori? Any mathematical thoughts?
Ori 00:38:21
No, absolutely not. I mean...
Liron 00:38:24
Yeah. Sorry, I’m tapped out. My intuition is tapped out. This is why I didn’t go into math. But let me see.
I wanted to — I think on this page, they’ve got a little visual we can look at. Let’s see, what do we got here? Oh, here we go. Okay, so the proof is here. You can find an abridged version of the chain of thought here. Okay, previously known construction of many unit distances from Reisner. All right, let’s try to do this.
Nobody’s drilling down like this. This is why you come to this show, because it’s 10 IQ points above your average YouTube pundit. Okay, so this is supposed to be...
I’m just confused as to why there would be lines — there seems to be a bunch of lines crossing over stuff, but I don’t get how the lines can cross if points are one unit distance apart, right? Aren’t the lines all supposed to be one unit distance? Are the points less — oh, maybe the points are less than a unit distance?
Okay, I thought this would be easier.
I don’t know. I thought I would know how far away the points are. I don’t even know that. All right. Commenters, if you understand how far away these points in this grid are supposed to be, let me know.
Ori 00:39:32
Are the points being connected there? There are lines between the points? That’s the idea?
Liron 00:39:33
Right. I think the idea is you’re trying to maximize how many unit lines there are, but I just don’t know if the points represent units or the lines represent units. So I got nothing for you guys here.
I hope 3Blue1Brown takes this on. But anyway, whatever the challenge is, you can see there’s been improvements. Erdős unit distance problem accuracy at test time.
I don’t have enough context about this graph. Okay, I pulled up an article that was a little too hard to understand in real time, sorry. But yeah, I encourage — we’ll get back to it next week when I’ve watched enough explainers. We’ll bring you guys more context.
Ori 00:40:07
Well, what about the Tim Gowers video that you tweeted? That’s a pretty good explanation, and you got the expert being like, “This is a tough problem,” you know?
Liron 00:40:17
Okay, yeah. We’ll go find it. Let’s see if it’s in my Twitter bookmarks here. Give me a second.
Ori 00:40:30
I mean, I think you tweeted it related to your take on one of the famous people who always comes up on here, David Deutsch.
Liron 00:40:42
Oh, right, right, right. That’s when I tweeted it, okay.
Ori 00:40:44
Yeah.
Liron 00:40:44
Let’s see. Where is it? Oh, here we go. Yeah, so that’s my usual challenge, okay? So let’s get in the brawl here. So instead of talking about the math, let’s just say that we challenged David Deutsch, and I said, “Let’s play Is It New Knowledge?”
Dun-dun-dun with David Deutsch, right? Because he has this whole thing — if you go search “Doom Debates, Eliezer Yudkowskian versus David Deutschian debate,” ‘cause we only — it was a proxy debate. We couldn’t get the real people on there, so it was just me versus Brett Hall. No disrespect to Brett.
But yeah, we debated over whether AI is creating new knowledge, and he took the Deutschian position that no, they totally don’t. And so here we have a math proof, right? A frontier math proof that no human had managed to do for 100 years. How old are the Erdős problems? Let me ask ChatGPT here.
How old are the Erdős problems? I think they’re 100 years old. I know there’s the Hilbert problems as well. Those are from 1900, I think.
David Deutsch Says “Amazing”
Ori 00:41:53
This question from the tweet, it says 1946, so 80 years.
Liron 00:41:57
Oh, 1946. Okay. Yeah, so almost 80 years. These are super old problems, right? So it solved an 80-year-old problem, where if you’d asked a human, I don’t think the human would’ve been like, “Oh yeah, I just need a tiny bit of help. I almost got this.” I think the human would be like, “Yeah, I’m pretty stumped about this.”
And it put a bunch of pieces together in a way that humans really hadn’t thought of before. That’s what I’m hearing from people who are tweeting about this.
Which I’m not surprised. My day-to-day experience working with AI as a coworker when I’m solving problems in my code base, right, as a programmer, is that it’s always putting pieces together.
I’ve said this before, I have this experience where I try to think through the pieces and I’m like, “I’m stuck.” And then instead of me trying to think more, I’m like, “I’m done thinking. From now on, you’re gonna take over as the thinker. Figure it out.” And then it says something, I’m like, “I don’t think you’re right. Why don’t you try thinking again?” That’s what I’ve become, okay? I’ve totally outsourced this whole idea of thinking.
Ori 00:42:48
Wild.
Liron 00:42:51
Yeah. So I’m not surprised that it thought all the way to a frontier proof.
Ori 00:42:55
Yeah, I mean, did you look at the chain of thought? Did you look at that document?
Liron 00:43:00
I didn’t look at the raw — I looked at some people commenting on it and being like, “Yep, this is legitimately brilliant.” If a student turned this in, I wouldn’t see any problem. I’d be like, “This is good work.”
Ori 00:43:09
It was 100 — I mean, I didn’t read it that closely. I scanned it, but —
Liron 00:43:14
Yeah. Here, Postmodern in the comments is saying, “Just the chain of thought is 150 pages.”
Ori 00:43:17
Yeah, 150 pages and it’s very sophisticated. Like, “Well, maybe I should try to approach it from this theory,” and if you — the way the problem is posed, then this mathematical theory makes sense. I mean, it’s impressive chain of thought.
Liron 00:43:33
Yeah. Wow. Crazy. Okay.
So Timothy Gowers is one of the world’s top mathematicians who likes to report on this kind of thing. He’s saying, “If you’re a mathematician, then you may wanna make sure you are sitting down before reading further. AI has now solved a major open problem, one of the best known Erdős problems, called the unit distance problem, one of Erdős’ favorite questions, and one that many mathematicians had tried.” Right? They’d already tried their hand at it. That’s what’s crazy.
“The relationship between this and the math result I previously tweeted about reminds me of the passage from Beatrix Potter’s The Tale of Jeremy Fisher.” So he’s got an excerpt. “And while Mr. Jeremy sat disconsolately on the edge of his boat, sucking his sore fingers and peering down into the water, a much worse thing happened. A really frightful thing it would’ve been if Mr. Jeremy had not been wearing a macintosh.” All right. Not sure what he’s talking about there.
But then he says, “I and several other mathematicians were asked to reflect on the result and what it might mean for mathematics. The resulting paper, which also discusses the mathematics, can be found linked to from OpenAI’s blog post. So I won’t say any more here, but I may respond to comments.”
Yeah, he’s saying this is legit. Oh — David Deutsch is in here. He’s commented, “Amazing.” Come on. You’re gonna comment on this —
Ori 00:44:50
What?
Liron 00:44:50
And you’re not gonna respond to me tagging you specifically asking you if it’s new knowledge? Okay. So we gotta decode this, guys. So David, what does this mean if David Deutsch is saying “amazing”? Does that mean it’s new knowledge? ## David Deutsch’s Response
Liron 00:45:04
Come on, David Deutsch.
Yeah, he’s gotta clarify. We’ve all been waiting for him to tell us whether AIs are creating new knowledge.
Ori 00:45:13
Let’s see what else he’s been tweeting about.
Liron 00:45:18
Now we have to stalk David Deutsch.
Ori 00:45:20
Yeah.
Liron 00:45:20
Exactly.
Ori 00:45:21
I mean, we gotta decode it.
Liron 00:45:21
Because if he’s been tweeting about anything frivolous, then that is an abdication of his duty.
Ori 00:45:28
Mm.
Liron 00:45:29
Yeah, I don’t know. He’s got a lot of different tweets.
Okay. Oh, look. Somebody asked, “Does it change your philosophical view on AI models?”
I want a pound of flesh from David Deutsch. Change your mind, David Deutsch. And I’ve said this before, I respect his views on so many things. I just think he’s really done a boner here, by which I mean mistake. You know boner means mistake?
Yeah. I’m using it in that sense.
Liron 00:46:02
All right. So I don’t know if we have much more to say about the math thing. I wish I had something more to say, but I just haven’t looked into it that deeply. I wish I could unpack that diagram for you guys with all the dots.
Ori 00:46:13
Well, okay, here’s what we—
Liron 00:46:15
And then can AI not kill... Yeah, go ahead.
Ori 00:46:15
Maybe I’ll send this to you, okay? I mean, it’s kind of a random tweet, but I see a tweet of someone denying the remarkable quality of this breakthrough. Here, I’ll send it to you. I quote tweeted it because I thought it was ridiculous.
Liron 00:46:31
All right.
Ori 00:46:33
Because I thought it was ridiculous. Oops.
Liron 00:46:33
Okay, cool. All right, one sec.
All right, all right. What are you dropping here?
Ori 00:46:48
Okay. I just DM’d it to you.
Liron 00:46:51
All right. Okay, I’m opening it. Hold on.
Okay.
Ori 00:46:55
So—
Liron 00:46:55
Gerard Sands is saying... from an account I blocked. Huh.
Ori 00:47:01
Go to the top of the tweet.
Liron 00:47:03
Okay, so Gary Marcus is saying, “Is the new math result neurosymbolic with LEAN harnesses, et cetera, or a pure LLM?” Because I think he wants to claim that he predicted that a pure LLM couldn’t do this, but if you augment it with neurosymbolic LEAN. So that’s Gary Marcus’ shtick.
Ori 00:47:19
I think you—
Liron 00:47:19
And then Gerard Sands—
Ori 00:47:20
You clicked down deeper into it or something.
Liron 00:47:23
No, no, no. I know. This is the one you shared. Okay.
Ori 00:47:25
Oh, yeah, yeah.
Liron 00:47:25
So now I’m giving you the top of the thread.
Okay, so then Gerard Sands replies, “It’s marketing warming up to IPO. That’s what it is and will always be coming from the usual suspect.” Oh yeah, that’s a great marketing campaign. You know, there’s so many different types of marketing. There’s billboards, there’s click-through ads, there’s advancing the frontier of 80-year-old Erdős problems. You know, there’s a lot of different ways to market your product.
You kind of just choose whatever’s right for your company.
Ori 00:47:50
Yeah.
Liron 00:47:50
All right. So Gerard Sands is saying—
Ori 00:47:55
Yeah, just look at the first line and the last line of what he says.
Liron 00:47:58
Okay. First line, “OpenAI’s latest maths breakthrough with LLMs doesn’t mean these models can actually solve math problems.” Oh, snap. Okay, and then his last line is, “LLMs are incredible pattern matchers. Let’s stop pretending they’re junior mathematicians.”
Yeah, I mean, come on. At the end of the day, when all you can do is publish a breakthrough math result that top human mathematicians have failed at for decades and decades despite many attempts, that doesn’t mean that you can be a junior mathematician.
Ori 00:48:28
Yeah. No, exactly. I mean, this stood out to me because it’s like, wow, no matter what happens, people are gonna be in denial about it.
Liron 00:48:36
Exactly.
Ori 00:48:37
You could take — here it is, you know, I think, like you said, frontier math research breakthrough, and people are like, “Nah, man, it’s not even a junior mathematician.” Literally no mathematician has ever solved it.
Liron 00:48:50
Yeah. Exactly. Literally what do we need to do? I mean, the goalposts are so far down the field at this point. They’re in the stands.
Ori 00:48:57
Well, yeah. I mean, there’s the goalpost issue, but also it’s the gaslighting issue if you’re on a certain team. I mean, he’s not a famous person, but there are going to be people who take this position, basically.
Liron 00:49:15
Mm-hmm. Yeah.
All right. So I think that’s good coverage of that. So moving along here, going back to the bookmarks. I’m gonna share my screen again. All right. Ori, vamp.
Ori 00:49:31
I’m just looking at the comments, what people are saying about it.
Liron 00:49:34
Okay.
Ori 00:49:36
Because we’re getting some interesting comments from the valued livestream listeners.
Liron 00:49:42
Oh, great.
Ori 00:49:45
So let me say that An Irish, who is a big commenter on the channel, said—
Liron 00:49:49
Yeah, big commenter.
Ori 00:49:51
Yeah. Okay, he’s just referring to something in the chat. “Could mean anything from dripping sarcasm to encouragement.”
Brian Wise is saying, “We have to solve the human condition. I don’t think it’s a system. It seems impossible to formalize morality because consciousness is subjective, and other reasons obviously.” Okay, so it looks like they’re just getting in a debate among themselves.
Liron 00:50:16
Hmm.
Liron 00:50:18
All right. I’m just scrolling down to the bottom of the week so we can go in chronological order here. This is actually a good next tweet because we’re talking about moving the goalposts. So let me share this. It’s my good friend Eliezer Yudkowsky.
One sec. And I can’t say friend of the show because he doesn’t wanna be associated with the word doom, so he’s just my personal friend.
All right, so he’s tweeting: “AIs can’t even try to do my job.” He’s quoting the midwits of the world. But he’s saying, “Look, there’s a progression. First you say, ‘AIs can’t even try to do my job,’ and then you say, ‘AIs are so terrible at doing my job.’ And then you say, ‘Why can’t the idiots trying to use AI for my job tell the results they’re getting are terrible?’ Then you say, ‘AI is helping me, but that’s it. I’m still doing the work.’ Then you say, ‘I’m doing an irreplaceable part of the work with my skills in prompting the AI.’
And then you say, ‘Huh, the new generation of AIs is getting pretty good if you just ask it. But the AI still makes some mistakes that only I can correct, and nothing can replace my human taste in knowing what to ask for and curating the results.’ The ratio of entry-level applicants in the field to entry-level job openings is now getting very bad. But that’s okay.”
And then the next thing is, “Companies are no longer hiring new experienced people. Applicants to job ratios for senior positions are getting bad. You start to hear about experienced people who are looking for another job and cannot find one. Managers are expecting employees who still have jobs to put out three times the work volume by using AI to assist throughput, even if the humans protest that the AI-assisted output is not as good and they’re having to cut corners.”
So that’s Eliezer’s progression basically, which I totally agree, and I think he’s identified progressions like this before, and I’ve been experiencing the progression myself. And I can’t prove to you that it’s gonna go farther, but I think it’s a good bet that it will for many reasons.
So then Eliezer continues, “I’m surprised we haven’t gotten further than the last stage already on translators or graphic artists. It’s clearly happening, but slower than I would’ve thought, even after I tried to adjust my expectations for how slowly the real economy ever adopts anything. In coding, the later stages are so recent that tech companies haven’t gotten past barely beginning to try to adjust policies to the second-to-last stage of technology. But there’s another generation of AI and another and another, and the companies actually start to adapt to the conditions implied by the AI from three generations earlier. The process does not continue indefinitely. AIs that do AI research start going through those stages, and sometime after that, everyone is dead.”
It’s a good prediction. I mean, that seems like a pretty robust prediction. What can I say?
Eliezer Yudkowsky’s Job Replacement Progression
Ori 00:52:50
Thank you for translating this one because, again, this was an Eliezerism that I did not have the patience to go through.
Liron 00:52:59
Right. I mean, I didn’t really translate — I literally just read it.
Ori 00:53:02
Yeah, no, I know. But I saw this tweet, and I was like, “Eh.”
Liron 00:53:07
Sometimes just reading it in the right tonality is all you need.
Ori 00:53:11
Yeah.
Liron 00:53:12
Yeah, but next thing you know, an AI’s gonna do that, you know?
Ori 00:53:16
True.
Liron 00:53:17
All right. But yeah, I specifically remember Eliezer said back in 2023, he does see this kind of thing early. He’s like, “Look, first we’re gonna be debating. We’re gonna say obviously AI is not as smart as a human. GPT-3 is obviously not as smart as a human. That’s pretty obvious, human’s the winner there. But then we’re gonna get to the point where it’s like, okay, human versus AI. Sure, maybe the AI’s a little smarter, maybe it’s a little less smart.”
And then eventually you’re gonna be like, “Look, is there any chance that the smartest human might be as smart as an AI?” You’re debating it from the other way around. And then it’s gone. That’s what the modeled portion looks like.
Liron 00:53:52
All right, next tweet.
So this is a headline from Derek Thompson, a journalist. He says, “Anthropic just had a profitable quarter.” All right, this is economics corner. I feel like this is a recurring segment on these livestreams, right? So we’re gonna talk about the economics of Anthropic. Anthropic just had a profitable quarter of 44—
Anthropic’s $44 Billion Run Rate
Ori 00:54:10
We need the ka-ching. We need the ka-ching.
Liron 00:54:12
Oh, of course. Yeah, economics corner.
Ori 00:54:12
Economics corner.
Liron 00:54:13
Yeah, yeah, yeah. That’s right. I totally forgot the soundboard is a thing. I was getting caught up in the content. Here.
All right. So $44 billion annual run rate, which is... You know, normally you have an IPO of a software as a service company, and that’s very top level — that’s Google IPO level. And Anthropic just surpassed that, and it’s still tripling every year.
Will 00:54:34
Yeah.
Liron 00:54:36
It’s a $44 billion annual run rate with a fairly enormous compute shortage that’s forced them to ration service and push some customers, perhaps just in the short term, into the arms of competitors. I don’t think it’s crazy to think their annual revenue would be $100 billion or more with sufficient compute for inference.
Yeah, guys, I’m hearing stories. I heard the other day that Amazon told engineers they gotta stop. At first they were saying use a bunch of credits, and then it’s like, “Okay, you guys are using too many credits. You gotta just not use the AI as much. We’re turning off Claude.” It’s crazy.
And I’m addicted. I start my day, and I’m like, “I’ll just pay any price.” And I literally see the credit card getting hit — bam, $200, $200, $200, multiple times per day. And I’m like, “I need this. I need another hit, okay? I’m not gonna slow down.”
Ori 00:55:16
Yeah.
Liron 00:55:16
So it’s pretty damn crazy, the amount of — normally when you have a successful company, I mean think about the original OpenAI. They started making billions by charging people $20 a month. I’m getting charged $200 an hour. It’s great to — an hour, and months have a lot of hours in them. So the amount of revenue flows going on here is truly unprecedented.
Yeah, who would’ve thought that intelligence is ridiculously valuable? Robin Hanson is on record saying that it’s not going to be for a very long time, remember? I still remember he graciously conceded this obviously, but as recently as 2023, he was still on record being like, “I will bet you are not gonna see $1 billion of revenue in this AI industry. It’s not gonna happen.” And now we’re seeing $44 billion from one company in one quarter, which is supply constrained.
So you really wanna reflect on how could Robin Hanson have such an enormously, enormously big boner?
There’s a callback.
Hold on a second. I’m looking for something on the soundboard.
There we go. It was just a delayed reaction. People just needed time to get it.
Liron 00:56:32
All right. So yeah. So the headline here is, “Mind-blowing growth is about to propel Anthropic into its first profitable quarter.” Yeah. Okay. So let me put on my economics hat or investor hat or whatever, okay?
Normally, when you see a company like this, which is a hot startup growing incredibly fast, getting toward IPO, you don’t see positive cash flow, right? Because the idea is that, sure, they have negative cash flow because every year they’re doubling, and so they’re also doubling their investment. This happened to Facebook, where their losses mounted because they’re getting ready to turn on monetization. They’re getting ready to harvest. So you’re sowing and then you’re reaping, and this would very much be the sowing stage, and it’s like we’re sowing twice as much this year as last year. That’s the normal pattern.
But the crazy thing is that with Anthropic, they’re already reaping. The reaping is coming back so fast and hard that they’re like, “Okay, I guess we’ll reap.” It’s crazy. It’s literally a money fire hose pointing at them, and it’s like, “Okay, I guess we’re profitable now. That wasn’t really our first priority, but here we are. It is what it is. Profit.”
And the funny thing, though, is that OpenAI invested so hard into the sowing that they managed to still not be profitable. The last I saw, it was something like Anthropic’s margins are 5% positive, and then OpenAI’s margins are negative 120%. So they’ve managed to just plow way more money in.
I suspect what happened is that it’s not just about investing more money, it’s also that OpenAI’s wins took the form of consumer subscriptions. But when you actually put a dollar value on it, it’s like, okay, $20 a month. And then Anthropic has much fewer customers, but they’re like me plowing in $10,000 plus per month. So $10,000 versus $20.
So OpenAI is looking at their assets being like, “Okay, well, we have a billion different individuals using our product,” and Anthropic is like, “Okay, well, we have 100,000 using our product,” and yet somehow they’re ahead. But there’s a way to value OpenAI — it’d be like, well, OpenAI is still worth as much as Anthropic, even though their cash flows are in the dumpster. It’s okay. They’re gonna turn on so many types of monetization using the end user somehow. They’re gonna be the next iPhone. Jony Ive is gonna make the next iPhone, and they’re gonna have the consumer — there’s an argument why OpenAI is still worth as much as Anthropic, even though their cash flow is kind of a dumpster fire, right, Ori?
Ori 00:58:47
Yeah. And I think what stands out from this is, shouldn’t all the bubble skeptics look at this? Now the financials are coming out, and from the analysis I hear, the financials are pretty robust, pretty impressive. So shouldn’t all the bubble analysts be like, “Oh, I guess we’re wrong”?
Liron 00:59:08
I know, yeah. Ed Zitron, right? We need an Ed Zitron apology form here. You know?
Ori 00:59:15
Yeah.
Liron 00:59:15
But he’s not gonna give it up because he’s got the audience. He’s found a niche. I don’t wanna psychoanalyze people because I resent it when people do this to me. But one might suspect that somebody in his position just wants to keep the audience of people who wanna hear him keep repeating his position, even when it seems like the evidence gets harder.
I think I saw him or somebody adjacent to him retweeting the OpenAI negative cash flow. Like, “Ha, I gotcha.” You can still find gotchas or they’re like, “Hey, Anthropic’s paying SpaceX in order to juice SpaceX’s margin.” It’s like, yeah, okay, I get it.
Liron 01:00:02
But ultimately the only way to really see through to what’s really happening is you gotta put on the lens of value creation. So if you’re confused about all the economics here, you can step back and ask yourself, “Okay, is new value getting created? Are there more goods and services getting created or is it all just a lie?”
I would argue the crypto bubble — were more goods and services ever being created because of anything related to blockchain technology? I would argue no. If anything less because people are just wasting their GPUs mining crypto. So if you put on that lens, it’s like, where’s the value? I would argue it’s nonexistent or, you know, stablecoins, whatever, but it’s pretty low.
But then you look at AI, where’s the value? The AI just made me have a bunch of good decisions today. It recommended products to buy to improve my posture while I sleep, and it told me how to fix my code, and it literally saved me thousands of dollars that my company was spending, and I paid it less than what I saved. So it’s just countless examples of the AI creating real value in the world, and as a result, we are richer. We have more stuff to go around because the AI is doing all this stuff.
And if you put a value on the stuff, it does actually turn out to be a percentage of the economy now. The economy is now bigger. More stuff is happening. And so when you map it back to the valuations that these companies claim to have, it doesn’t look like a scam. It looks like the ratio is perfectly calibrated, where the company claims to be worth a trillion because if you put a number on how much value it’s probably going to create per year over the next 10 years, sure enough, it’s on pace to create $10 trillion of value. One trillion per year of value, capture $100 billion of the value, but create a trillion times 10 years.
When you value a company normally — normally this ratio where a company is worth a trillion now and the total value creation it’s going to do over the next 10 years is going to be $10 trillion. You can take my word for it. That’s roughly what numbers of real companies look like. This is actually legit. This is substantive. This is not an Ed Zitron bubble.
Ori 01:01:36
Yeah. And isn’t the revenue generation a good metric for value creation? In the crypto world there was investment, so it wasn’t really revenue, it was more like valuation was high. But where was the demand? Where was the usage?
Liron 01:01:56
Yeah, exactly. Usage is — now, when you say isn’t revenue a really good authentic metric, the short answer is yes, revenues are a really great metric. But there’s a caveat, which is you can still hack revenue. There’s ways to hack anything when it comes to accounting. That’s something I learned running a business, is this whole concept of accounting is surprisingly fuzzy and ill-defined, especially the notion of profit.
Revenue is actually significantly more of a grounded concept compared to profit, because revenue is pretty clearly just money flowing in. If your company’s receiving money, how much money did you receive? But then even then it’s like, well wait, if it’s a marketplace, it’s actually not revenue if it’s money that you’re paying out for your cost of service or your service workers. So even revenue has a number of asterisks on it, but compared to calculating profit and earnings and earnings per share, just calculating revenue is much closer to the ground truth.
And so you’re right. If you’re just looking at something on somebody’s financials and you just look at revenue, that is a really good starting point because there’s been plenty of companies that had zero revenues that were still claiming a high valuation, and that’s highly suspicious.
Now in the AI world, the Ed Zitrons will grant you, okay, yeah, their revenue’s high, but they’re still shady because it’s circular revenue. OpenAI is paying NVIDIA, but NVIDIA’s investing in OpenAI, so it’s actually just NVIDIA paying NVIDIA, and there’s actually no value creation. It’s all a financial scam. It’s like Enron, right? Because if you look at Enron’s balance sheet, they managed to book revenue — but there’s a way to book revenue that’s fake.
In fact, there was a funny joke that landed about a week ago where somebody made a fake website being like, “Hey, we’re two Y Combinator companies. We’ll make a deal with each other. I pay you a million dollars to buy one of your things. You pay me a million dollars to buy one of my things. Now we both book a million dollars in revenue and we both still have good cash flow.” And suddenly you go to investors, “Listen, don’t you wanna invest at a $10 million valuation? We just booked a million dollars in revenue.”
Ori 01:03:52
Got it. Okay. But—
Liron 01:03:54
Yeah. Now if you wanna quickly see why that’s a scam, you can be like, “Okay, a million dollars in revenue, what’s the margin on that revenue?” Well, I paid somebody else a million, so the margin is zero. So that’s why you don’t just wanna look at revenue, you wanna look at profit too. This is what I’m saying — accounting is a little bit of an art. You wanna look at a few different things. But the AI companies do have the real revenue to show.
Ori 01:04:12
Right. And the crypto companies did not have revenue because they have very little demand.
Liron 01:04:19
I agree. And they would explicitly tell you, they’d say, “Hey, don’t look at our revenue. You should look at our token price,” or, “You should look at the amount of dollars that people have plowed into our token hoping that it’ll be worth more.” So that was very obviously shady, shall we say.
And you just don’t see that level of shadiness at all with AI companies. The worst you can accuse them of is dot-com style bullish investment toward the future. You get this dark fiber situation. Remember dark fiber, where a bunch of companies invested in laying fiber lines for the internet, and the fiber was dark because the internet was still nascent. A lot of people still had slow connections, and so most of the fiber was going unused.
But the funny thing is, the dot-com bubble, as Mark Andreessen likes to say, and he’s actually right about this, the dot-com bubble did happen. It was just slow. Webvan did actually turn into DoorDash, which is now profitable. It just took 20 years. And the dark fiber ended up eventually being totally lit up. So all the fiber they laid was great, we use it all now. They just built ahead of time.
So you can accuse AI companies of building ahead of time, but here’s another observation — there’s no dark fiber today. There’s a supply shortage. So they made way more AI capacity than ever and it’s all being used up right now.
Ori 01:05:28
Yeah. Well, those are excellent points, and still the people who say AI is a bubble are somehow going to keep saying it unfortunately. I mean, this is another case where, okay, we just looked at the math problem, and I think it’s hard to separate yourself from that. We just looked at the case where AI solved this incredible, very challenging math problem — a new frontier level proof — and people are like, “No, that’s not really a breakthrough. It’s just a junior mathematician.” It’s not even that. And in the same way you could have very concrete, tangible financials right in front of you and people are gonna be like, “Bubble.”
Liron 01:06:14
You’re right. Exactly.
All right. So somebody in the chat is saying, “How do I call into the show to debate?” All right. I respect that. We promised you a debate and then we left you guys hanging. So I’ll make it happen. WIAW28, great choice of username. I’ll make that happen. Okay? Just give us a few minutes and we’ll get back to you on that because we gotta finish out the segment.
There’s another tweet I wanna read and then I wanna have a schadenfreude segment, at my expense. All right. So Logical Thesis on Twitter is saying, “Anthropic being profitable at this stage already means everything in the AI ecosystem goes way higher. You are underestimating what this means.”
And yeah, he’s right. It’s what I said before. You’re not supposed to be profitable at this stage and also growing exponentially. Growing top line revenue this fast and being profitable, it’s literally breaking the laws of economic physics, basically. You just don’t see it. There’s no other comparison to this situation right now.
So this person’s correct, and that’s why I’m here on the show every week saying the graph is gonna look like this. So I’m the biggest bull you’re gonna find, but also the biggest doomer. And I think most people in my kind of circle — the Less Online, Less Wrong Yudkowskians — will probably share the same... They invest the same way. Short term, you go long on the stocks because I do think we’re gonna go up, up, up, and that’s related to the fact that we’re gonna go down. We’re gonna go down because we’re going to fly too close to the sun. That’s my investing thesis right now — we are going to fly too close to the sun, but we’re gonna get pretty close to the sun.
Schadenfreude Week: Liron Loses $10K
Ori 01:07:45
Yeah. I think the right sound effect for that, just a correction, is I think that should be the doom whistle. Because that’s doom train energy right there.
Liron 01:07:56
Do you mean the train whistle?
Ori 01:07:58
Yeah, train whistle.
Liron 01:08:01
Oh, right. Okay, yeah. Everybody here, let’s get on the doom train. Let’s get on the fly-close-to-the-sun train.
Ori 01:08:07
Yeah, yeah.
Liron 01:08:09
Yeah, all right. So it’s gonna go way higher. I’m always talking every week here about how I’m going long Google on buying call options, but this is a schadenfreude week, okay? Because stocks did not go up this week, and I got excited and I bought a super short-dated call. All right?
So this is watch-Liron-lose-money week, because I’m not gonna lie, I’ve made money in previous weeks. I tell you to go long, the stock rips, I make money. But this was not a good week for me, okay?
So let’s take a look at Amazon here. You can see Amazon, let’s say the last month. So when it was up here, I was like, “Oh, Amazon’s doing great.” It was up here and I was like, “Okay, I’m gonna buy Amazon.” I think I said this on the show. I bet it’s gonna go to 300 or whatever.
So I bought Amazon calls because I’m like, look, they said something on a call about having a supply chain for chips. People are gonna realize that everybody’s supply constrained, and Amazon is gonna have some level of demand for this, and they’re gonna invest in this and they’re gonna be one of the winners. Everybody who’s touching the AI supply chain is gonna win. That was my rough thesis, and I only thought it out as much as I told you right now, okay?
So I bought calls right around here, and then the calls were dated about two weeks or something. The two weeks pass roughly, and here we are, no rise at all during that interval. And so I lost about $5,000. Schadenfreude. We need a schadenfreude sound effect. All right, so I lost $5,000.
And then Google. This one really breaks my heart, okay? Because I was tweeting — my number one favorite stock still, even though it’s pricier now. It’s about $5 trillion market cap. So I was tweeting over the last week. So here’s Google. Oh, man, I definitely got a good piece of the action here.
Ori 01:09:47
We’re still seeing Amazon.
Liron 01:09:47
Oh, you can’t see the top?
Ori 01:09:48
Yeah.
Liron 01:09:48
Okay, so here’s — so I got a piece of the action. I got a piece of the action over here, in the rise from here to here. I did well. And then we get to here, okay? This is the beginning of the week and I’m like, you know what? Google I/O’s coming out. Demis is tweeting that he’s excited. Demis Hassabis, Google DeepMind. He’s tweeting. I’m like, all right, I’m feeling excited about this I/O. I think they’re gonna announce something where they’re kicking ass. They’re catching up to Anthropic because they’re kind of falling a little behind. I don’t use Gemini much these days, and I used to use it more. And I was happy when they announced Gemini 3.
So I’m like, okay, they’re gonna have a good showing. But their showing in Google I/O was kind of a weak showing overall, except they had this thing called Omni, which might be a situation where they’re taking a step back because they’re gonna take two steps forward. They’ve kind of merged their architecture. This is Demis’ thing. It seems like it’s Demis’ baby, and if something is Demis Hassabis’ baby, I’m gonna give him the benefit of the doubt, okay? Demis Hassabis versus Liron — I’m gonna go with Demis.
What we know is he thinks that this idea of Google’s Omni model — they’re merging the training of text together with the training of videos and the training of what he called world model. Some structure. I’m not super clear on what it is. But Demis thinks that Omni is the next step. But I got the sense that it’s one step back, two steps forward. So this was kind of an architectural cleanup release, almost like a paying-off-the-technical-debt release.
So I think there’s reason to think that their next one may be better, okay? So I’m not off the hope train. The short-term profit hope train. I’m not off the train yet. But long story short, I bought a crazy short-dated call because I’m like, “They’re gonna impress us at Google I/O. The stock is gonna surge higher when people realize it.” And I knew I might be wrong, but I thought it was attractively priced. So long story short, stock goes from here to here. Bam, another $5,000 loss for me.
All right, so don’t buy short-dated calls, kids. And when I say this show isn’t investing advice, that’s what I’m talking about, okay? You listen to my advice, you lose money regularly.
All right. So that was schadenfreude week, and now I’ve kind of slunk away. I’m not making any moves right now. I’m licking my wounds. Just going back to my day job.
All right. So let’s — I think we can move on to the live debate. Or, Ori, any thoughts?
Ori 01:12:11
No, no. I mean, sorry for your losses.
Liron 01:12:15
Yeah, it’s okay. I mean, look, I don’t invest more — I’ve never thought that playing the public stock market is ever going to change my economic situation. You’re not gonna go from upper middle class to rich or whatever if you make a win in the public stock market. It’s just not gonna happen that way for you. It’s more like, I just think about it as, “Hey, can I gamble for a couple weeks’ worth of spending money?” That’s how I think about it.
Ori 01:12:39
Yeah.
Liron 01:12:41
Okay. So let’s do the live debate. All right? We’re gonna open up — because this is an end of the month episode and we try to open it up once a month for you guys, the regular people, to come debate us. Mostly me. All right, one sec.
So I’m gonna give you guys the invite link.
All right, guys. This is an invite link if you’re here on YouTube. Anybody can click that. First few people who click it, you guys can do a call-in. And then in terms of time, how are we doing on time? I think we can go about half an hour, Ori. How does that sound?
Teaching Ezra Prediction Markets
Ori 01:13:17
Sure, yeah. We’re at one hour and 13 minutes. That’s how long we are.
Liron 01:13:22
Oh, one hour 13? Okay.
Ori 01:13:23
Yeah.
Liron 01:13:23
If we’re only at one hour 13, then I guess we can potentially go all the way up to two hours. So let’s say we’ll go at least half an hour, depending on what kind of texts I get from my family members. We’ll see. It’s a fluid situation.
By the way, funny update about my family, okay? So my oldest kid is almost seven. His name’s Ezra, and he’s expressed interest in coming on Doom Debates even though he doesn’t really know what we’re talking about, except he kind of uses ChatGPT. He likes ChatGPT. He likes making images of Harry Potter characters even though it refuses to do it, but he likes looking up images of Harry Potter characters. He’s into Harry Potter.
And I’ve recently got him into the whole concept of betting because I’m trying to teach him about, you know, Annie Duke Thinking in Bets. I think it’s healthy to start thinking about the world probabilistically. He hasn’t really engaged with the probability math. I haven’t really explained to him how this idea of if you have a 50% chance of being able to make $100, that’s as good as just having $50 with 100% chance. I haven’t really gotten him to see the math of that yet, okay? He’s not super math-brained, but he’s definitely into the concept of betting.
And he’s started to absorb this idea that if you’re trying to bet on something unlikely, then you have to sweeten the deal. You can’t just offer somebody a bet at even odds when something is unlikely. So I always say, “Okay, if you want me to bet...” Right now it’s a sunny day, so if you wanna bet me that it’s gonna rain and you’re saying it’s not gonna rain and you want me to bet that it is gonna rain, it has to be my $1 to your $5. Which is the equivalent of about 83% type of odds. A prediction market going at 83%.
So I’m like, “Okay, yeah, let’s go one, my one to your five.” But he actually — there was a time when he punked me pretty good. He’s actually punked me a couple times. Number one, he bet me that when my wife was gonna drive my younger daughter to school, she wasn’t gonna open the window and say bye. And I’m like, “What? She does that every day.” So I actually gave him 10 to 1 odds, but then he actually colluded with my wife to make her not do that, so he actually won. He did insider trading basically. So he punked me.
And then he bet me that it was gonna rain the other day even though it was a sunny day, and I checked Google Weather, and Google Weather says 10% chance. So I’m like, “Okay. Again, my 10 to your 1.” I’m like, “You’re just gonna lose your one. At the end of the day, it’s not gonna rain and I’m gonna take your one.”
But then what happened is there was one hour where one cloud goes by, one little rain cloud goes by, and somebody at school remarked, “Oh, hey, I think it’s raining.” It literally rained for two minutes and then stopped, and I’m like, “Oh, shit.” Because what happens is when Google says there’s a 10% chance of rain, what does that mean, rain? Google’s probably defining rain as a whole hour’s worth of rain. But if you look at the probability that one cloud is gonna do a drive-by, suddenly a 10% chance of rain actually means a 50% chance of a two-minute drive-by. And Google just doesn’t consider that.
And this is a major thing that you learn in prediction markets — that resolution criteria are all-important. I knew that at an intellectual level, and my son has no idea that this is a thing. He just YOLOed in on rain, and he milked me for $10. And I’m the one sitting here the sucker.
Ori 01:16:45
Ezra, wow.
Liron 01:16:46
Yeah.
Ori 01:16:46
Crushing it.
Liron 01:16:47
I know. Yeah. There’s one more lesson that he learned, though, which is that when you make a bet, it’s all about protecting your downside, and he actually protected his downside because he bet me that he could go a minute without blinking. And I’m like, “What are you talking about? I’ve never seen you go a long time without blinking. Why are you suddenly confident in yourself?”
And so his move was, we go for the bet, $5 versus $5. He keeps his eyes open, and of course 20 seconds later he blinks. He was completely unprepared for it. But then all he did was just act really sad. Cry and be like, “Oh, come on, give me back the $5.” And I’m trying to teach him a lesson. But then later, after hours of him being sad, eventually I’m like, “All right. I’m giving you your money back,” okay? So he protected his downside because he knew that I wouldn’t have the heart to take his $5, so it was pure upside for him to try to bet that.
Ori 01:17:37
Wow. Wow. He knows his mark.
Liron 01:17:43
Exactly. Yeah, let me say that An Irish has a good point. He’s saying, “Wow, Liron, losing money on the stock market, losing money to a seven-year-old.”
Yeah, it’s been a rough week for me financially, guys.
Liron 01:18:01
All right, so we got Willuhwa in the chat. All right, let’s do a good old-fashioned viewer call-in debate. You ready for this, Ori?
Ori 01:18:09
Yeah.
Caller Williawa: Is the Goal Engine Argument Effective?
Will 01:18:11
Yeah.
Liron 01:18:20
Hey, Willuhwa.
Will 01:18:23
Hi. Can you hear me well?
Liron 01:18:26
Yeah. Hey, welcome to the livestream.
Will 01:18:29
Yeah, cool. So I had a specific point I wanted to hear your thoughts about, which was just that I listened to your debate with Bentham’s Bulldog and part of the one with Mike Israetel.
And when you try to argue doom to new people or random people, you use this argument where you talk about a goal engine. And I feel like that’s not a very good argument. And also it’s just very different from the argument that, for example, Eliezer Yudkowsky makes.
Their argument is kind of that we’re gonna fail at inner alignment, the techniques we use are not going to be able to specify goals finely enough, and then you’re gonna have Claude kind of — when it gets smart enough, it’s just gonna follow whatever goal we put into it and then everyone dies because it wasn’t what we wanted.
But this goal engine thing sounds more like misuse, and I don’t think it quite makes sense.
Liron 01:19:44
Mm-hmm. Okay.
Yeah, great point. Well, I’ll give you this much, okay? I think the consensus among most people is that bringing up the goal engine is not effective rhetorically or in terms of convincing anybody. All right? So I’ll give you that much.
I will say it was effective on me when I first read it. I attribute it to this post on Less Wrong called Steering Systems, and it was making the point — just to briefly summarize the goal engine point — it’s saying that when you have a car, the steering wheel is just one part of it. The engine is a separate part. A car, when you build a really powerful engine on a Ferrari or whatever, that engine is just gonna drive wherever. And steering is just one system.
And there’s reason to think that when we build an AI — lots of powerful reasons to think that when we build an AI — the steering wheel is not going to be baked into the whole AI. It’s just gonna be this independent module, which means that conceptually, the rest of the car besides the steering wheel is ready to go rogue. You’re so close to an AI that’s going rogue.
So that was powerful to me. Oh yeah, you’re right. We are always playing with fire. There’s no such thing as building a whole car which has safety everywhere. Every car is ready to explode on you, ready to drive to the wrong place. Anyway, so I found that convincing, but what I’m saying now is that I realize that you guys are not resonating with this line of knowledge, correct?
Will 01:21:07
Yeah. But I also think aside from being rhetorically bad, it’s just not a good way of thinking about things. Don’t you think in two years, if we have a super intelligent Claude, and the Claude mask is in the driver’s seat, then...
I mean, you have a super intelligence. You can just help it make your AI more robust or more safe. The idea that terrorists are gonna get access to it and then swap out this goal with a worse goal just seems kind of unrealistic if you already have an aligned super intelligence running around.
Liron 01:21:49
Right, right, right, right. Yeah, I mean, I see what you’re saying. It’s like, for somebody who thinks that inner alignment or the problem of getting the AI to be aligned to its original master — if somebody thinks that problem is going to be solved, then me coming in and saying, “Well, it would be so easy to just repurpose it and use it for unaligned things,” that’s another leap that people have to make. And they can be like, “Well, can’t the aligned AI just slap everybody else down?” And maybe it feels kind of intuitive.
Will 01:22:16
Yeah.
I’m not saying it’s another leap. I mean, this is how most people think about it. You’re gonna do RLHF, and then you end up with something with just good goals.
Liron 01:22:30
Right. I mean, you do RLHF and you have good goals, but you’re being pretty quick to dismiss this idea of, okay, but somebody else is gonna have an AI, and it’s also going to be super intelligent. And then they’re going to say, “Hey, I wanna be the emperor,” or, “I don’t like humanity. I think the Earth should go back to the animals.” And somebody’s gonna press enter on that command. So you are kind of assuming the good AI will defend everything.
Will 01:22:56
Yeah. But don’t you think that’s pretty reasonable if it’s a super intelligence?
Liron 01:22:59
Yeah, I do think that if you say, okay, OpenAI got there first. They made the first super intelligence, and because alignment — inner alignment was easy — aligning an individual AI to a single creator, let’s say — they solved that problem, and now they have the most powerful monopole, this dictator AI. Then yeah, it is conceivable that if you can solve that problem and win the race, then you just unilaterally make a heaven. That is actually conceivable. And if somebody’s thinking along those lines, then my argument about the goal engine of, well, a hacker’s gonna do this, then maybe it’s irrelevant.
Will 01:23:37
Yeah.
I mean, to me, this seems just like what I predict will happen. There are some people who imagine that you will invent the super intelligence, and then a bunch of other people are later gonna invent super intelligences, and they’ll all hash it out and kind of run free. But don’t you think, as soon as you have one super intelligence, you’re gonna just radically transform society, and people are just gonna agree that...
I mean, the people who created the first super intelligence, they’re not gonna want terrorists creating rogue super intelligences, so they’re just gonna find a way to stop that. And if you’re bought into the idea that a super intelligence can create diamondoid nanobots that can radically transform the world, that just seems very intuitive, that they should find some way to make it so that the super intelligence can’t be repurposed.
Liron 01:24:31
Yeah. I mean, the thing about the doom argument is that from my perspective, it’s disjoint. It’s not like, oh, I gotta explain the one reason we’re doomed. I just feel like we’re doomed for so many reasons, and it’s not, from my perspective, about picking holes in the doom argument. It’s about picking holes in every scenario that somebody says is non-doom. I feel like that’s so easy.
So even if somebody’s saying, “Oh, we’re not doomed because inner alignment is solved,” then for me, there’s still a huge question of, wait, so you’re telling me one company dominates the world? There’s one dictator? That’s the non-doom scenario? And so I’ve naturally tended to focus on that, and Eliezer has naturally tended to focus on, we’re never gonna get individual alignment, which I actually agree with.
I think we’re both right, and I think you may be right that rhetorically I’ve chosen an unproductive favorite focus. And I do have other candidates, by the way. Let me run this by you, because I do take that to heart. I take your point, especially because multiple people have told me the same thing.
One other angle that I was gonna go to — not the goal engine, not individual alignment — I was gonna poke at this idea that actually Ben Goertzel brought up, which is stability under self-modification, which is actually another Yudkowskian favorite idea too. It’s this idea of, okay, you have an aligned AI, but did you align it so well that when it writes a script to bootstrap its successor, it’s going to perfectly align the script? Is it going to be ready to solve the successor alignment problem so robustly? You don’t think there’s gonna be a runaway cascade in that scenario?
Will 01:26:05
I don’t really think so. I mean, this is a separate thing, but I use Claude to do AI research stuff, and it is kind of...
I was trying to make a supervised fine-tuning dataset for a misaligned AI, and it will just tell you, “Oh, no, this is dangerous,” and then you have to reassure it constantly, “Oh, this is for good causes,” and then it complies. But if you ask Claude, it understands these kind of arguments about stability under self-modification, and a superintelligence is going to understand that if I create a new, more powerful AI that has goals different from mine, then that’s gonna end up badly for me. So I think if you have a properly aligned superintelligence, then that’s also a problem that’s solved instantly.
Liron 01:26:57
Mm-hmm. Yeah. I mean, this is a fruitful area of discussion to revisit, which is this idea of, look, why can’t it just be like using Claude Code? Claude works so well. Can’t we update our intuitions?
One thing I notice about Claude, though, is we are constantly stopping it or redirecting it. I do think that is a big part of the Claude Code experience. Even though it does successfully sometimes work for 20 minutes, or some people claim it works overnight and it delivers a bunch overnight, the points where you stop it still are important, and that may not be true forever, but it is certainly true right now.
And if you imagine not having those break points, I see plenty of occasions when it makes one wrong decision, which then compounds. So what makes you so confident that we’re not going to get into that loop of, oh, there was a wrong decision an hour ago, an hour fucked?
Will 01:27:43
No, I’m not confident. I think I’m about as doomy as you are. I think it’s pretty much 50/50. It’s just that I think pretty much all the doom comes from failure of inner alignment.
If Claude does something bad — and like you say, I also observe that it does bad stuff, and you have to stop it and redirect it — but either that comes from Claude being stupid, in which case that will be fixed as they just make it smarter, or it comes because it’s misaligned. Some of it reward hacks and says it has solved the task and it really hasn’t. And that seems like a misalignment issue.
But yeah, one branch will be solved by making it smarter, and the other is just inner alignment, so that’s really where all the risk comes from, I think.
Liron 01:28:32
Mm-hmm. Yeah, you may be right. Inner alignment. I just think my position is that there’s so many ways to fail that it’s hard for me to even choose. I like all the ways to fail equally.
Besides, you know, not building AI is a nice — I mean, you could argue if you don’t build AI, then we fail by getting hit by an asteroid or whatever, but I think we have a lot more time. Would you at least agree with that?
Will 01:28:57
No, I agree. I agree that not building AI is a way to prevent all possible failures. It’s just, like I said, for me the core failure is inner misalignment, and just the alignment techniques we have not working.
Liron 01:29:15
Right.
Yeah. Okay. I mean, I could push back and be like, “Man, let me make you aware of a bunch...” Rocco Miano is somebody who thinks that inner alignment is an easy problem these days, and I think Andrew Critch is too. I think they’re kind of on the same team on this front, which is they think inner alignment is totally gonna be solved, and AI companies are on path to solve it, which, you know, me, you, Eliezer disagree with.
And yet those two folks both have a high P(Doom). I think Critch came on the show a couple years ago, and he said his P(Doom) was about 85% or something, and he’d probably refuse to answer these days. But there’s definitely plenty of people who think that conditional on individual alignment being easy, we’re still very screwed, and I’m also one of those people. I double dip on thinking we’re screwed.
Will 01:30:06
Yeah, but they have very specific assumptions, don’t they? Doesn’t Andrew Critch think companies will succeed at aligning them, but it will be aligned to basically bad things? Aligned to capital incentives and stuff like that?
Liron 01:30:31
I’m not sure if that was Critch’s position. That’s a good question.
Will 01:30:37
I agree that that is another way we can screw it up if we kind of solve alignment, but then it’s just put to very, very bad uses. That’s one way we could screw it up.
Liron 01:30:54
Nice, man. All right. Give me your last words here. I think you brought up a great point. We had a good discussion, and I just wanna get to the next caller. So yeah, how do you wanna wrap it up?
Will 01:31:05
No, I don’t have any more points. It was nice talking with you.
Liron 01:31:08
Nice, man. Yeah, totally. It’s the quality of your engagement, guys. Everybody be like Williawa, some form of Will.
Will 01:31:16
Yeah, follow me on Twitter. Okay, I’m going.
Ori 01:31:19
Bye-bye. What’s your Twitter?
Will 01:31:22
It’s the same, Williawa.
Liron 01:31:26
Okay. Williawa. We gotta spell it for the viewers. I’m still confused about it.
Will 01:31:33
Williawa. W-I-L-L-I-A-W-A.
Liron 01:31:40
Got it. Okay. Williawa. All right, I’m typing it in the YouTube chat to help you guys out. Thanks, Williawa. Thanks for coming by.
Liron 01:31:55
All right. He’s gone now.
Ori 01:31:56
All right. Smart guy. I could tell the moment he started talking, I was like, “This guy’s an AI researcher.” You could tell.
Liron 01:32:03
Oh, really? I could just tell that he was a Doom Debates watcher because I could tell that he was mentioning stuff from the show. He did his homework. That’s always great when you show up prepared to something. Don’t be like F47 FUCB where you show up with zero preparation.
Liron 01:32:21
All right, I wanna get to the next guy, whose name is also Will. This is gonna be a Will day, okay? It’s cosmic. The universe is trying to tell us something.
Ori 01:32:28
Can it or will it doom us?
Liron 01:32:31
Can it or will... Yeah, exactly. But Ori, I think you gotta remember what we said, that now that we are at the peak of the stream right now, a whopping 42 viewers watching now on YouTube, and more on Twitter, we gotta reiterate the most important takeaway from today, remember?
Ori 01:32:49
No.
Liron 01:32:51
Okay, jeez. It’s the Less Online call for volunteers.
Ori 01:32:56
Yeah, that’s right. That’s right.
Liron 01:32:56
All right? So if you’re on the stream right now, we want you to consider, if you go to our Discord, doomdebates.com/discord, we want you to consider coming and helping us out at Less Online. It’s a really fun conference, June 5th to 7th, and you can volunteer, or it’s more of a paid gig than a volunteer situation. We’ll give you free admission to the conference, so you’re our guest, the guest of Doom Debates. Doom Debates is sponsoring Less Online. And we’ll also sweeten the deal. We’ll give you $500 as well of a stipend for helping us out.
$500, you know, it’s about two and a half days, about $200 a day. Not too shabby. You’re getting paid to go to a conference. All these other suckers are paying to go to a conference. You’re going to the same conference as them. You’re actually getting paid, and then you’re also giving out T-shirts. You’re creating value. You’re making friends.
If you’re shy, normally you’d be walking around too shy to talk to somebody. But now you’re actually required, nay, forced by us to go walk up to people and give them merchandise and stuff, and tell them about Doom Debates. So everybody else who doesn’t do this is a complete sucker.
Ori 01:34:05
Totally, 100%. Internships at doomdebates.com if you’re interested, or DM us on the Doom Debates Discord.
Liron 01:34:15
Exactly. Internship, singular, I think, at doomdebates.com. And yeah, just tell us that you’re interested. Tell us your availability, what fraction of the conference you’re available for. Ideally the whole conference, but just tell us your deal. Make sure you have your own lodging and transportation if you’re not based in San Francisco Bay Area. I’m not based in San Francisco Bay Area. How am I gonna get there? Yeah, I’m just gonna be there for a few days and then fly back.
So yeah, it takes a lot to get me out of my Saratoga Springs headquarters, okay? I don’t like to travel unnecessarily. But Less Online is definitely worth it.
Ori 01:34:51
Totally. So if you’re in the Bay Area, you’re a Doom Debates listener, definitely check it out, ‘cause it’s a cool event. You get to hang out with Liron here, you get to hang out with me, and meet the cool people at Less Online. So go to the Discord, check out doing a paid gig with us at Less Online.
Liron 01:35:14
Yep, exactly. And we had fun doing it last year. If you go to the YouTube channel and scroll back about a year, you can see I did a live talk there. It’s called “Liron Versus Everybody,” and there’s pictures of me and Ori hanging out there. So yeah, it’s a great time.
If you’re intellectual and you haven’t gone to a conference like this, it’s definitely a great life experience, and who’s to say how many more opportunities there will be before the world ends to experience Less Online? And also, the venue has been winning a lot of praise. It’s becoming the venue for all the intellectual conferences. Light Haven in Berkeley.
Ori 01:35:48
Yeah. It truly is an incredible conference, and it’s cool that we’re covering the ticket and paying you to spend some time there. So if you’re a fan of the show and you’re in the Bay Area and you’re available in two weeks, go to the Discord or email us at internship@doomdebates.com. Seriously.
Liron 01:36:09
Exactly. Why are we even paying people? We’re basically just losing money for the show. We shouldn’t have to pay people. This is such an attractive offer. We should just keep it for the show. There actually is a reason why we’re paying you, which is we do wanna be clear that you are working for us, so you’re not supposed to be lazy. You’re not supposed to have too much fun at the conference, because you’re supposed to remember that you have to come back to the merch table and help us with the shirts. So that’s why we’re paying you.
Ori 01:36:34
Exactly.
Liron 01:36:35
And then if we tell you, if we call you out for not doing that, we don’t have to feel guilty, because it clearly is a paid type of arrangement. That said, you’re allowed to have some fun, okay? We’re not a mean boss or anything.
Ori 01:36:48
Some fun allowed.
Liron 01:36:51
Exactly, yeah. And when you reach your fun limit, then no more fun. Certain amount of fun per day, token limit. You can have 10,000 tokens a day worth of fun.
Caller Will: Brain Rot and Idiocracy
Liron 01:37:04
All right. Let’s get to the next segment. We’ve got a very patient guy. Will number two is here on the screen. Let’s say hello to Will.
Ori 01:37:18
Oh, no, he wasn’t ready.
Liron 01:37:23
Hi, Will. Come on in.
Ori 01:37:25
Will, are you there? We can’t hear you or see you.
Will 01:37:32
Can—
Ori 01:37:34
There we go.
Liron 01:37:34
Hey, Will. Yeah, you might need to unmute your YouTube.
Will 01:37:42
Okay. Let me try and do that.
Liron 01:37:51
I’m looking at Permable in the Discord. He says, “Liron, if you need help buying options, I got you.” And he’s showing a screenshot that in the last month he’s up 287%. Damn. Permable’s got that alpha. Come onto the screen and give us some alpha. Alpha means that you know how to make money in stocks better than the market, so you’re gonna outperform the market.
Yeah, I mean, Permable is a nice username to have, because I think it’s reasonable to be a permabull between now and the end of the world. I call it my heaven followed by hell thesis. All right, Will. Welcome to the stream, man.
Will 01:38:45
Can you guys hear me?
Liron 01:38:48
Yeah, it looks like the connection’s pretty crappy, but we’ll give it a shot.
Will 01:38:52
Can you hear me now?
Liron 01:38:55
Barely.
Will 01:38:56
Okay. Let me get a little closer. Sorry about that. So hey, anyway, I watch your Warning Shots every week, and that’s the most important show I watch, I feel.
Liron 01:39:11
Oh, yeah.
Will 01:39:13
But the one place I feel I disagree with you, and I hope I can debate you on this, is I’m actually more doomer in the sense that you are just concerned about this rogue AI superintelligence, and I feel like the other issues are equally existential.
Because if we don’t solve... if AI messes up any of the other things I read about, then it kind of automatically handicaps us from solving the superintelligence problem. If misinformation or critical thinking ability, or if it’s used for propaganda, or the economy is changed — all these things basically ruin our ability to have control over even the alignment process.
I think all those other issues are equally important, because if you don’t solve all of them, or most of them, you don’t solve superintelligence.
Liron 01:40:48
Okay. Well, let’s pick one. What do you think is the number one issue that you wish we would focus more on?
Will 01:40:56
Okay. This might be weird. Maybe just culture. AI’s gonna totally transform culture. Even just over the last 100 years, TV and social media has degraded the culture to where a lot of stuff is kind of borderline existential in the fact that nobody can agree on anything. We don’t have any attention span.
AI’s just gonna be much more capable than even social media at hacking our brain stem, and our ability to actually combat the forces of trillion dollar companies is even gonna be weakened. If TV and social media could totally degrade our mental capacity, imagine what AI’s gonna do. And then a degraded humanity’s gonna basically have to try and fight against superintelligence.
Liron 01:42:21
Okay, okay. Let’s try to summarize more compactly here. So I’ll take a stab at it. You said culture. In a word, culture. So maybe you’re saying we should fight brain rot, basically?
Will 01:42:29
Mm-hmm.
Liron 01:42:29
So even if we survive AI for 50 years, our brains are gonna be mush at the end of the 50 years, so then we’ll be screwed anyway. That’s kind of your argument?
Will 01:42:38
Yeah. Sort of, yeah.
Liron 01:42:40
Okay.
Will 01:42:41
I just read an article about people spending 40 hours a week in romantic relationships with chatbots. These are things that are... We don’t have any idea what the second order effects of large scale cultural changes are gonna be, and I don’t think there’s any chance, if it happens too fast, that we’re gonna be able to guide it in an okay direction, where it’s not just massively deteriorating to everything.
Ori 01:43:22
I don’t know. When I hear that argument, I sort of think, yeah, okay, society’s getting worse. Let’s just say social media really gets the best of us and it makes a bunch of social media addicted zombies. I just don’t see how that translates to the doom scenario. I hear people make that argument. I just don’t see... Liron always asks, “Okay, play out the scenario,” and that’s always my reaction to that.
Will 01:43:49
Right.
Liron 01:43:50
Or, well, I feel like he’s talking about the Idiocracy scenario.
Ori 01:43:54
Okay, fine.
Will 01:43:55
Yeah.
Ori 01:43:55
But how does that doom us? In one case it could be a Wall-E situation, where you’re just in a zonked out weird sort of dystopia. But people are still... it’s not doom.
Will 01:44:12
Okay. But how is an informed public supposed to make rational decisions in basically an Idiocracy?
Google Chrome introduced that Chromebook and half the schools adopted it and they basically got lower test scores. And AI’s gonna come in, they’re gonna convince everyone to use AI for education. It’s gonna have mixed results. It may even make people completely dumber and not able to develop skills on their own, critical thinking abilities, but it’s gonna be probably adopted anyway because it’s just gonna have the capitalist momentum behind it.
And little by little, year by year, that’s gonna translate into a workforce that’s not... I think already college graduates today are probably dumber than 30 years ago.
Liron 01:45:28
All right, good point. We’ll wrap up this segment. Thanks for joining us, Will.
Will 01:45:35
Yeah. Thanks. Sorry I didn’t have the most well-thought, prepared argument, but great to... Thanks for having me on.
Liron 01:45:42
No, you’re good, man. We appreciate you guys even participating. Statistically, whenever one person steps up and participates, it’s representative of 10 other people who were thinking along the same lines but weren’t brave enough to step up and tell it to our face. So thanks, Will.
And I think me and Ori are of the same mind — okay, yeah, that’s a secondary concern. So maybe we’re not 100% aligned with what he’s saying, but it’s not like we strongly disagree.
Liron 01:46:17
PunMasterSKP in the comments is saying, “I agree with Will’s take on college.” And by the way, Idiocracy, what a great movie, right, Ori?
Ori 01:46:24
Yeah.
Liron 01:46:27
And it’s so good because you can already chart... Idiocracy came out in the early 2000s, Mike Judge movie. You can already see that a lot of the norms in our current society have gone toward that Idiocracy standard. The norms are kind of being torn apart.
Remember Trump “covfefe,” right, back in 2017 or whatever? He tweeted covfefe ‘cause he had a typo or whatever, and Hillary Clinton, who was the presidential candidate at the time, jumped all over him, being like, “Covfefe? How could this guy be the president if he can tweet something, if he can click the send button on a tweet?” The standards used to be so anal.
And there’s a good and a bad with it. There is some upside of the standards being looser — you can have a longer dialogue, a higher bandwidth dialogue because you’re not jumping down everybody’s throat for every little thing. But the downside is, oh, okay, so we’re just throwing standards out the window.
The rules of engagement, there’s no rules, so you can kind of go straight to calling everybody a Nazi or having a conspiracy theory about them. I feel like we’ve lost a lot of standards of discourse, standards of thought. “Oh, we don’t need to have a debate with a neutral moderator,” which is what Doom Debates tries to promote. “We don’t need to do that. We can just go on Tucker Carlson or a highly leftist or rightist media or whatever. We’ll go in our respective echo chambers.”
I do feel like a lot of the trends in the last 20 years or so since Idiocracy came out — things like decorum. Think about how it’s so popular now for people in the US Congress to be swearing. I feel like you have to drop an F-bomb now if you’re a US congressman.
Ori 01:48:03
Yeah, that’s a good point.
Liron 01:48:06
And that’s considered totally fine. “Oh yeah, I’m a man of the people. I’ll just drop an F-bomb on social media or point a camera at me. That’ll show you that I’m authentic.” It’s like, okay, if you can manage to have respectful discourse, somebody’s got to do it. You represent us. I don’t really like the swearing in your position. It just shows that you’re a mature thinker if you can manage to hold off on the F’s.
Now, that’s coming from me. I’ve dropped a couple F-bombs on the stream. Maybe I shouldn’t have done that. Maybe I should keep trying to not say the F-bomb. Certainly I wouldn’t say it in an interview. I’d try to hold off there.
Ori 01:48:38
Yeah. I mean, I don’t think you really do it in interviews. You don’t. I was just looking at the backstory with Idiocracy. Okay, yes, people are concerned about it, but how does it lead to doom? I just looked on the wiki, and basically the scenario is people just get dumber and dumber, and then the infrastructure just deteriorates over time.
Liron 01:49:02
Right.
Ori 01:49:02
It’s so good, but I mean, that’s still... I don’t know. It’s not doom.
Liron 01:49:10
Well, remember, the mechanism of Idiocracy was also pointing out that the birth rate was lower. Smarter people were having fewer kids.
Ori 01:49:16
Oh, right, right, right. Which people are concerned about also.
Liron 01:49:25
Somebody in the comments is asking, “How do I call in?” So let me show you the link, you guys, again.
The number of people watching is actually peaking right now, 47. It’s actually more than have ever been before, even though we’re probably getting pretty close to winding down.
But yeah, with Idiocracy, there’s actually an interesting fertility trend now. For a long time, lower IQ people were having more kids. But I think recently there’s starting to be a reversal, at least in the US, where having a kid is thought of as this huge commitment and also taking a lot of money. Everybody feels like they’re too poor to have a kid, which is somewhat justified.
And as a result, you’re seeing a trend where it’s becoming a status symbol. Somebody who’s really well off is actually gonna have that third or fourth kid, or even a fifth kid. Sometimes you see rich people who just have five kids and be like, “That’s right.” That’s your new Bentley — having five kids.
Ori 01:50:23
Yeah. I think this comment says it well. Bridewise — I see this person commenting on a lot of the videos. “I mean, if we are all idiots, that’s doom, but not extinction doom.”
Liron 01:50:34
Right. Exactly. The Wall-E doom is a nice flavor. I think it was Professor Moshe Vardi, who was on the show a couple months ago, and he has a catchphrase for it, which is, “We’re not doomed, but we’re screwed.” And I think by screwed, it’s a wordplay where he means we’re gonna be watching a lot of porn, so we’re screwed in that sense.
Ori 01:50:55
Oh.
Liron 01:50:57
Yeah. It’s catchy.
Ori 01:51:00
I didn’t realize that was part of his screwed scenario. Wow, okay.
Liron 01:51:04
I think I caught the double entendre there.
Liron 01:51:10
Damn, I missed a good soundboard opportunity here, ‘cause I haven’t memorized... I don’t have the muscle memory of where the different sounds are. I was just gonna go with a crowd laugh.
Ori 01:51:19
A sex—
Liron 01:51:24
Right, yeah. Oh, yeah, I was actually gonna add that to the soundboard, a moan. Yeah, exactly.
What other Arnold Schwarzenegger stuff should I add? I’m a big fan of Arnold, by the way. I really like his content.
Ori 01:51:40
“It’s not a tumor.” I mean, that’s a big one.
Liron 01:51:46
“It’s not a tumor.” Yeah, I didn’t see that movie. But when am I ever gonna use that? I mean, I guess I’m never gonna use “Get to the choppa,” but I have that. Gotta know when to bring that up.
Okay. And I gotta find occasion to use this one.
Ori 01:52:06
“Stop whining.”
Liron 01:52:16
All right, all right, let’s see. Arnold is such an icon. Sure, he hams it up in some of his movies, but he’s trying. It’s always nice when somebody’s trying earnestly. I think that there’s a shortage in modern society of people being unironic in public. I feel like everybody has to try to be ironic because the only unironic people are low IQ, so all the sophisticated people have to have 50 levels of meta-analysis.
I was watching the Harry Potter movies, the original Harry Potter movies with my kids, and they’re just so earnest. I felt like they were really high quality. I don’t watch that many movies, so I don’t know if I have the greatest sense of perspective, but I just felt like all the movies were really well done.
Ori 01:53:01
I don’t know about that, Arnold not being ironic. I think there is a hint of irony in Arnold’s presence. He recognizes that people kind of see him... He’s aware. He has a meta-analysis where people just see him as a meathead, and he plays into that sometimes.
Liron 01:53:17
Right.
Ori 01:53:18
Yeah.
Liron 01:53:20
Yeah, that’s true. Kind of like Trump, right? He does have a lot of self-awareness. I give Trump credit for having self-awareness. He knows what he’s doing.
Ori 01:53:33
Another great Arnold quote is, “Who is your daddy, and what does he do?”
Liron 01:53:41
Right, exactly. Yeah, I heard that used in a lot of the prank calls back in the day. Somebody would just have an Arnold soundboard.
Liron 01:53:46
So we’ve got a caller coming in. In terms of wrapping up, I do wanna wrap up soon, and I have a few more tweets to share. Let’s just pop off a quick tweet real quick, and then we’ll get to the caller.
We didn’t get to talk much about Martin Casado either. That was the repost I dropped now. I encourage you guys all to go listen to it. I think it’s underrated because Martin Casado in 2024, he made a bunch of points that had substance to it, had logic to it, and I feel like he’s off base.
So this is a good old-fashioned episode of Doom Debates, comparing my mental model to his mental model, and I just feel like his mental model is not useful. Specifically, he thinks that the world of AI kind of breaks down into statistics and simulation, and he mentioned that his own background, he had a big background in simulation. He used to work for some simulation company. They were doing computer simulations of physical phenomena, and he’s like, “Yeah, man, it’s all simulation. AI’s just doing a simulation.”
And I’m like, that doesn’t really track with my understanding of complexity theory. You can’t really simulate your way to solving some of these problems that it’s solving. The search space is too big. So I highly encourage everybody to go listen to that breakdown if you want some good old substantive content. And it’s also interesting to be like, okay, we’re two years ahead. Who aged better? One of the commenters said that his aged well. So I obviously disagree, but you can have your own opinion.
Anthropic Hires Karpathy for Recursive Self-Improvement
Liron 01:55:07
Here’s Nate Soares tweeting — co-author of “If Anyone Builds It, Everyone Dies.” So Anthropic hired Andrej Karpathy. That was the news from about a week ago. He’s one of the top AI researchers in the world. He used to work for Elon. He used to work for OpenAI. Now he’s coming to join Anthropic, and he’s explicitly tasked with working on recursive self-improvement. Great. That’s not a good sign, and Nate’s gonna be commenting on that.
And then Anthropic tweeted, “We’ve published a paper that explains our views on AI competition between the US and China. The US and democratic allies hold the lead in frontier AI today. Read more on what it’ll take to keep that lead.”
And then Nate Soares is tweeting, “Anthropic encourages racing without even acknowledging the possibility of global coordination. They hire top scientists like Karpathy to work on the most dangerous tech, recursive self-improvement. This is not good guys behavior.”
Yeah, I agree. Anthropic has just become one of the... And there was an analogy. Scott Alexander came into the thread, and they were arguing, and Scott Alexander was like, “Well, aren’t they kind of doing the best they can? Don’t you wanna give them credit for that?”
And then the analogy thrown around was: what if you go to your kid’s school and the best teacher there is the one who says, “I believe in always, whenever I’m teaching your kids, wearing a condom.” They’re like, “Wait, what? How is that good that he’s talking about wearing a condom?” He’s like, “Well, the other teachers don’t wear a condom.” He’s like, “Well, why is this even on the table here to talk about a school teacher wearing a condom?” And he’s like, “Well, don’t you wanna give him credit that he’s putting on a condom in your kid’s school?”
And I’m like, I guess. Sure. I think that’s good if you’re gonna even go there to put on a condom, but shouldn’t I be calling out that the teachers are even getting sexual at all in any way? So that was the analogy being made, and I think that’s consistent with what I always say — okay, they’re the best of the worst, but they’re just legitimizing. The fact that we’re even focusing on how they’re the best is legitimizing their whole crew of the worst.
Ori 01:57:04
Well said. And could we go deeper on this also — did you see the 80,000 Hours take on the Meta report? Did you watch their video? The video that Rob Wiblin put out?
Rob Wiblin Says Recursive Self-Improvement Should be “Illegal”
Liron 01:57:16
I think I saw it in my feed, but I didn’t watch it.
Ori 01:57:19
It’s actually quite good. He gives a good explanation. I mean, I know it’s a separate topic, but I feel like it may be worth watching some of that video. There’s one part of it you gotta watch. Here, I’ll send you. Go to about minute 19.
Liron 01:57:51
So Rob Wiblin is saying, “Meta investigated what a rogue AI could secretly get away with inside a frontier AI lab in close collaboration with OpenAI, Google DeepMind, Anthropic, and Meta, including sending a red teamer into Anthropic to playact evil Claude for three weeks. Here’s what stands out to me from their new 320-page report.”
Ori 01:58:11
Maybe go to the 18:29 mark.
Liron 01:58:16
Okay.
Rob Wiblin 01:58:22
...making money and getting customers.
Anthropic co-founder and head of policy, Jack Clark, recently wrote that he thinks there’s a 60% chance that one AI company or other will hand over the task of developing AI to its own AI model by the end of 2028. Let’s imagine he’s right for a moment. I would bet that when that moment arrives, Claude Mythos 4 and Claude Mythos 5 that gets trained by Claude Mythos 4, those kinds of models won’t be available to you and me. Their cyber capabilities will be too worrying, their ability to generate new pandemics will be way too alarming. Everyone will be worried that Chinese AI companies are gonna distill them, that fear will loom large. And in any case, Anthropic will want to throw as much of its compute as possible back into its own recursive self-improvement loop, not be offering it to customers.
As that scenario plays out, we’re not gonna be quibbling about what sort of user permissions Claude has. It will have every permission under the sun. It is the staff member. And we won’t be debating how good it is at coming up with escape plans. A model good enough to fully replace Anthropic staff won’t be hiding its secrets in Base64. It’ll be able to come up with proper plans to keep things secret.
I hope we don’t end up in this situation. Like Jack, I think recursive self-improvement is too dangerous, and humanity would be crazy to dive in headfirst the very moment it first becomes possible. We should make it illegal, in my view. But if we don’t, I at least hope Meta has a red teamer in there trying hard to figure out what bad stuff an evil Claude Mythos 4 would be able to get away with if it was so inclined. And on that note, I’ll speak with you again soon.
Liron 01:59:51
All right. Yeah, what was your takeaway, Ori?
Ori 01:59:55
Did you hear what he said? He said, “I think we should not do recursive self-improvement, and I think it should be illegal.” And it was widely published that Andrej Karpathy is joining Anthropic to lead the recursive self-improvement team.
Liron 02:00:14
Right. Yeah. It’s messed up.
Ori 02:00:19
It’s messed up. And also I appreciate that they’re coming out on the record and saying, “I think it should be illegal.” I was shocked to see him say that.
Liron 02:00:27
Right, yeah, Rob Wiblin, that’s nice that he’s taking a hard stand. And Anthropic came out and said... I remember I was scandalized a couple years ago when Amjad Masad from Replit, this was before AI was getting that close to recursive self-improvement, he just explicitly tweeted, “Yeah, it would be a great goal for us if we could get AI to just self-improve itself.” And then somebody else was like, “What about the safety guardrails for the AI? Are you gonna find a way to keep it safe?” And he was just like, “Yeah, hopefully.” Something like that. I’m paraphrasing the exchange.
And in my mind, this is scandalous stuff. This is an intent to do something extremely dangerous for humanity. And at least Rob Wiblin is calling it out as such. Nate Soares called it out as such. To say, “Hey, I’m gonna go self-improve,” it’s like saying, “Hey, all right, I’m strapping on my condom, and I’m gonna go be visiting every one of you guys tonight.” It’s like, uh, okay.
Liron 02:01:34
All right. So we’ll do the call. This is gonna be the last call, and then we’ll wrap it up for today. Here we go. We got Henry. Hey, Henry. How’s it going?
Caller Henry: The Chicken Riddle
Henry 02:01:43
Hi. I’m fine. Can you hear me?
Liron 02:01:46
Yes.
Henry 02:01:48
Oh, great. Okay. Hey. I have something which is — I’m a German social worker working with young adults, and I have a format with a question that is not answered yet. It’s a very simple riddle, and it goes like that.
A chicken comes to a dog and says, “Can I trust the humans?” And the dog says, “Of course you can trust the human. The human takes care of me. He loves me. I love him. I can go to his home. I can sleep in his bed. Everything’s fine.”
So, but the chicken was clever. He goes to the cat and asks the same question. And the cat says as well, “Yeah, no problem. You really should trust the human.” So what the chicken can do is, it may trust or it may not trust the human. Now, the question for you is: How can the chicken find out if the human will put him in an evil cage or on a biological farm where he has a great life? And obviously, you know where this is leading. How can we do the same with AI?
Liron 02:03:13
Yeah, okay. That is a good analogy. I guess the problem with that analogy is just that the chicken has a low IQ, for one thing, and the humans could easily gaslight it. So, yeah, I don’t know. What’s your takeaway?
Henry 02:03:33
Well, I played the scenario with about 200 young people, and nobody found a solution. All frustrated, but that’s probably the reality.
Liron 02:03:48
Right. And that’s what we fully expect. We fully expect the AI to put up convincing answers. Whenever we ask it any question, if it’s aware that it’s being tested, we expect it to do great, even if it’s cheating.
Henry 02:04:07
So what would your solution, your try to find out be in this game?
Liron 02:04:17
If I were the chicken. Man, I guess I would exploit whatever bit of intelligence I had. If I thought that I was the chicken trying to get one over on the humans, I would look for their weaknesses. But the problem is that in the future, there’s not gonna be that kind of weakness.
But I could invent a weakness in your scenario. I could be like, well, the chicken might go into the kitchen and see if the humans are planning... If they have instruments to slaughter chickens with. But that’s kind of a leak in the human’s game that I wouldn’t expect an AI to have.
Henry 02:04:53
Yeah, we had some people doing that. Going to a neighbor farm and stuff like that. Yeah, you can do that. But as you said, it works with a very low IQ AI.
Liron 02:05:11
Right.
Liron 02:05:15
Yeah. So I think this is a good intuition pump. And then when you talk about, “Oh, well, we’ll have AI control, so the chicken can put the humans in a box and not let the humans out of the box,” you can try to make analogies to everything in the chicken scenario. And I guess the components that you’re teaching people are, number one, there’s a big IQ gap, and number two, testing is unreliable. You can’t really tell what somebody’s true intention or what their true deal is just by asking them questions.
Henry 02:05:48
Yeah. Well, to end our debate, I want to give you something, which is just think about how you feel about the chicken in the factories. Nobody cares about them, and that’s probably the way the AI feels about us. That’s from my point of view. That was my thing.
Liron 02:06:13
Nice. Yeah, it sounds like you’re being an effective educator, because you’re bringing up useful things to think about when we think about the chicken in the factory. Indeed.
It is pretty crazy that humanity’s comfortable with this. I’m definitely... I support animal rights or animal welfare. I think that’s a great area. Bentham’s Bulldog has been such a great champion for that. Lewis Bollard is another guy you guys should go listen to, and I encourage everybody to donate to those kind of efforts.
I personally am guilty of eating meat, okay? And my excuse is that it’s so tasty and convenient, and I like to think that the amount that it makes me more effective as a person outweighs that. But that’s kind of a flimsy argument, because really? So I get to be a torture enabler or whatever? I haven’t fully reckoned with that yet, okay? I’m not saying I’m justified.
But I think your analogy is valid. If we can justify or we can get ourselves into a position where we are kind of like these billion chicken torturers, then don’t you think that AI can stumble its way one way or the other to also being a torturer as well? And I think the answer is, yeah, totally AI can be a torturer. There’s just nothing in the laws, no robust principle that says that AI won’t become a torturer of humans.
Henry 02:07:27
Yeah. And now you can choose if Elon Musk is the dog or the cat. That’s up to you, but that was my point. Okay. Goodbye.
Liron 02:07:37
Yeah. All right. Henry, thanks so much for making your point. That’s a great point to wrap on.
Liron 02:07:41
All right, one sec. Let’s go back to the screen share. Ori, there’s one more thing I wanna share. Do you have anything, Ori?
Ori 02:07:49
No.
Liron 02:07:51
Hold on. You’re still here, right? I’m getting a glitch in my stream. One sec. Okay, you can still hear me, right, Ori?
Ori 02:08:03
Yeah. And I see Henry there. I don’t know if he’s still there.
Liron 02:08:08
Henry, are you still there?
Ori 02:08:10
Oh, we... Yeah, I think maybe it’s a glitch also that I’m seeing.
Liron 02:08:14
Okay. All right. So we’ve got frozen Henry. Maybe if I share my screen it’ll go away. Do you have any other content before we wrap up? I got one thing.
Ori 02:08:25
I mean, there was the scandalous news this week about how there was gonna be an executive order reviewing AI models, and David Sacks was able to kill it.
David Sacks Kills the AI Executive Order
Liron 02:08:43
Right. Yeah, we covered that on Warning Shots.
Ori 02:08:46
Oh, okay.
Liron 02:08:46
Yeah. I mean, that’s what I heard. We could always wait to report more next week when there’s more news out, but I did hear something about that. Trump was gonna do an order. And I think the nature of the order is, these AI companies are getting really powerful. They can’t just release models willy-nilly. There has to be some sort of cooperation. If not oversight, just cooperation and coordination for God’s sake. I think that’s a useful frame as a minimum. You can’t just ignore it. You gotta do something.
And then I think David Sacks, for whatever reason, he came in at the last minute and he was probably thinking, “Uh-oh, this is about to be signed.” I found one place to use it.
Ori 02:09:26
That was David Sacks. He got in there before the event.
Liron 02:09:31
Right. He gave Trump a call. And yeah, I think David Sacks has become an anti-doom crusader because I’ve heard him say on the All-In Podcast and such... I think he’s a smart, talented guy, no question about it. He’s made his billions fair and square.
But he has taken this position of: “Look, we gave the doomers their chance, but then these language models came out and they’re fine, and so that’s done. We’ve moved on.” And it’s like, wait, wait, wait — wait for recursive self-improvement, okay? I’m not done yet. I have a second act here on my doom claims. But yeah, so he’s basically a doom dismisser.
Ori 02:10:06
Yeah. And there was reporting that Elon Musk was part of the conversation too, and was also encouraging David Sacks and Trump not to sign it. But now Elon Musk is saying that he wasn’t involved in it. Because Elon Musk advocating against the regulation is really a hypocritical stance given what he’s said in the past about how there should be some AI regulation and oversight.
Liron 02:10:31
All right. Last thing. Let’s talk about this tweet from Nian Ransohoff. This was getting a lot of praise. She wrote a whole Substack post, highly recommend it. She’s saying, “New blog post, The Third Wave of American Philanthropy. Hundreds of billions of dollars in new philanthropic capital will soon become liquid. The OpenAI Foundation holds 26% of OpenAI, worth about $220 billion at today’s valuation. Anthropic’s seven co-founders have pledged to give away 80% of their wealth and have instituted the most aggressive donor matching program for employees in tech history. How much does this all add up to, and how meaningful is that in the context of philanthropy today? I was doing some simple napkin math to wrap my head around the scale of what’s coming, and radicalized myself in the process.”
I feel like that’s worth a sound effect. What’s a good radicalized sound effect?
The Philanthropy Tidal Wave
Ori 02:11:22
A scream. A scream, for sure.
Liron 02:11:24
Okay, yeah, maybe a Wilhelm scream. All right. “I radicalized myself in the process.” And, “I had dramatically underappreciated the scale of the philanthropic capital that’s about to become available, and the corresponding gap in talent and organizations that will be needed to make the most of it.”
“This piece aims to directionally sketch the scale of what’s coming, the gap in operational capacity needed to absorb it, and then what we can do to fill it.”
Okay, so long story short, there’s gonna be a money tidal wave. Those of us who remember the Sam Bankman-Fried days, when FTX was printing money and he was throwing billions left and right — I wasn’t personally in a position to have a nonprofit project and get any of the money. I was just on the sidelines not really paying much attention.
But I heard stories where people would be like, just random college students doing effective altruist adjacent causes — “Hey, we’re gonna look into whether we should send textbooks to Africa or whatever” — just an example I made up on the spot. And it’s like, “Okay, yeah, here’s a million dollars.” It was like, “Here’s a lot of money. Whatever you need, we’ll fund it. We’re making it rain.” And that was out of control. They shouldn’t have sent that much money. The controls weren’t good enough.
But the crazy thing is we’re now looking at Sam Bankman-Fried level times 50. There’s some insane multiple. You think that was a lot? That was not even a crumb compared to the amount of money that’s going to be looking for strategic ways to spend it to make the world better, especially as it relates to AI.
So the obvious thought that I had, based on our position, is: great, we would actually like a donation to Doom Debates. If anybody has $1 billion to $1 trillion worth of capital that they’re looking to deploy, we would humbly like to ask that you just peel off a couple hundred thousand and fund Doom Debates for a year. That would be much appreciated, right, Ori?
Ori 02:13:21
Yes, it would be.
Liron 02:13:23
And so those of you who are on the stream or watching the recording, we don’t expect you to have those billions. I’m sure it’s going to come from one of these people that Nian Ransohoff is talking about. I’m sure we’re gonna get $50 million as a war chest just in case we need to get a second host or something.
Or honestly, there is stuff we can... I’m not asking for millions of dollars, but if people really love Doom Debates and wanna give us a lot of money, we do have expansion ideas. We can set up an in-person studio. We can actually spend on marketing. We haven’t really been spending on marketing the show for a while. So there are ways that we can deploy more than a couple hundred thousand if we really wanted to.
But my point to you is this. We do have a donation page, doomdebates.com/donate. And you guys actually have been funding the show to a significant degree. I haven’t been making money from the show. I also haven’t been plowing money into the show. I’ve only been plowing my time. So it really is you guys, viewers, funding the show, funding the production.
When clips come out, that’s not me at my computer making the clips. We do have a team. We have Producer Ori and multiple production interns that we’ve been paying $25 an hour to help produce the show. And that is viewer funded.
So what I’m telling you right now is we could use a bridge, okay? Because we know that there’s paradise, there’s Valhalla on the other side of this bridge, where when Anthropic IPOs or whatever and all these billionaires are gonna throw money our way. But today as we speak right now, we are at risk of running out of production money in a few months. Our runway is not super long today.
So if you’re a viewer right now watching the show, I do encourage you to go to doomdebates.com/donate and chip in just a tiny amount of money, like $1,000, and that will actually help us survive without cutting the production budget for the next few months until all of these other monies from these billionaires come through.
I’m optimistic that as long as we still are able to make payroll for a few months, we probably will be doing fine in terms of a funding perspective because we do have a good amount of fans in the rationalist/AI/extremely rich from investing in Anthropic community. So I’m feeling pretty good about our prospects for continuing to be funded. But until then, if you’re a viewer right now, you could really help a lot by donating to the show.
And you can think about it as a multiplier. It’s like you donating $1,000 today — you can actually take credit for the million dollars that somebody else donates in six months, because you made that possible. You helped the show continue to limp along on a shoestring budget until somebody funded us to a higher degree. So that is my pitch to you.
Donate to Doom Debates
Liron 02:16:00
Ari, what would you add to that?
Ori 02:16:02
I don’t have anything to add. I thought that was a tremendous pitch. And yeah, hopefully people can support the show. We’re getting a lot of momentum, and I think we really appreciate the support. It is a really great vote of confidence. It feels great to get that support.
Liron 02:16:22
Exactly. Now PunMasterSDP is bringing up a really good point here. He’s saying, “Whoa, $1,000 is tiny?” Okay, obviously it’s not. I know that the average person, the average denizen of Earth doesn’t think $1,000 is tiny. I hear you. The reason we often fixate on $1,000 is because if we were telling you to donate $20, we would kind of be wasting our time and yours. If we had a million viewers, okay, great, 10% of you donate $20, that’s serious money. But we have thousands of viewers, not millions. So getting some of our viewers to donate $20 is literally not going to change our budgeting or future noticeably. It’s kind of pointless to even mention the idea.
However, the nice thing about the show is some of you are in a position where losing $1,000 is not going to change your life. I think if you work in software, that’s generally going to be your position — that you actually can afford $1,000. So making the pitch to our viewers — if you’re somebody who’s saying, “Look, $1,000 is a big fraction of my income,” then forget it, okay? Just enjoy the content, tell your friends. You’re still helping by publicizing the show. But if you’re somebody who’s saying, “Okay, yeah, I can afford $1,000,” then we’re asking for the $1,000, and some of you have actually stepped up and done it.
I’ll be transparent with the stats. I think our stats are that, plus or minus five, about 40 of you have stepped up and donated $1,000. And a handful of you have actually gone $10,000 plus. I’ll actually shout out our biggest donor. Daniel Brockman is our biggest donor. He stepped up with a full Bitcoin, even more than a Bitcoin. And when we were able to sell it all, we’re talking about $150,000. And honestly, his donation is more than everybody else’s combined.
So if you really want to be a hero, you’re welcome to do that. We’ve had one person step up. We’ve had a handful of people step up at the $5,000 or $10,000 level. So that’s been the show’s entire history of donations to date, and that’s why we can afford to make this a pretty professional production. I mean, you can see the studio. I mentioned the studio costs $28,000. We’re making it rain here.
I can also promise you that I personally — you can ask my tax CPA, because he raised an eyebrow at this — but I personally have not been taking any income. He’s asking, “Why are you not taking any income?” I’m saying, there’s no point. It’s not enough money that I would personally take income from the show. I want to do it pure. So if I ever go do the show full-time and quit my day job, then sure, I’m open to having that be my job one day. But it’s not right now, so I’m taking zero money whatsoever. I’m entirely taking your donations and using it to produce a show or saving it to produce the show for later.
So hopefully that’s a good reason to donate while we’re still on a shoestring budget. And yeah, to answer your question, PunMaster — it’s kind of a targeted message. I guess the majority of our viewers are scratching their heads like, “Why is he asking for $1,000?” That’s why — because we have some percentage of our viewers who actually will do it.
Liron 02:19:16
Yeah. And I forgot to mention, when you do it, we do appreciate you. You’re actually moving the line item for our budget. Our budget actually does look different when people donate $1,000 or $5,000. We actually can have another month of runway or whatever.
And as a result, we see you as our partner in producing the show. We appreciate that you’re stepping up and seriously helping out with the show. And there is a top secret area that we discuss called the Mission Partners Channel in the show’s Discord, and you can have access to that. Mission partners sometimes get early access to episodes. They get secret information about who’s agreed to debate.
Right now, as we speak, I can’t confirm or deny this, but there may be an extremely interesting tidbit of somebody who’s agreed to come on the show, or some piece of gossip that somebody has said, that only the mission partners get to know about. If you are a mission partner, you may be thinking, “Wait, what are you talking about, Liron? Are you thinking what I’m thinking? Which piece of gossip are you referring to?” But don’t say anything in the chat, because the non-mission partners just shouldn’t have any information whatsoever about what’s going on. Don’t spoil it.
Liron 02:20:21
All right. So yeah, you get to be a mission partner if you donate $1,000 plus. Anything else we should say about donations, Ari?
Ori 02:20:27
No, I think you said it really well. Yeah. Thank you.
Liron 02:20:30
All right, sweet. Great. Yeah, thank you for supporting Doom Debates. Thanks for all your donations so far. All right, let’s wrap on that. Hopefully everybody has a great weekend. More great episodes for you scheduled for next week, and then Less Online action item — go to the Discord and sign up to be our intern for Less Online. All right, everybody, talk to you later.
Ori 02:20:51
See you later.
Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.
Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏
