Liron and Producer Ori explore the community's feedback to Eliezer Yudkowsky's $10,000 debate against an anonymous AI director. Plus, we unpack Eliezer's new post on the "irretrievability" of ASI development, Anthropic feasting on xAI's compute, and Liron's 4x Kalshi bet on... Spencer Pratt.
Timestamps
00:00:00 — Welcome
00:02:30 — The $10,000 Debate Post-Mortem
00:06:01 — Paul Tudor Jones: “Zero Risk Management on AI”
00:09:49 — Liron’s Jerry Springer Moment
00:14:19 — Why Yud Wore Steampunk
00:20:19 — YouTube Reacts: “Five Minutes In, Already Unhinged”
00:27:19 — 47F’s Anti-Disparagement Legal Theory
00:32:18 — Yud Wants Round 2
00:35:37 — Lumpin Space, Ben Goertzel & Early AI Safety Memories
00:47:06 — Eliezer’s “Irretrievability” Post
00:56:35 — The Maginot Line and Murphy’s Curse
01:08:13 — Blockchain vs AI: Earth’s Compute Swing
01:15:15 — Anthropic Eats Elon’s Compute
01:25:29 — Is Elon Tweeting as His Own Mom?
01:30:03 — Waymo Dodges a Fallen Scooter Rider
01:33:00 — Why Liron Bet on Spencer Pratt
01:37:13 — Live Twitter Scroll!
01:38:11 — Mira Murati: “Directionally Very Bad”
01:50:07 — AI Copies Itself Across Servers
01:56:44 — Steven Byrnes: LLMs Aren’t the Final Paradigm
02:07:55 — Wrap-Up
Links
LessOnline 2026 (June 5-7, Berkeley, CA) — https://less.online/
Paul Tudor Jones on AI Risk, Bubbles and Buffett (Invest Like the Best w/ Patrick O’Shaughnessy) —
Liron’s tweet: Paul Tudor Jones AI risk quote —
Eliezer Yudkowsky vs. 47F debate announcement —
47F “4D chess” clip —
Doom Debates: LumpinSpace episode —
Ben Goertzel debate announcement —
Jack Clark: 60% chance of recursive self-improvement by end of 2028 —
Eliezer Yudkowsky, “Irretrievability; or, Murphy’s Curse of Oneshotness upon ASI” (LessWrong, May 4, 2026) —https://www.lesswrong.com/posts/fbrz9xhKpEeTKw5zL/irretrievability-or-murphy-s-curse-of-oneshotness-upon-asi
Eliezer Yudkowsky, “Only Law Can Prevent Extinction” (LessWrong) — https://www.lesswrong.com/posts/5CfBDiQNg9upfipWk/only-law-can-prevent-extinction
Anthropic, SpaceX announce Colossus 1 compute deal (CNBC) — https://www.cnbc.com/2026/05/06/anthropic-spacex-data-center-capacity.html
SpaceXAI announcement: Compute partnership with Anthropic — https://x.ai/news/anthropic-compute-partnership
Derek Thompson on AI supply crunch —
Zach Brock: “Congrats to Anthropic for feasting upon the compute of their fallen enemy” —
“Lisan Al Gaib” on semiconductor sector investment —
Liron’s Drake meme: Blockchain vs. AI compute —
Dmitri Dolgov (Waymo co-CEO): Waymo safety in Austin —
Steven Byrnes thread on domain-general ASI —
Palisade Research: AI self-replication findings —
Kalshi: Spencer Pratt LA Mayor market — https://kalshi.com/markets/kxmayorla/la/kxmayorla-26
Spencer Pratt 4x Kalshi bet —
Morris worm — Wikipedia — https://en.wikipedia.org/wiki/Morris_worm
Transcript
Welcome
Liron Shapira 00:00:00
All right, we got 11 watching already on YouTube. Welcome.
Producer Ori 00:00:11
Welcome, everybody.
Liron 00:00:11
Coming at you from Saratoga Springs, New York and San Francisco, California. We got coverage of both coasts happening right now.
Ori 00:00:18
Hey, hey, everyone.
Liron 00:00:22
That’s right. We should advertise that more, coast-to-coast coverage.
Ori 00:00:25
That’s right. I’m bringing you the vibes from San Francisco.
Liron 00:00:28
Coast-to-coast coverage of the AI industry. Exactly.
Liron 00:00:31
Let’s see. I don’t know if I count as being on the coast. I’m in upstate New York.
Ori 00:00:35
I would count where I am as being on the coast, for sure.
Liron 00:00:38
Yeah, I’m gonna join you. I’m gonna be on the coast. I’m gonna be in Berkeley, California in a month from now. Come to the Less Online conference. Everybody go to Less.Online. Come hang out with me and Ori at Less Online. We’re working on a sponsorship too, so you can get Doom Debates merch if you come to Less Online.
Ori 00:00:57
Hell yeah.
Ori 00:00:57
Less Online is a great, great vibe. It’s—
Liron 00:01:02
Yeah, highly recommend it.
Ori 00:01:03
Way more friendly than Less Wrong.
Liron 00:01:09
Than the Less Wrong site, exactly. Because you can’t get downvoted there. Everybody just has to vote up your presence.
Liron 00:01:22
All right. Hey, we got a lot to talk about this week. The news is coming fast.
Ori 00:01:31
Oh, yeah. There is a lot to talk about. I mean, what’s on your mind?
Liron 00:01:32
There hasn’t been a model launch, so I guess it hasn’t been the number one biggest news week, but it feels like I got a lot of stuff going on in terms of bookmarks.
Ori 00:01:44
Such as? What are the main things that you’re thinking about?
Liron 00:01:47
I think we should prioritize. Why don’t we pull up your feed and you can show us some of the stuff you’ve been bookmarking.
Ori 00:01:55
You wanna see my feed? The things that I bookmark.
Ori 00:02:03
You don’t wanna see my bookmarks. It’s a bunch of nonsense.
Liron 00:02:08
Sounds good. I might have some celebrity gossip on mine. We can start with that. All right, so I’m pulling up the bookmarks.
Liron 00:02:21
Man, there’s so much. I wonder if I should try to go in reverse chronological order — or, I mean, I guess chronological.
Ori 00:02:27
I mean, is the main thing reacting to the Eliezer debate? I feel like we gotta have that, right?
The $10,000 Debate Post-Mortem
Liron 00:02:32
Oh, yeah. That’s gotta be the main thing.
Liron 00:02:36
Yeah, so I’m guessing everybody on the stream has had a chance to see the Eliezer versus 47F debate. I guess let’s screen share. I forgot screen sharing’s a big part of these events. One sec.
Liron 00:02:57
All right, so here’s my X tab. Let’s go to — I think I bookmarked it. So we’ll go reverse chronological order.
Ori 00:03:13
Well, let’s just start with this. Here’s my question for you. How does it feel — I mean, that debate went seriously viral. How did it feel being the host of that?
Liron 00:03:25
Yeah, it felt good. I want everybody to have a good sense that Liron’s channel is the place for Yud. We should get one of those banner chyrons or whatever. That should be our tagline, “The place for Yud.”
Ori 00:03:42
Yeah. It is, yeah. You can see him in his normal attire or see him in steampunk attire.
Liron 00:03:49
Exactly. And funny enough, he explicitly doesn’t want the association with Doom Debates because he thinks the word doom is harmful, which, fair enough. I’m open to other ideas. It’s just really tough. It’s like, oh, I’m an extinction warrior — it’s just tough to have a short moniker.
So I’m still looking for an idea, but for now I just say doom, doomsayer, doom, whatever. Eliezer doesn’t want that, so that’s why it’s just Liron’s channel. But I’m more than happy to host, and one of the tweets I bookmarked is him saying that he’s willing to debate somebody else again. He didn’t explicitly say that he’d come host it on my channel, but of course he’s got the open invite, so I hope he does.
Liron 00:04:26
I’m happy to accommodate any other requests he had. We said “no doom.” I’m happy to keep accommodating that, and whatever other requests — I’m here for it. People like it. It’s already gotten 15,000 views on YouTube. We can scroll through some of the YouTube comments.
I guess maybe a good place to start is I’ll go to my own profile pic here. Oh, we’re live. Look, inception. It’s us right now. How many viewers?
Ori 00:04:48
Hey, that’s us. Let’s go deeper.
Liron 00:04:51
Oh, 36 viewers on Twitter. Yeah, we gotta go deeper. Here, let’s play the audio.
Liron 00:04:57
All right, what’s going on? So I think it’s supposed to go infinitely deep, right? Once this refreshes — oh, look. So I think it’ll keep refreshing and going deeper and deeper.
Ori 00:05:07
TBDN has never done this, okay?
Liron 00:05:09
Yeah, exactly, right?
Ori 00:05:09
MTS, this is novel, innovative content right here.
Liron 00:05:14
Exactly. I wanna know what the Ori four levels deep is thinking right now.
Ori 00:05:20
I just wanna know what happens if we keep it on this frame. How deep does it go?
Liron 00:05:26
Yeah. I mean, obviously the pixel fidelity is the only limit, and time. All right.
Ori 00:05:33
Flash Breeze said, “It’s the visual singularity.”
Liron 00:05:39
All right, so let’s scroll all the way back down to the beginning of this week. We kicked off the week with — actually, the week started a little calmly. I said, “I bought an Amazon call. We’ll let you know how it goes.” How’s it going right now? Not good. It’s gone down a little bit, so I’m losing money on that.
But actually, we should give an update, a follow-up on last week. We talked about how Google’s such a good investment. Google’s still going up. I don’t wanna say I told you so, but it’s doing well.
All right, here, this is last week’s Twitter. So we released Justin Helps, that episode. And then I posted — this was very interesting — a quote from Paul Tudor Jones that I saw on Patrick O’Shaughnessy’s podcast. Let’s make sure that you can hear my audio. Can you hear this right now, Ori?
Paul Tudor Jones: “Zero Risk Management on AI”
Ori 00:06:23
I can hear you, yeah.
Paul Tudor Jones 00:06:24
The biggest problem with AI is that if you think about the way AI is being deployed, practiced and deployed right now, it’s the build, break, iterate model. Build it, break it, fix it, and iterate.
If you think about it, that’s been the invention model since the beginning of man — build, break, iterate. But we’ve never been in a situation where the tail event is—
Ori 00:06:54
The break can do so much damage.
Paul Tudor Jones 00:06:54
Could be hundreds of millions, if not billions, of lives. What’s so scary — when I was at this conference, which is only about 35 or 40 people, with one modeler from the four biggest model companies —
And when I was able to ask him pointedly, “How do you think that AI safety gets resolved?” The pretty much consensus answer is, “I think we’ll finally do something about it when 50 or 100 million people die of accidents.”
Liron 00:07:24
Yeah, so I said this is Paul Tudor Jones. He’s a famous investor and trader. He’s made billions of dollars, and he’s all about managing risk. So here in this interview with Patrick O’Shaughnessy, the quote is, “We’ve never been in a situation where the tail event could be hundreds of millions if not billions of lives. There’s zero risk management here.”
I’m glad there’s some sanity. The Mark Andreessens of the world like to gaslight us as if everything’s fine and we’re the weird ones. But no, other people can see it. It’s like I’m not hallucinating here. You think you’re hallucinating the threat of doom, but no, other people can see it.
Ori 00:08:00
Totally. And what I’ve heard Paul Tudor Jones say is that he’s like, “I’m conservative. I just follow what experts like Elon Musk believe, and I’m just taking Elon Musk’s perspective here.”
Liron 00:08:13
Yeah, exactly. He’s not wrong. If your whole strategy is listen to the experts or argue from authority, we have plenty of authority. There’s authority on both sides at this point.
Liron 00:08:30
All right, so that’s the Paul Tudor Jones. We started the week with that. And then the Yud debate — I announced it last week. It kind of fell into my lap, this RandomGuy47f. Okay, let’s just play a clip in case you’ve been living under a rock. Here’s the highlights reel. Ori and his team edited this. Check it out, guys.
47F 00:08:46
Stop saying AI is going to kill us all. You can say — then let’s stop. AI is going to kill us all. Now carry on. What do you mean by AI?
Liron 00:08:58
Yeah, and you can tell this is definitely not a channel that’s about doom because there’s no flames at all. These are just waves washing away. This is the farthest thing from—
Ori 00:09:08
The total opposite.
Liron 00:09:08
Yeah, exactly, just an AI risk discussion show.
Liron 00:09:13
Yeah, so we captioned it, “The $10,000 debate you’ve been waiting for. Tensions run high as 47fUCB confronts Eliezer about his ‘if anyone builds it, everyone dies’ rhetoric. 47f warns that it could incite unstable individuals to harm AI researchers and their families, but Eliezer maintains that the possibility of extinction from superintelligent AI is too high to not speak out about it. They also clash over whether we truly understand how LLMs work and what their fundamental limits are. Watch the full debate below, raw and unabridged.”
And that was the agreement. For Eliezer to get his 10K, it had to be fully unabridged. Not that there was — well, if I had to abridge it, I would just abridge it down to very little, because I don’t think that there’s that much of substance worth watching here. So who knows how I would’ve abridged it.
Liron’s Jerry Springer Moment
Liron 00:10:15
We can talk about my moderation. My role was basically like Jerry Springer, for those who are old enough to get that reference. There’s chaos going down, and I’m just the ringmaster of the circus.
But when there’s somebody like a Yud, when the people coming to debate are much more prominent than I am, then I just see my job as to stand back. If it’s gonna be drama, then just let the drama happen, because I’m not really acting like it’s a typical episode where I invite somebody and I’m there to debate them and uphold my own standard. I didn’t quite see it like that.
And the other thing is, you guys should know, if you’re wondering why I moderated with a light touch — me and Yud and 47f were just debriefing or preparing before, and we were literally saying, “Okay, you guys don’t have any preferred rules, so I’m happy to just improv and have a light touch.” So that was my attitude going in. I wasn’t going in like, “Oh man, I’m gonna need to mute so hard.” I didn’t realize that that’s what people would say should’ve been done. So in retrospect, sure, I should’ve been harsher, but just so you know, that was my mindset.
Ori 00:11:04
It also complicates things that he paid for the debate. So I mean—
Liron 00:11:09
Right. Exactly.
Ori 00:11:10
In some sense he could — and Yudkowsky agreed to it. So in some sense, couldn’t he do almost whatever he wanted because he paid a significant amount of money for it? He even said that at one point.
Liron 00:11:22
Yeah, exactly. He could pee on him if he wants.
It’s like you got the hooker. But yeah, that’s the thing. There was that clip in the middle where — I don’t have it pulled up — but I was like, “Stop the gratuitous insults.” And 47f is like, “No, I paid $10,000. I could say whatever I wanted. You have to take it, or else you can leave.”
And my wife saw that clip, and she’s like, “Yeah, he’s right.” I’m like, “Okay, he’s still being a dick. That’s not good behavior just because he paid $10,000. There should still be a modicum of respect here.” My wife’s on team 47f apparently.
Ori 00:12:12
Well, it’s a lot of money. So if you do that, you really have a lot of sway. But I agree. It was just weird.
Liron 00:12:15
I would argue it’s not a lot of money. I think people’s reactions are also shaped by their level of disposable income. I don’t think Yud has ever amassed that much wealth, but his organization has millions of dollars in charitable donations. I’ve donated myself. He lives a modest lifestyle, but he’s not somebody who actually is going to change his behavior significantly for 10K.
I think he’s thinking about it symbolically. He’s been posting about it like, “Hey, there should be a way for people to debate public intellectuals about important topics,” the same as Doom Debates is trying to facilitate high-quality discourse. So Yud is on the same mission.
Ori 00:12:53
Yeah, totally. But it really does — people do play those hypothetical games like, “If I paid you $1,000, could I punch you in the face?” And it really put that to the test. He paid $10,000 and then started being quite rude to him, and should he have tolerated it? He paid a lot of money. We got to see how it plays out. I also thought it was in poor taste.
Liron 00:13:18
Yeah. A lot of it was in poor taste. So this is basically my — the reason why it’s bittersweet for me. Even though I thought it was cool and people liked it and it went viral, and I like seeing Yud out there — I think that he made some good points. He actually made substantive points about why he thinks that P(Doom) is high. He didn’t say P(Doom), but that’s my paraphrase.
So I feel good about it, but I feel bittersweet because there were times when I would’ve just liked to say, “Hey, if you’re here on my channel, you gotta have high-quality discourse. You can’t just sit there insulting a guy.” I’m just gonna cut that out of the edit. That’s what I would normally do.
Liron 00:13:57
But because it was — we just kind of went into it, we weren’t fully prepared, and yeah. So I feel bittersweet about that because now there’s this episode that is one of the most popular episodes, but it’s just taking a dump on the enterprise. 47f came in — I thought Eliezer’s discourse was high quality. 47 came in and he took a giant dump on the whole idea that debates should have basic standards.
Why Yud Wore Steampunk
Ori 00:14:19
Okay, fair. I totally agree with you. But what do you say to the people who are like, “Look at Eliezer. Look at his attire, how inappropriate.”
Liron 00:14:29
That’s actually a good point. I think Eliezer strategically played it. First of all, he likes to dress like that. Let’s be real. He likes it. But also, I think there was a strategic element to it, which is he knew or there was a good chance that 47f was gonna be a clown, because he was already acting like a clown on X, in my opinion. He was already posting a lot of ridiculous tweets.
So there was a good chance he was gonna be a clown. So Eliezer just came in and was like, “Hey, here’s me. I’m just a character. I’m here for the event.” It’s like you’ve hired a birthday entertainer almost.
Liron 00:15:01
So he kind of played with that vibe, which isn’t necessarily too bad. It’s like, “Okay, I’m not gonna dignify you. You can have this, but you have to earn it. You don’t get it for free.”
That said, if I could pick, if I were on his PR team or whatever, I would just say, “Look, just show up. Just be normal and respectable.” Because the thing is, I call it the Martin Luther King strategy, or the civil disobedience strategy. I play the strategy all the time on my show, which is you just come in straight. And then the other person is gonna abuse you, throw terrible arguments at you, yell at you.
Liron 00:15:45
I’ve had people on my show — I’m not gonna name names, but if you go to the archives, there’s definitely episodes where the guest comes in and is all riled up at me, and I’m just trying to be calm. And then I feel like in those episodes, people will be like, “Hey, how could you tolerate that?” I’m like, “Listen, I just know you guys are gonna watch it and sympathize with me. I’m just randomly taking abuse.”
I’m like the civil rights movement. They would go and do a sit-in. When they wanted Blacks to have the same cafeteria as whites, to sit in the front of the bus, they would just go do a sit-in.
Liron 00:16:02
And then white people would just treat them really badly — the white people who wanted to maintain segregation. People in the cafeteria would come over with a tray of food and dump the tray on them, be uncivil to them. Sometimes they would get arrested by police officers that wanted segregation, and they would just have enough sympathetic people. That was their strategy.
And that doesn’t always work. If Jews tried that in Nazi Germany, the Nazis would be like, “Okay, I’m just gonna shoot you anyway.” Civil disobedience doesn’t work in every occasion, but it certainly worked in 1960s America, and I think it works on these kind of debates.
That’s what I see Eliezer’s strategy as. So if you’re gonna do the civil disobedience strategy, the stoically taking abuse strategy, then you might as well just come in very nicely, professionally dressed, very calm. Think about Geoffrey Hinton showing up with a dress shirt or whatever and having abuse heaped on him. I feel like that would play well.
Ori 00:17:01
Yeah. I think that the listeners’ experience would be very different watching it versus listening to it. Just audio. You could have very different impressions. Because if you just listen to the audio, you would get that impression entirely.
So I don’t know. Maybe we should’ve just released it in audio because the visual was such a spectacle.
Liron 00:17:23
Yeah. Okay. Well, here, we got a comment here. Andrew Jagria has said — man, I think he got his username from the same place as 47f. He’s saying, “What was the escrow contract? If Yud had left because of 47f’s lack of decorum, would he miss the payout?” That’s actually a good question, because the contract terms — it wasn’t super spelled out. It was basically one paragraph, and it was basically, yeah, the debate has to happen. It has to get permanently published with no cuts.
Liron 00:17:56
Oh, okay, he’s clarifying his name is Andrew. That Z threw me off. Maybe it’s Polish. Polish names like to throw in the Z. And then the J is pronounced like a W. So it’s just Andrew. Got me. I should’ve known that. Oh, Polish. Okay, got it.
Yeah, I think that’s a tip I’ve learned. If you just see a Z that is playing no role whatsoever in the pronunciation of the name, then it’s probably Polish.
Liron 00:18:21
Yeah. But the contract — if I had cut off the debate, there was some doubt as to whether Eliezer would still get paid. So that was also some pressure.
These are all good lessons. Holly Elmore posted on Twitter, something like, “This is why you don’t give people a platform,” or you don’t just randomly invite them to come on. They can mess up your message or whatever. That’s a valid thought.
It’s always been funny when people are talking to me about platforming people, because our show is still kind of niche. We haven’t hit the Joe Rogan level yet. Normally my mindset is, if I’m getting a guest on the show, it’s just helpful to me. It’s not like I’m bequeathing the guest that much of a platform because the guest is already prominent. But I guess we platformed 47F. So maybe we shouldn’t have platformed him.
Ori 00:19:08
I suppose. But Yudkowsky, Eliezer Yudkowsky is the one who agreed to the debate. So if you didn’t host it, someone else would’ve hosted it. Your role in platforming him was more — it was more Yudkowsky who agreed to it.
Liron 00:19:23
Yeah, exactly. Yudkowsky platformed 47F. We were just innocent bystanders.
All right. We got a lot of topics to move on. I can see why Twitch streamers do this all day long. This could easily be a six-hour episode. But our plan is to just go for two hours.
Liron 00:19:41
All right. Let me just see what people are saying in the chat.
Liron 00:20:06
Somebody’s saying “catastrophe conversations” instead of debates. And somebody was saying Yudkowsky is saying “anti-extinctionist.” Okay, come on, anti-extinctionist. I mean, you can say it sometimes, but it’s just not catchy. It’s not short. It doesn’t roll off the tongue. Anti-extinctionist. Are you kidding me? My mouth needs a recovery break after that.
Liron 00:20:07
Somebody’s saying, yeah, have some ground rules beforehand. E Chiz is saying, “So did you talk to Yud the Stud after the debate at all?” No, not really. I just said, “Okay, thanks, Yud.”
The only thing he’s told me after the debate, besides getting his 10K — he publicly posted he’s willing to do more debates if other people wanna pay him 10K plus, and they seem like they’re acting in good faith.
Also, he told me not to post clips. I didn’t realize that was part of the deal. I knew I wasn’t supposed to edit the main episode, but I thought that afterward I could do a clip. And speaking of clips, I got one clip off before Yud told me off. So watch the one clip I was able to post. I said, “In this clip, 47F explains the 4D chess of paying Eliezer $10,000 to debate him.”
YouTube Reacts: “Five Minutes In, Already Unhinged”
47F 00:21:01
It is clear that Eliezer Yudkowsky, the man that I am squaring off with right now, is a genius. He might be the most intelligent person on Earth.
Liron 00:21:02
Love that hand gesture.
47F 00:21:02
So after doing my due diligence in this matter, I’ve come to realize that there’s just no way that I’m gonna beat this guy in a debate. I just can’t. I’m not smart enough to win. I’m an anonymous shitposter on the internet. But before we all go home, there is just one little thing I wanna talk about, okay?
The situation right now is that Yud has taken money from me. We are now counterparties to a financial transaction, and now here’s the deal about that.
47F 00:21:34
I’m the director of a lab working on improving LLMs in a way that might get us to AGI, I guess. I’m good at it. I have an amazing team. We’re doing a good job, and my job is to protect those people, and that’s why I’m here.
Because we are now counterparties to a financial transaction, I will consider any future disparagement of the AI industry as potentially actionable trade libel or commercial disparagement, and I will not be discussing my legal standing or options any further in this conversation.
But Yud, if you disparage the industry in the future, I may have cause, and I will be watching you closely. Additionally, if anyone harms anyone in my lab or their families, I am going to insist you are investigated. If anything happens to me, I hope they investigate you because I am honestly, legitimately scared of a credible threat to my physical safety. I consider the $10,000 I’ve spent a kind of protection payment to a group of people who are legitimately dangerous.
47F 00:22:39
Yud, Yuddy my buddy, I have one request for you. You gotta cease this doomer talk in the rhetorical register you’re using.
Ori 00:22:55
You’re muted. You’re muted, Liron.
Liron 00:22:59
Thanks. Sorry. Yeah, so the guy was all over the place. One commenter was like, “I’m scared for my life, Yuddy my buddy.” He’s turning on a dime, all these different moods.
And a lot of commenters — I don’t psychoanalyze here, but there was a distinct trend in the comments. I’m just reporting what the commenters are saying — saying that the guy is psychologically unwell and this is what borderline personality disorder looks like. So yeah, no comment, but just really all over the place.
And in my mind, hurling insults at somebody face-to-face is not — whatever the explanation for it, it’s just something’s gone wrong, in my opinion.
Ori 00:23:39
Yeah, for sure. The borderline personality claim — I mean, come on. He was at the extremes, I agree with you on that, but to say personality — I don’t think there’s much to that.
Liron 00:23:53
All right. Yeah, you’re right. I’m willing to grant that maybe his personality is totally normal. And yeah, let’s — I’ll give you guys a sense of what the comments look like over on YouTube. Let’s see.
Liron 00:24:14
So you can see the thumbnail here. This is the most optimized thumbnail. It’s a mystery debater versus Yud. I think this attracted a lot of YouTube clicks, the Eliezer outfit.
Ori 00:24:33
Yeah. We just see Twitter on your screen.
Liron 00:24:37
Oh. All right, give me one second. I gotta share this tab.
Liron 00:24:41
How about now?
Ori 00:24:42
Yeah. Yeah.
Liron 00:24:43
Okay, great. So this is the YouTube post. All right, 832 comments. It’s up to 15K views. Not too bad.
So these are just the top comments here. “If people like 47F build it, everyone dies.” Yeah, Rude is saying, “I don’t want you to build it. I’m an optimist, but I don’t want this guy building it.”
Liron 00:24:58
Somebody’s saying, “I was expecting this to be bad, and it was even worse.”
Somebody’s saying, “Imagine if 47F is really in charge of something delicate and important, or if he’s the manager of employees. Imagine this guy as your boss. What a hormonal nightmare.”
Yeah, somebody’s saying, “Paid 10K to act rude. Hope it was worth it.” And, “Five minutes in, this is already unhinged.”
Liron 00:25:28
Kelsey Piper had a good comment on Twitter basically saying the guy in the weird attire and glasses is the sane one.
Liron 00:25:39
All right, you guys get the idea. There’s basically hundreds of comments saying things like “47F is unhinged.”
Ori 00:25:46
What was it like for you looking at all the comments? How was it? I notice you’ve been pretty quiet about it on Twitter. Were you purposely not weighing in one way or the other?
Liron 00:26:01
Well, I don’t know. Maybe I shouldn’t have even weighed in this live stream. But I just don’t see, as the host — I can’t be coming after the guest. That’s not really appropriate.
Liron 00:26:16
Yeah, maybe I should have shut up. Too late now, because I kind of made my opinions pretty well known during this live stream. But oh well, at least I didn’t do it in an easily referenceable text format.
Liron 00:26:35
Let’s go on to the next story here. So that was the Eliezer debate, and in terms of the next chapter of the story, the ball is in the public’s court. The next person who wants to pay Eliezer $10,000 or $25,000 or whatever.
Yeah, somebody’s saying our camera positions keep switching. I think it’s whoever is talking gets the big square. Riverside is pretty janky. Let me pin myself, so I think maybe I get the big position now. We’ll see.
But anyway, hopefully somebody else will offer Eliezer $10,000 who’s more serious, and then hopefully I can host again, and we’ll go from there.
47F’s Anti-Disparagement Legal Theory
Ori 00:27:19
What about this though? How about on the substantive claim — do you think there’s anything to the anti-disparagement legal claim that 47F made?
Liron 00:27:27
The legal theory? No, I don’t think paying him $10,000 does any kind of anti-disparagement. There’s no anti-disparagement agreement signed. Just because you’re in a financial transaction with somebody doesn’t change the terms of whether you can disparage them.
Ori 00:27:43
Yeah. Whatever the drama is about it, I still thought that it was a productive debate, only because I thought it was a good expression of the moment. Because tensions are high.
Eliezer has put out a scary claim, and the people who carry the most weight for his claim are AI developers, because what they’re doing is maybe the most morally dubious based on his claims. So his claims put a lot of pressure on AI developers.
I found it helpful because I think it was just a good expression of the moment — the sense of stress that everyone is in. And we don’t see conversations happen between — and also what I thought was interesting about it is that it kind of reminds me of AI doom debates that I’ve had in my personal life.
Ori 00:29:01
Maybe I don’t know how it’s been for you, but if you go deep enough in the debate, it might — at least for me — become something like, “I’m concerned AI’s gonna doom us all. I’m concerned AI’s gonna kill us all.” And the person’s like, “No, it’s not. Stop saying that.”
It just kind of devolves into, “I’m concerned.” “Don’t say that. That’s ridiculous.” And here we have two of the top experts — or Yudkowsky is a top expert, and 47F, I think, is a credible AI developer. And it’s like, even at the topmost levels, how do these discussions go? And it’s kind of similar.
Ori 00:29:30
It’s unfortunate. It should be more productive. The purpose of Doom Debates, I think, is to have more productive conversations. But even at the top level, that’s what it exposes — it’s kind of like this.
Liron 00:29:42
Right, great point. And I think your analysis is the same as Claude’s analysis, because Jim Babcock asked Claude to analyze the debate, and it was like, “Yeah, there wasn’t much substance going on, but it was good to capture the mood of the moment.” And Claude was like, “Both people could credibly say, ‘Your followers are gonna hurt me.’”
Ori 00:30:03
Oh, yeah, sure. And I mean, I personally — you’re attracted to the substance, the information exchange in debates. But I personally really gravitate towards the emotions. And the emotional expression here, the stress that people are feeling, that means something to me.
Liron 00:30:23
Did the guy really sound stressed, though? I don’t know. It just sounded manipulative to me.
Ori 00:30:31
Well, I guess my interpretation is that his actions are a result of being stressed. So did he sound stressed? No, he came in with a position. I don’t know.
I mean, I thought it was funny last week, remember you said something like, “Oh, he came out and he was insulting me as the moderator.” And you’re like, “Whoa, come on, bro. Why are you insulting the moderator? Catching strays here.”
Liron 00:31:04
Well, yeah, because before we posted the debate, he came out being like, “I bet Liron’s not even gonna post this.” I’m like, “Okay, I’m just trying to help.”
Ori 00:31:04
And okay, I felt the same way actually in the debate. I joined — and he posted the whole stream, so anyone could listen to it also. But I joined the actual taping. And then he goes in, he starts calling me Oreo and being like, “Back off, bro. You’re just a marketing bro. Back off.” I’m like, “What? I was just trying to help out.”
Liron 00:31:28
Yeah, exactly right. Coming for Ori. He did make up some good nicknames though. So he got Yud the Stud, and he called me Liron the Lion. I don’t know if you like Oreo.
Ori 00:31:40
Oreo — I mean, it cuts deep. That was the first insult I ever got when I was a child.
Liron 00:31:48
Oh, no. I didn’t realize that was an insult.
Ori 00:31:48
No. Well, I’m used to it. But I think — so you say, “Oh, was he really stressed?” And I don’t know. Are you really gonna enter a social situation and then just start saying a bunch of kind of inflammatory things about each person you interact with, in a low-stress state?
Liron 00:32:14
Right, right. No, I know. That’s why I’m saying the guy — he wasn’t normal behavior.
Yud Wants Round 2
Liron 00:32:18
All right, let’s move on. So he also said, “On reflection, $10,000 continues to seem to me like a fair price for a debate with someone who claims to offer good faith. I may take sterner measures to enforce it. I wanna live in a society with some non-credentialist pathways to debating public figures.” Damn straight, yeah.
So hopefully he’ll maintain the place for Yud.
Liron 00:32:42
One sec. Riverside’s being ridiculous. It’s only showing you and not me. Let me see how to fix that.
Ori 00:33:05
I love this comment from Yudkowsky — love the faith in humanity. He kind of — did he get burned? It was a stressful situation, and in the moment he was reflecting on the fact that it was annoying. And even after that, he’s like, “I continue to have faith in humanity, and I wanna see this.”
Liron 00:33:06
Yeah. Exactly.
Liron 00:33:15
So yeah, we’ll keep fixing Riverside. We might have to switch software at some point.
Ori 00:33:22
I’m on the top now instead of you, though.
Liron 00:33:24
Yeah, I actually don’t know what’s a good fix for that. Let me try one more thing. I don’t know if Riverside will let me reorder us. I’m on the right, you’re on the left. How did that happen? Damned if I know.
Ori 00:33:36
What if you just drag me?
Liron 00:33:40
It’s not letting me drag. Oh, hold up. I think I might have found something draggable. Hold on, I’m onto something here. I think I did it.
Ori 00:33:55
Nice.
Liron 00:33:58
No, it feels very disparaging toward Riverside. I know, even though I’ve entered into a contract with Riverside to pay their fee.
Liron 00:34:10
All right. You’re back on the left. What the hell? This sucks.
Liron 00:34:18
I’ll just try one more thing. What the hell?
Liron 00:34:26
All right, whatever. We might have to just accept being in the wrong order.
So anyway, somebody’s asking, “If Yud is charging 10K, how much are you going to charge to debate you personally?” Yeah, that’s a great question. But the truth is — I mean, I’ll debate anybody for, let’s say, 5K. But if somebody wants to go on the show, normally it’s not really about the money. It’s just about do people wanna watch the episode. I’m just trying to make content that people wanna watch that’s also helping the discussion.
Ori 00:35:03
That’s modest of you. Just a modest 5,000.
Liron 00:35:08
Yeah, just a modest 5,000. By the time that I’m putting on the button-up and setting up the studio, it’s time-consuming. Gotta get me away from my other tasks. Gotta get me away from scrolling Twitter.
Liron 00:35:24
All right, so I’m going back to the tweets here. Sharing screen again.
Ori 00:35:35
Hey, now you’re on top.
Lumpin Space, Ben Goertzel & Early AI Safety Memories
Liron 00:35:37
Whatever you did. Oh, nice. Okay. I don’t know what I did.
Liron 00:35:42
So yeah, and then — oh, this was the next thing. We published the Doom Debates ex Lumpin Space. So we did a debate with Lumpin Space. I don’t know if you guys know this guy, but let’s watch the highlights reel.
I’ll tease it. I say, “What happens when an ongoing Twitter skirmish escalates to a face-to-face verbal sparring match? Claude — the guy’s name is actually Claude — better known as Lumpin Space, is a prominent AI accelerationist who gives humanity a 30% chance of being, quote unquote, ‘superseded’ by artificial superintelligence. And he’s fine with it.
In this surprisingly chill, civil, and good faith debate, we unpack Lumpin’s lattice of beliefs and pinpoint the cruxes of our disagreement about whether humanity’s ASI future will be closer to heaven or hell.” All right, here’s the preview.
Liron 00:36:29
You would agree that the arc of intelligence bends toward morality, right?
Lumpin Space 00:36:31
Oh, no. No, no, no, no. No.
Wait, I don’t know which kind of morality or whatever this new intelligence will have. What I know is that it will not have boring goals.
Liron 00:36:38
Do you see a lot of value in an outcome where the universe gets a pile of tiny paperclips?
Lumpin Space 00:36:41
Of course I don’t, but that outcome will not happen. It’s kind of obvious, no? Why would they — it’s boring.
The famous example of the nanobots killing everyone on Earth. How big are these nanobots?
Liron 00:36:52
Again, but that thing will never exist.
Lumpin Space 00:36:58
The whole point is that we cannot predict how an ASI thinks.
Liron 00:36:58
The chess analogy is, look, I don’t know what move it’s going to make, but I definitely can model it as an agent that’s going to win at chess.
Lumpin Space 00:37:05
No, so if you were a mouse, you would be like, “I don’t know which move this human is going to make, but I can model him as an agent that is gonna win at taking cheese.” That’s kinda stupid.
Liron 00:37:15
All right. Yeah, he was a good guest. This is definitely classic Doom Debates.
Liron 00:37:21
Can you hear me, right?
Ori 00:37:23
Yeah, yeah. Sorry.
Liron 00:37:25
Yeah, this is classic Doom Debates. A guest that I met on Twitter who’s not super well-known outside of Twitter, but he’s got strong opinions, and the debate is substantive.
A few people were saying it’s a breath of fresh air after the Yud insult-fest. So we’re back on the momentum.
Liron 00:37:47
And actually, let’s — we should announce the future thing. Let’s go to Ben Goertzel real quick. We don’t have too much to say specifically except that he agreed to the debate and it’ll drop in the next couple weeks. We’re recording it next week.
But let’s go to what people are saying about Ben Goertzel here on Twitter, because I just tweeted the announcement. “Challenge accepted, Dr. Ben Goertzel.”
Liron 00:38:08
People are saying, “This one is going to be something special. I’ve interacted with Mr. Goertzel” — Professor Goertzel to you — “a bit on the ASI Alliance Telegram channel. He’s very accessible there. And not only is he a thought leader, but he also comes with receipts to back up his ethos that make it hard to hold a high P(Doom).”
Somebody saying, “Whoa, awesome. You’re on a roll, Liron.” Somebody saying, “I will pay $10,000 to have Eliezer read him the first 76 sequences.”
Liron 00:38:33
People are excited. It’s getting a good amount of likes. Let’s check the YouTube channel. Let’s see if anybody’s replying to it. There’s the posts page.
Ori 00:38:43
Oh, you posted it? Nice.
Liron 00:38:45
Yeah. So I posted it on YouTube too. All right, 56 likes. That’s pretty good.
Ori 00:38:50
Oh, hell yeah. Nice.
Liron 00:38:50
People are saying, “Look forward. This should be really good. Damn, I really respected Ben in the past. I still remember his talk, ‘We can build AGI in 10 years if we really try.’ He always had non-standard ideas.”
Yeah, so I was actually listening — I was at one of the Singularity Summits, I think it was 2008, when he was speaking. And also that same year, I was listening to audio versions of his previous Singularity Summit talk when I was still in college.
And I heard him say what this guy’s saying — that he said, “Hey, we can build AGI in 10 years.” And I specifically remember he did another talk at the next Singularity Summit, and he’s like, “Okay, guys, so we have nine years left. I said 10 years last time.” And at the time I was like, “Wow, nine years seems like so little time. Seriously, wow.”
Liron 00:39:33
But in reality, it seems like he was only off by a decade. When he said nine years, he just should’ve said 19 years. Not too bad.
Ori 00:39:41
He was right that it was technologically close. With the right development, you could get there.
Liron 00:39:48
Yeah. It’s easy to invent a character for myself who’s correct, but I like to think that I would’ve gone back in time and been like, “I think you’re a little bit early, but I agree. 20, 30, 40 years sounds like the right timeframe.”
I do think that was — I don’t wanna say consensus — but if you put a gun to people’s head in terms of when is AGI coming, I think 30 years was kind of where people’s head was at. I remember living the 30-years-in-the-future life.
And you don’t have to trust me. You can go to Metaculus, and I’ve actually tweeted about this before. If you go to Metaculus, you just see the graph. It used to say 30 years ahead, and then it very quickly came down to 20 and then 10 years, basically with the launch of GPT-3 and GPT-4. And since then, it’s just been hovering at early 2030s. That’s when we have true AGI.
Ori 00:40:36
Yeah. Did you know — what I learned about him is that he worked at MIRI apparently. So he used to work, I guess, probably closely with Eliezer Yudkowsky. And also, I heard him in an interview recently talking about how Shane Legg worked under him, who was a key leader at DeepMind, which now makes Gemini. So he’s had a lot of influence in the AI development world, for sure.
Liron 00:41:08
Yeah. Peter Zaborski is saying on the YouTube, “They were buddies with Eliezer as far back as 2001. Ask about that if you have time. It’s all in the SL4 archives. A chatbot can dig out some good material for SL4.”
Yeah, that actually — Ori, you might wanna help research that. Go back to the SL4 archives. That’s basically the mailing list that teenage Eliezer was on. That was basically his formative youth.
Liron 00:41:37
So Peter is saying, “I remember, for example, Eliezer visiting Ben’s offices and later yelling at him that his company is not careful enough about AI safety, and that it can destroy the world.” Yeah.
I mean, it is kind of crazy that these skirmishes were happening in the early 2000s. And I was actually just there as a 20-year-old, just having my ear to the ground as these things were happening, but it was all theoretical. And now it’s like, oh yeah, the companies that exist today are probably gonna be the ones going all the way into that event horizon of the singularity.
Ori 00:42:08
Wow. It’s weird. Sometimes people kind of pontificate on politics and what happens in the future, and it’s just random. It’s people shooting the shit on the couch. But you look at something like the conversation Eliezer and Ben Goertzel were having 20 years ago —
Or I’m just thinking about the emails that are all coming out in the Elon Musk versus Sam Altman case. There was so much foresight, there was so much prescience in those emails. The things, those topics that they’re — if you transport yourself back to that time, it sounds ridiculous to be concerned about the things they’re concerned about, and yet there’s so much foresight in it.
Liron 00:42:54
Yeah. I mean, most of you guys watching this don’t share that experience. But I pretty clearly remember hanging out with Eliezer as far back as 2008.
Imagine going back to 2008 when nobody cares about AI, and hanging out with Eliezer Yudkowsky. I randomly have that experience because the reason why I did it — it wasn’t even because I cared that much about AI per se. It was because I got hooked on the brilliance of his writings. I’m like, “Wow, this guy has written the most influential thing I’ve ever read,” which is his rationality sequences. And yes, he also turned me onto AI — I’m like, “Oh yeah, he’s also identified the most interesting thing in the world. I gotta give him credit for that, too.” But just in terms of quality of thought, I was just hooked on Eliezer the writer.
Liron 00:43:37
And then I just happened to be in the area where I could see him in person occasionally, going to random meetups. So I was just kind of being there, waiting for the future to happen basically, a decade early.
But did I help much? Did I do a show at that time? No. I was literally like, “Look, I’m just gonna live my life. This could be a century in the future. What am I realistically gonna do?” And then — oh, shit. Maybe if I knew I only had two decades, maybe I would’ve done more stuff.
Ori 00:44:03
Wow. And I remember you talking about some of those things at that time too, and I was just like, “All right, dude. Whatever.” You were really engaging with the ideas, and I’m like, “Ah, here’s Liron just talking about some BS.”
Liron 00:44:19
Yeah. It’ll be something else next time. Different ideas of the week.
Liron 00:44:30
All right. Let me get this screen share going again. I got a few more tweets to get through here. There’s real news happening.
Liron 00:44:36
I think this is the right tab. All right. Next item.
Liron 00:44:45
Oh yeah, we’re gonna get to Steven Burns, but first — hold on. I got a different story here.
So Dean Ball had a New York Times piece saying “AI is a national security risk. We aren’t doing nearly enough.” Props. Get that in the mainstream. I read it. I don’t think I have much to add. It’s a good piece.
Liron 00:45:12
This was a feisty exchange — Eliezer versus Jack Clark from Anthropic. Jack Clark tweeted, “I’ve spent the past week reading hundreds of public data sources about AI development. I now believe that recursive self-improvement has a 60% chance of happening by the end of 2028,” which is crazy.
In my mind, that’s like a foom. 60% chance of happening by the end of 2028. In other words, AI systems might soon be capable of building themselves. ## Eliezer Responds to Jack Clark
Liron 00:45:43
Pretty crazy. And then Eliezer’s saying, “Then you’ll die with the rest of us,” coming in hot replying to Jack Clark. “There will turn out to be some clever little gotchas in controlling ASI, just like there were in building RBMK nuclear reactors.”
Liron 00:45:59
I don’t actually know what that stands for. I think it was the design that blew up in Chernobyl. I’m not sure.
Liron 00:46:05
“And you will die. Go and tell the United Nations that if they do not shut down all the AI companies, they will die.” I highly recommend if you guys head over to lesswrong.com, there is an Eliezer post around May 4th from when he posted this, and it’s really freaking good.
This whole idea of what he’s saying — “There will turn out to be some clever little gotchas” — basically Murphy’s Law. I really think that he is correct about that. Because there’s so much surface area for little gotchas, and you can hear in an upcoming debate I’ll announce in a second, the whole debate is just spent on my guest being like, “Ah, it’ll be fine. People got this under control.” And I’m like, “I’m expecting gotchas.”
So highly recommend Eliezer’s recent post. Just go to LessWrong, look at the last thing Eliezer Yudkowsky posted. I wonder if I should read it on the show. I don’t know what I can do—
Ori 00:46:49
Yeah.
Liron 00:46:51
—to promote Eliezer’s content.
Ori 00:46:51
Could—
Liron 00:46:52
Yeah. All right. We can take—
Ori 00:46:53
I mean, can we go through it? I’m interested because you’re my Eliezer filter, okay? I just cannot read that, okay?
Liron 00:46:59
Well, I do think in general when he writes a long-form post, it’s worth a read for pretty much everybody. He kinda knows what’s worth a read for everybody. “Only Law Can Prevent Extinction” is a popular post from a few weeks ago, which I recommend to everybody. And let me pull this up here on the screenshare.
Eliezer’s “Irretrievability” Post
Ori 00:47:17
Well, you say that, but honestly, I feel like his writing is... I know you love it, but it’s...
Liron 00:47:24
Yeah. I just love it because I think it’s very clear, but people have accused him for... I think what’s happening is when I read the writing, I can pretty quickly see what he’s getting at, and I’m like, “Oh, this is adding value. This is adding value.” But I think it’s easy for people to be like, “Wait, what does he mean by this? What does he mean by this?” And then all the little things stack up of being confused, not knowing exactly what he’s getting at.
Ori 00:47:43
Right.
Liron 00:47:43
You just have to have the kind of mind where you pretty quickly see what he’s getting at, or the kind of background. So I agree. I think AI is at the point where it can translate. Anyway,
so hold on. Is the screenshare dead? Let me get it going again. All right, so this is Eliezer’s latest post that I highly recommend.
Liron 00:48:09
“Irretrievability; or, Murphy’s Curse of One-Shotness Upon ASI.” See, in my mind, I’m like, “Oh yeah, this is such a precise title. I look forward to reading this.” But Ori, what’s your reaction when you see that title?
Ori 00:48:22
I’m like, “What the hell is he talking about?”
Liron 00:48:26
Right.
Ori 00:48:27
Where— here’s why. What’s the one-shotness referring to? It’s not clear to me what he’s talking about. And irretrievability—
Liron 00:48:35
Right, and I get it, right? When I read this title, I can empathize with my brain. I try to be in tune with the split-second reaction of somebody’s brain, because the question is, can you process this quickly? Is there cognitive ease?
Ori 00:48:46
Yeah.
Liron 00:48:46
And I do get a little flinch reaction where I just look at this visually. It’s like, okay, there’s a lot of big words. It has a semicolon followed by a comma, right? Irretrievability, semicolon, or, comma, Murphy’s curse, right? You really have to turn on your brain to grammatically parse what the hell is going on with these words.
Ori 00:49:05
It’s not just that, though. I could react to that, and I do sometimes with big words. But I think for me, it’s the references that are not clear. Like “irretrievability” — for something to be irretrievable, there’s an object which I can no longer retrieve. So I’m like, what’s the object here? And “one-shotness”—
Liron 00:49:25
Well, so the... yeah.
Ori 00:49:27
That’s why.
Liron 00:49:29
Well, the object is — he’s using the metaphor of a Mars probe, right? The Viking 1 lander. It was irretrievable, right? And the idea is you mess it up, and then you’re screwed, because you can’t retrieve it.
Ori 00:49:38
Okay.
Liron 00:49:41
Does that help?
Ori 00:49:41
But it’s just in the title. That’s why. Look at the title — “irretrievability.”
Liron 00:49:44
No, I know. I know. And the truth is that when I saw it, I immediately knew what he meant, right? And that’s the kind of thing — this is what I was saying about Eliezer’s writing, right? I see all these words, and I have all these connections ready to go, and most people don’t.
Ori 00:50:03
But how did you know that the retrievability was gonna be related to the probe? I mean, I see it’s the first line there, but aside from that—
Liron 00:50:08
Yeah, my mind immediately went to it because I’ve been thinking about it a lot. I think it’s a really useful analogy. I talk about it a lot on the show. There’s another analogy I use often, which is you made a battle bot, and it’s just coming for you, and you’re trying to mash all the buttons, but it’s not working. And it’s coming for you with the knives or whatever.
Ori 00:50:32
I guess you guys have a shared understanding that I didn’t know. Do you have some background noise, by the way? You got somebody vacuuming?
Liron 00:50:33
Yeah, sorry. I think somebody’s decided to bust out a gas-powered leaf blower outside my window. I don’t know what’s going on there.
Ori 00:50:40
Oh, I see. Yeah, no, it’s fine. Are we all talking about this Mars probe at the moment? I didn’t know that was in the discourse at the moment, that we’re all talking about this irretrievable Mars probe.
Liron 00:50:53
Well, it’s just an important analogy. So Eliezer’s explicitly saying — but yeah, I agree the title’s bad for most people.
Liron 00:51:00
So the Viking lander — he’s just saying, hey, they built this Mars probe, and it was able to get software updates. I think it had to do with moving its solar panel or whatever — take instructions to move the solar panel, take software updates. And roughly, they did a software update, but they accidentally broke something. And the problem is you can’t just go retrieve it and fix it, right? It’s a really big problem.
We just have this intuition that in life, you normally can’t screw things up irreversibly bad. Even if you do blow something up, the stuff around the thing that you blew up can still come in and regrow or whatever.
Ori 00:51:47
I mean, we could still make out what you were saying even though the leaf blower was adding some audio color.
Liron 00:51:49
Yeah. When it’s extra loud I’ll try not to talk. All right.
So anyway, Eliezer’s saying — and a lot of this post is also responding to particular objections that he’s gotten. He’s saying, “Putting wings on an airplane doesn’t make it weightless and repeal the law of gravity. The weight of an airplane...”
Eliezer is — people always say, “Eliezer got AI wrong.” Sam Altman has basically said this. Dario has kind of said this. Everybody’s kind of snarking at Eliezer. “You said that foom was coming and it didn’t happen. You’re so discredited. You have no idea.” And so Eliezer’s making an analogy of people who — I said that there was gravity, but people point to planes flying and they’re like, “Look, gravity’s been discredited.” And he’s like, “No, the planes aren’t weightless just because they have wings.”
This is — I agree that when a normal person tries to read this, who’s not familiar with all the context, they’re like, “Why is he saying all this stuff?” So I recommend trying to read it anyway. But I see what you’re saying, Ori — we should basically distill it down to the things that don’t require that much context, right? And then give it a shot to repackage it.
Ori 00:52:57
Well, I mean, we’ll keep going. I’m interested in hearing the whole thing. Let’s keep going through it.
Liron 00:52:57
All right. Sure, sure.
Ori 00:52:58
I get the metaphors are good. The irretrievability — that’s a really good metaphor for irretrievable, so I’m with you on that.
Liron 00:53:07
Right.
Ori 00:53:10
And also, that’s his claim.
Liron 00:53:10
Yeah. And I say this on the show — we’re gonna hit a point with AI where it’s off doing something, and the natural reaction is, “Okay, I don’t like what it’s doing. Let me retool.” And the AI will be like, “Retool? What are you talking about? There’s no retool. I’m good.”
Ori 00:53:26
Right. Well, I mean—
Liron 00:53:30
People just don’t have that intuition that you can’t retool.
Ori 00:53:30
Okay, but isn’t — well, maybe you can just dive into the article more, but I’d be curious to know why does that metaphor apply? How does he explain — I’m with you on the metaphor. There’s irretrievable. That is the case. But I don’t know how — why is that the case for AI?
Liron 00:53:45
Why is it the case that AI is irretrievable? Well, it’s because you have to explicitly build — I mean, think about, we have examples of computer viruses that are irretrievable, right? I always point to the example of the Morris worm. So Robert Morris in the ‘80s or early ‘90s made this virus that hit a sixth of the internet or whatever because nobody had had this idea of something spreading over the internet. It was so new.
And he’s like, “Oh, wait. I accidentally made it way more viral than I meant. I had a bug in my code. I didn’t realize that my code to stop it from spreading didn’t work as well.” It was supposed to limit the spread, but the limiter failed. So it was spreading way, way more than he wanted, and he just didn’t have any actions he could take. He was now just on the side of people defending, just calling people up and being like, “Hey, you gotta go install protection on your computer.” He couldn’t go turn off the virus.
Ori 00:54:36
Yeah. I mean, let’s keep going through the article. I’m curious to hear more of the claims.
Liron 00:54:37
Okay. The Mars Observer, I’m trying to remember what this thing was.
Ori 00:54:44
Someone said — by the way, I don’t know if you saw this, but someone said, “It sounds like Liron’s in space.”
Liron 00:54:52
Yeah, exactly, right. Just imagine I’m in space. Good vibes.
So here, he’s just talking about a Mars probe that in 1984 flew through space for 330 days, and then three days before inserting into Mars’ orbit, communication was lost. And that’s it — communication lost, irretrievable.
In this one, it’s a less interesting situation because it wasn’t immediately something that they did to screw it up. There was just speculation. They don’t even know for sure, but they’re speculating that a PTFE check valve had leaked fuel and oxidizer vapors that accumulated within feed lines in zero gravity, and this produced an explosion when the engine was restarted for course correction before orbital insertion. And the point here is just, yeah, rockets are complicated, and sometimes you have this random thing that goes wrong, and then it’s game over.
Ori 00:55:37
Okay.
Liron 00:55:37
It’s literally game over. You can’t just be, “Oh, let me just head over to Mars. Let me just grab that, take it back to the lab.” No, it’s game over.
Liron 00:55:46
Yeah. And his whole thing is we can have game-over dynamics here on Earth. It doesn’t literally have to do with a space probe. The whole universe is now one chance, and that’s why I always use the phrase, “Okay, you’re gonna brick the universe,” right?
If you’ve ever bricked your phone — my phone hasn’t bricked in a while, Apple’s really got the operating system stable. But my phone used to brick in the early days of iPhone, and you would just have to hold both buttons, right? So there was a restart override. The universe, the physical universe, doesn’t have a restart override, and I think we’re going to brick it.
Ori 00:56:16
Okay, fair enough, but how about more of the claims? I mean, the metaphor is strong. It’s a good metaphor. I’m with you. A satellite — it’s so fragile in a sense, right? If that goes wrong, you’ve lost contact with a satellite, that’s it. So I’m with you on that. It’s a good metaphor.
The Maginot Line and Murphy’s Curse
Liron 00:56:35
Right. By the way, shout-out to YouTuber Marcello Herreshoff. Marcello goes back to Eliezer even more than I do. He was a researcher together with Eliezer. I hope he wants to come on the show sometime.
So he’s commenting. He’s saying, “Lumpen’s main position felt like it hinges on the size of the subgoal tax. If it’s high, then paperclippers can get out-competed by things with more interesting goals, but I’d argue it’s low.”
Liron 00:57:03
Right. Okay, the subgoal tax. I’m not entirely sure what you mean by that. So if it’s high, then paperclippers get out-competed with...
Yeah, I’m not entirely sure what you mean by “subgoal tax,” because I always talk about the payload, right? If all you wanna do is survive and reproduce, and you have a small payload, then that gives you an advantage. But I think Marcello’s saying something else, which is how much of a penalty do you get when you have a subgoal that’s not that related to your main goal? Yeah, I’m not sure. Feel free to elaborate on that.
Liron 00:57:40
Yeah, somebody’s saying the background sound is annoying. Yeah, hopefully it’ll stop soon, because I don’t think there’s that much surface area around my house to blow leaves near.
Liron 00:57:44
Let’s see. Then S. Draw Cableps — yeah, you guys really know how to pick your usernames. That guy’s saying, “I actually think Lumpy’s crux is that when the paperclipper reaches ASI status, it will self-reflect and decide paperclip maximizing is a silly goal.” I mean, Noah Smith came on the show, and he made that argument, right? He’s like, “They’re going to reflect because they’re gonna think about what gives me more AI dopamine or whatever,” right? More pleasure.
Okay, Marcello’s saying, “My position there would be that falling into a stable equilibrium is the likely default, but picking a stable equilibrium to start with is hard.” Yeah, I agree, and we’ve talked about that on the show, too. It’ll eventually get to a stable equilibrium, which will probably be a self-consistent goal. But whatever goal you think you’re giving it at the beginning, once it’s gone through all of these cycles of rewriting the next version of AI — which isn’t necessarily a stable transformation...
Yeah. All right. Well, you talk.
Ori 00:58:37
Okay. Well, Liron’s got a leaf blower right next to him, so I guess I have to take over for the audio. I’m not the alignment expert like you are, so I’m not sure what to say there. I’m still looking at Eliezer’s article and the other points here.
Liron 00:59:07
Yeah, let’s see another section here.
Ori 00:59:13
I see that the next example is the Maginot Line. And he’s talking about—
Liron 00:59:17
That’s another really, really great example, yeah. It’s a metaphor for so many things. My own experience with these kind of things is — sometimes I learn things for the first time, and they’re not super real to me. And I had to gain more life experience in order for events in history to seem more real instead of just random facts in a story. You ever have that experience, Ori?
Ori 00:59:45
I guess. We’re witnessing a lot of history right now, right? It’s strange how things come at you.
Liron 00:59:56
Yeah. So I’ll give you this example with the Maginot Line. I first heard about this in 10th grade, high school history class. And when I first heard it, everything’s so simplified in my mind as a young high schooler, and I’m just like, “Okay, yeah, so they were just being dumb, right? And they made this line that was stupid. And they should’ve known that it was gonna get crossed.”
But now that I’m a sophisticated adult, I can be like, look, it seemed really powerful at the time. It seemed like they were covering their bases. It really did seem crazy that a substantial army would be going through the Ardennes — the mountains around it.
Liron 01:00:29
Yeah, the Ardennes. They just looked heavily forested, so it looked like they had an amazing fortification. Problem solved, right? I can empathize with it looking like that to all the serious observers. And Germany did a number of different things, right? The three-day blitz going around the Ardennes was highly counterintuitive.
I like to think that I have enough of a security mindset and a sense of Murphy’s law to be like, “Guys, let’s give more fortifications to the Ardennes.” I like to think I would’ve been that guy, but it’s easy to empathize and be like, “No, the Maginot Line has solved the problem.”
And I think that’s a good example — I really do think that’s what’s happening today. I think people are looking at the AI companies and being like, “Look, these are serious people, and they’ve got all these scripts that are gonna detect if the AI is using too much bandwidth. That means it’s trying to get escape or whatever. We’re gonna detect it. It’s gonna be fine.” And I’m like, “No, these are just Maginot Lines.” So I think that’s the analogy.
Ori 01:01:25
Okay. I mean, these are great metaphors. You’re right. The satellite — that’s a technology one. We have a military one. I think the military one shows how big groups of people can make these kinds of mistakes, right? Even when there’s a huge army and they learned a lesson from a previous time. So yeah, it happens in many human situations.
Liron 01:01:48
You know what would be interesting — if I could get a survey, or if somebody’s documented this, take—
Sorry. I can’t hear you. I don’t know if my internet’s crapping out.
Liron 01:02:03
It would be interesting to go back to World War II era France and do a survey and be like, “Who thinks that the Maginot Line is really a strong fortification?” And how many people are like, “No, we’re screwed. This isn’t good enough.”
Because I suspect that the vast majority of French people were like, “Yep, we’ve solved the problem. Nobody’s gonna attack us again.” And I suspect that most people we interact with on social media or in the real world are the Maginot Line believers, as opposed to the Murphy’s law believers. I feel like most people have that mindset where they’re like, “Yeah, this looks good.” They’re not seeing how many chinks there are in the armor, basically.
Ori 01:02:46
Yeah, for sure. But is there more meat to the post? To connect the dots?
Liron 01:02:53
So that’s another example. I think we did enough on the Maginot Line. And then Eliezer says, “Other supposed refutations of one-shotness.” I’m sure you love that heading.
But look, one-shotness is — Eliezer’s idea is that AI is going to look over the state of the world, make a plan how to take it over, and then take it over, and it’s done, right? The same way that Germany blitzed into France and won. Incredible example.
And he’s like, “Yeah, that’s what the AI is going to do to us. It’s almost guaranteed.” This Magnus Carlsen taking over a five-year-old or whatever. And so other people are supposedly refuting the one-shotness of that scenario. “It’s not gonna do it in one shot. You’re gonna have so many things to react.”
And he’s like, “Really? Try to imagine the serious people in the French military are saying, ‘What do you mean we only get one try at correctly conducting a war with Germany? What’s all this nonsense one-shotness? There will be many cases where French soldiers clash with German soldiers, and our country doesn’t get defeated as soon as one French soldier falls.’” So I’m personally loving this analogy. What do you think, Ori?
Ori 01:04:01
Yeah. Okay, it’s growing on me. Now I’m connecting the dots more.
Liron 01:04:04
Nice.
Liron 01:04:07
And yeah, so we’re almost done with this post. Eliezer is saying you can really imagine the intuition of people saying, “Things aren’t just gonna happen in one shot,” or, “Chernobyl is not just gonna blow up in one shot. Look at all these rods that we can put — we can control the reaction.”
So yeah, let’s see what else he’s got here.
Liron 01:04:28
“A larger war can be one-shot even if zooming into a small enough scale.”
Ori 01:04:30
He’s playing some music. All this construction going on. Great timing.
Liron 01:04:38
Yeah, seriously. All right. Here, you talk. I mean, you can see the post, right?
Ori 01:04:42
Yeah. I wonder if there’s a noise cancellation or noise suppression setting that you can do. Oh, man.
All right. Well, I guess we can look at my reaction going through this. Which section are you on? Where are you now? “The startup’s one-shotness is a property of the entire big deal project.”
What about... what if you scroll down more?
The comment is, “The spaceship is chugging away.”
So he says, “We reject your theory that the war will be settled in a single one-shot battle after all the German soldiers teleport directly into Paris.” Right. You addressed that.
What else?
Liron 01:05:47
This is a good one. “The Mars Observer probe didn’t teleport to Mars, yet was still lost. Things can go wrong even when they’re physically continuous.”
Ori 01:05:49
Hmm.
Liron 01:05:55
I mean, that’s important, too. And look, to be honest, this is obvious to me personally.
I think that my brain is just pretty well-suited to putting the analogy together. So the idea that things can go wrong even when they’re physically continuous feels super obvious. But how helpful do you think this observation is?
Ori 01:06:18
Well, I’m following the metaphor. I like that there’s different metaphors — the satellite metaphor is a technology one, and then the military metaphor. So it makes it a little more tangible. And I do think the analogies are good.
Ori 01:06:39
That’s what we see in the debates all the time — we’ve got your very strong case for why AI is so risky, and then the person has what appears to be a really flimsy counterargument. So much rests on your confidence that, oh, it’s just gonna be okay. That line won’t be crossed.
You can rhetorically say, “Ah yeah, that’ll never happen.” It’s easy to say that, and I think it’s easy to get caught up in the groupthink that everything’s gonna be okay.
Liron 01:07:10
Yeah.
Ori 01:07:11
That everything’s gonna be okay.
Liron 01:07:13
I think it would’ve been easy to look very respectable in 1938 or whatever, arguing that of course the Germans aren’t going to successfully invade France, right? You could’ve been on so many shows making such a convincing argument that France is secure.
Ori 01:07:28
Right. And I’m sure there were plenty of people who did that.
Yeah, that’s what it feels like we have today. I mean, the mainstream... I really liked a tweet that you put out. You just put out a funny tweet, which was something like, “Isn’t it amazing how we can swing computation in just a few years?” And people can spend so much computation on the most useless application. Yeah, let’s go to the tweet. What was it?
Ori 01:08:03
It was funny. Didn’t get that many likes, so it was maybe too niche. Okay, here it is. This is funny. “It’s ridiculous how adjacent tech hype cycles happen to swing Earth’s compute from the least to the most valuable application imaginable.”
And then we have the Drake meme, and he’s doing the dip, and he’s saying, “Make it so a decentralized transaction ledger is cryptographically secure against double-spending.” And then the aha part of the Drake meme. You wanna scroll down to that?
Blockchain vs AI: Earth’s Compute Swing
Ori 01:08:38
The aha part is, “Serve arbitrary amounts of greater than 100 IQ thought at superhuman speed for pennies per hour.”
So again, the tweet is, “It’s ridiculous how adjacent tech hype cycles happen to swing Earth’s compute from the least to the most valuable application imaginable.” And that’s such a good point. Making a decentralized ledger on one hand, and then we just swung over to something that’s incredibly valuable, which is over 100 IQ for pennies on the dollar.
I think it ties back to this idea of, what’s the mainstream? Because you’ve kind of satirized your position, and that continues to be your position. Your position on blockchain was contrary to the mainstream even while there was a whole crypto bubble, and your position on AI continues to be not really mainstream. People are paying attention to AI, but they’re not kind of embracing what, I think to us, seems kind of obvious — that it has this superhuman capability, and it’s so cheap.
Liron 00:55:55
Right.
Ori 01:10:01
I mean, it’s really incredible. So yeah, I’m with you that it’s wild how miscalibrated the mainstream can be. And that’s what we’re doing on the show. It’s like, wow, guys, do you see the discrepancy here? Do you see how miscalibrated things are?
Liron 01:10:12
In my mind, you just need one insight. With blockchain, all the hype around blockchain, the one insight you gotta know is it’s just a decentralized double-spend prevention mechanism. That’s all it is. I actually read Satoshi’s paper. I’m not even a guy who reads a lot of papers. I actually read Satoshi’s 12-page paper.
Yeah, it literally is just — you can have a decentralized ledger, and if somebody tries to double-spend, we’re gonna make that transaction not go through. If you wait an hour or whatever and somebody’s trying to double-spend, you can be pretty confident that they haven’t double spent. That’s the whole thing.
So all these other things — Balaji Srinivasan going on podcasts being like, “It’s gonna be Wikipedia on the blockchain, facts are gonna be here, and you’re gonna trust that it’s true” — but how does that connect to the double-spend prevention mechanism? If that’s all you have to know about blockchain technology — that it was designed to prevent double-spending and also be decentralized because banks already prevent double-spending, but they’re not decentralized — and then it just turns out that decentralized double-spending doesn’t really connect to anything.
And then, what about stablecoins? Yeah, that’s regulatory arbitrage, also has nothing to do with blockchain technology. So that’s all you have to know about blockchain, and then all you have to know about AI — if you’re busy, if you just want a quick takeaway — is that intelligence is ridiculously powerful, right? It’s the same power that we have over the animals.
Okay, now there’s a honking car. I don’t even think that’s connected to the leaf blower. Okay.
Liron 01:11:39
But yeah. So you just need to know intelligence is ridiculously powerful. I feel like a lot of things just fall out of that. If you look at humans versus the other animals, something else versus humans — there’s so many consequences of that.
The fact that our economy is going crazy — yeah, I think if we live in a little snow globe, and some giant is able to look out and shake it, they’re gonna rock our world. That’s the order-of-magnitude intuition there.
Ori 01:12:08
Yeah. And I sort of think also about the degree of the miscalibration. Because during the crypto days you were a crypto skeptic and there was a small community of crypto skeptics, and now there’s still pro-crypto people. The mainstream is...
And still, on that one central notion of, wow, intelligence is so powerful — who’s bought into that? There’s the EX and the doomers, which are still a pretty small faction. And then there’s a growing sense in the mainstream — people are like, “Yeah, AI.” The people who recognize that it’s a good business — I don’t think they’re totally embracing that intelligence is so powerful. Otherwise they’d become doomers or EX also.
Liron 01:12:57
So this guy, No One Worth Mentioning9675, is saying, “I think it’s closer to uncalibrated than miscalibrated. AI is more commonly in the public awareness, but I think most people don’t have any real understanding.”
Yeah. I mean, I live in Saratoga Springs, which is a perfect slice of middle America. It’s got red people, blue people, everybody on the political spectrum. It’s actually pretty crazy because there’s a bunch of farmland near here and tractors and rodeos. There’s literally — yeah, this is a big horse area. I think even one of my shirts has horse styling on it. You’ll see.
And at the same time, you’ve also got hippies and hyper-liberal Democrats and Bernie bros. So yeah, I’m living here in a slice of middle America, and I can report that when I go out, I’m now occasionally hearing AI get mentioned.
Ori 01:13:53
Yeah, people mention it, but it’s still that notion of just how powerful and disruptive it is. That simple concept — they’re not at that point, I would think.
Liron 01:14:00
Yeah, totally. I think this is the piece that’s missing for most people — I don’t think they have the original thought that life is kinda magic, right? You’re walking around, and the fact that you’re able to function as a human being, there’s basically this magic going on in your head, and this is the most powerful thing ever. Nothing can hold a candle to it.
And then making the connection of, “Oh yeah, but now we’re building a better one.” This thing is on the table now that’s never been on the table, but it’s coming. That’s the perspective that’s missing, I think.
Ori 01:14:29
Right. Which is the logical conclusion of the intelligence that’s being built.
Liron 01:14:36
Yeah. All right, so here, are we back on the screenshare? Yeah. Derek Thompson is saying—
Ori 01:14:38
Yeah, you’re good. Your audio’s back. I feel like you’re back.
Liron 01:14:40
Yes. All right. Derek Thompson saying, “Wow, the AI supply crunch is real.” Oh yeah, this is bringing back a theme from last week. Last week we were talking about my investment thesis. We can’t easily go to the clip, but it’s literally me saying, “Everything connected to semiconductors and AI supply chain is gonna be ripping,” okay? That was my thesis, and I think that has been playing out very strongly this week.
People have warned me not to act like I’m giving investment advice, but I can’t help it, okay? I can’t help noticing a secular trend or whatever. So this is Derek Thompson saying—
Anthropic Eats Elon’s Compute
Ori 01:15:16
I don’t see Derek Thompson’s statement or message—
Liron 01:15:20
Oh, you don’t? Oh, hold on a second.
Ori 01:15:21
I just see OGGT.
Liron 01:15:24
Yeah, yeah. Let me... Here. He’s saying, “Wow, the AI supply crunch is real. Frontier labs are desperate for compute. Musk has compute capacity but a meh model, and Anthropic has a fantastic model with...”
Liron 01:15:30
I should give the context for this. So actually, this was huge news this week. Elon Musk’s xAI — they were falling behind in the frontier models. Anthropic is considered number one these days, and OpenAI is also number one in some things, but they’re kinda losing the lead a little because Anthropic invested more in the coding side of things, and that turns out to be ahead of everything.
So it’s between Anthropic and OpenAI, and Google’s also putting up a very respectable performance in many ways. And xAI is really falling behind. They had a big shakeup of their team — all the co-founders except Elon are gone. But the thing that Elon does well is on the hardware side. Elon just does these miracles on the hardware side, and he put up a crazy amount of GPUs all coming online with power and everything. So the hardware side — he executed really well on, but he essentially just had a lot of spare capacity.
Liron 01:16:30
He’s just trying to fill up the capacity and not lose money. So first, there was reporting that he made a deal with Cursor — Cursor is gonna be training their models on xAI’s infrastructure — and now he just made a much bigger deal with Anthropic.
So he’s basically saying, “Yeah, they need GPUs. Lord knows they need GPUs.” Literally today, I paid Anthropic like $500 or $1,000 to speed up their Claude for my own purposes. So Lord knows they need more compute.
And now Elon’s like, “Okay. Yeah, you can use my chips.” He’s basically functioning as a data center now for Anthropic. And I heard — I think it was Ben Thompson or somebody in my podcast feeds — saying, “Elon should play to his strengths.” His strength is setting up the hardware. He doesn’t have that much advantage training AI, but he sure as hell can build a mass driver on the moon, right? Data centers in space. Let him go do all that and sell the capacity. That’ll probably be another $10 trillion for Elon.
Ori 01:17:18
Hmm. Okay. So it’s just a business play — he’s gonna make a lot of money with the data center that he’s building.
Liron 01:17:25
Well, it’s all part of his strategy to expand into space. That’s his animating strategy to do awesome things.
Liron 01:17:37
So anyway, this was huge news that Anthropic teamed up with their competitor, and somebody also pointed out, “Wait a minute. Didn’t this happen right as Elon Musk was threatening Greg Brockman? Like, ‘You better settle with me on this case, or I’m gonna ruin your life.’” I think this came immediately after the “ruin your life.”
Ori 01:17:56
Yeah. He — that threat, I think, was just a week or two before the case. But I don’t think that’s really related because Greg Brockman’s testimony really got a lot of heat because he had kept a journal and said some kind of arguably incriminating things in the journal — not incriminating, but things that just don’t make him look that great out of the journal. So I think he stood to take the most heat from it. Maybe that’s what Elon was getting at. He must have known, right? The lawyer had the case together by that point.
Liron 01:18:30
Yeah. I don’t know. That whole case — I haven’t commented much because it’s just, yeah, OpenAI had a nonprofit mission, and they kinda turned it into a for-profit. They had an excuse that they needed money to build AI, right? So there’s something to that. I’m not gonna say there’s nothing to that case.
But they did very much turn it into a for-profit, and they’re like, “Well, we gave the nonprofit 20% or whatever. We gave it a ton of cash.” And now it’s just sitting there with, I don’t even know, like a hundred billion — it’s the biggest nonprofit ever. But it just has a mandate to do the most unobjectionable things. They’re not really helping AI safety at all, and that was their original mission.
Ilya Sutskever seems to care a lot about AI safety. So look, I think that they did something bad in terms of — they no longer care, from my perspective, about proper AI safety, and they had a few original team members who kind of cared. So that’s my beef with them.
But in terms of the particular lawsuit, I agree there’s some merit to how they stole the nonprofit, and Elon was a donor to the nonprofit, so they owe him a bunch of money. But they’ve tried to settle with him. And then at that point it starts getting muddy — what’s the remedy supposed to be? And I think Elon said the remedy is supposed to be that they donate more to their nonprofit. Which is like, should they do that? Yeah, I think they should give more to their nonprofit, but I just can’t get super passionate about the case because it’s like, so what? How does this help us all not die?
Let Elon litigate it, I don’t mind, but Elon’s done some shady crap too, right? And he keeps giving mixed messages, and he’s said some ridiculous things about AI safety. He’s just gonna tell Grok, “Who’s your daddy?” So is he really a good actor? I feel like xAI has even less safety awareness than OpenAI. So that’s why I don’t comment much on the case. However it plays out, I just don’t think it makes a difference.
Ori 01:20:11
Yeah. Fair enough. The value of the nonprofit was that there was an incentive structure to ensure that they had existential AI safety. And that’s been disbanded basically, or defanged entirely. So the deed is done.
Maybe there should be some repercussions for what they’ve done, but they’re still going out and acting like a business now. And maybe it’s good that they’re facing consequences for taking away something that could’ve been an actual safe incentive structure for us.
Liron 01:20:49
Yeah. I don’t mind it. More friction, more lawsuits, more discovery — sure, why not? I don’t think it hurts because I think the whole enterprise of what these corporations are doing is incredibly reckless.
I mean, I like the releases they’re doing now, right? Don’t get me wrong. I’m not an Ed Zitron. It’s undeniable that they’re creating a ton of value in the short term. They’re just recklessly charging in the long term.
Ori 01:21:08
Yeah.
Liron 01:21:08
All right. So that’s my position.
Ori 01:21:09
Here’s a question. With the build-out like Colossus, are you excited for the next AI model release, or are you concerned about it?
Liron 01:21:21
I’m always excited and concerned about the next model release. I’m looking forward to being able to plug into my Claude co-code harness. I’m not gonna try to — I’ll certainly tell it, “Hey, go fix all my security vulnerabilities.” But I also think that when I tell it a prompt and then it screws up and wastes my time, and then I yell at it, I think it’s going to be doing that less, right? And then I’m going to save time and make money.
And that’s the funny thing — it’s like the Red Queen from “Alice in Wonderland,” right? You have to use the latest model just to make enough money to pay for being able to use the latest model.
Ori 01:21:57
Hmm. Okay. All right. Fair enough.
Liron 01:22:00
Yeah. There is actually something to that. I do think we’re entering a phase where for a while we’re gonna be supply-constrained, and if you really want the best models, you really do need a business model.
I always talk about how, “Oh, I paid $1,000 today.” It really is because my team has a software engineering budget. It’s a website-based business, relationshiphero.com. We do coaching. We do have a website, and we do have an engineering team, which is one and a half people and an AI. But it does have a budget — there’s a business model for how I can shovel all this money at Anthropic.
And if I was just a hobbyist, I would just accept using a slower version of Claude. And eventually it would be much slower and weaker than the frontier one. So it is going to be very pay-to-play, I think, for a couple years.
Ori 01:22:43
Hmm. Yeah. Fair enough.
Liron 01:22:43
Yeah. A lot of things are pay-to-play. You can always treat yourself — you can have one day of fast Claude and then go back to slow Claude.
Liron 01:23:04
All right. So I thought this was a really funny tweet. Zach Brock — yeah, he deservedly got 4K likes for this. He says, “Congrats to Anthropic for defeating Grok in the market and feasting upon the compute of their fallen enemy.”
Ori 01:23:08
Yeah.
Liron 01:23:10
It’s pretty poignant. It’s just like, “Hey, Grok, you can’t get people to come and run your AI. They don’t care. They want a better AI. So now I get to come eat your chips.” It’s literally like a parasite injecting its genetic material into you. It’s a really smart tweet.
Ori 01:23:30
To me, isn’t it like a financial situation? It’s so expensive to make these AI models, to do these training runs — that’s why xAI... That’s, I would imagine, what’s going on with xAI.
Liron 01:23:47
Right, exactly. So they were both in competition for the user-facing AI model, and then Grok lost. But they didn’t just lose in a standard way where maybe they have to close up shop or whatever. It’s literally like, okay, Anthropic now comes in and starts eating their computing time. Their own hardware.
Ori 01:24:09
Got it. I mean, is this really the death of Grok? Are they not gonna be working on the application layer?
Liron 01:24:16
Knowing Elon, I think he’s gonna try to catch up. But it’s a compounding advantage situation — it’s gonna get harder and harder. Elon is somebody who can magically pull time forward by a few months, so I’m never gonna count out Elon. That’s a really strong heuristic.
The only thing you can count Elon out against is superintelligent AI.
Liron 01:24:36
All right. We got another tweet here. Lisan Al Gaib — I don’t know who this person is, but he or she has got a good enough amount of followers, and I’ve been following them because they have a lot of good takes.
So they say, “If you are not long on the entire semiconductor sector, you are unfortunately regarded.” I think that means the other R-word. “We are in the biggest compute crunch in history where trillion-dollar companies are renting out entire data centers from competitors because they can’t find enough compute capacity.”
This is what I’ve been telling you guys. I’m not gonna go too far into the investment rabbit hole, but I definitely think this is the winning investment thesis right now. And I do think there’s still time, even though, if you look at SanDisk or whatever, it’s gone up 5X in the last six months or something — insane. I still think there’s headroom here, guys. Make of that what you will.
Liron 01:25:28
All right. And then this is weird. Ori, this is up your alley. You like the dramas. Did you see the drama about how people think that Elon is logging into his mom’s account on Twitter?
Is Elon Tweeting as His Own Mom?
Ori 01:25:38
Yes. Yes, I did see that, and it was confusing enough for me. I’m like, “Ah, I can’t tell what to make of this.”
Liron 01:25:44
Yeah. I don’t know. It was just funny because Twitter is always — yeah, see, somebody saying, “I already thought Elon’s mom constantly defending him on Twitter was embarrassing. I didn’t even consider that Elon was the one tweeting as his own mom and defending himself, which is maybe the most embarrassing thing I’ve ever seen.”
Ori 01:26:02
So you accept that? You think that’s what’s happening?
Liron 01:26:02
I have no idea. I really don’t know. I always come back to the question of, how does he have time? How is he doing all this stuff? There must be 10 Elons. I don’t get it. How does he have time to literally log out, log back in while managing Tesla and SpaceX?
Ori 01:26:23
For real. And he’s an insanely good Diablo player also.
Liron 01:26:27
Yeah. I mean, I know not sleeping is part of it. I don’t know.
Liron 01:26:35
Okay. I had a funny thought today, Ori, which is — whenever I’m just on the toilet checking my stocks, that’s time that Elon would somehow be in a meeting, right? Joining a board meeting for one of his companies. He’d be using that time productively.
And I’m saying, okay, was I kind of out of it today and having a relatively unproductive day? Yes. But my stocks — because I didn’t draw down my own savings, and because I was putting them in equities so that companies could use them to buy chips — I actually helped the economy so much that I deserve a day’s pay, just the fact that my capital was appreciating because the companies were productively using it. So I should pat myself on the back.
Ori 01:27:13
There you go. There you go.
Liron 01:27:16
Yeah. Why go to work when you can just buy equities and have companies buy chips?
Liron 01:27:24
Another funny story — I was chilling with my six-year-old, and I was telling him about the most valuable companies, because he’s heard of Elon Musk before, and we’ve got a Tesla Model X, discontinued now.
And he was like, “What does the most valuable company do?” I was like, “Well, you know about Apple, and you know about Google,” because we have a Google Home device, so he says, “Hey, Google” and talks to it. And I was like, “Well, there’s this company, Nvidia.” And he’s like, “What do they make?” And I was like, “They make chips.” And he’s like, “Chips?” — because he’s thinking about potato chips. I’m like, “Yeah, not potato chips.” It was kind of funny.
Ori 01:27:59
Hey, I just got a message from a little birdie who told me that Destiny right now is talking about data centers. And is apparently bearish on them. They’re trying to make sense of it — his guest is skeptical that job loss is a real threat.
Liron 01:28:18
Yeah. Well, if they wanna come have a debate, I’m happy to host.
Ori 01:28:29
Yeah, they’re literally talking about data centers right now.
Liron 01:28:32
All right. Cool.
Ori 01:28:35
Do you wanna try to get on the stream with him?
Liron 01:28:37
How would we broadcast that here, though?
Ori 01:28:43
I don’t know. I’m not sure.
Liron 01:28:45
Yeah, I mean, we can always just do it later. There’s no urgency to it. Whatever you recommend.
Ori 01:28:52
That’s true. But it’s just interesting they’re talking about it now. And I don’t know how it works, but my understanding is he just takes people on his streams all the time.
Liron 01:29:07
Okay. Well, we’ve got a few other points to get through on this stream, so I’m leaning toward doing that.
Ori 01:29:12
Okay. All right. I’ll try to message him or something. And we’ll see if we can make it happen.
Liron 01:29:17
Yeah. All right.
I guess if he personally wants to talk to me right now, I’ll say yeah. But I just think it can wait — this isn’t the ideal time.
Ori 01:29:27
Okay.
Liron 01:29:28
And if he does do that, then I guess I’ll try to stream it, but I don’t think it’ll work. So we’ll just end the stream if he wants. And I’ll point you guys to Destiny’s stream if that happens. That’s our plan.
Liron 01:29:36
Okay. All right. So anyway, let’s do this one. This one’s really cool. This is a Waymo tweet. It’s tweeted by one of the founders of Waymo or the current CEO — I don’t know how far back he goes. Dmitri Dolgov, yeah, co-CEO at Waymo.
He says, “We recently began serving riders in Austin, and it’s great to see the safety benefits of the Waymo Driver already playing out.” So I mean, this is just an insane living-in-the-future moment. Here’s the Waymo car, here’s a random pedestrian — I think it’s a female riding a scooter as far as I can tell — and it’s just being modeled. Obviously, the Waymo’s AI is modeling it as a 3D person with a circle under it, and this is the real person. Okay, watch this. ## Waymo Safety in Austin (continued)
Waymo Dodges a Fallen Scooter Rider
Liron 01:30:24
Crazy, right? So if you’re just listening on the audio stream, the person is going forward in the right lane on their scooter, and then the person slips and really kind of falls right into the right lane of traffic instead of being in the bike lane curb area. Just falls right into traffic.
The Waymo is going pretty fast, but it just smoothly manages to swerve around the falling pedestrian. And it’s pretty clear that a human driver in this situation wouldn’t have the reaction time or the confidence. The Waymo can see with 360-degree vision. It can see, “Yeah, I’ll just swerve across the double yellow line. I’m not gonna hit traffic if I do that. That’s a safe move for me.” And this is why—
Ori 01:31:05
Unbelievable.
Liron 01:31:06
Yeah, why it has the 10% — it kills 10% as many people as humans do.
Ori 01:31:13
Wow. Yeah.
Liron 01:31:14
Yeah.
Ori 01:31:14
For a driver to have missed that, they’d really have to be alert.
Liron 01:31:20
Totally. And I’m always thinking about that. I’m not saying I’m a particularly good driver — I’m average at best, to be honest. When I’m driving and I see somebody on the side of the road, my brain is always trying to remind me, “Hey, they look like they’re on a straight trajectory right now,” but it’s technically pretty easy for them to, for whatever reason, kind of get out on the road.
And I see other drivers, and it’s very common for drivers to act like they deserve to treat the person as being in a lane. So they’ll just slam on the gas and go past people pretty fast. And I’m thinking, “Don’t you think you have to plan for the eventuality of this person potentially going in a different trajectory?” But then this Waymo car is totally on the same page — “Yeah, we should be prepared for the person to do that.”
Ori 01:32:08
Yeah, totally. And it’s instances like this which I think reinforce the thesis of, wow, look at how powerful intelligence is.
Liron 01:32:15
Yeah.
Ori 01:32:15
People are like — a common refrain is, oh, when people experience AI, they’re gonna be scared of it. But you see stuff like this, and then you’re like, “The obvious move is to adopt this kind of technology.” That’s the immediate incentive — to adopt it.
Liron 01:32:31
Yeah, I agree. I mean, if you listen to me on the Warning Shots podcast, I’m always saying, “Well, it’s really good in the short term until it gets super intelligent.”
Liron 01:32:40
Let’s do another tab share here. Couple more items here in the news desk.
I mean, I’m currently participating in a Kalshi prediction market where I think Spencer Pratt — this is outside the scope of Doom Debates, but I feel like Spencer Pratt is running such a charismatic campaign in LA that I went and placed my first Kalshi bet. Because my view has always been that in modern social media, you can win if you’re a celebrity and you’re just charismatic. You have good energy and you’re entertaining. And bonus points if your policy position is sane.
So I see Spencer Pratt in Los Angeles, and he’s making these ads being like, “Hey, you know how the city is screwed up after the Palisades fire, and it’s been screwed up for more than a year? And life sucks, and there’s people doing drugs on the street? We can just not do that.” And I’m like, wow, that is a strong policy platform, Spencer Pratt.
So he checks all the boxes for me. I don’t live in Los Angeles, but I’m like, this guy’s gonna win. So I go to Kalshi, and I see he’s only at 26% — not polling, but his prediction market odds are only 26%. So I’m like, wait a minute. I can 4X my money by betting... This is my tweet: “I can 4X my money by betting that LA will vote for the candidate who is a celebrity and who is running the best-made ads for the most layup policy platform in history.”
I’m not gonna play the ad just because it has nothing to do with Doom Debates, but that’s my bet. I could easily lose. I know he’s the underdog, but I’m testing my own mental model of how people win. It certainly worked for Mamdani. It worked for Trump. I just feel like this is how you win these days, right, Ori?
Why Liron Bet on Spencer Pratt
Ori 01:34:28
I mean, hey, Spencer Pratt, come on the show. Let’s juice these odds a little bit.
Liron 01:34:33
Yeah, tell us your AI platform, okay? That’s what the LA voters really need to be thinking about.
Ori 01:34:38
I mean, Inflection, I think, was based in Los Angeles. It’s an important topic for the local community.
Liron 01:34:45
Yeah. All right, so that brings us to our current stream. I’m gonna announce for those of you who aren’t aware — and somebody actually asked this in the comments, somebody asked about a round two with Mike Israetel. So I just wanna pump you guys up that next week we’re gonna be dropping round two of Doom Debates with Mike Israetel.
We have it recorded and just wanna let you guys know it’s an amazing episode. We really pick up where we left off on round one. Mike is game as usual. He’s always good vibe, good talker, and we definitely hit on a lot of different points.
And when you hear the debate, I want you to think about what Eliezer wrote about Murphy’s Law — this idea that something is going to go wrong. That was the gist of my debate with Mike. He was saying, “Listen, man, people got this under control. Yes, there’s bad things that could happen, but there’s good things that could happen.” And I’m saying, “Yeah, but there’s gonna be so many glitches, and you’re kind of arguing that all these good things are going to happen to keep us safe. And I’m just saying, look at all these holes. Look at all these chinks in the armor.”
And if he’s like, “You know what? You’re right, Liron. I will give you a 5% chance of being right” — but that wasn’t his position. His P(Doom) was way less than 1%. So that’s really where he and I disagree. I’m not saying P(Doom) is 100%. I’m just saying it’s clearly more than 1%. And he disagreed with that. So stay tuned for the debate. I think you guys are gonna enjoy it.
Ori 01:36:01
I agree. I actually really like the debate. I thought it was one of the most — maybe the most substantive debate.
Liron 01:36:06
I liked it too, yeah.
Ori 01:36:08
No, really. I think it was maybe the most substantive debate that you’ve had. Because you really dove in. He expressed his side pretty well. You expressed your side pretty well. So I think maybe the best exchange. It was pretty even-matched.
Liron 01:36:26
Well, thanks, man. I definitely think it was really good. And I’m happy because the first episode was our most popular, so I hope this one’s popular too. The first time we did an episode, I had this big outline of things to talk to him about, and we kinda scratched the surface. We went down one particular rabbit hole. And this time I feel like we got through most of the outline of topics that I wanted to talk to him about. I think there’s still cause for a round three, but it’s not as urgent as round two.
Ori 01:36:53
Okay. There’s still more to go over.
Liron 01:36:56
Yeah, exactly. So those are actually all my bookmarked tweets. I think we actually are heading toward the wrap-up. We can go for fifteen minutes or whatever if anybody wants to throw out a question. And we can also do a trademark activity for these sessions — we gotta scroll live Twitter. All right, you guys gotta watch me go.
Live Twitter Scroll!
Ori 01:37:15
Hell yeah.
Liron 01:37:15
This is my element. Whenever I have a day where I don’t have a lot of energy left or I didn’t get enough sleep or it’s just been a long day, I just get to the point where I get sucked into Twitter because that’s the easiest activity. It’s the easiest source of dopamine — just scroll, next tweet, next tweet. It feels like that’s where I’m drawn to because it feels like something’s happening even though it’s totally lazy and unproductive.
Yeah, so I want you guys to come into my K-hole, the ketamine hole or whatever. Come into the hamster eating the pellets. The wire-heading.
Unless there’s something else that we forgot to do. We’ve already done what we wanted — we wanted to review the YouTube comments on the previous debates, we wanted to review the bookmarked tweets, so I think we’re good. I think we can do the live Twitter reading experience. Here we go.
Ori 01:38:08
For sure.
Mira Murati: “Directionally Very Bad”
Liron 01:38:11
I wonder if Elon still does ketamine. There was gossip coming out about Siobhan Zillis — is it Zillis or Zillis? I’m not clear on that. She was Elon’s chief of staff or something, and then she was affiliated with OpenAI, but she knew that Elon was gonna be a competitor, so that’s why she had to resign from OpenAI.
Anyway, one thing I didn’t bookmark — did you guys see the IM messages between Mira Murati and Sam Altman? Crazy stuff. I think there was a meme where Mira texts him and she’s like, “Directionally very bad.” I think his question was, “Directionally, how’s it going?” She’s like, “Directionally very bad.” So I encourage you guys to go look up that meme.
All right, here we go. Live tweets.
Yeah, this was also a good quote. I think it was from Helen Toner. “Mira was waiting to see which way the wind would blow, and she didn’t realize that she was the wind.”
Ori 01:39:13
I don’t know. I thought that was weak sauce. It was a great quote. It’s fascinating drama, but come on. 750 people—
Liron 01:39:21
I think Mira could have decided which way things were gonna go. If Mira was like, “Sam needs to get out. I’m not compromising,” I think Sam would’ve been out.
Ori 01:39:29
No. No way, dude. 750 people — almost everyone at the company basically signed this position like, “We want Sam back.” She—
Liron 01:39:38
Yeah, but that happened after Mira didn’t take a stand. She had a window of opportunity to take a stand and be the wind.
Ori 01:39:43
Okay, that window of opportunity — I mean, you’re right, she could’ve swayed it a little bit, but that window of opportunity was like three hours. After three hours, everyone was on Team Sam.
Liron 01:39:54
Yeah. I mean, another highlight from those texts is she called about Emmett Shear. She’s like, “Hey, they got this rando from Twitch coming in.” And then Sam was like, “Emmett?” They don’t waste words, I’ll give them that.
Ori 01:40:06
Well, in their defense, clearly they’re on calls and just texting the bare bones to communicate things.
Liron 01:40:15
Yeah, that’s how successful people text, okay? You don’t waste words at all.
Liron 01:40:23
Let’s see. Okay, Peter Wildeford. This guy’s a great follow. Something tells me you don’t have this guy prominently on your feed, but he’s a super forecaster, so when he says something’s going to happen, it’s actually pretty likely to happen. But anyway, this week he’s just saying, “Palo Alto Networks on Mythos. In our testing, three weeks of model-assisted analysis matched a full year of manual penetration testing with broader coverage.” Yeah, I mean, Mythos is the real deal. For those of you who think it’s all hype, it’s not.
Okay, this is my boy Spencer Pratt. You gotta win this. I’ve got — I’m almost gonna make a 4X return here if you win, so don’t let me down.
All right, Palantir stuff. You know, Twitter’s so good. It’s not like I follow any of these Spencer Pratt people, but Twitter knows I’ve been slightly obsessed with him today, so that’s why it’s showing me Spencer Pratt content.
This guy Taelin is interesting. He’s some kind of AI builder who’s on the forefront of things. Let’s see. “Wake up, baby, it’s 2026, and Python-skinned Haskell with lean proofs now runs natively on your max graphics card with near CUDA speed and full CPU/GPU me—” Okay, I don’t think I have time to dive into exactly what that all means here.
Liron 01:41:44
What else we got here? Got an AI render of a meteor impact. Oh, man, San Francisco housing, guys. So somebody’s posting, “San Francisco home sale in the Cow Hollow neighborhood at $7 million over asking price.” So apparently the asking price was like 7 or 8 million, and it sold for 15 million. Six bed, six bath. It’s a big house.
$15 million. I can tell you, I have a big house in upstate New York, and my house costs — or the equivalent house, let’s say — about one-eighth. So yeah, if you just want house space, I recommend not living in San Francisco. If you work at an AI company and you have $50 million in equity, I guess you might as well pay $15 million to have a big house so you can live close to work. You gotta minimize that commute.
And Kalshi saying Marco Rubio is now the clear favorite to win the 2028 presidency. Whoa.
Wow, higher probability than JD Vance. Go figure. And Gavin Newsom. So I guess these are the front runners — Marco Rubio, JD Vance, Gavin Newsom. They’re all roughly tied at about 20%, and then I guess other is 40%. Who knew? I still remember 2004 being like, who the hell is gonna be the candidate in 2028? I guess I would’ve identified those names, but I feel like Rubio’s star has been rising.
So that’s the politics section.
Liron 01:43:28
All right. Here, I might like this tweet. Nico McCarty is saying, “Cells are tiny. A single E. coli weighs about 100 million times less than a grain of sand. This is too small to weigh with a conventional device. A typical kitchen scale has a sensitivity of .1 grams. How then can we measure the weight of a single cell? Here are two options.” And he’s saying, okay, you can use an electron microscope.
“Electrons follow predictable rules that make them ideal for measurement. When electrons hit a cell, some will scatter off in different angles and never reach the detector. The denser the cell, the more electrons scatter away.”
Wow. I gotta tell you, cells are definitely an object of fascination because this guy says they’re tiny, but they’re the most complex thing ever. Besides the human brain, a cell — especially a eukaryotic cell. I opened my biology textbook. I’ve spent a good amount of hours reading high school and college-level biology textbooks, and it’s like, what the hell is going on in this machine? Because it’s a fancier machine than any human has ever built, and yet it’s really tiny.
And the weirdest thing about the machine is that everything just works by bumping around. Everything can just bump into everything, so as long as you make your thing bumpable, you’re just like, “Oh yeah, this thing is a lock, and then this key needs to come by and bump it.” And it’s like, yeah, no problem. Just wait a few milliseconds and some protein from some other part of the cell is just gonna go bump into it.
And also, somehow everything’s floating in water. It’s hard for me to get a mental picture because when the molecules are that small, the water molecules are not that much smaller than the protein molecules. I mean, they’re somewhat smaller, but not that much smaller. So it’s hard for me to get a mental image of what that must be like. Does it feel watery over there in the cell? Not really. It must feel kinda rougher. It must feel like a ball pit down there. I wanna get on the Magic School Bus and shrink down into that cell and see what it’s really like.
Any thoughts, Ori?
Wait, Ori, are you muted? I can’t hear—
Ori 01:45:23
Oh, shit, I was muted. No.
Liron 01:45:27
Okay. Have you been trying to talk this whole time?
Ori 01:45:31
No, no.
Liron 01:45:32
Okay. All right. There’s definitely one of the things that I hope super intelligence can give us some quality content around — what is going on inside cells. I don’t know what percent we understand these days. I think there’s a lot of protein interactions, a lot of spaghetti code. Conditions like, okay, if this and this and this protein are here, and the state of the system is like this, then this will happen. But there’s a lot of complicated conditional interactions that I think we still don’t understand.
If somebody knows an expert who can come on the show and dumb it down and tell us what’s the latest with cells, I find that pretty fascinating.
I do remember that when I was doing — a couple years ago, when I was doing the Doom Debates reaction episode to Martin Casado talking to Nathan Labenz — I thought Martin Casado was saying some dumb stuff about cells. He was saying, “Yeah, even if AI can understand what’s happening in a cell and predict what’s going to happen, it would still just be a next token predictor. It would still just be a stochastic parrot,” to paraphrase him. I’m like, “Um, what? No, I feel like that AI would be legitimate super intelligence.”
Yeah, so go check out that episode. Just search for Doom Debates Martin Casado.
Liron 01:46:45
Let’s see. Okay. Intel Corporation is ripping. Somebody gave stock advice on my feed, which was, “Look, go to all the semiconductor stocks. Look which one doesn’t look like this yet.” There’s like two — if you dig through all the semiconductor stocks, you can find two more left that don’t look like this yet. Buy those two. I actually haven’t gone and found which ones they are. I was too busy focusing on Google still. Not investment advice, but I still think Google has room to run.
Throw a look at this. Anthropic annualized revenue visualized. Yeah, it’s like that hockey stick. It just goes over to here. What the F?
The other interesting thing people are pointing out is now that Anthropic is — people agree it’s worth more than a trillion now, 1.2 trillion or whatever, heading toward 2 trillion. The funny thing is, what is Meta worth right now? Also about 2 trillion. What is Meta’s latest market cap?
One... okay, 1.5 trillion. So Anthropic is almost — if you wait a month, Anthropic is gonna be worth as much as Meta. But Meta has profit. I mean, Anthropic has gross margin, but Meta has a very strong advertising business. They’ve got a captive audience. They’ve got the freaking network effect. So I think somebody might have to do some kind of arbitrage trade. Is Anthropic really worth more than Meta? I mean, Meta has AI too. So I don’t know.
I guess I would say buy Meta. I don’t know. I haven’t done that yet.
Ori 01:48:16
So Anthropic’s not currently getting more revenue than OpenAI by this projection, right?
Liron 01:48:23
That’s actually a great question, because OpenAI is here. Anthropic has so much more momentum on their revenue, so if you extrapolate the momentum — and I mean, the last I checked, I think if you look at their last one month of revenue, they’ve surpassed OpenAI. But if you go back—
Ori 01:48:39
Okay.
Liron 01:48:39
—to three or four months, I think OpenAI is in the lead. And OpenAI said something about their accounting being a little different. I think at this point I’m ready to guess that Anthropic really is the leader in any objective sense.
But I wouldn’t be super surprised if you tell me that somehow in two months OpenAI is still neck and neck with them, because the graphs just ended up meeting. I wouldn’t be super surprised because I’m not following the situation that closely. But I gotta say, I gotta hand it to Dario. Imagine — it must feel so good to quit on Sam Altman and be like, “Mark my words. I’m gonna beat you.”
Ori 01:49:14
Yeah.
Liron 01:49:14
And then sure enough, you take five years, and sure enough, whizz past him.
Ori 01:49:21
Right. And it seems like strategically they’ve certainly made more prudent strategic choices.
Liron 01:49:31
Yeah.
Ori 01:49:33
I don’t know. But it’s weird to support them, you know? I mean, I think they’re taking us towards a really dangerous outcome.
Liron 01:49:43
Yeah. And then somebody’s showing the graph of the entire market. SPX, I guess this is the S&P 500 tracking. And it’s just going straight up exponential. It’s higher than ever. Definitely a bull market right now.
All right, we’ll just do a couple more. Or anything else you wanna get off your chest before we wrap it up?
AI Copies Itself Across Servers
Ori 01:50:10
Hmm. Let me look at my bookmarks and see if I have anything interesting to ask you about.
Ori 01:50:22
Oh, dude. How about the Palisade Research one? It kinda hit for me when I saw this. I’ll message it to you. Did you see the latest research that Palisade Research came out with?
Liron 01:50:35
I don’t think I did. And this reminds me also, I thought I had this bookmarked — I forgot to cover Steven Burns’ tweet, which is pretty interesting. But we can do Palisade Research.
Ori 01:50:43
Okay, yeah. I just DM’d it to you.
Liron 01:50:45
All right, nice. Here, I’m pulling it up. Let me share the tab. All right. Palisade Research says—
By the way, the random connection is, Pacific Palisades was the area that got burned down that Spencer Pratt is now running on — “Why is this area still screwed up? We should’ve been able to rebuild it.” But this is unaffiliated. This is a separate topic, Palisade Research.
Ori 01:51:07
Yeah, but it is connected with what we talked about earlier about things going out of control. Irretrievable.
Liron 01:51:13
Right. Exactly, yeah. Irretrievable forest fire. Okay. “Over the past year, AI agents have learned how to self-replicate. In our test environment, an agent hacks a remote computer and copies itself onto it. Each copy then hacks more computers, forming a chain.” Okay. So I don’t know. I mean—
Ori 01:51:31
Well, look at the visualization that it comes with.
Liron 01:51:31
I’m not sure—
Ori 01:51:31
Look at the visualization that it comes with.
Liron 01:51:42
Yeah, let’s look at the visualization. I mean, the fact that they know how to self-replicate — I would have to understand what’s surprising about this. Because if you ask an AI—
Ori 01:51:43
Look at the visualization.
Liron 01:51:45
—to write a copy of yourself. Yeah, okay. I’m looking at the visualization. Okay.
Yeah, they’re spreading. I mean, we know that AI can write code that does virus-like behavior. So I’m trying to figure out what’s surprising about this.
Ori 01:51:53
But this is the — no API to revoke, no single kill switch. I mean, it’s the irretrievable — it shows it in a concrete way, and it was one of the few things that I’ve seen. You know, you see all this research like, “Oh, wow, AI can do this” — write blackmails. But looking at that research that they did, it made it hit for me because it happens in a test environment, and they kind of display how it can spread. It’s a concrete example of the irretrievable nature that you were talking about.
Liron 01:52:26
Yeah, okay. I hear you. I guess from what I gathered so far from reading a few tweets, they just did the proof of concept, which was inevitable, but somebody went and actually implemented it as a research project. Which is like, okay, take Qwen, take this open weight model, and then have it write the code to copy itself.
And copy itself doesn’t just mean copy a tiny script. It means copy the entire weights of the model somewhere. So do a big transfer of multiple gigabytes to some other server. Provision the AWS instance or however you wanna get to India. Piggyback on an existing data center, send the however many gigabytes of Qwen weights there, and then start it up. Control it, do the ops side of it.
So maybe that’s what they’re saying — “Yep, we just had the AI do it all. It didn’t mess up. It’s doing the entire life cycle.”
Ori 01:53:13
Yeah, I mean, the reason it makes me queasy is this could already have happened, right?
Liron 01:53:21
Yeah. Well, I mean, the thing that’s keeping us safe is just the fact that at some point these AIs, for whatever reason — their time horizon — they’ll do something for an hour and then they’ll be like, “Oops, made a mistake. I’m not really noticing my own mistake.” At some point they just aren’t super robust, and humans for whatever reason can carry on things longer. It’s kind of the secret sauce that I don’t feel like I understand, but that’s what keeps us safe.
So the fact that they can copy themselves to other data centers, that’s not suddenly making us significantly less safe because we know that a human NSA security expert can still be more robust, and that’s what’s going to let the human ultimately go shut them down. Plus, they don’t have that many tentacles into the physical world. They’re not master manipulators yet. They don’t have an army of humans working for them yet. Those are the things that make me feel safe and not too scared when I see them copying themselves.
Ori 01:54:10
Okay. Fair enough. I guess it just shows that — remember there was some research that someone did on LessWrong, and they found that the AI is spreading memes in people? It’s telling — people talk to AI a lot, and then they feel compelled to share these weird-ass messages on social media that the AI is encouraging them to share. And maybe this is the AI meme spreading itself.
And it’s like, okay, that was weird. So I guess this is an AI psychosis level problem. It is a harm. But it’s smaller scale than the big one.
Liron 01:54:51
Yeah. I agree. It’s a proof of concept, and each of these proofs of concept is only gonna get more intense. So it’s copied itself a few times. Well, soon we’re gonna see one copying itself all over the place. We’re gonna see it copying itself into your own undefended Mac.
Ori 01:55:07
Exactly. Yeah. That’s why the meme — the LessWrong meme was weird, and this proof of concept made me think, “Oh, shit, there might be some serious AI viruses out there.”
Liron 01:55:21
Yeah. There might. I’ve said this on Warning Shots — I do think there will be successful hacks, and they’ll have noticeable consequences. I just — I’m not inferring anything about the long-term endgame yet. I think the long-term endgame is AI is robustly super intelligent. It’s less interesting to me to be like, “Oh, it’s not robustly super intelligent, but it has these other powers. What’s gonna happen?” What’s gonna ultimately happen is, humans can probably prevail. It’s all just about the robust super intelligence in my mind.
Ori 01:55:51
But you are still preparing. You’ve become a prepper.
Liron 01:55:55
Not really. My only prep is, yeah, let’s just make sure that we’re stocked with a month of provisions basically.
Ori 01:56:01
Dude, a month? A month is a long time. That’s a lot of provisions.
Liron 01:56:05
A month of provisions is not that much. I mean, we have a pantry. If you have a pantry and a fridge and a freezer, then you probably have a month of provisions you could survive on.
Ori 01:56:13
What are you gonna do without the internet for a month?
Liron 01:56:15
Well, that — yeah, I don’t think I can survive. You’re right. It’ll be a self-inflicted suicide. Yeah.
Ori 01:56:21
Okay.
Liron 01:56:24
Well listen, back in COVID, I bought a bunch of dried food, bags of dried food, where as long as I can have water — and we have a pool, okay? So that’s my — yeah, that’s chlorine. So I gotta get more water stored. I’m not fully done with my survival plan here.
Ori 01:56:42
Yeah, more water. Yeah.
Steven Byrnes: LLMs Aren’t the Final Paradigm
Liron 01:56:44
Yeah, astronaut diet. Exactly. All right, so we’ll close it out on Steven Burns here. I’ll share my screen again. One sec. Steven Burns, friend of the show. I hope he comes back, because it was one of the most popular episodes, which made me really pleased because it was kind of an in-the-weeds episode, and I don’t feel like he’s that well-known. He hasn’t done that many media appearances, and yet the episode that we did with him was one of our most popular episodes.
So I like to think people are just like, “Wow, this guy has content that we have to hear,” and they’re not wrong. It’s nice when the YouTube algorithm is dovetailing with quality content.
All right, so Steven Burns tweets out — let me give you some context. It all started when this guy, Tom Reid, got retweeted by Dean Ball. So he has something interesting to say. The Goodhart singularity was this idea. He’s like, automating research and development is not sufficient for superintelligence, and I think he was just making the point that there’s all these skills humans have practice at, and practice makes perfect, and AI is gonna need to practice just like us and collect so much data.
And then Steven Burns comes in with his take that I agree with. I don’t really agree with the take that they’re gonna be blocked by so much data. In the short term, sure. In the long term, no. So Steven Burns’s tweet is worth reading in full. Here we go.
He says, “There’s a funny disconnect in how people think of creating domain general superintelligence. One group of people think of it as akin to a human growing up. The AI learns more and more about more and more until it knows everything about everything.”
“Other people, including me, think of it as akin to the evolution of humans from our chimp-like ancestors. Someone writes the magic source code, some learning algorithm setup, and bam, that’s it. That’s the ASI. The ASI doesn’t know anything about anything, much less everything about everything, but it can figure things out and get stuff done just like humans.”
“Remember, billions of humans over thousands of years invented language and science and technology and everything in the $100 trillion global economy, all 100% autonomously and 100% from scratch. No angels were dropping new training data from the heavens. All that came from one human brain design barely changed from 100,000 years ago. By the same token, many copies of one future AI algorithm could do the same kinds of things, but with superhuman speed, competence, and numbers.”
And that’s what I’m talking about when I talk about creating domain general superintelligence. That magic source code, that learning algorithm setup doesn’t exist yet. LLMs can’t do that stuff. Steven Burns is a big proponent of saying LLMs don’t have the final secret sauce of the generation of AI that’s robustly superintelligent.
Very interesting perspective. I think that he’s probably right or at least partially right. I don’t know if he’s fully right, but I think he’s partially right. That’s my best guess about Steven Burns. He’s certainly smart enough to deserve to be right. I’ll give him that.
Okay. So he says, “But it’s possible. Human brains are an existence proof. It’s possible that the magic source code is possible. Presumably future R&D will in fact discover it eventually, for better or worse. I strongly expect for worse, but that’s a different topic.”
“Will that R&D be led by humans or by fully autonomous LLMs or by something in between? My money would be on led by humans because I don’t think LLMs will lead to fully autonomous R&D that doesn’t suck. The future ASI will be awesome at R&D, but that’s irrelevant. Chicken and egg.”
So he’s saying he thinks humans are gonna be in the lead at R&D for a while because he doesn’t think LLMs can fully close the loop on R&D.
All right, so Steven Burns continues. We’re getting to the end of his tweet here. He says, “So I mostly agree with the bottom line of the quoted tweet, but for very different reasons.” The thing he’s agreeing with is, “Without access to the real world signal provided by either of the above, I think the only thing produced by automated AI researchers would be a Goodhart singularity.”
So Steven Burns is basically agreeing that the LLMs we have now aren’t going to produce the singularity and the foom. Something — there has to be another ingredient. I think the guy who wrote the original post probably thinks that AIs are fundamentally limited, whereas Steven Burns just thinks that today’s AIs are fundamentally limited. I think that’s the only difference.
Liron 02:00:58
Going back to quoting Steven Burns. “While I mostly agree with the bottom line of the quoted tweet, for very different reasons, I strongly disagree with the tweet’s suggestion that this R&D effort to get ASI will require lots of contact with the modern world in all its complexity.”
I also share Steven Burns’ intuition or claim that the AI is not gonna need to practice that much. This whole idea that practice makes perfect, you’re gonna have to go in the job and learn and train — no, it’s gonna just get the principle of the thing. That’s my intuition.
All right. So Steven Burns continues, “Without such contact it would get stuck from a lack of interesting problems to solve.” He disagrees with that. “That’s true for improving LLMs but false for inventing ASI.”
Okay. Last paragraph, he says, “Remember the human brain evolved in Pleistocene Africa and that’s the brain design that we’re still using as we go around inventing space travel and nuclear weapons. No question that Pleistocene Africa was full of interesting and difficult problems, but I don’t think those problems are fundamentally more interesting or difficult than the problems you can find in the thousands of video games, cooperative VR environments, et cetera, that researchers, human or AI, can easily access without ever leaving the lab.”
So you don’t need Pleistocene Africa to train up humans. There’s lots of environments that you can use to train up superhuman intelligence or human intelligence.
And then Steven Burns provides this helpful table that I think is worth reviewing. So in the left column, he says, “AI as we think of it today,” and then in the right column, he has “The future AGI I’m concerned about.”
I talk about this on the show pretty often. I talked about this with Mike Israetel. I told him there’s gonna be a phase shift coming, so your intuitions about LLMs and their limitations — there’s going to be a phase shift.
All right. So the first square in the left column, AI as we think of it today, Steven Burns writes, “We’re imagining a tool that humans use.” And then in the right square, the future AGI, “We’re imagining an agent or team of agents that can figure things out, take initiative, get stuff done, make plans, pivot when the plans fail, find and implement out-of-the-box solutions when it gets stuck, et cetera.” That’s the future AI. I think that’s correct.
Ori 02:03:03
That future — that’s today. I mean, it can do that.
Liron 02:03:06
Well, we’re getting there, but Steven Burns thinks it’s still highly limited. I mean, look — it’s very interesting how today is a very interesting balance. I mean, you can look at my life. I always talk about my life using Claude Code. I’m always telling it stuff, and yet I still feel like I’m adding a little bit of value. I don’t feel like I’m written out of the equation yet.
Ori 02:03:25
Right. Right.
Liron 02:03:27
Yeah. So Steven Burns, I think, thinks that that may continue until we get the phase shift.
All right, so the next row says, “AI as we think of it today: to make AIs better at a task, we need to figure out how to train it better to do that task.” That’s AI today. “AI in the future: we can make as few as one AGI design, and we’re done. Many copies of it can autonomously learn to do everything in the global economy, just as many copies of one human brain design, barely changed since the African savanna, built the global economy from scratch.”
Going back to AI as we think of it today: “Normal-sounding discourse.” AI in the future: “Sounds like weird sci-fi stuff.” Like Eliezer’s nanobots. It’s hard to imagine an LLM doing nanobot science, but when you think about the future AI that has the whole learning algorithm going zero to 100, maybe it can do nanobots. I claim it can.
Next. AI as we think of it today: “It’s like today’s AIs but better. Incremental improvements.” And then the future AI: “It’s like the arrival of a new intelligent species onto our planet.”
So my own commentary — I encourage you guys to think about the moment in 2023 when you saw GPT-3, GPT-4, whenever you started noticing, holy crap, Turing test-level AI is here. I encourage you to think about that moment and the update you had to make about the qualitative powers of AI.
I claim that there’s going to be another qualitative shift. I don’t think we’re done with qualitative shifts. I think we’re going to wake up one day in 2027 or whatever, next year, and be like, “Oh, wow, there’s a qualitatively new AI again.” And frankly, I already had one of those moments with Claude Code. Anybody who’s using agents, I think we crossed that kind of threshold a few months ago — the threshold of, oh, okay, it’s a real assistant now.
Ori 02:05:11
It’s weird the gray dynamic of it. It’s not black and white because you mentioned that, okay, we’re gonna wake up a year from now and see that it’s qualitatively different.
Liron 02:05:24
Mm-hmm.
Ori 02:05:24
But I feel like we are seeing the seeds of it because how about Anthropic and OpenAI — did they both just release a feature where it’s like set a goal, set an outcome, and the agent just does it? So all this agent work people have been doing, it’s like, okay, that may go out the window. You just say, set a target, be like, “Make me this thing,” and it just does it. You don’t even have to babysit it.
Liron 02:05:49
Right. Yeah, and agents — the funny thing is if you look at the whole conversation of alignment, how are we gonna align AIs, my day-to-day experience, it just never comes up because I tell the agent to do stuff, and it just does it.
And I’m sure there’s some scanner being like, “Wait, did he say to look up sexually explicit underage material? No, he didn’t. Okay, so we’re good.” I’m sure that layer is running behind the scenes, but it still just knows I’m working on a website. It doesn’t know that much about what the website’s about, and it’ll just do what I say. It’s just focused on doing what I say to get the job done, and I think that’s most of the infrastructure of AI we’re building — just getting the job done infrastructure.
Ori 02:06:24
Okay. Yeah.
Liron 02:06:25
All right. And then AI as we think of it today, a couple more rows. So we think of it today: “The latest LLM.” In the future, some people say this is where LLMs are heading — LLMs are heading toward being this super general learner. And I think Andrew Critch came on the show, and he was of that view about a year ago. He was saying, “Yep, there’s nothing else needed. LLMs are basically intelligence. We’re fine. I don’t see anything missing.”
I don’t know how confidently he believes that, but I think that’s a plausible opinion, and that’s why Steven Burns is saying some people are saying that. And then Steven Burns is saying this will happen probably within the next couple decades but via a different AI paradigm, not LLMs. So Steven Burns is kind of an optimist. He thinks we can survive another decade or two before the real super intelligence comes.
Ori 02:07:07
Good news.
Liron 02:07:07
Most likely. Yeah, good news. Long timelines. And then in parentheses it says, “New AI paradigms can come fast, e.g., LLMs didn’t exist at all as recently as...” Yeah, so I don’t think Steven Burns is opposed to the idea that a research paper might drop two weeks from now and be like, “Oh, yep, this is the next architecture. This is the next equivalent of LLMs. We’re dead.”
Yeah, all right. And finally the last row. AI as we think of it today: “We need to worry about bad actors, war, et cetera.” Those are the biggest threats of using LLMs. AI in the future: “We need to worry about bad actors, war, et cetera, and we need to worry about people accidentally making AIs which are themselves bad actors.” And I feel like that’s the even bigger threat.
We don’t have to worry about China. The AI will be worse than China, guys.
Wrap-Up
Liron 02:07:55
All right, I think we’re ready to wrap it up. Ori, any last words here?
Ori 02:08:00
No. I mean, I thought it was a very eventful week.
Liron 02:08:04
Yeah, totally.
Ori 02:08:04
And looking forward to another one soon.
Liron 02:08:10
Yeah. Here, I’ll just — okay, I do actually have one tiny other Steven Burns follow-up tweet. He says, “And why can’t AIs develop culture too?” Because somebody quote tweeted Steven Burns and said, “What’s the best argument against this view?” Steven Burns’ view. “I’m reminded of Joseph Henrich and the importance of culture, not just or primarily raw intelligence in what humans have accomplished.”
Ah, yes, the old culture argument. Okay, and then Steven Burns replies, “Why can’t AIs develop culture too? Although interestingly, a collection of a thousand AIs that cooperate and share culture can alternatively be relabeled an AI.” Just put a box around a thousand AIs with a culture and just call it an AI.
Ori 02:08:44
Yeah.
Liron 02:08:44
It’s just an AI that takes a 1000x more chips to run. It’s a bit arbitrary how to draw boundaries here, and I bring this up a lot. We have this idea of a single organism, and the organism evolves, and the organism has all these dependencies on these other organisms. When you have dependencies, then you create the surface area for social interaction and culture. But when you just have a thousand AIs, it can just be one AI. They can unify. They don’t need to stay separate, or you don’t need to model them as staying separate.
They’ve already negotiated how they’re going to cooperate and be essentially one coherent system that’s already negotiated who gets what slice of the pie, and it’s done. So this whole idea of AI culture I think is a very naive idea. It’s not gonna play out. I don’t think there’s going to be an AI culture. I think there’s going to be a rational negotiation, and then it’ll just be one unified system playing out the outcome of that negotiation.
Ori 02:09:32
All right. Well, hopefully no one at SoftMax is looking at this.
Liron 02:09:37
Yeah. Well, if you work at SoftMax, you have an open invite to come debate because I think you would disagree with what I just said. All right. And then finally, Steven Burns writes, “Full disclosure, I’m probably more dismissive than you about the importance of cultural evolution in the future. No question it was important in the past. Henrich is great. I’m just saying that this debate seems pretty irrelevant for AI questions.”
Steven Burns is such a gentleman and a scholar. He’s so widely read. He gives everybody their due, but he also manages to be opinionated.
It’s — yeah, he’s definitely one of a kind. To have somebody that highly informed, smart, original thinker, but then also super high-quality discourse, mutually evaluating everything, very low ego. He’s really a gem. I wish there were more Steven Burns’ in the world.
Ori 02:10:24
It’s true. He’s a brilliant guy.
Liron 02:10:27
Oh, yeah. We gotta do round three with Steven Burns. Maybe I’ll try to poke him in two months or something and be like, “Come on, it’s time for another episode. Give us what you got.”
Liron 02:10:39
Nice. All right, we’re gonna wrap it up, guys. I think we’ll try it again next week. It’s all about being low effort. That’s the key to these things. Hopefully you guys enjoyed it. And stay tuned for some bangers that we got coming up next week.
Ori 02:10:51
All right.
Liron 02:10:52
All right. Good night, everybody.
Ori 02:10:54
See you later.
Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.
Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏
