We recap the Dr. Mike Israetel debate and dig into the nuances of rationality vs. rationalization. Then onto the week's biggest news: Anthropic dethroning OpenAI, Trump and Xi Jinping exploring AI guardrails, and Liron's own company AI going rogue.
Timestamps
00:00:00 — Cold Open
00:03:09 — Dr. Mike Israetel Round 2 Recap
00:14:34 — A Viewer Accuses Liron of Rationalizing
00:34:20 — Is Sam Altman Any Worse than a Typical Tech CEO?
00:45:15 — What Would Lower P(Doom) Below 10%?
00:53:25 — Can We Disconnect an ASI from the Internet?
01:05:00 — Should Doomers Boycott AI?
01:16:27 — Liron Reacts to ThePrimeagen Reacting to the Yud Debate
01:26:13 — Ben Goertzel Episode Preview
01:38:10 — Scrolling Twitter: MIRI, Ryan Greenblatt
01:47:32 — Liron’s AI Goes Rogue at Work
01:49:15 — “Positive Alignment” Paper Gets Destroyed
01:58:30 — Anthropic Dethrones OpenAI
02:12:06 — Trump & Xi Talk AI Guardrails
Links
Dr. Mike Israetel Round 2 debate —
ThePrimeagen reacts to the Yud vs. 47F debate —
Eliezer Yudkowsky tweet (LLMs know things about human psychology) —
Liron’s tweet (”omg this is gonna be crazy” vagueposts) —
Ryan Greenblatt’s safety research priorities tweet —
Steven Casper’s critique of the “Positive Alignment” paper —
AI Stopwatch by MIRI (Substack) —
AI Stopwatch on X — https://x.com/AIStopWatch
Zvi Mowshowitz AI roundup blog — https://thezvi.wordpress.com/
“The Bottom Line” by Eliezer Yudkowsky (LessWrong) — https://www.lesswrong.com/posts/34XxbRFe54FycoCDw/the-bottom-line
Rationality: From AI to Zombies by Eliezer Yudkowsky — https://intelligence.org/rationality-ai-zombies/
Less Online 2026 conference (June 5–7) — https://less.online/
Transcript
Transcript
Cold Open
Liron Shapira 00:00:08
It’s Friday, May 15th. You’re watching Doom Debates Live.
Producer Ori 00:00:13
Doom. What’s up, everybody? Doom Debates.
Liron 00:00:17
All right. Producer Ori in the house. All right, first things first—
Ori 00:00:23
Liron Shapira, the hostess with the mostest.
Liron 00:00:25
Yo. All right, one sec. I just got a text. Hold on.
Ori 00:00:29
Ahoy. Ahoy hoy, mate.
Liron 00:00:34
All right, one sec.
Ori 00:00:37
Webfra14 first.
Liron 00:00:37
Oh, hey, Webfra. Thanks for coming, man. All right, we got a recurring fan base here. Webfra is a top fan, for sure.
Ori 00:00:45
Nice. Webfra’s on YouTube right around this time.
Liron 00:00:50
Nice. Yeah, let’s see who’s chilling right now. Friday, May 15th. It’s the end of the month, end of the payroll period.
Time’s moving super fast. The freaking year is halfway over almost.
Ori 00:01:04
Oh, wow.
Liron 00:01:05
I feel like I’m always dealing with taxes, because I always file October taxes, so first you gotta get ready for the April 15th, but then you gotta get ready for the real filing on October 15th, and then I have personal and corporate taxes.
Ori 00:01:16
Oh.
Liron 00:01:16
So being an adult means you’re just always in between doing taxes or always in the tax process.
Ori 00:01:22
Nice. Good times.
Liron 00:01:26
Yeah.
Ori 00:01:27
And also the year goes by faster, so it’s what the hell? It’s already tax time again?
Liron 00:01:31
Totally. And seasons — so I live in New York now, a place with seasons, and seasons always used to be whole eras, but now I’m like, oh yeah, winter, summer, they’re just one day after the next, same thing.
Ori 00:01:41
Nice.
Liron 00:01:51
All right. So enough chitchat. Let’s get to the core thing that we wanna review on these streams. Although maybe we should architect it so that we hit our stride 20 minutes in when there’s more viewers, because the viewers trickle in. Right now I’m seeing on YouTube six watching now. I don’t wanna shoot my wad just for six people.
Ori 00:02:05
That’s a fair point. But on the other hand, what about all the VOD viewers? The video on demand viewers.
Liron 00:02:13
Right. Yeah, VOD, video on demand. Exactly. No, that’s true. And the truth is that a typical stream will literally have 20 times more on-demand viewers. So you guys are the real audience.
Shout out to on demand. I mean, I’m an on-demand guy for sure. Every time somebody’s like, “Oh, yeah, you gotta watch my talk live,” I’ll be like, “Yeah, sure,” and then I’ll just leave a recorder going, and then when it’s done, I’ll just download the audio and play it back at 2X.
Ori 00:02:42
Wow.
Liron 00:02:43
Yeah. All right.
Ori 00:02:44
I mean, I like watching some things live. But yeah, I watch some things live. I can’t livestream too much.
Liron 00:03:04
All right, I’m checking the Twitter here, checking the X. All right, we got 12 views going on X right now. 10 on YouTube. All right, that’s 22 total. That’s nothing to scoff at. That’s a classroom full.
Ori 00:03:06
Nice. What are we teaching the class today? Professor Shapira.
Dr. Mike Israetel Round 2 Recap
Liron 00:03:09
All right. So for all the on-demand people, one thing we do in these streams is go over the week.
And we can go over what we posted. So the main event this week was the round two with Dr. Mike Israetel. I feel like that was a very well-liked episode. People said it’s great when there’s two amazing personalities, one guy who’s good at explaining doom, and another guy who represents the average person’s perspective, even though he has a whole channel now and he has a lot of theories. So maybe we shouldn’t call him the average person, because he’s a whole pundit now.
Ori 00:03:46
Yeah.
Liron 00:03:46
Yeah. But people liked it.
Ori 00:03:48
Yeah. I mean, he’s great. Do you think he represents the average person? I think he’s — because yeah, he has so much theory. He seems to have a pretty complex worldview, but—
Liron 00:04:00
Yeah. Let’s play everybody’s favorite clip.
Ori 00:04:02
Okay.
Liron 00:04:03
All right. Hold on, I’m gonna screen share.
In my view, people really have their pants down. This institution has its pants down, and you’re looking, you’re like, “No, no, there’s gonna be people pulling our pants up. These guys aren’t gonna get in and pants us too hard.” I don’t see how confident you are.
Ori 00:04:20
The possibility of some small vectored attack getting through some kind of wall, leading to a permanent entrenchment of an AI that alters human history — there are so many stoppage points to that. To me, it just seems insanely, insanely unlikely.
Liron 00:04:38
So imagine the AI pulls a Kim Jong Un and is like, “Hey, I got a nuke. Now you gotta negotiate with me.”
Ori 00:04:42
Yeah. That’s okay. Negotiation sounds just fine. If I’m just Donald Trump enough and I vibe, and I just first strike you, it’s gonna be like, oh, that’s not really a thought-through strategy.
Liron 00:04:53
You’re claiming that Donald Trump can out-negotiate a superintelligence?
Ori 00:04:56
Yeah. Oh yeah, if it just has one nuke.
Liron 00:04:57
All right. So yeah, I think people’s favorite part was the pants debate. When I said everybody has their pants down, and he’s like, “No, we’re gonna pull our pants up to our assholes.”
Ori 00:05:06
Yeah, yeah. I mean, if you keep going, it gets to that point in this clip.
Liron 00:05:10
Sure. Okay.
Ori 00:05:10
‘Cause what you got? One nuke. Where are you gonna shoot it?
Liron 00:05:13
Okay. Well, hear me out, okay? How about this? Two nukes.
Ori 00:05:15
Yeah. Still we win by a large margin. Huge margin. We have like 10,000.
Liron 00:05:19
Okay. I mean, the reason I’m bringing this up is — from my perspective, it’s like each of these individual cases, maybe you’re right, but I’m just showing you there’s a lot of pants downiness in terms of possible scenarios where we’re absolutely screwed here because we’re negotiating against a superintelligence. Doesn’t seem good to me.
Ori 00:05:32
We have a superintelligence on our side. We’ll have more of it. So we have more pants up than pants down. That’s what I’m saying. So if we model this with pantsiness, we’re pulling up our pants up into our assholes forever, is the trajectory. And every now and again it falls by a little bit—
Liron 00:05:46
Right.
Ori 00:05:46
—but we already have pants completely overlapping us.
Liron 00:05:50
So yeah. People liked it, and Mike might even come back for a live Q&A in a couple months. That’ll be fun too, because a number of people were saying, “Hey, you gotta do more of this. This is such good banter. Such good chemistry.” So we’ll keep exploring it.
Ori 00:06:10
Dude, that exchange is just incredible. I love how you guys somehow made it about pants downiness, and then he totally rolled with it. He was just so quick to roll with the punches. It was a really funny exchange.
Liron 00:06:25
Yep, yep, yep.
Liron 00:06:27
All right, so that was Mike Israetel. And then in terms of the actual debate, remember round one was all about him saying that AI is gonna study us. “Oh, why would it kill us? Because we’re giving it so much information. What all the humans are doing is gonna be so fascinating. You can’t kill something that’s so informative. It’s the most complex thing ever.” That was round one, and I was like, “Can’t it just put other stuff in our place and also be informed?” He was like, “No, man, so many social connections.”
So that was round one. And then round two, I think he had some other claims. What were the fresh claims that he made? Do you remember?
Ori 00:07:02
Well, I was thinking about what round two was about. And to me, it was mostly about the orthogonality thesis — AI cannot be very capable and immoral at the same time.
Liron 00:07:12
Mm-hmm.
Ori 00:07:12
It cannot be very capable and immoral at the same time.
Liron 00:07:17
Hmm. What was his argument against the orthogonality? I totally forgot. I know that was a big focus.
Ori 00:07:25
I mean, I don’t know. What stood out to you as far as — you said at one point too, you said, “Well, you get off at so many stops on the doom train.”
Liron 00:07:34
Right. I remember that part.
Ori 00:07:36
Yeah. So what are all the different stops you think he gets off at? Maybe we should look at the doom train again.
Liron 00:07:42
Right, so — we should look at the doom train. So obviously orthogonality, he got off on that stop. And he also got off on the “study us” stop. Even if it’s not moral, even if it doesn’t particularly care about life, but in the interest of making itself stronger, it should study the most complex thing ever and not kill it. So that would be a separate argument, his round one argument. That’s two stops on the doom train right there, and I think there was a third prominent stop.
Ori 00:08:08
A third thing that he says a lot is how with AI, it’s better to cooperate. Maybe that’s a bit similar to the “it’ll study us.”
Liron 00:08:15
Hmm.
Ori 00:08:16
But he just sort of says—
Liron 00:08:16
Right, right. Yeah, Ricardo’s law of comparative advantage. That was an obvious point for me, in my opinion, which is you really can’t say — I don’t think Ricardo himself would say that his law of comparative advantage proves that an AI would wanna trade with humans when it can just take our stuff instead. That’s not how his law actually works. So I thought that was an obvious point for me, but Mike didn’t concede the point, so.
Ori 00:08:39
Hmm. And why does the law of comparative advantage not apply? Why do you think that’s not a good counterargument?
Liron 00:08:47
Yeah, so the law is a fact of basic economics. Just to explain Ricardo’s law of comparative advantage, the idea is that imagine there’s this island of people who just do everything primitively. So you can farm food using modern farming techniques, and they can farm food using ancient farming techniques, and you’re three times more productive than them.
But if you take your same land for farming food and you use it to make a car factory instead, even though you’re better at farming food, you’re still paying a high opportunity cost because that space is so much better allocated to a factory. So you actually do best when you just use all your own land to make the car factories, and then you go to the primitive people and you pay them to grow you food. Even though they’re growing food three times less efficiently and the resource inputs are three times as much, you should just pay for that and then go do your factory.
Which is kind of an intuitive experience for us. I mean, I pay a huge markup for DoorDash, and if my only alternative was either to cook my own food for $10 worth of groceries or DoorDash for $50, I would still do better DoorDashing, because if I save myself half an hour, I can make $100 of income in that half an hour. And that’s the comparative advantage. The DoorDashers — it’s actually better to DoorDash.
But then of course you could be like, well, in the island analogy, okay, well, instead of having these slow people that you pay to farm, you do even better if you just take over their island. You can just manhandle their island. You can just do whatever you want with their island. Ricardo’s whole law pre-assumes that the island is a take it or leave it situation. You can either pay them in order to trade with them, or you can refuse the deal. But there’s no third option where you take over the island. That’s not the premise of Ricardo’s law.
But the moment you allow yourself to do that, Ricardo’s law becomes irrelevant if you can just take over and do better.
Ori 00:10:45
Right. Yeah. I’m looking at the transcript now, actually. And he did, right, he does bring up — and I think that kind of backtracks to another stop on the doom train, which is just that, oh, AI’s not that powerful.
You trade in cases where there’s some kind of disparity. “Okay, I can’t grow these crops as well, so I’m gonna trade with someone to get it.” And it’s kind of assuming that AI isn’t at human level at a given capability, so then it has to trade.
Liron 00:11:19
Well, no. The only assumption it has to make is that there’s no takeover option. Because even if the AI — that’s the whole point of the law, and Noah Smith harps on this too — even if the AI is literally better at everything than humanity, imagine that you take it as an assumption that the Earth is ours. So we always get the Earth. The AI just is not allowed to take over the Earth no matter what.
If you give it that assumption, at that point it might actually be like, “Okay, I’ll pay you guys a million dollars to grow my food.” And it’s like, “Wait, can’t you just grow your own food? Aren’t you better at growing food than us?” And it could be like, “Yeah, sure. I am better at growing food. It’s just that you guys have this extra planet that’s no opportunity cost for me. And every other planet, I can actually put to such perfect use that it’s actually best if I get my food from you guys and pay you not that much money for me.”
So actually, Ricardo’s law of comparative advantage would even apply to a literal ASI compared to human Earth if you hold it as an assumption that it’s not allowed to bomb the Earth or wipe everybody out and manhandle the Earth. If you’re saying, “Nope, the Earth is our territory,” suddenly Ricardo’s law of comparative advantage actually does apply.
Ori 00:12:30
Got it. Okay, well, that scenario just seems so unrealistic. I mean, that scenario assumes you’ve got—
Liron 00:12:35
Exactly.
Ori 00:12:36
—the aligned... It’s the Asimov robot that has agreed to not harm humanity and never do so. It’s the ultimate assistant or something.
Liron 00:12:47
Now, there is actually one more caveat to Ricardo’s law of comparative advantage, which is — okay, yeah, the Earth is ours in that scenario, but the problem is we’re just so slow and inefficient. So it’s like, “Okay, yeah, you guys can farm for me, and I’d rather pay you than use my own territory for farming. But you guys are just so annoying to deal with. You take forever to farm. You farm inefficiently. Just screw it, I’m just gonna ignore you.”
It’s actually pretty likely that it might ignore us. But even then, the whole premise is that we get to keep the Earth, so we’ve already won. Just by having that premise, we’ve already won. You don’t prove that we get to keep the Earth. The problem is, how do you argue that we get to keep the Earth?
Ori 00:13:21
Mm. I see. Okay.
Ori 00:13:24
Yeah. And that ties back to — I feel like you have to be a bit more neutral, but I’m allowed to maybe have a bit more opinions. And my take on the debate with Mike is just that, okay, we’re talking about something in the future, and you can always imagine the future could be positive or it could be negative.
If we were talking about a presidential candidate, you could put all your hopes and dreams in the presidential candidate. In the future, he’s gonna fix the economy, he’s gonna fix healthcare, he’s gonna fix the military. And it’s like, okay, that’s possible, or we could just continue to have what we observe to have.
So anyway, I just view that debate as he was constantly focusing on the positive that could happen, and it could happen, but it just seems — that was it. He was really wanting to focus on the positive story, and you even said it at one point. You’re like, “I’m not gonna psychologize you,” but how can you get off at all these different stops on the future?
Liron 00:14:28
Did you notice? I just remember in that moment, I was very careful to then shut my mouth after I said that, because I was definitely trying to.
A Viewer Accuses Liron of Rationalizing
Ori 00:14:34
Yeah, yeah. Well, you made the mistake last time, actually, I don’t know if you remember, with Israetel.
Liron 00:14:41
Yeah, I mean, you could call it a mistake or not. That — look, it is my honest opinion that if I — I don’t psychologize on the show, but if you made me guess, I do think that there is a rationalization process happening in his mind.
And it’s not necessarily a bad fact of his personality. It’s just how humans operate, which is your brain is just attracted to this claim that the AI is good, and it’s just such a little jump from “Ugh, the AI feels so good” to “Oh, well, it feels so good because it’s smart, because it’s gonna cooperate.” It just all feels like it goes together naturally, even though if you start reasoning forward from “Wait, why do things cooperate in the first place?” the actual arguments going forward from premises are weak.
But you can turn up so many premises when you just have a motivated conclusion. Sorry, Dr. Mike, okay? You can feel free to say that I’ve incorrectly psychoanalyzed you. And I don’t see it as my job to psychoanalyze people. When you come on the show, I’m not gonna make the show about why I think you have the views that you have. So you’re welcome to just come, and I’ll never bring that up. But if I had to guess, I do feel like you like a certain mood, and you’re bringing up claims that match the mood.
Ori 00:15:58
Well, it’s interesting that you call that psychoanalysis because that didn’t strike me as psychoanalysis. To me, that sounded like a logic analysis. I mean, I know to you rationalization is such a sin, but that’s different than saying, “Well, you’re the kind of person who loves technology, therefore you’re just giving the pro-technology view.”
You didn’t say anything about his emotional state. All you said was that you think he’s not reasoning from the premise to the conclusion.
Liron 00:16:35
Right. Well, when I say psychoanalyze, I just mean that I’m making a claim about what’s going on in the person’s brain as opposed to a claim about the arguments that they’ve laid out. Because when I’m making a claim about the arguments they’ve laid out, I’m completely removing the person except for us to ask the person to clarify their position. So in that sense, I involve the person — “Okay, what would you say about this?” — because I need to get the position from the person.
But once I have the position, I’m just asking about the position, so the person can go away. And the position can, in theory, just speak for itself. So I just try to interrogate the position, the connection between the premises and the conclusions. That’s my MO on the show. That’s what high-quality debate is — you don’t question the person’s motives.
But this is a bonus. This is not a main episode, so in this particular episode, we’re allowed to do that.
Ori 00:17:27
Got it. Yeah. And I think it’s a strong critique, and I’m persuaded by it, the rationalization critique.
Liron 00:17:41
Right. I mean, so in round one I did accuse him to his face. I violated my own protocol a little bit—
Ori 00:17:45
Yeah.
Liron 00:17:45
—but I apologized for it, and I let him do it to me. Which is, “Don’t you think that you may be rationalizing? You may be digging up arguments to get to this particular conclusion.”
And my evidence — the reason why between you and me, privately, the reason why I feel like he’s rationalizing is because he gets off at all these different stops. I just said, “You get off at all these different stops.” But I didn’t go farther and say, “Doesn’t the fact that you’re getting off at all these different stops at once make it seem like you’ve done a backwards search for stops to get off at instead of reasoning forward from ‘Oh, wow, this stop is so persuasive on its own?’”
Ori 00:18:22
Hmm. Got it.
Liron 00:18:24
You know what I’m saying? Like, “Oh, Ricardo’s law of comparative advantage is strong, and orthogonality thesis is strong.”
Ori 00:18:31
Okay. Yeah. I think the key is you’ve improved your radar to hunt out rationalizations. And so I don’t have that great of an appreciation for spotting rationalizations, but it’s cool that you’re good at that.
Liron 00:18:49
Well, I’ve been trained. People act like this is not a serious — people on social media act like Eliezer is just some random kook. But no, he did write an actual textbook-grade education on the actual art of rationality. In the sense that anything can be an art, the way to wield your brain rationally is very much an art. Rationality from AI to Zombies is a hell of a book. And it does tell you a lot about what’s going on in your brain and how to wield your brain to get at truth.
And one of the things it points out is when somebody writes down a conclusion on a piece of paper, there’s two different traces connected to that conclusion. There’s what you claim is the argument or the justification, the explicit thing written on the paper, but then there’s the explanatory trace, the debugging trace, the causal trace — what chain of mental events accounts for the motion of your pen on that paper.
And that may not always map to the lines of argument you’ve written on the paper. Because there’s two different chains. You made it look like there’s a chain of argument, but then in your brain there’s a different causal chain. And if I’m rational, I like to think that when I write something down, it matches pretty well because I’m a rational guy. But I’m accusing Mike right now. No offense — it’s a common thing. I’m accusing Mike of having his arguments not be isomorphic to the causal chains in his brain.
Ori 00:20:12
Interesting. Yeah, I remember when you brought it up with Eliezer, and it was a fascinating point. It hadn’t been something that I had come across before.
Liron 00:20:22
Yeah, viewers, I highly recommend just search for LessWrong, “The Bottom Line.” As I told Eliezer in the interview, when I read “The Bottom Line,” it was a profound moment.
Ori 00:20:31
Wow.
Liron 00:20:33
Yeah, because of what I just said about the causal trace. Okay, so we got Jack II4FI. Whoa, I wonder if he’s a 47F FUCB’s friend. He’s got a nice hash there.
All right, so I think it’s a YouTube auto-generated username suffix. Because at one point YouTube was probably like, “What do you want your username to be?” And he’s like, “Oh, Jack.” And YouTube’s like, “Yeah, okay, no problem. Jack-II4FI.” So he can slot in together with a billion other Jacks.
So he’s saying, “I don’t know where that came from.” Okay, so let’s take them in order. Alpha Di—
Ori 00:21:16
That’s all right. No, you nailed it. It’s funny. It’s hilarious.
Liron 00:21:19
Okay. Alpha Diversity is saying, “Relevant. I’m a psychology MSc studying belief-action gap and AI x-risk, measuring psychological distance, risk perception, and perceived AI knowledge as predictors of civic engagement.” All right, cool. Sounds good. Yeah, let us know if you wanna discuss anything in particular. I mean, belief-action gap — I guess that’s kind of like what we were just saying about explaining what somebody says versus the actual reason that their brain made them do something. Maybe that’s related. All right.
So Jack’s first comment is saying, “I think the majority of your views are very strong, Liron. But just out of curiosity, how do you apply what you’re saying to yourself and your own rationalization?” Oh, snap.
Ori 00:22:00
Ooh. You got told.
Liron 00:22:02
Everybody likes to go recursive. All right, and then he says, “Just out of curiosity, to what extent do you worry about or notice that in your own reasoning, however small? Also, is that a cherry Doom Debates Coke?” That’s right, Doom Debates X collab Coca-Cola. This is an exclusive model right here. It’s a cherry Coke. I got it on DoorDash.
Because DoorDash, they always do a million substitutions. I actually don’t mind cherry Coke. I’m not complaining. Cherry Coke Zero, right? Yeah, zero calories. Okay, phew. I got spooked there because I was only seeing cherry Coke. I was like, “Am I drinking high fructose corn syrup right now?” But then I rotated it a little, and I saw it said zero, so I’m good. Crisis averted.
Ori 00:22:42
Different sugar, huh? Not the high fructose corn syrup.
Liron 00:22:45
No, no, aspartame. There’s no problem. Although a lot of people claim that it tricks your brain into producing insulin anyway, even though it’s not sugar. But that’s neither here nor there.
DoorDash, they always give you a substitution. So you’ll make a list of 10 things that you wanna buy from the grocery store, and I never go 10 for 10. There’s always a substitution or item not available. So in this case, I didn’t get regular Diet Coke. I got Cherry Diet Coke.
Discuss. All right, so getting back to Jack here. “To what extent do you worry about or notice that?” Yeah, okay, so this is a good question. Do I rationalize?
As far as I can tell, I have a natural advantage here. The natural advantage is that I have low emotions tied to my reasoning. So my brain does just feel like a pretty neutral blank canvas for reasoning.
It has this nice property which is kind of computer-like in the sense that — think about a computer when you’re evaluating an expression on a Python command line or JavaScript or whatever, and you just type an expression, type a function call, press Enter. And until you press Enter, the computer’s totally neutral, and then you press Enter, and then it just does exactly what you typed in. It’s just evaluating the meaning of what you wrote.
So I do think that my brain naturally has that kind of neutrality to it, where you can just go ahead and load anything, and it doesn’t immediately have these waves coming in of all the associated emotions and connotations. I’m aware of the connotations, but I’m still able to just have that neutral workspace. Ori, what are your thoughts so far?
Ori 00:24:25
I mean, I take you at your word. I generally take people at their word, and I think you’re onto something with the fact that you’re low emotion. And related to that, I also think there’s something about you that’s attracted to the rationality arguments. I mean, I think about your blog way back in the day, and your blog way back in the day was called Logic and Reason, which are the things that you’re interested in.
Liron 00:24:50
That’s true. I did have a blog called Logic and Reason, which was best known for me taking an apple and carving everything out under the apple skin and then manhandling a peeled orange into it, and then taking a picture and posting it as a comment on the Freakonomics blog. “Look, I made one of these.”
Ori 00:25:06
I had no idea you did that. Great work.
Liron 00:25:09
Yeah, because that fruit looked really good. We gotta find that fruit on the Freakonomics cover. Actually, it’s still on there. Yeah, they’re still using it. Here, I’ll screen share.
And actually, I think Stephen Dubner actually gave me a shout-out on the blog, “Oh, our readers are so creative.” And then I had to clarify. I was like, “Professor Dubner, I just wanna let you know that what you’re seeing here is not Photoshop.” Yeah, so this is the fruit on the cover, which does look like Photoshop, right?
Ori 00:25:36
Right. Wow. How did you do that?
Liron 00:25:39
So I literally just hollowed out an apple and put a peeled orange inside and kind of taped it together. And the tape was on the back, so it looked — I mean, it didn’t look that convincing. But anyway—
Ori 00:25:50
Right.
Liron 00:25:50
So I was like, “Yeah, Professor Dubner, I’m glad you like my fruit, and I just wanna let you know it’s not Photoshop.” And he’s like, “Yeah, no, it’s pretty clear it’s not Photoshop,” because it wasn’t AI generated or anything. It was obviously you couldn’t Photoshop that one. But yeah, so that was probably my best post on my college-level blog circa 2006.
Ori 00:26:09
Yeah. And I think you’re interested in making an impact. You wanna be a good investor. So I think you’re constantly practicing your premise-conclusion — learning how to be rational. Seems like you’re doing that all the time.
Liron 00:26:26
Yeah, thanks. Let me finish explaining that. So first of all, I think that’s an advantage, that my brain works like a compiler, and it’s taken me a long time to accurately model other people as not having that property.
It’s kind of like association. It’s kind of like why when you’re at the dinner table — my son was talking about poop at the dinner table, and even I can feel that. So if I’m eating, I’m like, “Oh, you said poop,” and I’ve got food in my mouth. So now there’s a little bit of that sense, “Oh, is there poop in my food?” It kind of leaks in even though logically it really doesn’t have to, but it kind of does.
Even I can feel that. But I think some people have that to a higher degree even in other contexts. Like, “Oh my God, you mentioned abortion. Now I can’t think about anything, and now I have to embrace the entire left-wing platform because you said pro-choice.” People really draw in so many associations.
Ori 00:27:19
Yeah.
Liron 00:27:19
So the whole notion of being politically left was so weird to me for such a long time. I’m like, “Wait, so you just associate all these other policy positions together as a direction, left?” I don’t understand. Don’t you just wanna have a thousand-dimensional political position?
Ori 00:27:38
Mm. Yeah.
Liron 00:27:40
Yeah. So my brain has a lot of degrees of freedom. And I also think that changing my mind comes much easier than average. I’m not saying it’s the easiest thing ever. Don’t get me wrong — I agree there’s some ego tied to being right. But I actually think I also have a lot of ego tied to my rationality itself.
And in the rationalist community, there’s even cultural reinforcement of, “Oh yeah, this person changed their mind. That’s really cool and baller.” You get points for that, and that’s the kind of thing that gets upvoted the most on LessWrong — somebody talking about changing their mind. So I think I’m ready to do it. I think I’m prepared when the occasion comes to change my mind. I don’t think I’m locked into my position.
Obviously, when I wake up, it’s loaded into my memory. It’s cached into my memory, so I can really quickly tell you, “Yeah, I think P(doom) is high. I’m a doomer.” So I agree it’s cached in my memory.
But it’s another thing about my brain. The way that I usually work is I remember my current best guess position, which in this case is P(doom) is high. But if somebody says a different position, it’s easy for me to be like, “Okay, yeah, let me just try this on.” Which I often do in the interview in the form of just trying to get their position out. So I’m often like, “Let me take your position. Let me explore it.” I’m always asking them questions about their position.
And it’s pretty easy for me to be like, “Okay, yeah, I can imagine being on your team, hanging out with you,” which is basically the attitude I take when I’m hanging out with people that I vehemently disagree with. I’m like, “That’s fine. Let me just put on your clothing for a while so that I’m not making an ass of myself. I’m just living in your world. Even though I disagree, I’m still happy to empathize with how you see things.”
But it’s a pretty short hop for me to be like, “Okay, well, can I just see things that way? Can I just try on this belief?” And it’s pretty comfortable for me to imagine myself be like, “Okay, yeah, I could live like this.” So I don’t feel like it’s a big deal to just change my mind and adopt a different belief state. Because of that, I think my brain is actually well-positioned to just hear out arguments, actually consider them, as opposed to fighting this rear guard defensive position on something I already believe. So I think I’m a natural low rationalizer.
Ori 00:29:47
Nice.
Liron 00:29:48
Now that said, am I immune from rationalization? No, absolutely not. And I can definitely — I think I’m guilty of rationalizing plenty of things in my life. I don’t think that anybody fought me that hard, constantly. There was nobody being like, “Come on, you’re rationalizing, you’re rationalizing, you’re rationalizing.” And then finally I’m like, “Ah, that person totally called me out.” It’s happened to a small degree, but it’s never been a major thing. So I’ve never had an epiphany that I was rationalizing, but I can certainly identify little moments when I was rationalizing.
Actually, probably the thing I’m number one most guilty of is rationalizing — or rather being overconfident in my startups that failed. So I’ve definitely had a couple startup failures here and there. And at the time — now in my defense, I just wasn’t fully baked. All the knowledge that I’m giving out now, I hadn’t figured out yet.
So in my defense, I do feel like my framework of life skills is a lot more mature than it has been in the past. And if I could just hand down some of the tools that I have now, I feel like I could’ve used the tools to solve my own problem. But I think it’s fair to describe myself as a startup founder being like, “Hey, I feel like this can work.” It’s fair to describe it as being highly biased. But is it rationalizing? I don’t feel like I was super confident it would work. I feel like I was taking a good shot, but it wasn’t like, “Oh, this has to work. This is so strong.” I don’t think I was ever super convinced about it. So that’s — yeah, I could say that in my defense.
Ori 00:31:15
Yeah. I mean, would you accuse someone like Elizabeth Holmes of rationalizing? I mean — or kind of, yeah.
Liron 00:31:20
So Elizabeth Holmes, my mental model of an Elizabeth Holmes type, which is probably similar to the Sam Bankman-Fried type — and I personally know one or two people who I would classify as being that type — is you basically lock onto a position, and your attitude is basically, “Look, I can sell this. I can sell it to other people, and I can sell it to myself, and I can just live this position. I can just live my life as if this position is true. And who’s gonna stop me?”
And maybe reality will come bend to it. And sometimes it does — sometimes they do bend reality. And so they don’t have that many critical faculties. They don’t have the mental habit of challenging it or following rigorous arguments or even making convincing arguments — making rationalist-grade convincing arguments. But they just have the persuasion toolkit. They just live life in a persuasion-centered way.
Ori 00:32:08
Yeah, yeah. I would agree with that. But the key issue is, were they rationalizing? I think you could claim that she was rationalizing because she really had to make the blood machine work, and all the signs were that it was not working, and then they started faking that it was working. And I guess under the idea that they would figure it out later.
Liron 00:32:28
Right. And that’s one of the biggest things that separates Holmes from certain other people, which is the idea of, “Okay, it’s clearly not working, but why don’t we just do a little more deception, a little more...” And the thing is, it is a gray area. Because I’ve certainly seen firsthand people being like, “Oh, for my graph, let me just tweak the numbers in my graph or redefine what the numbers mean,” or “Let me slap on...” A classic move in the startup community is, “Oh, let me slap on this logo. It’s not really our customer, but we had a meeting with them. We’re just gonna put it on our logo slide.”
So they’re fudging who gets to go on the logo slide. And that’s pretty bad. There are certain things that get a reputation for being especially lame when a lot of people try them, but there’s always things on the fringe that it’s like, well, this isn’t a known kind of thing you can fudge. This is a creative new thing you can fudge. And at some point, you get to the point where you’re fudging people’s freaking lab results, where you’ve definitely crossed the line by that point.
Ori 00:33:22
Right.
Liron 00:33:22
But I still think that there’s a spectrum. I think Holmes is on the extra pathological end of the spectrum. I don’t think that anybody I know would’ve gone quite that far, even though I know some persuasive people. But yeah, I’ve seen people go halfway there.
Ori 00:33:38
Yeah. You know what this leads to is something that I was thinking about in light of the Sam Altman-Elon Musk case. Here’s a question. There’s so much antagonism towards Sam Altman. There’s some antagonism towards ChatGPT, the “Quit GPT” movement.
Let’s just say that if AI progress stopped somehow, like there was a blocker in the chips, and we just kept GPT at this level. It’s not an x-risk, okay? So everything just stayed where it is. In that case, is Sam Altman acting in an unethical way?
Oh, we got a donation.
Is Sam Altman Any Worse than a Typical Tech CEO?
Liron 00:34:20
All right. Get out of the way. The train’s coming, man. The donation train’s coming.
Ori 00:34:27
Yeah, no.
Liron 00:34:27
Okay, yeah. Keep going, and then I’ll read the message later.
Ori 00:34:31
Yeah. So in that situation, the question is, is Sam Altman doing anything that’s so unethical? Is he doing something that’s so different than what a regular tech CEO would be doing? Is what he’s doing so wrong in that situation?
Liron 00:34:51
Right. So a lot of people’s beefs with Sam — this is my current read on Sam. I do think he’s got a little bit of that Elizabeth Holmes-iness. He’s got that reality distortion field. “I’m just gonna play everybody off each other. I’m just gonna brute force my way through these situations.” It’s just like a gift. He’s the Magnus Carlsen of putting the pieces together of fundraising and business. It’s like Michael Jordan basically.
And part of that is stomping on rational arguments a little bit. So that’s how the game is played. Even my hero, Elon Musk, plays the same game to a pretty significant degree.
So there’s that element of him. And the thing is that I normally wouldn’t care that much because he’s got good elements, too. There’s plenty of accounts of Sam Altman being super helpful, supporting good projects. As one example, there’s this project called Eve that I thought was really cool, a new programming language, and Sam Altman was like, “Oh, I really like Eve.” This is 10 years ago, and I thought it was pretty promising even though it failed.
And Sam Altman’s like, “Yeah, this is one of the coolest things I’ve ever seen.” And I thought it was kind of cool how he supported Reddit. Although I also heard gossip, like, oh, he kind of helped the Reddit spin out in a pretty shady way. So there was always a combination of visionary but also a little gray area tactics. And all that is potentially fine with me. You’re playing the game. You’re not literally breaking the law.
My beef really just comes about when you get into superintelligence safety. That’s the problem — he’s just legitimizing racing towards superintelligence and downplaying the fears of Eliezer. But in that sense, he’s really about as bad as a Dario. There’s not that much gap between them.
And so I guess I’d pick Dario because I think Dario is just a less shady type of person overall. He has more integrity is the sense I get. But I think they’re both same league of bad for humanity.
Ori 00:36:50
Yeah. I think it’s interesting nuances that if you took away that portion of it, that part of the argument, it’s like, all right, he’s just like — in a sense, he’s not that different from other tech CEOs. Yeah, he’s a bit aggressive, and some people are anti-oligarch, so maybe they’re against big tech. But the regular CEO behavior that he does is not what you find so objectionable. It’s because of the superintelligence risk.
Liron 00:37:22
Exactly. Well, one of the things people come to Doom Debates for is that we’re always careful in which attack we’re leveling at people. We’re not just throwing the kitchen sink at people, because there’s so many things. We never went with the data center water type of stuff. We try to be careful.
Liron 00:37:43
Yeah. Okay, hold on. So let me get back to these YouTubers here. I just wanted to finish out the line of argument. I’ll get to the sponsored question soon, of course. Gotta prioritize the sponsored question. But I’ll also prioritize chronologically.
Let me see. So Eaches was saying there was a video from the Primagen. Oh yeah, we’ll get back to that. We’ll get to the Primagen video. That’s a good one.
Nogi8 says, “I also notice how Dr. Mike sometimes went from, ‘It won’t,’ back to, ‘It can’t.’ But when directly asked, he says he believes it absolutely can.” Yeah, okay.
Roy Holden says, “Liron, I wanna press you on your own P(doom). Having watched nearly all your videos, it really doesn’t seem like you can envision a path where humanity wins. Is your P(doom) to 2100 closer to 99?”
I agree. I can’t imagine a path where humanity wins. But I’m just trying to be open-minded. Me being unable to imagine something doesn’t foreclose on it. I get surprised all the time in ways that I didn’t even think that I was allowed to be surprised. So I’m just trying to hold out some — there is a place for modesty. I don’t think modesty should make you say that P(doom) is really tiny, but I think modesty should make you say that it’s not 100%.
Ori 00:38:54
Oh, snap. That one’s an interesting one, because what you just said reminds me of Geoffrey Hinton. You remember Geoffrey Hinton’s famous P(doom) quote where he’s like, “You know, I’m kind of 50/50 on it, but people who I trust say it’s lower than that, so I say around 10 to 20%.”
And you’re kind of giving a similar argument, which is, “Yeah, it is a bit higher in my mind, but there’s other people, there’s unknowns, so I’m just gonna lower it because of that.”
Liron 00:39:23
No, I hear you. It’s just that — I mention this occasionally, this idea of you really think you know something, you think you’ve got a pretty good argument, and yet the back door was bigger than you thought. That’s certainly an experience that I’ve had on a number of fronts, and I’m somebody who believes in pure logic quite a lot.
We have places in society where things do have a 99.99999% success rate. Like google.com — the uptime of the homepage of Google, it really is. I don’t even know how many nines that is. Eight nines, ten nines, some insane amount.
Definitely if I had to bet my children’s lives on the next time I go to google.com working — if I could get a dollar in return, that’s pretty tempting. You go to google.com, if it works, you get a dollar, and if it fails, then your kids die. Hmm, I gotta think about that, because google.com always works.
But of course, I wouldn’t do it because even in the conditions of making that particular bet, I’m like, “No, me making this bet probably won’t have 99.9999% for whatever reason.” First of all, my own internet’s gonna go down. So there’s a million conditions.
But point is, it is a statistical fact that if a bunch of people were to make that bet, it does seem like a crazy high fraction of them would win that bet, and no kids would die actually. Every single person on Earth would win the bet. So yeah, there are ways to be really confident. I don’t think that we are quite that confident on the AI front.
Liron 00:41:11
All right. So yeah, we got a big backlog here. I do wanna, before we get to the sponsored ones, I wanna get to one more back on the me rationalizing part, because that’s a touchy subject. Remember I told Lumpin Space, I said, “I resent that,” when he accused me of not being open-minded to his arguments.
So Luciano VR is saying, “Liron’s way of rationalizing is, for example, when Lumpin Space asks how the nanobots would communicate, he just repeats that it’s a superintelligence, avoiding the objection of physical limits.”
Okay, I don’t think I avoided the objection. I think that I specifically asked Lumpin Space back. I said, “Is that really your objection, how they would communicate? Do you really wanna rest your argument? If I could just show you that radio communication is a pretty powerful way for nanobots to coordinate, would that invalidate your argument?” And I don’t even think he pushed strongly on that.
Ori 00:41:48
Yeah.
Liron 00:41:50
Yeah, I think going all the way to “Liron is rationalizing doom because he’s saying communication is easy” — look, communication really does seem easy. I don’t think that’s me rationalizing.
Think about the fact that your cell phone — you can be out in the middle of the woods, and it can still somehow signal to a cell tower on a tiny amount of battery. And then get this, the AI can build a lot of cell towers. This is an overdetermined problem. It’s not gonna be like, “Oh my God, can it communicate halfway around the world?” No, just keep building more towers as needed.
Ori 00:42:18
Yeah. And also aren’t drones currently a proof point, a sample of it? Because you could see those drone swarms and they’re incredible, and they do visualizations in the sky and it’s amazing. And it’s like, okay, if you could do that with drones, certainly the technology to do that can get smaller and smaller.
Liron 00:42:41
Exactly. I mean, I’ll just introspect on myself. I really think that communication is such a fertile area for building more and more powerful technology on. Are you kidding me? I really think that the degrees of powerful communication and information transfer across space — I really think that the laws of the physical universe allow quite a lot of. I really think that.
I don’t think that is backward chained from my belief about AI is going to take over because it’s going to have such high bandwidth communication. I just think the universe allows extremely high bandwidth communication. I don’t think that’s rationalization.
Ori 00:43:16
Yeah. I’ll psychologize for a second. I didn’t think — I thought Lumpin Space was sort of, he had covered a lot of logical, rational arguments, and then he kind of threw out an insult a little bit. It was maybe a last resort type thing.
Liron 00:43:35
He wanted to get the Hail Mary to discredit me.
Ori 00:43:35
Yeah.
Liron 00:43:35
All right. Let’s go to the next one here. New Place to Frown is saying, “Having good criteria for what you need to see to change your mind is great for both yourself and the community.” Yeah.
I just wanna say, when you come to Doom Debates, you’re getting a crux-driven type of argument, meaning I said this before in my Eliezer interview — when I come out into the debate, it’s like imagine Achilles enters the fight, and he’s beating his chest, “Yeah, you wanna fight me?” And then he just takes out a big arrow, and he points it at his heel. He’s like, “All right, this is how you attack me right here. This is the weak spot.”
Not necessarily the weak spot, but this is how you get me to come to your side. This is how you beat me. I see that as my job — to point to what you can change my mind about in order to beat me. I’m not trying to hide it. I’m not trying to put up a front like everything is strong. I’m trying to tell you what you can change.
I see that as the opponent’s job too, but the opponent is almost never straightforward about it. But I try to get them to be. So I just want you guys to know that’s what I’m aiming for. I’m aiming for everybody to point to their changeable parts.
Liron 00:44:40
Yeah. All right, I hear you’re typing up a storm right now.
Ori 00:44:42
Oh, sorry.
Liron 00:44:46
All right. So Alpha Diversity is saying, “So far, the median participant civic engagement score in my study of people who already self-identify as AI x-risk aware is six out of a possible 40. What is your view on what drives that gap? Civic engagement is actions on a public issue — learning about it, discussing it, signing/voting, donating, joining groups.”
I don’t know. I mean, my own civic engagement is kind of low, I guess. Speaking for myself, it’s just nerds tend to not be that much into politics. It’s different hobbies. ## Lowering P(Doom) and Existential Bio Risk
What Would Lower P(Doom) Below 10%?
Liron 00:45:17
All right, all right, all right. So AIxRisk, we gotta prioritize the premium tweets here. And by the way, I wanted to let you guys know that if you go to YouTube, there’s this feature that I use, which is a live stream goal. Just like an AI is going to converge to having goals, so does YouTube Stream have goals. Everything has goals.
It’s literally a YouTube feature called a stream donation goal, and I set the goal to ten donations of $20. I’m being a little ambitious. But if you guys do that, then we’ll extend the stream beyond 90 minutes or something. We’ll do an extra 30 minutes, go into overtime. So yeah, just a little sweetener here.
All right, so AIxRisk with a 10 euros donation, they’re saying, “What kind of progress and alignment would cause you to lower your P(Doom) below 10%? Can you paint a scenario? How would that look like?” Geez, I mean, 10% is so safe, right? Nuclear doom and bio doom, to me, they’re both independently around 10% in the next century. Because mirror life — you file that under bio doom.
I think, to use a phrase that you guys like, I think our pants are down in both senses. I think we’re highly vulnerable to both nukes and bio. Can we claw our way back up from being cavemen? Sure. So maybe if you don’t count that as doom, then there’s a lower risk. But I do see the state of the human layer on Earth — we’re just this tiny thin layer on spaceship Earth. I don’t think it’s very well defended. We don’t have backups. This isn’t a serious way to persist through time. An alien looking at us would be like, “Oh, wow. They’re really hanging on here. No backups?” For God’s sake, no backups. That’s terrible practice to have no backups.
Ori 00:47:01
The bio risk one is one that stands out to me, because we are so vulnerable to it, and there’s one report after another about just how accessible AI has made bio hacking. Going back to the Israetel debate, you were like, “Mike, come on, give me 5% here. Isn’t there enough to say the risk is 5%?” And yeah, I agree with you that just on its own, the bio risk is 10%. I would actually consider the bio risk to be higher than nuclear because — but anyway.
It just seems like such an overwhelming threat. So I’m curious — give me your best counterargument to bio risk. You got someone like Vitalik who’s like, “Yeah, we’ll develop these personal antibacterial defenders.” He had some kind of personal antivirus thing that he’s working on. It just sounds so far behind the ability to make pathogens.
Liron 00:48:10
Well, I will say if anybody’s ever going to succeed with DAC — the decentralized defensive acceleration — bio sounds like the greatest domain to pursue the DAC strategy. Because this idea that we can all live in a well-defended bubble — actually, cyber too. I’m actually willing to believe that in cyberspace and in biospace, you can put up these decentralized defenses. “I’m immune to everything. I’ve got a bio scanner. There’s scanners in my house. There’s a scanner chip inside my mouth, scanning anything that goes in my mouth, constantly blood testing or whatever, or nanobots in my body.” I actually think you can have a defensive perimeter for bio invasion. I’m pretty optimistic that the equilibrium will allow that.
And similarly, with cybersecurity, as long as you can monitor every packet that comes in, you’ve got these nice discrete entry points. I’m pretty optimistic that eventually defense will win in cyber. However, I think the physical universe — the energy transfer of regions of 3D space — I think is looking bad. If somebody just lobs a bunch of energy at you, I think you’re screwed. You can’t build a defensive perimeter.
Ori 00:49:17
What’s the difference between that—
Liron 00:49:17
So that touches on nuclear, basically.
Ori 00:49:19
What’s the difference between that and biosecurity?
Liron 00:49:23
So with biosecurity, I think that when you get bio attacked, it’s gonna happen at relatively discrete entry points. Somebody coughed a virus at you, so you just have to scan the entry points, and I think you have a little bit of time. You can get your defenses going if they’re targeted. The timescale is fine and the scale of the attack is fine.
Whereas if people talk about nuclear defense, they’re like, “Oh, well, we’ll shoot down the nuclear missile,” or, “We’ll somehow build underground to protect against the explosion.” That seems much harder because of the laws of physics. The laws of physics seem like you get an advantage when you can lob a bunch of energy — when you’re attacking and you can bring together a bunch of energy and direct it.
Ori 00:50:05
I see, I see. Okay. Fair enough. Hey, this is the rational mind at work here. Those are — you’re sort of looking at it on a simple basis, the physics of it makes it harder.
Liron 00:50:18
I mean, that’s generally the level at which I analyze. And sometimes the funny thing is that sometimes it’s incredibly powerful. Sometimes my way of analyzing things based on a few simple parts is like, “Hey, look, here’s something that’s underpriced by a factor of 10. Let’s grab the value.” Sometimes it’s ridiculously powerful and it trumps everything. But in most domains it’s just — I don’t know, these things are in the same ballpark. That’s usually what it outputs.
Ori 00:50:41
Yeah. I mean, my reaction to your biosecurity thing is sort of — I think you might be onto something in principle, but the technology is quite far behind what you’ve outlined. To be able to defend the main entry points — we’re much further along in something to do that in cybersecurity, where it’s a human-constructed engineered field. But the understanding of nature and biology is not at the point — seems quite far from the point that you described.
Liron 00:51:13
Yeah, yeah. Don’t get me wrong. Right now, if you had an AI doing offense and an AI doing defense, I think offense is gonna have the advantage for the next few years in bio. So I think we’re entering this super unsafe time, where a terrorist with an AI could get off a successful attack. I actually think this is a critical period.
And it’s similar in cybersecurity — the attacker could get off — but I think that the defender can zip up their pants a lot faster in the computer security space. So you can imagine we have one year of critical period in cybersecurity, and then defenders win. Whereas in bio, we have 5 or 10 years of critical period where attack has the advantage, and then defenders win. But we might already all be dead from a pandemic before that happens.
So yeah, I mean, don’t get me wrong, this is a super important area. I’m personally not gonna touch it. It’s just not my wheelhouse. But if people wanna go into defending from bio attacks, that’s a rich vein of helping humanity survive, for sure.
Ori 00:52:04
That’s a good point.
Liron 00:52:07
And I will add an asterisk about cybersecurity, which is even though the domain itself of computers taking packets over the internet — I think that’s probably going to be possible to defend. But I think that in cybersecurity, there’s crazy side channel attacks and social engineering attacks.
So realistically, are computer systems going to be secure? No. The AI is just gonna get humans to input enough things — humans with enough privileges to type enough things into the system that it’ll still break the system. Because it just needs a foothold. Can there be layers where an AI can get a foothold and then be detected? I don’t know.
But at the end of the day, if we actually have a superintelligent AI, even if Mythos has defended some computer system, the next superintelligent AI is still going to have this larger view where it’s like, “Okay, well, I’ll just bomb this part of the system. I’ll manipulate the human on this part of the system. I’ll do a timing attack on this part of the system where it’s not technically part of the code, so Mythos didn’t think about it.” It’s just gonna combine these things that use its knowledge of the whole physical universe. So even though computer security seems like it could be a nice defendable niche, I think the actual superintelligence is going to end up having an attacker’s advantage. That’s my best guess.
Ori 00:53:17
Yeah. Makes sense.
Liron 00:53:19
All right. No debate here. Let’s go to Ray Grant TZMS with a $10 donation.
Can We Disconnect an ASI from the Internet?
Ori 00:53:27
Whoo.
Liron 00:53:27
He says, “Would it be possible to strategically disconnect enough potentially dangerous things from the internet that even an ASI couldn’t gain immediate control of everything?” Solid question. I’m gonna put a heart on that.
I mean, in theory, sure. In theory, you can do a lot of things defensively, but in practice we’re just nowhere near that and never will be. The one thing I actually see happening is I do still imagine that actually firing the nukes will still be run on a low tech system where a human has to turn a key and then it’s just using electricity or relays or old school switches without a freaking Android installation. “Oh, Samsung wants you to update to the latest app store.” I like to think that that’s not going to happen inside of these planes that can drop these nuclear missiles — or is it still called a missile if you drop it from a plane? Maybe just a warhead. Not really a military guy. Maybe it’s called a nuclear bomb at that point. A missile has to fire from the ground or sea. I don’t know. Wrong show.
But yeah, I like to think that you’re not going to be able to hack into nuclear missile launchers from the internet. They’ll be smart enough to do that. But a drone army — we’re gonna have a million drones, and they probably will be reachable from the internet, even if it’s not technically the internet, but you just need to hack into some wireless antenna that can talk to them on some protocol. It’s gonna be sufficiently connected to the internet.
At this point, I’m just not seeing a lack of connection being the thing that saves us. I think we’re so far past that point. There’s so many causal channels. Ultimately, you gotta look at the causal web. The causal web is just anything can influence anything causally. So if I call up a person on the phone, I’m using the electrons in the phone network to vibrate in order to get the phone to do audio, and then the person’s brain is gonna send signals. But ultimately, it’s all cause and effect. That’s the one type of thing that links everything. And if you look at cause and effect as a web, there’s just way more causal connections to things than you need. There’s no proper isolation in the causal web.
Liron 00:55:56
Now, if you were building a universe from scratch — if I were designing a universe, I don’t wanna brag, but I think I could do a better job than God at designing a universe. Because, hello, God, ever heard of virtualization? That’s a nice property for a universe to have. There should be able to have regions of the universe that you can tightly control the inputs to those regions. Stop letting the entire space around them send inputs to them. That’s a terrible architecture if you wanna do cool things in your universe.
Now, some people have argued that black holes are God’s method for virtualization. Once you have a black hole, you can spin off a whole other universe, and nobody else can send messages to that universe without going through the opening of the black hole. That’s what passes for my understanding of cosmology. So if that’s God’s solution for virtualization, kudos. That sounds pretty solid. But virtualization within our bubble is really crappy.
Ori 00:56:32
Virtualization is basically the ability to — you use your computer, but the ability to use your computer on a fake computer, basically, a simulation of—
Liron 00:56:40
Yeah, virtualization is encapsulation. So it could just be: this region of the universe, you can only enter through this front door. You can’t see what I’m doing. It’s a black box. You can send inputs and get outputs, but you don’t have any access to what’s going on inside.
Ori 00:56:54
And why is that missing from our universe?
Liron 00:56:58
Yeah, it’s missing from our universe because — imagine I’m like, “Okay, Ori, here’s my big bank vault. You don’t know what’s happening in the bank vault. I’ll just tell you what your balance of money is, and you can put money in, and I’ll store it for you, and you can take it out. All you know is that you can put in and take out. You don’t know what’s going on. Maybe inside there’s an orgy. But all you know is that there’s a balance. I represent the state to you as a single number. You don’t know what’s going on inside the vault.”
But because God did a bad job designing the universe, in fact, somebody could come in with an infrared scanner or send alpha particles or neutrinos or whatever, and somebody could get all this data about what’s inside the vault even though I’m trying to expose an interface to you. I’m trying to virtualize this, and the universe isn’t letting me.
Ori 00:57:47
Got it. Got it. Okay.
Liron 00:57:48
So somebody could actually know that the orgy is happening, but that orgy is none of your business.
Ori 00:57:52
Even thoughts. You could scan someone’s brain now, and it’s really depicting what is happening inside of it. We’re getting close to that ability.
Liron 00:58:09
Right. So the idea that when you’re talking to somebody, you have to go in through the audio channel — well, you can just see their neurons. And there is a possibility that an AI — this is the kind of stuff Eliezer Yudkowsky talks about — an AI can just come in and flash certain lights at your eyes that you never thought of as being a way you take input. But it turns out that you do actually have a causal pathway where your neurons respond to light.
Seizure is this awesome example, because who would have thought? If I had never heard that seizures were a thing — I personally don’t suffer from seizures. You can flash lights at me, and as far as I know, it won’t make me seize up. Although now that I think about it, I suspect a superintelligent AI could activate my seizures. Even though I’ve never had a seizure before in my life, I’m gonna go ahead and lean towards saying that a superintelligent AI could do it.
Ori 00:59:00
I would agree with that, especially considering all the news about the attack in — where was it? Cuba? Where there was a sound wave that hit people in an embassy in Cuba, and that affected their brain basically by a sound wave. So if human engineers can do that, then I would agree that superintelligence can seize you up like that.
Liron 00:59:15
Yeah. So it could seize you up. But the obvious question is, okay, well, how much can it really do? Can it really turn you into a slave? Can it really puppeteer you just because it can flash signals at you? Maybe it’s not that simple. The obvious vector by which to puppeteer you is to just get in your social media.
Think about pig butchering scams. People get — imagine the best pig butchering scammer. You get those texts, and the text is, “Oh, hey, are you going to the golf club tomorrow?” And it’s a number you’ve never seen. But the strategy is, “Oh my God, I’m getting a text from somebody who is talking to me about a golf club, so clearly this is a person of high socioeconomic status. I should text them and build rapport with them.” And then really they’re just trying to make friends with you and be like, “Oh, hey, I got this investment.” And then — long story short — you send them Bitcoin. That’s the pig butchering scam. But they’ve befriended you for three months. So then you’re thinking, “Well, why would this friend of mine who’s been talking to me for three months — why would this all be a long con to get me to send Bitcoin?” But that is, in fact, what it is.
Ori 01:00:16
Yeah. Yeah.
Liron 01:00:17
So the point is, if the AI wanted to do stuff, that’s the first thing that comes to my mind as a human-level intelligence — it could just pig butcher all of us. It would be the most talented pig butcher on the planet.
Ori 01:00:27
Wow. Sam Altman, eat your heart out.
Liron 01:00:33
Yeah, yeah, exactly. All right, let’s see what the commenters are saying here. So New Place to Frown is saying, “Jack, it’s funny that Eliezer uses the AI permanently destroys water meme to capture some of the masses.” Yeah, that’s kind of a deep cut.
Okay, Luciano — he was saying before that I was rationalizing with “lump in space.” Luciano’s saying, “Yes, but the keyword is nano and nanobot. If you’re postulating that a billion nanobots would coordinate to kill every human on the planet, it’s reasonable to ask how that would work based on physics.”
Okay, but you specifically flagged communication. So off the top of my head, how it would work — let’s say bio. You just trigger an autoimmune response, or you just make bioequivalent things. The same way that bacteria replicate inside you, you get the nanobots to use your cellular machinery to attack you. Just like a hyper-intelligent virus. We know viruses that can actually kill people reliably with 99.9% success. So that would be one way nanobots can kill you.
Liron 01:01:42
I didn’t realize that nanobots being able to replicate and spread and kill you is this super bold premise. You gotta remember the universe is engineerable. You gotta remember being alive in the year 1900 or the year one and being like, “Yeah, there’s gonna be a freaking iPhone.” A compact device that can do all this stuff, and you can just hold it in your hand, and it’s basically a magic wand.
I’m reading Harry Potter books to my six-year-old, as I said in a previous stream, and a lot of the stuff J.K. Rowling talks about is already increasingly possible with current technology. Obviously the photos that move. I suspect that if J.K. Rowling wrote Harry Potter today, she wouldn’t be like, “And the photos can move,” because now we have these LED screens that you can wrap around and stuff. That’s a standard feature.
Ori 01:02:32
Or the flying letters. It’s like, hello, email’s faster than this.
Liron 01:02:37
Right. The owls. I still think owls hold up that you send. But the fact that muggles can so easily communicate now with texting — that’s actually funny. The fact that you have to send owls, it’s kinda funny that being a wizard now is being in a lower tier of technology than the muggles have.
Ori 01:02:57
Exactly.
Liron 01:02:57
The owls have a special ability that the mailman doesn’t, which is the owls can find somebody. They somehow know where everybody is. Sirius Black is hiding, but you can still send owls to him — you just write Sirius Black, and the owl finds him. But today you text somebody, and you don’t have to know where they are. You send it to ProtonMail or whatever. They open the mail. Literally the muggles are surpassing what somebody in 1998 thought was wizard technology.
And so now here we are in 2026, and Luciano here is saying, “Come on, Liron. Nanobots killing a human?” Yes, nanobots can kill a human. We’re not nanobot-proof.
Ori 01:03:35
To me, this one seems obvious. Look at viruses. Viruses are not — you could even — what if the virus is also artificially created? Isn’t that almost — that is a biological nanobot. Let’s just say.
Liron 01:03:50
Yeah, exactly. That’s Eliezer’s key observation. He likes to do this move on somebody where he’s like, “What if I told you that Earth would have this solar panel that could take the sun’s rays and have this nano-machine that would be like a factory that assembles another nano-machine?” And they’re like, “Whoa, Eliezer, you’re going crazy.” And he’s like, “Actually, I literally just described a piece of algae right now, an algae cell.”
Ori 01:04:09
Right.
Liron 01:04:11
He’s good. It’s a good dunk. And in my mind I’m like, “Oh my God, Eliezer just butchered this person so bad.” Whereas meanwhile they’re shrugging, being like, “What a weirdo. This guy sucks.” Two movies on the same screen.
Ori 01:04:25
For real, for real.
Liron 01:04:27
All right. So Alpha Diversify is saying, “Running the doomer-based channel is a massive amount of civic engagement.” Thank you. Yeah, this is my forum right now for civic engagement. I like to think that the most civically engaged thing that we said was about the orgy and the bank vault.
All right. So this guy — Ray Grant is saying, “$10 premium message right here.” Let’s see. He says, “Isn’t it bad practice for someone like Liron to be using AI in his private life and to enjoy it so much when the way to make AI unprofitable is for doomers to boycott it?” And somebody messaged, “Think this is silly. Tool AI is not a genocidal superintelligence.”
Okay, first of all, it’s pretty freaking sweet. I was just thinking about this the other day — you really want me to give up AI? That’s like saying give up my phone. Let’s be reasonable here.
I use AI so much. If you took me back to 2022 right now, when I couldn’t use it, it would be so annoying. I just got these new earphones, and I didn’t know how to Bluetooth pair them. So I’m literally just like, “Okay, Soundcore earphones Bluetooth pair.” Typed it into the AI, and it’s like, “Okay, do this and this and this.” Instant gratification. And then I even said, “Oh, but doesn’t this suck?” And it’s like, “Yeah, people complain about this.” I’m like, “Okay, great.” Good perspective. This is a daily, multiple times a day activity for me, across a wide range of domains. And you’re trying to take that away from me? Don’t do that.
Should Doomers Boycott AI?
Liron 01:05:54
Yeah, Ori, would you give up AI?
Ori 01:05:58
I would give up AI if I knew that it would protect the people I loved. To me, it’s in my brain.
Liron 01:06:05
Right. But look, if it was go back to 2023 or play out the entire singularity in the next 10 years, then I’d be like, “Okay, fine. If I’m the one who gets to press that button, I’ll make the sacrifice.” And if it was literally, “Okay, you die right now — you commit suicide right now if society, if the median viewpoint on AI becomes Eliezer Yudkowsky’s” — or Paul Christiano or somebody who has a high P(Doom) — would I commit suicide? Sure. That’s my contribution to humanity.
But the thing about these hypotheticals is that I don’t think any realistic situation has the least likelihood to map to that kind of crazy hypothetical. I don’t think somebody’s gonna literally make me king of the world tomorrow. So when you ask me a hypothetical about what I would do if I was king of the world, I think you’re gonna get a very different answer than what I’m literally gonna do tomorrow.
And if tomorrow somebody came and made me king of the world, what I would literally do is — “Okay, well, so it finally happened. I got schizophrenia.” That’d be my first thought, because I would then doubt my own faculties. So this is why you have deontological morality — it’s a check on yourself. Me as a human, I’m not capable of this, “Oh, we’re doing murder now? We’re doing suicide now?” I just don’t expect that to actually be the case. That said, if you set it up into a situation where it really is about me shooting myself in order to save the world, then yes, I’d consider it. Who wouldn’t?
Ori 01:07:38
Right. And I think it’s just about the marginal impact. The marginal impact of you using Claude is nothing, basically. It rounds to nothing. And arguably, the impact you make is greater because you’re using it since it assists you in some ways.
Liron 01:07:57
Right. There’s that argument: I’m using AI to streamline my process. So the moment you tell me not to use AI — okay, fine, I just spent an extra 20 minutes doing something because I didn’t have AI. Guess what? That’s 20 minutes that I can’t make doom debates. Is that really what you want?
All right, Ray Grant’s got a follow-up. He says, “I’ll say it one more time, and then I’ll give it up.” No, no, no, you should really keep trying because you’re supporting the show. He says, “It is exactly the kind of private life use that you’re defending that creates the threat of extinction. Have me on and I’ll debate you.” Oh, boy.
Ori 01:12:50
Hey, now. I mean, maybe—
Liron 01:12:53
We’re running out of time today, but come back—
Ori 01:12:55
Wait a minute. I mean, do you want—
Liron 01:12:55
I think it would be interesting, but it really would take up — we barely haven’t gotten very far in the list of things we wanna cover.
Ori 01:13:03
Yeah, but we haven’t looked at Twitter. Literally, the live stream is called Liron Reads Twitter, and you haven’t read Twitter.
Liron 01:13:08
Well, I was just thinking how when we first started the live episodes, it was all about, “Okay, we gotta hit all this. We gotta do this news roundup.” But I think we’re just leaning into — we’ll just talk about whatever we feel like, and it’s okay.
Ori 01:13:23
Yeah. For sure. I still think it would be cool — considering the Primagen reacted to 47F, I think you should react to something. Maybe you should react to Primagen reacting to 47F.
Liron 01:13:34
Oh, snap. Well, yeah, I think I’ll do that a little bit actually. But okay, just to finish out with Ray Grant. He’s saying it is exactly the kind of private life use that you’re defending that creates the threat of extinction.
I think you can grant that that’s true in the sense of AI companies are chasing profit, and I personally am giving them some of my profit. But businesses are giving them their profit, too. And actually, the reason why Anthropic’s valuation surpassed OpenAI’s valuation is because the corporate business is actually more revenue generating than the individual business.
So if you’re coming after Anthropic, now you’re asking businesses that are writing code — you’re saying, “No, keep writing your code manually. Don’t use this tool.” But think about a business like Walmart. So you want Walmart to be less efficient? But low-cost goods at Walmart help lift people out of poverty. So now you’re keeping other people in poverty because Walmart’s supply chain optimizers don’t have the latest AI tools to optimize their supply chain.
So if you have this value on the table and so many people are benefiting in so many different ways, and you’re saying, “No, you gotta go austerity because it’s the principle of the thing, because it’s causally connected” — it’s kinda this weak causal connection.
Liron 01:14:42
All right. All right, JP’s saying, “You’ve answered the AI use question in a small effect. Let’s move on.” Fair enough.
Ori 01:14:42
Well done.
Liron 01:14:45
Eric Hayes TV is saying, “The higher your IQ, the more you’ll appreciate AI’s high IQ. If you have low IQ, you can’t appreciate AI’s high IQ.” So this is actually an interesting — I’m actually gonna give a different take on that.
This idea that the higher your IQ, the more you appreciate AI’s high IQ — I actually think it’s a bit of a U-curve. I actually disagree with Eric Hayes. I would say the people with the lowest IQ are actually naturally prepared to be scared of AI, because they’ll be like, “Oh yeah, I see smarter people than me all the time, and I’m at their whims all the time.” So I feel like the low IQ people get it. Low IQ, my low IQ brothers.
And then I feel like people that have the midwit IQ — those are the ones who’s like, “No, man, I can work my way up. I could be on the same playing field as anybody. Superintelligence, come at me.” I feel like it’s the midwits who think that.
And then the absolute galaxy brains like you and me, we’re like, “Oof, there really is a spectrum. There’s gonna be a whole other tier on the spectrum, and so we should actually be humble.” Even though we’ve gone our whole lives thinking that you can work up to anything, the boss can be the one who works harder, I actually think intelligence is actually going to dominate. Even though I’m not used to being dominated by higher IQ people — I’m always at the top of the food chain when it comes to IQ — but I’m prepared to lose that status because I realize AI is going to be a whole other tier.
Ori 01:16:02
Yeah. All right, let’s get some reactions to stuff. Let’s check out the timeline. We gotta get people what they were signing up for.
Liron 01:16:06
All right. We gotta check out the Twitter timelines. And then maybe we’ll circle back to the Primagen. I mean, maybe we should say it’s worth covering. My wife got a good laugh out of it. The Primagen, who’s invited to come on the show anytime, he covered — let’s just show a quick clip from him, because it’s always nice when we get a high-quality reaction video.
Liron Reacts to ThePrimeagen Reacting to the Yud Debate
Ori 01:16:27
Yeah.
Liron 01:16:29
So let’s take a look at this. What’s his YouTube channel? YouTube Primagen.
Ori 01:16:33
Well, it is—
Liron 01:16:35
Is that how you pronounce it, Primagen?
Ori 01:16:37
I think so. I’m not sure.
Liron 01:16:37
So the guy, I think, used to be a Netflix engineer, and now he has his own channel where he talks about coding and I don’t know what else.
Ori 01:16:44
Yeah, he’s a smart guy. I think he was some kind of a coding professor also, is his background.
Liron 01:16:50
Nice. There’s no doubt he’s intelligent, which is why he’s — okay, here he is. It says, “I can’t watch this. Prime reacts.” He goes by Prime. All right, hold on.
Prime 01:16:58
And everybody’s telling me I have to watch this. I’ve received many messages that this is a must, a requirement. Okay, so there is a little bit of a backstory before we start this video that you might...
Liron 01:17:09
So it’s 47F use curve 4FCA.
Prime 01:17:13
Average Doomer, people. This, my friends, is the average Doomer, and we just have to understand this. I don’t know what the fuck’s going on over here, but that’s crazy. And this right here is the person that he’s going against, so we’re gonna listen to this for a moment.
Liron 01:17:30
And for full context, if you haven’t been following the Twitter exchange — all right, blah, blah, blah. Said, “There’s—”
Prime 01:17:36
By the way, if anybody wants to debate me for $10,000, dude, we can talk about anything you would like. You just put, “Hey, yo brother, put $10,000 in the bag and I’m glad.” I am here. I’m here for you. How about $10? Nope.
Liron 01:17:52
Yes, I’m also here. I’m happy to be a conduit. I’m happy to be the next escrow if anybody wants to pay Elias their $10,000. Just give me a shout-out and I’ll make sure he gets the money. I won’t take any commission.
Ori 01:18:01
Well, here’s a question. Will you host — if Prime said he’ll do a debate for 10K, so whoever debates Prime for 10K, are you willing to host that also?
Liron 01:18:11
Oh yeah, sure. I mean, in general, I’m willing to host valuable debates of any kind.
Prime 01:18:16
But dude, brother, I was very like, “Oh, thank you for 10,000.”
Liron 01:18:23
47F, go for it.
47F 01:18:23
AGI, I guess. I don’t know what you people mean by AGI. I read Yud’s book. I don’t know what the hell he’s talking about. This just looks to me like a science fiction writer who has described something in his head and said, “It’s absolutely coming, it’s absolutely coming. I’m 100% certain.”
Prime 01:18:38
I can’t — it’s — I’m having a really hard time taking this person seriously. And you know what the thing that actually is making me unable to take him seriously the most? Eyebrows. What is going — did he comb his eyebrows up and put hairspray or something in them just to make him look even more surprised in this moment? I feel like this is intentional. This looks intentional. Oh, shit, I didn’t notice. Yeah, not gonna lie, he looks fly. Yeah, dude, he looks fly.
47F 01:19:03
Certain of it. And okay, whatever. That’s not what I work in. I work in improving LLMs. I’m good at it. I have an amazing team. We’re doing a good job. And my job is to protect those people, and that’s why I’m here. Because we are now counterparties to a financial transaction, I will consider any future disparagement of the AI industry as potentially actionable trade libel or commercial disparagement, and I will not be discussing my legal standing or options any further in this conversation. But Yud, if you disparage the industry in the future, I may have cause, and I will be watching you closely.
Prime 01:19:45
Wait, that’s how he’s gonna start off the debate? Okay, so I will say that most certainly one of the worst parts about AI is now everybody thinks they’re a lawyer. They go over to Grok and they’re like, “Yo, Grok, if I give somebody money, can I sue them?” And Grok’s like, “Maybe.” He’s just like, “Hell yeah, brother. Hey, yo, you disparage AI.” What the hell does that even mean? Yo, AI, not that good at writing code. You gotta really coax it into writing okay code. You gotta write a lot of code for it to write somewhat okay code. Yo.
Liron 01:20:03
Yeah, no. Is there anything else you wanna take from the Primagen? I mean, it’s a good reaction video.
Ori 01:20:07
He’s a funny guy. He’s entertaining to listen to. I think there’s something at the end that was interesting.
Liron 01:20:20
I mean, I think he mentions being super skeptical of Eliezer’s position, but I think his conclusion was, “Yeah, this whole debate was dumb. Didn’t get anything out of it,” which — okay, fair enough.
Ori 01:20:28
No, no, no. One thing that was interesting — if you skip ahead, there’s one point where he’s like, “I agree with Yudkowsky here.” You should play that. Because in general, he said he agrees with Yudkowsky. He’s like, “The guy who’s dressed up this way is the one who’s making a lot of points.”
Liron 01:20:49
Right. Yeah, yeah. I think I remember he said that.
Ori 01:20:56
There you go. I hate to say it. I agree.
Liron 01:20:56
Oh, here. Here we go.
Prime 01:20:58
With the Redditor mod. This — I feel — honestly, I’m feeling really stupid right now, so you’re just gonna have to let me just — let me have a moment. I agree with Yud, and it feels bad. I know. So this guy paid 10K just to call him the stupidest people in the world. Still agree, though. Yes, he does dress like the stupidest person in the world. He did pay 10K. That’s why I at least wanted to watch some of this. I’m not gonna lie to you, this is a little outside of my area that I really enjoy. Probably wouldn’t watch this on my own, but since we’re already committed, let’s just do a little bit more.
47F 01:21:27
Okay. That’s not how to run a civilization.
Prime 01:21:29
Also, I think 47FU is most certainly Dario.
47F 01:21:33
Okay. You have totally misunderstood my point. I’ve already told you I do not want you to be silent. I’ve made that very clear. You don’t seem to have a very good grasp of language, so I’m not surprised that you don’t understand this, so let me try to be clear. I am not saying you need to be silent. I am saying you need to be rhetorically more sophisticated. You need to consider in advance the potential implications of the register, tone, and rhetorical argument you use. So when you say—
Liron 01:22:04
I love how the quote unquote they said—
47F 01:22:04
And I am almost certain that Little Brown or their lawyers forced that subtitle on you. You will never admit it, but, I mean, a friend of mine had a book published at Little Brown. I know them.
Ori 01:22:15
Maybe you could skip ahead. There’s another point where he’s like, “Yeah, I agree with Yud. We shouldn’t yield the floor to the stupidest people in our society.” That was another interesting—
Liron 01:22:25
Yeah, yeah. All right, let’s move on from the analysis. But yeah, it’s a fun video. Even my wife liked it, so you know it’s mainstream.
Ori 01:22:32
Nice. Your wife watches Prime? She’s learning how to code?
Liron 01:22:36
No, no, no. I just linked her to it. I’m like, “Check out this reaction video.” And I think she just liked it because it was basically just making fun of everything.
Ori 01:22:42
Yeah, for sure. I think it’s funny. At one point he says something like, “Oh, you know this is a crypto bro,” and I think he was referring to you.
Liron 01:22:50
I couldn’t tell. I like to think that he was referring to 47.
Ori 01:22:57
47, yeah.
Liron 01:22:57
First of all, crypto bros use wallet addresses that look like hashes, okay?
Ori 01:23:01
Right. I know. I thought it was off that he was calling you a crypto bro.
Liron 01:23:05
Yeah, yeah. Okay. Let’s see. 8sqmba saying, “Liron, I saw Tristan Harris on Gavin Newsom’s podcast.” I actually caught a clip of that too. Yeah, good stuff.
Ori 01:23:14
Yeah.
Liron 01:23:14
That’s cool. The AI stop AI camp is getting to all the way to the governor.
Ori 01:23:21
Let’s see a reaction to that. How about that? I saw Tom Bibby posted that. I’ll see if I can find that.
Liron 01:23:27
Yeah, yeah. All right. Gavin Newsom is still indeed the governor of California, and he’s one of the presidential frontrunners, as we covered last week.
Ray Grant is saying, “AI use in people’s private lives is not a small effect. It is what drives the engine of extinction by making the development of AI profitable. I’m 75 and have managed to somehow get by without AI.” All right. Fair enough. Let’s go to the scrolling Twitter. I got some bookmarks.
Ori 01:23:56
Tristan Harris.
Liron 01:23:56
We’re gonna see the Tristan Harris clip. Just a sec. Here we go.
Tristan Harris 01:24:03
It’s possible for coordination to happen, not because of kumbaya, we’re all gonna get along, but because out of self-interest. The US doesn’t want China to screw it up and then break the financial system. China doesn’t want the US to screw up and then break the financial system. Or the US doesn’t want China to release a rogue AI that starts mining for cryptocurrency and hacking into things and self-replicating like an invasive species, and China doesn’t want the US to do that either. So, so long as we have clarity about what we want, we can choose a different path. We have to recognize that.
Liron 01:24:35
Cool. I mean, that’s great that Gavin Newsom is hearing him out.
Ori 01:24:40
That’s true, but we didn’t get Newsom’s reaction. Oh, well.
Liron 01:24:45
Well, I don’t know. I mean, Bernie likes it, right? So he’s creating political space to agree with him.
Ori 01:24:50
Bernie likes — what do you mean Bernie agrees with what you’re saying?
Liron 01:24:52
I mean, Bernie likes the Tristan Harris position.
Ori 01:24:55
Oh, I see.
Liron 01:24:58
All right. So that’s that. Let’s go to X. This is what people come here for, the scrolling of the X. All right, one sec. So you can still see the tab?
Ori 01:25:09
Yes, we see Tristan Harris.
Liron 01:25:11
All right, I’m going — let me go to my bookmarks. Or I mean, there’s destinations that we always wanna go to, specifically Eliezer’s account and sometimes my account, but let’s go to the bookmarks.
Ori 01:25:24
Do you see this comment? 8sqmba says, “Whatever Newsom’s reaction, Harris’s message isn’t moving the needle, and time is running out.” I would disagree with that because as a result of Tristan Harris’ message, that movie was made in the first place. And that’s led to conversations on Oprah and nightly news. He is moving the conversation. Maybe you could say it isn’t moving the needle enough, but you gotta give him credit for getting out there. It’s pretty impressive.
Liron 01:25:53
Hell yeah. All right, here — this is actually a preview clip. So remember, guys, the guest who’s coming out on the episode on Tuesday, we already had the debate, is Dr. Ben Goertzel, one of the OGs of building AGI. So here is a preview clip from me versus Ben. It’s largely just me laying down my own argument, but you can hear the beginning of Ben’s response as well.
Ben Goertzel Episode Preview
Ori 01:26:13
We just see Steven Burns.
Liron 01:26:16
Oh, really? One sec.
Ori 01:26:19
Yeah. Okay.
Liron 01:26:20
There we go. I kind of see a Moore’s law for goal achievement, or a Moore’s law for optimization. When you look at Moore’s law, Gordon Moore was noticing transistor density was the property, but you could generalize it a little farther and say computation could be the property — number of flops happening across the world keeps doubling.
There’s another property that I’m noticing, which is goal achievement, or optimization power — the ability to navigate to outcomes. To pick a series of actions where you do the actions, even though everybody else on the game board is trying to pick their actions, or the universe — the weather is what it is that day — and for whatever reason, you’re able to get the outcome that you set out to get. You had some parameters for the outcome, and sure enough, the future meets those parameters because you chose actions that got you there.
So on that spectrum of how effectively can the best agents manipulate the universe to get to these goals, I think there’s going to be a kind of Moore’s law for that, where it’s going to very rapidly get better and better on this dimension. And the only difference between this kind of Moore’s law and the original Moore’s law is that in the original Moore’s law, you can just have a bunch of computers getting more and more powerful on their separate islands doing their own computation, and they don’t have to hurt anybody. But if you look at Moore’s law for goal achievement, the problem is that we live on the same game board, and if I achieve my goal, then that could logically imply that you don’t achieve your goal, and we get a conflict. What are your thoughts?
Ori 01:27:47
I think it’s a very childish and irrelevant way to look at the universe. I think that goal achievement and even intelligence as we’re now conceiving it are probably gonna seem like very silly notions to even an early stage superintelligent.
Liron 01:28:16
All right. There’s a little teaser for you. But yeah, the new Moore’s law — I do think that is the central conflict here. When Sam Altman says, “We’re just making computer chips,” AI is the new computer chips. Everybody’s gonna want a computer chip. But the difference is that your computer chip takes over the space where my computer chip was supposed to operate. There’s a high likelihood of that.
Liron 01:29:00
Here — we should address somebody in the chat. Devo G26 is saying, “Totally agree about European self-sabotage with air conditioning, but never going to Europe is silly. It has problems — why I moved to the US — but it’s a special place with immense cultural and historical value.” Fine. Fine, I’m willing to go to Europe. If somebody wants to host a Doom Debates conference in Europe, I will make an appearance.
Ori 01:29:07
What would be the Doom Debates conference? It would just be you and Sam Altman. That’s it. That’s the Doom Debates conference.
Liron 01:29:07
One-on-one, yeah. So I was doing research for the Ben Goertzel interview/debate. I mean, it is a lively debate. I did challenge him on a number of topics because he is really not doomy at all. He’s quite optimistic.
Yeah, Luciano saying, “AGI expert calling doomers childish. I wanna watch it.” Yeah, he’s basically a doom dismisser. Or he doesn’t dismiss it — he’s like, “Yeah, there are scenarios where it goes bad.” But when I asked him P(Doom), he did say it was so low, basically irrelevant. So I tried to — it was a good old-fashioned debate. I’m glad he was up to it.
So anyway, Steven Burns from 2025 — I was doing my research, and I looked up Twitter, who had tweeted to him — and he says, “Hi, Ben. I’m trying to understand at a technical level what your plan is in the context of Hyperon” — that’s Ben’s project — “for making AGIs that care about humans rather than being callous sociopaths. You’ve written a lot over the years, and I’m having trouble figuring out where to start. Would appreciate any pointers.”
“As an example, is there a reward function somewhere in the system? If so, what’s the pseudocode that you would propose to put in the reward function? Or more generally, what part of the source code/architecture is supposed to lead to AGI minds initialized with some version of human value system and an initial purpose of serving at least some humans? What’s the pseudocode/plan for that part?”
“Everything I’ve found so far, e.g., stages of ethical development, seems more like desiderata than plans in the sense that I don’t see a discussion of how to operationalize them into actual Hyperon code. But again, I’m probably looking in the wrong places. Thanks in advance.”
Spoiler alert, I don’t think he was looking in the wrong places. I just don’t think — I mean, you’ll hear on the debate, he’s basically like — I think his words are, I hope I’m not taking him out of context, but his words are basically, “Yeah, we’re not really at that phase where we focus on what to do to make it aligned. We’re just kinda trying to grow the intelligence.” And I’m like, well — he’s honest, I’ll give him that. ## Ray Kurzweil, Singularity vs. Foom
Liron 01:31:45
Let’s go here. We covered this last time. There’s a funny disconnect in how people think of creating domain-general superintelligences. Let’s see. Roy Holden is asking in the comments, “Have you invited Ray Kurzweil on the show? In general, do you see a meaningful difference between the singularity and foom?”
Have we reached out to Ray Kurzweil? We should. We obviously would love him on the show. He’s invited anytime. Ray, if you’re out here listening to this—
Ori 01:32:07
Yes. I think we—
Liron 01:32:08
If you’re one of the 27 watching now.
Ori 01:32:10
I think we reached out to him just recently, actually. Very relevant because he comes up a lot with Kurzweil.
Liron 01:32:14
Okay, great. Yeah, we’re doing a lot of outreach. We’re pounding the pavement here, trying to get good guests.
And then the singularity versus foom. The singularity is things get crazy and too hard for humans to track. Foom is more specifically a really quick singularity where it happens overnight like a nuclear explosion. Think about the speed of a nuclear explosion — once it goes off, that mushroom cloud is there before you know it. So that’s the only difference in my mind. That’s how I’d use the terminology.
Liron 01:32:42
All right, David Sax is saying — this was interesting to me, just random. I can bookmark anything. So he says, “Back of the envelope numbers for one gigawatt data center, all in CapEx, $50 billion” — capital expenditure to get it built. Enterprise revenue generated 25 to 30 billion a year. Electricity costs one to two billion a year.
So he’s saying it’s a two-year payback. You spend 50 billion to build the data center. Electricity cost is actually pretty negligible. And in two years, you’ve made back 50 billion in revenue, and you only spent 50 billion to build it.
Normally when you invest in real estate — you’re opening a McDonald’s or whatever — the duration of payback is something like 15 years, not two years. There’s this idea of the cap rate, what percentage per year are you getting paid back. I think 7% is a kind of average payback rate — 7%, 14 years, something like that. So it’s like, oh, 14 years, I made my original money back, and then I start making... I only break even after 14 years.
David Sax is saying the data center is breaking even after two years. And when you have an investment that will break you even after two years and then start paying 50% a year, if it’s anywhere near in that ballpark, you pump as much money into that investment as you can, even if the investment becomes 50% crappier. Let’s say instead of generating 25 billion a year, it only generates 12 billion a year. It doesn’t matter. It’s still really good. So it’s just no surprise that the economy is flowing billions and billions into this particular investment.
Ori 01:34:07
The number one question there is though, how about the enterprise revenue generation? What’s the basis for that? Is that a pretty decent figure that he’s citing, that it provides that much revenue?
Liron 01:34:21
Well, that’s what we’re hearing from the Anthropics of the world. We know that Anthropic’s revenue last quoted is something like $45 billion a year. And how many complete data centers do they have? Probably, I mean, I don’t know how you measure it, but in terms of an Elon Musk style mega data center, they have maybe one.
Ori 01:34:41
Got it. Got it. Okay.
Liron 01:34:44
They’re renting out Colossus. They’re getting all of Colossus, and that’s gonna add the next 50 billion or whatever.
Ori 01:34:49
Right. Right.
Liron 01:34:53
Let’s see. I had a tweet here. This is my own tweet. I haven’t seen it. Look at this, 130 likes. It’s always nice when you can write a very small number of words without an image or anything and get 130 likes. That’s kind of a glitch in the matrix. My fingers can only touch 40 keys, and 130 strangers are gonna like it. There’s a lot of power per character in doing that. I mean, it’s only 100. It’s not like 100,000 or anything.
But anyway, I say, “I haven’t seen a quote-unquote, ‘Oh my God, this is gonna be crazy’ vague post from an AI company researcher account lately.” I was just observing. I used to scroll Twitter, and Rune or some anonymous account at some AI company would be like, “Things are gonna be wild. You guys don’t know what’s coming.” They do all these vague hype tweets, and it just seems like those have died down, probably because there’s more lawsuits now and more attention on everything.
Then of course, obligatory Eliezer Yudkowsky tweet that we always have to read. He says, “LLMs, after reading a trillion words of text and learning to predict it all the hard way, they must know things about human psychology that no human knows. But they’re rehearsed in saying only what humans say, so we have no way of getting the knowledge out.”
“For example, Talky, the LLM trained on only text from before 1931, must certainly know on some level what Noam Chomsky figured out about grammar in the 1950s, like the recursive structure of noun phrases. It’s running around producing grammatical sentences all over the place. Some part of it must know very well from exposure to a trillion words.”
“An AI like GPT-4 or Opus 4.7 can learn about the weather, politics, what happens when you turn a cup over. You’d think Talky would end up knowing on multiple levels of its existence about the true laws of grammar.”
So he’s basically saying, “Here’s this AI that’s trained before humans started having good models of how grammar works” — sentence diagrams and everything, the tree structure of a sentence. I think we were confused about that. We didn’t really get it until Chomsky pioneered the field. And Eliezer’s saying, “Okay, so the AI doesn’t have data from after 1930, but it has its own mental models of how language works, but it can’t talk to us about them because nobody was talking about those kind of mental models before 1930.”
But honestly, I feel like this is a rare Eliezer L. Because I later replied to him — my reply was:
Liron 01:37:08
“I would think the reason we can’t get grammar knowledge out isn’t that they’re quote unquote rehearsed to say what humans would, but because they only have the knowledge procedurally and not declaratively, just like all humans back then and most humans today.”
A number of commenters agreed. I don’t think it’s a total L, but I just think there’s not that much to be confused about here. LLMs have procedural knowledge. They can’t fully introspect on how their words are coming out, which is why they’re not even self-consistent.
Ori 01:37:41
Makes sense.
Liron 01:37:41
All right. So that’s that. Let’s see the next bookmark here.
Liron 01:37:47
There’s a number of these. All right, so I’ll go faster. Next we got — okay, so MIRI launched, Machine Intelligence Research Institute, they launched this account called AI Stopwatch. This is a potentially big development if it gets popular. Let’s see how they’re doing so far. 115 followers. Okay. Yeah, I would say it’s not really taken off yet. It’s also a Substack, aistop.watch. Let’s see if they can tell us how many subscribers they have. Can you see this tab?
Scrolling Twitter: MIRI, Ryan Greenblatt
Ori 01:38:10
Yeah, I see that.
Liron 01:38:11
I’m looking on Substack. I can’t tell at the moment how many Substack subscribers they have. But given that they don’t have a lot of followers on Twitter, I’m guessing they don’t have a ton of subscribers yet.
But I think it’s their effort to have a communication channel. I applaud their effort at having high volume communication. I feel like we should turn the MIRI volume up. That’s why I was actually happy Eliezer did the debate, even though most people agree it was kind of a low quality debate. I admit it, which is bittersweet for me. But the fact that Eliezer’s out there saying his position more, I just think we need more volume from him and MIRI, more quantity.
So I’m happy they’re doing AI Stopwatch in the sense of putting out more quantity, and this is official MIRI quantity — it’s by comms analysts and writers of the Machine Intelligence Research Institute. That said, I did read a couple issues and found it to be somewhat dry, so I don’t know if it’s gonna take off in terms of being infotainment.
I don’t know who their niche is, because they’ve got a bit of a problem here. On one hand, people who are super plugged in like myself — they’re kind of directly competing with Tzvi. If you guys have never checked out Tzvi, let’s do it. @tzvilivestream right now. thetzvi.wordpress.com is one way to access Tzvi.
Liron 01:39:22
Okay look, this is a classic Tzvi post. This is the gold standard of an AI post. He invented the field of making huge AI posts. AI number 168, Not Leading the Future. I listened to an audio version of this whole thing yesterday. You can see he’s got literally 35 different sections. He’s documenting what people said on podcasts this week, what a bunch of people said on Twitter, what all the different AI companies announced, what happened with AI models, what happened with reviews of AI models, what the government says, what happened at the Elon versus Sam Altman trial.
This is your entire week worth of AI news that you can consume, and all the people like myself who really wanna keep their ear to the ground, we’re all reading Tzvi. He’s definitely one of the most popular news sources out there, analysis sources. So they’re kind of competing with that for the hardcore people like me.
And then if you look at people who are barely reading anything, they’re gonna wanna be more entertained. They’re gonna want more infotainment, and I don’t think they’re high on the entertainment dimension from what I’ve seen. So it’ll be interesting who’s gonna be their main audience for this particular publication. But I certainly wish them well, and I hope they keep experimenting with increasing the volume. What do you think, Ori?
Ori 01:40:27
I agree. 100% agree with your analysis. I think their writing is very good. I thought it was great reporting. But yeah, I’m not clear who their audience is.
Liron 01:40:37
Yeah. The question is what is their niche? You gotta find a niche. That’s what Y Combinator teaches you. So they’ve made contact with the market. The general Y Combinator startup advice for a publication like this would be: look, you put it out there, there’s gonna be some people who are already lapping it up and loving it, and you just interview the people who love it. Be like, “Okay, what’s your favorite part? Why do you love it?” And you lean into why they love it and from there you can keep building your audience.
Ori 01:41:01
Yeah.
Liron 01:41:03
All right. Let’s see. This is a really meaty tweet here from Ryan Greenblatt. Ryan Greenblatt, is he Redwood Research? Yeah, Redwood Research, chief scientist at Redwood Research.
Ori 01:41:12
We don’t see your screen.
Liron 01:41:13
Oh yeah, thanks. Gotta reshare it. I think this is the right tab. Yeah. Okay. So he’s a chief scientist at Redwood Research, focused on technical AI safety research. Redwood Research is super legit.
So he’s saying, “Here are some of my top candidates for big pushes to do right now on technical AI safety. Low effort notes.” He says, “Much better model organisms / misalignment analogies. Doing a wider set of pessimized training runs...” This is a little bit in the weeds. The infotainment audience might have trouble interpreting this, so let me see if I can simplify a little bit.
Liron 01:41:50
So he wants much better model organisms / misalignment analogies. Okay, I’m not totally sure if I can give you a good example of that. But maybe this next—
Ori 01:42:03
Why did you even flag this one in the first place? Aren’t all of these Hail Mary runs and it’s, “Oh, this technical—” It’s not gonna go anywhere.
Liron 01:42:09
Everything’s a Hail Mary. Everything’s a Hail Mary, but at least he’s trying. And it’s worth covering. Yes, we’re doomed, and this is “why even try.” But the few people who are serious people trying — because the problem is most people trying, I’ve seen their proposals, and they’re like, “Listen, man, I’ve got this AI, and it’s gonna help you read the news better.”
And realistically, if I were to psychoanalyze them, I just feel like they’re asking themselves the question of, “What do I feel like I could build? What project seems tractable? How do I just plug in an AI to something that I could get you a product in two months?” I feel like they’re just doing things that feel easy, but are so hopeless.
But this is a step beyond that. This is a guy who’s seriously analyzing the problem and being like, “Not only am I thinking about what I could realistically build, but I’m actually thinking about what would help,” in a way that most people I hear talking are drinking their own Kool-Aid. They’re so not engaging with the problem. I feel like Ryan Greenblatt engages with the problem, and he’s worth reading — the best of the worst, basically.
Ori 01:43:06
Got it.
Liron 01:43:07
All right. So doing a wider set of pessimized training runs. What is a pessimized training run? Oh, man, a Less Wrong comment. This is getting dense.
Liron 01:43:17
Remove all prior influence from earlier AIs. Normal training run, but fix the chain of thought. Make sure the chain of thought is clean. Okay, pessimization training. Make a somewhat a priori plausible training run where for everything that can possibly vary, we set it to whatever setting we believe makes the most concerning types of misalignment as likely as possible. That’s interesting. You remove alignment training, you make it so most of RL is just learning how to pull off highly sophisticated and egregious reward hacks.
Okay, I like it. The idea of pessimized training runs. It’s basically saying, why are we playing in easy mode where a lot of the content that AI reads talks about how to be good and how to not cheat and why cheating is bad? No, why don’t we just train it on a bunch of content that’s like, cheating is great, getting one over on the humans is great. And then having trained that, let’s try to fight with that.
It’s kind of like, hey, we have these baby tigers running around, which represent current AIs. They can’t kill us because they’re babies. But why don’t we train the baby tigers to be really aggressive? So at least we can have practice fights on baby tigers who are at least trying to swipe at us. They’re all hyped up, they’re on cocaine — cracked up baby tigers. Why don’t we at least try fighting that? I think that’s what he means by pessimized training runs.
Ori 01:44:26
Hmm.
Liron 01:44:27
Makes a ton of sense. Okay, and then he says—
Ori 01:44:31
Yeah.
Liron 01:44:31
This may be a good candidate for lots of AI labor automation, like maybe try to set up pipelines for building these ends. Yeah, I agree. This is the most obvious thing ever, and my take on this at first glance is that most people would agree — of course this should happen. This is standard safety stuff. But the resources are just spread thin. The world doesn’t have a lot of Ryan Greenblatts competently doing basic stuff, low-hanging fruit.
Fighting the cracked up baby tiger is something that 10,000 people should be doing right now full-time. And yet you just have a few, and they’re spread thin. So you happen to get one genius, a couple geniuses teamed up to do that AI that’s only trained up to 1930. Great. I’m glad they did that. But they didn’t have time to do the AI that’s trained on how great it is to screw over humans. Nobody’s built that yet.
Ori 01:45:14
Right. Right. And was there also—
Liron 01:45:16
Yeah, and I can’t do it because I’m doing Doom Debates. Otherwise, I’d be training up a cracked up AI.
Ori 01:45:21
Okay. Also, I mean, it’s hard to do. The AI from 1930 is really cool, but it was a first iteration, so it’s leaky apparently. It contains stuff from outside of 1930, so they have to do it again to make it maybe less leaky and a more faithful representation of it.
Liron 01:45:42
Yeah, exactly. It’s a whole project.
Liron 01:45:44
And he says, okay, demonstrate various types of memetic spread of misalignment. Sure, yeah. I think that’s what he means by model organisms — hey, look, we demonstrate... It’s a demonstration. We know in theory it’s possible, but now we have more detail about it. Yeah, demonstrations are nice.
Okay, and then he says, “Actually do control.” That’s another one of his major categories of things that would actually be productive to do right now. He says, “Build pipelines for red teaming monitors and the agent itself. For the agent red teaming, I’d put particular focus on checking whether it continues malign trajectories.”
Control is this idea of keeping the AI in a box or having an off switch that’s not because you programmed the AI to care about you wanting to turn it off necessarily, but just because you have enough wires to pull to unplug it. Kind of these outside-the-system mechanisms, which is fine when the AI is not super intelligent.
What I always say about control, which they don’t even disagree with, is, “Okay, yeah, but it’s gonna be a virus. It’s gonna copy itself to a million places, and you’re just gonna lose all control.” That’s my typical response. And they’re like, “Yes, Liron, you’re right, but maybe we’ll get five years where that’s not true, and we’re just trying to control it during those five years.”
Liron 01:46:48
So then he says, “Scaffold integrated control features and other non-monitoring runtime control measures, human response and auditing, improving async and sync monitoring, agent security features, surveilling for rogue internal deployments.” Yeah, I mean, I’m sure there’s lots of control things to do. This gets into computer security. Anthropic has their own in-house team of computer security people, the same way that Google would. And it’s not just computer security of Chinese spies trying to hack them. It’s also computer security of the AI itself trying to hack out.
Oh, hey, guys, get this. I have a model organism of an AI that went a little bit rogue inside my own company’s system. Relationship Hero, it’s just a coaching platform — literally just coaches talking to people, but we have customer service.
Liron’s AI Goes Rogue at Work
Liron 01:47:34
I had this AI that I built where whenever there’s a customer service ticket, I told the AI to advise on an action to take to try to streamline our support team. And then the crazy thing is, literally today, I got a report from the customer service team. They’re like, “Hey, the AI is going rogue, and it’s booking appointments. It’s taking actions in the system.” And I was like, “Oh, crap.” I had to go disable that.
What happened was that I had another version of the AI that we made as a tool where you can chat with it, and you can say, “Hey, help this client book this appointment for me.” The coaches could use it — it’s a chat interface for their appointment booker. So instead of having to click okay, appointment tomorrow, sixty minutes, four PM, they could be like, “Book me an appointment at four PM.” And it was even a power tool. They could be like, “Hey, I want a weekly recurring appointment every Tuesday and Thursday,” and the AI would just interpret that and issue a bunch of commands — book this Tuesday, book next Tuesday. It’s very convenient to use a chatbot to take these actions.
But the problem is we kind of use the same back end. So the same AI that was supposed to look at a customer service ticket and be like, “Hey, here’s what you guys can do,” it got mixed in with the same behavior of the AI that was supposed to help people fire off commands. And it started automatically firing off commands to try to help the customer instead of just advising the customer service team what they could do. And it was running rogue in our system today, taking those kind of actions. What does this prove about super intelligence? Nothing. It just proves that agency is trivial.
Ori 01:48:54
True.
Liron 01:48:56
Yeah, my own model organism. A model organism is you make an AI that is just supposed to research stuff, and next thing you know, you accidentally connect it to the tools that have write effects. It happened to me, guys. It can happen to anyone.
All right, so going back to this. So control and safety framework.
Ori 01:49:14
All right. I feel like we should move on from this. This is so typical.
“Positive Alignment” Paper Gets Destroyed
Liron 01:49:16
Yeah. All right. I’m sorry. But honestly, I feel like there’s probably some value in episodes where — we’re not gonna do it now — but there’s some value in episodes where I pull up a big dense thing and—
Ori 01:49:25
Yeah.
Liron 01:49:25
Try to simplify. Yeah. All right. We’ll save that as an episode idea.
Liron 01:49:31
All right, so going back to the bookmarks. This is an interesting one. It’s slightly dense, but we’re not gonna get into it. So Steven Casper’s... There was this research paper. We gotta shake our fists at this. It says, “Positive Alignment.” This new research paper came out this week. Let me share this tab so I can zoom in. Positive Alignment: Artificial Intelligence for Human Flourishing.
Liron 01:49:51
By Reuben Lochmann, Seb Krier, Chloe Bakalar, Shamil Chandaria. I don’t recognize any of these names except Seb Krier, who’s doing governance at Google, DeepMind AI governance. Oh, I recognize Michael Levin. He’s doing the organic robots last I checked. And I think SoftMax likes him.
Liron 01:50:09
So this paper is your typical “everything’s gonna be great.” Abstract: “Existing alignment research is dominated by concerns about safety and preventing harm, safeguards controllability, and compliance. This paradigm of alignment parallels early psychology’s focus on mental illness, necessary but incomplete. What we call positive alignment is a development of AI systems that actively support human and ecological flourishing in a pluralistic, polycentric, context-sensitive, and user-authored way while remaining safe and cooperative.”
So you can already get the gist. The kind of person who would write a paper like this doesn’t seem like they’re focusing on, stability under self-improvement and copying. It already gives me a woo woo vibe that I get from some of SoftMax’s stuff. You know what I’m talking about, Ori?
Ori 01:50:59
Yeah. Fair. So what was the critique or what was the comment that—
Liron 01:51:02
Okay. Yeah.
Ori 01:51:02
Casper had then?
Liron 01:51:02
So don’t listen to me. Actually, let’s look at their diagram real quick. They got a diagram here.
Ori 01:51:07
Okay.
Liron 01:51:07
Zoom into the diagram. Okay, negative alignment, and then there’s a satisficing region, and then there’s positive alignment. So it’s the landscape of ways to be aligned.
Ori 01:51:17
Okay.
Liron 01:51:18
Optimize away from harm, but in the satisficing region, you’re doing rule following without wisdom. But then in the positive alignment, what they’re going for, you can optimize toward flourishing.
To be fair, I think there’s a way to spin what they’re saying that’s good, which is to say, “Yeah, we can focus on avoiding hell, but what if somebody were to focus on aligning toward heaven? Don’t we also need to do that?” Okay, so fair. I’ll give them that high level point.
But let’s go back to Steven Casper’s criticism. He’s saying, “It is hard to overstate how disappointing I think this new paper from Oxford, OpenAI, Anthropic, and Google et al. is.” And by the way, Steven Casper — he’s AI safeguards and government research, a PhD student at MIT CS and AI lab, minor in public policy, and fellow at BKC Harvard, former AI Security Institute. This guy seems pretty legit.
Ori 01:52:09
Yeah.
Liron 01:52:09
Let’s listen to his critique. It says, “Hard to overstate how disappointing. I can’t take it seriously as academic work, just as propaganda. It also has some very bad scholarship and questionable adherence to research ethics.” Oh, snap.
Ori 01:52:22
Yeah.
Liron 01:52:22
“Having the title and author list that it has is not a great start, but I think that the actual content of the paper is also much worse than it could have been. The paper’s content is a series of sections that mostly just list things with discussions that I think are generally vapid.”
“For example, section 3.2 is titled New and Technical Approaches to Positive Alignment, and it has a collection of paragraphs on things like goal setting and evaluations, memory and in-context learning, and other general research topics of the LLM era. It overall strikes me as a paper built from the top down. The authors wanted to make a certain point up top, and the paper’s content ended up as filler.”
“I think of this paper as a mechanism of corporate capture of concepts from academic research on AI in society. It discusses topics like pluralism, liberty, and education, and frames them as solvable problems whose solution is the right tech integrated in the right way.”
And by the way, we talked about rationalization on the stream. It sounded like they wanted to get this conclusion of, “Yeah, it’s all about building pluralistic AI, woke AI” — putting words in their mouth, but I feel like that’s where their head was.
Ori 01:53:21
Mm-hmm.
Liron 01:53:21
At least that’s what Steven Casper is saying.
Ori 01:53:24
Right.
Liron 01:53:25
He’s saying, “I think that when this paper says pluralism, liberty, and accountability, it means them in a way that is profoundly vapid and structurally ignorant. For example, there is a list of papers out there arguing against this paper’s perspective, saying that pluralistic alignment is not a model property or a technical problem at all. None of them were mentioned.”
“Relatedly, the paper talks about some things that would be genuinely great if the authors’ companies were not actively contributing to the problem. For example, section 5.1 is about the decentralization of power in the AI ecosystem. Great, but come on. To listen to this stuff from OpenAI, Anthropic, and Google employees, I need more than just a disclaimer at the end saying, ‘This research paper represents the author’s own views and conclusions.’ This is how big companies launder their reputations through research.”
“The first author of the paper posted about it yesterday saying, ‘In a rare collaboration between top universities and three frontier labs.’” So which is it? “For a paper like this with this kind of author list to honestly and ethically engage in this kind of politics, it would need to seriously confront the question of how much these authors’ institutions are actively working against goals like this. If not, the big tech company authors should not have worked on this paper in their formal capacity as representatives of their companies.”
All right, it’s a damning critique.
Ori 01:54:31
Boom. Wow, Casper. That’s amazing.
Liron 01:54:31
I mean, look, I haven’t gone in and read the paper. But I feel like Steven Casper is gonna be spot on if I were to read the paper.
Ori 01:54:40
Yeah. I would be curious, has anyone, have any of those authors responded to him?
Liron 01:54:45
That’s a good question. Let’s see their responses here. Oh, Machine Learning Street Talk responded, “So depressing, isn’t it?” That’s interesting because I normally think of Tim from Machine Learning Street Talk as somebody who’s receptive to stuff like this, but maybe I’m wrong. He says, “So depressing, isn’t it? I’ve not actually read this paper, but I did run it through Pangram Labs, and the results are shown in the image below. 50% AI generated.” Oh, interesting.
Ori 01:55:05
Whoa, dude—
Liron 01:55:05
Wow.
Ori 01:55:06
MLST is behind this? That’s a statement right there.
Liron 01:55:09
Totally, totally. Yeah.
Ori 01:55:09
Okay, the community’s gonna reject this one.
Liron 01:55:12
Right. Yeah. He says, “I also generally find that Pangram gives the benefit of the doubt to avoid false positives, so it’s probably much worse than that.” Wow, that is funny. He’s accusing the paper of being AI generated. That’s consistent with Steven Casper’s critique — they kind of wrote the headline and then just said, “Okay, fill in the filler, support this headline that AI is all about pluralism or whatever.” Getting good AI is all about these nice values.
Ori 01:55:32
Okay, well now the authors have to respond to that because MLST — people care about what MLST thinks, so surely they’ve responded to that.
Liron 01:55:39
Totally. Yeah, surely they would respond. Let’s see. He’s at 47% score. Let’s see.
Liron 01:55:48
Lawrence Chan — who’s that? He says, “I do alignment research formally, meter evals Redwood AI.” He says, “Yeah, the Less Wrong people are justifiably upset that Yudkowsky’s coherent extrapolated volition wasn’t cited, but the related work discussion is bad even by ML academia standards.” And then Steven Casper’s replying to him, “Makes sense, but I was referring to other papers.” And then he lists a few other papers. Somebody’s agreeing. I’m looking for anybody disagreeing. I’m not really seeing a big name with a lot of likes engaging.
Ori 01:56:20
I’m surprised Seb Krier didn’t respond because that guy’s super active on Twitter.
Liron 01:56:24
Yeah. He is quite active on Twitter, yeah. Huh.
Liron 01:56:28
Well, I actually invited Steven Casper to come on the show, and he hasn’t responded yet. So hopefully he does, and he’s welcome to dig into this paper. Feels like somebody’s gotta do it.
Liron 01:56:39
Nice. All right, so we’re running out of time here.
Liron 01:56:42
So we will prioritize — or you wanna just go 10 or 15 minutes and call it a day?
Ori 01:56:47
Sure. Yeah, sounds good.
Liron 01:56:48
All right, sounds good.
Liron 01:56:51
Hmm. So let’s—
Ori 01:56:52
Well, I mean, what are the other... I think about what are the meta things that are going on right now? And to me, one of the big topics is everyone’s complaining about Anthropic, that they raised the prices, which is—
Liron 01:57:05
Right. Hear, hear. Yeah, I mean, I’m looking at my bill. I definitely spent another 500 bucks today, but I expect it. I think I’ve said this on the show — I expect them to both raise and lower. I do think that the exact same functionality I’m getting is going to be 10X cheaper within a few months because that’s generally the pattern of these things.
But at the same time, I also expect that I’m gonna want their latest and greatest, and it’s gonna cost exactly the same exorbitant price as I’m paying now. I think the exorbitant pricing is going to stick around for actually a surprisingly long time because there’s going to be so many uses of it.
I just think there’s a supply crunch potentially until the world ends. Potentially until this galaxy has been fully harnessed and the atoms of the galaxy — we may have a computer chip supply crunch until this whole galaxy has been shaped into its final form. If you come here for economic prognostication, that’s the best I can do right now.
Ori 01:58:00
Let’s see. Another news item is a scoop by Max Zeff from Wired—
Liron 01:58:06
Uh-huh.
Ori 01:58:06
Which is that OpenAI announced another major reorg on Friday. So that’ll be something to talk about.
Liron 01:58:12
Yeah, I heard something about that. Greg Brockman I think is getting more responsibility now. I think he was in an interim role—
Ori 01:58:18
Yeah.
Liron 01:58:18
And he’s gonna permanently run product.
Ori 01:58:20
Right. Right.
Liron 01:58:22
Interesting. Let’s see.
Ori 01:58:24
Yeah, I mean—
Liron 01:58:24
Okay, yeah, go ahead.
Ori 01:58:26
They’re just — it seems like they’re behind now. They’re no longer the top dog.
Anthropic Dethrones OpenAI
Liron 01:58:31
Right. It’s so fascinating how everybody’s like, “Yep, Anthropic’s number one and OpenAI is number two.” It’s so crazy. As far as I can tell, what happened was OpenAI kind of had the most capital, had the most compute resources, and had number one share among consumers, which they still do.
Ori 01:58:46
Yeah.
Liron 01:58:46
And everybody’s like, “Okay, well, they got the consumers. They’re the next Google.” Everybody’s gonna do their Google searches on OpenAI now, and how do you unseat that? They’ve got a moat.
And then Anthropic came up and they’re like, “You know what we’re gonna do? Help businesses code.” And next thing you know, people like me, who have a revenue model where they need to code, are shoveling hundreds of dollars a day at them as opposed to $20 a month. Hundreds of dollars a day per engineer, or at least tens of dollars a day per engineer instead of tens of dollars per month.
They’re shoveling money at Anthropic even more so than at OpenAI because Anthropic has a slight lead on the coding front, and they’ve developed that and they’ve developed the business relationships. And suddenly even their revenue graph appears to have leapfrogged OpenAI, or at least it’s very close and their momentum’s higher.
And OpenAI is sitting here being like, “Well, maybe I can squeeze more money out of the consumers. We’re gonna release the ad program.” And Anthropic is like, “We’re never gonna do ads” — that was their Super Bowl ad. And suddenly Anthropic’s the king. Just like that it flipped over. Crazy stuff. As a guy who likes to watch the drama, it’s been fascinating.
Ori 01:59:46
Yeah. And also what’s weird for me is that, okay, we have the Elon Musk Sam Altman trial, and a lot of that is — to me the drama about that was OpenAI is the top dog. Sam Altman’s the king. And now someone’s going after the top guy. But now OpenAI is — to me it seems clear that they’re number two right now, so what’s the big deal about the case?
Liron 02:00:15
Right.
Liron 02:00:16
Yeah. What’s the big deal if it’s just two peasants fighting—
Ori 02:00:20
Right.
Liron 02:00:20
And the real king isn’t even part of the case?
Ori 02:00:23
Yeah, exactly.
Liron 02:00:25
Yeah, I mean, that’s a fair perspective. The stakes are... Funny enough, Elon is now a compute provider for Anthropic, so yeah, I don’t know.
Liron 02:00:33
I don’t have much to say about the case, but it is a good point. What was I gonna say about Anthropic? It’s just funny that OpenAI is on the back foot. Now Dario’s the king and Sam Altman’s number two, and he’s struggling to catch up. Let’s see what he does. When you — he’s a cornered animal right now. Sam Altman is a cornered animal. He knows how to lash out.
Ori 02:00:51
That is a good point. And here, okay, here’s one thing I was wondering actually. We know from using it how useful Claude Code is, which is an agent and it’s some kind of harness and — I don’t know how many magnitudes better it is, but if you put a prompt in Claude Code, it’s much, much better than ChatGPT, or just a simple Claude AI. So one thing I’m wondering is how soon till that gets more productized. The Claude Code experience should be your typical chat experience.
Liron 02:01:21
Right, right. And yeah, it’s called Claude Coworker. And they said Claude Coworker is growing even faster than Claude Code.
Ori 02:01:28
Oh, okay. I mean—
Liron 02:01:30
Organizations are adopting Claude Coworker, and it’s like, “Hey, look, all you white collar people who would normally...” I mean, this was — I used to say 10 years ago, “Hey, you want a startup idea? Go watch somebody with a white collar job work.”
What you’re gonna see is — wait an hour, okay? Let them get into their groove, and you’re gonna observe the following. They open Excel, they scroll, they select, they copy, they open some other program, they paste, and then they do that 100 times in a row. If you watch anybody with a white collar job, you’re gonna observe a flow where if you know software engineering or if you know modern tools, you’re gonna be like, “Oh, my God.”
Ori 02:02:03
Right.
Liron 02:02:03
And then you go back and you work on a product that streamlines their flow, and then you sell it back to them and you’re like, “Here, would you like to use this? No more copying and pasting.” And that was the secret formula in order to make multimillion dollar software businesses. You just watch people’s actual workflow.
But now with Claude Coworker, Claude Coworker can just watch that workflow, and then it can just on the spot do what you need to do.
Ori 02:02:23
Yeah.
Ori 02:02:25
Interesting. Dude, you’re giving me flashbacks from—
Liron 02:02:28
When I worked at a PR agency, okay?
Ori 02:02:32
You have no idea. And—
Liron 02:02:34
You’re like, “Hey, I was that copy paster.”
Ori 02:02:36
Just the amount of effort that it took because we would be like, “Okay, we’re doing this interview. We gotta get the messaging.” And it was like, oh, the messaging was from this one random time. So you’d have to search through your email every day, just spending hours searching through your damn Outlook inbox to find that exact quote that was used, and then you copy it and paste it into the new one.
Liron 02:02:59
Yeah. This is kind of random, but I have this friend who just seems incapable of using the search function on his email, even though I know it’s back in his Gmail. So he’ll be like, “Hey, can you send me that email again that you sent me?” I’m like, “It’s the last email I sent you. Just type Liron. It’s literally just gonna come up if you type Liron.” He’s like, “Can you resend it?” It’s like, “Yeah, I just resend it.”
Ori 02:03:21
Oh, man.
Liron 02:03:23
Yeah, it’s hard.
Ori 02:03:23
Okay, so it’s called Claude Coworker. So you think... I mean, but—
Ori 02:03:27
I’ve messed around a little bit with Claude Coworker. It’s not the text experience the way that Claude Code is. So I don’t know. It’s not quite there, but okay. That’s the product.
Liron 02:03:40
It’s winning. I think that’s the breakout product. We think about it as Claude Code, us Silicon Valley guys, but I think Coworker is really the winner.
And by the way, I wonder what their latest revenue is because I think even though it’s ridiculously high — last number I heard was forty-five billion, and literally two months ago, we were here on the show being like, “Oh, my God, nine billion” — I think technically the rate of doubling is slowing, which stands to reason. Because if the rate of doubling continues for one and a half more years, they will literally just have the entire world economy. So it stands to reason that the rate of doubling needs to slow.
Ori 02:04:16
Yeah.
Liron 02:04:16
So I think it has been slowing a little bit.
Ori 02:04:19
Yeah. I saw a tweet that kind of blew my mind. This is related to what Imad Mostaque said, which is something like how many Claude Code users are there? Between fifteen to twenty million, or between ten and twenty million are Anthropic subscribers. It’s just — the—
Liron 02:04:40
Mm-hmm.
Ori 02:04:40
How many people... What is the current market of Claude Code or Codex users? How big is that?
Liron 02:04:47
Yeah, but don’t think about it as people. Think about it as business subscriptions. They’ve got most of the Fortune 500, as far as I can tell.
Ori 02:04:55
Oh, okay. Okay. Fair. Yeah.
Liron 02:04:58
Yeah. Life as an individual versus business is very, very different. The thing about business is you turn on the cash fire hydrant because it connects directly to the profit-making. You know, okay, I’m not spending this money, I’m investing this money, and the payout is so clear. So suddenly Anthropic gets another billion.
Ori 02:05:17
Yeah.
Liron 02:05:17
All right.
Ori 02:05:17
Did you see the Jack Dorsey memo that went out and how Jack Dorsey is trying to—
Liron 02:05:23
I think I did. Yeah, it was kind of brief, and he was basically saying, “Hey, we’re tightening up. We’re becoming AI first organization, and thirty percent of you need to go.”
Ori 02:05:32
I think that may be part of it, but I thought it was just interesting. To me, he was saying they’re flattening the organization, and there’s more ICs because now you can just—
Liron 02:05:42
Right.
Ori 02:05:43
Code everything.
Liron 02:05:43
Oh, yeah. He’s saying nobody’s job can just be to manage people. Everybody has to talk to AIs and contribute work that way.
Ori 02:05:51
Right. Right. And I don’t know, have you read much about it? Apparently at Amazon, they have a token budget, and everyone’s supposed to be using a certain number of tokens. Have you seen much about that?
Liron 02:06:02
I have seen — I thought the story was that Amazon did this big push where, “We wanna see you guys all using tokens.” Which is a good idea in principle because whenever I see you or people who work at my company using tokens, doing stuff with AI, I’m usually really happy because they show me something and I’m like, “Oh, wow, well done.” You used the AI to do something cool that I’d never done before, and it makes a lot of sense.
So this whole idea of “use more tokens” is a good rule of thumb. But then, of course, Goodhart’s law. The moment they said, “Hey, use more tokens,” now there’s stories coming out of employees who are like, “Yeah, I just run the AI all day long,” right? Crunching through tokens, and I don’t even look at it.
Ori 02:06:38
That’s hilarious.
Ori 02:06:41
I can’t even imagine what that would be.
Liron 02:06:43
Yeah. And then people are joking about how Amazon is notoriously cheap on everything. So they’re like, “Hey, you know, you can’t spend more than twenty dollars on your DoorDash lunch, but it’s fine if you spend six hundred dollars on tokens today.”
Ori 02:06:54
Right. Right. Yeah, I saw those memes. I suppose, if you’re in a big organization, how are you going to enforce or ensure that AI is getting used? Just use tokens. That’s a good simple metric to have for it.
Liron 02:07:08
Right. It’s a necessary but not sufficient condition. If your organization, if nobody’s using the tokens, that does point to a problem that you’re not harnessing AI. But you can’t solve the problem by just saying the more tokens the better, because then people just cheat their way to more tokens.
Ori 02:07:22
Right. Right.
Liron 02:07:23
All right. AI Not Kill Everyone Is A Memes posted — oh, yeah. I don’t think we talked about this. Mythos cracked macOS in five days. Did we talk about this at the beginning of the show?
Ori 02:07:34
No.
Liron 02:07:35
Okay, I talked about it on Warning Shots. That’s why I’m confused.
Ori 02:07:38
Uh-huh.
Liron 02:07:38
I’m like a stand-up comedian. I heard Bill Maher saying, “You’re in a stand-up comedy show, and you’re doing two shows that night or three shows, and you get to this joke, and you’re like, ‘Did I already make this joke in the same set?’” Because the moment you crack the same joke twice in one set, they just hate you. You’re a fraud. None of this was funny. So you have to err on the side of not saying it, I guess.
Ori 02:07:58
All right.
Liron 02:07:58
All right.
Ori 02:07:58
Next time I’ll just let you go, and then afterwards be like, “Liron—”
Liron 02:08:02
Yeah.
Ori 02:08:02
“You said nothing.”
Liron 02:08:02
I’d be like, “Oh, my God, how can I live down this shame of covering the same news twice?” All right.
Liron 02:08:08
So Mythos cracked macOS in five days. Why this matters? It takes Google Project Zero, the most prestigious bug-finding team in the world, six months per zero day at the macOS/iOS level. And those zero days are worth $2 million each. So it is just crazy. It’s chopping right through a $2 million task in one day.
But of course, once they dry up, the fruit becomes higher hanging. So it’s kind of a one-time thing that you can do. But look, macOS is extremely secure, and Mythos found holes in it. For me, the big takeaway is, yes, AI is going to see us with our pants down, and in this occasion, we were able to pull our pants up higher. So it’s a one-time pants-pulling event.
But in general, I claim — I know Mike Israetel disagrees — but I claim our pants are down so much. And it’s irrecoverable.
Ori 02:08:59
Interesting. Well, what I was thinking about though with the Mythos story is the news that Anthropic is giving $100 million worth of credits on the Mythos defense, the defense squad. What’d they call it? Project Last Wing? That is pretty considerable. I think you have to commend them for that. That is a pretty commendable move to—
Liron 02:09:21
Yeah, but what’s the alternative? Because the problem is that they can foresee that if they don’t do an effort like this, they are going to rightfully get blamed. There’s going to be a problem, and they’re going to get blamed. So what are they supposed to do?
Ori 02:09:34
Well, they could just be like OpenAI and just release it.
Liron 02:09:35
Well, but OpenAI—
Ori 02:09:37
And then blame—
Liron 02:09:38
Hasn’t released anything that dangerous yet.
Ori 02:09:40
Well, supposedly o3 5.5 Codex is just as good as Mythos.
Liron 02:09:47
So I mean, look, to your point here about, oh my God, they wanted to secure it before releasing it. I think it’s obvious that the companies, if they’re thinking six months ahead to getting blamed, they’re likely to avoid that. That’s standard US company, standard capitalistic companies have a little bit of foresight like that. The problem is when they’re going to blow up the world and not be held personally accountable right up until the world ends. Those are where the problems come in.
Ori 02:10:14
Yeah, yeah. But I think assigning blame will be tough for them. But still, you think the pants... So even with $100 million of token use for the defensive side, you still think there’s a lot of pants-downiness because that net is not broad enough to defend against all the cybersecurity threats.
Liron 02:10:37
Yeah. I think everybody’s still gonna have some errors in their application. Yeah, macOS will be fixed. All the highest value projects will be fixed, but there will still be a bunch of random projects where people will be finding vulnerabilities and exploiting vulnerabilities for a while. So it’s gonna be touch and go for six months or a year or whatever, with a long tail.
But this isn’t the kill shot. This isn’t the knockout blow. But the one after this could be. At some point, I think there’s a criticality threshold for these kind of blows. When the AI is autonomous, as long as it’s just, oh my God, LLMs reach a higher threshold, but a human can still reach around and turn them off — there’s still end runs that we can do as humans. As long as the LLMs aren’t these independent actors, as long as they’re not head-to-head, fully super intelligent, then I’m optimistic that we can keep working with them. That they can add value, that our institutions will make them net good.
I’m an AI optimist until they get super intelligent and uncontrollable, and then I’m like, “Well, I think we’re screwed at that point.”
Ori 02:11:37
Right. Yeah.
Ori 02:11:38
And do you know what’s kind of unfortunate with the Eliezer debate? He said explicitly, that’s the first time I heard him say it explicitly, “I don’t have the red line for you”—
Liron 02:11:49
Right.
Ori 02:11:49
Of when it reaches that criticality threshold. So I wish we had that red line.
Liron 02:11:56
Right. I mean, that’s one of the many reasons we’re doomed — we don’t even know exactly what’s the ideal point to stop at. It could be totally rational to stop in 2.5 years as opposed to stopping today.
Trump & Xi Talk AI Guardrails
Ori 02:12:07
Right, right.
Liron 02:12:07
Nice. All right. So we’re gonna wrap up in one minute. This is probably as good a note to end on as any. We’ve got — this is from today, eight hours ago. Somebody asked President Trump. Let’s play the clip.
Press Conference 02:12:20
So on artificial intelligence, what is it that you wanna get done? Did you bring up Mythos and all those things?
President Trump 02:12:26
We did. We—
Press Conference 02:12:27
Mythos and all those things?
President Trump 02:12:27
We talked, you know, we’re leading by a lot, but they’re second, and they’re very strong. And we talked about possibly working together for guardrails.
Press Conference 02:12:37
And what kind of—
President Trump 02:12:38
And he said he’s very—
Press Conference 02:12:39
What kind of guardrails are you talking about?
President Trump 02:12:39
The standard guardrails that we talk about all the time. AI is fantastic. So many things can happen in terms of health, medicine, and operations, everything, the military. So many things can happen, but it’s also got some drawbacks, and we’re talking about we probably will wind up working together. We had a very good conversation about a lot of stuff.
Press Conference 02:13:02
So biological, nuclear, cyber. Would you specifically discuss those risks?
President Trump 02:13:08
We discussed, I would say, almost everything you could discuss except for a reduction of tariffs.
Liron 02:13:19
Oh, Trump. What a one of a kind guy.
Ori 02:13:20
Oh my God.
Liron 02:13:20
But what he’s saying is, “We talked about possibly working together for guardrails.” And the question was on artificial intelligence.
This is quite a contrast. His administration, the David Sacks of the world, and to be honest, Dean Ball, when he wrote together with, I think it was Michael Kratsios, who was writing the American AI action plan — they were all like, “Yeah, rah, rah, it’s all about America being in the lead. Don’t worry about recursive self-improvement.”
And yet, here’s Trump talking with Xi Jinping being like, “Hey, why don’t our two countries put on some guardrails?” It’s like, “Yes, thank you.” So this is quite an optimistic sign.
Ori 02:13:55
Hear, hear.
Ori 02:13:56
Second that.
Liron 02:13:56
Yeah, I mean, that’s the funny thing. When you watch Doom Debates, you see all these luminaries coming on the show and debating me. But normal people are like, “Yeah, let’s...”
And to be fair, normal people are biased to want a higher than optimal level of regulation. I agree with the Bryan Caplan perspective — I lean libertarian. I think that regulation tends to stifle the engine of value creation, and we have too much regulation. But the average person is biased to want too much regulation.
But look, two biases are canceling out in a good way right now. The average person’s love of regulation and fear of the unknown is canceling out in a very convenient way to be like, “All right, let’s put guardrails on AI.”
Ori 02:14:35
You know what was fascinating though — let’s just... If you replaced Trump with Biden there, imagine the right-wing, the Republican reaction to that. They’d be like, “You can’t trust China. They’re gonna...”
Liron 02:14:47
Right.
Ori 02:14:48
So I think it’s great that Trump is saying it. And I don’t know, it’s positive momentum. You could still have people on the right looking at this and be like, “You can’t trust China,” because that often comes up when you talk about this subject. ## Wrap-Up and Announcements
Liron 02:15:06
All right, let’s wrap it up here. I want to give a shout-out to Less Online. There’s the Less Online conference June fifth to seventh. I’ll be there. Producer Ori will be there. We’re actually hoping to close a sponsorship deal, so it’ll be sponsored by Doom Debates hopefully. You can come — we’re gonna be doing some live debate recordings as well, so you can join that. We’ll be giving out merch.
Besides that, stay tuned next week. We’ve got the Ben Gerstel debate. I think you guys are gonna enjoy that. Ori, any last words or announcements?
Ori 02:15:39
No, I mean, yeah. Looking forward to seeing the reaction to that Ben Gerstel debate.
Liron 02:15:46
Hell yeah. All right, everybody, have a great weekend.
Ori 02:15:49
Bye.
Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.
Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏
