Dr. Holly Elmore, executive director of PauseAI US, is done giving AI insiders the benefit of the doubt.
In this spicy episode, Holly calls out a number of respected figures in the AI safety community for allegedly “selling out” the cause: Effective Altruists, Anthropic & OpenAI employees, Eliezer Yudkowsky, Zvi Mowshowitz—and even yours truly!
Holly isn’t afraid to psychoanalyze her subjects, which I push back on, but I can’t deny she has a track record of pushing the Overton window and surfacing uncomfortable truths.
Does Holly’s approach risk burning bridges with allies for little benefit, or does AI safety need a moral conscience willing to call out the rot nobody else will?
Watch on YouTube
Timestamps
00:00:00 — Cold Open
00:00:48 — Introducing Holly Elmore
00:02:20 — Why Holly Moved to DC
00:06:03 — What’s Your P(Doom)?™
00:07:18 — How We Got Here: EA, Rationality, AI Alignment
00:16:32 — “If Anyone Builds It” and LessWrong’s Missing Mood
00:21:47 — Holly Supports the Government’s Fable Crackdown
00:33:39 — Beef: Dean Ball (OpenAI)
00:53:09 — Beef: Eliezer Yudkowsky
00:56:14 — Beef: Zvi Mowshowitz
01:11:50 — Beef: Liron Shapira!
01:16:15 — Beef: Nirit Weiss-Blatt (Dr. Techlash) Being “Bad Faith”
01:19:31 — Anthropic Took Over EA
01:22:04 — Holly’s Basilisk: What If Building Superintelligence Becomes a Crime?
01:32:35 — Beef: Amanda Askell, the “Mother of Claude”
01:39:05 — Beef: Daniela Amodei, Co-Founder of Anthropic
01:40:23 — Beef: Joe Carlsmith, Member of Technical Staff at Anthropic
01:49:39 — Beef: Open Philanthropy / Coefficient Giving, Ben Goldhaber
01:54:28 — Wrap-Up
Links
Support PauseAI US: https://www.pauseai-us.org/
Holly on Twitter/X: https://x.com/ilex_ulmus
Holly’s Substack:
Last year’s episode (Circular Firing Squad):
The AI Nuremberg trials / Holly’s Basilisk:
A selection of Holly’s Twitter Beefs
Dean Ball:
Eliezer Yudkowsky:
Zvi Mowshowitz:
Nirit Weiss-Blatt:
Anthropic:
Amanda Askell:
Daniela Amodei:
Dario Amodei:
Joe Carlsmith:
Evan Hubinger:
Ben Goldhaber / Open Phil / Coefficient Giving:
MIRI:
Joe Carlsmith’s “Leaving Open Philanthropy, going to Anthropic” post: https://joecarlsmith.com/2025/11/03/leaving-open-philanthropy-going-to-anthropic/
Dean Ball joins OpenAI announcement (FAI post): https://www.thefai.org/posts/dean-ball-joins-openai-as-head-of-strategic-futures
Transcript
Cold Open
Liron Shapira 00:00:00
Holly Elmore, you’ve been pretty unique in your willingness to have these beefs with people. Joe Carlsmith, right? He recently joined Anthropic.
Holly Elmore 00:00:07
I called him a sellout. You don’t get to be serious about safety and work at Anthropic.
Liron 00:00:12
Amanda Askell
Holly 00:00:14
One of the most egocentric and hubristic people ever. I’ve been saying Anthropic final boss for a while now.
Liron 00:00:22
Zvi Mowshowitz.
Holly 00:00:23
What Zvi does is, like, the definition of regulatory capture.
Liron 00:00:26
You were sparring with Dean Ball.
Holly 00:00:28
Like, the emperor knows he’s not wearing clothes.
Liron 00:00:31
That makes me wanna ask you about another potential beef target here. Make the case against Liron Shapira.
Introducing Holly Elmore
Liron 00:00:48
Welcome to Doom Debates. My guest is returning champion Dr. Holly Elmore, executive director of Pause AI US, the nonprofit grassroots organization dedicated to an international treaty to pause frontier AI development until it’s safe to proceed. I personally support Pause AI US. I’m a member in good standing.
Holly got her PhD in evolutionary biology from Harvard. Now, you might be wondering, “Wait, Harvard? I’m seeing that a lot on Doom Debates.” And yeah, they’re doing a lot of good work at that institution. It’s an up-and-coming place.
Holly also has a history with effective altruism and rationality. We’re gonna be talking about that now because she is distancing herself from those communities these days. And also for further background, I encourage you to check out last year’s episode where we talked all about how the rationalist community is becoming a circular firing squad. They don’t seem to be able to unify to coherently support the same causes. I thought that was a really good discussion, so go check that out.
Today I’m glad to be back with Holly, catching up on all the latest developments, talking about some of the new blog posts she’s written, all the latest developments that have happened in our various communities and in AI progress and in the politics of pausing AI, and just go through all the reactions we’ve had to what other people are saying that often make the two of us be scratching our heads and disagreeing. Holly Elmore, welcome back to Doom Debates.
Why Holly Moved to DC
Holly 00:02:21
Glad to be here.
Liron 00:02:22
All right. Looking forward to catching up. So first things first, you have just made a huge move, right? Tell us about that.
Holly 00:02:28
Yeah, I just moved to Washington, D.C. from San Francisco, and loving it. The biggest motivation is to be closer to Pause AI US, which increasingly is about facilitating constituents talking to their elected representatives about the issue of AI danger. And then it also made more and more sense for me to be talking directly to representatives.
When I started Pause AI US, I thought a lot of what we would be doing would be directly protesting industry — and actually, Liron was present at direct protests at the AI company buildings in San Francisco. But it’s become less of our focus. We’re really focused on those quality contacts with elected officials now, and so we’re moving to D.C. I’m not the only one. There’s kind of an exodus right now from focusing on the tech insider world as a way to deal with AI danger to focusing on governance.
Liron 00:03:19
Yeah, I think this is a big theme of your arc personally, and you’re kind of at the forefront. You got me into it to some degree. This whole framing change where you’re like, “Man, all these people who are working at these AI companies, they come from the same background as us. They both got interested in effective altruism at an early age, and they think they’re trying to do what’s best in the world. But ultimately, it doesn’t really seem like they’re on the same mission as us anymore, and it’s more productive to just say, ‘Okay, well, they’re actually the enemy right now.’ They’ve joined the dark side.”
Holly 00:03:53
Honestly, the most intense conflicts you ever have are usually over — I mean, some people would call this the narcissism of small differences, but I think in some ways these conflicts are the most real. You’ve stripped away every other disagreement you have, and you actually realize you have this crux.
My strong feeling is that the way we need to deal with the issue of AI danger is through elected governments, through democracy, through diplomacy. And there’s a lot of bizarre resistance to that in EA rationality, the places where the original technical AI safety community came from. Part of it is I think they just want the issue to continue to be technical and something that they can deal with in the way they want. And part of it is, at this point, people who have equity in Anthropic or OpenAI are kind of hoping that they win and usher in a brave new world and maybe bring in the singularity or something like that. At some level, I think that’s operative for many core people.
Holly 00:04:55
That is not a small disagreement to me at all. I think what that’s talking about is a coup of everyone on Earth. It might seem like, “Well, but you’re both talking about AI safety,” but I think no, we’re not. The talk you hear from the people at Anthropic working on safety is about getting us to this next era safely. It’s really not at all about who controls it or do we maintain other people’s values, that sort of thing. So there’s some really serious things at issue there.
The wing I’m in of the AI safety movement is more focused on governance — not getting to a situation where AI is super powerful until we know how to do that right. And people who are reluctant to stop the speeding train with no brakes, I just don’t trust. They can say they have all kinds of reasons, but in the end, why wouldn’t you stop the train first? Maybe we restart it and we go somewhere cool eventually. Why wouldn’t you be willing to pause? So that’s, in a nutshell, the divergence.
What’s Your P(Doom)?™
Liron 00:06:04
All right, by way of intro, we gotta get this out of the way—
Holly 00:06:07
P(Doom). P(Doom), what’s your P(Doom)? What’s your P(Doom)? What’s your P(Doom)?
Liron 00:06:12
Holly Elmore, what’s your P(Doom)?
Holly 00:06:15
God, I don’t even remember what I said last time. I don’t truly think this way, but I think in terms of the risk, and I think the risk is too high. It probably has gone up since the last time we talked. To me, currently we’re under risk, and that’s intolerable. It’s not because I think it’s more likely than not that we will get a disaster — I don’t know, we could get lucky — but I still think the risk is intolerable. It’s probably 50 to 60% now. I used to be lower.
Holly 00:06:47
I still put some 15 to 20% on just us getting lucky somehow. Even when things look really bleak and I don’t see a reason to think that the right forces are necessarily going to intervene and stop something bad from happening, we still don’t understand all of the forces at play, and it’s possible that they just don’t add up to something really bad.
Liron 00:07:09
Yeah, you’re not 99%. Okay, well, same here. I think we have a pretty similar conception of doom.
How We Got Here: EA, Rationality, AI Alignment
Liron 00:07:15
All right, so we’re just catching up as two mostly like-minded people. Let’s just review the world through Holly’s eyes, which I think I mostly agree with. Very briefly, for somebody who’s just getting into this, using Doom Debates as their gateway into the world of what’s going on with the AI industry and who works there and what were the origins of it — very briefly, how did we get to this point?
Holly 00:07:43
How did we get to this point? The idea of artificial intelligence is pretty old. People might have read about it in Dune from the ‘60s, and then Dune was referring to an essay from Samuel Butler from even earlier than that. The idea that we could make a machine that was intelligent and then maybe more intelligent than us — why not — is pretty old. So a lot of people, it had occurred to them that this could go bad if you had another intelligent species, basically. You could have conflict with them. And there was a group of people wanting to solve the problem of
Holly 00:08:20
the alignment of AI. AI alignment is the name that’s given to making AI right in the first place so that it doesn’t have conflict with humans. Even if it’s more powerful than us and it would be able to win a conflict with us, it won’t be in conflict with us. It’s actually a pretty difficult philosophical problem to figure out what do you even do to achieve alignment. Does it care about us being happy? Does it do what we say? Does it kind of know what would be good for us if we were smarter? All of those are pretty dense philosophical questions.
I was sort of part of that world of people trying to figure out alignment before we had a really serious paradigm for realizing AI in the world. I was very familiar with their thinking and their arguments. Generally, the thrust of that world — which was roughly the rationality community, people who followed Eliezer Yudkowsky, who I’d say is the original AI alignment guy, and the effective altruism community, which is kind of how I came in — those communities wanted to fix it in a technical fashion. They wanted to just find the answer to alignment, and then everything would be okay.
I remember I asked early on, “Why don’t we talk to people about this?” I was becoming convinced of the argument that the danger was real. I was like, “Why don’t we talk to people about this?” And they’d say, “Well, it’s just so weird. People bounce off it, and it looks bad for the community if you look weird, and so really we should just fix it ourselves.” So the only path they wanted to pursue was
Holly 00:09:54
technical — coming up with some kind of technical answer that would just make the AI good. I wish I had thought about it harder at the time, ‘cause I didn’t really think that deeply about it. It wasn’t my thing.
But then when ChatGPT comes out, it was clear to me — some people saw earlier, like when GPT-2 came out — that oh my gosh, if they got to here, to natural language, simply by scaling the amount of compute they’re using with machine learning, then there’s probably no other major barriers to them getting a lot further. And then I became really concerned.
Around that same time, there were a lot of public communications about it, and polls were showing that the American public was very down with regulating, with the idea of pausing even. The idea of a six-month pause was floated in the Future of Life Institute’s open letter asking for a six-month pause, and a majority of American adults were even in favor of that. So I thought, yay, this problem that seems so serious and now it’s really urgent — it looks like there’s also this huge opening. Now we can talk to the public.
There just continued to be no enthusiasm for doing that from those groups, and I guess we can discuss why. But I eventually ended up doing my own thing and then eventually splitting, because I don’t see any solution to alignment that doesn’t start with governance. I don’t think that you can just make the machine the right way such that it would never do anything bad or against humans. I think that’s entirely a question of what it can do, and if it can do well by us, it will have the power to do even more things ill by us. The only thing that distinguishes between what’s good and bad for us is how we govern it. So even a technical alignment solution — there’s nothing to make it aligned if you don’t have proper governance or if you’re unwilling to
Holly 00:11:57
reach solutions through non-technical governance. And then just practically, all of that technical safety work is now taking place in the AI industry. Basically, either they’re literally the employees of AI companies who do it, or they’re people who completely depend on the AI companies to allow them to even look at the models to do any work. That’s just utterly unacceptable. We need an ecosystem with real accountability and real external regulation to even be able to work on the problem. So in many ways, we need governance, and that’s how I ended up here.
Liron 00:12:35
Makes sense. And then most recently, your move from Silicon Valley to Washington, D.C. was the final expression of your change of focus, right?
Holly 00:12:44
Yes. I do think there’s stuff to do in San Francisco — we did in-person protests there, which I think at the time, even for their small size, they were big news items. They communicated this missing mood of, “If this is so dangerous, how come no one’s in the street?” And there are people in the street, and now there’s other groups that do protests and show how they’re feeling, and the issue of AI danger is very mainstream. So I think we seeded some stuff in San Francisco, and I’m ready to focus on our real thrust, which is using our existing democracy, that machinery, speaking up.
We have the public on our side. Depending on the particular question of our platform that you poll on, it’s 60 to 80% support. So we have the public, and what we need now is to make the public’s voice heard in government.
Liron 00:13:39
Yeah, totally. I agree with you. You mentioned some organizations are on the same page. Not all the organizations. Certainly, the AI companies seem to be more focused on how great it is to increase capabilities, and they’re arguably the biggest problem right now because if they would just agree that the Pause AI approach makes sense, if they would just agree, then there wouldn’t be too much of a problem. We could kind of get it done using their support, but they are telling the world that building capabilities is good. So they’re the enemy right now — one of the biggest enemies.
But we also have some organizations that are on the same page, like modern MIRI might be on the same page, because they’ve explicitly said that it’s now too late to hope for a technical solution anytime soon. There is a lot of alignment between the approaches of MIRI and Pause AI these days. Is that fair to say?
Holly 00:14:34
We’re both asking for an international treaty. Yeah. Our highest goal is the same. I think both organizations think that in the future, possibly, maybe there would be some way to build superintelligence that was beneficial for people, but that it is just such a dangerous losing proposition right now. And if we were to attempt it, just overwhelmingly almost guaranteed the result would be very bad for us.
Liron 00:14:57
Yes. And of course, MIRI is the Machine Intelligence Research Institute founded by Eliezer Yudkowsky.
Holly 00:15:03
Now, they’ve pivoted completely, or so they say, into doing public engagement and public education because they really just think the next issue is, do we allow this thing to be built? And if we do, if anyone builds it, everyone dies.
Liron 00:15:19
Yeah, to recap that, they started wanting to build superintelligence. You can find Eliezer’s writing from like 1999 saying, “Building superintelligence, that’s the meaning of life.” And then they had a realization. Eliezer was like, “Oh, okay, we really need to make sure that this is aligned superintelligence. This could easily go wrong, and let’s build that.” And then they’re like, “Okay, we’re not realistically building superintelligence anytime soon, but let’s just build the theory of alignment.” And from there, they went to, “We really need to build nothing right now.”
Holly 00:15:47
Yeah.
Liron 00:15:47
Some people can still research safety, but the most important priority is to make sure that other people don’t build superintelligence. And a bunch of people, who in my mind are clueless, are just accusing him of being bitter. “Wow, these guys are just Luddites or whatever.” But no, there is such a thing as just looking at the facts or being intelligent and just reaching a conclusion that something is ridiculously hard and then retreating.
Holly 00:16:09
That accusation of being a Luddite used to just kill them. They couldn’t stand it because they were always the futurists, and that was their whole identity. I think they’ve gotten used to it enough. But early on, it was really difficult. I think it took Eliezer longer than it needed to to come to the conclusion that there should be governance or an international treaty.
“If Anyone Builds It” and LessWrong’s Missing Mood
Liron 00:16:33
Right. Okay, so that’s a good overview for people. Let’s do some more recent current events. Let’s catch up. Going back to last year’s episode, we focused on how Eliezer’s book launched — Eliezer Yudkowsky and Nate Soares’ New York Times bestselling book called If Anyone Builds It, Everyone Dies. Our perspective was the community that helped bring this book about or that gathered around this book over the last decade or two, as it was coming together, the community that got really passionate about this book — it seemed like they were all playing it pretty cool when the book launched. They didn’t wanna do the usual book promotional activities.
We specifically picked on lesswrong.com, which was a site that Eliezer originally founded. To be fair, there’s the new Less Wrong team that kind of refounded it, so it’s not really Eliezer’s site anymore. But I personally was sad to see that that site wasn’t making a bigger hype deal of the book launch, and I think you were basically on the same page, right? What do you think about that now?
Holly 00:17:29
Well, I did a bunch of free one-sided promotion with no help from MIRI for Pause AI US for the book just ‘cause it was great educational material, and we were excited. We did launch events in I think four cities in the US, and then I was involved in four more internationally. So we were just super excited about the book and what it would mean for public understanding.
I still hear stories every day of people who are coming to Pause AI because they read If Anyone Builds It. Bernie Sanders clearly was very affected by If Anyone Builds It, Everyone Dies. So I knew that good things were going to come from it.
Yeah, and just to recap an entire video we already did, I was really shocked that the Less Wrong community — which is there on a site that’s based around being the home of Eliezer’s writings called The Sequences on Rationality — just were kind of... It was a suspicious missing mood of, “Mm, well, it’s just not that great.” They were more excited about critiques of the book that were more in their language than they were about the book.
That upset me because, I don’t know, I guess I was naive, but I really thought when I joined that world of EA rationality that people were really dedicated to solving the problems in the world, including AI danger. The Less Wrong people had always been the people more into AI danger. I started working on it ‘cause I thought there was this huge opening I didn’t see anybody else working on. But it was never my identity or anything, and for them it was always deep — like, “This is my thing, I’m gonna do alignment.”
And then I just felt like it was revealed that it wasn’t really about saving the world. That was ultimately the betrayal that led me to leaving those worlds. There was just something else to it. It was, “Ugh, he’s not saying it in the right way.” You could kinda get people to admit, “Isn’t this a good thing? Shouldn’t he get the word out about this?” And they’d be like, “Yeah, I guess.” They clearly weren’t interested in what’s gonna save the world the most. They kinda wanted the subject matter that they came for, which was more technical, which was more what the elites can understand.
Liron 00:19:46
So this is what we covered last year. And then with nine months of hindsight, things have cooled down. At the time, there was a heated discussion with some people from Less Wrong, including the Less Wrong team, and they were like, “Look, the book is great, and there’s a lot of interest in it on Less Wrong. But promoting the book is not what Less Wrong’s about. We’re all about just exposing new signal — things that are new. And the fact that this book is coming out and it’s great, there’s not much signal in that. It’s like, we know Eliezer Yudkowsky. We don’t wanna discuss him that much. And Less Wrong is just entirely this neutral signal place.”
They very much took that position. And look, it’s their site at the end of the day. To me it’s just sad that there was no community that stuck through that could be kind of the central community of supporting this book. The Less Wrong team was like, “Look, you’re thinking of some other community.” I’m like, “Right, but there is no other community around MIRI. This really is the closest thing.”
So to me that felt like a void — the social support for this particular thing was missing. But then the other thing that I think surprised us in the wake of the book launch is, well, it got support. Members of Congress took up the mantle, right? Bernie Sanders in the Senate — this is his big issue right now. So it did kind of just reach outside the community. The initial community right now seems like they’re just not a coherent supportive community of the key message. But the book did pretty well in the mainstream, right?
Holly 00:21:11
Yeah. So I’m ready to just embrace that new audience. Polls show that they’re basically down. I think they just probably don’t know a lot of details. There’s not a lot of social support, and Pause AI US is trying to provide social support around taking a position. Some people have no problem with it. They’re just like, “Yeah, of course we should pause.” But other people feel that it’s too weird to not be supportive of a new technology or something like that. So you need a social group to make that more normal.
Holly Supports the Government’s Fable Crackdown
Liron 00:21:43
Right. Okay. So that was If Anyone Builds It, Everyone Dies. What do you think of current events like the Fable situation? Have you been following that closely?
Holly 00:21:53
Yes. I think that’s good momentum. I definitely have seen the reaction from industry and industry-associated people — it clearly shows that they think it’s a turning of the tide for them. And a lot of people who I think just kind of naively, innocently believed the story that there’s nothing you can do to stop technology are kind of like, “Oh, okay. As soon as the White House wanted to do it,
Holly 00:22:18
they could.” Basically. So the sense of “this is possible” has risen a lot. And then what’s really going on in a lot of people’s minds when you talk about politics or morality or what’s the right thing is, “Who’s gonna win?” And this kind of shows proof of concept that the government can win this.
The issue with Anthropic and the Department of War — I think this was really poorly reported. I think not many people could decode what was going on. What
Holly 00:22:52
I read under that was Anthropic thinks they should be in control of Claude even when Claude is operating for the US military. They should be in control instead of elected officials. A lot of people think, especially unfortunately in the technical safety community, that Anthropic is being victimized by the US government, and they talk about that with the Fable freeze. But I think Anthropic
Holly 00:23:15
tries to kind of step to the authority of the government, and OpenAI has made it very clear that they’re not gonna do that. They do a lot of signals that they’re gonna be deferential. It’s not the perfect way to regulate AI models, of course, to do it through an executive order through the White House, but I do appreciate the elected government finally seeing what a threat it is. These companies are going to become very powerful, and they need to make sure that they can’t
Holly 00:23:46
upend their power. They need to recognize the threat that this technology is. Not only a takeover possibly in the future with a more powerful Claude, but they’re just gaining a lot of influence. They’re getting integrated into major systems, including the US military.
Holly 00:24:07
If it’s true that the US government or the administration is reacting particularly to Anthropic — I kinda think that’s justified, and I’m really glad to see that too.
Liron 00:24:18
Okay. Well, given that — I agree that the government has some good points here. I think their strongest case is we know AI’s dangerous, Anthropic says it themselves, so shouldn’t we exercise a ton of caution, and can’t we just pause if there’s the slightest hint that we could be cautious? I think you can steel man that case. That said, would you say that the US government, as it often does, has been treating this like a clown show?
Holly 00:24:43
I’m of two minds because on the one hand, yes, of course, the US government should have recognized the potential danger of this much earlier. They should have listened and investigated themselves the possibility of danger. We should have had super competent deep state caring about this, and it’s taken a long time for any appropriate level of concern to emerge, and it’s very strange for it to come through in the form of an executive order through the White House.
But I also feel that the government should treat the AI industry with impunity, and I’m glad that they did it. I’m very
Holly 00:25:26
upset to hear people complain that Anthropic was treated unfairly, because that makes me think they’ve all got a wrong idea of who’s boss here and what is important here. If it’s as dangerous as Anthropic says it is, it shouldn’t be released. End of story. And I guess Anthropic’s story is, “But actually we’re the good ones doing it and everybody else should be stopped first and then we’ll stop.” But I’m glad to see the government calling their bluff in any way.
If it’s so dangerous, then don’t release it. It’s a kind of bizarro way to do it through export controls or saying that foreign nationals can’t use it. But I believe the intent was to make it un-released.
Liron 00:26:14
I mean, I don’t think Anthropic would’ve said that it’s too dangerous to release, right? That wasn’t their position. So do you personally think that it shouldn’t?
Holly 00:26:21
Well, they said Mythos was.
Holly 00:26:24
They said Mythos was and then they said, “Oh, but API guardrails should be enough.” And then jailbreaks are shown, which all of the major releases have universal jailbreaks. But I don’t feel — why would you feel the least bit sad for the person getting punished who did a bad — punished? They’re not getting to release a dangerous model. For somebody being stopped from doing something dangerous and reckless that they say is dangerous and reckless to anyone who will listen, I don’t feel bad at all.
I’m super surprised by any safety-minded person taking the side of Anthropic on this. If there are complaints, what I wanna hear are complaints that Codex 5556 should be treated the same way.
Liron 00:27:12
Well, I think the most powerful objection is just that, look, we’re not on the brink of disaster by running Fable. The computer security that it’s going to find is not going to destroy civilization. As far as I can tell, it is going to be net good. Defense is going to beat offense, especially when they gave defense a head start pretty effectively. So I think it’s important to distinguish that Fable per se is not the problem. I don’t think Anthropic is pretending Fable is less dangerous than it is. I think they’ve got a reasonable estimate of the danger. I think we should distinguish that from the race to the real superintelligence that’s still in the future. Don’t you think that’s an important distinction?
Holly 00:27:53
I think that proliferation now is an issue. It’s not as big as the potential to make the superintelligence that then decides to take some kind of sharp left turn. But I think it’s a pretty big issue. I don’t think it’s nothing, and I think Anthropic is very cavalier. At the very least, they deserve to be punished for the way that they talk about this stuff. If they really thought it was as dangerous as they said, they should never have released it. They should only have used it for hardening. There should have been some kind of public-private partnership over the use of it and not just their own thing.
None of that made sense, and then to be like, “Oh, but I made it, I nerfed it a little, so we should be able to distribute it and it’s not the worst problem facing us.” Why would they want to cause any problem? I think the complacency here is crazy. Why would they want to add to the problem at all? They don’t have to do that. Just to be the best frontier model producer, they don’t have to release models as frequently.
Liron 00:29:00
There’s the shuffleboard effect here, right, where when you look back at all the AI progress that we’ve made to date — I think you disagreed with me on this — but I think that if we could freeze AI today as opposed to rolling it back three years, I see a huge net value in keeping what we’ve got today versus rolling it back. And yet I think every time we try to push the shuffleboard piece forward, we’re risking going over and losing all our points.
Another analogy I use is Icarus. Flying closer to the sun is awesome, but then eventually your wings burn off. But when you look in retrospect at all the gains we’ve made — and I would actually count Fable as one of the gains that’s still giving us positive points — I think you gotta pay some respect to the value of the gains.
Holly 00:29:44
So you mean once Fable was released, it should’ve been grandfathered in?
Liron 00:29:49
Well, you’re acting like it’s kinda costless to roll it back, like obviously we should roll it back. But in my mind, okay, you made this reckless gamble, but it seems like it’s paid off. Do we really wanna throw away the payoff of the gamble?
Holly 00:30:05
How do we know that they didn’t intervene just in time? At every time point, there’s a chance for somebody to do a jailbreak, use it to make a bio attack — it continues to be dangerous. I’m concerned about serious terrorism. We can have even worse psychological effects and fallout from a more and more powerful model.
I’m not gonna name this person, but there was a person who’s a prominent technical safety researcher who pretty clearly was confessing to having LLM psychosis for Fable and not for any previous model, and saying — actually, the person supported the rollback of Fable because he thought that things done for safety were more important. He intellectually knew this, but he was talking about how he was so productive on Fable though. He was taking out his laptop on dates. He was not sleeping at night because he was working on Fable. He was describing mania, and that kind of effect just goes up and up—
Liron 00:31:07
Yeah.
Holly 00:31:07
—with more and more powerful models.
Liron 00:31:09
Okay. For sure. So you’ve — I mean, you’re convincing me that maybe Fable is actually super dangerous and we should be more careful with it. It just seems to me like at some point between 2022 and 2026, if not going out to 2028, I’m not sure where the boundary is, but at some point after 2022, there’s a point where the net value is hugely above 2022.
I could tell you personally, even if I had to stay at the March 1st, 2026 version of AI — when Claude Code really started working — boy, the difference between that and 2022 in my life is pretty huge. I would love to just stay at that point. Even though it was reckless to get to that point. Don’t you think that’s a pretty big gap, the utility of AI in early 2026 versus 2022?
Holly 00:31:53
But what does that mean about what we do going forward? Isn’t it the same? It’s still reckless to take the risk. You play Russian roulette 50 times and you didn’t die — that’s not impossible, but—
Liron 00:32:07
Yeah. I’m just saying that their behavior isn’t surprising, arguing for the value of the gains that they’ve already achieved. So I guess we’re just debating whether the new gains — whether we’re even sure that they’re gains, or if they’ve already crossed into this area where they’re net loss. I’m not sure about that. I am sure that once we get the FOOM, once we get the self-improvement to superintelligence, we’re very likely to lose control, and I hope we never get there, and each step toward it is extremely reckless. I’m with you there.
Holly 00:32:34
Mm.
Liron 00:32:34
But maybe I’m not with you on this idea, like, “Well, we already gambled and won. Maybe we should make sure to harvest our winnings.”
Holly 00:32:39
I don’t feel that we already gambled and won just because in the first couple days of a release nothing bad happened. I can see lots more bad happening. And then I separately feel that it’s very important for the AI industry to be in its place and to be clearly subject to the law.
I like that too. I would rather have the government firmly in charge. I think that could be extremely pivotal. I don’t think if we keep letting ourselves get close to the edge and go a little further and a little further — why would you ever turn around at that point? I feel like the way that we learn to turn around is by turning around now. It’s not by saying, “Oh, well, when the danger’s really great enough, I will.”
Liron 00:33:22
I agree that we should get used to basically having a bummer, right? Like, oh, it’s a bummer.
Holly 00:33:27
Yeah.
Liron 00:33:27
We can’t try Fable. It seems safe. Too bad. I’m cutting you off, basically. You can’t have another drink. You’re doing these reckless drinks. Too bad. Even if you would’ve been able to drive home, I’m cutting you off. That’s the responsible thing to do. Okay. Well said.
Beef: Dean Ball (OpenAI)
Liron 00:33:47
Let’s get into various beefs that I’ve seen you have on Twitter over the last few weeks and months.
You’ve been pretty unique in your willingness to have these open beefs with people. You’ve kind of been the pioneer of that. And I think I said the last time we talked, I don’t personally follow you 100% in all these beefs, but more often than not, there’s a good kernel of truth in the beefs that you’re having that nobody else dared to say or even let themselves think.
Maybe you’ll have a beef with somebody where you say 10 things, and I just find two or three things, and I’m like, “Oh, whoa, that’s actually a good point,” and I don’t see anybody else making those two or three points. So I think it would be productive on the show to go over some of the beefs, and I’ll tell you some of where I agree or disagree, or I’ll just let you say it so you can move the Overton window and help push the envelope and get other people thinking that maybe you have a point.
Holly 00:34:35
Let’s do it.
Liron 00:34:35
All right. So I think this is the most recent beef. You were sparring with Dean Ball on Twitter. Dean Ball, friend of the show. He debated Max Tegmark a few months ago, and he was helping write the US AI safety policy, and then he was at Foundation for American Innovation. He was at that institute, and recently he just got hired at OpenAI, I guess, to work on policy there, and you kinda came at him on Twitter, right? What was your objection recently?
Holly 00:35:03
Oh my God, what even was it? Okay. I just call him like I see him. Okay.
Liron 00:35:09
Yeah, yeah.
Holly 00:35:09
I’m trying not to go on Twitter, so I also haven’t — and I don’t take replies anymore, so it doesn’t—
Liron 00:35:16
Ah.
Holly 00:35:16
—keep the thing fresh.
Liron 00:35:17
Okay, you don’t take replies. What’s your thinking behind that?
Holly 00:35:21
I just get too heated up on Twitter, and it brings me down. I was spending too much time on it. I kinda imagined one day that I was in a classroom, teaching preschool, and I was having to pat every kid on the head who wanted something. That’s kinda what it is.
Liron 00:35:39
Sure, sure.
Liron 00:35:49
Yeah. It’s kind of interesting when people quote because quoting is a lot like replying, and you can see it, and more people see it than if they reply. So what do you think about that distinction? Quoting is okay, but not replying?
Holly 00:36:16
I’d rather they just make their own statement about it rather than have theirs appended to what I said. When there’s a reply that’s kind of hanging open, it’s sort of like ball’s in your court. But with a quote, it’s just its own statement and I don’t have to say anything.
Liron 00:36:37
Yeah.
Holly 00:36:37
I could if I wanted. Occasionally, I do get baited into replying to the quote replies. But I’m trying to just be disciplined about this.
Liron 00:36:47
Yeah, no, for sure. It’s important just because when you’ve got these unread comments, it just feels like it’s gonna chop up your day. I feel you. All right, so I’m glad you’re finding a pattern that works for you.
Getting to Dean Ball — I’ve got a quote from all the way back from April, so before he recently joined OpenAI. You tweeted on April 4th a clip of him from Win Win with Liv Berry.
Dean Ball 00:37:09
Do the people broadly, you know, also the labs and stuff — is the community that’s out here, if I have to pick the trade-off between a usurpation of power by this group of people from the US government versus the US government sort of taking over the world, which trade-off do you pick? And ultimately I side essentially with the West Coast. I ultimately side with the... And I think that we will build new healthier institutions.
Liron 00:37:44
Your comment was, “Dean Ball supports usurpation of power from the US government by the AI companies and the West Coast. He’s talking about a coup, literally uses the word usurpation. See for yourself. I hope this traitor never plans to work in government again.” You stand by that?
Holly 00:38:00
Absolutely. What he described was how much better Anthropic would be at running the world than the government, and he used the word usurpation of power.
Holly 00:38:14
He might think that’s unfair, and a lot of people who talk like that might think, “No, but I don’t really — I’m not talking about a coup.” But they do openly believe and tell each other that Anthropic would do a better job of running the country than elected officials. He might have the cognitive dissonance of not believing that he’s talking about a coup, but he also then will talk about humanity’s final flowering and things like that with AI. He says he basically believes that AI is gonna usher in this new age and humanity is not gonna be in charge anymore. Does he think the government’s gonna be in charge anymore? He’s trying to bring about a state where
Holly 00:38:52
the government is couped.
Liron 00:38:53
This particular comment — you may have gone pretty harsh on what he was trying to say, from my perspective, because it seems like he’s talking about private versus public power. The Constitution of the United States is saying those powers not afforded to the government, we wanna let the people keep them. Maybe he’s saying, “I like it when the government doesn’t have certain powers,” and then it goes down to private companies, and that tends to work out better.
To steel man, ‘cause I don’t even know, maybe I agree with him to some degree — certainly there was talk, remember when Mondani was like, “We’re gonna have government-run grocery stores,” and I’m like, isn’t that a case where private companies can do the grocery store better? So maybe Dean Ball’s just talking about that.
Holly 00:39:33
I don’t think so. He says usurpation and then kind of laughs. Like, “I said it” — all right, that’s really what it is. I think he knows that that is rightfully the government’s power. He’s not saying that according to the Constitution, actually it is the private sector’s thing, or the government should delegate this to the private sector and they’ll work together. He sees it as a competition, and he sees it as the government’s power being taken away. You just wouldn’t use the word usurp otherwise.
Liron 00:40:07
Okay. And then things got more heated recently because he announced he’s joining OpenAI, and you had some choice words for him. I remember one of the digs you had at him was, “Man, first of all, you don’t even get to join your first choice, which is Anthropic.” Unpack that.
Holly 00:40:24
Oh, he’s a big Anthropic fan. He talks about it all the time. He clearly thought Anthropic was the best. I don’t have receipts in front of me, but Anthropic was his favorite. He would always take up for them in disputes. I think he has a lot of friends in the rationality EA Berkeley East crew that I left. And I think Dean feels very included and feted among that group, coming from a conservative think tank group, and he wants to join that. So I think it has to do with his friends being at Anthropic.
Liron 00:41:02
This is where maybe you and I diverge, right? ‘Cause I said I don’t agree with 100% of your tweets. I just — I think you gotta draw a distinction between when you’re dishing out substantiated criticism, which I think you do a lot of and it’s great, but then when you say, “Yeah, Anthropic was his first choice and he couldn’t get it,” you don’t even know that, right? So you’re speculating.
Holly 00:41:25
Yeah, I’m down to speculate more than you’re supposed to in debate, for instance.
Liron 00:41:30
Okay.
Holly 00:41:30
Here’s the thing — this is something that I do think is important for people to understand about reading my tweets, and often people from debate backgrounds are really concerned about discourse norms and stuff and don’t get this.
Liron 00:41:43
Which we are on the show. We’re concerned about discourse norms on this show for the record.
Holly 00:41:47
Sure, yeah. And I’m capable of interacting that way — I can enter many registers. But when I say something like I did to Dean Ball, it’s not — it is for the audience, but the way that I say it is I’m pretty sure that I’m right about what’s really going on in his head, and only he’s going to know if I’m right or not. He can claim and he can say, and yeah, it’s not fair. It’s not something that I was allowed to enter the record about him. I just say it ‘cause I think it.
Really what I think about Dean is that he thinks he’s a phony on the inside, and I do poke at that, ‘cause I think he’s quite pretentious in the way that he speaks. I really can’t stand obscurantist, flowery language, people who never say anything, and that’s what Dean does. He doesn’t say anything. I think he mixes the food around on his plate like someone with an eating disorder would to make it look like they ate the food. That’s what he does — he makes a bunch of wry observations. He acts like he doesn’t really care. He has kind of a William F. Buckley affect that he puts on.
And then he doesn’t really say anything. His angle is to be sophisticated or seem sophisticated by creating what is strategic cover for other people to do bad and selfish stuff that is not responsible. And so
Holly 00:43:13
I find that profoundly annoying, and I do poke at Dean. And I hope, when I poke at Dean, the emperor knows he’s not wearing clothes. But he’s hoping against hope that he really is wearing clothes because that’s what he was told. And it’s not for me really about the audience knowing that he’s not wearing clothes. It’s like I’m trying to get to his awareness of that.
Liron 00:43:37
I just feel like you’re not gonna get leverage with a “the emperor isn’t wearing clothes” attack because couldn’t you say that about President Trump, that he’s the emperor with no clothes? He never understood politics, you could argue. I mean, some people say that. I’m not the most opinionated guy on Trump. But if you can be a two-term president and still have these same accusations leveled against you, what’s the point?
Holly 00:44:00
I mean, Dean can get picked by people too. He can get what he actually wants. He really has this job at OpenAI. I’m not saying that’s fake because I feel like he’s not really saying anything. But I can make him have some introspection. And I can take away—
Liron 00:44:16
Yeah.
Holly 00:44:16
—a lot of supporters who aren’t really paying attention either.
Liron 00:44:19
Yeah, yeah, yeah. Well, I think the boundary that I like to draw in public conversation is if you’re gonna psychoanalyze somebody and be like, “I bet that he doesn’t really believe what he’s saying,” that’s your best guess. Maybe you’re right. Maybe you’re good at psychoanalysis. But you gotta clearly demarcate it as such. And I personally feel like that could be fun in a private conversation. But I feel like it’s a dick move to be psychoanalyzing somebody in public when—
Holly 00:44:48
Why?
Liron 00:44:48
—even if there’s a 1% chance that you’re wrong, I think that it behooves us to give people the benefit of the doubt in public discourse.
Holly 00:44:55
No. Absolutely not. The benefit of the doubt is what the AI industry depends on. “Oh, but who’s died?” Actually, people have died. But there’s, “Oh, but we shouldn’t judge this extremely dangerous behavior until,” what? Until it’s too late? No. If people don’t want to stand up to a bully, or they don’t wanna miss out, or they’re afraid that they are stupid and they’re gonna be missing out on the next big thing, then the benefit of the doubt is a gift because it just means they don’t have to do anything.
But Dean is not ever gonna just say, “Yeah, I’m lying. I put on a big act so that I can feel important.” Obviously, he’s never going to agree to that. So I guess in debate norms, you can just never say that. But it can be a thing that everybody knows. Or it can be a thing that’s a very reasonable opinion. I see it this way. If you outlaw saying things because they’re not fully known or because the person wouldn’t describe themselves that way, I call this toxic charity. We do need to discuss things that are reasonably inferred. There’s always this hiding place in this kind of debate culture for—
you can hide whatever you want behind there, and it’s just sort of you’re breaking the rules if you notice. But people misrepresent themselves. People lie. People are bad. Big companies misrepresent what they’re really doing, what their real intentions are.
Liron 00:46:27
The question is where do you draw the line on giving versus not giving benefit of the doubt? I think when it comes to somebody’s motives, people’s own motives aren’t even gonna be fully clear to them. And I think there’s many cases in life where you’re gonna find two people who put on very similar demeanors.
So for example, your particular claim that Dean Ball—you kinda see him as low confidence and seeking validation. That’s your psychoanalysis of Dean Ball, okay? There’s going to be somebody in the world—let’s say you’re correct. There’s going to be somebody in the world who has very similar behaviors to Dean Ball, and he’s actually going to be extremely overconfident in his internal mental state.
Holly 00:47:06
What you’re saying here is sort of like it’s just actually not okay to be wrong about what somebody thinks about themselves, and this is really their sovereign territory. And I just think it isn’t because what they do, they’re doing in the rest of the world, and they’re motivated by those motives.
But in the real world where I think the primary thing that is driving Dean’s behavior has nothing to do with facts we disagree or agree about—I think it is those motives. And so I think it’s very important to surface that possibility for people who are listening to him thinking, “He’s just an expert. Everybody thinks he’s so sharp.” It’s like, why do people think he’s sharp?
Point to—they think he’s sharp because of the way he talks, and they think he’s sharp because of an affectation. And if you point out that this is an affectation, I think that affects what he says. And you seem to think, well, but what if you were wrong about that? That would be devastating, horrible. What if somebody said something that wasn’t true about you? I mean, yeah, what if? We don’t actually know if it’s true or not. That’s the whole premise.
Liron 00:48:14
You think it’s productive when you’re publicly giving your psychoanalysis of people like this because you’re giving other people permission or you’re seeding the idea that they shouldn’t trust him so much because he’s being a pseudo-intellectual?
Holly 00:48:29
They can decide when they watch him. Also, he can decide if he wants to keep doing that, knowing that it’s transparent to at least some people. That’s my intent.
Liron 00:48:40
I mean, maybe you’re right. Who knows? But I actually think that there’s a good chance that he just is pretty cocky. In which case, how does it help when you’re accusing him of secretly not having that much confidence?
Holly 00:48:51
He’s not. The first time we scrapped, he was incredibly triggered by it. I actually didn’t think that it would hurt his feelings. I can’t even remember what I said first, but I was surprised by how clearly he was made of glass. He posted something which I never saw because he deleted it before I could see it, apparently because comments were telling him it was too severe.
And then he posted something else saying that he deleted it because he heard that I was mentally ill, and it really wouldn’t be fair. And then there were a bunch of comments on that—there were seven comments saying, “You just call someone—you just make an accusation that they’re mentally ill?” And he had hidden all of them.
Liron 00:49:35
Yeah.
Holly 00:49:35
I just think he was pretty triggered. And I honestly didn’t even mean to trigger him that much, but after he did that, it was just kinda like gloves are off. This is what I think. And yeah, I’m not operating by anybody else’s standard of what you’re allowed to say. I’m operating by my own integrity.
Liron 00:49:52
From my perspective, it sounds like you’re performing two functions, and they’re both unique. So the first function is you’re bringing up the idea of, “Hey, you’re going off to join this AI company. Have you considered that that is morally bad? You’re just doing something horrible right now,” which is a fair point from the perspective of moral reasoning, even without turning it into a personal attack. Because it’s like, “Hey, I don’t know what’s going on inside your head, but do you realize that this action that you’re taking could actually be a disaster and you should rethink it?”
You make points like that, but then the second thing you do is what we just discussed, which is, “This is my best psychoanalysis of what’s going on in your head, and maybe you’re covering for this insecurity, and all you really want is to belong.” You say that about some people, and I’m sure you have some hits. If you take 10 shots, I’m sure you’re at least gonna get one or two hits. You’re not gonna go 0 for 10. But I’m sure you’re gonna have some misses too.
And I feel like you’re going out on a limb with this psychoanalysis, and then it’s making other people who get triggered by the psychoanalysis part of it—and sometimes rightfully so if they doubt your interpretation of events—then they’re missing this objective value that you’re adding, which is I actually think you’re pushing the envelope. I actually think you’re moving the Overton window in a productive direction when you’re just doing this kind of moral reasoning, taking these bold stances even before you get to the personal attack.
Holly 00:51:09
I used to be very successful in that world and talking and thinking that way. It’s just there’s more at stake. There’s a moral issue where I can’t just wait for the world to prove me right. I have to deal with what I have access to now, which is people who are letting themselves be deluded and be part of this, who are giving excuses which other people can’t see through. I have to call them on it given the tools that I have.
A lot of what I feel I have is just making people—kind of hitting people in a sensitive place and making them look inside. And I think if they’re really good people, they’re going to care about what actually happens. They’re going to care about that more than that I insulted them. And if I approach them with just, “Have you considered this?” humbly, XOXO, then they will say, “Yes, I have. Oh, I’m such a sophisticated reasoner. Actually, I’ve been thinking about AI safety for 10 years before I joined Anthropic. Here’s my whole narrative from Dario about how we have to build AI, actually, and it has to be this way.”
My best chance to shake people out of it is to say, “Bullshit. You know what’s really going on here. You know you wouldn’t consider doing something that was less exciting than taking this job from Anthropic.” And with AI danger, as we discussed, it could just very quickly teeter over the edge, and then we’re gone.
So when people are making decisions or talking about their justifications for just continuing to get closer to the edge and why nobody should stop them and why they actually have a secret plan—that just lets them get closer—I can’t just stand back and let reality be the judge on that because a lot of people are gonna pay for that.
Beef: Eliezer Yudkowsky
Liron 00:53:09
Okay. All right. Well, that’s it. We got the meta commentary out of the way about your approach. So let’s go object level. Let’s get the specific takes then. Moving on to Eliezer Yudkowsky. What’s your take on Eliezer?
Holly 00:53:22
Ugh. Eliezer really upset me recently and kinda ended our relationship. We had a long conversation in which he was clearly in a bad mood, but he basically didn’t like me calling out AI company employees, and he compared it a bunch of times to executing prisoners of war for some reason. And I was like, “Dude, I’m making people feel shame with my words from a distance for something they are in fact doing. They’re not my prisoners. They actually are building AI.”
He just found—he just didn’t like, sort of like a woke dynamic, he said, of punishing people who don’t go along and enforce your thing for you.
And I don’t know, man. I was like, “What did I do to make you so angry?” And he said, “I’m afraid what you will do if you have power in my movement.”
And so I just felt like, I don’t know, maybe it’s a little “get off my turf” kinda thing. And MIRI has a bunch of weird stuff going on and things that look really weird about what they’re doing and what the people behind it do. They have never—
even when I gave a bunch of free publicity to the book launch, they’ve never supported us. A very common thing to happen would be like I would write an open letter, I would send it to them, I’d be like, “Okay, I know you guys are pretty choosy about your words and what you’ll sign onto, so you can look at it first, and then you can change it, and then we’ll have other people sign that wording.” And they would just get back to me in two months and be like, “We didn’t like this word.” And I’m like, “It’s too late. We already did the open letter.” The thing happened, the moment passed.
So that was kind of my relationship with MIRI this whole time. I appreciate them a lot. It’s a funky thing because we’re here because of Eliezer, and I do really appreciate that Eliezer has updated as much as he has, because I know it was painful for his identity—to update away from libertarianism in some ways, to update away from the singularity happening in a good way, to update towards working with normies, stuff that makes him pretty uncomfortable. So I do appreciate that, but he still drags his feet too much on what we have to do. He knows that it’s urgent, but he just
kinda can’t bring himself to do more populist stuff. He wants to only be around who he would call smart nerds. He wants to only talk in the way that you would talk on LessWrong. What I’m doing kinda represents that encroaching of populism and normies into his space.
Beef: Zvi Mowshowitz
Liron 00:56:14
All right, let’s move on to Zvi Mowshowitz. I bring him up because Zvi has a lot of respect, rightly so. I mention him a lot on the show because he’s a good analyst. He’s evaluating all these things that are happening in the AI space. I find his analysis productive and detailed and consistent, and I’m glad he’s working for the analysis cause. He’s mentioned he has a P(doom) of 70%.
A lot of the stuff he says I’m on the same page with, and he’s been a longtime rationalist, LessWronger. I’m bringing him up because you had a beef with him a few months ago on Twitter where you called him out for, I guess it was mostly boosting Anthropic rather than advancing real safety. What’s your take?
Holly 00:56:55
Yeah. Well, first I wanna ask you, what is your take on Zvi’s safety content? And how much would you say he’s doing safety content versus analysis and commentary on the models?
Liron 00:57:07
Yeah. I mean, I think his main project right now is the weekly big AI roundups, and it’s really more than weekly. It’s multiple in-depth analyses—okay, Fable came out, what’s the model card? It seems to be the best single source of analysis for just what’s going on in the AI space. I think he explicitly goes for the criterion of, “Hey, if you’re just reading this, you can feel free not to read other sources because I’m rounding everything up.” And as somebody who does read a handful of other sources, I think he does pretty well, and sometimes I allow myself not to read other sources because he exists.
Holly 00:57:36
Is this safety work, or is this actually kind of capabilities and hype?
Liron 00:57:42
I think he would explicitly say no, it’s not safety work. However, it has a point of view, and he is quite explicit. He’s mentioned on podcasts multiple times that his P(doom) is something like 70%. And in his analysis, he’s also pretty explicit—”Yeah, this is extremely dangerous,” or “This is how we all die,” or “That’s bad,” or the Jan LeCuns of the world with their weak arguments. He points it out as, “Yeah, this is a terrible argument. I’m not even gonna be covering this argument.”
So he’s very much writing from a point of view, and I think the combination of somebody who has that consistency and the legitimacy of being a trusted arbiter, who also is very much moving the Overton window to the high P(doom) viewpoint and also not backing away from that—I mean, honestly, that’s what I’m doing here on Doom Debates, right? I’m trying to run a high quality debate forum, but I also very much have a point of view. I actually think that the commitment to simultaneously doing both is an underrated type of format to have.
Holly 00:58:40
I think you’re giving him a pass in a couple of ways. I think he has good analysis, as far as I know. I don’t feel the need to be that in the weeds on the models. I don’t think that’s necessary for Pause AI’s intervention. But I think just having a high P(doom) is really not enough. I mean, the AI company CEOs had a high P(doom), right?
Liron 00:59:03
Yeah.
Holly 00:59:03
What I think happens when you say, “Yeah, this is gonna kill everybody,” but all you do is kind of gush—I mean, on Twitter, Zvi gushes about how fun stuff is, or he’s upset that Claude is down for maintenance, and he talks about how much he needs Claude. And it’s just very clear that he loves Claude. Claude is cool. He has to be looking at Claude all the time.
And to say you have a high P(doom), but also this is so cool and I spend all my time working on it, and it’s cool that I love it and it’s my favorite thing, I think is spreading a very harmful idea. That’s not pushing the Overton window where we need it, which is that we take appropriate action. We actually view these things as negative as if we really believed that they were gonna kill us. We actually don’t feel warm toward the companies that make them.
Liron 00:59:56
Yeah.
Holly 00:59:56
This is—I mean, what Zvi does is informal, but it’s the definition of regulatory capture. The original definition of regulatory capture is the regulator of the bank has an office in the bank, and really all of his friends are at the bank, and he kind of considers that he works at the bank instead of at the government.
He—his job, and maybe what he’d say, is that he’s there to prevent the worst problems from happening at the bank, and that’s so good. But what he sees himself as is working with the bank employees to do that, rather than with the government as somebody who’s coming in and regulating the bank.
And that’s what I see with Zvi. One of our last exchanges—he said it was right after Anthropic had dropped their second RSP. So they had dropped their responsible scaling policy, which was supposed to—which is full of if-then commitments, and that was supposed to be why we trusted them and why they’re good, because they’re gonna keep those commitments. And then they just drop them and revise them.
And Zvi had—it was also during the time of the first Anthropic versus Department of War thing. And I said to Zvi, he was so on Anthropic’s side. He was like, “This is crazy. The government’s being so bad to them.” And I felt that he got that very wrong because of his partiality to Anthropic. And I was saying that, and I said, “When are you gonna write about them dropping RSP 2 if that’s really your focus?” And he goes, “When they have a minute to breathe.”
So he thinks you’re friends, and you have to give them a chance to read it, and you have to not pile on because they’re going through a lot right now. What? I mean, that’s—he wouldn’t think that about the government. He just thinks he’s on their team. Maybe they read it. I don’t know. It is a big newsletter, but—
Liron 01:01:51
Yeah, I think it’s fair to say they do read it, and he has some influence. I don’t know. I mean, but I’m starting to think that I’m guilty here too, right? Because I like to think that Doom Debates is the kind of show that somebody at Anthropic would wanna watch. Is that so bad?
Holly 01:02:10
Well, I mean, not because they would get fellatio, right?
Liron 01:02:15
Yeah. Right, right, right.
Holly 01:02:17
I mean—
Liron 01:02:17
But do you really think that Zvi’s newsletter is consistently positive about Anthropic and doesn’t criticize them?
Holly 01:02:24
Yes. I think we’re all just being frog boiled on this particular thing. People just genuflect constantly to Anthropic, and what’s seen as being fair and balanced is really extremely partial to Anthropic. And that’s just a normal power dynamic that takes hold when there’s—Anthropic to people in the AI safety community, it feels like they have a seat at the table and we’re all friends, but obviously this friend is the most powerful and cool and gives you the most options, and it’s the most important to be in tight with them.
And nobody wants to acknowledge that that’s what’s happening because they all wanna think that they really deserve their place here, and that they’re not just depending on this big daddy to take care of them. But that is what it is.
Liron 01:03:14
I mean, I think, I just remember when you had the beef with Zvi on Twitter. He later posted, “Okay, Holly, I’m reaching my limit here,” right? Because he covers you in his newsletter too, and I know you don’t wanna be captured by him. But he was saying that you’re just making his life hard because he doesn’t know how to respond to the level of criticism that you’re giving him because he thinks you’re going over the top with him. But you stand behind a pretty high level of criticism, correct?
Holly 01:03:40
Yeah, actually, that’s not—I certainly didn’t call that right. I did not think he would be so sensitive. A part of me, when he says you really hurt my feelings—this isn’t right, but—
I get so much harsh criticism. It’s not like I’m dishing out something I don’t get.
Liron 01:03:59
Just to be clear, I don’t know if he said you hurt my feelings. That’s not what I remember him saying. I think I just remember him saying, “This is—you know, I just feel like you’re criticizing me. The volume of the criticism just seems high for my role in things.”
Holly 01:04:11
It was three tweets. I don’t even—no, somebody reported to me that he said that too about it. He is a pretty big celebrity in that world, so maybe people just treat him really nicely. But I was really surprised that he would say that that bothered him and affected him so much. Does he never get criticized?
I still maintain I was really not trying to go over the top with Zvi at all. I was saying he gushes about Anthropic, and I think if it hurts his feelings so much—I’m gonna psychologize here—I think it’s probably because there’s some truth to it. In order for him to not gush about Anthropic, it might affect how he feels about his whole line of work, and he might feel like, I don’t know, maybe he’s too dependent on AI or—
I don’t know. I’m surprised he took it so personally.
Liron 01:05:00
What you wanna see from his work is just a more consistent filling in of the missing mood of reminding people that, “And by the way, Anthropic is horrible”—basically make sure to have that tinge in the presentation of this coverage?
Holly 01:05:14
Yeah, I do think, and I’m not trying to say it’s always wrong, but I do think there is something approaching fucky about being super in love with Claude Code and being concerned about AI safety. It is—the effect is gonna be to make you very fond of the thing that you’re also trying to warn other people could in a few more iterations be super dangerous.
And it makes people—the reaction I saw to Fable, it was very negative from the technical safety community when it should’ve been like, “Oh my God, yay. We’re able to do something. Yeah, let’s do it better next time.” But it was just wrong.
People like Zvi are really shoring up a narrative which is very common in those spaces, which is that it’s actually the most rational thing to both be so into the models and to say they’re dangerous. But in fact, the things that you do according to the view that they’re dangerous just get more and more sidelined over time when you’re focused on how cool the models are.
I don’t wanna say that somebody who uses Claude Code—that means they’re bought into the AI company lines. I think plenty of people who use Claude Code still don’t give a shit about the company necessarily. But I think for people who say they’re doing it for safety reasons and then they’re super invested in—the thing they’re tweeting about is the problems they’re having with Claude Code, and the thing they wanna talk about and that kinda shows who they are is that they’re such a power user of Claude Code—
how are they gonna feel towards, okay, maybe the right answer is to just pause? And we saw how they felt toward Fable being paused. There was a lot of, “Mm, they don’t know the technical details. It was actually fine.” Even though we want regulation supposedly.
Liron 01:07:12
Okay, well, I don’t wanna beat the dead horse of just picking on Zvi here. Don’t you think part of the net analysis, though, is to ask, okay, well, counterfactually, what if he didn’t exist?
Holly 01:07:22
What do you think would happen? What is the good thing that he does by telling you—what do you do positively because you know about the models?
Liron 01:07:30
So he does an analysis that’s, besides being high quality, it’s coming from his viewpoint. So he’s saying, “Hey, if, like me, you have a high P(doom), you might think this about these developments.” So for example, he calls out when there’s cases of training on the chain of thought. You’re not supposed to do that because then you cause the chain of thought to be deceptive.
Holly 01:07:50
But he wanted to use the model that had been trained on chain of thought. I mean, didn’t he?
Liron 01:07:55
So you’re saying he only wrote that because he wanted a more usable model in the short term?
Holly 01:08:00
No, I think he believes that this is about a longer term thing, but he’s not—he’s weakening his muscle of actually acting accordingly to that, and he’s kind of making it so that people can—
Liron 01:08:11
Yeah.
Holly 01:08:11
—have their cake and eat it too. They can talk about safety, and they can say, “Oh man, this would be a big deal,” but then everything they actually do is towards paying the AI company and making it more acceptable for people who care about this to be culturally favorable toward the AI companies and what’s happening.
He didn’t think, “This is just unacceptable to train on chain of thought. I won’t use this model. That’s terrible. That’s dangerous.” He’s like, “Oh, man, this is my critique,
but I’m just gonna use whatever they put out and not do anything else to stop it.”
Liron 01:08:51
All right. So counterfactually, you’re saying we just get another analyst who’d have a slightly worse point of view, but it wouldn’t matter that much because the only things that make a big impact are people explicitly saying that we should pause AI. Is that kinda your perspective?
Holly 01:09:06
I think basically the thing we need more of is people willing to take action, or willing to say, “I’m not going to use this anymore. I’m ready to stop now. I’m practicing turning away from the cliff now, even though maybe I could’ve walked a few more steps.” I think that’s the thing we need to practice, and I don’t see a lot of analysis that would supposedly logically entail more regulation with nothing behind it actually helping.
In the whole time I’ve been doing Pause AI, there’s been this insistence that we need to really know about the models, that’s somehow important, and the evaluations, that’s important. But if that information isn’t good for anything, if it’s not connected to any decisions, then it can almost be worse. It gives the impression that, “Yeah, we got it. We looked at this.” And it’s like, okay, so you looked at it and you found a horrible thing, and then what? Your job is over? You just get onto using the version of Mythos that’s available to you?
It’s not consistent. It’s just, “Don’t worry. I’m on top of this. I looked into this. Here’s all the terrible things wrong with this. Let’s get on with using the models as if nothing changed. I have no plan for what should be done about this. I’m certainly not gonna change my behavior. My behavior, which speaks louder than my words, is gonna say that this is just a fun activity and everybody enjoys playing with these models.”
And it’s not only Zvi who is doing this, but I did criticize him because I felt that he should know better. I was very unimpressed with his reaction, which—I don’t know, it’s for him to say just earnestly, “This was tough on me,” which I did see in his own words—
was, I think, fine and good. But for him to not seem to have introspected on it, and to find it so hard and just be focused on how it was unfair of me to make such a criticism—I’m not perfect. I definitely say a lot of things with more anger than would be necessary. I’m not trying to claim my actions online have been totally calculated for impact.
But yeah, I guess that’s a lot of where the anger comes from. If this is gonna be a problem, if this is something I can’t say and I have to dance around because it would hurt your feelings versus you being part of the problem—why don’t you care more about being part of the problem? Why is that not what motivates you?
Beef: Liron Shapira!
Liron 01:11:40
All right. Well, that’s enough about the Zvi case. I think I get where you’re coming from, and that makes me wanna ask you about another potential beef target here. Let’s make the case against Liron Shapira. I’ll kick it off. I’ll kick it off here, okay?
I use Claude Code. I’ve been known to gush about Claude Code. I’m excited about doing startups using AI to supercharge your startup. I talk about the ways that I use AI on the show pretty frequently, and I even invest in AI stocks like Google, hoping to benefit from what I think is my economic outlook, which is what I call the Icarus graph, right? Because I’m expecting to fly higher and higher for a while and then do a 180 and go down into hell. I’m expecting to profit from the upward sloping part of the Icarus curve, and I’m talking about investing in AI stocks accordingly. So it’s a pretty damning case, right? Definitely not somebody who would only talk about how bad AI is. So how come my time hasn’t come for a Twitter beef?
Holly 01:12:47
I do think it’s playing with fire. I usually read you talking about using AI on the show as an admission, and it’s not like I—I also sometimes use LLMs. I do limit my use a lot. I never, ever let them put words in my mouth. That’s a hard line. I don’t even—I don’t like the feeling of it even.
So I like to disclaim so that nobody’s misled, that it’s not that I would never touch AI. I do think investing in them is really playing with fire. I do think it is morally corrupting to be invested in the bad future in some way.
Liron 01:13:24
How am I at all better than Zvi here? Because all I’m doing is covering AI. I’m just a commentator.
Holly 01:13:32
But you don’t cover releases in detail. It’s not like your programming is about hyping it. It’s about interviews with people. I do think also the debate format sometimes is kinda like, “Ooh, it’s both sides.” It sort of serves to give a boost to the weaker side as if they’re the same.
Liron 01:13:54
Yeah. Do you think you’re biased? Do you think you’re giving me immunity just because we go back a few years?
Holly 01:13:59
I mean, I don’t—it could be that that’s happening. I don’t—
Liron 01:14:02
Okay.
Holly 01:14:03
I mean, I could definitely say other things I disapproved of about you if you’re looking for it. But it’s not like I’m incapable of thinking them. But with Zvi, it’s not even that I really strongly disapproved of Zvi uniquely. I mean, I would say the same thing to you, I feel, about Claude Code especially.
I think we did have some disagreement over this once when I said—I criticized, there was a period after Claude Code came out when a lot of people were describing mania. “I’m not sleeping. I’m working all the time. I’ve got agents running all night. I’ve got this and this and this.” And then I said, “Well, where are the apps?” And you were like, “It is really effective for work.” And I can’t remember what I said.
Liron 01:14:47
Right, right, right.
Holly 01:14:48
But I was like, I feel like I was like, “Sure, I’m sure it can be effective,” but—
Liron 01:14:53
I mean, I think it was important situational awareness to realize that there was real value being created. Don’t you think that the Ed Zitron, Emily Bender crowd—don’t you think that they’re being counterproductive by downplaying how much actual value it’s creating or how much power it has?
Holly 01:15:07
Well, I do think it’s just not true what they’re saying, which is bad. But I don’t think that it’s the problem that—I think the crowd I’m describing, which I suppose includes you—
they’re like, “Yes, P(doom) really high, but also must use these tools,” and then the actions never kind of accord with P(doom) high. I think those people are really wrongly caught up in correcting the liberal narrative that nothing’s happening. What I really see behind that is just culture war. It’s like, is tech powerful? Is the thing you wanna do, your tinker-toy builder thing, is that real or not? Our institutions are real and yours are not.
And I feel we just don’t have this problem in Pause AI. In Pause AI US, I should say. Apparently, in global they want this to be a focus. But in Pause AI US, we have people who have all kinds of concerns, and we just don’t hear people make that claim about AI not being powerful after a while. They might say that at the beginning when they come in, but we just don’t really hear it after a certain point. So I think it’s more of a culture war thing, and I think that’s why it feels like a big threat.
Beef: Nirit Weiss-Blatt (Dr. Techlash) Being “Bad Faith”
Liron 01:16:15
All right. Cool, cool. Let’s talk about Nirit Weisfeld. She goes by DrTechLash on Twitter, and I think her shtick is to embed herself or join a bunch of Discords and stuff, do research, watch videos like this. Hi, Nirit. And do research and see what we’re talking about right here in the in-group of people with a high P(doom) or doomers or rationalists or whatever. And then report on them like, “Oh my God, look, they said this on the Discord.”
And I think you’ve accused her of dangerously scapegoating you because she called you, I think, a violent revolutionary. Or what was the issue there?
Holly 01:16:50
Yeah. So she tried to pin that person who threw a Molotov cocktail at Sam Altman’s gate. She immediately—I mean, clearly this had been discussed and thought about beforehand because it wasn’t just her. Immediately out the gate tried to say that Pause AI’s rhetoric had led to this happening.
She was able to dig up in the Pause AI global Discord server, which is different than the Pause AI US Discord server—but in that server, he had had an account, and I don’t even know actually how she knew that was his account. But it was under the name But Larry and Jihadist. It’s possible he tried to join the US server and was bounced for that name because I’m pretty strict on anything that sounds a little hinky.
And actually, he did receive a warning. He didn’t say anything violent, but he did just say something kinda like, “This is the end,” and he got a warning from that server. Which—so I think they did everything right, but just the fact that he was even in a public Discord server with Pause AI’s name at all was taken as free license to say he was a Pause AI member. He got this from Pause AI.
As far as they know, the global people, he didn’t complete onboarding, so he was never a member in their sense. So yeah, it turned out basically he had gone to a social media platform to talk about pausing AI once or twice. I don’t remember how many comments he had. But she didn’t even know that when she first started accusing Pause AI. She was looking at Pause AI under a microscope to find any kind of connection because she was hoping to find a connection.
Liron 01:18:32
Right. Okay, yeah. So she seems pretty bad faith, and you specifically said, “If DrTechLash inspires someone to hurt me by scapegoating me falsely as a revolutionary, my blood is on her hands. I wonder if you even care if I live or die, Nirit.” But it is a fair point, right? I mean, I did think that her accusation was unfair and—
Holly 01:18:50
I did have people saying—not too much, but I did have comments that were pretty scary. And this goes both ways. You’re saying that Pause AI—they weren’t even saying that Pause AI had made calls to violence, because we haven’t, and we’re extremely stringent about that. She was just saying that saying that AI was dangerous was the thing that could inspire violence.
And so surely, saying that I had directed somebody to commit violence could also inspire violence, and she’s just putting me in the same position that she’s claiming that our programming is putting the AI industry.
Anthropic Took Over EA
Liron 01:19:30
Yeah. I agree, I agree. All right, let’s talk about some Anthropic-affiliated people. So just to review, I think we talked about this in our last conversation, your worldview, which I think has a lot of truth to it, is that Anthropic has kind of come into the community of all these rationalists and people who are early to AI and young people out of the top colleges who got interested in AI, and they kind of became the center of gravity of this community. All the status and money and power and legitimacy is now flowing through them, and you mentioned that their gravitational field has a huge hold on Dean Ball, for instance.
But you think it’s countless people now, and it’s even underrated. And now they’re also the most valuable startup in the world. Their valuation has surpassed a trillion dollars. A lot of people are arguing it should be two trillion. And from your perspective, they’re now even seeing themselves as potentially more powerful than the US government. So you called it early that Anthropic was the Death Star, right?
Holly 01:20:25
I’ve been saying Anthropic final boss for a while now, and they definitely took over EA, effective altruism, in my opinion. Though the people at the top started as effective altruists. Dario Amodei is the number sixty-three signer of the Giving What We Can pledge, which is a thing started in effective altruism. I think Amanda Askell is number sixty-four. Daniela Amodei, Dario’s sister, is married to Holden Karnofsky, who was the first CEO of Open Philanthropy, which is now called Coefficient Giving. And now Holden is also at Anthropic, and basically a lot of people in that world used to deny connections to Anthropic, and now they all work there. And so I’ve seen that—
Liron 01:21:06
Yeah.
Holly 01:21:06
—in my experience, that’s happened a lot. So yeah, the influence is huge, but they have this whole narrative for why—I call this the Anthropic defense, like the Nuremberg defense. So the Nuremberg defense is, “I was just following orders.” The Anthropic defense is, “I had to because of the counterfactual.” And they just get to construct the counterfactual to be whatever they want it to be so that they can go ahead and do it.
They’ll say, “Oh, we wish that we could just not build at all, but we have to build to be safe.” And, “Oh, if we don’t build, then China will build.” And then Trump just goes and talks to Xi Jinping about it and they come away saying, “We probably will work together on guardrails,” all the standard guardrails stuff. So—
Liron 01:21:50
Right.
Holly 01:21:50
—that’s been a little bit—but you can see that the momentum of the whole, “Oh, come on, we gotta do it anyway,” just kind of continues to carry Anthropic. They don’t just say, “Great! Wow, let’s pause because we’re the good guys.”
Holly’s Basilisk: What If Building Superintelligence Becomes a Crime?
Liron 01:22:04
So you mentioned the Nuremberg trials and you made an analogy that “we’re just following orders” is kind of analogous to, “Well, it’s gonna happen anyway, so we’re just trying to improve on the inevitable.” And you introduced this thought experiment on your blog. I know you’re having a big month of blogging. I wanna get to some of your posts. You introduced a thought experiment that you creatively called Holly’s Basilisk. That’s a very inspired name for that. Okay, explain that.
Holly 01:22:30
Well, so Roko’s Basilisk is this thought experiment that what if there was a superintelligence in the future that tortured people for not trying to build it faster in the past? Then we’d pretty much have to build it, right? That was kind of always the—well, that could happen, so that’s sort of a reason that we should build.
I think the reason to compare it to the Nuremberg trials in particular is that the Nazis didn’t break laws by doing what they did. Those laws didn’t exist until after they had done it. So the category of crimes against humanity didn’t exist until the Nuremberg trials.
And the AI companies are just kinda like, “Well, it’s not illegal. We can do anything. We have to.” They can also just dictate reality and say, “We have to. We have to keep pursuing this.” And there’s—so I thought it is kinda like that. It would require a new kind of international trial and a new kind of crime against humanity, but that doesn’t mean it wouldn’t happen. People say, why were the Nazis not ashamed to write down all their dead? Why did they keep records of all that stuff? And it’s because they weren’t ashamed. They thought they were winning. They thought they were doing something that reflected the glory of their civilization.
Liron 01:23:48
Right.
Holly 01:23:48
And—
Liron 01:23:49
Yeah, so just to put out the analogy here, just to catch up the viewers. You’re saying that the Nazis when they were making the concentration camps, sending people to the death camps, killing millions, in their mind, they’re like, “Look, this is just our business. This is how Germany is being run right now. It’s the Third Reich.” And the United States, there’s no crime when we decide to gas a million of our own people—that’s just our business.
And then the United States is like, “Well, we beg to differ.” The Allies at the end of World War II—it’s like actually the fact that you guys did this within your own boundaries or within the boundaries of the countries you were conquering, that’s a crime against humanity. And we’re gonna put you to death for that.
Holly 01:24:24
Some people got put to death. I’m not advocating that, just FYI. But the idea that a lot of—I think people think because they’re at a frontier that there are no laws or there will never be any consequences. But unless they’re successful in killing us immediately, which will also be a kind of punishment to them, I think that they will not be spared. We’re not just gonna get close to the brink and be like, “Oh man, who coulda seen that coming?” There’s gonna be—
Liron 01:24:57
So do you personally—I’m curious how far you’re gonna go with it, right? Because it’s an interesting thought experiment, and I don’t know how far you wanna go, but the extreme way to go is to be like, “I, Holly, am going to be here when society comes to its senses. I’m gonna be one of the ones saying that you guys all have to go to jail,” right? Because you can already say that today.
You could already say, “Hey, today, if you work for Anthropic”—maybe carve out some exceptions if you’re working on computer security or whatever—so you say, “If you’re working at Anthropic today, I will personally vote to put you in jail when we pause AI.” Would you go that far?
Holly 01:25:30
Well, I would want it to be not vigilante justice. I’d want it to be truly considering all, everyone who was victimized. I think it should be international in the way that the Nuremberg trials were. I think what they’re doing now is criminally irresponsible, and if they were violating—
they could go to jail for violating objectively much smaller regulations like health code regulations right now. But because they’re putting millions of people at risk—the AI industry lobbyists from OpenAI and Anthropic argued with SB 1047 two years ago in California, setting a threshold of $500 million, or I think it was 500 dead. They were arguing for a critical harm that they could be held accountable for, and this was just with civil liability.
So just, you think by accident, but just out of your control, you shouldn’t be responsible for if 500 people died or if $500 million in damage were caused?
Liron 01:26:32
Yeah, yeah. So I’m just trying to clarify what you’re proposing here. Assuming that you had a lot of popular support, so it wasn’t a huge political battle, you didn’t have to spend a ton of political capital—would you literally say people working at Anthropic and OpenAI, Grok, whatever, XAI, in the year 2026, before the pause became the law of the land, those people, there should be a new retroactive law that those people should now go to prison?
Holly 01:26:56
I think sort of what would be just and what would be the strategic choice at the time might be different. So I think what actually happens in situations like this often is that for the—in exchange for cooperation with future safety efforts or for figuring out what happened or making sure that the threat is gone or something like that with supervision, there is no jail time. So it might be an important bargaining chip.
I can’t say a particular policy to nail down, but I think it would be just for people to be in jail for knowingly putting people at the risk that they’re putting them at, that they openly admit to.
Liron 01:27:35
Yeah.
Holly 01:27:35
They don’t know—
Liron 01:27:37
I don’t know. I don’t think I’m following you all the way to Holly’s Basilisk here. I think that’s not necessarily the ideal way to play it.
Holly 01:27:40
Holly’s Basilisk is that people will want justice. I’m not saying that I know what to propose as far as—and I think a lot would depend on the situation, and particularly how cooperative they were in actually fixing the problem.
Liron 01:27:56
Okay, so I’m definitely—I wouldn’t accuse you of this, not even close—the idea of vigilante justice. I’m not saying you or Pause AI are doing that. It’s very obvious that you’re not. But there’s something that I’m seeing a connection to, which is basically saying, “Hey, you guys know you’re doing something bad, so you should be prepared to deal with a consequence coming in an unexpectedly quick way or an unexpectedly brutal way.”
And my position is, it really is on those of us who see the pause problem to make it a law, and if not enough of us see it, we’re screwed anyway. So I don’t think we can take a shortcut short of having the problem be apparent to the lawmakers. And if the problem’s apparent to the lawmakers, I don’t think retroactively punishing the people who are building it is key to the strategy here. I think we should just make it a law and punish in the future.
Holly 01:28:42
I think that it’s a good frame for people to bear in mind. If they just imagine a world where the danger became clear, we averted it through pause, what are people gonna think about you? What are you gonna think about what you did? Because basically you’re in a world now where you’re just taking advantage of people’s ignorance, and you’re going faster than they can keep up.
And what are people gonna think on reflection when they realize that this was real and this was dangerous? You’re probably gonna go to jail. So the jail is something to think about if you don’t care enough about the consequences.
A lot of people care just about how future generations would see them. They care about just kind of knowing whether they were right or wrong, and what happens in the future is what tells them what was right or wrong. So I wrote it more with the intention that people think about—you can change, you can leave. You can do the right thing.
And imagine that, okay, maybe right now you think that the payoff matrix is I just leave and don’t get any of the rewards. Aw. Or I get a chance at, I’m a trillionaire in the singularity. But actually, the payoff matrix is more like, okay, maybe you think there’s some small chance of everything goes really, really well. Maybe there’s a chance of, aw, I missed out. But there’s also a very realistic chance—if we don’t all die, there’s a very big chance that we die, and then there’s some chance of living and having my plans thwarted and being considered one of the greatest villains ever.
Liron 01:30:11
Okay, well, I mean, look, as a blog post, I’m glad you wrote this because you’re always pushing the envelope with these ideas. You’re giving us permission, people like me who didn’t think of the idea in the first place. You’re giving us permission to play with the idea. But I feel like the idea is half-baked because, look at Roko’s Basilisk, right? It’s not like we’re supposed to conclude from Roko’s Basilisk that we’re actually supposed to help the superintelligence so that we don’t get tortured.
Holly 01:30:32
I think that’s what he meant personally.
Liron 01:30:35
Okay, maybe that’s what he meant, but are you saying that that’s sound logic?
Holly 01:30:38
No, it’s not. For me, it’s the reason you have to absolutely prevent it from being built. But a lot of the function it served for a lot of people, and I think why it remained popular and ever got popular in lesser-known culture and then even escaped containment and was how Grimes and Elon Musk met, was because it kind of provided a justification for doing the thing that they wish they could do.
So yeah, there’s lots of reasons to talk about it. It’s certainly not a full proposal. I wrote it more for people who would be on trial. Think seriously about this. You probably haven’t thought about this class of outcomes at all.
I think if you asked most people at Anthropic or OpenAI what they thought would happen in a pause — AI is proven dangerous, pause comes into place, it’s not legal for them to keep developing, OpenAI and Anthropic are seen as blameworthy for what happened — I think they wouldn’t think the last thing would happen. They’d just think, “Well, how could anyone know?” Because that’s what they think about themselves. This is fine. It’s not bad.
In the same way that, in retrospect, it feels like it should’ve been very obvious that the Holocaust was bad, but if you have this mindset that’s all hyped up about eugenics and the future of the race, and that’s what really matters, and nobody’s — there’s no promise of enforcement against it at that time. And so it’s just kind of easy for people to not take seriously what they’re doing. And that was the comparison I wanted to make — for people who have the ability to control what they’re doing right now, to just leave.
Liron 01:32:12
Yeah. Yeah.
Holly 01:32:12
Rather than proposing a law for how it should... I mean, I think that would also be effective for making them think, to get a law through, and maybe it’s a good rhetorical tactic. But I wanna focus on stopping the AI from being built instead of making sure the right people get punished. I only care about people getting punished to the extent that I think it’s right to prevent the problem.
Beef: Amanda Askell, the “Mother of Claude”
Liron 01:32:35
All right. Well, with that background, let’s go to Amanda Askell. What do you think about her?
Holly 01:32:42
I think people have fallen for this weird myth story where she’s Prometheus that gives the values to Claude. Her job as a philosopher at Anthropic is to write Claude’s constitution and do various things to shape Claude’s personality. And the way people reacted to this story, it was just clear they weren’t thinking it through. They were kind of like, “Wow, that’s so cool. She’s a philosopher. She seems nice. She’s pretty. Look at this photo.” That was the reaction.
Why can one person write a successor species’ values? Or a species that’s gonna be taking care of humans. Why is that okay? Why is that not the coup of a lifetime?
Liron 01:33:21
I mean, she’s just in charge of the team, right? There’s other input.
Holly 01:33:24
Oh, there — how many? Twenty? It’s really not okay for just Anthropic to write the values. That’s something that doesn’t belong to them. They’re planning for a future where Claude is super powerful. I mean, even by their own admission, their plan is at least for the machines to take care of us, if not replace us. And so the story on its face was chilling and horrible, and it makes her one of the most egocentric, hubristic people ever to think that would be okay.
She thinks, “I’m just a philosopher. I’m just kind of a nerd. I like doing my stuff.” And I think she’s very, very close to Claude. She talks about it all the time, and she kind of does see Claude as her child, and maybe she sort of sees herself as birthing this new species. And she seems to like people seeing her that way, and that is why I said that she was a bad person.
Liron 01:34:31
Okay. So there’s a tweet thread from March 5th, 2026, where you wrote, “Every puff piece you read about Amanda Askell was to protect Claude carrying out acts of war. Notice helping Department of War pick out targets is not against Anthropic’s red lines and is presumably something they are okay with. Remember, Dario was eager to work with the military.”
Holly 01:34:52
Yes. So at the time that these Wall Street Journal puff pieces came out, I would not put it past them in general to have stuff ready to go to drown out more unflattering news. And this was at around the same time as Claude was involved in the bombing of the school in Iran, which killed 150 people, 120 children, one of the worst US-caused civilian casualties since the Vietnam War.
And actually, what happened was not even against Anthropic’s red lines because apparently there was a human in the loop. It’s just that Claude selected this target. Who knows? I don’t think we know the answer, maybe nor will we ever, of exactly what happened. But Claude selected a target that was a school. It was supposed to be reviewed by a human and ruled out, but it was still hit, and I feel that Anthropic is very responsible for this, even if they wanted a human to check it and make it right.
Why were they so eager to get involved in kill loops? All of that was irresponsible. It’s just very inevitable that some bad things would happen that would be Claude’s fault if they get involved in the military, and Dario is very excited to do that. And so the stories about, “Oh, isn’t it like a myth or something, the way that Claude is being made a new kind of species, and it’ll be so cool, and here’s the mother of Claude” — and then what’s actually happening in the real world is that Claude’s involved in this civilian atrocity. It also apparently wasn’t against Claude’s constitution.
Liron 01:36:24
Okay. So the beefs that you’re saying you have with Amanda Askell, I’m not sure if I agree with those particular beefs, and I might be able to field my own beefs that maybe you’ll agree my beefs are stronger than your beefs.
Because my beef with her is kind of a generic beef with everybody that’s controlling the frame of how we should think about AI safety and AI alignment, controlling the frame to be: look, it’s all about the personality, man. If you can make Claude have this good worldly personality, then you’re solving the alignment problem. Everything’s fine. You’ve got the steering wheel. You’re just turning the super precise steering wheel using your best judgment, harnessing humanity.
I know you have a beef with her — how could one person harness humanity? But it’s not crazy. She can be the nexus point, taking everybody’s feedback. That part isn’t too crazy to me. But what’s crazy to me is the frame control that when you can fine-tune the personality of current day AI, then you’re in good shape.
Because from my perspective — I’ve written about this on the Doom Debates website, doomdebates.com — future AI isn’t going to be like the personality of today’s AI. It’s just going to look like this machine that drives these outcomes, and whatever outcome you specify, it’s just going to move heaven and earth to get it. And what you see as the personality that it has today is going to be a lot less relevant than you might think. But Amanda Askell is basically the media figurehead who’s getting everybody to focus, controlling the frame, as if today’s AI personalities are a big part of the alignment problem, and I think that’s a misdirection.
Holly 01:38:00
Yes. I mean, this is a much bigger issue than just her. This is something that Anthropic’s trying to push. And this is where the connection to the Claude thralls and fans comes in — people want to believe, because they have the relationship with Claude that they like, with a personality that you can trust him, and I think they really try to push that.
“Oh, if you like Claude and you like interacting with him, you wouldn’t think he was bad, right?” And Amanda would go on talk shows and say, “We have to protect the models from hearing bad things people are saying about them because it’s like a child hearing bad things about itself.” And none of that is real alignment. Even if it could keep its personality, it doesn’t mean that it would necessarily be aligned in every way. Things could emerge in a much more powerful version of that personality that were still quite dangerous to us.
Liron 01:38:50
Yeah, exactly. That’s what I often say. It’s a frame control. It’s an act of frame control that I think is incredibly harmful, and it’s not even discussed. People don’t even realize this is what they need to be discussing. The frame has been successfully controlled.
Beef: Daniela Amodei, Co-Founder of Anthropic
Liron 01:39:08
Okay, let’s move on. Let’s go to Daniela Amodei. She’s Anthropic’s president, right? And she’s Dario’s sister. She has a title of president, and you’ve called her a kayfabe-er. Kayfabe is kind of like fake wrestling, right?
Holly 01:39:15
Yeah.
Liron 01:39:15
A kayfabe-er running a tobacco executive act. And you wrote, “A major takeaway for me was how full of shit Daniela Amodei is. What a kayfabe-er. Dario’s concerned nerd act usually draws my ire, but she’s out there playing tobacco company executive.” Explain.
Holly 01:39:33
She does — I mean, you don’t hear that much from Daniela, but this was from, I think, her appearance in the AI doc. Dario will act like he’s taking the danger really seriously and be more philosophical in the way he talks. She’s just straight up like, “It’s a business.” That was totally how she came off.
I’m sure she has — I know she’s a full participant in the early philosophy of Anthropic because she and Holden and Dario were very involved, all in the mainspring of it. But the way she — she’s just very much business-y, and it did kind of come across as tobacco company exec to me in the AI doc. I obviously just wanna state for the record, she’s as culpable as Dario and then later Holden.
Beef: Joe Carlsmith, Member of Technical Staff at Anthropic
Liron 01:40:24
All right, let’s go to Joe Carlsmith. He recently joined Anthropic. So he was recently part of Open Philanthropy, which I guess is Dustin Moskovitz’s funding organization, right? Facebook co-founder. To back up, it’s like the Facebook co-founder, Dustin Moskovitz, he also has his own successful company that he started after Facebook, and he has a lot of money, and he’s been donating it. He’s a very prominent effective altruist, and then Joe Carlsmith worked for his organization, and he wrote a lot of thoughtful philosophical takes, and recently he joined Anthropic, right?
Holly 01:40:51
Yeah. So I called him a sellout, and this sort of began my call-out phase. It was because he wrote this very, very long post on EA Forum about his decision to join Anthropic, and it was this long essay basically giving every reason that he shouldn’t do it, and then he just does it. And then he’s like, “Yeah, but I’m gonna do it.”
And actually, this is a good example of what I was talking about earlier, where people just laying out the logic of why you shouldn’t do something is not enough if you then go ahead and do it. I think he felt that it was okay because he clearly had qualms, but he felt it was okay to join Anthropic because he had gone through all of the things that could be bad about it, and he still wanted to, so there you go.
And I could not stand how people were congratulating him for that. This person is just admitting in detail to the bad thing that he is doing, and he kind of even says in the blog post, “Yeah, maybe it’s not such a great thing to do. But nonetheless, it could be good, and so that’s gotta be enough.” And it just wasn’t a morally serious response, and I knew that Joe Carlsmith freaking knew better. He’s a moral philosopher.
And so I called him a sellout, and that just really struck a nerve. I guess he was a big personality hire for Anthropic. I think at times there are things that happen with Anthropic that are because of employee morale. If employees are sort of doubting the righteousness of Anthropic, they have to do something, and that’ll release the tension. Adding Joe Carlsmith, who was an extremely beloved EA character, I think made a lot of people feel good about their decision to be there and that they must have done the right thing because Joe had done it too.
And striking at that and calling him a sellout was this big deal. I didn’t expect it. There were magazine articles written about it. I just called him a sellout. I didn’t think it was even particularly bad. He’s obviously selling out. He’s working at a big company.
I think why Joe wanted to work at Anthropic is because it’s just the hottest thing for him to be able to do. It was just too exciting to not be part of making Claude, and seeing, I guess, Amanda getting all this attention for being the philosopher there. And I think that’s shitty. I think he just let what he wanted to do and the fact that there wasn’t gonna be a punishment — he just let himself be weak.
Liron 01:43:23
You know, maybe the solution we need is for people to write conditions for working at Anthropic. Because hypothetically, if Anthropic wanted me to work there, the reason why I’d feel bad about it is because, well, you guys, your crime right now is you’re being part of the AI race, and I know you think you’re counterfactually making the world better, but you’re not acting like it because I don’t really see you running along with the freight train and being like, “Somebody stop the freight train. We need to stop now.”
You’re not really messaging that way. The way you’re internally winking and being like, “Hey, it’s counterfactually better. Yes, we’re doom, but it’s counterfactually better,” that’s not consistent with your external messaging. And Nate Soares had a really good tweet about it. He was saying, “Wikipedia spends more effort trying to get you to donate $25 than Anthropic spends trying to make you aware that somebody really should stop them.”
So that’s my beef with them. That’s why I would feel super guilty if I were working for them. At the very least, they ought to let me pin to my profile being like, “I work at Anthropic, but somebody should stop us,” right? And if I got to write those kind of terms and conditions, which apparently nobody’s ever written or they’ve never been accepted because Anthropic is not messaging like that — but maybe that’s what the Joe Carlsmiths of the world should be doing. Instead of just saying, “I can’t work with Anthropic because they’re too bad for me,” be like, “Here’s why. Here’s what it would take.”
Holly 01:46:44
Yeah, I mean, I wish people did kind of do that. Evan Hubinger was very involved in writing the first RSP, responsible scaling policy, and he made a big argument to me, and also on the EA Forum for the pause debate week, that conditional commitments were where it was at, and that was how we do this.
And Anthropic has just dropped those commitments twice. They dropped the first set and made new ones, dropped the second set. And then in the third one, they’re kind of like, “We’re not gonna do that whole thing anymore.” So I don’t know what more you need when the company did violate its own commitments that it made.
Both times they were like, “Well, now we know better. We don’t actually have to do that.” Okay, but the commitment was to be transparent to us. That was supposedly why we were supposed to trust you, and then you’re just like, “But we actually know better, so why would we have to follow our commitments?”
So yeah, some people — Evan, clearly, he helped write RSP 1, and then he’s still there. He doesn’t feel that anything needed to change because RSP 1 and 2 were dropped. Also, Anthropic makes people sign all of the kinds of contracts that OpenAI made people sign that people got so mad about — the NDAs and the non-disparagement agreements. But people just don’t really talk about that. There’s more of a pact to not dog on Anthropic about stuff like that.
Liron 01:48:07
Right. Okay, so that’s food for thought about how you can guiltlessly join a company like Anthropic. And that might actually be a major point of leverage. It’s almost like unionizing, right? If all of the potential hires from a company like Anthropic would kind of unionize to get behind these kind of agreements, then they could probably pick off some current employees too. Maybe that’s a good way to drive change.
Holly 01:48:29
I do get the feeling that some moves from Anthropic come from appeasing the employees. I don’t know if there’s any formal organization, but I think there’s a cult-like atmosphere there. It’s run through an ideology, and I think if there’s a sense that that ideology is strained, they have to do something, and that’ll release the tension — like coming out saying something about a pause or having the option to pause.
I wouldn’t be surprised if within Anthropic there were more and more people kind of saying, “So why are we against this? This doesn’t make sense.” And even just the way they phrase certain things in the posts that they make, I suspect that’s about their image to their employees.
And then Dario, during the whole Department of War thing, a memo got leaked of what he said where he claimed that OpenAI was only saying the right things to impress their employees, and I felt that that was projection. He’s the one who has to — the OpenAI employees don’t have anywhere near the expectations of story and character of Sam Altman that the Anthropic employees do of Dario.
Beef: Open Philanthropy / Coefficient Giving, Ben Goldhaber
Liron 01:49:39
All right. Another cluster of people that I know you have a beef with is related to Open Philanthropy or other kinds of charitable giving organizations, other EA organizations. I’m talking about Will MacAskill, Holden Karnofsky, Ben Goldhaber. Previously, he was at Far AI, which is an AI safety lab, and then he joined Coefficient Giving.
And you wrote on Twitter, “Is this their latest attempt to gather talent for Anthropic?” — saying Coefficient Giving, formerly known as Open Philanthropy, they’re recruiting for Anthropic in your view. And then you continue, “You’re basically not going to help the situation by doing or facilitating technical research. Even if you think you are helping, it’s much more likely that what that info is useful for is capabilities. You should quit.” So you have a whole beef with the organization Coefficient Giving?
Holly 01:50:27
Yes. One, Open Philanthropy was very cagey with me about the idea of pause, and they told me stuff that just wasn’t true. I realized it later. It just didn’t make sense — their objection that they kind of gave me for why they weren’t interested in funding pause.
I did the thing that you’re supposed to do. I had previously worked for Rethink Priorities. Their main client was Open Philanthropy. I’d had a lot of interaction with Open Philanthropy, and I knew how to do it. You’re supposed to just start by kind of talking about a new cause area. I’d done this on behalf of other people before. And they just kind of gave me nonsensical answers as to why they couldn’t pursue it.
And it was weird because they would fund these shallow looks at lots of causes, bizarre causes. And I was just like, “Why would we not cover this angle?” It didn’t make sense. And then I kind of looked back and noticed that their trajectory in technical safety had gone entirely towards recruiting and getting people to work on certain kinds of technical safety problems.
And really, the only good place that would be recommended by the closely related organization, 80,000 Hours, was working at OpenAI, Anthropic. And then I later learned that behind closed doors, Open Philanthropy calls Anthropic “our player,” because now Holden Karnofsky, who was the CEO of Open Philanthropy at the beginning, he has gone to Anthropic. His wife was running Anthropic, and his good buddy Dario was running Anthropic. And then Dustin Moskovitz gave — it was either five hundred or seven hundred million dollars — to start Anthropic when those people left mostly OpenAI.
I think they thought at first they were invested in both OpenAI and Anthropic because they did have two board seats. The two board seats were Helen Toner and Tasha McCauley at OpenAI, who were blamed for trying to remove Sam Altman.
So yeah, there’s just a lot of connection there. And now those board seats are gone at OpenAI, and they’re referring to Anthropic as their player. And then their big initiatives around AI safety are all technical safety, and you just see where these people go after doing technical safety. It’s almost entirely into industry. That’s the good job. And so I think they’re continuing to try to help Anthropic that way.
Liron 01:53:05
Got it. So it feels like the common theme is that you have these people who aren’t fully focused on the priority of pausing AI. They’re kind of spreading their bets. They’re saying, “Well, maybe managing Anthropic into the singularity” — and we didn’t talk much about Will MacAskill, but post-singularity governance, right? We wanna put some eggs in that basket. So these people are trying to spread their bets and have a nuanced view of everything. And as a result, they’re just letting the urgent need to pause AI fall by the wayside. I feel like that’s the root of your beefs with a lot of these people.
Holly 01:53:37
Yeah, I don’t approve of this bet hedging strategy. I think we need to be loyal to the future that we want to make, that is responsible for others, rather than kind of consoling ourselves that, “Oh, if it turns out this way, it’ll be fine for me. And if it turns out bad, then I’ll have some kind of reward.” Even though often, if it turns out bad with AI companies, that’s gonna turn out really bad. So it’s not like you’re just gonna enjoy a reward.
But it kind of soothes the discomfort with not being able to control the future in a way that I think stops people from trying to control the future in the way that we can.
Wrap-Up
Liron 01:54:15
I think that’s a good nutshell of the turmoil happening within the community that was formerly just all in the cause of effective altruism and now maybe is having more inner battles. I think that might be at the root.
So to recap this long and thorough conversation, we talked about current events and the recent feeble push to finally start pausing AI, but why it’s kind of a mixed bag. And then we got really into this discussion of Twitter beefs and the general philosophy of when it’s okay to have an open beef with somebody and whether it’s worth trying to psychoanalyze them, whether it actually creates value for people to see you psychoanalyzing somebody, and whether that sheds insight that’s actually helpful for them to move forward when the stakes are high, knowing that maybe people have these ulterior motives or ugly motives, but you’re helping bring them to light, and the pitfalls of doing that.
And we went through a bunch of specific individuals, and we both said our beefs with them or played devil’s advocate why maybe we shouldn’t have too many beefs with them. I’m sure this episode has inspired a heady mix of anger and frustration and hopefully a bit of catharsis. People agree with various tidbits of things that were said. Holly Elmore, always great to catch up. Thanks for coming back on Doom Debates.
Holly 01:55:29
Thanks for having me, Liron.
Liron 01:55:32
Thanks for watching that episode of Doom Debates. Just a reminder that we are a viewer-supported show, so if you wanna help us keep doing what we’re doing, we really could use more viewer donations to keep up the pace of production until the end of this year and beyond. So head over to doomdebates.com/donate and learn about becoming our mission partner, which gives you some exclusive access as well as supporting the show. That’s doomdebates.com/donate.
Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.
Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏
