Eliezer Yudkowsky can warn humankind that If Anyone Builds It, Everyone Dies and get on the New York Times bestseller list, but he won’t get upvoted to the top of LessWrong.
According to the leaders of LessWrong, that’s intentional. The rationalist community thinks aggregating community support for important claims is “political fighting”.
Unfortunately, the idea that some other community will strongly rally behind Eliezer Yudkowsky’s message while LessWrong “stays out of the fray” and purposely prevents mutual knowledge of support from being displayed, is unrealistic.
Our refusal to aggregate the rationalist community beliefs into signals and actions is why we live in a world where rationalists with double-digit P(Doom)s join AI-race companies instead of AI-pause movements.
We let our community become a circular firing squad. What did we expect?
Timestamps
00:00:00 — Cold Open
00:00:32 — Introducing Holly Elmore, Exec. Director of PauseAI US
00:03:12 — “If Anyone Builds It, Everyone Dies”
00:10:07 — What’s Your P(Doom)™
00:12:55 — Liron’s Review of IABIED
00:15:29 — Encouraging Early Joiners to a Movement
00:26:30 — MIRI’s Communication Issues
00:33:52 — Government Officials’ Reviews of IABIED
00:40:33 — Emmett Shear’s Review of IABIED
00:42:47 — Michael Nielsen’s Review of IABIED
00:45:35 — New York Times Review of IABIED
00:49:56 — Will MacAskill’s Review of IABIED
01:11:49 — Clara Collier’s Review of IABIED
01:22:17 — Vox Article Review
01:28:08 — The Circular Firing Squad
01:37:02 — Why Our Kind Can’t Cooperate
01:49:56 — LessWrong’s Lukewarm Show of Support
02:02:06 — The “Missing Mood” of Support
02:16:13 — Liron’s “Statement of Support for IABIED”
02:18:49 — LessWrong Community’s Reactions to the Statement
02:29:47 — Liron & Holly’s Hopes for the Community
02:39:01 — Call to Action
Show Notes
PauseAI US — https://pauseai-us.org
PauseAI US Upcoming Events — https://pauseai-us.org/events
International PauseAI — https://pauseai.info
Holly’s Twitter — https://x.com/ilex_ulmus
Referenced Essays & Posts
Liron’s Eliezer Yudkowsky interview post on LessWrong — https://www.lesswrong.com/posts/kiNbFKcKoNQKdgTp8/interview-with-eliezer-yudkowsky-on-rationality-and
Liron’s “Statement of Support for If Anyone Builds It, Everyone Dies” — https://www.lesswrong.com/posts/aPi4HYA9ZtHKo6h8N/statement-of-support-for-if-anyone-builds-it-everyone-dies
“Why Our Kind Can’t Cooperate” by Eliezer Yudkowsky (2009) — https://www.lesswrong.com/posts/7FzD7pNm9X68Gp5ZC/why-our-kind-can-t-cooperate
“Something to Protect” by Eliezer Yudkowsky — https://www.lesswrong.com/posts/SGR4GxFK7KmW7ckCB/something-to-protect
Center for AI Safety Statement on AI Risk — https://safe.ai/work/statement-on-ai-risk
Other Resources Mentioned
Stephen Pinker’s new book on mutual knowledge, When Everyone Knows That Everyone Knows... — https://stevenpinker.com/publications/when-everyone-knows-everyone-knows-common-knowledge-and-mysteries-money-power-and
Scott Alexander’s “Ethnic Tension and Meaningless Arguments” — https://slatestarcodex.com/2014/11/04/ethnic-tension-and-meaningless-arguments/
Previous Episodes Referenced
Holly’s previous Doom Debates appearance debating the California SB 1047 bill — https://www.youtube.com/watch?v=xUP3GywD0yM
Liron’s interview with Eliezer Yudkowsky about the IABI launch — https://www.youtube.com/watch?v=wQtpSQmMNP0
Transcript
Introduction
Liron Shapira 00:00:32
Welcome to Doom Debates. My guest today is Holly Elmore. She’s the executive director of PauseAI US. She leads protests at AI companies that I’ve been a part of myself. She leads outreach to policymakers through the PauseAI US organization, and she also has a PhD in evolutionary biology from Harvard University.
Holly is a returning champion. Last year she was here on Doom Debates and she was debating Greg Tanaka, a Palo Alto City council member about the SB 1047 bill. Go check out that episode.
Today, Holly and I have a lot to talk about. We’ve been like-minded about a lot of the issues surrounding Eliezer Yudkowsky’s new book “If Anyone Builds It, Everyone Dies.” And we discussed it last week at the unofficial “If Anyone Builds It, Everyone Dies” launch party. So you can check that out as well.
But I wanted to bring Holly on for a full episode because we have so much to talk about in the aftermath of this book launch. I think she’s a great person to compare notes with. Just share our opinions about what we think a good launch would look like and how we can do better going forward.
Holly Elmore, welcome to Doom Debates.
Holly Elmore 00:01:41
Thanks for having me back.
Liron 00:01:43
All right, Holly. So we’ve got a lot to talk about. We’ve been going back and forth on Twitter having a lot of the same observations, and I’m gonna say criticisms because there’s a lot of stuff that we’re not happy about. I think this is gonna be a very interesting conversation.
So first we’re just gonna review the book’s main claim, and then we’ll talk about people’s reaction to the book, essentially reviewing the reviews. And then we’re gonna talk about something that you might call the circular firing squad, or my new coinage, the crab bucket, which is our kind of critical term for how the rationalist community sometimes reacts to these kind of things. And I think it’s really coming to a head this week in the wake of this book launch. So we’re gonna talk about that.
We’re gonna talk about the general challenge of building mutual knowledge when the most informed people are anti-signaling, which I think is also a very timely topic in the wake of Steven Pinker’s new book, which is all about mutual knowledge.
So wide ranging discussion. Because all of this stuff is all in your mind too, right?
Holly 00:02:44
Absolutely. I can’t wait. It feels like normally doing outreach and explaining about AI danger is work. I’m having to phrase things in a way to really meet the audience.
I feel like this is my real frustration that I feel every day and I’m really excited to have a chance to take it apart.
Liron 00:03:03
Exactly, exactly. All viewers, there’s gonna be some drama here. This is gonna be a juicy episode.
On If Anyone Builds It, Everyone Dies
Liron 00:03:10
But before we get there, let’s just back up. So what would you describe as the book’s main claim?
Holly 00:03:15
I would say the book’s main claim, it’s well advertised, is that if anyone builds it, everyone dies. The book’s main claim is that we just - not that it’s impossible to have aligned super intelligence ever, or that a utopia is impossible, but that it’s not possible given the level of technology we have today using current techniques.
They - it’s not, if somebody does that, tries to build something that powerful, it’s not going to actually be aligned and there will be disaster.
Liron 00:03:49
Yeah, I’ll read a quote from the book because it summarizes itself very well. It says, “Building an artificial super intelligence using anything remotely like current techniques based on anything remotely like the present understanding of AI will cause human extinction.”
Holly 00:04:08
Yeah. They say extinction. I’m a little more umbrella, like disaster would be bad too. But they feel pretty confident about extinction with what they said about our current understanding and our current techniques.
Liron 00:04:24
Yeah. So that’s the main claim viewers. If you haven’t gotten the book yet, please order it. Please read it. It’s definitely worth your time. It’s like five hours of reading or audio listening, distilling the last decade or two of Eliezer Yudkowsky and the thinking of people like me in the community, highly worth your time.
I mean, you agree it’s worth people’s time, right?
Holly 00:04:43
Definitely. People in particular are often very curious about Eliezer, the personality, and this is by far the easiest look into his thinking. He’s got a very particular style, which does still come across in this book, but a lot of people are kind of screened out or alienated by that style and he’s really made an effort.
And Nate Soares is also an author of the book. I should give Nate credit. But Eliezer’s style and his ideas are coming across in his own words, but in a much more accessible way than I think ever before.
Liron 00:05:20
Exactly. It’s like somebody asked Nate Soares, give me a distilled Eliezer Yudkowsky book without the “ow.”
Holly 00:05:32
But still with plenty of parables and dialogues and stuff.
Liron 00:05:35
Exactly. And fans still notice the real Eliezer Yudkowsky coming out. I mean, there’s plenty of references. Analogies he likes to make, particular anecdotes, so it’s not fully Eliezer-less.
But if you’d never known about Eliezer Yudkowsky and the rationalists and you just pick up a copy of this book, I think the reaction you would have is, oh yeah, this belongs in the nonfiction section. This is just like a nice, popular bestselling book, but it’s hitting me with banger after banger, which is rare for a nonfiction book to do.
Holly 00:06:00
Yeah, it’s definitely - it should be going for the cadence of popular science, where you’re sort of allowed to make - if you think about Neil deGrasse Tyson, he says a lot of kind of poetic things. Could he put that in a scientific paper? No, but he’s telling you about the facts and he’s giving you also his interpretation of the facts.
And Eliezer is very allowed to bring that level of takeaway to an audience. And he does, he just normally does it in a much more idiosyncratic way that was not mass marketable before.
And it was really hard for him behind the scenes. I just know it was really hard for him to do that. It was really difficult for him to alter his style. He didn’t wanna feel like he was sacrificing precision or accuracy at all.
And so he dug deep to be able to do this, to be accessible and also keep his standards of honesty. Which is why it’s particularly shitty that people are like - his own community that he built are being so negative about it and they’re criticizing exactly that thing.
Personal Book Reviews
Liron 00:07:11
You wanted to add that maybe you have some nitpicks with the degree of risk that it’s saying, or how would you finish...
Holly 00:07:21
I like this book. I think it’s accessible. It tells one account of AI danger and it’s very - you can hear Eliezer’s voice, the original AI alignment guy who’s been saying it for a long time. But it’s much more readable for just any new audience.
And I really have very little to say against it. The only thing that I would add just to clarify, just as a representative of PauseAI US, is that we don’t think it has to be extinction. And we don’t think that extinction has to be almost certain to implement pause as a solution.
There’s lots of negative things that can happen. And so I think Eliezer and Nate have a very high bar for interfering in society or regulating. Most people don’t have such a high bar and PauseAI US is about just us as the people asserting what we want and what we think is good for us, and if we wanna be safe, that needs to be taken into the democratic calculus.
So I don’t want people to get the idea from the book that you have to think that extinction is certain to merit the solutions that they’re talking about. The solutions they’re talking about are good for preventing gradual disempowerment, loss of connection between people, a generation that is emotionally stunted or has their mental illnesses exploited by ChatGPT. Any concern you have about this technology going too fast, a pause is meant to address that.
Liron 00:08:57
Yeah, that’s a good distinction on the PauseAI Coalition as it compares to the AI extinction level doomer coalition, which I myself am part of. I mean, I’m both, right? I’m both in the PauseAI coalition and I also think that AI is likely to cause extinction.
So if I were to just ask you the book’s central claim, if anyone builds it, everyone dies - I know that’s a narrower claim than necessary for membership in PauseAI, but do you think they’ve made a strong case for that claim?
Holly 00:09:26
I think so. I think there is a strong case and they could have told many other scenarios with different details that lead to the same conclusion. I think the real point they’re trying to make is that the world is very easy to disrupt in lots of ways that are not obvious at our level of capabilities. But if you raise the level of capabilities, there’s a lot of unforeseen ways in which that intersects with the current world and it can just break our system.
And because I believe that that’s how the world is, that it is vulnerable to higher capability agents coming in, or not even agents just higher capability stuff messing up our equilibria.
Liron 00:10:03
So on that note, I hope you don’t mind if I ask you, Holly Elmore, what’s your P(Doom)?
Holly 00:10:15
Ugh. I don’t love this question. My pat answer is that it’s like 20 to 40% on the worst - the variation is like 20% extinction and then to like 40%, the worst civilizational tragedies, but not everyone dies.
And then I’d put like 40 additional percent into just bad. We just have a society we don’t really want and we didn’t need to do it. And then I would put maybe 10 to 15% on the top of just, we get lucky and we don’t die. And then we’re also like, society’s not so bad.
So I have kind of a spread of outcomes. I think that just society not being good is plenty of reason to not rush ahead with this technology. I don’t love the obsession with P(Doom) as a number, but I would - so 20 to 40, put me down.
Liron 00:11:16
Okay. So like the book, talking about everyone dies, do you think there’s like a 20 to 40% chance that by 2050 everyone dies?
Holly 00:11:24
I find myself resisting these questions so hard. But I put that closer to like 20.
It really varies. I hate how much that’s by my feels, and my feels do vary every day. And sometimes I can tell my feels have to do with a Twitter interaction and that really shouldn’t be determining what I think is the likelihood of the technology going bad.
And there are always subjective estimates, but sometimes I feel like as low as five and then sometimes I feel like as high as 35, like for everyone dies. And it has a lot to do with - just to be honest, subjectively, I can tell it has a lot to do with how much I feel I can trust authority figures to take action and that swings very wildly.
Liron 00:12:24
Yeah. For me, I don’t have a strong opinion. If anybody wants to say 10 to 90%, that’s all fine. I mean, feels are going to determine it. I don’t claim to have single digit percentage precision. I think no human can calculate probability that precisely.
That said, I do think if somebody’s feels are like below 10% or especially below 5%, I just - that’s mind blowing to me. I just feel like the feels are in the wrong ballpark. Right? It’s as if somebody said their probability is like 0.05%. It’s like really? Is that really the ballpark that your feels should be outputting numbers in? I don’t think so.
Okay, so that’s your nuanced book review. I’ll give my book review. I’ve never written it down, and I wanna premiere my book review for the first time here on this episode of Doom Debates.
My book review is that there was a space of possible books that I would’ve really liked to exist. Think about it as like a target with circles inside of circles. I think this book hit quite close to the bullseye as far as I can tell.
But I also think that the target of books that I would’ve loved is pretty big. So I think this book is just well within the space of books that I would’ve loved. I think this book is really great. I think it’s much greater than the minimum standard of a book that I’d be really passionate about.
So to me it’s like people are asking, oh, what’s your least favorite part about it? And I’m like, I have no idea. It’s already insanely great.
And to the extent that I would change it, I would just change it in ways that consequently become more effective. So if you tell me people would react better if the book were like this, I can’t judge whether that’s true or not. But if that is true, then sure I’d wanna change it that way.
But as far as I can tell from reading the book itself, I don’t know a particular modification that I think would make it more likely to be well received or make the argument more clear. So basically, I think the book is amazing and I just don’t feel the need to nitpick, but I also don’t have particular nitpicks that come to mind. I think it’s like a nice smooth ride. It’s like a nice nonfiction book.
I think they’re not getting enough praise for this book.
Holly 00:14:16
No, I mean, and because we have a few books or a few resources that kind of hit this mark already. And like with Eliezer’s fame - they’ve created already, before it even came out, it was like the premier resource of this type and that’s amazing. I’m really, really happy that they did that.
I’m trying to do public communication. This kind of book being out there really fills a gap because people - I don’t know, just people look to certain signals of like, is this topic real? Is this topic serious?
And having a real trade book that they saw in the New York Times, like existing on the topic, whether they ever crack it open or not, does immense things for their belief about where this topic falls, how much they should care about it, how much credence they should give it. So that alone is amazing. It’s not even about the content of the book.
And then the book itself, I think if somebody reads it and they don’t know anything about the situation, they’ll come away getting it. I mean, I as a professional in this field might have some additional things to say and probably just selfishly, the only thing I would change about the book is that there’s a lot of letting people off the hook for not being early joiners to this cause. Maybe that is effective.
But we’re in the stage right now where we’re needing people to join to make the critical mass and they’re kind of like, well, of course it’s okay. You don’t want to join something that doesn’t have a critical mass. They say it at a few places and their marches are assurance contracts where they’re only gonna happen if enough people agree.
And I would like to emphasize, I guess that courage and standing up a little bit more, but that just has to do with me knowing this arena very intimately.
Liron 00:16:08
Here I think I know one example of what you’re talking about. So here’s the book’s website, ifanyonebuildsit.com. And there’s this link here that says March.
“If anyone builds it, everyone dies. So don’t build it.” March to Soft Super Intelligence in Washington DC. Pretty cool looking webpage.
And you can click “I pledge to attend.” And then I think this is what you’re talking about. It says, “If a hundred thousand other people also pledge to march in DC I will attend that March.”
Holly 00:16:37
Yeah. And then it says in the book something like, we believe that a big march is more effective than a small march. And I did a little bit think, oh, was that aimed directly at me?
Because I’m doing small protests and if people are around me or people from the AI safety community, literally deadass many times are like, well, have you ever considered having a big protest? I’m like, how do you think big protests happen?
They really, I think that they kind of believe on some level that if I approached it in the coolest way, then it wouldn’t be difficult. People wouldn’t be going through fear or that isn’t just the process of actually winning converts - that they have to deal with, oh, it’s not socially safe to have this position because there’s not a lot of people in it. Them joining is how we get big. And I feel like we can’t leapfrog that.
I really hope that this works. I will pledge to attend. But it does kind of take away from a personal responsibility duty message, which I do think is important for getting the critical mass at this point.
Liron 00:17:49
Fair enough. Yeah.
Holly 00:17:50
Small nitpick from also working in AI safety.
Liron 00:17:55
I’ve heard Eliezer a couple times - I think this was in his book, Inadequate Equilibria. He’s expressed appreciation for the social technology of Kickstarter, right? The idea that you first need a bunch of people to conditionally commit, and then it reaches some threshold and then they all act.
So he’s trying to use the Kickstarter model for this protest, but in practice it’s kind of interesting. I feel like Kickstarter, the company never really used the Kickstarter dynamic that much. It’s more like, okay, yeah, here’s a webpage where you can just donate and we’ll just take any amount. Right? I feel like the thresholds never made that much of a difference.
Holly 00:18:30
I think it’s a fantasy. People - because they’re not the first to talk about assurance contracts for demonstrations. I had a lot of rationalists tell me that I should do that. And then maybe they would come to a protest if they could be assured that it wasn’t cringe, essentially.
And I just don’t think that’s true. I think if the reason you’re coming is because you have that guarantee, you’re not the best recruit. What I want is for people to believe, to have conviction behind what they’re saying and represent that at a demonstration and just getting people to feel like it’s popular and safe to come to a protest is not really demonstrating the chutzpah that I want.
Liron 00:19:07
You don’t like me at my smallest, you shouldn’t join me at my biggest.
Holly 00:19:22
I feel that way about a lot of these people, honestly. I’ll take them, I think once we are bigger and once they’ll forget all about it. If the way that this goes is that PauseAI keeps getting bigger, we have more buy-in from the public, we have more jobs to offer - they’re going to forget that they ever ridiculed what we were doing or that they came up with excuses not to do it. And they’re gonna even think that they were there on the ground floor because they were aware of it.
That’s just how they’re gonna remember it. This is classic, this happens with vegetarianism all the time. People think they were big supporters because they knew you and they forget that they were giving you a hard time about not eating at dinner.
So yeah, I think absolutely. I’ll take ‘em back. It’s a big part of that kind of activism to be big about it and take people back and not rub it in their face that they have amnesia about this.
But it makes me annoyed and I really wish that they had taken this opportunity to valorize standing up and doing the right thing, whether it’s popular or not. And instead they kind of emphasize that, oh, of course, nobody could be expected to do this. This sucks.
MIRI’s Flawed But Good Execution
Liron 00:19:23
Yeah, exactly. So now we’re getting into one of the topics of discussion, which is we do have some criticisms with how Eliezer and MIRI have approached their PR, which I don’t think have to undermine how much we support their book. Right? I’m a hundred percent in support of the book. I’m actively working to help the book.
But it’s been a week after the book, there’s a bunch of drama. I think it’s still fun and interesting to lay some of the drama at the feet of Eliezer and MIRI. So like you’re saying, I don’t think that they lionized the act of doing the early protests as much as you and I would’ve liked.
And furthermore, when Eliezer did the interview with me, I brought up the question of protesting and he was really hemming and hawing. He is like, well, my preferred ideal scenario is if the politicians just immediately get the picture. That would be the smoothest situation with the least wasted motion to use one of his terms.
To paraphrase, it would be the least wasted motion if our leaders would just get it and take action. But he did reluctantly agree that yes, the world where people protest, that is a useful signal. That’s probably good. But it’s like the world where we smoothly get to the end state without the protest - it just seems so unrealistic to me.
Holly 00:20:51
Well, he would love to just complain about that, that the world is not adequate or whatever. But yeah, it’s a thing that - I do think Eliezer has put in a lot of work. And so I don’t wanna be too hard on him.
But I feel like his philosophy on this has really given license to a lot of just kind of lazy nerds to just be like, no, well I don’t have to do that. ‘Cause that’s inelegant.
They have this instinct that protesting is cringe and they don’t even - I don’t even know if they can introspect and know why that is. But they see that there’s few people doing this against this big building of people and they’re just yelling - don’t they realize that doesn’t do anything? Are they stupid? They feel vulnerable at that whole situation about being in the small group and caring, showing that they care, showing that they hold the position strongly. All of that makes them feel vulnerable.
I have no idea how much real insight they have into that, but they just say that’s cringe. And then the reaction is actually, you are wrong for doing it or for pressuring me to do it with your existence because you make us look cringe. And it’s really important that we not look cringe to decision makers, to AI company execs that we could influence. Totally. Because we are cool. They’ll see that we know a lot about ML and that will - they will have to listen to us once we’ve shown that we’re such excellent scientists.
And so you’re actually detracting from that because you’re so uncool. But they just feel vulnerable and Eliezer really pushes bravery in a lot of other ways, mostly being honest, and I think he is brave, but I don’t wanna - it feels unfair to be so hard on him. ‘Cause I think he’s also done an incredible amount of good.
But yeah, he’s giving cover to really very base instincts on this and I think he should reevaluate what he considers worthy of honor. It’s cringe and embarrassing and you don’t wanna do it and it’s against your aesthetic of how debate should happen or whatever. But doesn’t that make it all the more admirable?
I mean, especially the people in my protest, almost all of them come from a background somewhat influenced by Eliezer and they had to go against that in order to do a protest. I thought it was even...
Liron 00:23:11
Yeah, I mean, to push back a little. I mean, if I’m putting on my Eliezer and MIRI hat, I think they would say, look, it’s not really about our emotions. We agree that it’s honorable to just go stand there. Especially like the recent people who did the hunger strike, like Guido Reichstädter and Michael Trazzi and a couple other guys.
I think Eliezer would agree, yes, this is honorable and there’s a lot of dignity to it, but we at MIRI doing it just in terms of consequentialism and the various impacts of that - it might be net negative, right?
Holly 00:24:14
Well, that’s a different question. Whether their organization should do it is a different question than whether they should be supportive of the idea.
They have never - the best I’ve ever heard is apparently happened in that interview, which, sorry I haven’t watched. But is basically like, well, I have to admit that on my worldview, this seems like something good, but it’s just so against my aesthetic.
And my aesthetic is very close to what I consider morality about rationalist ideas about speech and discourse. That’s the main sacred thing that they do is speak in a certain way and follow very strict rules on how to speak. And they believe that has almost magical properties that propagate out to change people’s minds because you’re raising the sanity waterline.
And there’s so many phrases like that in rationalism that evince that idea. And they consider protesting to be not that. And so it’s very, very difficult for them to countenance it. And I think it’s just an aesthetic thing.
And I really think fine, I understand reacting that way initially. And then of course there are really times where your audience you’re trying to reach is gonna think the same. And so you shouldn’t do that. I don’t think MIRI could have led protests. But if they don’t want to, they want a different image, that’s fine. As an organization, I don’t think every organization needs to be out there protesting, and I don’t think every person needs to be out there protesting if they really, truly have another thing that they’re pursuing.
Critique of MIRI’s Coordination
Liron 00:26:37
So yeah, so just to get my last MIRI criticisms out of the way here, because I think most of my criticisms are really for the community. And the big picture here is I don’t think MIRI failed here or I don’t think that MIRI was inadequate here. I think MIRI was adequate. I think by creating this amazing book and seeding the community to begin with and then kind of throwing the book over the fence and agreeing to do interviews, putting in a good faith effort, right? They didn’t phone it in.
I think they did enough and they gave the community enough to run with and the community dropped the ball more so than MIRI. That said, let me just get in my little MIRI nitpicks in here. I do think MIRI is also often hard to cooperate with themselves.
Like you say, they’re not really supporting the protest movement. They’re not really supporting PauseAI in any way that I’ve seen. I don’t think that they helped out that many people in the community with interviews. I mean, I was fortunate enough to interview Eliezer. I know a number of other people who didn’t get that opportunity.
And in particular the number one recommendation I gave them was rally the troops in the form of a Discord before you launch the book. And just every week be like, okay, so these are our pre-orders. This is our best estimate of our pre-orders. We’re hoping we can get more this week. Can anybody else reach out to anybody? Do you need our help? What can we do to help?
Have the Discord group, which is, as I understand it, this is a standard tactic that a lot of organizations use when they’re trying to get numbers up. I don’t think they did that. I personally actually told them to do it. The response I heard was like, hmm, that sounds like a pretty good idea. And then just radio silence. So I do think that’s representative of...
Holly 00:28:02
Other people to approve it. So they have organizational issues. Yeah.
I’ve been pretty - PauseAI and this was my idea. We’ve been putting a lot of work into promoting this book because this is a great moment where we can - we know there’s gonna be a conversation and a news item about exactly this. And it’s an opportunity for just fun stuff instead of protests. That’s hard. That’s a big ask for everybody to do adversarial stuff. We’re just having a fun celebration where we’re teaching people about what’s in this book.
That’s coming up October 4th in San Francisco, by the way, at Manny’s five to eight. So we’ve been putting a lot of work into this and I was trying to get MIRI’s help with this all up to the launch and I would get kind of supportive...
I talked to - I guess I shouldn’t name names, but one person very high up was like, yes, great idea. Let’s do it. You should talk to these people. And then I would talk to other people and they’d be like, well, we’re not saying we’re against PauseAI, but we just don’t wanna couple the fates of PauseAI and MIRI. And I was like, we are just offering to do a service for you. And I’m asking you can you tell me some things about the book so we can promote it? And you’re like, mm, mm.
That’s very typical of their culture.
Liron 00:29:24
Right. Similar. I mean, some of the shows that didn’t get interviews, I’m like, this show has a half million dollar a year budget promoting your message and not sponsored by you. And they’re just asking for an interview. So I’ve heard - and they’ll take Nate Soares too. It doesn’t even have to be Eliezer. And Nate was a great co-author and has a lot of great opinions themselves, but no, they didn’t get anybody from MIRI. Right? So I know a number of...
Holly 00:29:49
They’ve done a really good job with publicity, considering how distrustful they are of journalists. This has been definitely an Eliezer position. He says this in his own voice online all the time, but also it’s a MIRI position that they just think mainstream journalists don’t understand anything and they just don’t understand what happened. They gave a very clear five paragraphs of information and the journalists took it all outta context. This is something that they’ve described over and over.
So I’m really happy with how far they did go with mainstream journalists and they got a lot of great endorsements from big publications and things like that. And that was big, I know that was hard for them.
But still, I’ve always found that worthy of criticism. Just the other day, Eliezer said something online about he wasn’t sure if he could believe this number in the New York Times because it was from a mainstream media source. And like, what are you trying to say? You’re not mainstream, you don’t trust - why would you - my dad’s a journalist. I’ve always been particularly sensitive to it because rationalists in general tend to have these - and they get their cues from Eliezer often. They have these beliefs that journalists just don’t understand anything and they’re so manipulative because they don’t do the speech act the way that they’re supposed to. They don’t...
Liron 00:31:06
Yeah, I wanna be careful to - when you say this stuff, I mean, you gotta also try to give Eliezer’s side of the picture, right? I mean, he was right a few weeks ago on Twitter when some mainstream outlet tweeted some academic paper or something, tweeted something and Eliezer rolled to disbelieve and Eliezer was totally right. So let’s try to be fair and balanced here.
Holly 00:31:27
It’s more though. It’s not just like they get stuff wrong or they don’t have the highest standards of accuracy. Yes. Of course. It’s different than Eliezer always compares it to nerds blogging. It’s different than that and it’s different than academic work.
But it’s just a distrust that’s very deep and up until now, Eliezer’s never been willing, never deigned to try to talk in a way that is more predictable what the journalist is going to understand and take from that. He is trying to do that now. They basically just reading scripts, which is good. I’m glad that they’re willing to do that.
Liron 00:32:06
Yeah. All right. And yeah, so last thing in the MIRI criticisms from me section. They always feel like they’re suffering from analysis paralysis. I think you mentioned something like that.
And then whenever we interact with ‘em, it’s just kinda like pulling teeth. It feels like there’s a long delay, emails don’t get responded to, and then we just get like a simple answer. It’s like, okay.
So I just get the sense they’re kind of energy vampires and I know they’re trying their best and they wish they could do better, but boy, it is just a little unpleasant dealing with ‘em. Okay. So I keep it honest on this show. I mean, let’s keep it in context. I’m the biggest MIRI fanboy that you’re going to find. Okay? So these criticisms are just coming from a good place.
Holly 00:32:49
Yeah, it’s not even just the delays. It’s like, and then getting kind of a little slap in the face. In my case a lot, every time I get an answer from MIRI, they’re just sort of essentially like, well, we don’t think you’re that good.
Liron 00:32:59
Right? Yeah, pretty much. Pretty much. I hear you. Yeah, exactly. Yeah. You’re not in the top, top of your field. Yeah. No, I can empathize.
Holly 00:33:09
So they’re arrogant. They think they do things better, even though they do things super slow and stuff. I feel like they make excuses for themselves. I also think in other ways they’re deservedly arrogant. So like...
Liron 00:33:25
Nice. All right.
Holly 00:33:27
I’m super happy with what they’ve done, but they are - they’re just, they’re real people. Yeah.
Liron 00:33:31
Let’s move past this section because this episode is actually supposed to be an episode about not nitpicking things that are good overall. And like I said, I’m a big MIRI fanboy.
My position on MIRI is that their work and their content is vastly underrated. And what they did in launching the book, despite all the nitpicks that we just went over, was more than adequate. So they have done a great job overall, and that should be the bigger takeaway here.
Holly 00:33:54
No, and especially I think I would give extra honor and dignity points to them for doing something that did go against a lot of their hangups. It was, I think it was quite difficult and it showed a lot of bravery and a lot of - I really believe in them and MIRI in a way that I don’t in the rationalist community and that old part of the AI safety community anymore because of demonstrating that. I really believe they care and they have their heart in the right place.
Reviewing the Reviews
Liron 00:34:21
All right. So that said, let’s go on to talk about other people’s reactions to “If Anyone Builds It, Everyone Dies.”
Maybe we can start with the most famous people reviewing the book. For example, there were the blurbs. The blurbs by various national security people, right? People in government. Ben Bernanke was there, like RFP Eddie. I could pull up the list myself.
Okay. We got Fiona Hill, the former senior director of the White House National Security Council. I feel like I’ve heard that name before, right? I’m not intimately familiar, but that’s great that these people are leaving positive reviews, right?
Holly 00:34:54
Lieutenant General Jack Shanahan. That was a big one. I remember when they got that.
Liron 00:34:56
Yeah. Hell yeah. Yeah. And the MIRI people told me, we mentioned these names to other people, and they were shocked. They couldn’t believe that we got these names. I’ll take their word for that, that they’ve done something really impressive in government. It’s just, the problem is that compared to a name like Sydney Sweeney in the general public, right? These people are like zeroes, right? But I trust that they’re...
Holly 00:35:17
They got Marc Rowan.
Liron 00:35:19
Okay. That’s true. They got Marc. Yeah, no, I mean, look, their titles seem pretty legit, right?
So, I mean, it’s definitely something, and I think a representative kind of review is like the one from Ben Bernanke, Nobel Laureate, and former chairman of the Federal Reserve. It says, “A clearly written and compelling account of the existential risks that highly advanced AI could pose to humanity. Recommended.”
And that’s, to me, that’s manna from heaven to be like, okay, here’s a person who’s respected in traditional institutions who’s saying this is a recommended book, and it’s a compelling account. Because meanwhile you’ve got the mob on Twitter being like, still trying to gaslight us, nobody believes this man. And then of course, when I pointed them to Ben Bernanke, it’s like Ben Bernanke, he’s stupid. He’s nobody.
Holly 00:36:00
I don’t respect Ben Bernanke. What’s his Twitter handle? Yeah.
But I think though this is actually part of what happened with the backlash and the nitpicking from the rationalist community. Rationalism is an alternative community. It’s about not being part of mainstream institutions. It’s about having its own institution on its own, transparent, but very difficult rules.
And they don’t like people being able to use mainstream credibility and status in their group. I mean, they feel like they’re gonna get mugged. And I think this backlash is mainly about protecting the exclusivity of AI safety because it used to be - because it was just a niche topic.
It would be a pretty good filter, and then it could be talked about in a very LessWrong way. It took me a very long time to peel off the LessWrong talk from AI safety, because that was how I’d been introduced to it, but it’s so unnecessarily complicated.
But that was a filter for their community and it gave them a certain status to be able to play in that arena. And I did not know this at the time that I started trying to get people to do protests and stuff and they started attacking me. I mean, I feel a lot of empathy for Eliezer. It feels very similar. Suddenly there’s something wrong with what you’re saying, even though it’s what everybody was saying.
They see this as a thing that mainstream people can get and that mainstream people - and they do not want normies invited into their space and they feel like it’s gonna be taken away and they’re gonna lose their special thing. And just the very topic of AI safety used to be enough of a natural filter to have the kind of little community that they wanted.
They don’t want it to be mainstream. They don’t want people to understand it. They don’t want people to see that it’s not that hard. They don’t like Eliezer giving away the secrets.
Liron 00:38:12
I mean, the thing is there’s a lot of psychological theories we can have of why we think the community is the way it is, and I think that’s an aspect of it. Sometimes there’s the question of which of these theories is more dominant in explaining the situation? And I’m not sure how dominant that is. I think there’s something to it, but it might be the third place explanation of what’s going on overall. So we’ll have to...
Holly 00:38:33
I think I increasingly feel that this is the main dynamic.
Liron 00:38:39
Okay. Yeah, I mean, I don’t - I think there’s something to it. Maybe it’s second place. I don’t know if I’d put it first, but we’ll...
Holly 00:38:45
I wanna know what you put first place, but I don’t wanna get ahead of you.
Liron 00:38:47
Yeah. Let me tell you, I’ll tell you soon. Next topic.
But let’s go back to this book here. So I noticed, this is kind of funny, but I agree with their choice here. So if you look at this book, “If Anyone Builds It, Everyone Dies” only $14.99 on Amazon.
If you look at the back, there’s 11 different blurbs. So it’s all about credibility for the book. They don’t even tell you what the book’s about. They just wanna show you the praise. Okay?
So then you open the book and there is, to be fair, the jacket does kind of explain what the book is about. But then you go to the first page and all you see is praise for “If Anyone Builds It, Everyone Dies.” So it’s a page of praise, another page of praise, another page of praise, two more pages of praise.
So it’s just - they really want to hit you with the praise to be like, listen, this is a real thing. Okay, what we’re about to say, we’re not cranks. Which look, I think that’s the right decision.
Holly 00:39:04
I mean, they seem to think this would be their worry, and I’m glad that they dealt with that. It’s more important to deal with what the public thinks than what was supposed to be the community of supporters thinks. But I’m glad they prioritized that.
Doom Debates Transcript - Part 3
More Reviews Discussion
Holly 00:39:23
I think that there’s a trade-off with that stuff. In LessWrong if you show that you’re caring at all what the outside world thinks, or it’s not just about the outside world, doesn’t include tech, but if you show at all that you care about what the outside world thinks and you’re trying to get their love, they punish that.
Liron 00:40:20
Right. All right. Well, we talked about the respectable people who gave positive reviews to the book, like all these people in government. So let’s start going down the scale of maybe people who weren’t a hundred percent positive.
This is an interesting pivot point. Let’s go to Emmett Shear here, because he’s one of the prominent reviews here. Emmett Shear, former interim CEO of OpenAI, also the former founder and CEO of wildly successful startup Twitch sold to Amazon.
He also was at the party, so very interesting, intelligent guy and he writes Yudkowsky and Soares lay in plain and easy to follow terms why our current path toward ever more powerful AIs is extremely dangerous, and that is consistent with what he told me at the party.
He basically says that the book is very likely accurate in most or all of its claims, and he personally does disagree. When you get to the question of okay, so how do we actually do alignment? What does alignment look like? He starts to have his own disagreements, but the claim that we’re on this default path that looks extremely scary and risky, he’s on the same page.
He rides the doom train past all of the early stops, which I think is awesome. Right? I mean, do you have any thoughts on Emmett Shear?
Holly 00:41:40
I think it’s very consistent with what he at least said before, which is not something I can say about a lot of other people reviewing the book. And I appreciate it. I think it’s great. I don’t know what to say about it.
Liron 00:41:54
Look, he wrote a great review and that’s really cool of him to notice that the book is making important points and want other...
Holly 00:42:03
Okay. Let me give my positive on Emmett Shear. So I appreciate that Emmett Shear has both been quite - in his public persona especially, he’s very pro alignment and safety and he is very accessible as well, which is something I respect a lot.
Liron 00:42:30
Yeah, exactly. And he left this career of doing startups, successful startups, in order to pivot to doing an AI alignment research company. And again, you and I have all these bones to pick with him about how he’s going about alignment, but just high level, the fact that he’s saying, hey guys, AI alignment is this big problem, and society isn’t exactly on track to solve it, so we need a new company to research it. We’re wildly in support of that.
Holly 00:42:57
He does put his money where his mouth is. That’s definitely - I have critiques, but it’s not of that. I think he’s willing to do what he thinks is right, which is kind of rare.
Liron 00:43:11
Exactly. All right. And continuing on this theme of people being super reasonable, even if they may have some beef, some nitpicks, but they’re still being super reasonable about the review. Let’s go to Michael Nielsen.
So he writes here. This is a difficult book to review. There are significant holes at some points in the argument. These holes will be used by people who dislike the conclusion to justify ignoring or dismissing the book. However, the book also contains a large and important set of likely correct arguments, yada, yada, yada. I recommend the book.
So he’s basically for the book. I mean, that’s important to say that’s less ambivalent than a lot of other people have been, right?
Holly 00:43:57
Yeah, I really appreciated it. From the beginning, I thought, oh God, am I gonna have to be annoyed at him too? But no, no. I appreciated it a lot.
Liron 00:44:12
Yeah. And he also says the quality of the arguments put forward by the people hypnotized by the seductive glitter of ASI are far weaker. Thank you. I mean, that’s really the contrast to notice here, is that the other side’s arguments are far weaker and also contradictory with each other.
Holly 00:44:36
I mean a big, probably the biggest meta goal that I have with PauseAI is to reframe the burden of proof is like, you need to prove it’s okay to do this. Not like, we need to show that definitely everyone would die in order to stop you.
We don’t know what can happen if you build this crazy dangerous thing. There are just - there are a million ways we’ll never anticipate all of the ways in which this could go wrong. We need a lot more time to make sure this is safe.
If we could ever decide to - it might not be possible to ever undertake this because of the amount of unknown unknowns and it should be - it’s easy I think, to make an accurate case. Especially if you’re just talking about one scenario of what could go wrong. That’s kind of the default. I expect it to go wrong in some way when you do something like this.
But I don’t want to wait for luck to protect us. I want our laws to enshrine an attitude toward creating dangerous technology that could coup everyone or kill everyone. I want our laws to be that you can’t be trying to do that, except with a lot of democratic buy-in and a lot of safety, like a CERN for AI type thing.
Liron 00:45:48
Great. Okay. Yeah, and I’ll just finish Michael Nielsen’s thoughts. He thinks the harder alignment problem will be that when you have multiple super powerful agents, even if you got ‘em all aligned with their creators, it’s still super destabilizing and maybe the first person to build it will feel the need to defend their turf and go to war with anybody else before other people can slow them down.
So he kind of sees a decentralized multi-agent world, whereas MIRI and Eliezer and I guess me, we think there are even bigger problems like, ah, you’re not even gonna control the first AI. So he’s got a pretty significant disagreement on the details of why, if anyone builds it, everyone dies. But I’m just happy that his review has a good sense of perspective.
Holly 00:46:27
Agreed. Yeah, I really appreciated it and I liked Michael, so I was kind of - it was a nail biter, whether I would have to be irritated at...
Liron 00:46:34
Yeah, I like Michael too. All right, well done. Well done, Michael.
All right, now let’s go to a terrible one. I think we covered this on the launch party. Maybe it was me with another guest, but it’s worth mentioning it’s the New York Times, Stephen Marche, who is clearly out of his element. I’m not familiar with Stephen Marche. I don’t really read the New York Times book review section normally because I don’t read mainstream media. Okay, Holly.
So the whole article is titled “AI Bots Are Us: Who Will End Humanity First.” So they just dumped Eliezer Yudkowsky in together with these other random books. He was like one of three. At least they featured him at the top.
Let me just skip to the part that I think is representative of the review as a whole. It says chapter three, “Learning to Want” is typical. It begins with a conversation between a professor and a student about whether a chess computer wants to win at chess.
The authors then continue with an inexcusably lazy explanation of why the computer chess program does what it does, quote, “We’ll describe it as wanting to win. In saying this, we’re not commenting one way or the other on whether a machine has feelings. Rather we need some word to describe the outward winning behavior and want seems close.”
And then Stephen Marche continues from that vague foundation. They proceed to extrapolate not only fantasies of the end of the world, but frantic pointless questions like, “But wouldn’t AI keep us around as pets?”
Okay, so Steven Marche was basically saying they gave a vague foundation by defining wanting vaguely. Whereas from my perspective, it’s like, no. I mean, the dangerous thing that we normally think of as wanting really is the ability to drive a system toward a goal. And then you perturb the system, you try to get it off course and it still gets back on course and drives toward the goal. I think they’ve correctly defined to a very high useful degree what the dangerous part of wanting is.
Holly 00:48:36
Agree. This is very - it’s like Philosophy 101 kind of errors. Yeah. But this is - I hadn’t read that one yet, but it just does feel utterly typical, but it is just sad to see people flounder at the earliest, the earliest...
Liron 00:49:07
Yeah. And I mean, to me it’s infuriating. The disconnect between - so this person, he’s like, okay, the whole Doom Train, you know how there’s like 83 stops on the Doom Train, like Stephen Marche was - and that’s what I think there’s so many places to get off.
Well, I know the one true place to get off, it’s their definition of wanting. Okay. That’s where they screwed up. They gave a vague definition of wanting. It’s like, what are you talking about, man? That is not the place to get off the Doom Train if you’re gonna get off, there’s so many stops where you can reasonably get off. But...
Holly 00:49:40
That was their whole case. Yeah.
Liron 00:49:41
It’s just like we’re so screwed. Like the amount of stops in the Doom Train. It’s just - they’ll never, and of course if you’re somebody like Richard Hanania. Yeah. Some people think the Doom Train is like a conjunction fallacy and they’re like, listen, it’s enough for me that there’s a lot of stops. Okay. One of ‘em bound to be the right stop.
Holly 00:50:03
Yeah, I don’t think about this as much as I did in 2023, after ChatGPT and like, oh my gosh, this is all - this is gonna happen now, in my lifetime instead of in the future. The amount of stuff that seemed like makes no difference, philosophical issues that have really surfaced as extremely important to understanding AI safety.
Huge. I mean, consciousness, the hard problem, there’s a real world thing. I’m talking about that every day now. Like there’s ideas about teleology that I was accustomed to talking about a lot in the context of evolutionary biology. That suddenly people will argue that means that AI can’t be dangerous, it can’t be smart enough, it can’t be real. And if it can’t be real then it can’t do any harm.
Liron 00:51:08
So, yeah, while we’re dissing the NY Times, I think it’s important for me to bring out my tier list of NY Times AI journalists. So at the bottom we have Steven Marche and Cade Metz. Those people are just consistently terrible at covering the AI space.
And then on the top we have Kevin Roose. I personally think Kevin Roose is decently good at covering the AI space. I just think the rest of the New York Times team is just insane.
And there were some good reviews of the review on Twitter where somebody had a nice job criticism where he is like, man, it must be great to be a New York Times book reviewer. You just don’t have to read the book at all. Because there’s some claim in the review that was just factually false about the book.
It’s not worth getting into more detail about it, but yeah, we’ve said enough there.
Let’s go to one that we’re more disappointed and surprised by. Let’s go to Will MacAskill. Right? I think you had a sharp reply to him.
Holly 00:51:50
I thought about this a lot. I was very upset because - so Will, just put a little context on Will. Will is a very big figure in effective altruism. He’s a professor at Oxford, think he’s the youngest philosophy professor at Oxford ever. He wrote the trade book for EA “Doing Good Better,” and then he wrote this big book about longtermism.
Liron 00:52:19
By the way, “Doing Good Better,” highly recommended book. One of the best books I’ve read in my life, quite frankly. I recommend it to everybody.
Holly 00:52:25
Yeah. So he’s the big EA guy and he really moved hard into AI safety or thinking about the future, protecting the future from existential risk was his real focus in the last like five years, more than that, eight years. So he knows a lot about this. So let’s just read the review.
Liron 00:52:48
So he says, here’s a mini review of “If Anyone Builds It, Everyone Dies.” I’ll at least read his TLDR. He says, I found the book disappointing.
I thought it relied on weak arguments around the evolution analogy, an implicit assumption of a future discontinuity in AI progress. I think he means discontinuity upward, right? Conflation of “misalignment” with catastrophic misalignment and that their positive proposal was not good.
I had hoped to read a Yudkowsky/Soares worldview that has had meaningful updates in light of the latest developments in ML and AI safety, and that has meaningfully engaged with the scrutiny their older arguments received. I did not get that.
I think if a younger version of me had read this as my first exposure to the field, I’d have either bounced off altogether if I’d correctly assessed that the arguments are not good or come away with substantively the wrong strategic picture. This is a shame because I think the risk of misaligned AI takeover is enormously important.
And then he does go on to elaborate on why he disagrees with the evolution analogy and why he disagrees with discontinuities in AI progress. So I just read you the TLDR of his review. What are your thoughts?
Holly 00:53:59
I didn’t even - I mean, you can look at my Twitter for if you wanna see more in depth, but I didn’t even engage with the evolution part, even though I have a PhD in evolutionary biology.
And it was this kind of - to be frank, I have never really understood what people in AI safety think that argument is or means. But he’s wrong here. I’ll just briefly say why he’s wrong because he’s giving a bunch of differences in natural selection of humans versus ML. But none of them are the crux, what he’s really claiming to disagree with is the principle of Goodhart’s Law.
And I know he wouldn’t disagree with that. So it just felt very - I don’t know, that felt - I don’t know if maybe because I understand that topic better that it feels bad faith to me. But that one felt kind of bad faith to me. I felt like that was just waffling. Like, oh, there’s a reason you could poke holes in this, so we don’t have to agree with it.
But the thing that really bothered me was further down, he talks about what he thinks is most likely to happen, and there’s this part where he talks about making deals with AIs and how we can know - we don’t have to know exactly what’s going on and exactly what they value and all that in order...
Liron 00:55:23
Yeah, let’s - ‘cause I agree that’s probably the single weakest thing out of all the different things he wrote. This seems like the clearly the weakest.
Okay, so Will MacAskill is saying that AI is likely to be risk averse because that way it can increase its odds of success by not coming out with a big risk that might fail. Like making a bid to take over the world when it doesn’t have a hundred percent chance of succeeding.
So, to quote part of his language, he says the risk averse AI still wants power and self preservation, but that didn’t result in human extinction because it prefers deals that give it a high chance of some amount of power to attempts at takeover that give it some lower chance of lots of power.
We can have misalignment without catastrophic misalignment. Maybe we won’t be able to make deals with AIs. I agree. That’s a worry. But then the right response is to make sure that we can. Won’t the super intelligence have essentially a hundred percent chance of taking over if it wanted to? But that’s again invoking the discontinuous jump.
So Will MacAskill, he has this mainline scenario where the jump is gonna be slow enough and the AI is gonna be risk averse and we’re gonna be making deals and you disagree.
Holly 00:56:56
So the main thread that I think the real crux of where we disagree is Will is representing this cluster of people who think this way. They’re trying - they’re taking very seriously, you give them a bunch of parameters for this scenario and then they’re like, how do you actually maximize the area under the curve?
Well given that this function behaves like this and if we could tweak it, then these parameters are acceptable and they’re really trying to maximize the area under that curve. And I’m trying to think about what’s going to happen.
Like, and to them, if they can make an argument that, no, but I have a scenario where in this scenario, if you take these starting conditions, these things happen. You could imagine a world where, because scaling progress is smooth and we can always check in on what’s happening, it would be possible for us to gauge appropriately what the power level of AIs are and make it so that the incentives are such for them to make deals with us even if we’re not aligned.
And this story is - they’re getting no ding for how many ifs there are in the story and how particular it is. But because they can kind of tell the story end to end, it’s fine. And they think, well you just have to deal with that. Whereas if you kind of reply with, I’m worried that that’s not robust at all. There’s a lot of assumptions, but even if they were all true, you could just fail to pull this off. It’s just not robust. It’s not giving any time for this whole thing to play out.
Liron 00:58:55
Yeah, exactly. Like everything Will is saying, right? It’s like, let’s even grant, okay, Will, that’s a great mainline scenario. I personally don’t think that’s a mainline scenario, but let’s say even if it is, aren’t you reserving 20% chance that that’s just not the scenario and the scenario is very different because your key assumptions don’t hold, because you shouldn’t be that confident in them?
Holly 00:59:10
And it also annoys me how at times it feels like in this review, he’s only addressing will everyone die? But then it comes across as saying, well this book is wrong. This shit’s not gonna happen.
When if you look closely, it’s like, will everyone die? Or will everyone die immediately? I don’t know, as if this is important, we need to know which scenarios will have everyone immediately obliterated versus dying slowly over 50 years. It’s so not the point, so not caring about what happens.
I don’t wanna be upset at him for making an academic review. ‘Cause that’s what it is. But the proportions are just so off and the thing that annoyed me more than anything else is this line of, well, what if the AIs don’t wanna make deals? That could happen. We just have to make sure they can.
Liron 01:00:08
Yeah.
Holly 01:00:09
How isn’t that the alignment problem?
Liron 01:00:12
It’s like you’re dealing with something that’s a problem you don’t understand, right? It’s like rocket engineers being like, oh yeah, okay, so we’re just gonna have to make sure that the rocket just always has enough weight that it can get it to orbit, just always has enough margin of error that it can get the weight to orbit. It’s like, okay, that’s always going to be a very hard constraint satisfaction...
Holly 01:00:32
It’s like what else do you think rocket science is? I know. This is - but you can’t. He knows you can’t just say, we have to make sure AI is aligned, but this is basically that. We just have to make sure we can make deals with the AIs.
Doesn’t that require significantly constraining them to be sure that it’s gonna be in their interest to make deals or that they’re gonna be competent to make deals according to what we think is the right calculus? Or it just - this is the alignment problem.
Liron 01:01:03
No, I agree. Look, I agree. I find this level of discourse evidenced in this tweet alarming for the same reason as you. Okay. Let’s go to some of the comments here. ‘Cause this is a whole discourse. This tweet got 104,000 views on Twitter, which is a pretty high amount. It got 559 likes.
All right, so the first comment in my sorting is actually from me, it’s saying, “Hey Will, you make some sharp points here. I’m planning a Doom Debates episode on the reception of ‘If Anyone Builds It, Everyone Dies.’” So he was invited to come be part of this episode or do another episode and just talk about his review. Basically debate it out. So I haven’t heard from him on that front.
But let’s go to your reply. You said, “These criticisms read to me as ‘development could go in a less catastrophic direction’ and ‘but pausing could conceivably have a bad outcome,’ rather than engaging with the overwhelming thrust of what society should do when toying with making catastrophically dangerous tech.”
And then you escalated it to a quote tweet. You quoted the part where he said, “Maybe we won’t be able to make deals with AIs. I agree. That’s a worry. But then the right response is to make sure we can,” and you’re like triple question mark. “This sentence alone should be disqualifying from serious discussion.”
Fair. Fair. Fair enough, fair enough.
And then you also had a quote tweet to me where you say, “Really disappointed that this is what Will MacAskill is doing now, but I guess when you wanna work on ‘post singularity governance,’ you don’t wanna switch to trying to stop the singularity. This whole description of super alignment reads like a parable on greed and hubris.”
Okay. I mean, I agree with the part that his focus on post singularity governance is pretty optimistic. Grace, definitely. Like assuming that there will be such a thing.
Holly 01:02:59
No, but it really disappointed me and I just felt like the lashing out is kind of just - I just like the frustration of realizing, oh, we’re just not in good faith anymore and this group is going to reinforce each other on that.
And it used to be a place where I really felt like we were all trying to figure things out and we were all really trying to do what’s good for the world. And I really did consider Will a huge ally on that, leader in that and to see this turn on AI safety where...
I mean, I don’t - I’m just gonna say my position on it. I mean, I feel like they just - that group kinda went in on a strategy of working with the labs or in the case of Anthropic, kinda being a lab and they just don’t wanna give that up.
And even though there’s a better solution now, which would’ve been a better solution from the beginning if we could have rallied public opinion or we could have had really engaged lawmakers on it to care about the substance of the issue and to care about our safety. And I just think a lot of people aren’t ready to give up on working hands-on with the...
With the machines, they love working with the models. They have these dreams of what’s gonna happen post singularity. That’s what Will does now is work on post singularity futures.
Liron 01:04:00
Okay.
Holly 01:04:02
And they’re just so invested in that, that I think it’s just - to say, oh hey, let’s do pause - I think pause is compatible with thinking about these things and also working on technical solutions. I think pause is helpful in two ways. One, if it was achieved, it would give more time for that to be done safely and well and under democratic control.
Two, advocating for a pause makes it easier for their solutions to be stomached in society. It’s more of a compromise. It becomes to just work on technical safety and I’m happy if in trying to get to a pause, we just make things a little safer, that way that’s still better than nothing.
So I think these things can work together, but in practice, they for some reason don’t wanna support pause related stuff. I don’t really know why. I can speculate that it makes them feel bad or they don’t feel that they can do that without undercutting their case or I don’t know. But it’s extremely disappointing to me ‘cause I think it’s just textbook motivated reasoning.
Liron 01:05:40
Yeah. Now, personally, there’s like - I mean, I just can’t follow you all the way down from okay. Will wrote this and then his motive is this. I mean, you gotta be careful to really only give an ounce of psychoanalyzing for a pound of more...
Holly 01:06:02
Not all based on this review. I have other reasons for thinking this. No. And especially in this discourse, it’s very outlawed to think about motives.
Liron 01:06:14
Yeah. I noticed that was a tweet you had so we can zoom out into that, right? So that, I mean, that is, to be honest, becoming kinda your specialty, right? That’s one of your special points that you think is underrated, right? And so you have to signal boost this point of like, look at these people’s motives.
Holly 01:06:29
Yeah. Well, I think - it wouldn’t fly in another community. I think that people - just mostly people call bullshit more than this, and they’re not as sensitive to accusations of you’re not being charitable to me.
But that’s deadly in rationalist discourse. You can’t be doing that, even though people do, of course. They do this kind of thinking all the time, but on certain topics you’ll be called out and it’s not allowed.
And whether someone is really doing their best to do the things they say about keeping people safe - one of these topics. You’re not really allowed to question that. Or if that person can string together something that could happen or could be real, this story, this crazy Rube Goldberg machine of the scaling story and how, but actually with deals and AI control and all of this, it’ll turn out - so it’s actually silly to try to stop it because...
So just in that review, there’s a lot - there’s good evidence for one comparison. There’s a lot of space given to this Rube Goldberg machine. Like how, because things are smooth and scaling and because the AI will be risk averse and all that. We can know we kind of got it. We should keep going.
Pause is dismissed with like, couldn’t that disrupt the race dynamics? Can’t do it. That’s not - and this is very hard to prosecute. You have to be willing - I have to be willing to take a hit to my debate score in order to say that.
But it’s just to the point where that is - that’s my true belief of what’s going on. It doesn’t - I believe in myself and I trust myself because I tried within in sincere, good faith and innocence for a very long time to make sense of these positions.
And they don’t make sense because it’s just people are not being - they’re not being honest. They’re not - maybe they’re not even aware of their own motivations. There’s not a lot that’s going to encourage them to be honest. Nobody can ever call them out. Nobody can ever access their - they’ve got this great position where it’s protected. They’re allowed to just have whatever dissonance or pursue whatever...
Liron 01:08:06
Right. I agree. It is - it’s a very socially comfortable niche right now to be like, I’m taking all the perspectives into account. I don’t think any one of the perspectives is that clear. I think it’s fine to just work on the good scenario. I think it’s fine to work on post singularity governance because nobody knows that it’s not going to go well.
That’s like - there should be some people working on post singularity governance and also if I do work at an AI company like Anthropic, look, there’s enough uncertainty. Maybe Anthropic will have a good role to play also. I make a ton of money and I’m the hottest part in the tech industry.
I mean, it’s just there’s a lot of social positive reinforcement and comfort. It doesn’t feel like you’re taking that big of a risk. I mean, there’s some people like me and you who are out there yelling at them and protesting. But so far the overall magnitude of the social bubble is just balanced at plenty of positive reinforcement.
Holly 01:09:06
Yeah, and you can be - they’re free to be as tough as they want on arbitrarily, unfairly tough on pause type positions and you can’t be - first, they’ll shout you down if you don’t have qualifications in ML in the first place. And try to say that because of that you can’t say what’s gonna happen. You also can’t even have moral and ethical opinions on AI because of that.
Now that’s not so bad, but they just - they continue to find ways that nobody can confront them. Nobody can take away - they have policing mechanisms within their own world, but they - nobody can take away the cachet of this thing, and nobody can talk about how precious it obviously is.
You can’t say like, isn’t it great to make all that money? No, they’re just doing it ‘cause it’s the right thing. They deserve a little something. It just - what about all the status of having this job? Just no, that doesn’t...
Liron 01:10:58
Right. Right. Okay. Well I’ll give you my take on your take and also tell the viewers where I stand. So I mentioned at the party. I think you’re a thought leader because you often come out with these statements that I hadn’t considered, or maybe even I hadn’t allowed myself to think. Right?
That’s kind of your specialty is thinking outside the box, moving Overton window. You were the one who a couple years ago was saying, “Hey, we should go out there with a megaphone and protest.” And now I’m on that page too, but if you hadn’t said it, I wouldn’t have said it. Right?
And I pride myself on sometimes noticing things like that. Right? I should have been the guy who said, let’s go out on megaphone and protest. But no, but it definitely wasn’t me. It was you and you’re the ones who actually planned the protest. I would’ve been like, yeah, that’s a good idea, but I wouldn’t have planned it.
But you planned it, so I thought you were a thought leader then, you’re still doing the protest and I think you’re a thought leader. A lot of the stuff you’re saying on Twitter of like, “Hey, these people are pretty messed up and a lot of their social dynamics and motivations are pretty messed up.”
I think you’re directionally correct. I just don’t think that I follow you all the way. I think you may be racing ahead with some of these particular points, and I know you’re saying that you tried not to, and you’ve been down a long slope where you just got more and more convinced. So maybe you’re right, but I just want the viewers to know that I’m just not following you all the way.
Holly 01:12:08
You don’t have to. I mean, yeah, this is a thing that I do now is Overton window pushing. And yeah. I go a little - I’m hoping to give shade for other people and other people don’t have to go as far as I do.
I don’t say stuff I don’t think is true, even though that could be effective for the Overton window pushing strategy. I don’t. I just really - to be who I am, to say the things the way that I say them, they need to be what I really think.
But yeah, and I have also probably where you can’t follow me is more the fiery passion of it. And that does have to do with specifically me being wronged and I don’t expect people who are just trying to figure out what’s true to feel that way about the issue.
I felt gaslit for a very long time and gaslit in a very painful way. I mean, so not only were people saying - they didn’t wanna say, I just don’t wanna do protests. They were telling me that it was bad and actually I was bad. And they really - they were going for the very thing that compelled me to do it, was doing the right thing.
And they were going for that motivation, and they were nasty about it and there was no repercussion for them.
Liron 01:13:28
So fair. That’s so fair. And if you look at it today in 2025, because there’s been this little trickle of protests, right? I mean, let’s be honest, it’s only a trickle, but it’s something because that’s been happening now for two years and it’s gotten a disproportionate amount of media coverage and everybody knows it’s a thing.
I feel like you don’t hear that many people being like, no, oh, don’t protest. Because everybody’s like, yeah, some people protest, right? You totally - that’s called moving the Overton window.
Holly 01:13:49
Oh yeah, no, totally. And it happened after the first protest, people were really trying to get me not to do the first one, and then as soon as it happened, they were kinda like, I guess this happens now.
Liron 01:13:59
Whatever, some people protest. Yeah.
Holly 01:14:00
And then they kind of moved to why it was cringe, but they weren’t like, you can’t, it’s gonna reflect on us somehow. And you can’t, and this is gonna...
Liron 01:14:10
And it’s all - I would also call it status quo bias. Right? Because if we had gone back in time and said, okay, imagine a future where the protests are already happening, would you really be like, stop the protests?
Holly 01:14:18
No. Some of my anger that pushes me to be willing to just be out there with a lot of these ideas is because I was very trusting and I really believed in the ideals of effective altruism and rationality and I really thought the people around me were upholding that standard and they weren’t ready for protests.
I thought they would be. I thought we were all just waiting to have the opportunity to talk to the public because that’s what people would say. I asked that question like 10 years ago, what about public stuff? And the answer I got was like, oh, you’ll look crazy. And it’s just not - it’s not time yet. But one day, for sure.
I thought that day had come when after we get these polls, after the pause letter from FLI showing such high agreement with the ideas in the letter. I thought, okay, yes, time to do it. Just suddenly nobody wants to do it, but they don’t wanna say, I don’t wanna do it.
They - and I don’t even know if they’re lying. I think they might just be not very honest with themselves on this, but they don’t wanna say, I don’t wanna do it. But they have to have some reason that actually it’s bad to do it. And they need to...
Liron 01:15:28
Yeah, totally, totally. So, yeah, so massive respect because you’ve demonstrated this ability - when you moved to the Overton window, it means you have to use your brain to logically come up with something that should be done.
Even though there’s no social proof right there, there’s no social feedback, but that’s the thing to do. But you just identify it as a thing to do and then you do it. That is super rare. So yeah. So props for doing that.
I think there’s a natural segue here to the review that I personally think is the most damning, which is Clara Collier from Asterisk Magazine. Asterisk Magazine - normally we like Asterisk Magazine, right? There’s a lot of...
Holly 01:16:00
I like it. Yeah.
Liron 01:16:01
Yeah. In the rationality community. I think even Clara Collier, I don’t know her well, but I think that she’s considered a thoughtful writer in general, right?
Holly 01:16:12
Yeah, I mean, I don’t know Clara that well, but I did hang out with her a few times at Constellation back when they would let me in there, which is a workspace for sort of the kind of research that Will was describing that happens there.
Liron 01:16:24
Exactly. So here the review title is called “More Was Possible: A Review of If Anyone Builds It, Everyone Dies.”
And the reason it’s a segue is ‘cause if you look back at Will MacAskill’s review, Will writes, “I had hoped to read a Yudkowsky/Soares worldview that has had meaningful updates.”
So he is using that same terminology of updating your belief. “I’d hoped to had meaningful updates in light of the latest developments in ML and AI safety, and that has meaningfully engaged with the scrutiny.” So Will’s pulling the same position of Eliezer Yudkowsky never updated, he’s just been stuck. All this evidence is coming at him that he’s been ignoring.
And that’s right in the first couple sentences here of Clara Collier’s review. Clara Collier’s language is, “The book is full of examples that don’t quite make sense and premises that aren’t fully explained, but its biggest weakness was described many years ago by a young blogger named Eliezer Yudkowsky.”
Oh snap. You got a colon. “Both authors are persistently unable to update their priors,” so everybody is suddenly a Bayesian reasoning expert in their reviews thinking that Eliezer has failed to notice that there’s more evidence of the hypothesis that we’re not doomed compared to evidence of the hypothesis that we are, and so that he should multiply an odds ratio and come out with a higher belief that we’re not doomed.
Okay? There is actually - as an intermediate Bayesian myself, somebody who’s looked a little bit into the math of Bayesian updates, guess what? You can still encounter different kinds of evidence and come away with the same probability that you had before you saw the evidence and still have updated. Okay, so that’s a technical correction right off the bat.
Holly 01:17:56
Yeah. Clara is saying that sassy, like he should have listened to Yudkowsky. I thought, are you really gonna call him a hypocrite? Be like - I felt like it was - it felt cruel to me to kind of imply how far you’ve fallen and you’re not living up to - it’s worse than just not updating, you’re not living up to what you said and it just felt - it was not necessary to get in that little flourish. So that I just felt was mean and then...
Liron 01:18:32
No, totally. Yeah. It was a low blow. I mean, I guess if you’re writing on a snark forum, then sure, it is good snark.
I mean, look, so the problem for me really is that technical Bayesian math - I mean, let me unpack that a little bit. So when you have a Bayesian prior, it’s not just a number that you then go higher when you encounter certain evidence.
A Bayesian prior is actually a larger mathematical object that encodes what different pieces of evidence will then mean to you. So Eliezer could have told you a priori, that if he sees a piece of evidence of an AI that hasn’t crossed the super intelligent threshold yet, and that AI is nice and safe, that doesn’t necessarily constitute evidence to update on. He could have told you that beforehand, his conditions for changing his probability weren’t met.
That was actually encoded into the structure of his Bayesian prior. So this is getting a little bit pedantic, but Clara Collier, you’re writing this review. He really should have had that level of rigor. When she says he didn’t update, no, sorry. That is not correct. I would argue that Eliezer has in fact updated his priors despite the fact that his probability of doom hasn’t gone down.
Holly Elmore 01:15:35
I’m gonna go even further and not implicate Liron when I do, but I sometimes wonder when you hear lines like this that are repeated among this whole group, if overtly or covertly, they are coming from Anthropic and they’re the party line. They want this population of EA rationalists, they want them for talent and they want their existing employees to believe in the mission and everything.
I don’t know how much enforcement it even takes, or if it’s just Dario kind of puts out, or somebody at the top puts out apologetics like this and then they all just pick them up and they sort of know that this is the right thing we’re supposed to think.
Holly 01:16:23
It feels propagandistic, I guess. It’s like saying Eliezer can’t update, Eliezer’s not engaging with the new information. Eliezer’s not really that smart anymore, he’s kind of a has-been. All of this for people who are there because of Eliezer and liking Eliezer’s style of thinking and who can judge themselves based on Eliezer’s more in-depth writing.
That’s more in the style they liked about how well he’s keeping up with these developments. He’s given answers to why he doesn’t think that current ML meaningfully updates him. He has given those answers before. I don’t even know what answers conceivably he hasn’t given that they might not have seen.
For them to push the line that he doesn’t get it - that if you’re a real expert, which now means essentially working at an AI lab, that is what’s required to be an expert - then you’ll see that actually this is how it works. So I’m just gonna throw that out there as part of the whole motivational ecosystem.
Liron Shapira 01:17:32
Yeah, this idea that the community has left Eliezer behind, we all updated, I guess Eliezer didn’t update, right? But all these other smart people, they updated, and Eliezer is just stuck here, not updating. He’s just gonna be shaking his fist forever. And we’re all so smart and we’re setting up post-singularity governance.
But from my own capabilities of just looking at how much sense people are making and looking at the various communities of people and seeing what do you think about AI - I listen to a ton of podcasts for one thing, so I absorb information that way. It’s not like I see all these other people and I’m like, wow, that person has so much insight from absorbing new evidence that Eliezer hasn’t absorbed.
No, from my perspective and of course Eliezer’s own perspective, to the extent that I can be objective as somebody who’s agreed with Eliezer in the past, I think Eliezer is keeping pace with the important developments. I think the people around him are incredibly intelligent.
Liron 01:18:26
As an example, I had Steven Byrnes on the show. Steven Byrnes is not like me, where I’m so much an Eliezer fanboy and I just think Eliezer is more right than I can ever be. I’m just never going to have as much knowledge and insight about the subject as Eliezer already does. That’s my perspective about myself.
Steven Byrnes is coming at it from a very different angle. He’s a math contest winner, Harvard genius, postdoc, won a bunch of awards, top caliber brain, came at it from his own direction. Does his own cognitive science research, has his own alignment program, but he still thinks that Eliezer’s conclusions are timely and accurate. There’s social proof of that. So I don’t understand how you can be so smug and call Eliezer somebody who hasn’t updated.
Holly 01:19:06
My impression from being in that world is that there’s kind of a conspiracy. I don’t know how accurate that is as far as somebody’s directing this. There are people who are benefiting from it and at some level directing what everybody’s supposed to think about what’s going on in machine learning alignment.
But there’s just a consensus, and a more neutral explanation of it is they’re just attracted to stuff that feels more complex. They’re attracted to stuff that feels like it’s more hands-on. And it ends up becoming like, whatever your research is, the cool shiny things are working with the models and working with big data sets and getting time from Anthropic to run experiments. That’s what’s really cool.
Holly 01:20:16
Richard Ngo once wrote a note on either EA Forum or LessWrong where he - I couldn’t believe he just said the quiet part out loud, but I think about this a lot because I think it’s going on a lot. He basically said right around the time of the first Pause AI protest, he said there’s two pathways we can go down. One is cool, empirical, we’re building things, we’re working on things, everybody’s excited, everybody’s having fun, we’re working with the models. Two is it’s like environmentalism and it sucks, everybody’s depressed because they have to do activism.
He just said it. But I think a lot of people have this view that it’s almost like a form of evidence, how much more fun of an environment it is to be working on, and how could it not be the right thing to do? They’re techno-optimists, they’re finding solutions, how could it not be right to be in this environment that’s invigorating and you’ve got a lot of support?
Holly 01:21:11
The whole beautiful EA community basically got co-opted into doing this stuff to support Anthropic. It was a beautiful community and I mourned it a lot and I couldn’t understand what was happening as it switched over until eventually I was out because I made the wrong moral call according to them.
Liron 01:21:34
What was the incident where you were out because you made the wrong moral call?
Holly 01:21:36
I felt like what they wanted was for me to make my wording so tortured and rat-style that basically it wouldn’t appeal to any member of the public. They said that I could do public stuff, of course we can do that, that’s good. But they would always have a problem with the execution.
People would say the signs gave them so many problems because they didn’t want to write a sentence that short. People would put footnotes on the signs!
Book Review Critiques
Liron 01:22:04
So finishing up with this review, I do think Clara Collier’s review has a lot of consensus that it was an especially weak review. And I think that the “not updating” is just a taste of it being weak. But let’s move on. Let’s go to Vox.
I’ll just read the title quote: “AI will kill everyone” is not an argument. It’s a worldview. Mic drop. That’s the title of this Vox article. It’s by Sigal Samuel, who I think normally is considered a respectable Vox writer who normally can analyze things decently. And then her subtitle is: How rational is Eliezer Yudkowsky’s prophecy? You know anything about Sigal Samuel?
Holly 01:22:46
No, she did write something that was not liked about the EA community, but which at that point I felt was fair. That was the only time I’ve paid attention to reading her.
Liron 01:22:58
Well, I think the title speaks for itself. “AI will kill everyone” is not an argument. Yeah, it’s not an argument because arguments connect claims to evidence.
Holly 01:23:07
It’s a conclusion.
Liron 01:23:08
That’s just a claim. It’s a claim, it’s a conclusion, right. So if your title is coming from the domain of epistemology, right? Epistemology is the study where we take those kinds of phrases and classify their epistemological status - is that of an argument or a claim or a worldview - and then you just bungle it in the title. How good is the post going to be? Turns out it’s not good.
Liron 01:23:34
I’ve got one clip from Paul Crowley who goes by CipherGoth on Twitter. Paul Crowley says: “I’m right to stop reading the Vox review at the point where she wildly misrepresents Eliezer’s article in Time.”
He quotes Sigal Samuel saying that Eliezer says “the only reasonable thing to do is stop all efforts to build it, including by bombing data centers that power the AI if necessary.”
So Paul Crowley is just pointing out that Sigal Samuel is misquoting Eliezer as wanting to escalate all the way toward bombing when really what Eliezer said is we should escalate to an international treaty. There’s no way to solve this short of an international treaty. And I don’t mean the kind of international treaty where you can violate it and nothing happens for 50 years. I mean the kind of international treaty where violating is severely punished, like we will take out your data centers if you violate the treaty.
That’s what Eliezer said, not “I’ll get angrier and angrier until I bomb you.” Very big difference.
Holly 01:24:34
I mean, to be fair, I really wish he hadn’t said it that way. I knew how to interpret the Time article because he has this long history of admiring Orwell. And Orwell is always like, if you’re talking about political violence, you can never euphemize it, you have to be honest about what it is and state violence and you have to call it what it is.
If he had just said we would have a treaty where kinetic action would be part of enforcement, nobody would’ve batted an eye because that’s the way you normally say it. But to him it was important to say that. I think he was wrestling with turning to the state to try to solve something. But it did confuse people.
Liron 01:25:15
There’s also a difference. Sometimes you get an international treaty that’s kind of toothless and it’s more just for show. And then you also have other international treaties like nuclear proliferation where somebody violates it and you’re like, okay, we really need to consider attacking immediately. So there’s different grades of international treaty and I think Eliezer wants it to be very clear - no, no, no, this is the serious kind where our lives are at risk.
Holly 01:25:40
He wanted to address when state violence is justified, because that’s a big issue to him as a strong libertarian. Most people just don’t need that at all. So I do understand why they would be like, why would you even say you bomb data centers if you mean that would be the enforcement of a treaty? I understand why some people became confused. I do feel that the Vox journalist had a higher responsibility.
Liron 01:26:04
Yeah, so I think that’s a low blow. And just to go back to the title of “it’s not an argument, it’s a worldview.” What the hell? It’s like, oh, I’m just going to dismiss anybody who would dare to say AI will kill everyone. You can’t say AI will kill everyone, okay?
This is what smarter, more technical people have made a similar claim. Like I have a whole episode from last year - Arvind Narayanan, one of the AI Snake Oil guys, he was very big on how we can’t make policy based on probability estimates, man, we have to make policy based on something else.
Holly 01:26:36
What do you ever make policy on?
Liron 01:26:37
Right? The same epistemic flavor as “AI will kill everyone is not an argument, it’s a worldview.” Listen, the epistemics are fine. We’re just debating what the probability is.
Holly 01:26:46
I think there’s a lot of cultiness in the AI safety world. And people are not wrong to pick up on that. But we need to distinguish the argument from that feeling. I think a great thing we can do is just have lots of people be aware of this so that it isn’t something that can be associated with one weird community.
I don’t understand how you could get it from this book. I do not think this book sounds culty at all. But the Sequences, HPMOR - Harry Potter and the Methods of Rationality, which Eliezer wrote to recruit people to work on AI safety - they are all an invitation to be in a cult. There’s sort of like you are now in the Bayesian conspiracy, now you’re special.
And I do understand people are right, I think, to have an immune reaction to that if they get that whiff. A lot of people have that whiff pretty strongly from knowing the rationality community or knowing EA. And yeah, they’re not crazy. The arguments need to stand on their own, but if you kind of only get 10% of it and you’re getting this huge wave of like, there’s something else, they want me to do something because of this and I don’t want to, then I do understand that.
The Circular Firing Squad
Liron 01:28:12
Now the last and maybe most important topic of the show that we’re gonna get to is the Circular Firing Squad. The idea that we have this community and we really are aligned to this mission - AI risk reduction, saving the world to be a little hyperbolic, not even that hyperbolic. We’re really just trying to survive the next decade or two.
And yet we spend most of our time nitpicking each other and sending ambiguous signals about what we believe. Do you agree with that general thesis?
Holly 01:28:40
Yes.
Liron 01:28:43
So you could call it the circular firing squad. I remember Barack Obama called his party a bit of a circular firing squad sometimes - Democrats don’t get on the same page the way Republicans do.
I would maybe use a different metaphor for LessWrong, because I don’t think that you get a food fight dynamic on LessWrong. It is quite polite and civil. LessWrong has a lot of stuff to its credit. It is quite open-minded. There’s actually always somebody who will stand up for the good side to steelman any position.
So I wouldn’t feel like it’s a circular firing squad, but I feel like it’s on the spectrum toward circular firing squad. It’s not all the way toward aligning on important things. It lands on a place that I call the crab bucket.
Liron 01:29:24
It’s like everybody can stay in the bucket and have a polite discourse, but the moment one crab tries to climb out and says, “Hey, I think there’s consensus here. I think there’s an important belief that we can build mutual knowledge that we all agree and we should shout out to the world” - everybody pulls them down and says, “Well, hold on, hold on, hold on. Let me give you a trillion caveats. Let me write a million footnotes on your sign. Stay in the bucket with us. Discuss forever. Never finish the discussion.”
And I guess in that sense, it’s what people criticize about the field of philosophy. You can never win the field of philosophy. You can never get the award where your philosophical theory is correct. You just have to wait for other philosophers to die. And then maybe in a couple hundred years, people will just forget the other philosophers and they’ll just think that your claims are obvious.
That feels to be the case on LessWrong. You just have to let time pass and see what’s obvious. But you can’t just unite people because it’s always going to be a crab bucket.
Holly 01:30:16
Yes, that rings true for me because it’s not like they kicked me out. I was welcome to stay in the bucket. But I was like, guys, why aren’t we leaving the bucket to do this thing? And they never say “we’re never gonna leave the bucket, that’s not our intention,” but that’s what happens.
It would’ve been fine if I had just been like, “Oh, yes, I see. There’s actually many compelling reasons not to leave the bucket” and sat back down. I would’ve been welcome to stay. And then they would’ve been nice to me and considered me an insider.
LessWrong’s Response to the Book Launch
Liron 01:30:54
I’ll give you some backstory on this because there is a personal element involved. I had this realization recently when I made a post of my interview with Yudkowsky. I posted the full transcript and the video on LessWrong, and I posted it right on book launch day. Check it out guys - it’s a longtime LessWrong person, myself, interviewing the founder of LessWrong with a deep dive through rationality and all the different concepts he talks about in the book and how the public’s way to respond to the book.
It was a textbook LessWrong guy interviewing the book author about LessWrong and the book. It was kind of a canonical LessWrong post to celebrate the book’s launch. And I post it and I’m like, okay, yeah, it’s getting some upvotes. I got like 90 upvotes total. But the thing is that a LessWrong post that’s doing well and has community support usually gets like 200 to 500 points. So this is kind of a lukewarm reception.
Liron 01:31:51
And then the next day happens, the book’s launch week transpires, and I look on the front page of LessWrong. And the post for a while that had the highest votes out of all the posts for a few days was somebody saying, “Yeah, I somewhat like this book. I was thinking whether it would’ve been better if somebody else wrote the book. But I think I’m pretty much a no on that. I think it’s probably fine that Eliezer wrote the book, even though I disagree with a bunch of stuff.”
And that was the highest voted post for a while. It got more than twice as many votes as my post saying, here’s the interview with Eliezer talking about all the stuff related to this book launch.
Liron 01:32:25
That’s what made me think - why can’t I go to the LessWrong homepage and see some kind of section that’s celebratory of the book launch? And then to be fair, there was a banner. The banner didn’t say it’s important that you read the book, we support this. It was just very calm, like this book is out. Oh look, this is what we’re highlighting this week, there’s a book out.
So I perceived it as kind of calm, not very flashy. But to be fair, a banner at the top, definitely worth something. But then you scroll down to the posts, and it just looks like the typical crab bucket melee. It didn’t look like LessWrong is here wanting this book to be a success. We recognize the importance of this occasion. We wish Eliezer the best. This is Eliezer’s community. There were none of those vibes on the LessWrong homepage.
Holly 01:33:20
Yes, all too familiar.
Liron 01:33:27
There was a personal aspect to it. Did it damage my ego that I posted on LessWrong and I didn’t even break a hundred points? I mean, a little bit. But it’s not like I’ve never felt the pain of not getting a hundred points.
It’s more like on Eliezer’s behalf and on behalf of the movement - is there ever going to be a time when we can just stand up out of the bucket and be like, “Alright, world, this is our chance”? I thought this book was supposed to unite the movement.
Holly 01:33:57
I can’t say I’m surprised. I mean, because of what I went through, I’m a little surprised. I thought that they would be a little more excited for it, or at least that they would want to take credit for some of the fame of the book or something.
In the past, when there were the first few TikToks where somebody totally unrelated from this community talked about AI risk, everybody, all this group on Twitter would be like, “Oh wow, we’re getting big in the world.”
Holly 01:34:37
I have very detailed thoughts about it from having lived this for years now. It didn’t really surprise me that they would feel a desire to disclaim the book, because that’s just consistent with everything so far.
They’re really, really anxious about anything compromising their objectivity. Self compromises the objectivity of LessWrong all the time, but it doesn’t present itself obviously enough usually. They sort of start noticing it when it’s not really something that they want to go along with.
And I think this slight rift with Eliezer has maybe is gonna widen now because there’s something about it - mainly the mainstream flavor - that they don’t like. And then what the people who run LessWrong always say is that whatever they’re doing is about maintaining objectivity.
Holly 01:35:49
There was a lot of stuff about could I advertise protests on LessWrong? Because was that recruiting and that’s not allowed? They let me post there, but then Oliver Habryka, the head of Lightcone, would post under it something basically saying that political speech is deceptive. And I’m like, thanks, don’t let me post this here and then just malign me.
It doesn’t really - I’ve known about this for a long time. I don’t know, how consistently popular have you been? Because I imagine you would kind of split them. They’d kind of feel like they want to claim you if you might be popular.
Liron 01:36:27
I rarely break into the hundred plus. I’ve had a small number of hundred plus posts over the years. It’s tough for me to work in the LessWrong environment.
I think my problem is just that my writing style is oriented toward a pretty mainstream tech-leaning audience, and I just don’t do the hedging. I’m just like, “Hey, check out this claim. I’m confident about this claim,” and I don’t think LessWrong people like that approach.
Of course, it’s hard for me to analyze myself - why didn’t I get more points? Because my posts were objectively low quality? Okay, maybe that’s a factor too.
Holly 01:37:02
Eliezer many years ago in the Sequences has a post about why our kind can’t cooperate, which I think still rings true. I know you linked this in the discussion.
Why Our Kind Can’t Cooperate
Liron 01:37:10
We can go to “Why Our Kind Can’t Cooperate” because I think he absolutely nailed it. Let’s go to “Why Our Kind Can’t Cooperate” because I think that was on full display this week.
This was written in March 2009, right at the tail end of his original LessWrong Sequences, which he was writing 2007 to 2009, which is so impressive. He called so many things so early. “Why Our Kind Can’t Cooperate” - I think today it holds up so well.
Liron 01:37:39
So I’ll just read a couple quotes that I think really nail it. First, there’s the anecdote he said: “I remember our synagogue’s annual fundraising appeal. It was a simple enough format. The rabbi and the treasurer talked about the synagogue’s expenses and how vital this annual fundraiser was, and then the synagogue’s members called out their pledges from their seats. Straightforward? Yes.”
“Let me tell you about a different annual fundraising appeal, one that I ran.” And then he talks about how whenever he would ask for money, everybody would immediately start arguing about “Wait, is it really worth this much? Why don’t I donate to something like this?” Which is like, okay, yeah, you can have your argument, but where’s the show of support? You guys are Eliezer’s fans, right? You’re constantly talking about how great his institute is and the shared mission of AI, and the moment he is like, “So I’m open to donations,” and it was like, “Well, blah, blah, blah, blah.”
It’s like there’s a time and place for these things and there’s a time and place to send a signal that you’re mostly positive about something.
Holly 01:38:26
Yes. I mean, there’s so many - I’m fighting a tide of anecdotes and stories about my time with the rationalists about this. A big one that comes to mind is vegetarianism. They have never liked it. They’ve never liked that EAs are vegetarians. There are some rationalist vegans, but it’s less of a cultural thing.
When it comes time for a demand or some kind of show of, or even just sort of a suggestion that they’re supposed to do something that they don’t want to do, it becomes very disagreeable. That’s really what they all come together over - just being disagreeable about lots of things and being able to communicate in that way.
Holly 01:39:15
So I guess it doesn’t surprise me that it’s hard for them to come together. In fairness, I have been impressed with the kinds of things they can come together to do. They can come together to do social stuff, they can come together to do Secular Solstice, which I think is a beautiful ceremony. It’s been incredibly well produced in Berkeley in the last several years.
They can come together for pretty advanced conferences and for pretty advanced discussion stuff, forum stuff. They can really agree on those rules of engagement, but they don’t really have a consensus about how to act in the world, and it’s not really part of their philosophy.
I always found it weird that they were hostile to EA, but instead of just saying “I don’t believe that” or “I’m not gonna do it,” the reaction to protesting was similar. It’s not just like “I’m not gonna do it,” but that’s fine. It’s like, “No, in our community, we don’t have to do things. We only agree on these certain things.”
Liron 01:40:22
Just to give you one more quote from Eliezer’s post about why our kind can’t cooperate. The positive appeal - he’s saying: “I think there are parts of life where we should learn to applaud strong emotional language, eloquence and poetry, when there’s something that needs doing. Poetic appeals help get it done and therefore are themselves to be applauded.”
“We need to keep our efforts to expose counterproductive causes and unjustified appeals from stomping on tasks that genuinely need doing.” So like in this case, waking up the public.
“You need both sides of it. The willingness to turn away from counterproductive causes and the willingness to praise productive ones, the strength to be unswayed by ungrounded appeals and the strength to be swayed by grounded ones.”
Liron 01:41:02
So I feel like that’s the issue with the crab bucket or the circular firing squad - we’ve lost that gear where once in a while there’s an appeal so good that you want to signal, “Okay, we have mutual knowledge that this is a good appeal. We do leave the bucket and tell society something.”
Holly 01:41:15
Yeah, I think there is a mutual agreement. It’s not simply that the crabs pull down whichever crab tries to leave, but it’s a mutual agreement to stay in the bucket and that there’s not gonna be any pressure to put your money where your mouth is.
I think that’s kind of how rationalism evolved. It’s not always been the case with this community. Sometimes they were more active and they did things like found this field of AI safety.
But I think they became dedicated to being right and that’s it. And when you do stuff or when you take action because you think something matters, you often have to sacrifice some amount of being right. You have to put things on the line.
Holly 01:41:57
They’ll think that they’re brave about putting stuff on the line because they’ll do forecasting bets or something. But it’s too exposing to do activist stuff. It’s too exposing to possibly be taken for saying something stupid by a stupid person who doesn’t know how to understand what you’re talking about. I’ve just seen again and again how vulnerable that is for this group.
I also want to be right. I totally get it. It’s a very strong motivation for me. But I do feel much more - I have a much stronger motivation to save the world from this danger and protect the world. And I just thought, I don’t know, I thought they did too.
But I think if you’d have to be wrong at all or you’d have to give up any options, you’d have to commit to something you’re not totally sure about, then they’re like, no. And there’s always a reason not to do that.
Holly 01:42:56
There’s a couple versions of rationalism today that are articulated in this way. And there’s one strain which is popular with the Lightcone people especially, which is basically, because of nth-order effects of actions, you should have the highest possible bar for doing actions. And also there’s no action/inaction distinction - not acting is fine.
So basically that’s just like, you can just sit around and be right, and as long as you don’t deceive anyone or do anything dishonorable from your computer chair, you are good. And the minute you start to take actions, you’re opening yourself up to taking an action that has a harmful nth-order effect. So there are some articulations of this view in rationalism.
Liron 01:43:49
I noticed that Paul Crowley left a comment on this post from 17 years ago - time flies because I was actually there when Eliezer was publishing this stuff back in 2009, and Paul Crowley was there. We saw a tweet from him earlier. So he’s one of the OGs.
So he writes: “In this community, agreeing with a poster such as yourself [talking about Eliezer] signals me as sycophantic and weak-minded. Disagreement signals my independence and courage.” There’s also a sense that “there are leaders and followers in this world, and obviously just getting behind the program is no task for so great a mind as mine.”
So Paul Crowley really gets it all the way back in 2009. Funny enough, he got 71 votes at the time. I wonder if he’d get 71 votes on that today as the crab bucket has embedded itself in the culture.
Liron 01:44:41
Now getting back to the subject of this particular book launch. So remember I was triggered to notice how bad the situation is when I noticed that my own post didn’t get upvoted enough. Okay, so I’m biased, but then I went and looked - well wait a minute, what about MIRI’s book launch posts?
It’s got 270 points after a week. When I checked it last week, it was more like 230. But yeah, this is the official release day post on the book launch day by Alex Vermeer who works at MIRI. Okay, it got 270 points.
But if we scroll through LessWrong, what else is on there getting 200 points or similar? There’s this cool short story called “The Company Man” that was posted after the book launch that has 541 points and one of these curated stars. Or this post about “The Rise of Parasitic AI” also posted after the book launch - 537 points.
So this is what it looks like when LessWrong is really like, “Hell yeah, this is good content.”
Liron 01:45:35
Now here’s one that is a review of Clara Collier’s review, which we said a lot of people were dunking on, including us. So that’s cool that this one has 215 points. But on the other hand, why is the official post announcing the book in the same league as a post that’s reviewing somebody else’s review?
There’s basically no coordination to say, here we are in this community supporting the book launch. Why can’t the book have 541 points like these posts got in the last week? It’s not unusual for a post to get 537. If you combine the 200-something points that the official book launch post got and you add to that the 90 points that my interview got, we’re still not even approaching typical levels of stars here on LessWrong. Why did LessWrong just let this occasion go by?
Holly 01:46:18
I have suppositions you will think are uncharitable. I don’t think they - I think specific individuals in charge of LessWrong and just the community doesn’t want to feel compelled to take the actions in the book and they don’t want discussion to move that way.
I think they just feel scared at the idea of committing themselves or exposing themselves to a general proposition like that, or just getting in a realm that they don’t really understand or feel mastery of, like public conversation or working in politics.
Holly 01:47:17
You’ll find plenty of evidence of that around the site. They’ll say the reason that you can’t bring this up or the reason they’re all piling on is because you’re encouraging political thought or self-deception, and that could be very contagious. So they just can’t have that.
There’s a lot of, especially with the book reviews, a lot of comments said things like, “Why can’t we just have one place, one place where we can just say what we want?” And it’s like, you’re saying that we have to perform because of AI safety.
I’m kind of like, you can be disagreeable together as long as the day is long. You can do that, but don’t steal the topic of AI safety. That’s what annoys me. Just be disagreeable, fine. Have a place where you’re disagreeable, but don’t - you know that when you do that the message you send is that you don’t approve of the book or you don’t approve of activism to stop AI danger. Even though you say AI danger is so great.
I’m like either let go of the claim to working on this real topic and this real important thing or get with the program. If you want to be part of the solution, you have to be part of the solution.
Liron 01:48:18
Yeah, expanding on what you just said - get with the program of working on a real thing. This is actually one of Eliezer Yudkowsky’s core teachings back when he was laying down the rationality foundations on a little site called LessWrong.
One of the teachings is “Something to Protect.” That was the name of the post. The idea is that you don’t just randomly have these superpowers that let you reason really well, and you just work on them for their own sake. It’s all in the service of some other thing that you value. Rationality skills are a way to systematically go out into the world and live your life to obtain more of what you value or to give to the species more of what you value for their future.
To do something real. And the problem with LessWrong, the problem with the crab bucket is that they’ve kind of forgotten about the mission to get out of the bucket for certain occasions.
Holly 01:49:09
I completely agree. When you read the Sequences, that idea comes up many, many times in the Sequences, and Eliezer’s moved a lot. He’ll talk about how people don’t understand how he could cry thinking about the future or what’s at risk.
There’s many disquisitions talking about how it’s rational to feel emotions related to things you value. There’s so much, but if you were to just comment that and people didn’t recognize it from the Sequences on LessWrong, you’d get torn apart today. Because that’s like, “Well that could compromise your judgment and then you wouldn’t be right.” Or like, “Oh, you’re just doing this because you’re so sentimental about people’s lives or whatever.”
Creating a Statement of Support
Liron 01:49:56
Let’s go to the discussion now. Let’s see what people are saying. So right after I posted my interview with Eliezer and I saw the way that the front page of LessWrong was going where random posts about other topics and even posts saying that they didn’t love the book were just on this level playing field with this huge book launch announcement - the pinnacle of how MIRI and Eliezer Yudkowsky are trying to affect the world after 17 years of this community - that was just kind of getting buried or getting mixed with all these other opinions.
So then I posted on Twitter. I said: “LessWrong is a circular firing squad. My deep dive with founder of site and associated movement to promote his potentially world-saving book got 83 votes. Book’s ‘tentatively supportive’ review of said book questioning whether someone more reasonable could have written it got 152 votes.”
Liron 01:51:01
According to the two highest voted comments on my interview, I should have made it “less about that one particular person, Eliezer Yudkowsky.” That’s what the commenter was saying. I should make it less about Eliezer Yudkowsky and also that I made various viewers feel “icky” and it was too much “fawning.”
Yes, I’m openly a big fan of the guy responsible for this uniquely smart movement and community. Yes, I am a fan of Eliezer. Yes, the interview of Eliezer’s book launch was about Eliezer. Yes, I admit it.
Anyway, check out my interview with Eliezer Yudkowsky. It’s one of the most popular ones on YouTube. Because to me it was mind-blowing that you can post an almost official - it’s not an official Eliezer book launch interview, it was on my own personal channel, but it was a LessWronger on LessWrong, recapping the last 17 years on this very special day.
Liron 01:52:00
So for me, it was just mind-blowing that I can post this on YouTube and it gets 30,000 views. People have spent over 10,000 hours on YouTube taking in this interview with Eliezer, organic views. We didn’t even advertise it. And then you go on LessWrong, and it’s like, “Uh, yeah, 80-something points,” just gets buried down. It’s not even on the front page after a day. It’s like, what the hell?
So then Oliver Habryka, who’s one of the founders of the New LessWrong - Oliver did a really great job, I think around 2014, kind of bringing LessWrong back to life after the community was kind of dying. I’m a big fan of Oliver Habryka. I think he’s generally adding a ton of value and having great opinions and helping LessWrong be good. But obviously we’re disagreeing on this one particular issue.
Liron 01:52:44
So he comes in, he says: “For what it’s worth, I agree with that rough ranking. Buck’s review is great. Eliezer isn’t that great on podcasts, especially compared to reading his essays and not that interesting to LessWrong readers.”
So he is basically saying, look, you have this one post where you interviewed Eliezer, but LessWrong isn’t really about interviews, and Buck’s post is kind of like good LessWrong material that we want to discuss. So that’s fine. Which look, I understand his perspective. If you’re just asking the question of what does upvoting mean on LessWrong? And you define it to mean content that people on LessWrong want to read right now because it’s stimulating to them right now, then yes, they don’t necessarily need yet another interview with Eliezer.
Holly 01:53:37
And they could watch it right on LessWrong, right? It was embedded.
Liron 01:53:40
It’s embedded into LessWrong. The only other place that it was on is my personal YouTube channel. It’s not even on Doom Debates. Not only that, but there’s a full transcript on LessWrong. So it’s like this is fresh Eliezer content.
Nathan Young is saying, “Buck says he tentatively supports it. Is that really a circular firing squad?” So I clarified, I don’t mind Buck’s post getting votes. I’m pointing out that the voting pattern for Yudkowsky and high AI risk content and awareness raising efforts is much lower than expected, given the historical and presently perceived prominence of these two factors in the community.
Liron 01:54:36
This guy named Galileo wrote: “To be honest, this reads as a complaint about a cult not behaving like a cult.”
And then I’m like, “I’m sure that’s what many think as they smile in satisfaction that the post didn’t get that many votes.” They’re like smiling in satisfaction.
Holly 01:54:51
They get a little hit of like, yes, non-conformity really affirms their identity. Even if it is a form of conformity, something that feels like oppositionality - this is a very, very common personality trait of LessWrongers.
Liron 01:54:41
Exactly. Another interesting moment was a couple days after the book launch, and we had this discussion on LessWrong with all kinds of opinions being shared and not really much of a rallying effect, somebody named Remmelt made a post called “This is a Review of the Reviews.” It got 95 points. So notably a little bit more than my interview of Eliezer.
If you read how the post starts, he’s saying this is a review of the reviews. It’s a meta-review, if you will. But first a tangent and then a history list. So, and then he goes into like previously I rode a motorcycle. I rode it for about a year. And his point is like, okay, the motorcycle actually had a pretty high risk of death, but it’s lower than what people are saying.
And then he is also saying: “I also noticed mental gymnastics in people’s book reviews. And it made me disappointed and almost not wanting to say anything. I think it’s virtuous to note disagreements.”
Liron 01:55:35
End of the post he’s saying: “It worries me that people who allegedly care about the future going well and are also at least 5% concerned that AI is not going to go well are also squandering opportunities to help wake the world up to the dangers that they themselves are seeing, even if they see them slightly differently from the authors.”
So I completely agree, this is very similar to my crab bucket observation, but he’s getting 95 points on LessWrong and people aren’t having that same reaction of “if I upvote this, that means I’m conformist.” It’s more like, “Yeah, that’s a good observation because you’re arguing with people who are arguing with the book. I like that.” That’s how you have to package it.
Holly 01:56:02
Yeah, and the quote you read - the less readable it is, the less memeable it is, the more they like it. They can sense if something is too easy to get or too easy to like.
I have an inner sense of kind of what is gonna play with that crowd well or not, and it has to feel - my belief is that they’re very nervous about somebody coming away with the wrong impression. Not because they care about that person understanding really, but because they want to be understood and they’re really nervous that people are not really gonna understand them. They’re just gonna pattern-match them.
Holly 01:56:39
And so they have to make things difficult to access in the first place. And they have to make it so that you can’t - Scott Alexander is always flipping out because a journalist quoted him in a way that he doesn’t like and he considers to be such a violation. This is a long saga with Scott Alexander. And this is always the thing that they have with journalists.
I think they feel safe and they don’t like somebody coming into their conversation where they have that safety that there’s a guarantee that the other side is required to follow these rules and listen to them and they’re gonna be punished if they get it wrong. And they don’t like any other style of conversation, especially the kind that could possibly reach other people.
Holly 01:57:28
Be intruding into that space is my take on it. And so somebody can take your same content. It’s not really about the content. They can take your same content and if they put it in a like, “Yes, I’m gonna quantify all my uncertainty, I’m gonna put an epistemic status, I’m gonna have a long, unrelated story” - but that feels like it’s more data that’s being given to the reader to understand themselves in their own way. That kind of puts them at ease enough to take in a message.
Liron 01:58:02
Right. You have to wrap it up in these layers of packaging, and then they look at the post. It’s like, “Oh, I guess I’m ripping into these layers. Look at all this nuance. Look at all this wrapping paper strewn around me. I have now unpacked the nuance. Oh, and there is a little bit of a point here. There is a little bit of a takeaway that this book is an urgent warning to the world. Okay, I’m ready to receive it now.”
Holly 01:58:21
That’s like the love of parables. Definitely strikes me that way. I love Eliezer. I follow him on Twitter, but when I open up his long tweet and it’s just like, how much do I get to know these characters before I can understand your point?
But I kind of get it. This mode of discussion evolves again and again. And now I kind of think on LessWrong you kind of have to get people comfortable, have them distracted a little bit on a story to kind of be able to - they want it to be earned. You can’t just give your conclusion. It needs to be earned and it needs to feel like all of their thoughts kind of have to be somewhat addressed. Like, “Oh, but what percent? Surely not anybody who thinks there is a chance of doom because you could have a really small chance of doom.” They want all of that to be anticipated. Like, “Yes, this post is for you, the kind of person who thinks this way.”
Liron 01:59:35
Let me show you another bank shot that got LessWrong readers to upvote a pro-book post. The first bank shot was the guy I showed you who’s saying “I’m reviewing all the reviews and I’m sad that they’re not more pro the book.”
Here’s a post by Max Harms who actually works for MIRI. Smart guy. His post is titled “Contra Collier on I-A-B-I-E-D.” So remember Clara Collier? She wrote that review that seemed like a low-quality review. So here’s somebody from MIRI saying, “I disagree with this review. Let me explain why this review is bad.”
And again, a disagreement post about a disagreement review about the book - that bank shot is more interesting to people or gathering more upvotes than just a straightforward launch day interview with Eliezer Yudkowsky, original content from a LessWronger. No, no, no, that’s not interesting LessWrong content. A review of somebody else’s review criticizing that saying the book is bad is bad - that is how you do it.
Liron 02:00:19
So I said: “Ah, yes, this is the way to get LessWrong to upvote a post that supports Eliezer Yudkowsky’s latest and most important work. Step one: have someone else attack it with a bad argument (Clara Collier). Step two: attack their attack. Well done, Max Harms.” Seriously, and I meant it. I’m like, you did it. You cracked the formula.
And then Oliver Habryka - I don’t think that he fully understood that I meant it, that well done. He says: “I don’t understand this tweet. What is up with you being weirdly aggro at anyone getting upvoted? Obviously the key difference between this and your content is that it’s a completely different form factor and aimed at a completely different audience. And it’s not even super upvoted.”
Max Harms’ post had 116 upvotes. “It’s not even that upvoted, it’s like reasonably popular. Also, I don’t know, this tweet feels like it’s just contributing to bad polarization. Max’s post is pretty good. Why would you attack him for writing it? I am glad people are responding to critiques and Clara is definitely someone whose critiques are worth responding to.”
Liron 02:01:20
So then I said: “I think it’s sad in an ‘our community can’t cooperate’ way that LessWrong isn’t upvoting Eliezer’s book launch content. The large community of people who haven’t already become 250+ hour Sequences readers could benefit from a natural, timely entry point. I don’t have a problem with the other posts.”
So like some people have misunderstood that I’m like hating on Buck and Max Harms for writing these posts. It’s like, no, those posts are fine. I just wish that there was some kind of big community support thing. And I know there was a little banner. But I would’ve liked to see more and I would’ve liked to see it come more directly from an aggregation of the community. Not a hundred percent of the community - I would’ve been happy if like 60% of people engaging in the front page would show their support. Or I would’ve even been happy if like 40% showed their support and the other 60% are like, “Sorry, we’re not ready to fully support it.” And there was just like this very important 40% segment.
Holly 02:02:08
It’s just a missing mood. There’s nowhere -
Liron 02:02:10
The mood of support.
Holly 02:02:11
Kind of like, “Yes, I’m so excited for this book and what it’s gonna accomplish.” It just - there’s really nowhere. Even the MIRI book announcement on LessWrong, I’m sure it’s kind of like “We are modestly excited for our book.” That’s because that’s how you gotta do it.
And so yeah, it’s just a missing mood. It’s not that other people are getting love. It’s like there is a mood of like, “Oh yeah, let’s get into it, these criticisms of the book or whatever.” But there isn’t like, “Yeah, let’s savor that Eliezer is doing this new thing and this is so cool and I’m excited to tell my family or be the insider who has relative connections to this whole world.” That’s all clearly missing. And you’re pointing at one instance that you’re like, this is an example.
Community Reactions to the Statement
Liron 02:03:05
There’s another piece of this puzzle that I’m confused about. So it’s like I think we all have a good grasp of what the LessWrong old-timers - the people who get high karma themselves and write these long, nuanced posts and like when other people write long nuanced posts.
We have a good read on them and we have a good read on this idea of the general public and whether LessWrong can successfully signal to the general public. And the general public likes to respond to pretty crude straightforward signals. I think we’re on the same page there.
But what about the newer LessWrong users? Because it seems to me like the median user of LessWrong isn’t an old-timer. It stands to reason that it would be somebody who joined in the last few years, probably in 2023 when AI started getting much more attention. That’s probably where the curve averages out.
If you’re taking the integral of all the users, I’m guessing 2023 is like the median time to join LessWrong, and somebody who’s joined in 2023, they probably don’t have 250 hours of reading Eliezer the way a lot of these old-timers do. So if they’re new to LessWrong, they’re interested in Eliezer, they haven’t read that much Eliezer - isn’t this kind of post the kind of thing that we want them to be checking out?
Holly 02:04:10
I don’t know this level of detail about what kind of new users there are. It certainly doesn’t seem to me like they want broad usership from LessWrong. I think they would say at Lightcone that they’re focused on quality of the community first, and that might mean being kind of inhospitable to new users. I don’t know.
Liron 02:04:36
Right, so basically I saw a failure of signaling both internally and externally. And there’s only two ways to explain it. So number one is, okay, it’s a failure of signaling because the community’s divided. There’s not that much unanimous support for Eliezer’s book. That could be one explanation. I suspect that’s not the case.
I suspect there’s probably more than 50% support of karma-weighted users thinking that this book is making a positive impact and wish that it would get more eyes on it. That’s what I suspect. But that is one plausible explanation - if it’s like, nope, only the minority of the community supports it.
But the other explanation is like, nope, everybody supports it. There’s just a lack of mutual knowledge, which was, by the way, that’s the topic of Steven Pinker’s new book. He just released a book that’s all about mutual knowledge. So there is a funny connection that’s like, okay, well here on LessWrong, one of the most rational places, it seems like we’re passing up this golden opportunity, this Schelling point, which is another concept that Pinker touches on - this Schelling point right when the book launches to signal mutual knowledge so that we all know a lot of us, most of us know that most of us know that most of us know that this book is a big deal.
And also when the outside world takes a look, it’s like, “Ah yes, LessWrong thinks this is a big deal.”
Holly 02:05:56
Yeah, I think about - not obviously not this exact thing since the book just came out, but I think about this dynamic every day since 2023.
Liron 02:06:05
If the failure mode is not that the community itself is divided, if the community itself is united, then we’ve got a failure mode in the aggregation of the site. And it could be intentional, it could be, “Look, we do not want the site aggregating people’s opinions into this outside-the-crab-bucket signal. That’s not what we’re here to do.”
There’s different reasons why there could be a failure to aggregate people’s opinions. But the problem that I see is, if LessWrong is not going to aggregate people’s opinions, then who is? Because we don’t have another MIRI-pilled rational community. This is it.
Holly 02:06:56
I do believe that they would tell you how mutual knowledge works, and Schelling point is one of the most popular words in this community. So they know what it means. Which leads me to conclude they don’t want to.
I think they would say - they would’ve sort of agreed if you gave them this example like 10 years ago, they would’ve known that the answer would be to support it. But I think when it comes down to it, they’re just for personal reasons. It’s just, they’re too squeamish about it.
Holly 02:07:32
So you get kind of explanations that like, “Oh, you get just as like what we saw from Habryka, it’s like, well, in this case, this is explained by the form factor of these two posts.”
Liron 02:07:41
So to steelman Habryka, right, as I was doing before, it’s like if you’re just really committed on this invariant that votes on LessWrong mean what these voting readers want to read more of for the purpose of their own education and entertainment and information, then it makes sense that an Eliezer interview didn’t get voted.
But it just seems like there’s some - there’s an aggregation function missing. Voting is one type of aggregation. I’m just not seeing another aggregation where there’s no climb-out-of-the-crab-bucket aggregation.
Holly 02:08:06
Well, the closest thing I think is these assurance contract type things. It’s telling what that changes is that people don’t have to become early adopters and be visibly different until there’s like a sea change.
People tell me a lot of rationalists tell me that that’s what I should do with protests, for example. They shouldn’t have to join when it’s cringe and small. There should - the decision should already be made before they have to stick their neck out.
But I don’t even know if that’s the situation with LessWrong. I don’t know if anybody would really think it was wrong of LessWrong to just have Eliezer content featured or make a bigger deal about the book release or have an event where they solicit posts about “If Anyone Builds It, Everyone Dies,” or things that would be supportive, or if they had somehow rewarded actions that were supportive of the book or something like that. To my knowledge, they didn’t do any of that other than the banner.
Holly 02:09:05
But I don’t think anybody would’ve thought it was not their right. LessWrong - the team does themed content all the time. Not sponsored - they don’t do outside sponsors, but they do like, they are having a conference at Lighthaven. They will put that up on LessWrong. They don’t think that biases it.
So in this situation, I don’t know. I feel like just based on my knowledge of the people working there, that it was probably more of a in the moment it was like a squeamishness thing. Like, “Ugh, this feels too partial.” Or like, “I don’t like the way this looks,” or “This is kind of committing me to this idea. I don’t want that to be visible. I don’t like inviting sort of the outside world into my -”
Liron 02:09:45
Well, no, I mean, I actually think Oliver is correct, where the reason my Eliezer interview post didn’t get that many votes is because a lot of LessWrong readers had the same reaction as you did of like, “Oh, this is cool, but I’m not going to watch it and I’m not going to upvote something that I don’t personally want to spend my time watching.”
Holly 02:10:09
I mean, seems fair. It’s just, but as you’re saying, I mean, it’s like, isn’t that a little bit of a LessWrong viewership issue? And then I don’t know, I’m not saying they had to, but I mean, I’m just kind of curious. It seems like a book launch is a good opportunity to have events and accompanying stuff on a forum and they didn’t see fit.
Liron 02:10:27
Right. I think that in most communities they would kind of get the hint of like, “Oh, I should upvote this post because I support Eliezer Yudkowsky. I support ‘If Anyone Builds It, Everyone Dies.’ I think this is a critical moment. This is an important event.”
And so as kind of basic signaling 101, I am pro this post, so let me just throw it an upvote, even though I’m not even going to watch it. So I think that dynamic would’ve prevailed in many communities.
I think it’s fair to say, look, this is LessWrong. A vote is sacred. A vote must only mean content that I personally will think is worth my time to watch. If you want to maintain that invariant, fine. Okay. So I don’t have to make this personally like “Liron’s post needs votes.” I’m willing to look beyond that.
Keep my posts at 80 votes or whatever. That’s fine. I’m just noticing that there is just a lack of some sort of aggregation. So even if my post got low votes, I really wish the front page of LessWrong would just send a clear signal that there’s a high amount of support for the book.
Holly 02:11:16
I mean, yeah, I guess, yeah, I’d take that even beyond LessWrong per se. I wish these people really wanted this book to succeed and saw that as something important to do.
But it seems like when it comes to their image in the community that they care about, they don’t want to do that. They want to avoid that. They kind of make excuses for why they’re not doing that. And it makes me think that they don’t really put the actual world-saving aspect of their favorite thing to talk about very highly in their considerations.
Liron 02:11:38
There’s one more chapter to this saga, which is I tried to do a statement of support, and this is modeled after the famous Center for AI Safety Statement on AI risk, which I say often on my show. It says: “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks, such as pandemics and nuclear war.”
When I saw this statement come out with all the signatories, I was like, holy crap. This is a mutual knowledge breakthrough. This is a game changer.
Holly 02:12:07
Yes. I mean, I use the CAIS statement every week in communication. It’s a very quick shorthand to be like, “And everyone knows.” Everyone, people have agreed and it’s really - this strategy of getting people on the record of saying stuff, it’s amazing too.
Even without having to whip it out again and say “but you said” or “but a lot of people say” - it just really commits people to this. They might say that they believe a lot of things at any given time, but if you sign your name to this and you’re saying I can be held accountable to this later, it changes how you see it.
And with what we’re talking about with the missing mood on LessWrong and that whole community - I think they don’t want to sign a statement that says, “I was really into If Anyone Builds It, Everyone Dies.” I think they’re sort of wanting to maintain that option value. And I think like why? Why do you need to maintain that option value?
Liron 02:13:09
Yeah. So with the CAIS statement - for viewers who have been living under a rock, there’s a bunch of prominent people: Geoffrey Hinton, Yoshua Bengio, Demis Hassabis, Sam Altman, Dario Amodei, and it goes on from there. It’s really a broad swath of people.
So you can divide the timeline into before and after 2023. Before 2023, you could be like, “AI risk is this niche thing. It’s not taken seriously. Anybody who cares about it is a freak. It’s fake. They just want attention.”
And then after 2023, it’s like really look at the signatories. How do you have signatories like this if it’s true that nobody takes it seriously? It’s insane. I mean, people will still try to make the case, don’t get me wrong, but I just point to the signatories and I’m like, I will not discuss this further.
Holly 02:13:50
Yeah. Well, I mean true because often people nominally were arguing about the object-level claim, but really were arguing about the warrant to even take a position on the claim. And something like the CAIS statement just abolishes that argument. It’s like definitely we can.
Liron 02:14:12
So I saw a strong analogy because if you talk to me in early 2023, there would be a lot of people saying, “Look, of course, all these researchers take it seriously. Remember that time when this one guy said this, or he argued against somebody else that was anti-AI safety?” So people would always point to all these anecdotes and it was quite a mess.
Then after the statement it’s like, “Oh, okay. Wow, this has really risen in prominence. This is very significant.”
So similarly with Eliezer’s book and Nate Soares, people are saying, “Oh yeah, there’s all this discussion happening on LessWrong,” and people explicitly told me, they’re like, “Look, why would there be so much discussion around it if it wasn’t important?” I’m like, that’s your idea of signaling? That there’s a lot of discussion around it?
We just need a statement where people build mutual knowledge because the problem with not having mutual knowledge is okay, maybe I personally know that a bunch of people are into the book, but I don’t even know that other people know that those people are into the book.
Holly 02:15:00
People are not necessarily gonna say the same thing in two years that they said was obvious.
Liron 02:15:06
Right. They might change their mind. They might say different things to different people. I mean, mutual knowledge is a thing. Look at Steven Pinker’s latest book. The thing about these kind of public statements is that now everybody knows that everybody knows that everybody knows. Like there is this cascading effect. It is powerful for mutual knowledge to be a thing. I do think this falls into the blind spot of a community like ours.
Holly 02:15:27
I think there’s still ways in which they do - there is mutual knowledge about certain things that people in the community are really into, and I think they have withheld it from this book. And that’s a statement.
They don’t - there’s something about they don’t want it to be written that they really supported this. I think they’re unsure of how things are gonna go societally. I think they’re also unsure of how things are going to go. Are they gonna continue to identify with this community? If things - if what the community does is different, if it’s more public-facing -
Liron 02:16:04
Are they gonna be part of the post-singularity governance?
Holly 02:16:05
Yeah. Are they somehow gonna be excluded from still working on AI if they want to?
Liron 02:16:12
Yeah, so that’s one hypothesis, but I wasn’t so quick to conclude that they’re withholding their mutual knowledge because I figured, look, maybe this is just like the situation in 2023. If you look at all of these experts in the AI field, they weren’t exactly withholding their support for the field. It’s just nobody had organized them together into signing a statement.
So I took a stab at it. I went on LessWrong yesterday, and I posted “Statement of Support for If Anyone Builds It, Everyone Dies.” And of course I had to have the preamble. I wanted to increase my chances. So I didn’t want to go right into it. I wanted to explain the concept that I’m building mutual knowledge.
My Statement of Support Attempt
Liron 02:16:43
I said: “The purpose of this post is to build mutual knowledge that many, most of us, on LessWrong support If Anyone Builds It, Everyone Dies. Inside of LessWrong, not every user is a long-timer who’s already seen consistent signals of support for these kind of claims. A post like this could make the difference in strengthening versus weakening the perception of how much everyone knows that everyone knows that everyone supports the book.”
“Externally, people who wonder how seriously the book is being taken may check LessWrong and look for an indicator of how much support the book has from the community that Eliezer Yudkowsky originally founded.”
Liron 02:17:16
“The LessWrong front page where high-voted posts are generally based on whether users want to see more of a kind of content wouldn’t by default map a large amount of internal support for I-A-B-I-E-D into a front page that signals support. What seems to emerge from the current system is an active discussion of various aspects of the book, including well-written criticisms, disagreements, and nitpicks.”
“So here is the statement of support: I support If Anyone Builds It, Everyone Dies. That is, I think the book’s thesis is likely or at least all too plausibly right - that building an artificial superintelligence using anything remotely like current techniques based on anything remotely like the present understanding of AI will cause human extinction.”
“I think the world where the book becomes an extremely popular bestseller is much better on expectation than the world where it doesn’t. I generally respect MIRI’s work and consider it underappreciated and underrated.”
Liron 02:18:00
And then I have a little section at the end where I say similarity to the Center for AI Safety Statement on AI risk, where I explained what we just went over. The famous 2023 Center for AI Safety Statement on AI risk reads: “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”
“I’m extremely happy that this statement exists and has so many prominent signatories. While many people considered it too obvious and trivial to need stating, many others who weren’t following the situation closely or are motivated to think otherwise had assumed there wasn’t this level of consensus on the content of the statement across academia and industry.”
“Notably, the statement wasn’t a total consensus that everyone signed or that everyone who signed agreed with passionately. Yet it still documented a meaningfully widespread consensus and was a hugely valuable exercise. I think LessWrong might benefit from having a similar kind of mutual knowledge building statement on this occasion.”
So that was my post. Alright, what did the LessWrong community say? Any guesses?
Holly 02:18:55
Well, I already saw.
Liron 02:18:58
All right. We are currently standing at 24 points, but the funny thing is it’s 24 points -
Holly 02:19:03
39 votes.
Liron 02:19:04
with 39 upvotes.
Holly 02:19:05
I both strong upvoted it, so.
Liron 02:19:08
You strong upvoted. Yeah, exactly. That’s pretty funny. I mean, if you compare it to like a random post on the front page, this one that has 83 points, this one has 83 points based on 30 votes. So I have more votes than this post that has 83 points. But you can tell there are probably people strong upvoting and strong downvoting. So this would rank as a controversial post.
Let’s be fair though. Okay. It’s controversial because people are confused about whether this kind of post is something they want to see again. So they’re not necessarily saying, “Eliezer’s book sucks.” Okay? So it’s not necessarily, that’s not what it represents.
Holly 02:19:41
What’s wrong with this post? It just like -
Liron 02:19:44
Yeah, so maybe the comments will tell us what’s wrong with this post. So Buck has the highest voted comment that relatively a lot of people agree with. I guess about 10 people agree with it, roughly.
He’s saying - first he was saying that my original - I had a different phrasing of my statement of support. Originally, I used the language, “the book’s thesis is basically right, that if anyone builds super intelligent AI in the next decade or two, it’ll have a terrifyingly high, let’s say 15%+ chance of causing everyone to die in short order.”
Holly 02:20:11
You leave a hook out like that. Yep.
Liron 02:20:12
Yeah, I love to hook out. So this, and I did rephrase it after Buck’s comment, but he said: “I think this is an absurd statement of the book’s thesis. The book is plainly saying something much stronger than that. How did you pick 15% rather than, for example, 80 or 95%?”
And look, it’s a fair point that I picked 15. The book was making much stronger statement. I mean, my intention with my statement is to be like, if you think the correct value is 15%, you should probably still be a supporter of the book. So I was kind of trying to draw a boundary there. But anyway, I agree that he’s at least right to criticize that this is probably not an ideal statement. So I did rephrase it.
Liron 02:20:43
So I told him, look, I’ve updated it right now. I’m just saying I’ve used the book’s own language: building artificial superintelligence using anything remotely like current techniques, based on anything remotely like the present understanding of AI will cause human extinction. So I changed the language.
And then what else was Buck saying? Here he is saying, “I’m basically on the fence about the statement you have made.” For reasons described here in this other post by somebody who works with him. He says: “I think P of human extinction given AI takeover is likely 30%. And I separately think P of AI takeover is like 45%.” Okay. So he is on the fence.
But then he’s saying that it’s probably right about what I said about the world where the book becomes an extremely popular bestseller is much better in expectation. He says it’s probably right, but it’s unclear. Okay.
Liron 02:21:22
And then finally with the thing about, I said about I respect MIRI’s work. I mean, maybe I should have just taken that out. It’s like, because this is bringing in additional assumptions. It would’ve been simpler for me to take out. I generally respect MIRI’s work and consider it underappreciated and underrated.
But to me, when I wrote it, I was just thinking, isn’t this implied by supporting the book? That MIRI’s buildup to this point has been very important and underrated?
Holly 02:21:41
This is Buck’s world professionally. So it’s like -
Liron 02:21:45
Right. It’s Buck’s -
Holly 02:21:46
He thinks about all day is in this map of ideas and MIRI’s over here and he’s over here. But like -
Liron 02:21:54
Right. So to be fair, he says it depends what you mean by work. “I think their work making AI risk arguments at the level of detail represented by the book is massively underappreciated and underrated. Eliezer’s work on making a rationalist and AI safety community is very underrated. I think that their technical work is overrated by LessWrong readers.”
So, you know what? I don’t actually want to nitpick Buck. I actually think Buck has a fine comment. I don’t think that it undermines the post. I think it’s just like a fine, almost tangential comment.
Holly 02:22:20
I guess, I mean, I’m still kind of thinking like, okay, but if you really cared about getting this message out or being clearly understood on where you fall in this book and you at all were taking into account the costs of people not understanding that you support it to the level that you do, instead of only taking into account the costs of people thinking you agree with it where you don’t, then I think you wouldn’t be equivocal.
I don’t know, maybe he’d be like, “I can’t do this because professionally it might confuse people about my professional positions or something.” But even so, I mean, I just - just because you can have a question about it or like - it’s just like when you choose to bring up objections, all very indicative of how you’re weighing the costs and benefits underlying.
Holly 02:23:16
And it is a common LessWrong rationalist thing to treat like no nitpick is too small. There’s not real - and it’s almost considered bad thinking to weigh the thrust of arguments such that you would overlook a nitpick. But I think that they wouldn’t do it if they really, really valued the aim. And I think they wouldn’t do it in their - I don’t know.
I think it’s - Eliezer puts it this way - but the cognition you use to go pick up milk at the store, they’re not using that. They’re using social cognition to make this decision, and Eliezer would say that was wrong.
Liron 02:23:50
Yep, yep. Alright, so let me just read you the second highest voted comment. It’s 22 votes and this is kind of interesting. It has 13 votes. So slightly controversial I guess. So it’s a guy named Garrett Baker. And I think this comment is actually representative of the community reaction. This comment probably gives you a good sense in my opinion of why people weren’t loving this post. That’s just my guess.
So the comment says: “I don’t think LessWrong is the right venue for this, or really, I don’t want it to become that venue. I basically agree with Steven Byrnes here but generalized to also include rallying others to agree to stuff.”
“I think it’s reasonable to make a post arguing that people should publicly declare their stances on such things.” So there was actually a pretty popular post that even quoted one of my discussions in Doom Debates. This was a few days ago. There’s a pretty popular post saying safety researchers should take a public stance.
Liron 02:24:38
So it’s kind of funny, the community is like, “Oh, somebody is saying that people should take a public stance on their beliefs. Yeah, great idea. Yeah, let’s upvote a post talking about the general concept of that.” But then when the actual object-level opportunity comes up, “Statement of Support for If Anyone Builds It, Everyone Dies,” it’s like, “Ah, ah, you know, I can’t, I can’t support this.”
Holly 02:24:56
We’re closed.
Liron 02:24:57
So Garrett Baker is continuing: “I think the sort of thing skirts too close to groupthink. Further, I don’t know what it means to ‘support’ a book. Does the book need help? How can that be? It’s a book. See also ‘Ethnic Tension’ and ‘Meaningless Arguments’ in particular.”
So he’s quoting Scott Alexander from a different post saying: “A question: are you pro-Israel or pro-Palestine? Take a second. Actually think about it. Some people probably answered pro-Israel, other people probably answered pro-Palestine. Probably very few people answered, ‘Huh? What?’”
And he is just talking about how arguments can be meaningless. But then when I clarified that later, it turned out that all he meant was that it was meaningless when I said “I support If Anyone Builds It, Everyone Dies.” But when I pointed out that I was just using it as a shorthand for what I wrote right below it, which was - I explained exactly what I meant by that - he was like, “Yeah, of course the long thing you wrote is meaningful, but ‘I support If Anyone Builds It, Everyone Dies’ is meaningless.” And I’m like, okay, well it’s a shorthand.
Holly 02:25:51
I mean, I like Garrett. I don’t want to be too hard on him, but is he really saying a book is an object? What is it? It doesn’t need support?
Liron 02:26:00
Yeah, I mean, look, I thought it was weakly argued. And the only reason I’m even reading it is just because I don’t see another post in this entire comment section, which to be fair, only has a handful of comments in it. I just don’t see any other post that explains the deal better than this post.
And then he continues. “I know you clarify what you mean by support later. And of course, you’re committing a much smaller sin than the religious person in the example. But if someone said the statement of support you outline, what they’re literally saying is meaningless. The statement can only be understood in the context of a political game they’re playing with the rest of society.”
And then I think this last statement is significant: “Now that political game may be necessary and useful, but I’d rather LessWrong not be a battlefield for such games and remain the place that everyone can go think, not fight.”
Liron 02:26:46
So then I reply to that, this idea of think not fight. I said: “Aggregating people’s existing belief states into mutual knowledge - that seems like good thinking to me. Part of the infrastructure that makes thinking more effective at climbing the tree of knowledge.”
So I just think that when people say, “Okay, yeah, we do support this, this is actually a piece of knowledge we think we have” - to me, that’s part of thinking, not fighting.
Holly 02:27:07
He seems to have already decided it’s political. So like, I see where you’re going. If you know what we think, then it’ll become political. But the thing that kills me about LessWrong is the idea that it’s not already political.
I do believe - I think the core LessWrong users tend to be very deficient in understanding frames. They just don’t - they’re not very good at noticing them. If they get the sense that maybe there is some kind of manipulative frame, they react really strongly and they don’t really know how to engage with it.
It’s a thing that there’s a lot of discourse on LessWrong about this mysterious stuff, frame control. And they believe that there’s such a thing as entering political thinking. Like they are not doing it. They believe that they’re not doing it and they think that they’re avoiding it by staying off of certain topics and by not talking in a certain way.
Holly 02:28:12
And they think they’re entering political thinking when they are agreeing too much. That is one thing that kind of makes them think that’s happening. And they think that trying to accomplish something often is political thinking, if that thing is not science.
But they don’t realize how political they are all the time. Their identity as LessWrongers, their belief that this is where we think - that is all political thinking in the same way, but they’re very insensitive to it.
It’s not perfectly overlapping with, but it is very close to the amount of stuff they just don’t want to do anyway or feel uncomfortable with. Or they think somebody, some other group is gonna be better than them or win the argument if they do it that way.
Holly 02:29:00
But a lot of it is just, I think they begin to think like, “Oh no, I like this person” or “They’re attractive, and so I might be inclined to agree with them for reasons that are not true.” That pings them as political or halo effect kind of thinking.
But they never think to themselves like, “I’m tempted to just disagree and nitpick and tear this apart.” They think like, “Yes, that’s good. That reassures them that they are doing the right thing and they’re not doing political thinking.”
But that can absolutely be political thinking. Absolutely. I mean, it can be bad in the same way. And I think that Garrett’s comment was displaying that. As long as we’re not agreeing, as long as we’re not rah rah rah, that means that we’re thinking without this influence. Whereas I think - why do you feel the need to tear it apart? Why does it feel wrong to -
Liron 02:29:37
Right, exactly. Which is why Eliezer back in 2009, he does do a good job of anticipating so much. He’s like, “By the way, you don’t want to get into this mode where you just can’t share your support for stuff.” So I think he called it.
Alright, so to end the saga here, this is really where things stand right now. There was a tiny little movement on Twitter when I was tweeting about how I posted this statement of support. It picked up a tiny bit of steam where a few people have retweeted it and they’re like, “I support If Anyone Builds It, Everyone Dies.”
So there was a little bit of standing up in support, but it was unfortunately the farthest thing from going viral. And similarly on LessWrong, we basically saw the extent of people’s reactions to it. It’s like, okay, yeah, I got some controversial up and down voting and I think it’s done. I just don’t think we’re going to see a bigger push from the community.
I think we’re seeing some interest from the larger world, but just - I think we’re kind of seeing the shape of where this is going. It just kind of - it was a bullet that got fired. It attacked the situation a little bit, but it seems to kind of now be - the momentum seems to be dying down. Is that your perception?
Final Reflections and Calls to Action
Holly 02:30:35
Well, I think it’ll - so for our events, we’re planning - we’re doing discussion events that are happening on October 4th, 5 to 8 PM in San Francisco. But also there’s gonna be a New York event, October 12th. And others, you should check out our website, which is pauseai.us.org. And /events will take you right there.
But another - it’s not just like, did this get the buzz that it should? Or did this reach as many people or should did it get as high on the bestseller list as it should have or whatever. It’s also - Nate and Eliezer, they haven’t indicated to me that they feel demoralized at all by the reception, but they’re dealing with it on Twitter. They’re responding to people.
Liron 02:31:26
I will say this, a couple people from MIRI did let me know that they’ve known about the circular firing squad issue for a long time, and they enjoyed me documenting it, but it’s not news to them at all.
Holly 02:31:36
Well, yeah. And basically that’s what I was told is like, “Oh, we expected a lot of this.” But I know that it’s a gut punch and we need them right now to be doing these interviews and to be promoting this book. And the book tour was a big part of the reason to do it.
And I feel like that - even if the community doesn’t think about these things that consciously, they know when they’re kind of giving their supportive energy or not.
Liron 02:32:01
Right, right. You know, I didn’t mention this part, but it’s a really good rule of thumb when you’re trying to get out there with a message: others aren’t going to be more passionate than you are.
And in the case of LessWrong, it’s the collective view. I mean, like it or not, even if you say, “Oh, LessWrong is something else, it’s this impartial discussion” - I’m sorry. LessWrong is by far the closest thing to the MIRI community.
And if LessWrong isn’t showing gusto at this launch, but they’re hoping - I believe the majority of people, even though there’s no mutual knowledge, I believe the majority of people are secretly hoping that the world embraces this book, even while this community isn’t modeling what it looks like to embrace this book.
And that certainly breaks my rule of thumb, which is that you can’t expect the larger world to get more passionate about something than you are.
Holly 02:32:47
I agree. I think a lot of these people would love to be able to be like, “Well, I was a slightly more sophisticated kind of AI safety person,” but it’s not going to get big if they don’t lend themselves to that.
Which is one of the things I complained about. One of - I don’t have many complaints about this book, but one complaint was that I felt that it didn’t properly valorize and just discuss the necessity of standing alone if you need to. You don’t get from zero to one without doing that. You need people who are willing to become part of the critical mass, which means people who joined before there is a critical mass. You need that, that’s really critical.
Holly 02:33:33
And we just absolutely do not have that from the LessWrong community. And I think what we’re talking about right now is people not willing to lend themselves to that critical mass, kind of holding back, waiting for things to kind of work themselves out or be more of a sure thing before they stake their reputation on it.
And yeah, we just need bravery to actually change things on this issue. And it’s crazy that we can have such high level of understanding of the issue and a high level of agreement about the issue. And for there to still be that level of disconnect between how you act - not even asking that much, just how do you offer your support? But I guess it is asking a lot. It’s asking a kind of reputation risking. It’s asking to enter a different mode than LessWrong’s been in where you’re not just nitpicking all the time, but you’re kind of trying to work together on something.
Holly 02:34:20
I’m shocked first of all that that’s - I don’t know, obviously I’m not like this. I’m what they call in the social change literature a “zero,” which is kind of a funny word, but I’m a zero because I don’t require anybody else around me to have adopted the change in order to do it.
So the people around - I activated a bunch of ones around me and then they can activate twos. The fact that there are ones and zeros activates twos.
Liron 02:35:01
Yeah.
Holly 02:35:02
Who needs this? I can’t think -
Liron 02:35:05
But I mean, but seriously, right. The way you influenced me, that’s a nice backhanded compliment, I guess.
Holly 02:35:10
I know it’s hilarious. But I had a sociologist write this about me and I thought it was so funny to read. He called me a “rational zero.”
Liron 02:35:20
Yeah, yeah, yeah, exactly.
Holly 02:35:21
So yeah, obviously I’m kind of weird in this way. But it is amazing to me that this is even a question. Like yeah, okay, maybe there’s some cost to it. Maybe you have to put something on the line, but how you could not do that?
And I think most people in this circular firing squad, if like 20% of people were zeros on this, and they were just like, “Of course I’ll support this,” then a lot of people would fall into line and a lot of people’s true feelings about what to do form a bell curve, I’m sure. If they had that 20% base, then they would fall into line too.
But they just consider it this big deal to be expected to do anything in that uncertain position. And I think Eliezer is kind of this way too. And I think the book kind of shows that with the assurance contract march and stuff. It’s not saying just go out and do protests. It’s saying if we all get the security that everybody else will show up, then we will do it.
Holly 02:36:20
I really do hope that works. I’ve signed up as I said. And I wouldn’t begrudge them at all. I’d just be thrilled if that worked. I do feel like it’s gonna be pretty crucial that we directly deal with this tough moral - which apparently a tough moral issue for a lot of people - of putting yourself on the line and actually being part of making that critical mass rather than just saying “if they build it, I will come.”
Liron 02:36:44
Right, right. Okay. Yeah, good points. And I’ll let you promote some of your events. I think that’s well deserved.
Just to recap here, there is some reason to be positive here, which is, like I said before, huge fan of MIRI. I’m overall satisfied with their performance. I don’t think the world needed MIRI’s performance to be better. I think they did great.
LessWrong - I’m kind of disappointed, but they certainly could have done worse. And overall, just like I’m a MIRI fan, I’m a big LessWrong fan. I think generally LessWrong is actually underrated. It’s kind of fun for people to dunk on LessWrong. I think they don’t respect LessWrong enough.
I just think in this episode we focused on something that is a significant weakness of LessWrong that I hope we do better with in the future. Because it’s probably the single biggest weakness and it really counted this week. And I feel like we didn’t fully come through, but we still did okay. We didn’t do that bad.
Liron 02:37:30
And the truth is that if LessWrong was super supportive about the book, would that mean that Donald Trump would be holding up a copy saying everybody has to read it? Would it make that big of an impact? No. Probably not.
So at the end of the day, we gotta also have some perspective and be like, LessWrong is still pretty awesome. MIRI is definitely pretty awesome. So I don’t want to make this into the biggest issue ever, but I just think it’s an issue.
Holly 02:37:53
For me, this issue this week is illustrative of all the weeks previous for the last two years. So I think that this is one of the bigger spiritual issues that we can deal with inside amongst people who already care and put significant resources toward AI safety.
Dealing with this reluctance to go out on a limb to form consensus is one of the biggest things that we can do. So it’s not the same - it’s not like if we just get our act together that suddenly Donald Trump starts writing a peace accord with Xi Jinping. But it’s a big lever that we have.
And it’s something that even though I try to focus on just the public and new people knowing more about this issue, I go back and back to the community and the insiders. And I want - I feel like I can help them and I know they have it in them to get stronger on this. And I think that would really multiply our impact a lot if we could all have a stronger base and we could work together more effectively.
Liron 02:39:03
Great. So if you’re listening to this and you support If Anyone Builds It, Everyone Dies, my call to action for you is go on your social media of choice, whether that’s Twitter, LessWrong, or whatever other platform I’m told exists. Post something like “I support If Anyone Builds It, Everyone Dies.” I do actually think it helps. I think people notice when this is happening repeatedly.
And yeah, Holly has another call to action that I also think you should listen to.
Holly 02:39:27
Well, my general call to action is we need to pause AI indefinitely, globally, until we know that it’s safe. If you’re interested in that, you can pursue that at the international version is pauseai.info and you can find your country, figure out where to go from there.
The US version if you’re in the US is pauseai.us.org. And you can see if there’s a local group nearby you. You could always express your interest in starting a local group, and you can get involved in our general take action - check out our take action page.
And then we’ve got an event for this book. The first of the series is gonna be in San Francisco, and it’s gonna be October 4th, 5 to 8 PM at Manny’s in the Mission. And there’s going to be a large event in New York as well that involves a march, and that’s gonna be on the 12th.
Holly 02:40:17
You should check out our website, our events page, to make sure you have the details for that. And we also have other events throughout the country and also throughout Europe and Australia if you want to check out Pause AI’s event pages.
So we would love to have you at Manny’s. I’ll be there at Manny’s, 5 to 8 on October 4th to discuss If Anyone Builds It, Everyone Dies. I’d love to see you there.
Liron 02:40:48
Great. And yeah, so for people who just want to do stuff online, like join the Discord or whatever, just because the logistics are complicated, they just go to pauseai.us.org and join the online community. Correct?
Holly 02:40:59
Yes. So we’ve got a community page and we have an events page where you can at a glance see your options. And if you’re not American, you go to pauseai.info and you can sort of disambiguate from there where you should be, where your local group is.
Liron 02:41:13
All right, Holly, I think this was a nice cathartic session where we let out some of our pent-up thinking over the week. Is that fair to say?
Holly 02:41:22
Yes, yes. And this is an indulgent kind of conversation for me, but I need to get it out.
Liron 02:41:28
Yeah, exactly. All right. So maybe we’ll let the next round build up and we’ll circle back in a couple months and this can be a regular thing.
Holly 02:41:35
Yeah, I’m sure there’ll be something.
Liron 02:41:37
Awesome. All right, well thanks very much for coming on.
Holly 02:41:40
Thanks, Liron.
Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.
Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏
